Re: Emitting JSON to file using COPY TO

Nathan Bossart <nathandbossart@gmail.com>

From: Nathan Bossart <nathandbossart@gmail.com>

To: Tom Lane <tgl@sss.pgh.pa.us>

Cc: Andrew Dunstan <andrew@dunslane.net>, Joe Conway <mail@joeconway.com>, Davin Shearer <davin@apache.org>, PostgreSQL-development <pgsql-hackers@postgresql.org>

Date: 2023-12-06T16:33:49Z

Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

Add option force_array for COPY JSON FORMAT
- 4c0390ac53b7 19 (unreleased) landed
json format for COPY TO
- 7dadd38cda95 19 (unreleased) landed
introduce CopyFormat, refactor CopyFormatOptions
- a2145605ee3d 19 (unreleased) landed
Doc: add IDs to copy.sgml's <varlistentry> and <refsect1>
- e4018f891dec 19 (unreleased) cited

On Wed, Dec 06, 2023 at 11:28:59AM -0500, Tom Lane wrote:
> It might be acceptable to plan on improving the performance later,
> depending on just how bad it is now.

On 10M rows with 11 integers each, I'm seeing the following:

	(format text)
	Time: 10056.311 ms (00:10.056)
	Time: 8789.331 ms (00:08.789)
	Time: 8755.070 ms (00:08.755)

	(format csv)
	Time: 12295.480 ms (00:12.295)
	Time: 12311.059 ms (00:12.311)
	Time: 12305.469 ms (00:12.305)

	(format json)
	Time: 24568.621 ms (00:24.569)
	Time: 23756.234 ms (00:23.756)
	Time: 24265.730 ms (00:24.266)

'perf top' tends to look a bit like this:

  13.31%  postgres                      [.] appendStringInfoString
   7.57%  postgres                      [.] datum_to_json_internal
   6.82%  postgres                      [.] SearchCatCache1
   5.35%  [kernel]                      [k] intel_gpio_irq
   3.57%  postgres                      [.] composite_to_json
   3.31%  postgres                      [.] IsValidJsonNumber

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com