Re: Emitting JSON to file using COPY TO
Andrew Dunstan <andrew@dunslane.net>
From: Andrew Dunstan <andrew@dunslane.net>
To: jian he <jian.universality@gmail.com>
Cc: Joe Conway <mail@joeconway.com>, Junwang Zhao <zhjwpku@gmail.com>,
Florents Tselai <florents.tselai@gmail.com>,
"Andrey M. Borodin" <x4mmm@yandex-team.ru>,
Dean Rasheed <dean.a.rasheed@gmail.com>,
Daniel Verite <daniel@manitou-mail.org>, Davin Shearer <davin@apache.org>,
PostgreSQL-development <pgsql-hackers@postgresql.org>
Date: 2026-03-06T12:33:31Z
Lists: pgsql-hackers, pgsql-general
Attachments
- v27-0001-introduce-CopyFormat-refactor-CopyFormatOptions.patch (text/x-patch) patch v27-0001
- v27-0002-json-format-for-COPY-TO.patch (text/x-patch) patch v27-0002
- v27-0003-Add-option-force_array-for-COPY-JSON-FORMAT.patch (text/x-patch) patch v27-0003
- v27-0004-COPY-TO-JSON-support-column-lists.patch (text/x-patch) patch v27-0004
On 2026-03-06 Fr 4:38 AM, jian he wrote:
> COPY (SELECT 1 UNION ALL SELECT 2) TO stdout WITH (format json);
>
> still fails with v25-0002, json_tupledesc_ready is not helpful.
> I think I figured it out. We need to use BlessTupleDesc in BeginCopyTo.
> Then let slot->tts_tupleDescriptor point to cstate->queryDesc->tupDesc
> in CopyToJsonOneRow
>
> * CSV, text and json formats share the same TextLike routines except for the
> * one-row callback.
> This comment is not useful, I want to delete it.
>
> CopyToTextLikeStart
> + /* JSON-specific initialization */
> + if (cstate->opts.format == COPY_FORMAT_JSON)
> + {
> + MemoryContext oldcxt;
> +
> + /* Allocate reusable JSON output buffer in long-lived context */
> + oldcxt = MemoryContextSwitchTo(cstate->copycontext);
> + initStringInfo(&cstate->json_buf);
> + MemoryContextSwitchTo(oldcxt);
> + }
> We ca just add
>
> cstate->json_buf = makeStringInfo();
>
> in BeginCopyTo.
>
> v25-0004-COPY-TO-JSON-build-JSON-per-column-support-colum.patch
> added several fields to the CopyToStateData.
> Actually, there is a simpler way (construct a new Tupdesc and let
> composite_to_json do the job), please see my v26-0004.
>
Yeah, I got some crashes using this, but reworked it a bit, and all now
seems good. I was apparently mistaken about the supposed speedup, but
the performance is steady.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
Commits
-
Add option force_array for COPY JSON FORMAT
- 4c0390ac53b7 19 (unreleased) landed
-
json format for COPY TO
- 7dadd38cda95 19 (unreleased) landed
-
introduce CopyFormat, refactor CopyFormatOptions
- a2145605ee3d 19 (unreleased) landed
-
Doc: add IDs to copy.sgml's <varlistentry> and <refsect1>
- e4018f891dec 19 (unreleased) cited
-
Refactor COPY TO to use format callback functions.
- 2e4127b6d2d8 18.0 cited