Re: Emitting JSON to file using COPY TO
jian he <jian.universality@gmail.com>
From: jian he <jian.universality@gmail.com>
To: Joe Conway <mail@joeconway.com>
Cc: "Andrey M. Borodin" <x4mmm@yandex-team.ru>,
Dean Rasheed <dean.a.rasheed@gmail.com>, Daniel Verite <daniel@manitou-mail.org>,
Andrew Dunstan <andrew@dunslane.net>, Davin Shearer <davin@apache.org>,
PostgreSQL-development <pgsql-hackers@postgresql.org>
Date: 2024-09-13T14:42:00Z
Lists: pgsql-hackers
Commits
Same data as JSON:
GET /api/v1/messages/:b64id/commits
the thread's linked commits as JSON, with link sources.
API reference →
-
Add option force_array for COPY JSON FORMAT
- 4c0390ac53b7 19 (unreleased) landed
-
json format for COPY TO
- 7dadd38cda95 19 (unreleased) landed
-
introduce CopyFormat, refactor CopyFormatOptions
- a2145605ee3d 19 (unreleased) landed
-
Doc: add IDs to copy.sgml's <varlistentry> and <refsect1>
- e4018f891dec 19 (unreleased) cited
Attachments
- scratch31.sql (application/sql)
Hi.
in ExecutePlan
we have:
for (;;)
{
ResetPerTupleExprContext(estate);
slot = ExecProcNode(planstate);
if (!TupIsNull(slot))
{
if((slot != NULL) && (slot->tts_tupleDescriptor != NULL)
&& (slot->tts_tupleDescriptor->natts > 0)
&& (slot->tts_tupleDescriptor->attrs->attname.data[0] == '\0'))
elog(INFO, "%s:%d %s this slot first attribute attname is
null", __FILE_NAME__, __LINE__, __func__);
}
if (TupIsNull(slot))
break;
if (sendTuples)
{
if (!dest->receiveSlot(slot, dest))
break;
}
dest->receiveSlot(slot, dest) is responsible for sending values to destination,
for COPY TO it will call copy_dest_receive, CopyOneRowTo.
For the copy to format json, we need to make sure
in "dest->receiveSlot(slot, dest))", the slot->tts_tupleDescriptor has
proper information.
because we *use* slot->tts_tupleDescriptor->attrs->attname as the json key.
For example, if (slot->tts_tupleDescriptor->attrs->attname.data[0] == '\0')
then output json may look like: {"":12}
which is not what we want.
in ExecutePlan i use
elog(INFO, "%s:%d %s this slot first attribute attname is null",
__FILE_NAME__, __LINE__, __func__);
to find sql queries that attribute name is not good.
based on that, i found out many COPY TO (FORMAT JSON) queries will either
error out or the output json key be empty string
if in CopyOneRowTo we didn't copy the cstate->queryDesc->tupDesc
to the slot->tts_tupleDescriptor
You can test it yourself.
first `git am v12-0001-introduce-json-format-for-COPY-TO.patch`
after that, comment out the memcpy call in CopyOneRowTo, just like the
following:
if(!cstate->rel)
{
// memcpy(TupleDescAttr(slot->tts_tupleDescriptor, 0),
// TupleDescAttr(cstate->queryDesc->tupDesc, 0),
// cstate->queryDesc->tupDesc->natts *
sizeof(FormData_pg_attribute));
build and test with the attached script.
you will see COPY TO FORMAT JSON, lots of cases where the json key
becomes an empty string.
I think this thread related issues has been resolved.