Re: [POC] Fast COPY FROM command for the table with foreign partitions

On 2/9/21 12:47 PM, tsunakawa.takay@fujitsu.com wrote:
> From: Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
> I guess you used many hash partitions.  Sadly, The current COPY implementation only accumulates either 1,000 rows or 64 KB of input data (very small!) before flushing all CopyMultiInsertBuffers.  One CopyMultiInsertBuffer corresponds to one partition.  Flushing a CopyMultiInsertBuffer calls ExecForeignCopy() once, which connects to a remote database, runs COPY FROM STDIN, and disconnects.  Here, the flushing trigger (1,000 rows or 64 KB input data, whichever comes first) is so small that if there are many target partitions, the amount of data for each partition is small.
I tried to use 1E4 - 1E8 rows in a tuple buffer. But the results weren't 
impressive.
We can use one more GUC instead of a precompiled constant.

> Why don't we focus on committing the basic part and addressing the extended part (0003 and 0004) separately later?
I focused only on the 0001 and 0002 patches.
>  As Tang-san and you showed, the basic part already demonstrated impressive improvement.  If there's no objection, I'd like to make this ready for committer in a few days.
Good.

-- 
regards,
Andrey Lepikhov
Postgres Professional