Horton

RE: POC: postgres_fdw insert batching

tsunakawa.takay@fujitsu.com <tsunakawa.takay@fujitsu.com>

View thread View on postgr.es
From: "tsunakawa.takay@fujitsu.com" <tsunakawa.takay@fujitsu.com>
To: 'Craig Ringer' <craig.ringer@enterprisedb.com>
Cc: Tomas Vondra <tomas.vondra@enterprisedb.com>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, "Andrey V. Lepikhov" <a.lepikhov@postgrespro.ru>, "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>, Andres Freund <andres@anarazel.de>
Date: 2020-11-27T06:05:55Z
Lists: pgsql-hackers 
From: Craig Ringer <craig.ringer@enterprisedb.com> 
> But in the libpq pipelining patch I demonstrated a 300 times (3000%) performance improvement on a test workload...

Wow, impressive  number.  I've just seen it in the beginning of the libpq pipelining thread (oh, already four years ago..!)  Could you share the workload and the network latency (ping time)?  I'm sorry I'm just overlooking it.

Thank you for your (always) concise explanation.  I'd like to check other DBMSs and your rich references for the FDW interface.  (My first intuition is that many major DBMSs might not have client C APIs that can be used to implement an async pipelining FDW interface.  Also, I'm afraid it requires major surgery or reform of executor.  I don't want it to delay the release of reasonably good (10x) improvement with the synchronous interface.)

(It'd be kind of you to send emails in text format.  I've changed the format of this reply from HTML to text.)


Regards
Takayuki Tsunakawa

Commits

  1. Fix tuple routing to initialize batching only for inserts

  2. Fix pointer type in ExecForeignBatchInsert SGML docs

  3. Fix initialization of FDW batching in ExecInitModifyTable

  4. Implement support for bulk inserts in postgres_fdw