Re: [HACKERS] PATCH: Batch/pipelining support for libpq

Alvaro Herrera <alvherre@2ndquadrant.com>

From: Alvaro Herrera <alvherre@2ndquadrant.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Justin Pryzby <pryzby@telsasoft.com>, pgsql-hackers@lists.postgresql.org, Craig Ringer <craig.ringer@enterprisedb.com>, Matthieu Garrigues <matthieu.garrigues@gmail.com>, Zhihong Yu <zyu@yugabyte.com>, Álvaro Herrera <alvherre@alvh.no-ip.org>, Andres Freund <andres@anarazel.de>, Aya Iwata <iwata.aya@jp.fujitsu.com>, Daniel Vérité <daniel@manitou-mail.org>, "David G. Johnston" <david.g.johnston@gmail.com>, Kirk Jamison <k.jamison@fujitsu.com>, Michael Paquier <michael.paquier@gmail.com>, Nikhil Sontakke <nikhils@2ndquadrant.com>, Vaishnavi Prabakaran <VaishnaviP@fast.au.fujitsu.com>
Date: 2021-03-15T18:54:54Z
Lists: pgsql-hackers

Attachments

Here's what seems a final version of the patch.  I renamed one more
function: PQsendPipeline is now PQpipelineSync.  I also reworded the
docs in a couple of places, added a few tests to the pgbench patch, and
made it work.

Note the pgbench results in pipeline mode:

./pgbench -r -Mextended -n -f /home/alvherre/Code/pgsql-build/pipeline/src/bin/pgbench/tmp_check/t_001_pgbench_with_server_main_data/001_pgbench_pipeline -c 100 -t10000
pgbench (PostgreSQL) 14.0
transaction type: /home/alvherre/Code/pgsql-build/pipeline/src/bin/pgbench/tmp_check/t_001_pgbench_with_server_main_data/001_pgbench_pipeline
scaling factor: 1
query mode: extended
number of clients: 100
number of threads: 1
number of transactions per client: 10000
number of transactions actually processed: 1000000/1000000
latency average = 2.316 ms
initial connection time = 113.859 ms
tps = 43182.438635 (without initial connection time)
statement latencies in milliseconds:
         0.000  \startpipeline
         0.000  select 1;
         0.000  select 1;
         0.000  select 1;
         0.000  select 1;
         0.000  select 1;
         0.000  select 1;
         0.000  select 1;
         0.000  select 1;
         0.000  select 1;
         0.000  select 1;
         1.624  \endpipeline

If I just replace the \startpipeline and \endpipeline lines with BEGIN
and COMMIT respectively, I get this:

tps = 10220.259051 (without initial connection time)

         0.830  begin;
         0.765  select 1;
         0.752  select 1;
         0.753  select 1;
         0.755  select 1;
         0.754  select 1;
         0.755  select 1;
         0.757  select 1;
         0.756  select 1;
         0.756  select 1;
         0.756  select 1;
         0.750  commit;

Yes, you could say that this is a liiiitle bit unfair -- but it seems
quite impressive nonetheless.

-- 
Álvaro Herrera                            39°49'30"S 73°17'W

Commits

  1. Add libpq pipeline mode support to pgbench

  2. Implement pipeline mode in libpq