Horton

Re: PATCH: Batch/pipelining support for libpq

Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>

View thread View on postgr.es
From: Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
To: Daniel Verite <daniel@manitou-mail.org>
Cc: Andres Freund <andres@anarazel.de>, Craig Ringer <craig@2ndquadrant.com>, Michael Paquier <michael.paquier@gmail.com>, David Steele <david@pgmasters.net>, "Prabakaran, Vaishnavi" <VaishnaviP@fast.au.fujitsu.com>, Haribabu Kommi <kommi.haribabu@gmail.com>, "Tsunakawa, Takayuki" <tsunakawa.takay@jp.fujitsu.com>, Dmitry Igrishin <dmitigr@gmail.com>, PostgreSQL Hackers <pgsql-hackers@postgresql.org>, Manuel Kniep <m.kniep@web.de>, "fujita.etsuro@lab.ntt.co.jp" <fujita.etsuro@lab.ntt.co.jp>, "Iwata, Aya" <iwata.aya@jp.fujitsu.com>
Date: 2017-08-10T05:23:06Z
Lists: pgsql-hackers

Attachments

Andres Freund wrote :
> If you were to send a gigabyte of queries, it'd buffer them all up in
memory... So some
>more intelligence is going to be needed.

Firstly, sorry for the delayed response as I got busy with other
activities.

To buffer up the queries before flushing them to the socket, I think
"conn->outCount>=65536" is ok to use, as 64k is considered to be safe in
Windows as per comments in pqSendSome() API.

Attached the code patch to replace pqFlush calls with pqBatchFlush in the
asynchronous libpq function calls flow.
Still pqFlush is used in "PQbatchSyncQueue" and
"PQexitBatchMode" functions.

Am failing to see the benefit in allowing user to set
PQBatchAutoFlush(true|false) property? Is it really needed?

Thanks & Regards,
Vaishnavi,
Fujitsu Australia.

Commits

  1. Add libpq pipeline mode support to pgbench

  2. Implement pipeline mode in libpq