Horton

Re: PATCH: Batch/pipelining support for libpq

Andres Freund <andres@anarazel.de>

View thread View on postgr.es
From: Andres Freund <andres@anarazel.de>
To: Daniel Verite <daniel@manitou-mail.org>
Cc: Craig Ringer <craig@2ndquadrant.com>, Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>, Michael Paquier <michael.paquier@gmail.com>, David Steele <david@pgmasters.net>, "Prabakaran, Vaishnavi" <VaishnaviP@fast.au.fujitsu.com>, Haribabu Kommi <kommi.haribabu@gmail.com>, "Tsunakawa, Takayuki" <tsunakawa.takay@jp.fujitsu.com>, Dmitry Igrishin <dmitigr@gmail.com>, PostgreSQL Hackers <pgsql-hackers@postgresql.org>, Manuel Kniep <m.kniep@web.de>, "fujita.etsuro@lab.ntt.co.jp" <fujita.etsuro@lab.ntt.co.jp>, "Iwata, Aya" <iwata.aya@jp.fujitsu.com>
Date: 2017-06-22T15:52:04Z
Lists: pgsql-hackers 
Hi,

On 2017-06-22 13:43:35 +0200, Daniel Verite wrote:
> But OTOH there are certainly batch workloads where it will be preferrable
> for the first query to reach the server ASAP, rather than waiting to be
> coalesced with the next ones.

Is that really something people expect from a batch API?  I suspect it's
not really, and nothing would stop one from adding PQflush() or similar
calls if desirable anyway.

FWIW, the way I did that in the hack clearly isn't ok: If you were to
send a gigabyte of queries, it'd buffer them all up in memory... So some
more intelligence is going to be needed.


> libpq is not going to know what's best.
> One option may be to leave that decision to the user by providing a
> PQBatchAutoFlush(true|false) property, along with a PQBatchFlush()
> function.

What'd be the difference between PQflush() and PQbatchFlush()?

Greetings,

Andres Freund

Commits

  1. Add libpq pipeline mode support to pgbench

  2. Implement pipeline mode in libpq