Re: PATCH: Batch/pipelining support for libpq

Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>

From: Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>
To: Daniel Verite <daniel@manitou-mail.org>
Cc: Michael Paquier <michael.paquier@gmail.com>, "Prabakaran, Vaishnavi" <VaishnaviP@fast.au.fujitsu.com>, Craig Ringer <craig@2ndquadrant.com>, Haribabu Kommi <kommi.haribabu@gmail.com>, "Tsunakawa, Takayuki" <tsunakawa.takay@jp.fujitsu.com>, Dmitry Igrishin <dmitigr@gmail.com>, Andres Freund <andres@anarazel.de>, PostgreSQL Hackers <pgsql-hackers@postgresql.org>, Manuel Kniep <m.kniep@web.de>, "fujita.etsuro@lab.ntt.co.jp" <fujita.etsuro@lab.ntt.co.jp>, "Iwata, Aya" <iwata.aya@jp.fujitsu.com>
Date: 2017-03-13T00:54:02Z
Lists: pgsql-hackers
On Sat, Mar 11, 2017 at 12:52 AM, Daniel Verite <daniel@manitou-mail.org>
wrote:

>   Hi,
>
> I notice that PQsetSingleRowMode() doesn't work when getting batch results.
>
> The function is documented as:
> " int PQsetSingleRowMode(PGconn *conn);
>
>   This function can only be called immediately after PQsendQuery or one
>   of its sibling functions, before any other operation on the connection
>   such as PQconsumeInput or PQgetResult"
>
> But PQbatchQueueProcess() unconditionally clears conn->singleRowMode,
> so whatever it was when sending the query gets lost, and besides
> other queries might have been submitted in the meantime.
>
> Also if trying to set that mode when fetching like this
>
>      while (QbatchQueueProcess(conn)) {
>        r = PQsetSingleRowMode(conn);
>        if (r!=1) {
>           fprintf(stderr, "PQsetSingleRowMode() failed");
>        }
>        ..
>
> it might work the first time, but on the next iterations, conn->asyncStatus
> might be PGASYNC_READY, which is a failure condition for
> PQsetSingleRowMode(), so that won't do.
>


Thanks for investigating the problem, and could you kindly explain what
"next iteration" you mean here? Because I don't see any problem in
following sequence of calls - PQbatchQueueProcess(),PQsetSingleRowMode()
, PQgetResult(). Am I missing anything?
Please note that it is MUST to call PQgetResult immediately after
PQbatchQueueProcess() as documented.



>
> ISTM that the simplest fix would be that when in batch mode,
> PQsetSingleRowMode() should register that the last submitted query
> does request that mode, and when later QbatchQueueProcess dequeues
> the batch entry for the corresponding query, this flag would be popped off
> and set as the current mode.
>
> Please find attached the suggested fix, against the v5 of the patch.
>

Before going with this fix, I would like you to consider the option of
asking batch processing users(via documentation) to set single-row mode
before calling PQgetResult().
Either way we need to fix the documentation part, letting users know how
they can activate single-row mode while using batch processing.
Let me know your thoughts.

Best Regards,
Vaishnavi,
Fujitsu Australia.

Commits

  1. Add libpq pipeline mode support to pgbench

  2. Implement pipeline mode in libpq