Re: PATCH: Batch/pipelining support for libpq

Andres Freund <andres@anarazel.de>

From: Andres Freund <andres@anarazel.de>
To: Craig Ringer <craig@2ndquadrant.com>
Cc: Daniel Verite <daniel@manitou-mail.org>, Vaishnavi Prabakaran <vaishnaviprabakaran@gmail.com>, Michael Paquier <michael.paquier@gmail.com>, David Steele <david@pgmasters.net>, "Prabakaran, Vaishnavi" <VaishnaviP@fast.au.fujitsu.com>, Haribabu Kommi <kommi.haribabu@gmail.com>, "Tsunakawa, Takayuki" <tsunakawa.takay@jp.fujitsu.com>, Dmitry Igrishin <dmitigr@gmail.com>, PostgreSQL Hackers <pgsql-hackers@postgresql.org>, Manuel Kniep <m.kniep@web.de>, "fujita.etsuro@lab.ntt.co.jp" <fujita.etsuro@lab.ntt.co.jp>, "Iwata, Aya" <iwata.aya@jp.fujitsu.com>
Date: 2017-06-22T01:07:21Z
Lists: pgsql-hackers
On 2017-06-22 09:03:05 +0800, Craig Ringer wrote:
> On 22 June 2017 at 08:29, Andres Freund <andres@anarazel.de> wrote:
> 
> > I.e. we're doing tiny write send() syscalls (they should be coalesced)
> 
> That's likely worth doing, but can probably wait for a separate patch.

I don't think so, we should get this right, it could have API influence.


> The kernel will usually do some packet aggregation unless we use
> TCP_NODELAY (which we don't and shouldn't), and the syscall overhead
> is IMO not worth worrying about just yet.

1)
					/*
					 * Select socket options: no delay of outgoing data for
					 * TCP sockets, nonblock mode, close-on-exec. Fail if any
					 * of this fails.
					 */
					if (!IS_AF_UNIX(addr_cur->ai_family))
					{
						if (!connectNoDelay(conn))
						{
							pqDropConnection(conn, true);
							conn->addr_cur = addr_cur->ai_next;
							continue;
						}
					}

2) Even if nodelay weren't set, this can still lead to smaller packets
   being sent, because you start sending normal sized tcp packets,
   rather than jumbo ones, even if configured (pretty common these
   days).

3) Syscall overhead is actually quite significant.

- Andres


Commits

  1. Add libpq pipeline mode support to pgbench

  2. Implement pipeline mode in libpq