Re: Add Pipelining support in psql
Anthonin Bonnefoy <anthonin.bonnefoy@datadoghq.com>
Attachments
- v03-0001-Add-pipelining-support-in-psql.patch (application/octet-stream) patch v3-0001
An updated version of the patch, still a bit rough around the edges. I've added \flushrequest, \flush and \getrequests meta-commands. \flushrequest and \getresults look interesting as they add an additional protocol message to test, but it also introduces the possibility to be in the middle of an aborted pipeline which adds some complexity to the patch. I'm a bit skeptical of \flush as \getresults already automatically flush through PQgetResult and I haven't thought of interesting tests that could rely on \flush. I've added a new prompt interpolation instead of trying to hijack the existing %x: %P reports the pipeline status with either '|num_syncs,num_queries,pending_results|' for an ongoing pipeline or '+num_syncs,num_queries,pending_results+' for an aborted pipeline. Those are definitely open to debate for those but I found those informations useful while debugging. With the new \flushrequest and \getresults, it's visible that %x doesn't reflect the transaction status when pipelines are involved. If I do: postgres=# \startpipeline postgres=|0,0,0|# BEGIN \bind \g postgres=|0,1,0|*# \flushrequest postgres=|0,1,1|*# \getresults BEGIN postgres=|0,0,0|# There's an ongoing transaction block running on the server but %x will be empty as there's no queued commands and the server results are consumed. I'm tempted to change %x to not display anything when within a pipeline. Calling PQgetResult while a command is piped and not followed by a flushrequest or a sync would block since the asyncstatus is busy. So \getresults has to keep track of the number of pending results to avoid calling PQgetResult if there's no pending results. This is to cover cases like: postgres=# \startpipeline postgres=|0,0,0|# select 1 \bind \g postgres=|0,1,0|*# \getresults No pending results to get postgres=|0,1,0|*# There are some interesting behaviors with pipeline + copy. Running the following: \startpipeline SELECT $1 \bind 'val1' \g COPY psql_pipeline FROM STDIN \bind \g \flushrequest \getresults 3 test3 \. \endpipeline Will complain with a "message type 0x5a arrived from server while idle" (but still work). It's also possible to trigger a protocol error with a query error after a piped copy: truncate pgbench_accounts; \startpipeline copy pgbench_accounts FROM stdin \bind \g SELECT 1 \bind \g \endpipeline 1 2 3 asd \. Copy will fail with "ERROR: unexpected message type 0x50 during COPY from stdin" and connection will be terminated.
Commits
GET /api/v1/messages/:b64id/commits
the thread's linked commits as JSON, with link sources.
API reference →
-
psql: Fix assertion failures with pipeline mode
- 3631612eae9c 18.0 landed
-
psql: Rework TAP routine psql_fails_like() to define WAL sender context
- 0ff95e0a5be1 18.0 landed
-
psql: Split extended query protocol meta-commands in --help=commands
- 78231baaf967 18.0 landed
-
psql: Improve descriptions of \\flush[request] in --help
- 5743d122fcf3 18.0 landed
-
psql: Fix incorrect status code returned by \getresults
- 5ee7bd944ee8 18.0 landed
-
psql: Allow queries terminated by semicolons while in pipeline mode
- 2cce0fe440fb 18.0 landed
-
psql: Add \sendpipeline to send query buffers while in a pipeline
- 17caf6644546 18.0 landed
-
psql: Fix memory leak with \gx used within a pipeline
- 54d23601b978 18.0 landed
-
psql: Add pipeline status to prompt and some state variables
- 3ce357584e79 18.0 landed
-
Add more tests for utility commands in pipelines
- a4e986ef5a46 18.0 landed
-
psql: Add support for pipelines
- 41625ab8ea3d 18.0 landed
-
Add braces for if block with large comment in psql's common.c
- 40af897eb777 18.0 landed