Re: [patch] libpq one-row-at-a-time API

On Mon, Jul 30, 2012 at 10:26 PM, Jan Wieck <JanWieck@yahoo.com> wrote:
> On 7/30/2012 10:31 PM, Leon Smith wrote:
>>
>> This is not necessarily true,  on multiple levels.   I mean,  some of
>> the programs I write are highly concurrent,  and this form of batching
>> would have almost no risk of stalling the network buffer.    And
>> the possible use case would be when you are dealing with very small
>> rows,  when there would typically be several rows inside a single
>> network packet or network buffer.
>
>
> With "highly concurrent" you mean multi-threaded? Like one thread reads the
> rows in batches and pushes them into a queue while another thread processes
> them from that queue?
>
> If that is the case, then you just added a useless layer of buffering and
> the need for thread/thread context switches to PQsetSingleRowMode. Libpq's
> "receiver thread" is the kernel itself. Libpq tries to never read partial
> kernel buffers already. It always makes sure that there are at least 8K of
> free space in the inBuffer. In the case you describe above, where several
> rows fit into a single packet, libpq will receive them with a single system
> call in one read(2), then the application can get them as fast as possible,
> without causing any further context switches because they are already in the
> inBuffer.

Yeah: with asynchronous query processing the query gets sent and
control returns immediately to your code: that's the whole point.
Even if some data races to the network buffer, libpq doesn't 'see' any
data until you tell it to by asking for a result (which can block) or
draining the buffers with PQconsumeInput.  So there is no race in the
traditional sense and I'm ok with the PQsetSingleRowMode as such.

Removing malloc/free on row iteration seems only to be possible via
one of two methods: either a) you introduce a non-PGresult based
method of data extraction or b) you preserve the PGresult across row
iterations.

merlin