Thread

Re: libpq: PQgetCopyData() and allocation overhead

Jeroen Vermeulen <jtvjtv@gmail.com> — 2023-03-03T16:16:05Z

Thank you.

I meant to try PQExpBuffer — as you say, it fits much better with existing
libpq style.  But then I hit on the callback idea which was even more
efficient, by a fair margin.  It was also considerably simpler both inside
libpq and in the client code, eliminating all sorts of awkward questions
about who deallocates the buffer in which situations.  So I ditched the
"dynamic buffer" concept and went with the callback.

One other possible alternative suggests itself: instead of taking a
callback and a context pointer, the function could probably just return a
struct: status/size, and buffer.  Then the caller would have to figure out
whether there's a line in the buffer, and if so, process it.  It seems like
more work for the client code, but it may make the compiler's optimisation
work easier.


Jeroen

On Fri, 3 Mar 2023 at 16:52, Jelte Fennema <postgres@jeltef.nl> wrote:

> On Thu, 2 Mar 2023 at 20:45, Jeroen Vermeulen <jtvjtv@gmail.com> wrote:
> > I'm attaching a diff now.  This is not a patch, it's just a discussion
> piece.
>
> Did you try with PQExpBuffer? I still think that probably fits better
> in the API design of libpq. Obviously if it's significantly slower
> than the callback approach in this patch then it's worth considering
> using the callback approach. Overall it definitely seems reasonable to
> me to have an API that doesn't do an allocation per row.
>