Allow substitute allocators for PGresult.
Kyotaro Horiguchi <horiguchi.kyotaro@oss.ntt.co.jp>
From: Kyotaro HORIGUCHI <horiguchi.kyotaro@oss.ntt.co.jp>
To: pgsql-hackers@postgresql.org
Date: 2011-11-11T09:18:43Z
Lists: pgsql-hackers
Attachments
- libpq_dupevents_alloc_r1.patch (text/x-patch) patch
- libpq_replasable_alloc_r1.patch (text/x-patch) patch
- testlibpq.c.gz (application/octet-stream)
Hello. This message is a proposal of a pair of patches that enables the memory allocator for PGresult in libpq to be replaced. The comment at the the begging of pqexpbuffer.c says that libpq should not rely on palloc(). Besides, Tom Lane said that palloc should not be visible outside the backend(*1) and I agree with it. *1: http://archives.postgresql.org/pgsql-hackers/1999-02/msg00364.php On the other hand, in dblink, dblink-plus (our product!), and maybe FDW's connect to other PostgreSQL servers are seem to copy the result of the query contained in PGresult into tuple store. I guess that this is in order to avoid memory leakage on termination in halfway. But it is rather expensive to copy whole PGresult, and the significance grows as the data received gets larger. Furthermore, it requires about twice as much memory as the net size of the data. And it is fruitless to copy'n modify libpq or reinvent it from scratch. So we shall be happy to be able to use palloc's in libpq at least for PGresult for such case in spite of the policy. For these reasons, I propose to make allocators for PGresult replaceable. The modifications are made up into two patches. 1. dupEvents() and pqAddTuple() get new memory block by malloc currently, but the aquired memory block is linked into PGresult finally. So I think it is preferable to use pqResultAlloc() or its descendents in consistensy with the nature of the place to link. But there is not PQresultRealloc() and it will be costly, so pqAddTuple() is not modified in this patch. 2. Define three function pointers PQpgresult_(malloc|realloc|free) and replace the calls to malloc/realloc/free in the four functions below with these pointers. PQmakeEmptyPGresult() pqResultAlloc() PQclear() pqAddTuple() This patches make the tools run in backend process and use libpq possible to handle PGresult as it is with no copy, no more memory. (Of cource, someone wants to use his/her custom allocator for PGresult on standalone tools could do that using this feature.) Three files are attached to this message. First, the patch with respect to "1" above. Second, the patch with respect to "2" above. Third, a very simple sample program. I have built and briefly tested on CentOS6, with the sample program mentioned above and valgrind, but not on Windows. How do you think about this? Regards, -- Kyotaro Horiguchi NTT Open Source Software Center