Horton

Re: snapbuild woes

Petr Jelinek <petr.jelinek@2ndquadrant.com>

View thread View on postgr.es
From: Petr Jelinek <petr.jelinek@2ndquadrant.com>
To: Stas Kelvich <s.kelvich@postgrespro.ru>
Cc: Erik Rijkers <er@xs4all.nl>, Andres Freund <andres@anarazel.de>, PostgreSQL-development <pgsql-hackers@postgresql.org>, Robert Haas <robertmhaas@gmail.com>, Craig Ringer <craig@2ndquadrant.com>
Date: 2017-05-11T14:59:18Z
Lists: pgsql-hackers 
On 11/05/17 16:33, Stas Kelvich wrote:
> 
>> On 10 May 2017, at 11:43, Petr Jelinek <petr.jelinek@2ndquadrant.com> wrote:
>>
>> On 09/05/17 22:11, Erik Rijkers wrote:
>>> On 2017-05-09 21:00, Petr Jelinek wrote:
>>>> On 09/05/17 19:54, Erik Rijkers wrote:
>>>>> On 2017-05-09 11:50, Petr Jelinek wrote:
>>>>>
>>>>
>>>> Ah okay, so this is same issue that's reported by both Masahiko Sawada
>>>> [1] and Jeff Janes [2].
>>>>
>>>> [1]
>>>> https://www.postgresql.org/message-id/CAD21AoBYpyqTSw%2B%3DES%2BxXtRGMPKh%3DpKiqjNxZKnNUae0pSt9bg%40mail.gmail.com
>>>>
>>>> [2]
>>>> https://www.postgresql.org/message-id/flat/CAMkU%3D1xUJKs%3D2etq2K7bmbY51Q7g853HLxJ7qEB2Snog9oRvDw%40mail.gmail.com
>>>>
>>>
>>> I don't understand why you come to that conclusion: both Masahiko Sawada
>>> and Jeff Janes have a DROP SUBSCRIPTION in the mix; my cases haven't. 
>>> Isn't that a real difference?
>>>
>>
> 
> Just noted another one issue/feature with snapshot builder — when logical decoding is in progress
> and vacuum full command is being issued quite a big amount of files appears in pg_logical/mappings
> and reside there till the checkpoint. Also having consequent vacuum full’s before checkpoint yields even
> more files.
> 
> Having two pgbench-filled instances with logical replication between them:
> 
> for i in {1..100}; do psql -c 'vacuum full' && ls -la pg_logical/mappings | wc -l; done;
> VACUUM
>       73
> VACUUM
>      454
> VACUUM
>     1146
> VACUUM
>     2149
> VACUUM
>     3463
> VACUUM
>     5088
> <...>
> VACUUM
>    44708
> <…> // checkpoint happens somewhere here
> VACUUM
>    20921
> WARNING:  could not truncate file "base/16384/61773": Too many open files in system
> ERROR:  could not fsync file "pg_logical/mappings/map-4000-4df-0_A4EA29F8-5aa5-5ae6": Too many open files in system
> 
> 
> I’m not sure whether this is boils down to some of the previous issues mentioned here or not, so posting
> here as observation.
> 

This has nothing to do with snapshot builder so this is not the thread
for it. See the comment section near the bottom of
src/backend/access/heap/rewriteheap.c for explanation of why it does
what it does.

-- 
  Petr Jelinek                  http://www.2ndQuadrant.com/
  PostgreSQL Development, 24x7 Support, Training & Services

Commits

  1. Fix thinko introduced in 2bef06d516460 et al.

  2. Avoid superfluous work for commits during logical slot creation.

  3. Fix race condition leading to hanging logical slot creation.

  4. Don't use on-disk snapshots for exported logical decoding snapshot.

  5. Preserve required !catalog tuples while computing initial decoding snapshot.