Horton

Thread

Commits

  1. Remove obsolete SLRU wrapping and warnings from predicate.c.

  2. Remove incorrect assertion in clog.c

  3. Keep track of transaction commit timestamps

  4. Handle 5-char filenames in SlruScanDirectory

  1. pg_serial early wraparound

    Thomas Munro <thomas.munro@enterprisedb.com> — 2016-11-08T22:07:42Z

    Hi hackers,

The SLRU managed by predicate.c can wrap around and overwrite data if
you have more than 1 billion active XIDs.  That's because when SSI was
implemented, slru.c was limited to four digit segment names, which
implied a page limit that wasn't enough for pg_serial to have space
for every possible XID.  We should probably rip that code out, because
SLRUs now support five digit segment names.  Something like the
attached.  I'll post a test script to demonstrate correct wraparound
behaviour around in time for one of the later CFs.

-- 
Thomas Munro
http://www.enterprisedb.com

  2. Re: pg_serial early wraparound

    Thomas Munro <thomas.munro@enterprisedb.com> — 2017-02-27T06:28:33Z

    On Wed, Nov 9, 2016 at 11:07 AM, Thomas Munro
<thomas.munro@enterprisedb.com> wrote:
> The SLRU managed by predicate.c can wrap around and overwrite data if
> you have more than 1 billion active XIDs.  That's because when SSI was
> implemented, slru.c was limited to four digit segment names, which
> implied a page limit that wasn't enough for pg_serial to have space
> for every possible XID.  We should probably rip that code out, because
> SLRUs now support five digit segment names.  Something like the
> attached.  I'll post a test script to demonstrate correct wraparound
> behaviour around in time for one of the later CFs.

Here is a shell script that shows a full rotation through xid space if
you build PostgreSQL with TEST_OLDSERXID, which you can do by
uncommenting a line in predicate.c.  On master we see the SLRU
segments go around the clock twice for each time xid goes around.
With the patch it goes around just once, adding an extra character to
the segment name to double the space.

By the way, I think the real number of xids it can hold today is
(65536 * 32 * 8192) / sizeof(uint64) = 2^31 xids, not 2^30 as
indicated by an existing comment.  So I think there is actually enough
space and wrapping is probably harmess, but it seems cleaner and
simpler not to do that and to rip out the scary warning code, so I'll
add this to the CF.

-- 
Thomas Munro
http://www.enterprisedb.com

  3. Re: pg_serial early wraparound

    Thomas Munro <thomas.munro@enterprisedb.com> — 2017-02-27T06:33:10Z

    On Mon, Feb 27, 2017 at 7:28 PM, Thomas Munro
<thomas.munro@enterprisedb.com> wrote:
> On Wed, Nov 9, 2016 at 11:07 AM, Thomas Munro
> <thomas.munro@enterprisedb.com> wrote:
>> The SLRU managed by predicate.c can wrap around and overwrite data if
>> you have more than 1 billion active XIDs.  That's because when SSI was
>> implemented, slru.c was limited to four digit segment names, which
>> implied a page limit that wasn't enough for pg_serial to have space
>> for every possible XID.  We should probably rip that code out, because
>> SLRUs now support five digit segment names.  Something like the
>> attached.  I'll post a test script to demonstrate correct wraparound
>> behaviour around in time for one of the later CFs.
>
> Here is a shell script that shows a full rotation through xid space if
> you build PostgreSQL with TEST_OLDSERXID, which you can do by
> uncommenting a line in predicate.c.  On master we see the SLRU
> segments go around the clock twice for each time xid goes around.
> With the patch it goes around just once, adding an extra character to
> the segment name to double the space.

I attached the wrong version.  Here is the right one.

-- 
Thomas Munro
http://www.enterprisedb.com

  4. Re: pg_serial early wraparound

    Anastasia Lubennikova <lubennikovaav@gmail.com> — 2017-03-24T14:11:22Z

    The following review has been posted through the commitfest application:
make installcheck-world:  not tested
Implements feature:       not tested
Spec compliant:           not tested
Documentation:            not tested

Hi, I've tried to review this patch, but it seems that I miss something essential.
You claim that SLRUs now support five digit segment name, while in slru.h
at current master I see the following:

 * Note: slru.c currently assumes that segment file names will be four hex
 * digits.  This sets a lower bound on the segment size (64K transactions
 * for 32-bit TransactionIds).
 */
#define SLRU_PAGES_PER_SEGMENT	32

/* Maximum length of an SLRU name */
#define SLRU_MAX_NAME_LENGTH	32

Could you please clarify the idea of the patch? Is it still relevant?

I've also run your test script.
pg_clog was renamed to pg_xact, so it need to be changed accordingly
echo "Contents of pg_clog:"
  ls $PGDATA/pg_xact/


The test shows failed assertion:

========== setting next xid to 1073741824 =========
Transaction log reset
waiting for server to start....2017-03-24 17:05:19.897 MSK [1181] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2017-03-24 17:05:19.981 MSK [1181] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2017-03-24 17:05:20.081 MSK [1183] LOG:  database system was shut down at 2017-03-24 17:05:19 MSK
2017-03-24 17:05:20.221 MSK [1181] LOG:  database system is ready to accept connections
 done
server started
vacuumdb: vacuuming database "postgres"
vacuumdb: vacuuming database "template0"
vacuumdb: vacuuming database "template1"
TRAP: FailedAssertion("!(TransactionIdPrecedesOrEquals(oldestXact, ShmemVariableCache->oldestXid))", File: "clog.c", Line: 669)
vacuumdb: vacuuming of database "template1" failed: server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request.
2017-03-24 17:05:21.541 MSK [1181] LOG:  server process (PID 1202) was terminated by signal 6: Aborted
2017-03-24 17:05:21.541 MSK [1181] DETAIL:  Failed process was running: VACUUM (FREEZE);

The new status of this patch is: Waiting on Author

  5. Re: pg_serial early wraparound

    Thomas Munro <thomas.munro@enterprisedb.com> — 2017-03-24T18:27:40Z

    On Sat, Mar 25, 2017 at 3:11 AM, Anastasia Lubennikova
<lubennikovaav@gmail.com> wrote:
> Hi, I've tried to review this patch, but it seems that I miss something essential.

Hi Anastasia,

Thanks for looking at this.

> You claim that SLRUs now support five digit segment name, while in slru.h
> at current master I see the following:
>
>  * Note: slru.c currently assumes that segment file names will be four hex
>  * digits.  This sets a lower bound on the segment size (64K transactions
>  * for 32-bit TransactionIds).
>  */
> #define SLRU_PAGES_PER_SEGMENT  32
>
> /* Maximum length of an SLRU name */
> #define SLRU_MAX_NAME_LENGTH    32

That comment is out of date.  Commit 638cf09e extended SLRUs to
support 5 digit names, to support pg_multixact.  And I see now that
commit 73c986ad more recently created the possibility of 6 chacater
SLRU file names for pg_commit_ts.

> Could you please clarify the idea of the patch? Is it still relevant?

The idea is simply to remove some strange old code including scary
error messages that is no longer needed.  In my study of predicate.c
for other reasons, I noticed this in passing and thought I'd tidy it
up.  Because I have tangled with pg_multixact and seen 5-character
SLRU files with my own eyes, I knew that the restriction that
motivated this code was no longer valid.

> I've also run your test script.
> pg_clog was renamed to pg_xact, so it need to be changed accordingly
> echo "Contents of pg_clog:"
>   ls $PGDATA/pg_xact/

Right.

> The test shows failed assertion:
>
> ========== setting next xid to 1073741824 =========
> Transaction log reset
> waiting for server to start....2017-03-24 17:05:19.897 MSK [1181] LOG:  listening on IPv4 address "127.0.0.1", port 5432
> 2017-03-24 17:05:19.981 MSK [1181] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
> 2017-03-24 17:05:20.081 MSK [1183] LOG:  database system was shut down at 2017-03-24 17:05:19 MSK
> 2017-03-24 17:05:20.221 MSK [1181] LOG:  database system is ready to accept connections
>  done
> server started
> vacuumdb: vacuuming database "postgres"
> vacuumdb: vacuuming database "template0"
> vacuumdb: vacuuming database "template1"
> TRAP: FailedAssertion("!(TransactionIdPrecedesOrEquals(oldestXact, ShmemVariableCache->oldestXid))", File: "clog.c", Line: 669)
> vacuumdb: vacuuming of database "template1" failed: server closed the connection unexpectedly
>     This probably means the server terminated abnormally
>     before or while processing the request.
> 2017-03-24 17:05:21.541 MSK [1181] LOG:  server process (PID 1202) was terminated by signal 6: Aborted
> 2017-03-24 17:05:21.541 MSK [1181] DETAIL:  Failed process was running: VACUUM (FREEZE);

My cheap trick for moving the xid around the clock quickly to test
wraparound scenarios no longer works, since this new assertion was
added in ea42cc18.  That was committed just a few hours before you
tested this.  Bad luck for me!

> The new status of this patch is: Waiting on Author

It's not urgent, it's just cleanup work, so I've now moved it to the
next commitfest.  I will try to figure out a new way to demonstrate
that it works correctly without having to ask a reviewing to disable
any assertions.  Thanks again.

-- 
Thomas Munro
http://www.enterprisedb.com

  6. Re: pg_serial early wraparound

    Thomas Munro <thomas.munro@enterprisedb.com> — 2017-06-28T01:11:29Z

    On Sat, Mar 25, 2017 at 7:27 AM, Thomas Munro
<thomas.munro@enterprisedb.com> wrote:
> On Sat, Mar 25, 2017 at 3:11 AM, Anastasia Lubennikova
> <lubennikovaav@gmail.com> wrote:
>> You claim that SLRUs now support five digit segment name, while in slru.h
>> at current master I see the following:
>>
>>  * Note: slru.c currently assumes that segment file names will be four hex
>>  * digits.  This sets a lower bound on the segment size (64K transactions
>>  * for 32-bit TransactionIds).
>>  */

I've now complained about that comment in a separate thread.

> It's not urgent, it's just cleanup work, so I've now moved it to the
> next commitfest.  I will try to figure out a new way to demonstrate
> that it works correctly without having to ask a review[er] to disable
> any assertions.  Thanks again.

Here's a rebased batch.

-- 
Thomas Munro
http://www.enterprisedb.com

  7. Re: pg_serial early wraparound

    Thomas Munro <thomas.munro@enterprisedb.com> — 2017-09-01T11:12:10Z

    On Wed, Jun 28, 2017 at 1:11 PM, Thomas Munro
<thomas.munro@enterprisedb.com> wrote:
> On Sat, Mar 25, 2017 at 7:27 AM, Thomas Munro
> <thomas.munro@enterprisedb.com> wrote:
>> On Sat, Mar 25, 2017 at 3:11 AM, Anastasia Lubennikova
>> <lubennikovaav@gmail.com> wrote:
>>> You claim that SLRUs now support five digit segment name, while in slru.h
>>> at current master I see the following:
>>>
>>>  * Note: slru.c currently assumes that segment file names will be four hex
>>>  * digits.  This sets a lower bound on the segment size (64K transactions
>>>  * for 32-bit TransactionIds).
>>>  */
>
> I've now complained about that comment in a separate thread.
>
>> It's not urgent, it's just cleanup work, so I've now moved it to the
>> next commitfest.  I will try to figure out a new way to demonstrate
>> that it works correctly without having to ask a review[er] to disable
>> any assertions.  Thanks again.

Rebased again, now with a commit message.  That assertion has since
been removed (commit ec99dd5a) so the attached test script can once
again be used to see the contents of pg_serial as the xid goes all the
way around, if you build with TEST_OLDSERXID defined so that
predicate.c forces information about xids out to pg_serial.

-- 
Thomas Munro
http://www.enterprisedb.com

  8. Re: [HACKERS] pg_serial early wraparound

    Michael Paquier <michael.paquier@gmail.com> — 2017-11-30T03:16:41Z

    On Fri, Sep 1, 2017 at 8:12 PM, Thomas Munro
<thomas.munro@enterprisedb.com> wrote:
> Rebased again, now with a commit message.  That assertion has since
> been removed (commit ec99dd5a) so the attached test script can once
> again be used to see the contents of pg_serial as the xid goes all the
> way around, if you build with TEST_OLDSERXID defined so that
> predicate.c forces information about xids out to pg_serial.

Moved to next CF per lack of reviews.
-- 
Michael

  9. Re: [HACKERS] pg_serial early wraparound

    Stephen Frost <sfrost@snowman.net> — 2018-01-22T22:50:14Z

    Greetings,

* Thomas Munro (thomas.munro@enterprisedb.com) wrote:
> On Wed, Jun 28, 2017 at 1:11 PM, Thomas Munro
> <thomas.munro@enterprisedb.com> wrote:
> > On Sat, Mar 25, 2017 at 7:27 AM, Thomas Munro
> > <thomas.munro@enterprisedb.com> wrote:
> >> On Sat, Mar 25, 2017 at 3:11 AM, Anastasia Lubennikova
> >> <lubennikovaav@gmail.com> wrote:
> >>> You claim that SLRUs now support five digit segment name, while in slru.h
> >>> at current master I see the following:
> >>>
> >>>  * Note: slru.c currently assumes that segment file names will be four hex
> >>>  * digits.  This sets a lower bound on the segment size (64K transactions
> >>>  * for 32-bit TransactionIds).
> >>>  */
> >
> > I've now complained about that comment in a separate thread.
> >
> >> It's not urgent, it's just cleanup work, so I've now moved it to the
> >> next commitfest.  I will try to figure out a new way to demonstrate
> >> that it works correctly without having to ask a review[er] to disable
> >> any assertions.  Thanks again.
> 
> Rebased again, now with a commit message.  That assertion has since
> been removed (commit ec99dd5a) so the attached test script can once
> again be used to see the contents of pg_serial as the xid goes all the
> way around, if you build with TEST_OLDSERXID defined so that
> predicate.c forces information about xids out to pg_serial.

I've taken a look through this and it seems pretty reasonable.  Would be
great to have someone actually try to duplicate the testing that Thomas
did (though I have little doubt that it works as described) and get it
to Ready-For-Committer state.

Anastasia, thanks for the previous review, any chance you could try
again with the latest patch (against the current state of git)?

Thanks!

Stephen

  10. Re: pg_serial early wraparound

    Anastasia Lubennikova <lubennikovaav@gmail.com> — 2018-03-12T11:26:27Z

    The following review has been posted through the commitfest application:
make installcheck-world:  tested, passed
Implements feature:       tested, passed
Spec compliant:           not tested
Documentation:            not tested

The patch doesn't break anything in regression tests and does the code cleanup.
As far as I understand, the removed code was dead, since SLRU size is large enough 
and the wraparound, described in the message is impossible.
So I mark it as Ready For Committer.

I didn't manage to repeat the attached test, though. 
Server doesn't start after xid reset. It throws an error:

server stopped
========== setting next xid to 1073741824 =========
Write-ahead log reset
waiting for server to start....2018-03-12 14:18:59.551 MSK [16126] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2018-03-12 14:18:59.625 MSK [16126] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2018-03-12 14:18:59.764 MSK [16127] LOG:  database system was shut down at 2018-03-12 14:18:59 MSK
2018-03-12 14:18:59.802 MSK [16127] FATAL:  could not access status of transaction 10737418
2018-03-12 14:18:59.802 MSK [16127] DETAIL:  Could not open file "pg_xact/000A": Нет такого файла или каталога.
2018-03-12 14:18:59.803 MSK [16126] LOG:  startup process (PID 16127) exited with exit code 1
2018-03-12 14:18:59.803 MSK [16126] LOG:  aborting startup due to startup process failure
2018-03-12 14:18:59.804 MSK [16126] LOG:  database system is shut down

  11. Re: [HACKERS] pg_serial early wraparound

    Tom Lane <tgl@sss.pgh.pa.us> — 2018-03-26T16:50:00Z

    Thomas Munro <thomas.munro@enterprisedb.com> writes:
> Rebased again, now with a commit message.  That assertion has since
> been removed (commit ec99dd5a) so the attached test script can once
> again be used to see the contents of pg_serial as the xid goes all the
> way around, if you build with TEST_OLDSERXID defined so that
> predicate.c forces information about xids out to pg_serial.

Couple thoughts here ---

Seems like if the patch is correct as-is, then the OldSerXidPage
macro could be simplified, as the modulo no longer does anything.
Also, OldSerXidSegment doesn't seem to be used.

I'm a little worried because Anastasia couldn't repeat the test;
why is that?

			regards, tom lane

  12. Re: [HACKERS] pg_serial early wraparound

    Thomas Munro <thomas.munro@enterprisedb.com> — 2018-03-27T04:01:36Z

    On Tue, Mar 27, 2018 at 5:50 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Thomas Munro <thomas.munro@enterprisedb.com> writes:
>> Rebased again, now with a commit message.  That assertion has since
>> been removed (commit ec99dd5a) so the attached test script can once
>> again be used to see the contents of pg_serial as the xid goes all the
>> way around, if you build with TEST_OLDSERXID defined so that
>> predicate.c forces information about xids out to pg_serial.
>
> Couple thoughts here ---

Thanks for looking at this!

> Seems like if the patch is correct as-is, then the OldSerXidPage
> macro could be simplified, as the modulo no longer does anything.

The patch already did that:

-#define OldSerXidPage(xid)     ((((uint32) (xid)) /
OLDSERXID_ENTRIESPERPAGE) % (OLDSERXID_MAX_PAGE + 1))
+#define OldSerXidPage(xid)     (((uint32) (xid)) / OLDSERXID_ENTRIESPERPAGE)

> Also, OldSerXidSegment doesn't seem to be used.

Right, thanks.  Removed.

> I'm a little worried because Anastasia couldn't repeat the test;
> why is that?

Hmm.  I'm not sure.  It works for me on a couple of machines and what I see is:

========== setting next xid to 65536 =========
...
Contents of pg_serial:
0002
========== setting next xid to 1073741824 =========
...
Contents of pg_serial:
8000
========== setting next xid to 2147483648 =========
...
Contents of pg_serial:
10000
========== setting next xid to 3221225472 =========
...
Contents of pg_serial:
18000
========== setting next xid to 65536 =========
...
Contents of pg_serial:
0002
========== setting next xid to 1073741824 =========
...
Contents of pg_serial:
8000

-- 
Thomas Munro
http://www.enterprisedb.com

  13. Re: [HACKERS] pg_serial early wraparound

    Tom Lane <tgl@sss.pgh.pa.us> — 2018-03-30T19:14:22Z

    Thomas Munro <thomas.munro@enterprisedb.com> writes:
> On Tue, Mar 27, 2018 at 5:50 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> I'm a little worried because Anastasia couldn't repeat the test;
>> why is that?

> Hmm.  I'm not sure.  It works for me on a couple of machines and what I see is:

Yeah, it works for me too, so not sure what the problem was.

The only other point I can think of is that by changing the storage
layout under pg_serial/, this'd create an issue for pg_upgrade, if
pg_upgrade tried to migrate this data.  But it doesn't.

Hence, pushed.

			regards, tom lane