Re: WAL format changes

On 18.06.2012 21:08, Heikki Linnakangas wrote:
> On 18.06.2012 21:00, Robert Haas wrote:
>> On Thu, Jun 14, 2012 at 5:58 PM, Andres Freund<andres@2ndquadrant.com>
>> wrote:
>>>> 1. Use a 64-bit segment number, instead of the log/seg combination. And
>>>> don't waste the last segment on each logical 4 GB log file. The concept
>>>> of a "logical log file" is now completely gone. XLogRecPtr is
>>>> unchanged,
>>>> but it should now be understood as a plain 64-bit value, just split
>>>> into
>>>> two 32-bit integers for historical reasons. On disk, this means that
>>>> there will be log files ending in FF, those were skipped before.
>>> Whats the reason for keeping that awkward split now? There aren't
>>> that many
>>> users of xlogid/xcrecoff and many of those would be better served by
>>> using
>>> helper macros.
>>
>> I wondered that, too. There may be a good reason for keeping it split
>> up that way, but we at least oughta think about it a bit.
>
> The page header contains an XLogRecPtr (LSN), so if we change it we'll
> have to deal with pg_upgrade. I guess we could still keep XLogRecPtr
> around as the on-disk representation, and convert between the 64-bit
> integer and XLogRecPtr in PageGetLSN/PageSetLSN. I can try that out -
> many xlog calculations would admittedly be simpler if it was an uint64.

Well, that was easier than I thought. Attached is a patch to make 
XLogRecPtr a uint64, on top of my other WAL format patches. I think we 
should go ahead with this.

The LSNs on pages are still stored in the old format, to avoid changing 
the on-disk format and breaking pg_upgrade. The XLogRecPtrs stored the 
control file and WAL are changed, however, so an initdb (or at least 
pg_resetxlog) is required.

Should we keep the old representation in the replication protocol 
messages? That would make it simpler to write a client that works with 
different server versions (like pg_receivexlog). Or, while we're at it, 
perhaps we should mandate network-byte order for all the integer and 
XLogRecPtr fields in the replication protocol. That would make it easier 
to write a client that works across different architectures, in >= 9.3. 
The contents of the WAL would of course be architecture-dependent, but 
it would be nice if pg_receivexlog and similar tools could nevertheless 
be architecture-independent.

I kept the %X/%X representation in error messages etc. I'm quite used to 
that output, so reluctant to change it, although it's a bit silly now 
that it represents just 64-bit value. Using UINT64_FORMAT would also 
make the messages harder to translate.

-- 
   Heikki Linnakangas
   EnterpriseDB   http://www.enterprisedb.com