Thread
-
Re: pg_dump and large files - is this a problem?
Giles Lean <giles@nemeton.com.au> — 2002-10-04T00:50:21Z
Philip Warner writes: > Yes, but there is no guarantee that off_t is implemented as such, nor would > we be wise to assume so (most docs say explicitly not to do so). I suspect you're reading old documents, which is why I asked what you were referring to. In the '80s what you are saying would have been best practice, no question: 64 bit type support was not common. When talking of near-current systems with 64 bit off_t you are not going to find one without support for 64 bit integral types. > Again yes, but the problem is the same: we need a way of making the *value* > of an off_t portable (not just assuming it's a int64). In general that > involves knowing how to turn it into a more universal data type (eg. int64, > or even a string). So you need to know the size of off_t, which will be 32 bit or 64 bit, and then you need routines to convert that to a portable representation. The canonical solution is XDR, but I'm not sure that you want to bother with it or if it has been extended universally to support 64 bit types. If you limit the file sizes to 1GB (your less preferred option, I know;-) then like the rest of the PostgreSQL code you can safely assume that off_t fits into 32 bits and have a choice of functions (XDR or ntohl() etc) to deal with them and ignore 64 bit off_t issues altogether. If you intend pg_dump files to be portable avoiding the use of large files will be best. It also avoids issues on platforms such as HP-UX where large file support is available, but it has to be enabled on a per-filesystem basis. :-( > Does the large file API have functions for representing > the off_t values that is portable across architectures? And is the API also > portable? The large files API is a way to access large files from 32 bit processes. It is reasonably portable, but is a red herring for what you are wanting to do. (I'm not convinced I am understanding what you're trying to do, but I have 'flu which is not helping. :-) Regards, Giles
-
Re: pg_dump and large files - is this a problem?
Tom Lane <tgl@sss.pgh.pa.us> — 2002-10-04T03:07:09Z
Giles Lean <giles@nemeton.com.au> writes: > When talking of near-current systems with 64 bit off_t you are not > going to find one without support for 64 bit integral types. I tend to agree with Giles on this point. A non-integral representation of off_t is theoretically possible but I don't believe it exists in practice. Before going far out of our way to allow it, we should first require some evidence that it's needed on a supported or likely-to-be-supported platform. time_t isn't guaranteed to be an integral type either if you read the oldest docs about it ... but no one believes that in practice ... regards, tom lane
-
Re: pg_dump and large files - is this a problem?
Bruce Momjian <pgman@candle.pha.pa.us> — 2002-10-04T03:10:19Z
Tom Lane wrote: > Giles Lean <giles@nemeton.com.au> writes: > > When talking of near-current systems with 64 bit off_t you are not > > going to find one without support for 64 bit integral types. > > I tend to agree with Giles on this point. A non-integral representation > of off_t is theoretically possible but I don't believe it exists in > practice. Before going far out of our way to allow it, we should first > require some evidence that it's needed on a supported or > likely-to-be-supported platform. > > time_t isn't guaranteed to be an integral type either if you read the > oldest docs about it ... but no one believes that in practice ... I think fpos_t is the non-integral one. I thought off_t almost always was integral. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania 19073
-
Re: pg_dump and large files - is this a problem?
Philip Warner <pjw@rhyme.com.au> — 2002-10-04T03:15:29Z
At 11:07 PM 3/10/2002 -0400, Tom Lane wrote: >A non-integral representation >of off_t is theoretically possible but I don't believe it exists in >practice. Excellent. So I can just read/write the bytes in an appropriate order and expect whatever size it is to be a single intXX. Fine with me, unless anybody voices another opinion in the next day, I will proceed. I just have this vague recollection of seeing a header file with a more complex structure for off_t. I'm probably dreaming. ---------------------------------------------------------------- Philip Warner | __---_____ Albatross Consulting Pty. Ltd. |----/ - \ (A.B.N. 75 008 659 498) | /(@) ______---_ Tel: (+61) 0500 83 82 81 | _________ \ Fax: (+61) 0500 83 82 82 | ___________ | Http://www.rhyme.com.au | / \| | --________-- PGP key available upon request, | / and from pgp5.ai.mit.edu:11371 |/ -
Re: pg_dump and large files - is this a problem?
Philip Warner <pjw@rhyme.com.au> — 2002-10-18T03:42:24Z
I have made the changes to pg_dump and verified that (a) it reads old files, (b) it handles 8 byte offsets, and (c) it dumps & seems to restore (at least to /dev/null). I don't have a lot of options for testing it - should I just apply the changes and wait for the problems, or can someone offer a bigendian machine and/or a 4 byte off_t machine? >was integral. ---------------------------------------------------------------- Philip Warner | __---_____ Albatross Consulting Pty. Ltd. |----/ - \ (A.B.N. 75 008 659 498) | /(@) ______---_ Tel: (+61) 0500 83 82 81 | _________ \ Fax: (+61) 0500 83 82 82 | ___________ | Http://www.rhyme.com.au | / \| | --________-- PGP key available upon request, | / and from pgp5.ai.mit.edu:11371 |/ -
Re: pg_dump and large files - is this a problem?
Tom Lane <tgl@sss.pgh.pa.us> — 2002-10-18T13:25:26Z
Philip Warner <pjw@rhyme.com.au> writes: > I don't have a lot of options for testing it - should I just apply the > changes and wait for the problems, or can someone offer a bigendian machine > and/or a 4 byte off_t machine? My HP is big-endian; send in the patch and I'll check it here... regards, tom lane
-
Re: pg_dump and large files - is this a problem?
Peter Eisentraut <peter_e@gmx.net> — 2002-10-18T22:07:07Z
Philip Warner writes: > > I have made the changes to pg_dump and verified that (a) it reads old > files, (b) it handles 8 byte offsets, and (c) it dumps & seems to restore > (at least to /dev/null). > > I don't have a lot of options for testing it - should I just apply the > changes and wait for the problems, or can someone offer a bigendian machine > and/or a 4 byte off_t machine? Any old machine has a 4-byte off_t if you configure with --disable-largefile. This could be a neat way to test: Make two installations configured different ways and move data back and forth between them until it changes. ;-) -- Peter Eisentraut peter_e@gmx.net
-
Re: pg_dump and large files - is this a problem?
Philip Warner <pjw@rhyme.com.au> — 2002-10-19T04:15:54Z
At 12:07 AM 19/10/2002 +0200, Peter Eisentraut wrote: >Any old machine has a 4-byte off_t if you configure with >--disable-largefile. Thanks - done. I just dumped to a custom backup file, then dumped it do SQL, and compared each version (V7.2.1, 8 byte & 4 byte offsets), and they all looked OK. Also, the 4 byte version reads the 8 byte offset version correctly - although I have not checked reading > 4GB files with 4 byte offset, but it's not a priority for obvious reasons. So once Giles gets back to me (Monday), I'll commit the changes. ---------------------------------------------------------------- Philip Warner | __---_____ Albatross Consulting Pty. Ltd. |----/ - \ (A.B.N. 75 008 659 498) | /(@) ______---_ Tel: (+61) 0500 83 82 81 | _________ \ Fax: (+61) 0500 83 82 82 | ___________ | Http://www.rhyme.com.au | / \| | --________-- PGP key available upon request, | / and from pgp5.ai.mit.edu:11371 |/