Horton

Thread

  1. tables >2GB

    Bruce Momjian <maillist@candle.pha.pa.us> — 1998-01-29T05:32:39Z

    Now that we know the storage manager code that splits tables over 2GB
into separate files doesn't work(Irix), can we rip out that code and
just use the OS code to access >2GB files as normal files.  Now, most
OS's can support 64-bit files and file sizes.

Because it is isolated in the storage manager, it should be easy.

-- 
Bruce Momjian
maillist@candle.pha.pa.us

  2. Re: [HACKERS] tables >2GB

    Vadim B. Mikheev <vadim@sable.krasnoyarsk.su> — 1998-01-29T07:25:18Z

    Bruce Momjian wrote:
> 
> Now that we know the storage manager code that splits tables over 2GB
> into separate files doesn't work(Irix), can we rip out that code and
> just use the OS code to access >2GB files as normal files.  Now, most
> OS's can support 64-bit files and file sizes.
> 
> Because it is isolated in the storage manager, it should be easy.

Someday we'll get TABLESPACEs and fixed multi-chunk code could
allow to store chunks in different TABLESPACEs created on _different
disks_ - imho, ability to store a table on > 1 disk is good thing.

And so, I would suggest just add elog(ERROR) to mdextend() now,
with recommendation to increase RELSEG_SIZE...

Vadim

  3. Re: [HACKERS] tables >2GB

    Bruce Momjian <maillist@candle.pha.pa.us> — 1998-03-19T16:19:10Z

    > 
> Now that we know the storage manager code that splits tables over 2GB
> into separate files doesn't work(Irix), can we rip out that code and
> just use the OS code to access >2GB files as normal files.  Now, most
> OS's can support 64-bit files and file sizes.
> 
> Because it is isolated in the storage manager, it should be easy.

Can someone knowledgeable make a patch for this for our mega-patch?

-- 
Bruce Momjian                          |  830 Blythe Avenue
maillist@candle.pha.pa.us              |  Drexel Hill, Pennsylvania 19026
  +  If your life is a hard drive,     |  (610) 353-9879(w)
  +  Christ can be your backup.        |  (610) 853-3000(h)

  4. Re: [HACKERS] tables >2GB

    Marc G. Fournier <scrappy@hub.org> — 1998-03-19T17:05:47Z

    On Thu, 19 Mar 1998, Bruce Momjian wrote:

> > 
> > Now that we know the storage manager code that splits tables over 2GB
> > into separate files doesn't work(Irix), can we rip out that code and
> > just use the OS code to access >2GB files as normal files.  Now, most
> > OS's can support 64-bit files and file sizes.
> > 
> > Because it is isolated in the storage manager, it should be easy.
> 
> Can someone knowledgeable make a patch for this for our mega-patch?

	*Only* if its in before 9am AST (or is it ADT?) on Friday
morning...:)

  5. Re: [HACKERS] tables >2GB

    David Gould <dg@illustra.com> — 1998-03-19T20:09:43Z

    > > Now that we know the storage manager code that splits tables over 2GB
> > into separate files doesn't work(Irix), can we rip out that code and
> > just use the OS code to access >2GB files as normal files.  Now, most
> > OS's can support 64-bit files and file sizes.
> > 
> > Because it is isolated in the storage manager, it should be easy.
> 
> Can someone knowledgeable make a patch for this for our mega-patch?
> 

There are still quite a few OS's out there that do not support >2GB files
yet. Even my beloved Linux (x86)...

So, how about we fix the storage manager instead?

A neat thing that Illustra does is allow you to stripe a table across
multiple directories. You get big tables, easy storage management, and
a nice performance boost.

create stripedir('stripe1', '/disk1/data/stripe1');
create stripedir('stripe2', '/disk2/data/stripe2');

create table giant_table (...) with (stripes 4, 'stripe1', 'stripe2');
 -- the '4' is the number of pages to interleave.

Then the smgr just distributes the blocks alternately across the stripes.

read_block(blockno, ...stripeinfo)
{
   ...
   stripe = (blockno / stripe_interleave ) % number_of_stripes;
   stripe_block = blockno / number_of_stripes;

   fd = stripe_info->fd[stripe];
   lseek(fd, stripe_block * BLOCKSIZE, SEEK_SET);
   ...
}

All vastly oversimplified of course....

-dg

David Gould            dg@illustra.com           510.628.3783 or 510.305.9468 
Informix Software  (No, really)         300 Lakeside Drive  Oakland, CA 94612
 - I realize now that irony has no place in business communications.

  6. Re: [HACKERS] tables >2GB

    Marc G. Fournier <scrappy@hub.org> — 1998-03-19T20:14:24Z

    Does this act as a notice of volunteering to work on this aspect of the
code? *Grin*


On Thu, 19 Mar 1998, David Gould wrote:

> > > Now that we know the storage manager code that splits tables over 2GB
> > > into separate files doesn't work(Irix), can we rip out that code and
> > > just use the OS code to access >2GB files as normal files.  Now, most
> > > OS's can support 64-bit files and file sizes.
> > > 
> > > Because it is isolated in the storage manager, it should be easy.
> > 
> > Can someone knowledgeable make a patch for this for our mega-patch?
> > 
> 
> There are still quite a few OS's out there that do not support >2GB files
> yet. Even my beloved Linux (x86)...
> 
> So, how about we fix the storage manager instead?
> 
> A neat thing that Illustra does is allow you to stripe a table across
> multiple directories. You get big tables, easy storage management, and
> a nice performance boost.
> 
> create stripedir('stripe1', '/disk1/data/stripe1');
> create stripedir('stripe2', '/disk2/data/stripe2');
> 
> create table giant_table (...) with (stripes 4, 'stripe1', 'stripe2');
>  -- the '4' is the number of pages to interleave.
> 
> Then the smgr just distributes the blocks alternately across the stripes.
> 
> read_block(blockno, ...stripeinfo)
> {
>    ...
>    stripe = (blockno / stripe_interleave ) % number_of_stripes;
>    stripe_block = blockno / number_of_stripes;
> 
>    fd = stripe_info->fd[stripe];
>    lseek(fd, stripe_block * BLOCKSIZE, SEEK_SET);
>    ...
> }
> 
> All vastly oversimplified of course....
> 
> -dg
> 
> David Gould            dg@illustra.com           510.628.3783 or 510.305.9468 
> Informix Software  (No, really)         300 Lakeside Drive  Oakland, CA 94612
>  - I realize now that irony has no place in business communications.
> 
>    
>    
>

  7. Re: [HACKERS] tables >2GB

    Mattias Kregert <matti@algonet.se> — 1998-03-20T11:59:06Z

    Bruce Momjian wrote:
> 
> >
> > Now that we know the storage manager code that splits tables over 2GB
> > into separate files doesn't work(Irix), can we rip out that code and
> > just use the OS code to access >2GB files as normal files.  Now, most
> > OS's can support 64-bit files and file sizes.
> >
> > Because it is isolated in the storage manager, it should be easy.
> 
> Can someone knowledgeable make a patch for this for our mega-patch?

But, could it not be useful to be able to use multiple files per
table? Suppose someone wants to spread them out on different
disks to increase access performance?

And what about tables over 2^64 bytes size? There will never be
disks of that size? Now, remember what people said about 2^32 byte
files, and years after 1999, and 64k RAM, and about all inventions
already being invented, and... :)

/* m */