Re: How To: A large [2D] matrix, 100,000+ rows/columns

On 6/8/23 22:17, Pat Trainor wrote:
> Imagine something akin to stocks, where you have a row for every stock, 
> and a column for every stock. Except where the same stock is the row & 
> col, a number is at each X-Y (row/column), and that is the big picture. 
> I need to have a very large matrix to maintain & query, and if not 
> (1,600 column limit), then how could such data be broken down to work?

  100,000 rows *
  100,000 columns *
  8 bytes (assuming float8)
= about 80 GB per matrix if I got the math correct.

Is this really a dense matrix or is it sparse? What kind of operations?

Does it really need to be stored as such or could it be stored as 
vectors that are converted to a matrix on the fly when needed?

Seems like using python or R makes more sense. Perhaps it might make 
sense to store the data in Postgres and use plpython or plr. But it is 
hard to say with more details.


-- 
Joe Conway
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com