Re: pg_dump / copy bugs with "big lines" ?
Daniel Verite <daniel@manitou-mail.org>
From: "Daniel Verite" <daniel@manitou-mail.org>
To: "Tom Lane" <tgl@sss.pgh.pa.us>
Cc: "Alvaro Herrera" <alvherre@2ndquadrant.com>,"Robert Haas"
<robertmhaas@gmail.com>,"Jim Nasby" <Jim.Nasby@bluetreble.com>,"Ronan
Dunklau" <ronan.dunklau@dalibo.com>,"pgsql-hackers"
<pgsql-hackers@postgresql.org>
Date: 2016-03-03T16:19:55Z
Lists: pgsql-hackers
Tom Lane wrote: > BTW, is anyone checking the other side of this, ie "COPY IN" with equally > wide rows? There doesn't seem to be a lot of value in supporting dump > if you can't reload ... Indeed, the lines bigger than 1GB can't be reloaded :( My test: with a stock 9.5.1 plus Alvaro's patch with my additional int64 fix mentioned upthread, creating this table and filling all columns with 200MB of text each: create table bigtext(t1 text ,t2 text, t3 text, t4 text, t5 text, t6 text, t7 text); # \copy bigtext to '/var/tmp/bigtext.sql' That part does work as expected, producing this huge single line: $ wc /var/tmp/bigtext.sql 1 7 1468006407 /var/tmp/bigtext.sql But reloading it fails: # create table bigtext2 (like bigtext); CREATE TABLE # copy bigtext2 from '/var/tmp/bigtext.sql'; ERROR: 54000: out of memory DETAIL: Cannot enlarge string buffer containing 1073676288 bytes by 65536 more bytes. CONTEXT: COPY bigtext2, line 1 LOCATION: enlargeStringInfo, stringinfo.c:278 # \copy bigtext2 from '/var/tmp/bigtext.sql' ERROR: 54000: out of memory DETAIL: Cannot enlarge string buffer containing 1073741808 bytes by 8191 more bytes. CONTEXT: COPY bigtext2, line 1 LOCATION: enlargeStringInfo, stringinfo.c:278 Best regards, -- Daniel Vérité PostgreSQL-powered mailer: http://www.manitou-mail.org Twitter: @DanielVerite
Commits
-
Fix overflow check in StringInfo; add missing casts
- 42f50cb8fa98 10.0 landed
-
Permit dump/reload of not-too-large >1GB tuples
- fa2fa9955280 10.0 landed
- 646655d264f1 9.5.6 landed
- 4e01ecae9827 9.6.2 landed