Horton

Thread

Commits

  1. Fix nbtsort.c's page space accounting.

  2. Fix potential assertion failure when reindexing a pg_class index.

  3. Make heap TID a tiebreaker nbtree index column.

  1. ERROR: failed to add item to the index page

    Andreas Joseph Krogh <andreas@visena.com> — 2019-04-30T09:03:15Z

    With master (3dbb317d32f4f084ef7badaed8ef36f5c9b85fe6) I'm getting this: 
visena=# CREATE INDEX origo_email_part_hdrvl_value_idx ON 
public.origo_email_part_headervalue USING btree 
(lower(substr((header_value)::text, 0, 1000)) varchar_pattern_ops);
 psql: ERROR: failed to add item to the index page
The schema looks like this: create table origo_email_part_headervalue ( 
entity_idBIGSERIAL PRIMARY KEY, version int8 not null, header_value varchar NOT 
NULL, header_id int8 references origo_email_part_header (entity_id), value_index
int NOT NULL DEFAULT0, UNIQUE (header_id, value_index) ); CREATE INDEX 
origo_email_part_hdrvl_hdr_id_idxON origo_email_part_headervalue (header_id); 
CREATE INDEXorigo_email_part_hdrvl_value_idx ON origo_email_part_headervalue (
lower(substr(header_value, 0, 1000)) varchar_pattern_ops); (haven't tried any 
other version so I'm not sure when this started to happen) -- Andreas Joseph 
Krogh

  2. Re: ERROR: failed to add item to the index page

    Tom Lane <tgl@sss.pgh.pa.us> — 2019-04-30T13:43:16Z

    Andreas Joseph Krogh <andreas@visena.com> writes:
> With master (3dbb317d32f4f084ef7badaed8ef36f5c9b85fe6) I'm getting this: 
> visena=# CREATE INDEX origo_email_part_hdrvl_value_idx ON 
> public.origo_email_part_headervalue USING btree 
> (lower(substr((header_value)::text, 0, 1000)) varchar_pattern_ops);
>  psql: ERROR: failed to add item to the index page

Hm, your example works for me on HEAD.

Usually, the first thing to suspect when you're tracking HEAD and get
bizarre failures is that you have a messed-up build.  Before spending
any time diagnosing more carefully, do "make distclean", reconfigure,
rebuild, reinstall, then see if problem is still there.

(In theory, you can avoid this sort of failure with appropriate use
of --enable-depend, but personally I don't trust that too much.
I find that with ccache + autoconf cache + parallel build, rebuilding
completely is fast enough that it's something I just do routinely
after any git pull.  I'd rather use up my remaining brain cells on
other kinds of problems...)

			regards, tom lane

  3. Sv: Re: ERROR: failed to add item to the index page

    Andreas Joseph Krogh <andreas@visena.com> — 2019-04-30T13:49:48Z

    På tirsdag 30. april 2019 kl. 15:43:16, skrev Tom Lane <tgl@sss.pgh.pa.us 
<mailto:tgl@sss.pgh.pa.us>>: Andreas Joseph Krogh <andreas@visena.com> writes:
 > With master (3dbb317d32f4f084ef7badaed8ef36f5c9b85fe6) I'm getting this:
 > visena=# CREATE INDEX origo_email_part_hdrvl_value_idx ON
 > public.origo_email_part_headervalue USING btree
 > (lower(substr((header_value)::text, 0, 1000)) varchar_pattern_ops);
 > psql: ERROR: failed to add item to the index page

 Hm, your example works for me on HEAD.

 Usually, the first thing to suspect when you're tracking HEAD and get
 bizarre failures is that you have a messed-up build. Before spending
 any time diagnosing more carefully, do "make distclean", reconfigure,
 rebuild, reinstall, then see if problem is still there.

 (In theory, you can avoid this sort of failure with appropriate use
 of --enable-depend, but personally I don't trust that too much.
 I find that with ccache + autoconf cache + parallel build, rebuilding
 completely is fast enough that it's something I just do routinely
 after any git pull. I'd rather use up my remaining brain cells on
 other kinds of problems...)

 regards, tom lane I built with this: make distclean && ./configure 
--prefix=$HOME/programs/postgresql-master --with-openssl --with-llvm && make -j 
8 install-world-contrib-recurse install-world-doc-recurse
It's probably caused by the data: visena=# select count(*) from 
origo_email_part_headervalue;
 count
 ----------
 14609516
 (1 row)
I'll see if I can create a self contained example. 
 --
 Andreas Joseph Krogh

  4. Re: Sv: Re: ERROR: failed to add item to the index page

    Tom Lane <tgl@sss.pgh.pa.us> — 2019-04-30T13:53:50Z

    Andreas Joseph Krogh <andreas@visena.com> writes:
> I built with this: make distclean && ./configure 
> --prefix=$HOME/programs/postgresql-master --with-openssl --with-llvm && make -j 
> 8 install-world-contrib-recurse install-world-doc-recurse

--with-llvm, eh?  Does it reproduce without that?  What platform is
this on, what LLVM version?

> I'll see if I can create a self contained example. 

Please.

			regards, tom lane

  5. Sv: Re: Sv: Re: ERROR: failed to add item to the index page

    Andreas Joseph Krogh <andreas@visena.com> — 2019-04-30T14:03:04Z

    På tirsdag 30. april 2019 kl. 15:53:50, skrev Tom Lane <tgl@sss.pgh.pa.us 
<mailto:tgl@sss.pgh.pa.us>>: Andreas Joseph Krogh <andreas@visena.com> writes:
 > I built with this: make distclean && ./configure
 > --prefix=$HOME/programs/postgresql-master --with-openssl --with-llvm && 
make -j
 > 8 install-world-contrib-recurse install-world-doc-recurse

 --with-llvm, eh? Does it reproduce without that? What platform is
 this on, what LLVM version?

 > I'll see if I can create a self contained example.

 Please.

 regards, tom lane Ubuntu 19.04 $ llvm-config --version 
 8.0.0
"--with-llvm" was something I had from when pg-11 was master. It might not be 
needed anymore? I'm trying a fresh build without --with-llvm and reload of data 
now. 
 --
 Andreas Joseph Krogh

  6. Sv: Sv: Re: Sv: Re: ERROR: failed to add item to the index page

    Andreas Joseph Krogh <andreas@visena.com> — 2019-04-30T14:27:05Z

    På tirsdag 30. april 2019 kl. 16:03:04, skrev Andreas Joseph Krogh <
andreas@visena.com <mailto:andreas@visena.com>>: På tirsdag 30. april 2019 kl. 
15:53:50, skrev Tom Lane <tgl@sss.pgh.pa.us <mailto:tgl@sss.pgh.pa.us>>: 
Andreas Joseph Krogh <andreas@visena.com> writes:
 > I built with this: make distclean && ./configure
 > --prefix=$HOME/programs/postgresql-master --with-openssl --with-llvm && 
make -j
 > 8 install-world-contrib-recurse install-world-doc-recurse

 --with-llvm, eh? Does it reproduce without that? What platform is
 this on, what LLVM version?

 > I'll see if I can create a self contained example.

 Please.

 regards, tom lane Ubuntu 19.04 $ llvm-config --version 
 8.0.0
"--with-llvm" was something I had from when pg-11 was master. It might not be 
needed anymore? I'm trying a fresh build without --with-llvm and reload of data 
now. Yep, happens without --with-llvm also. I'll try to load only the necessary 
table(s) to reproduce. --
 Andreas Joseph Krogh

  7. Re: Sv: Sv: Re: Sv: Re: ERROR: failed to add item to the index page

    Andrew Dunstan <andrew.dunstan@2ndquadrant.com> — 2019-04-30T16:34:31Z

    Please fix or abstain from using the MUA that produces this monstrosity
of a Subject: "Sv: Sv: Re: Sv: Re: ERROR: failed to add item to the
index page"


cheers


andrew


-- 
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

  8. Re: ERROR: failed to add item to the index page

    Andreas Joseph Krogh <andreas@visena.com> — 2019-04-30T16:44:19Z

    På tirsdag 30. april 2019 kl. 16:27:05, skrev Andreas Joseph Krogh <
andreas@visena.com <mailto:andreas@visena.com>>: [snip] Yep, happens without 
--with-llvm also. I'll try to load only the necessary table(s) to reproduce. I 
have a 1.4GB dump (only one table) which reliably reproduces this error.
 Shall I share it off-list? --
 Andreas Joseph Krogh

  9. Re: ERROR: failed to add item to the index page

    Peter Geoghegan <pg@bowt.ie> — 2019-04-30T16:45:26Z

    On Tue, Apr 30, 2019 at 9:44 AM Andreas Joseph Krogh
<andreas@visena.com> wrote:
> I have a 1.4GB dump (only one table) which reliably reproduces this error.
> Shall I share it off-list?

I would be quite interested in this, too, since there is a chance that
it's my bug.

-- 
Peter Geoghegan

  10. Re: ERROR: failed to add item to the index page

    Tom Lane <tgl@sss.pgh.pa.us> — 2019-04-30T16:47:03Z

    Andreas Joseph Krogh <andreas@visena.com> writes:
> I have a 1.4GB dump (only one table) which reliably reproduces this error.
> Shall I share it off-list? --

That's awfully large :-(.  How do you have in mind to transmit it?

Maybe you could write a short script that generates dummy data
to reproduce the problem?

			regards, tom lane

  11. Re: ERROR: failed to add item to the index page

    Peter Geoghegan <pg@bowt.ie> — 2019-04-30T16:48:45Z

    On Tue, Apr 30, 2019 at 9:47 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Andreas Joseph Krogh <andreas@visena.com> writes:
> > I have a 1.4GB dump (only one table) which reliably reproduces this error.
> > Shall I share it off-list? --
>
> That's awfully large :-(.  How do you have in mind to transmit it?

I've send dumps that were larger than that by providing a Google drive
link. Something like that should work reasonably well.

-- 
Peter Geoghegan

  12. Re: ERROR: failed to add item to the index page

    Andreas Joseph Krogh <andreas@visena.com> — 2019-04-30T16:56:02Z

    På tirsdag 30. april 2019 kl. 18:48:45, skrev Peter Geoghegan <pg@bowt.ie 
<mailto:pg@bowt.ie>>: On Tue, Apr 30, 2019 at 9:47 AM Tom Lane 
<tgl@sss.pgh.pa.us> wrote:
 > Andreas Joseph Krogh <andreas@visena.com> writes:
 > > I have a 1.4GB dump (only one table) which reliably reproduces this error.
 > > Shall I share it off-list? --
 >
 > That's awfully large :-(. How do you have in mind to transmit it?

 I've send dumps that were larger than that by providing a Google drive
 link. Something like that should work reasonably well. I've sent you guys a 
link (Google Drive) off-list. 
 --
 Andreas Joseph Krogh

  13. Re: ERROR: failed to add item to the index page

    Peter Geoghegan <pg@bowt.ie> — 2019-04-30T16:59:19Z

    On Tue, Apr 30, 2019 at 9:56 AM Andreas Joseph Krogh <andreas@visena.com> wrote:
> I've sent you guys a link (Google Drive) off-list.

I'll start investigating the problem right away.

Thanks
-- 
Peter Geoghegan

  14. Re: ERROR: failed to add item to the index page

    Peter Geoghegan <pg@bowt.ie> — 2019-04-30T17:58:31Z

    On Tue, Apr 30, 2019 at 9:59 AM Peter Geoghegan <pg@bowt.ie> wrote:
> I'll start investigating the problem right away.

I have found what the problem is. I simply neglected to make a
conservative assumption about suffix truncation needing to add a heap
TID to a leaf page's new high key in nbtsort.c (following commit
dd299df8189), even though I didn't make the same mistake in
nbtsplitloc.c. Not sure how I managed to make such a basic error.

Andreas' test case works fine with the attached patch. I won't push a
fix for this today.

-- 
Peter Geoghegan

  15. Sv: Re: ERROR: failed to add item to the index page

    Andreas Joseph Krogh <andreas@visena.com> — 2019-04-30T18:54:45Z

    På tirsdag 30. april 2019 kl. 19:58:31, skrev Peter Geoghegan <pg@bowt.ie 
<mailto:pg@bowt.ie>>: On Tue, Apr 30, 2019 at 9:59 AM Peter Geoghegan 
<pg@bowt.ie> wrote:
 > I'll start investigating the problem right away.

 I have found what the problem is. I simply neglected to make a
 conservative assumption about suffix truncation needing to add a heap
 TID to a leaf page's new high key in nbtsort.c (following commit
 dd299df8189), even though I didn't make the same mistake in
 nbtsplitloc.c. Not sure how I managed to make such a basic error.

 Andreas' test case works fine with the attached patch. I won't push a
 fix for this today.

 --
 Peter Geoghegan Nice, thanks! --
 Andreas Joseph Krogh

  16. Re: Re: ERROR: failed to add item to the index page

    Peter Geoghegan <pg@bowt.ie> — 2019-04-30T18:55:28Z

    On Tue, Apr 30, 2019 at 11:54 AM Andreas Joseph Krogh
<andreas@visena.com> wrote:
> Nice, thanks!

Thanks for the report!

-- 
Peter Geoghegan

  17. Re: ERROR: failed to add item to the index page

    Andres Freund <andres@anarazel.de> — 2019-04-30T18:59:43Z

    Hi,

On 2019-04-30 20:54:45 +0200, Andreas Joseph Krogh wrote:
> På tirsdag 30. april 2019 kl. 19:58:31, skrev Peter Geoghegan <pg@bowt.ie 
> <mailto:pg@bowt.ie>>: On Tue, Apr 30, 2019 at 9:59 AM Peter Geoghegan 
> <pg@bowt.ie> wrote:
>  > I'll start investigating the problem right away.
> 
>  I have found what the problem is. I simply neglected to make a
>  conservative assumption about suffix truncation needing to add a heap
>  TID to a leaf page's new high key in nbtsort.c (following commit
>  dd299df8189), even though I didn't make the same mistake in
>  nbtsplitloc.c. Not sure how I managed to make such a basic error.
> 
>  Andreas' test case works fine with the attached patch. I won't push a
>  fix for this today.
> 
>  --
>  Peter Geoghegan Nice, thanks! --
>  Andreas Joseph Krogh

Andreas, unfortunately your emails are pretty unreadable. Check the
quoted email, and the web archive:

https://www.postgresql.org/message-id/VisenaEmail.41.51d7719d814a1f54.16a6f98a5e9%40tc7-visena

Greetings,

Andres Freund

  18. Re: ERROR: failed to add item to the index page

    Andreas Joseph Krogh <andreas@visena.com> — 2019-04-30T19:23:21Z

    På tirsdag 30. april 2019 kl. 20:59:43, skrev Andres Freund <andres@anarazel.de 
<mailto:andres@anarazel.de>>: [...]
 Andreas, unfortunately your emails are pretty unreadable. Check the
 quoted email, and the web archive:

 
https://www.postgresql.org/message-id/VisenaEmail.41.51d7719d814a1f54.16a6f98a5e9%40tc7-visena

 Greetings,

 Andres Freund
I know that the text-version is quite unreadable, especially when quoting. My 
MUA is web-based and uses CKEditor for composing, and it doesn't care much to 
try to format the text/plain version (I know because I've written it, yes and 
have yet to fix the Re: Sv: Re: Sv: subject issue...). But it has tons of 
benefits CRM- and usage-wise so I prefer to use it. But - how use text/plain 
these days:-) --
 Andreas Joseph Krogh

  19. Re: ERROR: failed to add item to the index page

    Andres Freund <andres@anarazel.de> — 2019-04-30T19:26:52Z

    Hi,

On 2019-04-30 21:23:21 +0200, Andreas Joseph Krogh wrote:
> På tirsdag 30. april 2019 kl. 20:59:43, skrev Andres Freund <andres@anarazel.de 
> <mailto:andres@anarazel.de>>: [...]
>  Andreas, unfortunately your emails are pretty unreadable. Check the
>  quoted email, and the web archive:
> 
>  
> https://www.postgresql.org/message-id/VisenaEmail.41.51d7719d814a1f54.16a6f98a5e9%40tc7-visena
> 
>  Greetings,
> 
>  Andres Freund

> I know that the text-version is quite unreadable, especially when quoting. My 
> MUA is web-based and uses CKEditor for composing, and it doesn't care much to 
> try to format the text/plain version (I know because I've written it, yes and 
> have yet to fix the Re: Sv: Re: Sv: subject issue...). But it has tons of 
> benefits CRM- and usage-wise so I prefer to use it. But - how use text/plain 
> these days:-) --

The standard on pg lists is to write in a manner that's usable for both
text mail readers and the archive. Doesn't terribly matter to the
occasional one-off poster on -general, but you're not that...  So please
try to write readable mails for the PG lists.

Greetings,

Andres Freund

  20. Sv: Re: ERROR: failed to add item to the index page

    Andreas Joseph Krogh <andreas@visena.com> — 2019-04-30T19:33:26Z

    På tirsdag 30. april 2019 kl. 21:26:52, skrev Andres Freund <andres@anarazel.de 
<mailto:andres@anarazel.de>>: > [...]
> The standard on pg lists is to write in a manner that's usable for both > 
text mail readers and the archive. Doesn't terribly matter to the > occasional 
one-off poster on -general, but you're not that... So please > try to write 
readable mails for the PG lists.
> 
> Greetings,
 > 
> Andres Freund ACK. --
 Andreas Joseph Krogh

  21. Re: ERROR: failed to add item to the index page

    Peter Geoghegan <pg@bowt.ie> — 2019-05-01T01:28:11Z

    On Tue, Apr 30, 2019 at 10:58 AM Peter Geoghegan <pg@bowt.ie> wrote:j
> I have found what the problem is. I simply neglected to make a
> conservative assumption about suffix truncation needing to add a heap
> TID to a leaf page's new high key in nbtsort.c (following commit
> dd299df8189), even though I didn't make the same mistake in
> nbtsplitloc.c. Not sure how I managed to make such a basic error.

Attached is a much more polished version of the same patch. I tried to
make clear how the "page full" test (the test that has been fixed to
take heap TID space for high key into account) is related to other
close-by code, such as the tuple space limit budget within
_bt_check_third_page(), and the code that sets up an actual call to
_bt_truncate().

I'll wait a few days before pushing this. This version doesn't feel
too far off being committable. I tested it with some of the CREATE
INDEX tests that I developed during development of the nbtree unique
keys project, including a test with tuples that are precisely at the
1/3 of a page threshold. The new definition of 1/3 of a page takes
high key heap TID overhead into account -- see _bt_check_third_page().

-- 
Peter Geoghegan

  22. Re: ERROR: failed to add item to the index page

    Peter Geoghegan <pg@bowt.ie> — 2019-05-02T19:38:02Z

    On Tue, Apr 30, 2019 at 6:28 PM Peter Geoghegan <pg@bowt.ie> wrote:
> Attached is a much more polished version of the same patch. I tried to
> make clear how the "page full" test (the test that has been fixed to
> take heap TID space for high key into account) is related to other
> close-by code, such as the tuple space limit budget within
> _bt_check_third_page(), and the code that sets up an actual call to
> _bt_truncate().

Pushed, though final version does the test a little differently. It
adds the required heap TID space to itupsz, rather than subtracting it
from pgspc. This is actually representative of the underlying logic,
and avoids unsigned underflow.

-- 
Peter Geoghegan

  23. Re: ERROR: failed to add item to the index page

    Andreas Joseph Krogh <andreas@visena.com> — 2019-05-02T19:41:46Z

    På torsdag 02. mai 2019 kl. 21:38:02, skrev Peter Geoghegan <pg@bowt.ie>:
 > Pushed, though final version does the test a little differently. It
 > adds the required heap TID space to itupsz, rather than subtracting it
 > from pgspc. This is actually representative of the underlying logic,
 > and avoids unsigned underflow. Thanks! 
 --
 Andreas Joseph Krogh