Re: CREATE INDEX CONCURRENTLY does not index prepared xact's data

Hi Noah,

On Sun, Oct 31, 2021 at 12:49 AM Noah Misch <noah@leadboat.com> wrote:

> On Sat, Oct 30, 2021 at 01:44:56PM +0530, Sandeep Thakkar wrote:
> > On Sat, Oct 30, 2021 at 1:29 PM Sandeep Thakkar <
> sandeep.thakkar@enterprisedb.com> wrote:
> > > On Sat, Oct 30, 2021 at 12:56 PM Andrey Borodin <x4mmm@yandex-team.ru>
> wrote:
> > >> Some failed runs on this animal indicate SegFault in places changed by
> > >> bugfix relevant to this thread. Fails were observed only on HEAD, but
> this
> > >> may be a result of concurrent nature of possible new bug. If there is
> a new
> > >> bug - this would affect upcoming point release. But we do not know for
> > >> sure. Other animals show no signs of any related SegFault.
> > >> Can you please run make check phase on this animal on HEAD multiple
> time
> > >> (preferably ~hundred times)? If some of this runs fail it's vital to
> > >> collect backtraces of run.
> > >
> > > OK I got it now. I've scheduled "./run_build.pl HEAD" on this animal
> to
> > > run multiple times in a day. Unfortunately, I can't run it ~100 times
> > > because it's a legacy (slow) server and also we run another animal
> (anole -
> > > with a different compiler) on the same server. Depending on the time
> every
> > > run on HEAD takes, I'll increase the frequency. Hope this helps.
> >
> > I've used "--force" option so that it ignores the last running status.
> > ./run_build.pl --verbose --force HEAD
>
> Thanks, but I don't think that gets us closer to having a stack trace for
> the
> SIGSEGV.  If some run gets a SIGSEGV, the core dump will get unlinked.  I'd
> disable both buildfarm member cron jobs and then run this script (derived
> from
> gharial's buildfarm configuration):
>
> git clone https://git.postgresql.org/git/postgresql.git
> cd postgresql
> git checkout 70bef49
> export
> LD_LIBRARY_PATH='/opt/uuid-1.6.2/inst/lib:/opt/packages/uuid-1.6.2/inst/lib:/opt/packages/krb5-1.11.3/inst/lib/hpux64:/opt/packages/libmemcached-0.46/inst/lib/hpux64:/opt/packages/libevent-2.0.10/inst/lib/hpux64:/opt/packages/expat-2.0.1/inst/lib/hpux64:/opt/packages/gdbm-1.8.3/inst/lib/hpux64:/opt/packages/openldap-2.4.24/inst/lib/hpux64:/opt/packages/proj-4.7.0/inst/lib:/opt/packages/geos-3.2.2/inst/lib:/opt/packages/db-5.1.19/inst/lib/hpux64:/opt/packages/freetype-2.4.4/inst/lib/hpux64:/opt/packages/tcltk-8.5.9/inst/lib/hpux64:/opt/packages/openssl-1.0.0d/inst/lib/hpux64:/opt/packages/editline-2.9/inst/lib/hpux64:/opt/packages/mutt-1.5.21/inst/lib/hpux64:/opt/packages/libidn-1.20/inst/lib/hpux64:/opt/packages/libxslt-1.1.26/inst/lib/hpux64:/opt/packages/libgcrypt-1.4.6/inst/lib/hpux64:/opt/packages/libgpg_error-1.10/inst/lib/hpux64:/opt/packages/libxml2-2.7.8/inst/lib/hpux64:/opt/packages/zlib-1.2.5/inst/lib/hpux64:/opt/packages/grep-2.7/inst/lib/hpux64:/opt/packages/pcre-8.12/inst/lib/hpux64:/opt/packages/ncurses-5.8/inst/lib/hpux64:/opt/packages/termcap-1.3.1/inst/lib/hpux64:/opt/packages/gettext-0.18.1.1/inst/lib/hpux64:/opt/packages/libiconv-1.13.1/inst/lib/hpux64:/opt/packages/sdk-10.2.0.5.0-hpux-ia64/instantclient_10_2/lib'
> cat >temp.conf <<EOF
> log_line_prefix = '%m [%p:%l] %q%a '
> log_connections = 'true'
> log_disconnections = 'true'
> log_statement = 'all'
> fsync = off
> force_parallel_mode = regress
> EOF
> ./configure --enable-cassert --enable-debug --with-perl --without-readline
> \
>         --with-libxml --with-libxslt
> --with-libs=/opt/packages/zlib-1.2.5/inst/lib/hpux64:/opt/packages/libxslt-1.1.26/inst/lib/hpux64:/opt/packages/libxml2-2.7.8/inst/lib/hpux64
> \
>
> --with-includes=/opt/packages/zlib-1.2.5/inst/include:/opt/packages/libxslt-1.1.26/inst/include:/opt/packages/libxml2-2.7.8/inst/include
> \
>         --with-pgport=5678 CFLAGS=-mlp64 CC=gcc
> make
> while make check TEMP_CONFIG=$PWD/temp.conf NO_LOCALE=1; do :; done
> # When the loop stops, use the core file to make a stack trace.  If it runs
> # for a week without stopping, give up.
>

Yes, that makes sense. I did what you suggested (except that I used git://
instead of https:// during cloning as the latter returns with an error;
I'll fix it later) and in the first run the core files got generated:
bash-4.1$ ls -l
./postgresql/src/test/regress/tmp_check/data/core.postgres.*
-rw------- 1 pgbfarm users 6672384 Oct 30 22:04
./postgresql/src/test/regress/tmp_check/data/core.postgres.1267
-rw------- 1 pgbfarm users 3088384 Oct 30 22:09
./postgresql/src/test/regress/tmp_check/data/core.postgres.5422

Here is the stack trace:

bash-4.1$ gdb ./postgresql/tmp_install/usr/local/pgsql/bin/postgres
> ./postgresql/src/test/regress/tmp_check/data/core.postgres.5422
>
> HP gdb 6.1 for HP Itanium (32 or 64 bit) and target HP-UX 11iv2 and 11iv3.
>
> Copyright 1986 - 2009 Free Software Foundation, Inc.
>
> Hewlett-Packard Wildebeest 6.1 (based on GDB) is covered by the
>
> GNU General Public License. Type "show copying" to see the conditions to
>
> change it and/or distribute copies. Type "show warranty" for
> warranty/support.
>
> ..
>
> Core was generated by `postgres'.
>
> Program terminated with signal 11, Segmentation fault.
>
> SEGV_MAPERR - Address not mapped to object
>
> #0  0x3fffffffff3fdbf0 in <unknown_procedure> ()
>
> (gdb) bt
>
> #0  0x3fffffffff3fdbf0 in <unknown_procedure> ()
>
> warning: Attempting to unwind past bad PC 0x3fffffffff3fdbf0
>
> #1  0x40000000003fdc00:0 in equalTupleDescs (tupdesc1=0x60000000001f65e0,
>
>     tupdesc2=0x60000000001fba08)
>
> #2  0x40000000017f9660:0 in RelationClearRelation (
>
>     relation=0x60000000001f3730, rebuild=true)
>
> #3  0x40000000017fa730:0 in RelationFlushRelation
> (relation=0x60000000001f3730)
>
> #4  0x40000000017fabb0:0 in RelationCacheInvalidateEntry
> (relationId=27272)
>
> #5  0x40000000017c8f20:0 in LocalExecuteInvalidationMessage (
>
>     msg=0x60000000001a46b8)
>
> #6  0x40000000017c84e0:0 in ProcessInvalidationMessages (
>
>     group=0x60000000001a43e4, func=0x87ffffffef7b7250)
>
> #7  0x40000000017cb420:0 in CommandEndInvalidationMessages ()
>
> #8  0x40000000006b1c50:0 in AtCCI_LocalCache ()
>
> #9  0x40000000006b0910:0 in CommandCounterIncrement ()
>
> #10 0x4000000000807130:0 in create_toast_table (rel=0x60000000001f3730,
>
>     toastOid=0, toastIndexOid=0, reloptions=0, lockmode=8, check=true,
>
>     OIDOldToast=0)
>
> #11 0x4000000000805ac0:0 in CheckAndCreateToastTable (relOid=27272,
>
>     reloptions=0, lockmode=8, check=true, OIDOldToast=0) at toasting.c:88
>
> #12 0x4000000000805850:0 in AlterTableCreateToastTable (relOid=27272,
>
>     reloptions=0, lockmode=8) at toasting.c:62
>
> #13 0x4000000000aa9a30:0 in ATRewriteCatalogs (wqueue=0x87ffffffffffc3b0,
>
>     lockmode=8, context=0x87ffffffffffc590)
>

 Since the results are not posted to the dashboard, I've attached the
script log, regression.diffs and temp.conf for reference.

-- 
Sandeep Thakkar