Horton

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

Prabhat Sahu <prabhat.sahu@enterprisedb.com>

View thread View on postgr.es
From: Prabhat Sahu <prabhat.sahu@enterprisedb.com>
To: Robert Haas <robertmhaas@gmail.com>
Cc: Peter Geoghegan <pg@bowt.ie>, Tomas Vondra <tomas.vondra@2ndquadrant.com>, Tom Lane <tgl@sss.pgh.pa.us>, Amit Kapila <amit.kapila16@gmail.com>, Thomas Munro <thomas.munro@enterprisedb.com>, Rushabh Lathia <rushabh.lathia@gmail.com>, Heikki Linnakangas <hlinnaka@iki.fi>, Pg Hackers <pgsql-hackers@postgresql.org>, Corey Huinker <corey.huinker@gmail.com>, Andres Freund <andres@anarazel.de>
Date: 2018-03-08T15:53:05Z
Lists: pgsql-hackers 
On Wed, Mar 7, 2018 at 7:51 PM, Robert Haas <robertmhaas@gmail.com> wrote:

> On Wed, Mar 7, 2018 at 8:59 AM, Prabhat Sahu <
> prabhat.sahu@enterprisedb.com> wrote:
>>
>> 2018-03-07 19:24:44.263 IST [54400] LOG:  background worker "parallel
>> worker" (PID 54482) was terminated by signal 9: Killed
>>
>
> That looks like the background worker got killed by the OOM killer.  How
> much memory do you have in the machine where this occurred?
>
I have ran the testcase in my local machine with below configurations:

Environment: CentOS 7(64bit)
HD : 100GB
RAM: 4GB
Processor: 4

I have nerrowdown the testcase as below, which also reproduce the same
crash.

-- GUCs under postgres.conf
maintenance_work_mem = 8GB

./pgbench -i -s 500 -d postgres

postgres=# create index pgb_acc_idx3 on pgbench_accounts(aid,
abalance,filler);
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the
current transaction and exit, because another server process exited
abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and
repeat your command.
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.
!>

--

With Regards,

Prabhat Kumar Sahu
Skype ID: prabhat.sahu1984
EnterpriseDB Corporation

The Postgres Database Company


--

With Regards,

Prabhat Kumar Sahu
Skype ID: prabhat.sahu1984
EnterpriseDB Corporation

The Postgres Database Company

On Thu, Mar 8, 2018 at 7:12 AM, Andres Freund <andres@anarazel.de> wrote:

>
>
> On March 7, 2018 5:40:18 PM PST, Peter Geoghegan <pg@bowt.ie> wrote:
> >On Wed, Mar 7, 2018 at 5:16 PM, Tomas Vondra
> ><tomas.vondra@2ndquadrant.com> wrote:
> >> FWIW that's usually written to the system log. Does dmesg say
> >something
> >> about the kill?
> >
> >While it would be nice to confirm that it was indeed the OOM killer,
> >either way the crash happened because SIGKILL was sent to a parallel
> >worker. There is no reason to suspect a bug.
>
> Not impossible there's a leak somewhere though.
>
> Andres
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>

Commits

  1. Support parallel btree index builds.

  2. Report an ERROR if a parallel worker fails to start properly.

  3. Transfer state pertaining to pending REINDEX operations to workers.

  4. Add a barrier primitive for synchronizing backends.

  5. Allow DML commands that create tables to use parallel query.

  6. Refactor GetOldestXmin() to use flags

  7. Fix regression in parallel planning against inheritance tables.

  8. Don't create "holes" in BufFiles, in the new logtape code.

  9. Simplify the code for logical tape read buffers.

  10. Fix excessive memory consumption in the new sort pre-reading code.

  11. Implement binary heap replace-top operation in a smarter way.

  12. Cosmetic code cleanup in commands/extension.c.

  13. Speed up planner's scanning for parallel-query hazards.

  14. Read from the same worker repeatedly until it returns no tuple.

  15. Improve tuplesort.c to support variable merge order. The original coding