Horton

Thread

Commits

  1. Avoid catalog lookups in RelationAllowsEarlyPruning().

  1. old_snapshot_threshold's interaction with hash index

    Amit Kapila <amit.kapila16@gmail.com> — 2016-05-01T06:35:16Z

    Currently we do the test for old snapshot (TestForOldSnapshot) for hash
indexes while scanning them.  Does this test makes any sense for hash
indexes considering LSN on hash index will always be zero (as hash indexes
are not WAL-logged)?  It seems to me that PageLSN check in
TestForOldSnapshot() will always return false which means that the error
"snapshot too old" won't be generated for hash indexes.

Am I missing something here, if not, then I think we need a way to prohibit
pruning for hash indexes based on old_snapshot_threshold?


With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

  2. Re: old_snapshot_threshold's interaction with hash index

    Amit Kapila <amit.kapila16@gmail.com> — 2016-05-01T06:43:02Z

    On Sun, May 1, 2016 at 12:05 PM, Amit Kapila <amit.kapila16@gmail.com>
wrote:

> Currently we do the test for old snapshot (TestForOldSnapshot) for hash
> indexes while scanning them.  Does this test makes any sense for hash
> indexes considering LSN on hash index will always be zero (as hash indexes
> are not WAL-logged)?  It seems to me that PageLSN check in
> TestForOldSnapshot() will always return false which means that the error
> "snapshot too old" won't be generated for hash indexes.
>
> Am I missing something here, if not, then I think we need a way to
> prohibit pruning for hash indexes based on old_snapshot_threshold?
>
>
What I mean to say here is prohibit pruning the relation which has hash
index based on old_snapshot_threshold.



With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

  3. Re: old_snapshot_threshold's interaction with hash index

    Kevin Grittner <kgrittn@gmail.com> — 2016-05-02T21:02:35Z

    On Sun, May 1, 2016 at 1:43 AM, Amit Kapila <amit.kapila16@gmail.com> wrote:
> On Sun, May 1, 2016 at 12:05 PM, Amit Kapila <amit.kapila16@gmail.com> wrote:
>>
>> Currently we do the test for old snapshot (TestForOldSnapshot) for hash
>> indexes while scanning them.  Does this test makes any sense for hash
>> indexes considering LSN on hash index will always be zero (as hash indexes
>> are not WAL-logged)?  It seems to me that PageLSN check in
>> TestForOldSnapshot() will always return false which means that the error
>> "snapshot too old" won't be generated for hash indexes.
>>
>> Am I missing something here, if not, then I think we need a way to
>> prohibit pruning for hash indexes based on old_snapshot_threshold?
>
> What I mean to say here is prohibit pruning the relation which has hash
> index based on old_snapshot_threshold.

Good spot; added to the open issues page.

Thanks!

-- 
Kevin Grittner
EDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  4. Re: old_snapshot_threshold's interaction with hash index

    Bruce Momjian <bruce@momjian.us> — 2016-05-03T15:45:05Z

    On Mon, May  2, 2016 at 04:02:35PM -0500, Kevin Grittner wrote:
> On Sun, May 1, 2016 at 1:43 AM, Amit Kapila <amit.kapila16@gmail.com> wrote:
> > On Sun, May 1, 2016 at 12:05 PM, Amit Kapila <amit.kapila16@gmail.com> wrote:
> >>
> >> Currently we do the test for old snapshot (TestForOldSnapshot) for hash
> >> indexes while scanning them.  Does this test makes any sense for hash
> >> indexes considering LSN on hash index will always be zero (as hash indexes
> >> are not WAL-logged)?  It seems to me that PageLSN check in
> >> TestForOldSnapshot() will always return false which means that the error
> >> "snapshot too old" won't be generated for hash indexes.
> >>
> >> Am I missing something here, if not, then I think we need a way to
> >> prohibit pruning for hash indexes based on old_snapshot_threshold?
> >
> > What I mean to say here is prohibit pruning the relation which has hash
> > index based on old_snapshot_threshold.
> 
> Good spot; added to the open issues page.

Uh, I have no idea how this would be fixed if the PageLSN is zero.  Do
you?

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+                     Ancient Roman grave inscription +

  5. Re: old_snapshot_threshold's interaction with hash index

    Kevin Grittner <kgrittn@gmail.com> — 2016-05-03T15:46:47Z

    On Tue, May 3, 2016 at 10:45 AM, Bruce Momjian <bruce@momjian.us> wrote:
> On Mon, May  2, 2016 at 04:02:35PM -0500, Kevin Grittner wrote:
>> On Sun, May 1, 2016 at 1:43 AM, Amit Kapila <amit.kapila16@gmail.com> wrote:
>> > On Sun, May 1, 2016 at 12:05 PM, Amit Kapila <amit.kapila16@gmail.com> wrote:
>> >>
>> >> Currently we do the test for old snapshot (TestForOldSnapshot) for hash
>> >> indexes while scanning them.  Does this test makes any sense for hash
>> >> indexes considering LSN on hash index will always be zero (as hash indexes
>> >> are not WAL-logged)?  It seems to me that PageLSN check in
>> >> TestForOldSnapshot() will always return false which means that the error
>> >> "snapshot too old" won't be generated for hash indexes.
>> >>
>> >> Am I missing something here, if not, then I think we need a way to
>> >> prohibit pruning for hash indexes based on old_snapshot_threshold?
>> >
>> > What I mean to say here is prohibit pruning the relation which has hash
>> > index based on old_snapshot_threshold.
>>
>> Good spot; added to the open issues page.
>
> Uh, I have no idea how this would be fixed if the PageLSN is zero.  Do
> you?

Yes, I see three ways, the most obvious of which is what Amit
suggested -- don't do early vacuum on a table which has a hash index.

-- 
Kevin Grittner
EDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  6. Re: old_snapshot_threshold's interaction with hash index

    Robert Haas <robertmhaas@gmail.com> — 2016-05-03T16:09:05Z

    On Tue, May 3, 2016 at 11:46 AM, Kevin Grittner <kgrittn@gmail.com> wrote:
>> Uh, I have no idea how this would be fixed if the PageLSN is zero.  Do
>> you?
>
> Yes, I see three ways, the most obvious of which is what Amit
> suggested -- don't do early vacuum on a table which has a hash index.

What do you mean by "early VACUUM"?  Amit suggested disabling
HOT-pruning, but HOT-pruning happens completely outside of VACUUM.  It
also happens inside VACUUM, so if we disabled HOT pruning, how could
we VACUUM at all?  Sorry, I am confused.

Doesn't this issue also affected indexes on any unlogged table?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  7. Re: old_snapshot_threshold's interaction with hash index

    Kevin Grittner <kgrittn@gmail.com> — 2016-05-03T16:17:52Z

    On Tue, May 3, 2016 at 11:09 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Tue, May 3, 2016 at 11:46 AM, Kevin Grittner <kgrittn@gmail.com> wrote:
>>> Uh, I have no idea how this would be fixed if the PageLSN is zero.  Do
>>> you?
>>
>> Yes, I see three ways, the most obvious of which is what Amit
>> suggested -- don't do early vacuum on a table which has a hash index.
>
> What do you mean by "early VACUUM"?

Both vacuuming and hot-pruning adjust xmin based on calling
TransactionIdLimitedForOldSnapshots(TransactionId recentXmin,
Relation relation).  I'm talking about having that function, if all
other conditions for the override pass, checking for a hash index,
too.

> Amit suggested disabling
> HOT-pruning, but HOT-pruning happens completely outside of VACUUM.  It
> also happens inside VACUUM, so if we disabled HOT pruning, how could
> we VACUUM at all?  Sorry, I am confused.

I guess we were both talking a bit loosely since (as I mentioned
above) the function that adjusts the xmin is called for a vacuum or
pruning.  He mentioned one and I mentioned the other, but it's all
controlled by TransactionIdLimitedForOldSnapshots().

> Doesn't this issue also affected indexes on any unlogged table?

That's been covered all along.

--
Kevin Grittner
EDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  8. Re: old_snapshot_threshold's interaction with hash index

    Robert Haas <robertmhaas@gmail.com> — 2016-05-03T16:48:18Z

    On Tue, May 3, 2016 at 12:17 PM, Kevin Grittner <kgrittn@gmail.com> wrote:
> On Tue, May 3, 2016 at 11:09 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>> On Tue, May 3, 2016 at 11:46 AM, Kevin Grittner <kgrittn@gmail.com> wrote:
>>>> Uh, I have no idea how this would be fixed if the PageLSN is zero.  Do
>>>> you?
>>>
>>> Yes, I see three ways, the most obvious of which is what Amit
>>> suggested -- don't do early vacuum on a table which has a hash index.
>>
>> What do you mean by "early VACUUM"?
>
> Both vacuuming and hot-pruning adjust xmin based on calling
> TransactionIdLimitedForOldSnapshots(TransactionId recentXmin,
> Relation relation).  I'm talking about having that function, if all
> other conditions for the override pass, checking for a hash index,
> too.
>
>> Amit suggested disabling
>> HOT-pruning, but HOT-pruning happens completely outside of VACUUM.  It
>> also happens inside VACUUM, so if we disabled HOT pruning, how could
>> we VACUUM at all?  Sorry, I am confused.
>
> I guess we were both talking a bit loosely since (as I mentioned
> above) the function that adjusts the xmin is called for a vacuum or
> pruning.  He mentioned one and I mentioned the other, but it's all
> controlled by TransactionIdLimitedForOldSnapshots().

OK, I see now: the basic idea here is that we can't prune based on the
newer XID unless the page LSN is guaranteed to advance whenever data
is removed.  Currently, we attempt to limit bloat in non-unlogged,
non-catalog tables.  You're saying we can instead attempt to limit
bloat only in non-unlogged, non-catalog tables without hash indexes,
and that will fix this issue.  Am I right?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  9. Re: old_snapshot_threshold's interaction with hash index

    Kevin Grittner <kgrittn@gmail.com> — 2016-05-03T16:56:06Z

    On Tue, May 3, 2016 at 11:48 AM, Robert Haas <robertmhaas@gmail.com> wrote:

> OK, I see now: the basic idea here is that we can't prune based on the
> newer XID unless the page LSN is guaranteed to advance whenever data
> is removed.  Currently, we attempt to limit bloat in non-unlogged,
> non-catalog tables.  You're saying we can instead attempt to limit
> bloat only in non-unlogged, non-catalog tables without hash indexes,
> and that will fix this issue.  Am I right?

Right.

I was wondering whether there might be other avenues to the same
end, but that is the most obvious fix.  I'm hesitant to raise the
alternatives because people seem to have entered "panic mode", at
which point alternatives always look scary.

--
Kevin Grittner
EDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  10. Re: old_snapshot_threshold's interaction with hash index

    Amit Kapila <amit.kapila16@gmail.com> — 2016-05-04T03:51:15Z

    On Tue, May 3, 2016 at 9:47 PM, Kevin Grittner <kgrittn@gmail.com> wrote:
>
> On Tue, May 3, 2016 at 11:09 AM, Robert Haas <robertmhaas@gmail.com>
wrote:
> > On Tue, May 3, 2016 at 11:46 AM, Kevin Grittner <kgrittn@gmail.com>
wrote:
> >>> Uh, I have no idea how this would be fixed if the PageLSN is zero.  Do
> >>> you?
> >>
> >> Yes, I see three ways, the most obvious of which is what Amit
> >> suggested -- don't do early vacuum on a table which has a hash index.
> >
> > What do you mean by "early VACUUM"?
>
> Both vacuuming and hot-pruning adjust xmin based on calling
> TransactionIdLimitedForOldSnapshots(TransactionId recentXmin,
> Relation relation).  I'm talking about having that function, if all
> other conditions for the override pass, checking for a hash index,
> too.
>
> > Amit suggested disabling
> > HOT-pruning, but HOT-pruning happens completely outside of VACUUM.  It
> > also happens inside VACUUM, so if we disabled HOT pruning, how could
> > we VACUUM at all?  Sorry, I am confused.
>
> I guess we were both talking a bit loosely since (as I mentioned
> above) the function that adjusts the xmin is called for a vacuum or
> pruning.  He mentioned one and I mentioned the other, but it's all
> controlled by TransactionIdLimitedForOldSnapshots().
>

Yes, I think we are saying the same thing here.


With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

  11. Re: old_snapshot_threshold's interaction with hash index

    Kevin Grittner <kgrittn@gmail.com> — 2016-05-04T14:18:58Z

    On Tue, May 3, 2016 at 11:48 AM, Robert Haas <robertmhaas@gmail.com> wrote:

> OK, I see now: the basic idea here is that we can't prune based on the
> newer XID unless the page LSN is guaranteed to advance whenever data
> is removed.  Currently, we attempt to limit bloat in non-unlogged,
> non-catalog tables.  You're saying we can instead attempt to limit
> bloat only in non-unlogged, non-catalog tables without hash indexes,
> and that will fix this issue.  Am I right?

As a first cut, something like the attached.

-- 
Kevin Grittner
EDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  12. Re: old_snapshot_threshold's interaction with hash index

    Amit Kapila <amit.kapila16@gmail.com> — 2016-05-06T05:45:25Z

    On Wed, May 4, 2016 at 7:48 PM, Kevin Grittner <kgrittn@gmail.com> wrote:
>
> On Tue, May 3, 2016 at 11:48 AM, Robert Haas <robertmhaas@gmail.com>
wrote:
>
> > OK, I see now: the basic idea here is that we can't prune based on the
> > newer XID unless the page LSN is guaranteed to advance whenever data
> > is removed.  Currently, we attempt to limit bloat in non-unlogged,
> > non-catalog tables.  You're saying we can instead attempt to limit
> > bloat only in non-unlogged, non-catalog tables without hash indexes,
> > and that will fix this issue.  Am I right?
>
> As a first cut, something like the attached.
>

Patch looks good to me.  I have done some testing with hash and btree
indexes and it works as expected.


With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

  13. Re: old_snapshot_threshold's interaction with hash index

    Kevin Grittner <kgrittn@gmail.com> — 2016-05-06T13:03:31Z

    On Fri, May 6, 2016 at 12:45 AM, Amit Kapila <amit.kapila16@gmail.com>
wrote:
> On Wed, May 4, 2016 at 7:48 PM, Kevin Grittner <kgrittn@gmail.com> wrote:
>> On Tue, May 3, 2016 at 11:48 AM, Robert Haas <robertmhaas@gmail.com>
wrote:
>>
>>> OK, I see now: the basic idea here is that we can't prune based on the
>>> newer XID unless the page LSN is guaranteed to advance whenever data
>>> is removed.  Currently, we attempt to limit bloat in non-unlogged,
>>> non-catalog tables.  You're saying we can instead attempt to limit
>>> bloat only in non-unlogged, non-catalog tables without hash indexes,
>>> and that will fix this issue.  Am I right?
>>
>> As a first cut, something like the attached.
>
> Patch looks good to me.  I have done some testing with hash and
> btree indexes and it works as expected.

Pushed with the addition of a paragraph to the docs regarding this
and some other situations where people have been unclear about what
to expect.

--
Kevin Grittner
EDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company