Horton

Thread

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Updates to contrib/isn ISBN tables.

  2. Add a "LIKE = typename" clause to CREATE TYPE for base types. This allows

  1. Core Extensions relocation

    Greg Smith <greg@2ndquadrant.com> — 2011-06-09T04:14:53Z

    Following up on the idea we've been exploring for making some extensions 
more prominent, attached is the first rev that I think may be worth 
considering seriously.  Main improvement from the last is that I 
reorganized the docs to break out what I decided to tentatively name 
"Core Extensions" into their own chapter.  No longer mixed in with the 
rest of the contrib modules, and I introduce them a bit differently.   
If you want to take a quick look at the new page, I copied it to 
http://www.2ndquadrant.us/docs/html/extensions.html

I'm not completely happy on the wordering there yet.  The use of both 
"modules" and "extensions" is probably worth eliminating, and maybe that 
continues on to doing that against the language I swiped from the 
contrib intro too.  There's also a lot of shared text at the end there, 
common wording from that and the contrib page about how to install and 
migrate these extensions.  Not sure how to refactor it out into another 
section cleanly though.

Regression tests came up last time I posted this.  Doesn't look like 
there are any for the modules I'm suggesting should be promoted.  Only 
code issue I noticed during another self-review here is that I didn't 
rename contrib/pgrowlocks/pgrowlocks--unpackaged--1.0.sql cleanly, may 
need to do that one over again to get the commits as clean as possible.

Updated code is at 
https://github.com/greg2ndQuadrant/postgres/tree/move-contrib too, and 
since this is painful as a patch the compare view at 
https://github.com/greg2ndQuadrant/postgres/compare/master...move-contrib 
will be easier for browsing the code changes.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us

  2. Re: Core Extensions relocation

    Vinicius Abrahao <vinnix.bsd@gmail.com> — 2011-06-09T15:14:19Z

    Hello Greg, hello All,

This is my first post at Hackers, so sorry if I am been a noob here, but I
am pretty confused about
how to create the extension pg_buffercache.

First of all, I was trying to create using the old method by calling the
pg_buffercache--1.0.sql directly.
Then I discover the change that occurs recently to use CREATE EXTENSION, but
even now I am getting the weird error:

# select * from pg_available_extensions;
      name      | default_version | installed_version |             comment
----------------+-----------------+-------------------+---------------------------------
 plpgsql        | 1.0             | 1.0               | PL/pgSQL
procedural language
 pg_buffercache | 1.0             |                   | examine the
shared buffer cache
(2 rows)

postgres=# CREATE EXTENSION pg_buffercache SCHEMA pg_catalog;
ERROR:  syntax error at or near "NO"

Right now, talking with some fellows at #postgresql they tell that the error
is NOT occurring for they.
This was about 9.1beta from git.

But even so, I need to ask, because my production is on another versions:

What is the right way to install this contrib at 9.0.1, 9.0.2 and 9.0.4 ?



Many thanks,

Best regards,
vinnix



On Thu, Jun 9, 2011 at 1:14 AM, Greg Smith <greg@2ndquadrant.com> wrote:

> Following up on the idea we've been exploring for making some extensions
> more prominent, attached is the first rev that I think may be worth
> considering seriously.  Main improvement from the last is that I reorganized
> the docs to break out what I decided to tentatively name "Core Extensions"
> into their own chapter.  No longer mixed in with the rest of the contrib
> modules, and I introduce them a bit differently.   If you want to take a
> quick look at the new page, I copied it to
> http://www.2ndquadrant.us/docs/html/extensions.html
>
> I'm not completely happy on the wordering there yet.  The use of both
> "modules" and "extensions" is probably worth eliminating, and maybe that
> continues on to doing that against the language I swiped from the contrib
> intro too.  There's also a lot of shared text at the end there, common
> wording from that and the contrib page about how to install and migrate
> these extensions.  Not sure how to refactor it out into another section
> cleanly though.
>
> Regression tests came up last time I posted this.  Doesn't look like there
> are any for the modules I'm suggesting should be promoted.  Only code issue
> I noticed during another self-review here is that I didn't rename
> contrib/pgrowlocks/pgrowlocks--unpackaged--1.0.sql cleanly, may need to do
> that one over again to get the commits as clean as possible.
>
> Updated code is at
> https://github.com/greg2ndQuadrant/postgres/tree/move-contrib too, and
> since this is painful as a patch the compare view at
> https://github.com/greg2ndQuadrant/postgres/compare/master...move-contribwill be easier for browsing the code changes.
>
> --
> Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
> PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us
>
>
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>
>


-- 

Vinícius Abrahão Bazana Schmidt
Desenvolvimento
Dextra Sistemas
www.dextra.com.br
+55 19 3256-6722 Ramal 246

Este email é confidencial. Mais informações em:
This message is confidential. More information at:
www.dextra.com.br/confidencial.htm


--
vi[nnix]™
aka: Vinícius Abrahão Bazana Schmidt
vischmidt.wordpress.com
twitter.com/vischmidt

  3. Re: Core Extensions relocation

    Greg Smith <greg@2ndquadrant.com> — 2011-06-09T15:51:01Z

    Vinicius Abrahao wrote:
> This is my first post at Hackers, so sorry if I am been a noob here, 
> but I am pretty confused about
> how to create the extension pg_buffercache.

This list is for talking about development of new features, normally on 
the latest development version of the software (right now 9.1).  There 
is no such thing as CREATE EXTENSION in versions before that.  A 
question like "how do I install pg_buffercache for 9.0?" should normally 
get sent to one of the other mailing lists; any of pgsql-performance, 
pgsql-admin, or pgsql-general would be appropriate to ask that at.  This 
one really isn't.  It's also better to avoid taking someone else's 
discussion and replying to it with your questions.

> But even so, I need to ask, because my production is on another versions:
> What is the right way to install this contrib at 9.0.1, 9.0.2 and 9.0.4 ?

But since I happen to know this answer, here's an example from a RedHat 
derived Linux system running PostgreSQL 9.0.4, logged in as the postgres 
user:

-bash-3.2$ locate pg_buffercache.sql
/usr/pgsql-9.0/share/contrib/pg_buffercache.sql
/usr/pgsql-9.0/share/contrib/uninstall_pg_buffercache.sql
-bash-3.2$ psql -d pgbench -f 
/usr/pgsql-9.0/share/contrib/pg_buffercache.sql
SET
CREATE FUNCTION
CREATE VIEW
REVOKE
REVOKE
-bash-3.2$ psql -d pgbench -c "select count(*) from pg_buffercache"
 count
-------
  4096

The location of the file will be different on other platforms, but 
that's the basic idea of how you install it.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us

  4. Re: Core Extensions relocation

    Tom Lane <tgl@sss.pgh.pa.us> — 2011-06-09T15:54:19Z

    Please do not piggyback on an unrelated thread to ask a question.
Start a new thread.

Vinicius Abrahao <vinnix.bsd@gmail.com> writes:
> postgres=# CREATE EXTENSION pg_buffercache SCHEMA pg_catalog;
> ERROR:  syntax error at or near "NO"

This looks like a syntax error in the pg_buffercache--1.0.sql file ...
have you tampered with that at all?

I believe BTW that you cannot specify pg_catalog as the target schema
here.  When I try that, I get:

regression=# CREATE EXTENSION pg_buffercache SCHEMA pg_catalog;
ERROR:  permission denied to create "pg_catalog.pg_buffercache"
DETAIL:  System catalog modifications are currently disallowed.

but it goes through fine without the SCHEMA clause.

			regards, tom lane

  5. Re: Core Extensions relocation

    Dimitri Fontaine <dimitri@2ndquadrant.fr> — 2011-06-10T11:21:13Z

    Hi,

Greg Smith <greg@2ndquadrant.com> writes:
> Following up on the idea we've been exploring for making some extensions
> more prominent, attached is the first rev that I think may be worth
> considering seriously.  Main improvement from the last is that I reorganized
> the docs to break out what I decided to tentatively name "Core Extensions"
> into their own chapter.  No longer mixed in with the rest of the contrib
> modules, and I introduce them a bit differently.   If you want to take a
> quick look at the new page, I copied it to
> http://www.2ndquadrant.us/docs/html/extensions.html

Thanks a lot for working on this!

I have two remarks here.  First, I think that the “core extensions” (+1
for this naming) should not be found in a documentation appendix, but in
the main documentation, as a new Chapter in Part II where we list data
types and operators and system functions.  Between current chapters 9
and 10 would be my vote.

Then, I think the angle to use to present “core extensions” would be
that those are things maintained like the core server, but that you
might not need at all.  There's no SQL level feature that rely on them,
it's all “extra”, but it's there nonetheless, because you might need it.

Other than that, +1 for your patch.  I'd stress out that I support your
idea to split the extensions at the source level, I think that's really
helping to get the message out: that is trustworthy and maintained code.

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support

  6. Re: Core Extensions relocation

    Peter Eisentraut <peter_e@gmx.net> — 2011-06-11T05:09:45Z

    On tor, 2011-06-09 at 00:14 -0400, Greg Smith wrote:
> Following up on the idea we've been exploring for making some
> extensions 
> more prominent, attached is the first rev that I think may be worth 
> considering seriously.  Main improvement from the last is that I 
> reorganized the docs to break out what I decided to tentatively name 
> "Core Extensions" into their own chapter.  No longer mixed in with
> the 
> rest of the contrib modules, and I introduce them a bit
> differently.   

For the directory name, I'd prefer either src/extensions (since there is
more than one), or if you want to go for short somehow, src/ext.  (Hmm,
I guess the installation subdirectory is also called "extension".  But
it felt wrong on first reading anyway.)

There is some funny business in your new src/extension/Makefile.  You
apparently based this on a very old version of contrib/Makefile (if
still contains a CVS keyword header), it uses for loops in make targets
after we just got rid of them, and it references some modules that
aren't there at all.  That file needs a complete redo based on current
sources, I think.

Equally, your new extension-global.mk sets MODULEDIR, which is no longer
necessary, and has a CVS header.  What version did you branch this
off? :)

Perhaps a small addition to the installation instructions would also be
appropriate, to tell people that certain core extensions, as it were,
are installed by default.

  7. Re: Core Extensions relocation

    Greg Smith <greg@2ndquadrant.com> — 2011-06-11T16:38:03Z

    Peter Eisentraut wrote:
> For the directory name, I'd prefer either src/extensions (since there is
> more than one), or if you want to go for short somehow, src/ext.  (Hmm,
> I guess the installation subdirectory is also called "extension".  But
> it felt wrong on first reading anyway.)
>   

I jumped between those two a couple of times myself, settling on 
"extension" to match the installation location as you figured out.  
Assuming that name shouldn't change at this point, this seemed the best 
way to name the new directory, even though I agree it seems weird at first.


> What version did you branch this off? :)
>   

Long enough ago that apparently I've missed some major changes; Magnus 
already pointed out I needed to revisit how MODULEDIR was used.  Looks 
like I need to rebuild the first patch in this series yet again, which 
shouldn't be too bad.  The second time I did that, I made the commits 
atomic enough that the inevitable third one would be easy.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us

  8. Re: Core Extensions relocation

    Robert Haas <robertmhaas@gmail.com> — 2011-07-05T15:31:38Z

    On Sat, Jun 11, 2011 at 12:38 PM, Greg Smith <greg@2ndquadrant.com> wrote:
> Peter Eisentraut wrote:
>> For the directory name, I'd prefer either src/extensions (since there is
>> more than one), or if you want to go for short somehow, src/ext.  (Hmm,
>> I guess the installation subdirectory is also called "extension".  But
>> it felt wrong on first reading anyway.)
>
> I jumped between those two a couple of times myself, settling on "extension"
> to match the installation location as you figured out.  Assuming that name
> shouldn't change at this point, this seemed the best way to name the new
> directory, even though I agree it seems weird at first.
>
>> What version did you branch this off? :)
>
> Long enough ago that apparently I've missed some major changes; Magnus
> already pointed out I needed to revisit how MODULEDIR was used.  Looks like
> I need to rebuild the first patch in this series yet again, which shouldn't
> be too bad.  The second time I did that, I made the commits atomic enough
> that the inevitable third one would be easy.

Are you going to do this work for this CommitFest?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  9. Re: Core Extensions relocation

    Bruce Momjian <bruce@momjian.us> — 2011-10-14T16:48:03Z

    Is this going to be done for 9.2?

---------------------------------------------------------------------------

Greg Smith wrote:
> Following up on the idea we've been exploring for making some extensions 
> more prominent, attached is the first rev that I think may be worth 
> considering seriously.  Main improvement from the last is that I 
> reorganized the docs to break out what I decided to tentatively name 
> "Core Extensions" into their own chapter.  No longer mixed in with the 
> rest of the contrib modules, and I introduce them a bit differently.   
> If you want to take a quick look at the new page, I copied it to 
> http://www.2ndquadrant.us/docs/html/extensions.html
> 
> I'm not completely happy on the wordering there yet.  The use of both 
> "modules" and "extensions" is probably worth eliminating, and maybe that 
> continues on to doing that against the language I swiped from the 
> contrib intro too.  There's also a lot of shared text at the end there, 
> common wording from that and the contrib page about how to install and 
> migrate these extensions.  Not sure how to refactor it out into another 
> section cleanly though.
> 
> Regression tests came up last time I posted this.  Doesn't look like 
> there are any for the modules I'm suggesting should be promoted.  Only 
> code issue I noticed during another self-review here is that I didn't 
> rename contrib/pgrowlocks/pgrowlocks--unpackaged--1.0.sql cleanly, may 
> need to do that one over again to get the commits as clean as possible.
> 
> Updated code is at 
> https://github.com/greg2ndQuadrant/postgres/tree/move-contrib too, and 
> since this is painful as a patch the compare view at 
> https://github.com/greg2ndQuadrant/postgres/compare/master...move-contrib 
> will be easier for browsing the code changes.
> 
> -- 
> Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
> PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us
> 
> 


> 
> -- 
> Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

  10. Re: Core Extensions relocation

    Thom Brown <thom@linux.com> — 2011-10-14T17:13:06Z

    On 14 October 2011 17:48, Bruce Momjian <bruce@momjian.us> wrote:
>
> Is this going to be done for 9.2?
>
> ---------------------------------------------------------------------------

I didn't spot this thread before.  I posted something related
yesterday: http://archives.postgresql.org/pgsql-hackers/2011-10/msg00781.php

-- 
Thom Brown
Twitter: @darkixion
IRC (freenode): dark_ixion
Registered Linux user: #516935

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  11. Re: Core Extensions relocation

    Greg Smith <greg@2ndquadrant.com> — 2011-11-02T15:25:01Z

    On 10/14/2011 01:48 PM, Bruce Momjian wrote:
> Is this going to be done for 9.2?
>    

Refreshing this patch is on my list of things to finish before the next 
CommitFest starts later this month.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us

  12. Re: Core Extensions relocation

    Josh Berkus <josh@agliodbs.com> — 2011-11-02T17:59:50Z

    On 11/2/11 8:25 AM, Greg Smith wrote:
> On 10/14/2011 01:48 PM, Bruce Momjian wrote:
>> Is this going to be done for 9.2?
>>    
> 
> Refreshing this patch is on my list of things to finish before the next
> CommitFest starts later this month.

Put me down as reviewer.


-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

  13. Re: Core Extensions relocation

    Greg Smith <greg@2ndquadrant.com> — 2011-11-14T09:08:29Z

    I've revived the corpose of the patch submitted in May, now that it's a 
much less strange time of the development cycle to consider it.  
http://archives.postgresql.org/message-id/4DF048BD.8040302@2ndquadrant.com 
was the first attempt to move some extensions from contrib/ to a new 
src/extension/ directory.  I have fixed the main complaints from the 
last submit attempt, that I accidentally grabbed some old makesfiles and 
CVS junk.  The new attempt is attached, and is easiest to follow with 
the a diff view that understands "moved a file", like github's:  
https://github.com/greg2ndQuadrant/postgres/compare/master...core-extensions

You can also check out the docs changes done so far at 
http://www.highperfpostgres.com/docs/html/extensions.html  I reorganized 
the docs to break out what I decided to tentatively name "Core 
Extensions" into their own chapter.  They're no longer mixed in with the 
rest of the contrib modules, and I introduce them a bit differently.  
I'm not completely happy on the wordering there yet.  The use of both 
"modules" and "extensions" is probably worth eliminating, and maybe that 
continues on to doing that against the language I swiped from the 
contrib intro too.  There's also a lot of shared text at the end there, 
common wording from that and the contrib page about how to install and 
migrate these extensions.  Not sure how to refactor it out into another 
section cleanly though.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us

  14. Re: Core Extensions relocation

    Thom Brown <thom@linux.com> — 2011-11-14T22:06:39Z

    On 14 November 2011 09:08, Greg Smith <greg@2ndquadrant.com> wrote:
> I've revived the corpose of the patch submitted in May, now that it's a much
> less strange time of the development cycle to consider it.
>  http://archives.postgresql.org/message-id/4DF048BD.8040302@2ndquadrant.com
> was the first attempt to move some extensions from contrib/ to a new
> src/extension/ directory.  I have fixed the main complaints from the last
> submit attempt, that I accidentally grabbed some old makesfiles and CVS
> junk.  The new attempt is attached, and is easiest to follow with the a diff
> view that understands "moved a file", like github's:
>  https://github.com/greg2ndQuadrant/postgres/compare/master...core-extensions
>
> You can also check out the docs changes done so far at
> http://www.highperfpostgres.com/docs/html/extensions.html  I reorganized the
> docs to break out what I decided to tentatively name "Core Extensions" into
> their own chapter.  They're no longer mixed in with the rest of the contrib
> modules, and I introduce them a bit differently.  I'm not completely happy
> on the wordering there yet.  The use of both "modules" and "extensions" is
> probably worth eliminating, and maybe that continues on to doing that
> against the language I swiped from the contrib intro too.  There's also a
> lot of shared text at the end there, common wording from that and the
> contrib page about how to install and migrate these extensions.  Not sure
> how to refactor it out into another section cleanly though.

I'm all for removing all mention of "modules".  It's ambiguous and
used inconsistently.

In my previous post in this area
(http://archives.postgresql.org/pgsql-hackers/2011-10/msg00781.php) I
suggested that bundling tools, libraries and extensions together in
the same category is confusing.  So those are still a problem for me.
And auto_explain appears in your new "Core Extensions" section, but
it's not an extension in the terminology PostgreSQL uses, so that's
also potentially confusing.

-- 
Thom Brown
Twitter: @darkixion
IRC (freenode): dark_ixion
Registered Linux user: #516935

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  15. Re: Core Extensions relocation

    Dimitri Fontaine <dimitri@2ndquadrant.fr> — 2011-11-14T22:34:12Z

    Thom Brown <thom@linux.com> writes:
> I'm all for removing all mention of "modules".  It's ambiguous and
> used inconsistently.

The module is the shared library object.  It should be possible to use
that consistently.  And I have some plans on my TODO list about them
anyway, so making them disappear from the manual would not serve my
later plans :)

> And auto_explain appears in your new "Core Extensions" section, but
> it's not an extension in the terminology PostgreSQL uses, so that's
> also potentially confusing.

This is a related problem, we should have a terminology for contrib
tools such as pg_standby or pg_archivecleanup, for modules like the one
you talk about, that provide new features but nothing visible from SQL,
and extensions, that are all about SQL --- and if I can work on my plans
will get even more about SQL in a near future.

It's too late for me today to contribute nice ideas here though.

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support

  16. Re: Core Extensions relocation

    Josh Berkus <josh@agliodbs.com> — 2011-11-15T00:46:40Z

    > This is a related problem, we should have a terminology for contrib
> tools such as pg_standby or pg_archivecleanup, for modules like the one
> you talk about, that provide new features but nothing visible from SQL,
> and extensions, that are all about SQL --- and if I can work on my plans
> will get even more about SQL in a near future.

I see nothing wrong with "Tools" and "Extensions".  I'm not sure that
having one catch-all name for them serves the user.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

  17. Re: Core Extensions relocation

    Josh Berkus <josh@agliodbs.com> — 2011-11-15T00:56:05Z

    Greg,

So I'm a bit unclear on why most of the optional data types were
excluded from your list of Core Extensions.  I would regard the
following as stable and of general utility:

btree_gin
btree_gist
citext
dblink
file_fdw
fuzzystrmatch
hstore
intarray
isn
ltree
pgcrypto
pg_trgm
unaccent
uuid-ossp

These should, in my opinion, all be Core Extensions.  I'd go further to
say that if something is materially an extension (as opposed to a tool
or a code example), and we're shipping it with the core distribution, it
either ought to be a core extension, or it should be kicked out to PGXN.

Am I completely misunderstanding what you're trying to accomplish here?

... also, why is there still a "tsearch2" contrib module around at all?

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

  18. Re: Core Extensions relocation

    Thom Brown <thom@linux.com> — 2011-11-15T01:14:08Z

    On 15 November 2011 00:56, Josh Berkus <josh@agliodbs.com> wrote:
> Greg,
>
> So I'm a bit unclear on why most of the optional data types were
> excluded from your list of Core Extensions.  I would regard the
> following as stable and of general utility:
>
> btree_gin
> btree_gist
> citext
> dblink
> file_fdw
> fuzzystrmatch
> hstore
> intarray
> isn
> ltree
> pgcrypto
> pg_trgm
> unaccent
> uuid-ossp

Greg clarified on the core extensions page text:

"These core extensions supply useful features in areas such as
database diagnostics and performance monitoring."

None of those others perform such a role.  Instead they add additional
functionality intended to be utilised as part of general data usage,
adding new types, operators, query functions etc.  Maybe the term
"core" is inappropriate.  Instead we might wish to refer to them as
"utility extensions" or something like that, although that may be just
as vague.

> ... also, why is there still a "tsearch2" contrib module around at all?

Backwards compatibility.  No-one will use it except if they're coming
from an older version.

-- 
Thom Brown
Twitter: @darkixion
IRC (freenode): dark_ixion
Registered Linux user: #516935

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  19. Re: Core Extensions relocation

    Peter Geoghegan <peter@2ndquadrant.com> — 2011-11-15T01:24:23Z

    On 15 November 2011 00:56, Josh Berkus <josh@agliodbs.com> wrote:
> So I'm a bit unclear on why most of the optional data types were
> excluded from your list of Core Extensions.  I would regard the
> following as stable and of general utility:

> isn

I consider contrib/isn to be quite broken. It hard codes ISBN prefixes
for the purposes of sanitising ISBNs, even though their assignment is
actually controlled by a decentralised body of regional authorities.
I'd vote for kicking it out of contrib.

-- 
Peter Geoghegan       http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training and Services

  20. Re: Core Extensions relocation

    Greg Smith <greg@2ndquadrant.com> — 2011-11-15T01:44:50Z

    On 11/14/2011 07:56 PM, Josh Berkus wrote:
> So I'm a bit unclear on why most of the optional data types were
> excluded from your list of Core Extensions.

I was aiming for the extensions that seemed uncontroversial for a first 
pass here.  One of the tests I applied was "do people sometimes need 
this module after going into production with their application?"  The 
very specific problem I was most concerned about eliminating was people 
discovering they needed an extension to troubleshoot performance or 
corruption issues, only to discover it wasn't available--because they 
hadn't installed the postgresql-contrib package.  New package 
installation can be a giant pain to get onto a production system in some 
places, if it wasn't there during QA etc.

All of the data type extensions fail that test.  If you need one of 
those, you would have discovered that on your development server, and 
made sure the contrib package was available on production too.  There 
very well may be some types that should be rolled into the core 
extensions list, but I didn't want arguments over that to block moving 
forward with the set I did suggest.  We can always move more of them 
later, if this general approach is accepted.  It only takes about 5 
minutes per extension to move them from contrib to src/extension, once 
the new directory tree and doc section is there.  But I didn't want to 
do the work of moving another 15 of them if the whole idea was going to 
get shot down.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us

  21. Re: Core Extensions relocation

    Robert Haas <robertmhaas@gmail.com> — 2011-11-15T03:09:38Z

    On Mon, Nov 14, 2011 at 8:44 PM, Greg Smith <greg@2ndquadrant.com> wrote:
> On 11/14/2011 07:56 PM, Josh Berkus wrote:
>>
>> So I'm a bit unclear on why most of the optional data types were
>> excluded from your list of Core Extensions.
>
> I was aiming for the extensions that seemed uncontroversial for a first pass
> here.  One of the tests I applied was "do people sometimes need this module
> after going into production with their application?"  The very specific
> problem I was most concerned about eliminating was people discovering they
> needed an extension to troubleshoot performance or corruption issues, only
> to discover it wasn't available--because they hadn't installed the
> postgresql-contrib package.  New package installation can be a giant pain to
> get onto a production system in some places, if it wasn't there during QA
> etc.
>
> All of the data type extensions fail that test.  If you need one of those,
> you would have discovered that on your development server, and made sure the
> contrib package was available on production too.  There very well may be
> some types that should be rolled into the core extensions list, but I didn't
> want arguments over that to block moving forward with the set I did suggest.
>  We can always move more of them later, if this general approach is
> accepted.  It only takes about 5 minutes per extension to move them from
> contrib to src/extension, once the new directory tree and doc section is
> there.  But I didn't want to do the work of moving another 15 of them if the
> whole idea was going to get shot down

I continue to think that we should be trying to sort these by subject
matter.  The term "core extensions" doesn't convey that these are
server management and debugging tools, hence Josh's confusion.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  22. Re: Core Extensions relocation

    Greg Smith <greg@2ndquadrant.com> — 2011-11-15T03:47:39Z

    On 11/14/2011 10:09 PM, Robert Haas wrote:
> I continue to think that we should be trying to sort these by subject
> matter.  The term "core extensions" doesn't convey that these are
> server management and debugging tools, hence Josh's confusion.
>    

I'm not attached to the name, which I just pulled out of the air for the 
documentation.  Could just as easily call them built-in modules or 
extensions.  If the objection is that "extensions" isn't technically 
correct for auto-explain, you might call them core add-ons instead.  My 
thinking was that the one exception didn't make it worth the trouble to 
introduce a new term altogether here.  There's already too many terms 
used for talking about this sort of thing, the confusion from using a 
word other than "extensions" seemed larger than the confusion sown by 
auto-explain not fitting perfectly.

The distinction I care about here is primarily a packaging one.  These 
are server additions that people should be able to count on having 
available, whereas right now they may or may not be installed depending 
on if contrib was added.  Everything I'm touching requires our RPM and 
Debian packagers (at least) make a packaging change, too.  I can't 
justify why that's worth doing for any of the other extensions, which is 
one reason I don't try to tackle them.

The type of finer sorting you and Thom are suggesting seems like it's 
mainly a documentation change to me.  I'm indifferent to the idea; no 
plans to either work on it or object to it.  The docs could be made 
easier to follow here without any change to the directory tree, and 
trying to push out a larger packaging change has downsides.  Useful 
reminder reading here is 
http://wiki.postgresql.org/wiki/PgCon_2011_Developer_Meeting#Moving_Contrib_Around  
To quote from there, "Users hate having loads and loads of packages. We 
do need to be careful not to oversplit it."  There's some useful notes 
about dependency issues there too.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us

  23. Re: Core Extensions relocation

    Dimitri Fontaine <dimitri@2ndquadrant.fr> — 2011-11-15T10:50:05Z

    Thom Brown <thom@linux.com> writes:
> None of those others perform such a role.  Instead they add additional
> functionality intended to be utilised as part of general data usage,
> adding new types, operators, query functions etc.  Maybe the term
> "core" is inappropriate.  Instead we might wish to refer to them as
> "utility extensions" or something like that, although that may be just
> as vague.

The term “core” here intends to show off that those extensions are
maintained by the core PostgreSQL developer team. If needs be, those
extensions will get updated in minor releases (crash, bugs, security,
etc).

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support

  24. Re: Core Extensions relocation

    Josh Berkus <josh@agliodbs.com> — 2011-11-15T17:53:40Z

    Greg,

> I'm not attached to the name, which I just pulled out of the air for
> the
> documentation.  Could just as easily call them built-in modules or
> extensions.  If the objection is that "extensions" isn't technically
> correct for auto-explain, you might call them core add-ons instead.
>  My
> thinking was that the one exception didn't make it worth the trouble
> to
> introduce a new term altogether here.  There's already too many terms
> used for talking about this sort of thing, the confusion from using a
> word other than "extensions" seemed larger than the confusion sown by
> auto-explain not fitting perfectly.

Well, I do think it should be *something* Extensions.  But Core Extensions implies that the other stuff is just random code, and makes the user wonder why it's included at all.  If we're going to rename some of the extensions, then we really need to rename them all or we look like those are being depreciated.

Maybe:

Core Management Extensions
Core Development Extensions
Additional Database Tools
Code Examples
Legacy Modules

I think that covers everything we have in contrib.

Given discussion, is there any point in reporting on the actual patch yet?

--Josh Berkus

  25. Re: Core Extensions relocation

    Josh Berkus <josh@agliodbs.com> — 2011-11-15T17:54:55Z

    Peter,

> I consider contrib/isn to be quite broken. It hard codes ISBN
> prefixes
> for the purposes of sanitising ISBNs, even though their assignment is
> actually controlled by a decentralised body of regional authorities.
> I'd vote for kicking it out of contrib.

Submit a patch to fix it then.  

I use ISBN in 2 projects, and it's working fine for me.  I'll strongly resist any attempt to "kick it out".

--Josh Berkus

  26. Re: Core Extensions relocation

    Robert Haas <robertmhaas@gmail.com> — 2011-11-15T18:03:03Z

    On Tue, Nov 15, 2011 at 12:54 PM, Joshua Berkus <josh@agliodbs.com> wrote:
>> I consider contrib/isn to be quite broken. It hard codes ISBN
>> prefixes
>> for the purposes of sanitising ISBNs, even though their assignment is
>> actually controlled by a decentralised body of regional authorities.
>> I'd vote for kicking it out of contrib.
>
> Submit a patch to fix it then.

It's not fixable.  The ISBN datatype is the equivalent of having an
SSN datatype that only allows SSNs that have actually been assigned to
a US citizen.

> I use ISBN in 2 projects, and it's working fine for me.  I'll strongly resist any attempt to "kick it out".

That's exactly why contrib is a random amalgamation of really useful
stuff and utter crap: people feel justified in defending the continued
existence of the crap on the sole basis that it's useful to them
personally.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  27. Re: Core Extensions relocation

    Greg Smith <greg@2ndquadrant.com> — 2011-11-15T18:19:11Z

    On 11/15/2011 12:53 PM, Joshua Berkus wrote:
> Given discussion, is there any point in reporting on the actual patch yet?

I don't expect the discussion will alter the code changes that are the 
main chunk of the patch here.  The only place the most disputed parts 
impact is the documentation.

I like "Management Extensions" as an alternate name for this category 
instead, even though it still has the issue that auto_explain isn't 
technically an extension.  The name does help suggest why they're thrown 
into a different directory and package.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us

  28. Re: Core Extensions relocation

    Bruce Momjian <bruce@momjian.us> — 2011-11-15T18:43:09Z

    Robert Haas wrote:
> > I use ISBN in 2 projects, and it's working fine for me. ?I'll strongly resist any attempt to "kick it out".
> 
> That's exactly why contrib is a random amalgamation of really useful
> stuff and utter crap: people feel justified in defending the continued
> existence of the crap on the sole basis that it's useful to them
> personally.

Agreed.  Berkus must have one million customers to have X customers
using every feature we want to remove or change.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

  29. Re: Core Extensions relocation

    Kevin Grittner <kevin.grittner@wicourts.gov> — 2011-11-15T18:52:27Z

    Robert Haas <robertmhaas@gmail.com> wrote:
> Joshua Berkus <josh@agliodbs.com> wrote:
 
>>> I consider contrib/isn to be quite broken. It hard codes ISBN
>>> prefixes for the purposes of sanitising ISBNs, even though their
>>> assignment is actually controlled by a decentralised body of
>>> regional authorities.
 
By an international standard which says what numbers are valid in
the "prefix element" and "registration group element" of the ISBN
for each of those regional authorities, and how the check digit is
to be calculated.
 
>>> I'd vote for kicking it out of contrib.
>>
>> Submit a patch to fix it then.
> 
> It's not fixable.  The ISBN datatype is the equivalent of having
> an SSN datatype that only allows SSNs that have actually been
> assigned to a US citizen.
 
Certainly it would make sense to go so far as to support the overall
standard format as described here:
 
http://www.isbn-international.org/faqs/view/5#q_5
 
Beyond the broad strokes there, perhaps it would make sense for the
type to be able to digest a RangeMessage.xml file supplied by the
standards organization, so that the current ranges could be plugged
in as needed independently of the PostgreSQL release.
 
http://www.isbn-international.org/page/ranges
http://www.isbn-international.org/pages/media/Range%20message/RangeMessage.pdf
 
Hard-coding ranges as of some moment in time seems pretty dubious.
 
-Kevin

  30. Re: ISN was: Core Extensions relocation

    Josh Berkus <josh@agliodbs.com> — 2011-11-15T19:01:34Z

    >> Submit a patch to fix it then.
> 
> It's not fixable.  The ISBN datatype is the equivalent of having an
> SSN datatype that only allows SSNs that have actually been assigned to
> a US citizen.

Nothing is "not fixable".  "not fixable without breaking backwards
compatibility" is entirely possible, though.  If fixing it led to two
different versions of ISN, then that would be a reason to push it to
PGXN instead of shipping it.

It's not as if ISN is poorly coded. This is a spec issue, which must
have been debated when we first included it.  No?

> That's exactly why contrib is a random amalgamation of really useful
> stuff and utter crap: people feel justified in defending the continued
> existence of the crap on the sole basis that it's useful to them
> personally.

Why else would we justify anything?  It's very difficult to argue on the
basis of theoretical users.  How would we really know what a theoretical
user wants?

Calling something "crap" because it has a spec issue is unwarranted.
We're going to get nowhere in this discussion as long as people are
using extreme and non-descriptive terms.

The thing is, most of the extensions in /contrib have major flaws, or
they would have been folded in to the core code by now.  CITEXT doesn't
support multiple collations.  INTARRAY and LTREE have inconsistent
operators and many bugs.  CUBE lacks documentation.  DBlink is an
ongoing battle with security holes.  etc.

Picking out one of those and saying "this is crap because of reason X,
but I'll ignore all the flaws in all these other extensions" is
inconsistent and not liable to produce results.  Now, if you wanted to
argue that we should kick *all* of the portable extensions out of
/contrib and onto PGXN, then you'd have a much stronger argument.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

  31. Re: Core Extensions relocation

    Peter Geoghegan <peter@2ndquadrant.com> — 2011-11-15T19:04:38Z

    On 15 November 2011 18:03, Robert Haas <robertmhaas@gmail.com> wrote:
> It's not fixable.  The ISBN datatype is the equivalent of having an
> SSN datatype that only allows SSNs that have actually been assigned to
> a US citizen.

That isn't even the worst part. isn is basically only useful in the US
at the moment, because in every other country there are a number of
bar code symbologies that co-exist within supply chain management for
various reasons. Only in the US can you reasonably assume that all
articles that you come across will be UPC, and even that might be a
shaky assumption these days.

In the E.U. and much of the rest of the world, products that you buy
in the store will bear one of the following symbologies: EAN-8 (for
small articles like chewing gum), UPC (the American one, 12 digits),
EAN-13 and GTIN-14. Some, but not all of these are available from
contrib/isn. There is no datatype that represents "some known barcode
format", even though writing an SQL function that can be used in a
domain check constraint to do just that is next to trivial. I guess
that means that you'd either have to have multiple columns for each
datatype, each existing in case the article in question was of that
particular datatype, or you'd need to make a hard assumption about the
symbology used for all articles that will ever be entered.

The way that these formats maintain backwards compatibility is through
making previous formats valid as the new format by padding zeros to
the left of the older format. So a UPC is actually a valid EAN-13,
just by adding a zero to the start - the US EAN country code is zero,
IIRC, so the rest of the world can pretend that Americans use
EAN-13s/GTIN-14s, even though they think that they use UPCs. The check
digit algorithms for each successive symbology are built such that
this works. This is why a DIY bar code bigint domain can be written so
easily, and also why doing so makes way more sense than using this
contrib module.

To my mind, contrib/isn is a solution looking for a problem, and
that's before we even talk about ISBN prefixes. By including it, we
give users a false sense of security about doing the right thing, when
they're very probably not.

-- 
Peter Geoghegan       http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training and Services

  32. Re: Core Extensions relocation

    Alvaro Herrera <alvherre@commandprompt.com> — 2011-11-15T19:18:10Z

    Excerpts from Robert Haas's message of mar nov 15 15:03:03 -0300 2011:
> On Tue, Nov 15, 2011 at 12:54 PM, Joshua Berkus <josh@agliodbs.com> wrote:
> >> I consider contrib/isn to be quite broken. It hard codes ISBN
> >> prefixes
> >> for the purposes of sanitising ISBNs, even though their assignment is
> >> actually controlled by a decentralised body of regional authorities.
> >> I'd vote for kicking it out of contrib.
> >
> > Submit a patch to fix it then.
> 
> It's not fixable.  The ISBN datatype is the equivalent of having an
> SSN datatype that only allows SSNs that have actually been assigned to
> a US citizen.

Interesting.  Isn't it possible to separate it into two parts, one
containing generic input, formatting and check digits verification, and
another one that would do the prefix matching?  Presumably, the first
part would still be useful without prefix validation, wouldn't it?
Surely the other validation rules aren't different for the various
prefixes.

Perhaps the prefix matching code should not be in C, or at least use a
lookup table that's not in C code, so that it can be updated in userland
without having to recompile.  (BTW, this is very similar to the problem
confronted by the E.164 module, which attempts to do telephone number
validation and formatting; there's a generic body of code that handles
the basic country code validation, but there's supposed to be some
per-CC formatting rules which we're not really clear on where to store.
We punted on it by just having that in a GUC, so that the user can
update it easily; but that's clearly not the best solution).

-- 
Álvaro Herrera <alvherre@commandprompt.com>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

  33. Re: ISN was: Core Extensions relocation

    Peter Geoghegan <peter@2ndquadrant.com> — 2011-11-15T19:39:34Z

    On 15 November 2011 19:01, Josh Berkus <josh@agliodbs.com> wrote:
> Nothing is "not fixable".

My idea of fixing contrib/isn would be to remove so much of it that it
would obviously make more sense to use simple, flexible SQL. It simply
makes way too many assumptions about the user's business rules for a
generic C module.

> Calling something "crap" because it has a spec issue is unwarranted.
> We're going to get nowhere in this discussion as long as people are
> using extreme and non-descriptive terms.

It is warranted, because contrib/isn is extremely wrong-headed.

> The thing is, most of the extensions in /contrib have major flaws, or
> they would have been folded in to the core code by now.  CITEXT doesn't
> support multiple collations.  INTARRAY and LTREE have inconsistent
> operators and many bugs.  CUBE lacks documentation.  DBlink is an
> ongoing battle with security holes.  etc.

The difference is that it's possible to imagine a scenario under which
I could recommend using any one of those modules, despite their flaws.
I could not contrive a reason for using contrib/isn.

-- 
Peter Geoghegan       http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training and Services

  34. Re: ISN was: Core Extensions relocation

    Robert Haas <robertmhaas@gmail.com> — 2011-11-15T20:32:42Z

    On Tue, Nov 15, 2011 at 2:01 PM, Josh Berkus <josh@agliodbs.com> wrote:
>>> Submit a patch to fix it then.
>>
>> It's not fixable.  The ISBN datatype is the equivalent of having an
>> SSN datatype that only allows SSNs that have actually been assigned to
>> a US citizen.
>
> Nothing is "not fixable".  "not fixable without breaking backwards
> compatibility" is entirely possible, though.  If fixing it led to two
> different versions of ISN, then that would be a reason to push it to
> PGXN instead of shipping it.

Well, the way to fix it would be to publish a new version of
PostgreSQL every time the international authority that assigns ISBN
prefixes allocates a new one, and for everyone to then update their
PostgreSQL installation every time we do that.  That doesn't, however,
seem very practical.  It just doesn't make sense to define a datatype
where the set of legal values changes over time.  The right place to
put such constraints in the application logic, where it doesn't take a
database upgrade to fix the problem.

> It's not as if ISN is poorly coded. This is a spec issue, which must
> have been debated when we first included it.  No?

I think our standards for inclusion have changed over the years - a
necessary part of project growth, or we would have 500 contrib modules
instead of 50.  The original isbn_issn module was contributed in 1998;
it was replaced by contrib/isn in 2006 because the original module
became obsolete.  I think it's fair to say that if this code were
submitted today it would never get accepted into our tree, and the
submitter would be advised not so much to publish on PGXN instead as
to throw it away entirely and rethink their entire approach to the
problem.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  35. Re: ISN was: Core Extensions relocation

    Kevin Grittner <kevin.grittner@wicourts.gov> — 2011-11-15T20:59:14Z

    Robert Haas <robertmhaas@gmail.com> wrote:
> Josh Berkus <josh@agliodbs.com> wrote:
 
>> Nothing is "not fixable".  "not fixable without breaking
>> backwards compatibility" is entirely possible, though.  If fixing
>> it led to two different versions of ISN, then that would be a
>> reason to push it to PGXN instead of shipping it.
> 
> Well, the way to fix it would be to publish a new version of
> PostgreSQL every time the international authority that assigns
> ISBN prefixes allocates a new one, and for everyone to then update
> their PostgreSQL installation every time we do that.  That
> doesn't, however, seem very practical.
 
Having just taken a closer look at contrib/isn, I'm inclined to
think the current implementation is pretty hopeless.  ISBN seems
common enough and standardized enough that it could perhaps be
included in contrib with the proviso that ranges would only be
validated through pointing to a copy of the XML provided by the
standards body -- it wouldn't be up to PostgreSQL to supply that.
 
The other types in contrib/isn are things I don't know enough about
to have an opinion, but it seems a little odd to shove them all
together.  It would seem more natural to me to have a distinct type
for each and, if needed, figure out how to get a clean union of the
types.
 
-Kevin

  36. Re: ISN was: Core Extensions relocation

    Tom Lane <tgl@sss.pgh.pa.us> — 2011-11-15T21:53:32Z

    Josh Berkus <josh@agliodbs.com> writes:
> The thing is, most of the extensions in /contrib have major flaws, or
> they would have been folded in to the core code by now.  CITEXT doesn't
> support multiple collations.  INTARRAY and LTREE have inconsistent
> operators and many bugs.  CUBE lacks documentation.  DBlink is an
> ongoing battle with security holes.  etc.

> Picking out one of those and saying "this is crap because of reason X,
> but I'll ignore all the flaws in all these other extensions" is
> inconsistent and not liable to produce results.  Now, if you wanted to
> argue that we should kick *all* of the portable extensions out of
> /contrib and onto PGXN, then you'd have a much stronger argument.

There's a larger issue here, which is that a lot of the stuff in contrib
is useful as (a) coding examples for people to look at, and/or (b) test
cases for core-server functionality.  If a module gets kicked out to
PGXN we lose pretty much all the advantages of (b), and some of the
value of (a) because stuff that is in the contrib tree at least gets
maintained when we make server API changes.

So the fact that a module has conceptual flaws that are preventing it
from getting moved into core doesn't really have much to do with whether
we should remove it, IMV.  The big question to be asking about ISN is
whether removing it would result in a loss of test coverage for the core
server; and I believe the answer is yes --- ISTR at least one bug that
we found specifically because ISN tickled it.

			regards, tom lane

  37. Re: ISN was: Core Extensions relocation

    Peter Geoghegan <peter@2ndquadrant.com> — 2011-11-15T23:44:09Z

    On 15 November 2011 21:53, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> There's a larger issue here, which is that a lot of the stuff in contrib
> is useful as (a) coding examples for people to look at, and/or (b) test
> cases for core-server functionality.  If a module gets kicked out to
> PGXN we lose pretty much all the advantages of (b), and some of the
> value of (a) because stuff that is in the contrib tree at least gets
> maintained when we make server API changes.

The isn module is patently broken. It has the potential to damage the
project's reputation if someone chooses to make an example out of it.
I think that that's more important than any additional test coverage
it may bring. There's only a fairly marginal benefit at the expense of
a bad user experience for anyone who should use isn.

-- 
Peter Geoghegan       http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training and Services

  38. Re: ISN was: Core Extensions relocation

    Robert Haas <robertmhaas@gmail.com> — 2011-11-15T23:57:51Z

    On Tue, Nov 15, 2011 at 6:44 PM, Peter Geoghegan <peter@2ndquadrant.com> wrote:
> On 15 November 2011 21:53, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> There's a larger issue here, which is that a lot of the stuff in contrib
>> is useful as (a) coding examples for people to look at, and/or (b) test
>> cases for core-server functionality.  If a module gets kicked out to
>> PGXN we lose pretty much all the advantages of (b), and some of the
>> value of (a) because stuff that is in the contrib tree at least gets
>> maintained when we make server API changes.
>
> The isn module is patently broken. It has the potential to damage the
> project's reputation if someone chooses to make an example out of it.
> I think that that's more important than any additional test coverage
> it may bring. There's only a fairly marginal benefit at the expense of
> a bad user experience for anyone who should use isn.

I agree.  The argument that this code is useful as example code has
been offered before, but the justification is pretty thin when the
example code is an example of a horrible design that no one should
ever copy.  I don't see that the isn code is doing anything that is so
unique that one of our add-on data types wouldn't be a suitable
(probably far better) template, but if it is, let's add similar
functionality to some other module, or add a new module that does
whatever that interesting thing is, or shove some code in
src/test/examples.  We can't go on complaining one the one hand that
people don't install postgresql-contrib, and then on the other hand
insist on shipping really bad code in contrib.  It's asking a lot for
someone who isn't already heavily involved in the project to
distinguish the wheat from the chaff.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  39. Re: ISN was: Core Extensions relocation

    Peter Geoghegan <peter@2ndquadrant.com> — 2011-11-16T00:46:01Z

    On 15 November 2011 23:57, Robert Haas <robertmhaas@gmail.com> wrote:
> We can't go on complaining one the one hand that
> people don't install postgresql-contrib, and then on the other hand
> insist on shipping really bad code in contrib.  It's asking a lot for
> someone who isn't already heavily involved in the project to
> distinguish the wheat from the chaff.

ISTM that any sensible sanitisation of data guards against Murphy, not
Machiavelli. Exactly what sort of error is it imagined will be
prevented by enforcing that ISBN prefixes are "valid"? Transcription
and transposition errors are already guarded against very effectively
by the check digit.

If we can't put isn out of its misery, a sensible compromise would be
to rip out the prefix enforcement feature so that, for example, ISBN13
behaved exactly the same as EAN13. That would at least result in
predictable behaviour. The "strike" that separates each part of the
ISBN would be lost, but it is actually not visible on large ISBN
websites, presumably because it's a tar-pit of a problem.

-- 
Peter Geoghegan       http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training and Services

  40. Re: ISN was: Core Extensions relocation

    Josh Berkus <josh@agliodbs.com> — 2011-11-16T01:09:30Z

    All,

> I agree.  The argument that this code is useful as example code has
> been offered before, but the justification is pretty thin when the
> example code is an example of a horrible design that no one should
> ever copy.

People are already using ISN (or at least ISBN) in production.  It's been around for 12 years.  So any step we take with contrib/ISN needs to take that into account -- just as we have with Tsearch2 and XML2.

One can certainly argue that some of the stuff in /contrib would be better on PGXN.  But in that case, it's not limited to ISN; there are several modules of insufficient quality (including intarray and ltree) or legacy nature which ought to be pushed out.  Probably most of them.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com
San Francisco

  41. Re: ISN was: Core Extensions relocation

    Peter Geoghegan <peter@2ndquadrant.com> — 2011-11-16T01:49:17Z

    On 16 November 2011 01:09, Joshua Berkus <josh@agliodbs.com> wrote:
> People are already using ISN (or at least ISBN) in production.  It's been around for 12 years.

contrib/isn has been around since 2006. The argument "some unknowable
number of people are using this feature in production" could equally
well apply to anything that we might consider deprecating.

I am not arguing for putting isn on PGXN. I'm arguing for actively
warning people against using it, because it is harmful. Any serious
use of the ISBN datatypes can be expected to break unpredictably one
day, and the only thing that someone can do in that situation is to
write their own patch to contrib/isn. They'd then have to wait for
that patch to be accepted if they didn't want to fork, which is a very
bad situation indeed. This already happened once.

-- 
Peter Geoghegan       http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training and Services

  42. Re: Core Extensions relocation

    Greg Smith <greg@2ndquadrant.com> — 2011-11-16T03:09:19Z

    Well, this discussion veering off into ISN has certainly validated my 
gut feel that I should touch only the minimum number of things that 
kills my pain points, rather than trying any more ambitious 
restructuring.  I hope that packaged extensions become so popular that 
some serious cutting can happen to contrib, especially the data type 
additions.  If something as big as PostGIS can live happily as an 
external project, surely most of these can too.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us

  43. Re: ISN was: Core Extensions relocation

    Tom Lane <tgl@sss.pgh.pa.us> — 2011-11-16T03:40:49Z

    Peter Geoghegan <peter@2ndquadrant.com> writes:
> If we can't put isn out of its misery, a sensible compromise would be
> to rip out the prefix enforcement feature so that, for example, ISBN13
> behaved exactly the same as EAN13.

That might be a reasonable compromise.  Certainly the check-digit
calculation is much more useful for validity checking than the prefix
checking.

			regards, tom lane

  44. Re: Core Extensions relocation

    Robert Haas <robertmhaas@gmail.com> — 2011-11-16T13:00:47Z

    On Tue, Nov 15, 2011 at 5:50 AM, Dimitri Fontaine
<dimitri@2ndquadrant.fr> wrote:
> Thom Brown <thom@linux.com> writes:
>> None of those others perform such a role.  Instead they add additional
>> functionality intended to be utilised as part of general data usage,
>> adding new types, operators, query functions etc.  Maybe the term
>> "core" is inappropriate.  Instead we might wish to refer to them as
>> "utility extensions" or something like that, although that may be just
>> as vague.
>
> The term “core” here intends to show off that those extensions are
> maintained by the core PostgreSQL developer team. If needs be, those
> extensions will get updated in minor releases (crash, bugs, security,
> etc).

Everything in contrib meets that definition, more or less.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  45. Re: Core Extensions relocation

    Dimitri Fontaine <dimitri@2ndquadrant.fr> — 2011-11-16T14:20:41Z

    Robert Haas <robertmhaas@gmail.com> writes:
>> The term “core” here intends to show off that those extensions are
>> maintained by the core PostgreSQL developer team. If needs be, those
>> extensions will get updated in minor releases (crash, bugs, security,
>> etc).
>
> Everything in contrib meets that definition, more or less.

Yeah? It would only mean that Josh Berkus complaint about the naming is
to be followed.

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support

  46. Re: Core Extensions relocation

    Robert Haas <robertmhaas@gmail.com> — 2011-11-16T15:25:50Z

    On Wed, Nov 16, 2011 at 9:20 AM, Dimitri Fontaine
<dimitri@2ndquadrant.fr> wrote:
> Robert Haas <robertmhaas@gmail.com> writes:
>>> The term “core” here intends to show off that those extensions are
>>> maintained by the core PostgreSQL developer team. If needs be, those
>>> extensions will get updated in minor releases (crash, bugs, security,
>>> etc).
>>
>> Everything in contrib meets that definition, more or less.
>
> Yeah? It would only mean that Josh Berkus complaint about the naming is
> to be followed.

I am not sure I'm quite following you, but I'm unaware that there are
some contrib modules that we maintain more than others.  Bugs and
security vulnerabilities in any of them are typically fixed when
reported.  Now, sometimes we might not be able to fix a bug because of
some architectural deficiency, but that also happens in the server -
consider, for example, the recent discussion of creating a table in a
schema that is concurrently being dropped, which is likely to require
far more invasive fixing than we are probably willing to do anywhere
other than master.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  47. Re: ISN was: Core Extensions relocation

    Josh Berkus <josh@agliodbs.com> — 2011-11-16T18:17:16Z

    On 11/15/11 7:40 PM, Tom Lane wrote:
> Peter Geoghegan <peter@2ndquadrant.com> writes:
>> If we can't put isn out of its misery, a sensible compromise would be
>> to rip out the prefix enforcement feature so that, for example, ISBN13
>> behaved exactly the same as EAN13.
> 
> That might be a reasonable compromise.  Certainly the check-digit
> calculation is much more useful for validity checking than the prefix
> checking.

Sounds good to me.  FWIW, I know that ISBN is being used for some
library software, so a backwards-compatible fix would be very desirable.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

  48. Re: ISN was: Core Extensions relocation

    Tom Lane <tgl@sss.pgh.pa.us> — 2011-11-17T00:13:24Z

    Josh Berkus <josh@agliodbs.com> writes:
> On 11/15/11 7:40 PM, Tom Lane wrote:
>> Peter Geoghegan <peter@2ndquadrant.com> writes:
>>> If we can't put isn out of its misery, a sensible compromise would be
>>> to rip out the prefix enforcement feature so that, for example, ISBN13
>>> behaved exactly the same as EAN13.

>> That might be a reasonable compromise.  Certainly the check-digit
>> calculation is much more useful for validity checking than the prefix
>> checking.

> Sounds good to me.  FWIW, I know that ISBN is being used for some
> library software, so a backwards-compatible fix would be very desirable.

How backwards-compatible do you mean?  Do you think we need to keep the
existing prefix-check logic as an option, or can we just drop it and be
done?

(I'm a bit concerned about the angle that the code has some smarts
currently about where to put hyphens in output.  If we rip that out
it could definitely break application code, whereas dropping a check
shouldn't.)

			regards, tom lane

  49. Re: ISN was: Core Extensions relocation

    Josh Berkus <josh@agliodbs.com> — 2011-11-17T02:32:33Z

    > (I'm a bit concerned about the angle that the code has some smarts
> currently about where to put hyphens in output.  If we rip that out
> it could definitely break application code, whereas dropping a check
> shouldn't.)

Right.  Do we need to dump the hyphen logic?

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

  50. Re: ISN was: Core Extensions relocation

    Peter Geoghegan <peter@2ndquadrant.com> — 2011-11-17T03:04:37Z

    On 17 November 2011 02:32, Josh Berkus <josh@agliodbs.com> wrote:
>
>> (I'm a bit concerned about the angle that the code has some smarts
>> currently about where to put hyphens in output.  If we rip that out
>> it could definitely break application code, whereas dropping a check
>> shouldn't.)
>
> Right.  Do we need to dump the hyphen logic?

Only if you think that it's better to have something that covers many
cases but is basically broke. Perhaps it's different for code that is
already committed, but in the case of new submissions it tends to be
better to have something that is limited in a well-understood way
rather than less limited in a way that is unpredictable or difficult
to reason about.

-- 
Peter Geoghegan       http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training and Services

  51. Re: ISN was: Core Extensions relocation

    Tom Lane <tgl@sss.pgh.pa.us> — 2011-11-17T03:54:14Z

    Peter Geoghegan <peter@2ndquadrant.com> writes:
> On 17 November 2011 02:32, Josh Berkus <josh@agliodbs.com> wrote:
>> Right.  Do we need to dump the hyphen logic?

> Only if you think that it's better to have something that covers many
> cases but is basically broke. Perhaps it's different for code that is
> already committed, but in the case of new submissions it tends to be
> better to have something that is limited in a well-understood way
> rather than less limited in a way that is unpredictable or difficult
> to reason about.

Well, as was stated upthread, we might have bounced this module in toto
if it were submitted today.  But contrib/isn has been there since 2006,
and its predecessor contrib/isbn_issn was there since 1998, and both of
those submissions came from (different) people who needed the
functionality bad enough to write it.  It's not reasonable to suppose
that nobody is using it today.  Ergo, we can't just summarily break
backwards compatibility on the grounds that we don't like the design.
Heck, we don't even have a field bug report that the design limitation
is causing any real problems for real users ... so IMO, the claims that
this is dangerously broken are a bit overblown.

			regards, tom lane

  52. Re: ISN was: Core Extensions relocation

    Josh Berkus <josh@agliodbs.com> — 2011-11-17T05:03:56Z

    On 11/16/11 7:54 PM, Tom Lane wrote:
> Heck, we don't even have a field bug report that the design limitation
> is causing any real problems for real users ... so IMO, the claims that
> this is dangerously broken are a bit overblown.

This is why I mentioned clients using ISBN in production.  I have yet to
see any actual breakage in the field.  Granted, both clients are
US-only.   I don't believe either of these clients is depending on the
prefix-check, and that's the sort of thing we could announce in the
release notes.

I do get the feeling that Peter got burned by ISN, given his vehemence
in erasing it from the face of the earth.  So that's one bug report.  ;-)

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

  53. Re: ISN was: Core Extensions relocation

    Peter Geoghegan <peter@2ndquadrant.com> — 2011-11-17T15:44:25Z

    On 17 November 2011 03:54, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> It's not reasonable to suppose
> that nobody is using it today.

I didn't suppose that no one is using it, but that those that are
using it are unaware of the risks with prefix validation, and that
there will be a rude awakening for them.

> Ergo, we can't just summarily break
> backwards compatibility on the grounds that we don't like the design.
> Heck, we don't even have a field bug report that the design limitation
> is causing any real problems for real users ... so IMO, the claims that
> this is dangerously broken are a bit overblown.

I think that's it's rather unlikely that removing hyphenation and
prefix validation would adversely affect anyone, provided that it was
well documented and wasn't applied to stable branches. If it were up
to me, I might remove validation from stable branches but keep
hyphenation, while removing both for 9.2 . After all, hyphenation will
break anyway, so they're worse off continuing to rely on hyphenation
when it cannot actually be relied on.

On 17 November 2011 05:03, Josh Berkus <josh@agliodbs.com> wrote:
> I do get the feeling that Peter got burned by ISN, given his vehemence
> in erasing it from the face of the earth.  So that's one bug report.  ;-)

Actually, I reviewed a band-aid patch about a year ago, and I was
fairly concerned at the obvious wrong-headedness of something in our
tree. I have a certain amount of domain knowledge here, but I've never
actually attempted to use it in production. For all its faults, it is
at least obviously broken to someone that knows about GS1 standards
(having separate bar code datatypes is just not useful at all),
although that tends to not be Americans. This may account for why
we've heard so few complaints. It's also really easy and effective to
create your own domain, and the flexibility that this affords tends to
make an off-the-shelf solution unattractive (I've done things like
store "compacted" bar codes that will subsequently be used to match a
full bar code with an embedded price - I didn't want to enforce a
check digit for the compacted representation).

If I had a lot of time to work on fixing contrib/isn, I still
wouldn't, because the correct thing to do is to produce your own
domain.

-- 
Peter Geoghegan       http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training and Services

  54. Re: ISN was: Core Extensions relocation

    Robert Haas <robertmhaas@gmail.com> — 2011-11-17T17:40:35Z

    On Thu, Nov 17, 2011 at 10:44 AM, Peter Geoghegan <peter@2ndquadrant.com> wrote:
> I think that's it's rather unlikely that removing hyphenation and
> prefix validation would adversely affect anyone, provided that it was
> well documented and wasn't applied to stable branches. If it were up
> to me, I might remove validation from stable branches but keep
> hyphenation, while removing both for 9.2 . After all, hyphenation will
> break anyway, so they're worse off continuing to rely on hyphenation
> when it cannot actually be relied on.

Well, keep in mind that most people test their code.  It seems likely
that it actually DOES work pretty well for the people who are using
the module.  The ones for whom it didn't work presumably would have
complained (and, mostly, they haven't) or abandoned the module (in
which case they're irrelevant to the discussion of who might be hurt
by this change).  I'd be willing to bet a nickle that we'll get
complaints if we rip that hyphenation behavior out.

At the same time, I still think we should push this out to PGXN or
pgfoundry or something.  The fact that it's useful to some people does
not mean that it's a good example for other people to follow, or that
we want the core distribution to be in the process of tracking ISBN
prefixes on behalf of PostgreSQL users everywhere.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  55. Re: ISN was: Core Extensions relocation

    Tom Lane <tgl@sss.pgh.pa.us> — 2011-11-17T17:50:01Z

    Robert Haas <robertmhaas@gmail.com> writes:
> At the same time, I still think we should push this out to PGXN or
> pgfoundry or something.  The fact that it's useful to some people does
> not mean that it's a good example for other people to follow, or that
> we want the core distribution to be in the process of tracking ISBN
> prefixes on behalf of PostgreSQL users everywhere.

I wouldn't object to that as long as we replaced it with some other
module that had a similar relationship to core types.  We'd never have
realized the need for CREATE TYPE's LIKE option, until it was far too
late, if we'd not had contrib/isn to show up the problem (cf
commit 3f936aacc057e4b391ab953fea2ffb689a12a8bc).

But really I think this discussion is making a mountain out of a
molehill.  If tracking ISBN prefix changes is such a time-consuming
activity, why have we not seen a steady stream of update patches from
users?  By my count there's been just one such patch since 2006
(commit 6d1af7b2180719102a907bd3e35d218b43e76ad1).

			regards, tom lane

  56. Re: Core Extensions relocation

    Peter Eisentraut <peter_e@gmx.net> — 2011-11-17T20:18:49Z

    On mån, 2011-11-14 at 20:44 -0500, Greg Smith wrote:
> The very specific problem I was most concerned about eliminating was
> people discovering they needed an extension to troubleshoot
> performance or corruption issues, only to discover it wasn't
> available--because they hadn't installed the postgresql-contrib
> package. 

Who's to say that after this, the core extensions won't end up in a new
separate package postgresql-extensions (or similar) or might even stay
in postgresql-contrib, for compatibility?

  57. Re: Core Extensions relocation

    Greg Smith <greg@2ndquadrant.com> — 2011-11-18T09:31:59Z

    On 11/17/2011 03:18 PM, Peter Eisentraut wrote:
> Who's to say that after this, the core extensions won't end up in a new
> separate package postgresql-extensions (or similar) or might even stay
> in postgresql-contrib, for compatibility?
>    

I don't know why packagers would make an active decision that will make 
their lives more difficult, with no benefit to them and a regression 
against project recommendations for their users.  The last thing anyone 
packaging PostgreSQL wants is more packages to deal with; there are 
already too many.  Each of the current sub-packages has a legitimate 
technical or distribution standard reason to exist--guidelines like 
"break out client and server components" or "minimize the package 
dependencies for the main server".  I can't think of any good reason 
that would inspire the sort of drift you're concerned about.

There's little compatibility argument beyond consistency with the 
previous packages.  The reason why this is suddenly feasible now is that 
the under the hood change are almost all hidden by CREATE EXTENSION.

And if some wanted to wander this way, they'll end up having to maintain 
a doc patch to address the fact that they've broken with project 
recommendations.  This text in what I submitted will no longer be true:

"This appendix contains information regarding core extensions that are 
built and included with a standard installation of PostgreSQL."

One of the reasons I picked the name I did was to contrast with the 
existing description of contrib:

"porting tools, analysis utilities, and plug-in features that are not 
part of the core PostgreSQL system, mainly because they address a 
limited audience or are too experimental to be part of the main source 
tree."

That says it's perfectly fine to make these optional in another 
package--they're not "part of the core".  That scary wording is 
practically telling packagers to separate them, so it's easy to keep the 
experimental stuff away from the production quality components.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us

  58. Re: Core Extensions relocation

    Tom Lane <tgl@sss.pgh.pa.us> — 2011-11-18T14:35:05Z

    Greg Smith <greg@2ndQuadrant.com> writes:
> On 11/17/2011 03:18 PM, Peter Eisentraut wrote:
>> Who's to say that after this, the core extensions won't end up in a new
>> separate package postgresql-extensions (or similar) or might even stay
>> in postgresql-contrib, for compatibility?

> I don't know why packagers would make an active decision that will make 
> their lives more difficult, with no benefit to them and a regression 
> against project recommendations for their users.

Why do you figure that, exactly?  The path of least resistance will
be precisely to leave everything packaged as it is, in a single
postgresql-contrib module.  I'm pretty likely to do that myself for
Fedora and RHEL.  Subdividing/rearranging contrib makes the packager's
life more complicated, *and* makes his users' lives more complicated,
if only because things aren't where they were before.  It seems unlikely
to happen, at least in the near term.

> And if some wanted to wander this way, they'll end up having to maintain 
> a doc patch to address the fact that they've broken with project 
> recommendations.  This text in what I submitted will no longer be true:

You're assuming anybody will notice or care about that text, if indeed
it gets committed/released with that wording at all.

The upstream project can't force these decisions on packagers, and it
doesn't help to go about under the illusion that we can.

			regards, tom lane

  59. Re: Core Extensions relocation

    Dimitri Fontaine <dimitri@2ndquadrant.fr> — 2011-11-18T20:27:15Z

    Tom Lane <tgl@sss.pgh.pa.us> writes:
> Why do you figure that, exactly?  The path of least resistance will
> be precisely to leave everything packaged as it is, in a single
> postgresql-contrib module.  I'm pretty likely to do that myself for
> Fedora and RHEL.  Subdividing/rearranging contrib makes the packager's
> life more complicated, *and* makes his users' lives more complicated,
> if only because things aren't where they were before.  It seems unlikely
> to happen, at least in the near term.

Then if we want packagers to move, what about removing all the
extensions not listed by Greg from the contrib/ directory and inventing
another place where to manage them, which is not automatically built,
but still part of buildfarm tests, if at all possible.

Then the only change we suggest to packagers is to have the main
PostgreSQL package install the contrib one by means of dependencies.

I don't much like this solution, but that's how I read your email.  The
status quo is not a good place to live in.

> The upstream project can't force these decisions on packagers, and it
> doesn't help to go about under the illusion that we can.

Really? You are packaging for RHEL, Dave is responsible for the windows
packaging, Devrim is covering the other RPM systems (apart from SuSE
maybe and I'm not even sure) and Martin is caring for debian and ubuntu
and following along. We're missing BSD ports packagers, and we're
covering like 90% or more of the servers and developers installs.

We can't force everybody hands to doing it our way, but I'm pretty sure
we can talk to them and see what they think about the usefulness of this
proposal and how they intend to react.  We're not *that* disconnected.

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support

  60. Re: Core Extensions relocation

    Josh Berkus <josh@agliodbs.com> — 2011-11-18T20:36:59Z

    On 11/18/11 12:27 PM, Dimitri Fontaine wrote:
> Tom Lane <tgl@sss.pgh.pa.us> writes:
>> Why do you figure that, exactly?  The path of least resistance will
>> be precisely to leave everything packaged as it is, in a single
>> postgresql-contrib module.  I'm pretty likely to do that myself for
>> Fedora and RHEL.  Subdividing/rearranging contrib makes the packager's
>> life more complicated, *and* makes his users' lives more complicated,
>> if only because things aren't where they were before.  It seems unlikely
>> to happen, at least in the near term.
> 
> Then if we want packagers to move, what about removing all the
> extensions not listed by Greg from the contrib/ directory and inventing
> another place where to manage them, which is not automatically built,
> but still part of buildfarm tests, if at all possible.

Actually, the whole idea is that the "Core Management Extensions" should
move from the -contrib module to the -server module.  That is, those
extensions should always get installed with any server.

Of course, packagers may then reasonably ask why that code is not just
part of the core?

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

  61. Re: Core Extensions relocation

    Greg Smith <greg@2ndquadrant.com> — 2011-11-19T14:56:50Z

    On 11/18/2011 03:36 PM, Josh Berkus wrote:
> Of course, packagers may then reasonably ask why that code is not just
> part of the core?
>    

Let me step back from the implementation ideas for a minute and talk 
about this, and how it ties into release philosophy.  The extensions 
infrastructure for PostgreSQL is one of its strongest features.  We can 
use it as a competitive advantage over other databases, one that can 
make this database stable and continuously innovating at the same time.  
But that's not happening enough yet; I feel this change is a major step 
in that direction.  There's no demonstration that extensions are edible 
dog food like the core database visibly eating a can.

To see why this matters so much, let's compare two stereotypes of 
PostgreSQL users at different extremes of upgrade tolerance.  First we 
have the classic enterprise DBA.  Relative to this person's 
expectations, PostgreSQL releases are way too fast.  They can't upgrade 
their database every year; that's madness.  This is the person who we 
yell at about how they should upgrade to the latest minor point release, 
because once they have a working system they touch *nothing*.  For this 
user, the long beta period of new PostgreSQL releases, and its general 
conservative development model, are key components to PostgreSQL being 
suitable for them.

At the other extreme, we have the software developer with a frantic 
development/release schedule, the one who's running the latest stable 
version of every tool they use.  This person expects some bugs in them, 
and the first reaction to running into one is to ask "is this fixed in 
the next version?"  You'll find at least one component in their stack 
that's labeled "compiled from the latest checkout" because that was the 
only way to get a working version.  And to them, the yearly release 
cycle of PostgreSQL is glacial.  When they run into a limitation that is 
impacting a user base that's doubling every few months, they can't wait 
a year for a fix; they could easily go out of business by then.

The key to satisfying both these extremes at once is a strong extensions 
infrastructure, led by the example of serious tools that are provided 
that way in the PostgreSQL core.  For this to catch on, we need the 
classic DBAs to trust core extensions enough to load them into 
production.  They don't do that now because the current contrib 
description sounds too scary, and they may not even have loaded that 
package onto the server.  And we need people who want more frequent 
database core changes to see that extensions are a viable way to build 
some pretty extensive server hacks.

I've submitted two changes to this CommitFest that are enhancing 
features in this "core extensions" set.  Right now I have multiple 
customers who are desperate for both of those features.  With 
extensions, I can give them changes that solve their immediate crisis 
right now, almost a full year before they could possibly appear in a 
proper release of PostgreSQL.  And then I can push those back toward 
community PostgreSQL, with any luck landing in the next major version.  
Immediate gratification for the person funding development, and peer 
reviewed code that goes through a long beta and release cycle.  That's 
the vision I have for a PostgreSQL that is simultaneously stable and 
agile.  The easiest way to get there it is to lead by example--by having 
extensions that provide necessary, visible components to core, while 
still being obviously separate components.  That's the best approach for 
proving this development model works and is suitable for everyone.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us

  62. Re: Core Extensions relocation

    Greg Smith <greg@2ndquadrant.com> — 2011-11-19T16:44:46Z

    On 11/18/2011 09:35 AM, Tom Lane wrote:
> Subdividing/rearranging contrib makes the packager's
> life more complicated, *and* makes his users' lives more complicated,
> if only because things aren't where they were before.  It seems unlikely
> to happen, at least in the near term.
>    

Users are visibly suffering by the current packaging.  Production DBAs 
are afraid to install contrib because it's described as untrustworthy.  
They are hit by emergencies that the inspection tools would help with, 
but cannot get contrib installed easily without root permissions.  They 
have performance issues that the contrib modules I'm trying to relocate 
into the server package would help with, but company policies related to 
post-deployment installation mean they cannot use them.  They have to 
always be installed to make this class of problem go away.

If you feel there are more users that would be negatively impacted by 
some directories moving than what I'm describing above, we are a very 
fundamental disagreement here.  The status quote for all of these 
extensions is widely understood to be unusable in common corporate 
environments.  Packagers should be trying to improve the PostgreSQL 
experience, and I'm trying to help with that.

In the face of pushback from packagers, I can roll my own packages that 
are designed without this problem; I'm being pushed into doing that 
instead of working on community PostgreSQL already.  But I'd really 
prefer to see this very common problem identified as so important it 
should get fixed everywhere instead.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us

  63. Re: Core Extensions relocation

    Bruce Momjian <bruce@momjian.us> — 2011-11-21T16:40:39Z

    Greg Smith wrote:
> I've submitted two changes to this CommitFest that are enhancing 
> features in this "core extensions" set.  Right now I have multiple 
> customers who are desperate for both of those features.  With 
> extensions, I can give them changes that solve their immediate crisis 
> right now, almost a full year before they could possibly appear in a 
> proper release of PostgreSQL.  And then I can push those back toward 
> community PostgreSQL, with any luck landing in the next major version.  
> Immediate gratification for the person funding development, and peer 
> reviewed code that goes through a long beta and release cycle.  That's 
> the vision I have for a PostgreSQL that is simultaneously stable and 
> agile.  The easiest way to get there it is to lead by example--by having 
> extensions that provide necessary, visible components to core, while 
> still being obviously separate components.  That's the best approach for 
> proving this development model works and is suitable for everyone.

I think a question is how often people are waiting for features that
actually can be addressed in a contrib/plugin way.  My gut feeling is
that most missing features have to be added to the server core (e.g.
index-only scans) and are not possible to add in a contrib/plugin way.  

I am not saying this would not help, but I am saying that this is going
to address only a small part of the goal of getting features to users
quicker.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

  64. Re: ISN was: Core Extensions relocation

    Bruce Momjian <bruce@momjian.us> — 2011-11-21T17:25:54Z

    Peter Geoghegan wrote:
> On 17 November 2011 03:54, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >?It's not reasonable to suppose
> > that nobody is using it today.
> 
> I didn't suppose that no one is using it, but that those that are
> using it are unaware of the risks with prefix validation, and that
> there will be a rude awakening for them.
> 
> > Ergo, we can't just summarily break
> > backwards compatibility on the grounds that we don't like the design.
> > Heck, we don't even have a field bug report that the design limitation
> > is causing any real problems for real users ... so IMO, the claims that
> > this is dangerously broken are a bit overblown.
> 
> I think that's it's rather unlikely that removing hyphenation and
> prefix validation would adversely affect anyone, provided that it was
> well documented and wasn't applied to stable branches. If it were up
> to me, I might remove validation from stable branches but keep
> hyphenation, while removing both for 9.2 . After all, hyphenation will
> break anyway, so they're worse off continuing to rely on hyphenation
> when it cannot actually be relied on.

Clarification:  Our policy for patching back-branches is that the bug
has to affect many users, be serious, and the fix has to be easily
tested.

For a user-visible change (which this would be), the criteria is even
more strict. 

I don't see any of this reaching the level that it needs to be
backpatched, so I think we have to accept that this will be 9.2-only
change.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

  65. Re: ISN was: Core Extensions relocation

    Josh Berkus <josh@agliodbs.com> — 2011-11-21T17:57:54Z

    Bruce,

> I don't see any of this reaching the level that it needs to be
> backpatched, so I think we have to accept that this will be 9.2-only
> change.

Agreed.  If users encounter issues with the prefix in the field, it will be easy enough for them to back-patch.  But we don't want to be responsible for it as a project.

--Josh

  66. Re: Core Extensions relocation

    Greg Smith <greg@2ndquadrant.com> — 2011-11-21T19:59:16Z

    On 11/21/2011 11:40 AM, Bruce Momjian wrote:
> I think a question is how often people are waiting for features that
> actually can be addressed in a contrib/plugin way.  My gut feeling is
> that most missing features have to be added to the server core (e.g.
> index-only scans) and are not possible to add in a contrib/plugin way.
>    

Good question; let's talk about 9.0.  We were building/distributing 
three things for that version that poked into the server:

-Replication monitoring tools that slipped from the 9.0 schedule, 
similar to what became pg_stat_replication in 9.1
-An early version of what became hot_standby_feedback in 9.1.
-pg_streamrecv

While these weren't all packaged as extensions per se, all of them used 
the PGXS interface.  And they all provided deployment blocking features 
to early adopters before those features were available in core, in some 
cases after the issues they address had been encountered in production 
deployments.  As I was ranting on my blog recently, I'm seeing more 
complaints recently about monitoring and management features--exactly 
the sort of thing that you can improve as an extension, and that the 
extensions I've proposed provide--than I am over missing big features.

Index-only scans are a good example, as one of the most requested 
performance feature you can only get in core (I'd put them at #2 behind 
materialized views for the customers I talk to).  I wouldn't bet that 
they are considered more important by a typical deployment than good 
built-in query profiling though.  I get complaints about query 
monitoring from every single PostgreSQL install, while complaints about 
not having index-only scans only come from the bigger installations.  
Note how demand is high enough that we have two pg_stat_statements 
replacements submitted right now.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us

  67. Re: Core Extensions relocation

    Bruce Momjian <bruce@momjian.us> — 2011-11-21T20:02:10Z

    Greg Smith wrote:
> On 11/21/2011 11:40 AM, Bruce Momjian wrote:
> > I think a question is how often people are waiting for features that
> > actually can be addressed in a contrib/plugin way.  My gut feeling is
> > that most missing features have to be added to the server core (e.g.
> > index-only scans) and are not possible to add in a contrib/plugin way.
> >    
> 
> Good question; let's talk about 9.0.  We were building/distributing 
> three things for that version that poked into the server:
> 
> -Replication monitoring tools that slipped from the 9.0 schedule, 
> similar to what became pg_stat_replication in 9.1
> -An early version of what became hot_standby_feedback in 9.1.
> -pg_streamrecv
> 
> While these weren't all packaged as extensions per se, all of them used 
> the PGXS interface.  And they all provided deployment blocking features 
> to early adopters before those features were available in core, in some 
> cases after the issues they address had been encountered in production 
> deployments.  As I was ranting on my blog recently, I'm seeing more 
> complaints recently about monitoring and management features--exactly 
> the sort of thing that you can improve as an extension, and that the 
> extensions I've proposed provide--than I am over missing big features.
> 
> Index-only scans are a good example, as one of the most requested 
> performance feature you can only get in core (I'd put them at #2 behind 
> materialized views for the customers I talk to).  I wouldn't bet that 
> they are considered more important by a typical deployment than good 
> built-in query profiling though.  I get complaints about query 
> monitoring from every single PostgreSQL install, while complaints about 
> not having index-only scans only come from the bigger installations.  
> Note how demand is high enough that we have two pg_stat_statements 
> replacements submitted right now.

Agreed much of the edge stuff, e.g. monitoring, can be done as plugins.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

  68. Re: Core Extensions relocation

    Robert Haas <robertmhaas@gmail.com> — 2011-11-21T20:07:49Z

    On Fri, Nov 18, 2011 at 9:35 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Why do you figure that, exactly?  The path of least resistance will
> be precisely to leave everything packaged as it is, in a single
> postgresql-contrib module.  I'm pretty likely to do that myself for
> Fedora and RHEL.  Subdividing/rearranging contrib makes the packager's
> life more complicated, *and* makes his users' lives more complicated,
> if only because things aren't where they were before.  It seems unlikely
> to happen, at least in the near term.

When we discussed this topic at the developer's meeting, I thought we
had general consensus that it would be a good idea to package a
limited number of important and stable debugging tools with the core
server, and I had the impression that you (Tom) thought this was a
reasonable thing to do.  If you don't, or if you did then but don't
now, then it seems to me that this conversation is dead in the water
for so long as you're the one packaging for Red Hat, and we should
just move on; you pretty much have unassailable personal veto power on
this issue.  But let's not pretend that the conversation is about what
packagers in general will do, because I don't think it is.  I think
it's about what you personally will do.

I think that if we move a few things into src/extension and set things
up such that they get installed even if you just do "make install"
rather than requiring "make install-world", packagers who don't have
any terribly strong personal agenda will decide that means they ought
to be shipped with the server.  However, if you're personally
committed to making sure that all of that stuff remains in
postgresql-contrib in Red Hat/Fedora, regardless of where we move it
to on our end, then that's where it's going to be, at least on all Red
Hat-derived systems, which is a big enough chunk of the world to
matter quite a lot.  Note that I'm not necessarily saying anything
about whether your reasons for such a decision might be good or bad;
I'm just pointing out that a good deal of our ability to make a change
in this area is within your personal control.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

  69. Re: Core Extensions relocation

    Josh Berkus <josh@agliodbs.com> — 2011-11-30T02:24:37Z

    Tom,

> I think that if we move a few things into src/extension and set things
> up such that they get installed even if you just do "make install"
> rather than requiring "make install-world", packagers who don't have
> any terribly strong personal agenda will decide that means they ought
> to be shipped with the server.  However, if you're personally
> committed to making sure that all of that stuff remains in
> postgresql-contrib in Red Hat/Fedora, regardless of where we move it
> to on our end, then that's where it's going to be, at least on all Red
> Hat-derived systems, which is a big enough chunk of the world to
> matter quite a lot.  Note that I'm not necessarily saying anything
> about whether your reasons for such a decision might be good or bad;
> I'm just pointing out that a good deal of our ability to make a change
> in this area is within your personal control.

Any response to this before I take it to the other packagers?

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

  70. Re: Core Extensions relocation

    Josh Berkus <josh@agliodbs.com> — 2011-12-09T18:43:54Z

    All,

This is currently awaiting a check by gsmith that the 7 named extensions
do not add any new dependancies.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

  71. Re: Core Extensions relocation

    Josh Berkus <josh@agliodbs.com> — 2012-02-08T22:14:25Z

    Greg,

> This is currently awaiting a check by gsmith that the 7 named extensions
> do not add any new dependancies.

Are you going to investigate this?  If not, I'll give it a try this weekend.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

  72. Re: Core Extensions relocation

    Josh Berkus <josh@agliodbs.com> — 2012-02-10T19:59:36Z

    All,

Andrew ran crake on these modules, and they do not add any links not
added by core postgres already.

As such, can we proceed with this patch?  Greg, do you have an updated
version to run against HEAD?

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com