Horton

Thread

Commits

  1. Add --no-blobs option to pg_dump

  1. Exclude pg_largeobject form pg_dump

    Andreas Joseph Krogh <andreas@visena.com> — 2016-03-08T09:53:39Z

    Hi all.
 
Is there a way to exclude pg_largeobject from pg_dump? -T pg_largeobject 
doesn't work. I've tried to exclude tables using OID-datatype also but that 
didn't work either.
 
Thanks.
 
-- Andreas Joseph Krogh
CTO / Partner - Visena AS
Mobile: +47 909 56 963
andreas@visena.com <mailto:andreas@visena.com>
www.visena.com <https://www.visena.com>
 <https://www.visena.com>

  2. Re: Exclude pg_largeobject form pg_dump

    Adrian Klaver <adrian.klaver@aklaver.com> — 2016-03-08T14:43:37Z

    On 03/08/2016 01:53 AM, Andreas Joseph Krogh wrote:
> Hi all.
> Is there a way to exclude pg_largeobject from pg_dump? -T pg_largeobject
> doesn't work. I've tried to exclude tables using OID-datatype also but
> that didn't work either.

Well pg_largeobject is a system catalog so pretty sure it cannot be 
excluded.

What tables are you trying to exclude with OID?

Do you care about not dumping the pg_largeobject table or not dumping 
the data it contains?

> Thanks.
> --
> *Andreas Joseph Krogh*
> CTO / Partner - Visena AS
> Mobile: +47 909 56 963
> andreas@visena.com <mailto:andreas@visena.com>
> www.visena.com <https://www.visena.com>
> <https://www.visena.com>


-- 
Adrian Klaver
adrian.klaver@aklaver.com

  3. Re: Exclude pg_largeobject form pg_dump

    Andreas Joseph Krogh <andreas@visena.com> — 2016-03-08T15:04:54Z

    På tirsdag 08. mars 2016 kl. 15:43:37, skrev Adrian Klaver <
adrian.klaver@aklaver.com <mailto:adrian.klaver@aklaver.com>>:
On 03/08/2016 01:53 AM, Andreas Joseph Krogh wrote:
 > Hi all.
 > Is there a way to exclude pg_largeobject from pg_dump? -T pg_largeobject
 > doesn't work. I've tried to exclude tables using OID-datatype also but
 > that didn't work either.

 Well pg_largeobject is a system catalog so pretty sure it cannot be
 excluded.

 What tables are you trying to exclude with OID?

 Do you care about not dumping the pg_largeobject table or not dumping
 the data it contains?
 
I have several tables with OID-columns and I'd like to dump my DB without any 
data in pg_largeobject (> 95% of the space is occupied by data in 
pg_largeobject).
I've tried to exclude (using -T) the tables containing OID-columns but 
pg_largeobject is still dumped containing the data it seems.
 
-- Andreas Joseph Krogh
CTO / Partner - Visena AS
Mobile: +47 909 56 963
andreas@visena.com <mailto:andreas@visena.com>
www.visena.com <https://www.visena.com>
 <https://www.visena.com>


 

  4. Re: Exclude pg_largeobject form pg_dump

    Tom Lane <tgl@sss.pgh.pa.us> — 2016-03-08T15:30:36Z

    Andreas Joseph Krogh <andreas@visena.com> writes:
> P tirsdag 08. mars 2016 kl. 15:43:37, skrev Adrian Klaver <
> adrian.klaver@aklaver.com <mailto:adrian.klaver@aklaver.com>>:
>  Do you care about not dumping the pg_largeobject table or not dumping
>  the data it contains?
> 
> I have several tables with OID-columns and I'd like to dump my DB without any 
> data in pg_largeobject (> 95% of the space is occupied by data in 
> pg_largeobject).
> I've tried to exclude (using -T) the tables containing OID-columns but 
> pg_largeobject is still dumped containing the data it seems.

A look at the pg_dump source code says that it skips blobs if any of
-s, -n, -t are used.  There's a -b switch to undo that and include
them anyway, but no "inverse -b" to skip them in an otherwise-complete
dump.

So you could do something along the lines of pg_dump -t '*' ...
although this will result in *all* non-schema-named objects being
excluded, I believe, which might be a problem.

			regards, tom lane

  5. Re: Exclude pg_largeobject form pg_dump

    Andreas Joseph Krogh <andreas@visena.com> — 2016-03-08T15:46:11Z

    På tirsdag 08. mars 2016 kl. 16:30:36, skrev Tom Lane <tgl@sss.pgh.pa.us 
<mailto:tgl@sss.pgh.pa.us>>:
Andreas Joseph Krogh <andreas@visena.com> writes:
 > P�� tirsdag 08. mars 2016 kl. 15:43:37, skrev Adrian Klaver <
 > adrian.klaver@aklaver.com <mailto:adrian.klaver@aklaver.com>>:
 >  Do you care about not dumping the pg_largeobject table or not dumping
 >  the data it contains?
 >
 > I have several tables with OID-columns and I'd like to dump my DB without 
any
 > data in pg_largeobject (> 95% of the space is occupied by data in
 > pg_largeobject).
 > I've tried to exclude (using -T) the tables containing OID-columns but
 > pg_largeobject is still dumped containing the data it seems.

 A look at the pg_dump source code says that it skips blobs if any of
 -s, -n, -t are used.  There's a -b switch to undo that and include
 them anyway, but no "inverse -b" to skip them in an otherwise-complete
 dump.

 So you could do something along the lines of pg_dump -t '*' ...
 although this will result in *all* non-schema-named objects being
 excluded, I believe, which might be a problem.

 regards, tom lane
 
Hm:
 
pg_dump -v -t '*' > andreak-noblob.dmp
 pg_dump: reading extensions
 pg_dump: identifying extension members
 pg_dump: reading schemas
 pg_dump: reading user-defined tables
 pg_dump: [archiver (db)] query failed: ERROR:  permission denied for relation 
pg_authid
 pg_dump: [archiver (db)] query was: LOCK TABLE pg_catalog.pg_authid IN ACCESS 
SHARE MODE
 
 
What I'm looking for is "inverse -b" in an otherwise complete dump. Any plans 
to add that?
 

-- Andreas Joseph Krogh
CTO / Partner - Visena AS
Mobile: +47 909 56 963
andreas@visena.com <mailto:andreas@visena.com>
www.visena.com <https://www.visena.com>
 <https://www.visena.com>


 

  6. Re: Exclude pg_largeobject form pg_dump

    Adrian Klaver <adrian.klaver@aklaver.com> — 2016-03-08T15:54:19Z

    On 03/08/2016 07:46 AM, Andreas Joseph Krogh wrote:
> På tirsdag 08. mars 2016 kl. 16:30:36, skrev Tom Lane <tgl@sss.pgh.pa.us
> <mailto:tgl@sss.pgh.pa.us>>:
>
>     Andreas Joseph Krogh <andreas@visena.com> writes:
>      > P�� tirsdag 08. mars 2016 kl. 15:43:37, skrev Adrian Klaver <
>      > adrian.klaver@aklaver.com <mailto:adrian.klaver@aklaver.com>>:
>      >  Do you care about not dumping the pg_largeobject table or not
>     dumping
>      >  the data it contains?
>      >
>      > I have several tables with OID-columns and I'd like to dump my DB
>     without any
>      > data in pg_largeobject (> 95% of the space is occupied by data in
>      > pg_largeobject).
>      > I've tried to exclude (using -T) the tables containing
>     OID-columns but
>      > pg_largeobject is still dumped containing the data it seems.
>
>     A look at the pg_dump source code says that it skips blobs if any of
>     -s, -n, -t are used.  There's a -b switch to undo that and include
>     them anyway, but no "inverse -b" to skip them in an otherwise-complete
>     dump.
>
>     So you could do something along the lines of pg_dump -t '*' ...
>     although this will result in *all* non-schema-named objects being
>     excluded, I believe, which might be a problem.
>
>     regards, tom lane
>
> Hm:
> pg_dump -v -t '*' > andreak-noblob.dmp
> pg_dump: reading extensions
> pg_dump: identifying extension members
> pg_dump: reading schemas
> pg_dump: reading user-defined tables
> pg_dump: [archiver (db)] query failed: ERROR:  permission denied for
> relation pg_authid


Off hand I would say you are running pg_dump as a user that is not a 
superuser:

aklaver@panda:~> pg_dump -v -d test -U aklaver -t '*'
pg_dump: reading extensions
pg_dump: identifying extension members
pg_dump: reading schemas
pg_dump: reading user-defined tables
pg_dump: [archiver (db)] query failed: ERROR:  permission denied for 
relation pg_authid
pg_dump: [archiver (db)] query was: LOCK TABLE pg_catalog.pg_authid IN 
ACCESS SHARE MODE


aklaver@panda:~> pg_dump -v -d test -U postgres -t '*'
pg_dump: reading extensions
pg_dump: identifying extension members
pg_dump: reading schemas
pg_dump: reading user-defined tables
pg_dump: reading user-defined functions
pg_dump: reading user-defined types
pg_dump: reading procedural languages
pg_dump: reading user-defined aggregate functions
pg_dump: reading user-defined operators
pg_dump: reading user-defined operator classes
pg_dump: reading user-defined operator families
pg_dump: reading user-defined text search parsers
pg_dump: reading user-defined text search templates
pg_dump: reading user-defined text search dictionaries
....


> pg_dump: [archiver (db)] query was: LOCK TABLE pg_catalog.pg_authid IN
> ACCESS SHARE MODE
> What I'm looking for is "inverse -b" in an otherwise complete dump. Any
> plans to add that?
> --
> *Andreas Joseph Krogh*
> CTO / Partner - Visena AS
> Mobile: +47 909 56 963
> andreas@visena.com <mailto:andreas@visena.com>
> www.visena.com <https://www.visena.com>
> <https://www.visena.com>


-- 
Adrian Klaver
adrian.klaver@aklaver.com

  7. Re: Exclude pg_largeobject form pg_dump

    Tom Lane <tgl@sss.pgh.pa.us> — 2016-03-08T15:57:01Z

    Andreas Joseph Krogh <andreas@visena.com> writes:
> What I'm looking for is "inverse -b" in an otherwise complete dump. Any plans 
> to add that?

[ shrug... ]  Nobody ever asked for it before.

			regards, tom lane

  8. Re: Exclude pg_largeobject form pg_dump

    Andreas Joseph Krogh <andreas@visena.com> — 2016-03-08T15:58:07Z

    På tirsdag 08. mars 2016 kl. 16:54:19, skrev Adrian Klaver <
adrian.klaver@aklaver.com <mailto:adrian.klaver@aklaver.com>>:
On 03/08/2016 07:46 AM, Andreas Joseph Krogh wrote:
 > På tirsdag 08. mars 2016 kl. 16:30:36, skrev Tom Lane <tgl@sss.pgh.pa.us
 > <mailto:tgl@sss.pgh.pa.us>>:
 >
 >     Andreas Joseph Krogh <andreas@visena.com> writes:
 >      > P�� tirsdag 08. mars 2016 kl. 15:43:37, skrev Adrian Klaver <
 >      > adrian.klaver@aklaver.com <mailto:adrian.klaver@aklaver.com>>:
 >      >  Do you care about not dumping the pg_largeobject table or not
 >     dumping
 >      >  the data it contains?
 >      >
 >      > I have several tables with OID-columns and I'd like to dump my DB
 >     without any
 >      > data in pg_largeobject (> 95% of the space is occupied by data in
 >      > pg_largeobject).
 >      > I've tried to exclude (using -T) the tables containing
 >     OID-columns but
 >      > pg_largeobject is still dumped containing the data it seems.
 >
 >     A look at the pg_dump source code says that it skips blobs if any of
 >     -s, -n, -t are used.  There's a -b switch to undo that and include
 >     them anyway, but no "inverse -b" to skip them in an otherwise-complete
 >     dump.
 >
 >     So you could do something along the lines of pg_dump -t '*' ...
 >     although this will result in *all* non-schema-named objects being
 >     excluded, I believe, which might be a problem.
 >
 >     regards, tom lane
 >
 > Hm:
 > pg_dump -v -t '*' > andreak-noblob.dmp
 > pg_dump: reading extensions
 > pg_dump: identifying extension members
 > pg_dump: reading schemas
 > pg_dump: reading user-defined tables
 > pg_dump: [archiver (db)] query failed: ERROR:  permission denied for
 > relation pg_authid


 Off hand I would say you are running pg_dump as a user that is not a
 superuser:
 [snip]
 
Yes, since when should I not be able to dump a DB (owned by a non-superuser) 
as that user?
 
-- Andreas Joseph Krogh
CTO / Partner - Visena AS
Mobile: +47 909 56 963
andreas@visena.com <mailto:andreas@visena.com>
www.visena.com <https://www.visena.com>
 <https://www.visena.com>


 

  9. Re: Exclude pg_largeobject form pg_dump

    Andreas Joseph Krogh <andreas@visena.com> — 2016-03-08T16:02:45Z

    På tirsdag 08. mars 2016 kl. 16:57:01, skrev Tom Lane <tgl@sss.pgh.pa.us 
<mailto:tgl@sss.pgh.pa.us>>:
Andreas Joseph Krogh <andreas@visena.com> writes:
 > What I'm looking for is "inverse -b" in an otherwise complete dump. Any 
plans
 > to add that?

 [ shrug... ]  Nobody ever asked for it before.

 regards, tom lane
 
It surely helps testing production-datasets which contain lots of BLOBs where 
one wants to dump the production-data into a test-env. We have >1TB databases 
containing > 95% blobs so it would help us tremendously to have this option.
 
-- Andreas Joseph Krogh
CTO / Partner - Visena AS
Mobile: +47 909 56 963
andreas@visena.com <mailto:andreas@visena.com>
www.visena.com <https://www.visena.com>
 <https://www.visena.com>


 

  10. Re: Exclude pg_largeobject form pg_dump

    Melvin Davidson <melvin6925@gmail.com> — 2016-03-08T16:04:15Z

    On Tue, Mar 8, 2016 at 11:02 AM, Andreas Joseph Krogh <andreas@visena.com>
wrote:

> På tirsdag 08. mars 2016 kl. 16:57:01, skrev Tom Lane <tgl@sss.pgh.pa.us>:
>
> Andreas Joseph Krogh <andreas@visena.com> writes:
> > What I'm looking for is "inverse -b" in an otherwise complete dump. Any
> plans
> > to add that?
>
> [ shrug... ]  Nobody ever asked for it before.
>
> regards, tom lane
>
>
> It surely helps testing production-datasets which contain lots of BLOBs
> where one wants to dump the production-data into a test-env. We have >1TB
> databases containing > 95% blobs so it would help us tremendously to have
> this option.
>
> --
> *Andreas Joseph Krogh*
> CTO / Partner - Visena AS
> Mobile: +47 909 56 963
> andreas@visena.com
> www.visena.com
> <https://www.visena.com>
>
>

Probably you need to redesign the schema.

Move the blobs to a new/separate child table. Then you can exclude them.

-- 
*Melvin Davidson*
I reserve the right to fantasize.  Whether or not you
wish to share my fantasy is entirely up to you.

  11. Re: Exclude pg_largeobject form pg_dump

    Tom Lane <tgl@sss.pgh.pa.us> — 2016-03-08T16:10:43Z

    Andreas Joseph Krogh <andreas@visena.com> writes:
> P tirsdag 08. mars 2016 kl. 16:54:19, skrev Adrian Klaver <
> adrian.klaver@aklaver.com <mailto:adrian.klaver@aklaver.com>>:
>  Off hand I would say you are running pg_dump as a user that is not a
>  superuser:

> Yes, since when should I not be able to dump a DB (owned by a non-superuser) 
> as that user?

The problem is that -t '*' is being interpreted as matching system
catalogs.  You might be able to get somewhere with

pg_dump -t '*' -N pg_catalog ...

Probably we should fix pg_dump so it doesn't try to dump system catalogs
as tables, even if the switches seem to ask it to.

			regards, tom lane

  12. Re: Exclude pg_largeobject form pg_dump

    Andreas Joseph Krogh <andreas@visena.com> — 2016-03-08T16:13:55Z

    På tirsdag 08. mars 2016 kl. 17:10:43, skrev Tom Lane <tgl@sss.pgh.pa.us 
<mailto:tgl@sss.pgh.pa.us>>:
Andreas Joseph Krogh <andreas@visena.com> writes:
 > P�� tirsdag 08. mars 2016 kl. 16:54:19, skrev Adrian Klaver <
 > adrian.klaver@aklaver.com <mailto:adrian.klaver@aklaver.com>>:
 >  Off hand I would say you are running pg_dump as a user that is not a
 >  superuser:

 > Yes, since when should I not be able to dump a DB (owned by a non-superuser)
 > as that user?

 The problem is that -t '*' is being interpreted as matching system
 catalogs.  You might be able to get somewhere with

 pg_dump -t '*' -N pg_catalog ...

 Probably we should fix pg_dump so it doesn't try to dump system catalogs
 as tables, even if the switches seem to ask it to.

 regards, tom lane
 
That didn't work either:
pg_dump -t '*' -N pg_catalog
 pg_dump: [archiver (db)] query failed: ERROR:  permission denied for relation 
pg_authid
 pg_dump: [archiver (db)] query was: LOCK TABLE pg_catalog.pg_authid IN ACCESS 
SHARE MODE

 
-- Andreas Joseph Krogh
CTO / Partner - Visena AS
Mobile: +47 909 56 963
andreas@visena.com <mailto:andreas@visena.com>
www.visena.com <https://www.visena.com>
 <https://www.visena.com>


 

  13. Re: Exclude pg_largeobject form pg_dump

    Joshua D. Drake <jd@commandprompt.com> — 2016-03-08T16:38:04Z

    On 03/08/2016 08:02 AM, Andreas Joseph Krogh wrote:
> På tirsdag 08. mars 2016 kl. 16:57:01, skrev Tom Lane <tgl@sss.pgh.pa.us
> <mailto:tgl@sss.pgh.pa.us>>:
>
>     Andreas Joseph Krogh <andreas@visena.com> writes:
>      > What I'm looking for is "inverse -b" in an otherwise complete
>     dump. Any plans
>      > to add that?
>
>     [ shrug... ]  Nobody ever asked for it before.
>
>     regards, tom lane
>
> It surely helps testing production-datasets which contain lots of BLOBs
> where one wants to dump the production-data into a test-env. We have
>  >1TB databases containing > 95% blobs so it would help us tremendously
> to have this option.

I have quite a few customers that would benefit from the ability to not 
have blobs present in dumps.

JD


-- 
Command Prompt, Inc.                  http://the.postgres.company/
                         +1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.

  14. Re: Exclude pg_largeobject form pg_dump

    Andreas Joseph Krogh <andreas@visena.com> — 2016-03-08T16:45:14Z

    På tirsdag 08. mars 2016 kl. 17:38:04, skrev Joshua D. Drake <
jd@commandprompt.com <mailto:jd@commandprompt.com>>:
On 03/08/2016 08:02 AM, Andreas Joseph Krogh wrote:
 > På tirsdag 08. mars 2016 kl. 16:57:01, skrev Tom Lane <tgl@sss.pgh.pa.us
 > <mailto:tgl@sss.pgh.pa.us>>:
 >
 >     Andreas Joseph Krogh <andreas@visena.com> writes:
 >      > What I'm looking for is "inverse -b" in an otherwise complete
 >     dump. Any plans
 >      > to add that?
 >
 >     [ shrug... ]  Nobody ever asked for it before.
 >
 >     regards, tom lane
 >
 > It surely helps testing production-datasets which contain lots of BLOBs
 > where one wants to dump the production-data into a test-env. We have
 >  >1TB databases containing > 95% blobs so it would help us tremendously
 > to have this option.

 I have quite a few customers that would benefit from the ability to not
 have blobs present in dumps.
 
Great! So how do we proceed to get "--no-blobs" added to pg_dump?
Maybe CommandPrompt and Visena should co-fund development of such an addition, 
if it's accepted by -hackers?
We'd be willing to pay for such an addition for the 9.5 branch, as a patch.
 
-- Andreas Joseph Krogh
CTO / Partner - Visena AS
Mobile: +47 909 56 963
andreas@visena.com <mailto:andreas@visena.com>
www.visena.com <https://www.visena.com>
 <https://www.visena.com>


 

  15. Re: Exclude pg_largeobject form pg_dump

    David G. Johnston <david.g.johnston@gmail.com> — 2016-03-08T20:03:01Z

    On Tue, Mar 8, 2016 at 9:45 AM, Andreas Joseph Krogh <andreas@visena.com>
wrote:

> På tirsdag 08. mars 2016 kl. 17:38:04, skrev Joshua D. Drake <
> jd@commandprompt.com>:
>
> On 03/08/2016 08:02 AM, Andreas Joseph Krogh wrote:
> > På tirsdag 08. mars 2016 kl. 16:57:01, skrev Tom Lane <tgl@sss.pgh.pa.us
> > <mailto:tgl@sss.pgh.pa.us>>:
> >
> >     Andreas Joseph Krogh <andreas@visena.com> writes:
> >      > What I'm looking for is "inverse -b" in an otherwise complete
> >     dump. Any plans
> >      > to add that?
> >
> >     [ shrug... ]  Nobody ever asked for it before.
> >
> >     regards, tom lane
> >
> > It surely helps testing production-datasets which contain lots of BLOBs
> > where one wants to dump the production-data into a test-env. We have
> >  >1TB databases containing > 95% blobs so it would help us tremendously
> > to have this option.
>
> I have quite a few customers that would benefit from the ability to not
> have blobs present in dumps.
>
>
> Great! So how do we proceed to get "--no-blobs" added to pg_dump?
> Maybe CommandPrompt and Visena should co-fund development of such an
> addition, if it's accepted by -hackers?
> We'd be willing to pay for such an addition for the 9.5 branch, as a patch.
>

​Unfortunately this doesn't qualify as a bug fix - it is a new feature and
thus is ineligible for inclusion in official 9.5

David J.

  16. Re: Exclude pg_largeobject form pg_dump

    Andreas Joseph Krogh <andreas@visena.com> — 2016-03-08T20:06:30Z

    På tirsdag 08. mars 2016 kl. 21:03:01, skrev David G. Johnston <
david.g.johnston@gmail.com <mailto:david.g.johnston@gmail.com>>:
On Tue, Mar 8, 2016 at 9:45 AM, Andreas Joseph Krogh <andreas@visena.com 
<mailto:andreas@visena.com>> wrote:
På tirsdag 08. mars 2016 kl. 17:38:04, skrev Joshua D. Drake <
jd@commandprompt.com <mailto:jd@commandprompt.com>>:
On 03/08/2016 08:02 AM, Andreas Joseph Krogh wrote:
 > På tirsdag 08. mars 2016 kl. 16:57:01, skrev Tom Lane <tgl@sss.pgh.pa.us 
<mailto:tgl@sss.pgh.pa.us>
 > <mailto:tgl@sss.pgh.pa.us <mailto:tgl@sss.pgh.pa.us>>>:
 >
 >     Andreas Joseph Krogh <andreas@visena.com <mailto:andreas@visena.com>> 
writes:
 >      > What I'm looking for is "inverse -b" in an otherwise complete
 >     dump. Any plans
 >      > to add that?
 >
 >     [ shrug... ]  Nobody ever asked for it before.
 >
 >     regards, tom lane
 >
 > It surely helps testing production-datasets which contain lots of BLOBs
 > where one wants to dump the production-data into a test-env. We have
 >  >1TB databases containing > 95% blobs so it would help us tremendously
 > to have this option.

 I have quite a few customers that would benefit from the ability to not
 have blobs present in dumps.
 
Great! So how do we proceed to get "--no-blobs" added to pg_dump?
Maybe CommandPrompt and Visena should co-fund development of such an addition, 
if it's accepted by -hackers?
We'd be willing to pay for such an addition for the 9.5 branch, as a patch.
 
​Unfortunately this doesn't qualify as a bug fix - it is a new feature and 
thus is ineligible for inclusion in official 9.5

 

David J.




 
Of course. That's why I mentioned that, if possible, an unofficial patch to 
9.5 could be developed, funded partly by Visena (my company). Given that 
someone is willing to do this of course.
 
-- Andreas Joseph Krogh
CTO / Partner - Visena AS
Mobile: +47 909 56 963
andreas@visena.com <mailto:andreas@visena.com>
www.visena.com <https://www.visena.com>
 <https://www.visena.com>


 

  17. Re: Exclude pg_largeobject form pg_dump

    Guillaume Lelarge <guillaume@lelarge.info> — 2016-10-23T15:06:57Z

    2016-03-08 21:06 GMT+01:00 Andreas Joseph Krogh <andreas@visena.com>:

> På tirsdag 08. mars 2016 kl. 21:03:01, skrev David G. Johnston <
> david.g.johnston@gmail.com>:
>
> On Tue, Mar 8, 2016 at 9:45 AM, Andreas Joseph Krogh <andreas@visena.com>
> wrote:
>
>> På tirsdag 08. mars 2016 kl. 17:38:04, skrev Joshua D. Drake <
>> jd@commandprompt.com>:
>>
>> On 03/08/2016 08:02 AM, Andreas Joseph Krogh wrote:
>> > På tirsdag 08. mars 2016 kl. 16:57:01, skrev Tom Lane <
>> tgl@sss.pgh.pa.us
>> > <mailto:tgl@sss.pgh.pa.us>>:
>> >
>> >     Andreas Joseph Krogh <andreas@visena.com> writes:
>> >      > What I'm looking for is "inverse -b" in an otherwise complete
>> >     dump. Any plans
>> >      > to add that?
>> >
>> >     [ shrug... ]  Nobody ever asked for it before.
>> >
>> >     regards, tom lane
>> >
>> > It surely helps testing production-datasets which contain lots of BLOBs
>> > where one wants to dump the production-data into a test-env. We have
>> >  >1TB databases containing > 95% blobs so it would help us tremendously
>> > to have this option.
>>
>> I have quite a few customers that would benefit from the ability to not
>> have blobs present in dumps.
>>
>>
>> Great! So how do we proceed to get "--no-blobs" added to pg_dump?
>> Maybe CommandPrompt and Visena should co-fund development of such an
>> addition, if it's accepted by -hackers?
>> We'd be willing to pay for such an addition for the 9.5 branch, as a
>> patch.
>>
>
> ​Unfortunately this doesn't qualify as a bug fix - it is a new feature and
> thus is ineligible for inclusion in official 9.5
>
> David J.
>
>
> Of course. That's why I mentioned that, if possible, an unofficial patch
> to 9.5 could be developed, funded partly by Visena (my company). Given that
> someone is willing to do this of course.
>
>

That probably should look like the patch attached. It applies cleanly on
HEAD, and works AFAICT. If this patch seems interesting enough, I'll add it
to the next commit fest (note for myself, update the ref/pg_dump.sgml
documentation file).

For Andreas' information, it also applies on 9.5, though I didn't check if
it worked afterwards.


-- 
Guillaume.
  http://blog.guillaume.lelarge.info
  http://www.dalibo.com

  18. Re: Exclude pg_largeobject form pg_dump

    Andreas Joseph Krogh <andreas@visena.com> — 2016-10-23T17:15:17Z

    På søndag 23. oktober 2016 kl. 17:06:57, skrev Guillaume Lelarge <
guillaume@lelarge.info <mailto:guillaume@lelarge.info>>:
2016-03-08 21:06 GMT+01:00 Andreas Joseph Krogh <andreas@visena.com 
<mailto:andreas@visena.com>>: På tirsdag 08. mars 2016 kl. 21:03:01, skrev 
David G. Johnston <david.g.johnston@gmail.com 
<mailto:david.g.johnston@gmail.com>>:
On Tue, Mar 8, 2016 at 9:45 AM, Andreas Joseph Krogh <andreas@visena.com 
<mailto:andreas@visena.com>> wrote:
På tirsdag 08. mars 2016 kl. 17:38:04, skrev Joshua D. Drake <
jd@commandprompt.com <mailto:jd@commandprompt.com>>:
On 03/08/2016 08:02 AM, Andreas Joseph Krogh wrote:
 > På tirsdag 08. mars 2016 kl. 16:57:01, skrev Tom Lane <tgl@sss.pgh.pa.us 
<mailto:tgl@sss.pgh.pa.us>
 > <mailto:tgl@sss.pgh.pa.us <mailto:tgl@sss.pgh.pa.us>>>:
 >
 >     Andreas Joseph Krogh <andreas@visena.com <mailto:andreas@visena.com>> 
writes:
 >      > What I'm looking for is "inverse -b" in an otherwise complete
 >     dump. Any plans
 >      > to add that?
 >
 >     [ shrug... ]  Nobody ever asked for it before.
 >
 >     regards, tom lane
 >
 > It surely helps testing production-datasets which contain lots of BLOBs
 > where one wants to dump the production-data into a test-env. We have
 >  >1TB databases containing > 95% blobs so it would help us tremendously
 > to have this option.

 I have quite a few customers that would benefit from the ability to not
 have blobs present in dumps.
 
Great! So how do we proceed to get "--no-blobs" added to pg_dump?
Maybe CommandPrompt and Visena should co-fund development of such an addition, 
if it's accepted by -hackers?
We'd be willing to pay for such an addition for the 9.5 branch, as a patch.
 
​Unfortunately this doesn't qualify as a bug fix - it is a new feature and 
thus is ineligible for inclusion in official 9.5

 

David J.




 


Of course. That's why I mentioned that, if possible, an unofficial patch to 
9.5 could be developed, funded partly by Visena (my company). Given that 
someone is willing to do this of course.
 


 
That probably should look like the patch attached. It applies cleanly on HEAD, 
and works AFAICT. If this patch seems interesting enough, I'll add it to the 
next commit fest (note for myself, update the ref/pg_dump.sgml documentation 
file).

 For Andreas' information, it also applies on 9.5, though I didn't check if it 
worked afterwards.



 
+1 for adding it to the commitfest.
 
It's almost scary how simple this patch is and noone ever got around to 
implement it.
 
Thanks, I'll test it on 9.5 soon.
 
-- Andreas Joseph Krogh
CTO / Partner - Visena AS
Mobile: +47 909 56 963
andreas@visena.com <mailto:andreas@visena.com>
www.visena.com <https://www.visena.com>
 <https://www.visena.com>


 

  19. Re: Exclude pg_largeobject form pg_dump

    Andreas Joseph Krogh <andreas@visena.com> — 2016-10-23T18:37:30Z

    På søndag 23. oktober 2016 kl. 19:15:17, skrev Andreas Joseph Krogh <
andreas@visena.com <mailto:andreas@visena.com>>:
På søndag 23. oktober 2016 kl. 17:06:57, skrev Guillaume Lelarge <
guillaume@lelarge.info <mailto:guillaume@lelarge.info>>:
2016-03-08 21:06 GMT+01:00 Andreas Joseph Krogh <andreas@visena.com 
<mailto:andreas@visena.com>>: På tirsdag 08. mars 2016 kl. 21:03:01, skrev 
David G. Johnston <david.g.johnston@gmail.com 
<mailto:david.g.johnston@gmail.com>>:
On Tue, Mar 8, 2016 at 9:45 AM, Andreas Joseph Krogh <andreas@visena.com 
<mailto:andreas@visena.com>> wrote:
På tirsdag 08. mars 2016 kl. 17:38:04, skrev Joshua D. Drake <
jd@commandprompt.com <mailto:jd@commandprompt.com>>:
On 03/08/2016 08:02 AM, Andreas Joseph Krogh wrote:
 > På tirsdag 08. mars 2016 kl. 16:57:01, skrev Tom Lane <tgl@sss.pgh.pa.us 
<mailto:tgl@sss.pgh.pa.us>
 > <mailto:tgl@sss.pgh.pa.us <mailto:tgl@sss.pgh.pa.us>>>:
 >
 >     Andreas Joseph Krogh <andreas@visena.com <mailto:andreas@visena.com>> 
writes:
 >      > What I'm looking for is "inverse -b" in an otherwise complete
 >     dump. Any plans
 >      > to add that?
 >
 >     [ shrug... ]  Nobody ever asked for it before.
 >
 >     regards, tom lane
 >
 > It surely helps testing production-datasets which contain lots of BLOBs
 > where one wants to dump the production-data into a test-env. We have
 >  >1TB databases containing > 95% blobs so it would help us tremendously
 > to have this option.

 I have quite a few customers that would benefit from the ability to not
 have blobs present in dumps.
 
Great! So how do we proceed to get "--no-blobs" added to pg_dump?
Maybe CommandPrompt and Visena should co-fund development of such an addition, 
if it's accepted by -hackers?
We'd be willing to pay for such an addition for the 9.5 branch, as a patch.
 
​Unfortunately this doesn't qualify as a bug fix - it is a new feature and 
thus is ineligible for inclusion in official 9.5

 

David J.




 


Of course. That's why I mentioned that, if possible, an unofficial patch to 
9.5 could be developed, funded partly by Visena (my company). Given that 
someone is willing to do this of course.
 


 
That probably should look like the patch attached. It applies cleanly on HEAD, 
and works AFAICT. If this patch seems interesting enough, I'll add it to the 
next commit fest (note for myself, update the ref/pg_dump.sgml documentation 
file).

 For Andreas' information, it also applies on 9.5, though I didn't check if it 
worked afterwards.



 
+1 for adding it to the commitfest.
 
It's almost scary how simple this patch is and noone ever got around to 
implement it.
 
Thanks, I'll test it on 9.5 soon.
 
It's totally OK for me to use 9.6 (now that it's released) to dump 9.5 DBs, so 
I'm all good with this patch, thanks!
 
-- Andreas Joseph Krogh
CTO / Partner - Visena AS
Mobile: +47 909 56 963
andreas@visena.com <mailto:andreas@visena.com>
www.visena.com <https://www.visena.com>
 <https://www.visena.com>


 

  20. Re: Exclude pg_largeobject form pg_dump

    Guillaume Lelarge <guillaume@lelarge.info> — 2016-10-23T18:44:34Z

    2016-10-23 20:37 GMT+02:00 Andreas Joseph Krogh <andreas@visena.com>:

> På søndag 23. oktober 2016 kl. 19:15:17, skrev Andreas Joseph Krogh <
> andreas@visena.com>:
>
> På søndag 23. oktober 2016 kl. 17:06:57, skrev Guillaume Lelarge <
> guillaume@lelarge.info>:
>
> 2016-03-08 21:06 GMT+01:00 Andreas Joseph Krogh <andreas@visena.com>:
>>
>> På tirsdag 08. mars 2016 kl. 21:03:01, skrev David G. Johnston <
>> david.g.johnston@gmail.com>:
>>
>> On Tue, Mar 8, 2016 at 9:45 AM, Andreas Joseph Krogh <andreas@visena.com>
>> wrote:
>>
>>> På tirsdag 08. mars 2016 kl. 17:38:04, skrev Joshua D. Drake <
>>> jd@commandprompt.com>:
>>>
>>> On 03/08/2016 08:02 AM, Andreas Joseph Krogh wrote:
>>> > På tirsdag 08. mars 2016 kl. 16:57:01, skrev Tom Lane <
>>> tgl@sss.pgh.pa.us
>>> > <mailto:tgl@sss.pgh.pa.us>>:
>>> >
>>> >     Andreas Joseph Krogh <andreas@visena.com> writes:
>>> >      > What I'm looking for is "inverse -b" in an otherwise complete
>>> >     dump. Any plans
>>> >      > to add that?
>>> >
>>> >     [ shrug... ]  Nobody ever asked for it before.
>>> >
>>> >     regards, tom lane
>>> >
>>> > It surely helps testing production-datasets which contain lots of BLOBs
>>> > where one wants to dump the production-data into a test-env. We have
>>> >  >1TB databases containing > 95% blobs so it would help us tremendously
>>> > to have this option.
>>>
>>> I have quite a few customers that would benefit from the ability to not
>>> have blobs present in dumps.
>>>
>>>
>>> Great! So how do we proceed to get "--no-blobs" added to pg_dump?
>>> Maybe CommandPrompt and Visena should co-fund development of such an
>>> addition, if it's accepted by -hackers?
>>> We'd be willing to pay for such an addition for the 9.5 branch, as a
>>> patch.
>>>
>>
>> ​Unfortunately this doesn't qualify as a bug fix - it is a new feature
>> and thus is ineligible for inclusion in official 9.5
>>
>> David J.
>>
>>
>> Of course. That's why I mentioned that, if possible, an unofficial patch
>> to 9.5 could be developed, funded partly by Visena (my company). Given that
>> someone is willing to do this of course.
>>
>>
>
> That probably should look like the patch attached. It applies cleanly on
> HEAD, and works AFAICT. If this patch seems interesting enough, I'll add it
> to the next commit fest (note for myself, update the ref/pg_dump.sgml
> documentation file).
>
> For Andreas' information, it also applies on 9.5, though I didn't check if
> it worked afterwards.
>
>
> +1 for adding it to the commitfest.
>
>
>
Done, https://commitfest.postgresql.org/11/833/

It's almost scary how simple this patch is and noone ever got around to
> implement it.
>
>
Nobody had the time (like me, till now) or the motivation.


>
> Thanks, I'll test it on 9.5 soon.
>
>
> It's totally OK for me to use 9.6 (now that it's released) to dump 9.5
> DBs, so I'm all good with this patch, thanks!
>

Remember that, if it gets commited, it'll be for next release (aka 10), and
not 9.6 and earlier.


-- 
Guillaume.
  http://blog.guillaume.lelarge.info
  http://www.dalibo.com

  21. Re: Exclude pg_largeobject form pg_dump

    Guillaume Lelarge <guillaume@lelarge.info> — 2016-10-23T18:49:11Z

    2016-10-23 20:44 GMT+02:00 Guillaume Lelarge <guillaume@lelarge.info>:

> 2016-10-23 20:37 GMT+02:00 Andreas Joseph Krogh <andreas@visena.com>:
>
>> På søndag 23. oktober 2016 kl. 19:15:17, skrev Andreas Joseph Krogh <
>> andreas@visena.com>:
>>
>> På søndag 23. oktober 2016 kl. 17:06:57, skrev Guillaume Lelarge <
>> guillaume@lelarge.info>:
>>
>> 2016-03-08 21:06 GMT+01:00 Andreas Joseph Krogh <andreas@visena.com>:
>>>
>>> På tirsdag 08. mars 2016 kl. 21:03:01, skrev David G. Johnston <
>>> david.g.johnston@gmail.com>:
>>>
>>> On Tue, Mar 8, 2016 at 9:45 AM, Andreas Joseph Krogh <andreas@visena.com
>>> > wrote:
>>>
>>>> På tirsdag 08. mars 2016 kl. 17:38:04, skrev Joshua D. Drake <
>>>> jd@commandprompt.com>:
>>>>
>>>> On 03/08/2016 08:02 AM, Andreas Joseph Krogh wrote:
>>>> > På tirsdag 08. mars 2016 kl. 16:57:01, skrev Tom Lane <
>>>> tgl@sss.pgh.pa.us
>>>> > <mailto:tgl@sss.pgh.pa.us>>:
>>>> >
>>>> >     Andreas Joseph Krogh <andreas@visena.com> writes:
>>>> >      > What I'm looking for is "inverse -b" in an otherwise complete
>>>> >     dump. Any plans
>>>> >      > to add that?
>>>> >
>>>> >     [ shrug... ]  Nobody ever asked for it before.
>>>> >
>>>> >     regards, tom lane
>>>> >
>>>> > It surely helps testing production-datasets which contain lots of
>>>> BLOBs
>>>> > where one wants to dump the production-data into a test-env. We have
>>>> >  >1TB databases containing > 95% blobs so it would help us
>>>> tremendously
>>>> > to have this option.
>>>>
>>>> I have quite a few customers that would benefit from the ability to not
>>>> have blobs present in dumps.
>>>>
>>>>
>>>> Great! So how do we proceed to get "--no-blobs" added to pg_dump?
>>>> Maybe CommandPrompt and Visena should co-fund development of such an
>>>> addition, if it's accepted by -hackers?
>>>> We'd be willing to pay for such an addition for the 9.5 branch, as a
>>>> patch.
>>>>
>>>
>>> ​Unfortunately this doesn't qualify as a bug fix - it is a new feature
>>> and thus is ineligible for inclusion in official 9.5
>>>
>>> David J.
>>>
>>>
>>> Of course. That's why I mentioned that, if possible, an unofficial patch
>>> to 9.5 could be developed, funded partly by Visena (my company). Given that
>>> someone is willing to do this of course.
>>>
>>>
>>
>> That probably should look like the patch attached. It applies cleanly on
>> HEAD, and works AFAICT. If this patch seems interesting enough, I'll add it
>> to the next commit fest (note for myself, update the ref/pg_dump.sgml
>> documentation file).
>>
>> For Andreas' information, it also applies on 9.5, though I didn't check
>> if it worked afterwards.
>>
>>
>> +1 for adding it to the commitfest.
>>
>>
>>
> Done, https://commitfest.postgresql.org/11/833/
>
> It's almost scary how simple this patch is and noone ever got around to
>> implement it.
>>
>>
> Nobody had the time (like me, till now) or the motivation.
>
>
>>
>> Thanks, I'll test it on 9.5 soon.
>>
>>
>> It's totally OK for me to use 9.6 (now that it's released) to dump 9.5
>> DBs, so I'm all good with this patch, thanks!
>>
>
> Remember that, if it gets commited, it'll be for next release (aka 10),
> and not 9.6 and earlier.
>
>
New patch, this time with the documentation.


-- 
Guillaume.
  http://blog.guillaume.lelarge.info
  http://www.dalibo.com

  22. Re: Exclude pg_largeobject form pg_dump

    Andreas Joseph Krogh <andreas@visena.com> — 2016-10-23T21:58:45Z

    På søndag 23. oktober 2016 kl. 20:44:34, skrev Guillaume Lelarge <
guillaume@lelarge.info <mailto:guillaume@lelarge.info>>:
[snip]
Remember that, if it gets commited, it'll be for next release (aka 10), and 
not 9.6 and earlier.



 
The patch working for 9.6 is fine for me, but getting this in master is of 
course the best.
 
-- Andreas Joseph Krogh
CTO / Partner - Visena AS
Mobile: +47 909 56 963
andreas@visena.com <mailto:andreas@visena.com>
www.visena.com <https://www.visena.com>
 <https://www.visena.com>


 

  23. Re: Exclude pg_largeobject form pg_dump

    Amul Sul <sulamul@gmail.com> — 2016-11-03T07:22:32Z

    Hi Guillaume,

With your v2 patch, -B options working as expected but --no-blobs
options is still unrecognized, this happens is because of you have
forgot to add entry for 'no-blobs' in long_options[] array.

Apart from this concern patch looks good to me. Thanks

Regards,
Amul

The new status of this patch is: Waiting on Author

On Mon, Oct 24, 2016 at 12:19 AM, Guillaume Lelarge
<guillaume@lelarge.info> wrote:
> 2016-10-23 20:44 GMT+02:00 Guillaume Lelarge <guillaume@lelarge.info>:
>>
>> 2016-10-23 20:37 GMT+02:00 Andreas Joseph Krogh <andreas@visena.com>:
>>>
>>> På søndag 23. oktober 2016 kl. 19:15:17, skrev Andreas Joseph Krogh
>>> <andreas@visena.com>:
>>>
>>> På søndag 23. oktober 2016 kl. 17:06:57, skrev Guillaume Lelarge
>>> <guillaume@lelarge.info>:
>>>
>>> 2016-03-08 21:06 GMT+01:00 Andreas Joseph Krogh <andreas@visena.com>:
>>>>
>>>> På tirsdag 08. mars 2016 kl. 21:03:01, skrev David G. Johnston
>>>> <david.g.johnston@gmail.com>:
>>>>
>>>> On Tue, Mar 8, 2016 at 9:45 AM, Andreas Joseph Krogh
>>>> <andreas@visena.com> wrote:
>>>>>
>>>>> På tirsdag 08. mars 2016 kl. 17:38:04, skrev Joshua D. Drake
>>>>> <jd@commandprompt.com>:
>>>>>
>>>>> On 03/08/2016 08:02 AM, Andreas Joseph Krogh wrote:
>>>>> > På tirsdag 08. mars 2016 kl. 16:57:01, skrev Tom Lane
>>>>> > <tgl@sss.pgh.pa.us
>>>>> > <mailto:tgl@sss.pgh.pa.us>>:
>>>>> >
>>>>> >     Andreas Joseph Krogh <andreas@visena.com> writes:
>>>>> >      > What I'm looking for is "inverse -b" in an otherwise complete
>>>>> >     dump. Any plans
>>>>> >      > to add that?
>>>>> >
>>>>> >     [ shrug... ]  Nobody ever asked for it before.
>>>>> >
>>>>> >     regards, tom lane
>>>>> >
>>>>> > It surely helps testing production-datasets which contain lots of
>>>>> > BLOBs
>>>>> > where one wants to dump the production-data into a test-env. We have
>>>>> >  >1TB databases containing > 95% blobs so it would help us
>>>>> > tremendously
>>>>> > to have this option.
>>>>>
>>>>> I have quite a few customers that would benefit from the ability to not
>>>>> have blobs present in dumps.
>>>>>
>>>>>
>>>>> Great! So how do we proceed to get "--no-blobs" added to pg_dump?
>>>>> Maybe CommandPrompt and Visena should co-fund development of such an
>>>>> addition, if it's accepted by -hackers?
>>>>> We'd be willing to pay for such an addition for the 9.5 branch, as a
>>>>> patch.
>>>>
>>>>
>>>> Unfortunately this doesn't qualify as a bug fix - it is a new feature
>>>> and thus is ineligible for inclusion in official 9.5
>>>>
>>>> David J.
>>>>
>>>>
>>>> Of course. That's why I mentioned that, if possible, an unofficial patch
>>>> to 9.5 could be developed, funded partly by Visena (my company). Given that
>>>> someone is willing to do this of course.
>>>>
>>>
>>>
>>> That probably should look like the patch attached. It applies cleanly on
>>> HEAD, and works AFAICT. If this patch seems interesting enough, I'll add it
>>> to the next commit fest (note for myself, update the ref/pg_dump.sgml
>>> documentation file).
>>>
>>> For Andreas' information, it also applies on 9.5, though I didn't check
>>> if it worked afterwards.
>>>
>>>
>>> +1 for adding it to the commitfest.
>>>
>>
>>
>> Done, https://commitfest.postgresql.org/11/833/
>>
>>> It's almost scary how simple this patch is and noone ever got around to
>>> implement it.
>>
>>
>> Nobody had the time (like me, till now) or the motivation.
>>
>>>
>>>
>>> Thanks, I'll test it on 9.5 soon.
>>>
>>>
>>> It's totally OK for me to use 9.6 (now that it's released) to dump 9.5
>>> DBs, so I'm all good with this patch, thanks!
>>
>>
>> Remember that, if it gets commited, it'll be for next release (aka 10),
>> and not 9.6 and earlier.
>>
>
> New patch, this time with the documentation.
>
>
> --
> Guillaume.
>   http://blog.guillaume.lelarge.info
>   http://www.dalibo.com
>
>
> --
> Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-general
>

  24. Re: Exclude pg_largeobject form pg_dump

    Guillaume Lelarge <guillaume@lelarge.info> — 2016-11-03T14:01:18Z

    Hi,

2016-11-03 8:22 GMT+01:00 amul sul <sulamul@gmail.com>:

> Hi Guillaume,
>
> With your v2 patch, -B options working as expected but --no-blobs
> options is still unrecognized, this happens is because of you have
> forgot to add entry for 'no-blobs' in long_options[] array.
>
>
You're right. v3 (attached) fixes this.


> Apart from this concern patch looks good to me. Thanks
>
>
Thanks.


> Regards,
> Amul
>
> The new status of this patch is: Waiting on Author
>
> On Mon, Oct 24, 2016 at 12:19 AM, Guillaume Lelarge
> <guillaume@lelarge.info> wrote:
> > 2016-10-23 20:44 GMT+02:00 Guillaume Lelarge <guillaume@lelarge.info>:
> >>
> >> 2016-10-23 20:37 GMT+02:00 Andreas Joseph Krogh <andreas@visena.com>:
> >>>
> >>> På søndag 23. oktober 2016 kl. 19:15:17, skrev Andreas Joseph Krogh
> >>> <andreas@visena.com>:
> >>>
> >>> På søndag 23. oktober 2016 kl. 17:06:57, skrev Guillaume Lelarge
> >>> <guillaume@lelarge.info>:
> >>>
> >>> 2016-03-08 21:06 GMT+01:00 Andreas Joseph Krogh <andreas@visena.com>:
> >>>>
> >>>> På tirsdag 08. mars 2016 kl. 21:03:01, skrev David G. Johnston
> >>>> <david.g.johnston@gmail.com>:
> >>>>
> >>>> On Tue, Mar 8, 2016 at 9:45 AM, Andreas Joseph Krogh
> >>>> <andreas@visena.com> wrote:
> >>>>>
> >>>>> På tirsdag 08. mars 2016 kl. 17:38:04, skrev Joshua D. Drake
> >>>>> <jd@commandprompt.com>:
> >>>>>
> >>>>> On 03/08/2016 08:02 AM, Andreas Joseph Krogh wrote:
> >>>>> > På tirsdag 08. mars 2016 kl. 16:57:01, skrev Tom Lane
> >>>>> > <tgl@sss.pgh.pa.us
> >>>>> > <mailto:tgl@sss.pgh.pa.us>>:
> >>>>> >
> >>>>> >     Andreas Joseph Krogh <andreas@visena.com> writes:
> >>>>> >      > What I'm looking for is "inverse -b" in an otherwise
> complete
> >>>>> >     dump. Any plans
> >>>>> >      > to add that?
> >>>>> >
> >>>>> >     [ shrug... ]  Nobody ever asked for it before.
> >>>>> >
> >>>>> >     regards, tom lane
> >>>>> >
> >>>>> > It surely helps testing production-datasets which contain lots of
> >>>>> > BLOBs
> >>>>> > where one wants to dump the production-data into a test-env. We
> have
> >>>>> >  >1TB databases containing > 95% blobs so it would help us
> >>>>> > tremendously
> >>>>> > to have this option.
> >>>>>
> >>>>> I have quite a few customers that would benefit from the ability to
> not
> >>>>> have blobs present in dumps.
> >>>>>
> >>>>>
> >>>>> Great! So how do we proceed to get "--no-blobs" added to pg_dump?
> >>>>> Maybe CommandPrompt and Visena should co-fund development of such an
> >>>>> addition, if it's accepted by -hackers?
> >>>>> We'd be willing to pay for such an addition for the 9.5 branch, as a
> >>>>> patch.
> >>>>
> >>>>
> >>>> Unfortunately this doesn't qualify as a bug fix - it is a new feature
> >>>> and thus is ineligible for inclusion in official 9.5
> >>>>
> >>>> David J.
> >>>>
> >>>>
> >>>> Of course. That's why I mentioned that, if possible, an unofficial
> patch
> >>>> to 9.5 could be developed, funded partly by Visena (my company).
> Given that
> >>>> someone is willing to do this of course.
> >>>>
> >>>
> >>>
> >>> That probably should look like the patch attached. It applies cleanly
> on
> >>> HEAD, and works AFAICT. If this patch seems interesting enough, I'll
> add it
> >>> to the next commit fest (note for myself, update the ref/pg_dump.sgml
> >>> documentation file).
> >>>
> >>> For Andreas' information, it also applies on 9.5, though I didn't check
> >>> if it worked afterwards.
> >>>
> >>>
> >>> +1 for adding it to the commitfest.
> >>>
> >>
> >>
> >> Done, https://commitfest.postgresql.org/11/833/
> >>
> >>> It's almost scary how simple this patch is and noone ever got around to
> >>> implement it.
> >>
> >>
> >> Nobody had the time (like me, till now) or the motivation.
> >>
> >>>
> >>>
> >>> Thanks, I'll test it on 9.5 soon.
> >>>
> >>>
> >>> It's totally OK for me to use 9.6 (now that it's released) to dump 9.5
> >>> DBs, so I'm all good with this patch, thanks!
> >>
> >>
> >> Remember that, if it gets commited, it'll be for next release (aka 10),
> >> and not 9.6 and earlier.
> >>
> >
> > New patch, this time with the documentation.
> >
> >
>



-- 
Guillaume.
  http://blog.guillaume.lelarge.info
  http://www.dalibo.com

  25. Re: Exclude pg_largeobject form pg_dump

    Amul Sul <sulamul@gmail.com> — 2016-11-04T06:52:15Z

    Hi Guillaume,

I found following issues with this patch, sorry missed in previous post:

#1 :
 43 @@ -392,6 +393,10 @@ main(int argc, char **argv)
 44                 dopt.outputBlobs = true;
 45                 break;
 46
 47 +           case 'B':           /* Don't dump blobs */
 48 +               dopt.include_everything = false;
 49 +               break;
 50 +

Touching dopt.include_everything flag does not seems to be a good idea
for '--no-blobs' option, our intension is to exclude blob only, but
this excluds other dump too (e.g COMMENT ON DATABASE, CREATE
EXTENSION, COMMENT ON EXTENSION, .., etc)  that what we don't want,
right?

#2 :
We should add note for default behaviour if --no-blobs & --blobs both
are specified.

Regards,
Amul Sul

  26. Re: Exclude pg_largeobject form pg_dump

    Guillaume Lelarge <guillaume@lelarge.info> — 2016-11-04T08:35:49Z

    Hi Amul,

2016-11-04 7:52 GMT+01:00 amul sul <sulamul@gmail.com>:

> Hi Guillaume,
>
> I found following issues with this patch, sorry missed in previous post:
>
>
You don't have to be sorry for me doing shitty things :)


> #1 :
>  43 @@ -392,6 +393,10 @@ main(int argc, char **argv)
>  44                 dopt.outputBlobs = true;
>  45                 break;
>  46
>  47 +           case 'B':           /* Don't dump blobs */
>  48 +               dopt.include_everything = false;
>  49 +               break;
>  50 +
>
> Touching dopt.include_everything flag does not seems to be a good idea
> for '--no-blobs' option, our intension is to exclude blob only, but
> this excluds other dump too (e.g COMMENT ON DATABASE, CREATE
> EXTENSION, COMMENT ON EXTENSION, .., etc)  that what we don't want,
> right?
>
>
Agreed. I was afraid of that, but for some reason, didn't find that. I'll
fix this.


> #2 :
> We should add note for default behaviour if --no-blobs & --blobs both
> are specified.
>
>
Right. I don't know how I will handle this, but you're right that the
behaviour should be specified. I'll also fix this.

I'll try to work on this today but as I'm in pgconf.eu 2016, it may be only
for tomorrow.

Thank you.


-- 
Guillaume.
  http://blog.guillaume.lelarge.info
  http://www.dalibo.com

  27. Re: Exclude pg_largeobject form pg_dump

    Guillaume Lelarge <guillaume@lelarge.info> — 2016-11-06T20:33:20Z

    2016-11-04 9:35 GMT+01:00 Guillaume Lelarge <guillaume@lelarge.info>:

> Hi Amul,
>
> 2016-11-04 7:52 GMT+01:00 amul sul <sulamul@gmail.com>:
>
>> Hi Guillaume,
>>
>> I found following issues with this patch, sorry missed in previous post:
>>
>>
> You don't have to be sorry for me doing shitty things :)
>
>
>> #1 :
>>  43 @@ -392,6 +393,10 @@ main(int argc, char **argv)
>>  44                 dopt.outputBlobs = true;
>>  45                 break;
>>  46
>>  47 +           case 'B':           /* Don't dump blobs */
>>  48 +               dopt.include_everything = false;
>>  49 +               break;
>>  50 +
>>
>> Touching dopt.include_everything flag does not seems to be a good idea
>> for '--no-blobs' option, our intension is to exclude blob only, but
>> this excluds other dump too (e.g COMMENT ON DATABASE, CREATE
>> EXTENSION, COMMENT ON EXTENSION, .., etc)  that what we don't want,
>> right?
>>
>>
> Agreed. I was afraid of that, but for some reason, didn't find that. I'll
> fix this.
>

Fixed in v4.


>
>
>> #2 :
>> We should add note for default behaviour if --no-blobs & --blobs both
>> are specified.
>>
>>
> Right. I don't know how I will handle this, but you're right that the
> behaviour should be specified. I'll also fix this.
>
>
I checked other options, such as --format, and there's nothing noted as a
default behaviour. Last one wins, which is what this patch does.


> I'll try to work on this today but as I'm in pgconf.eu 2016, it may be
> only for tomorrow.
>
>
It tooks me more time than expected. v4 attached.

>

-- 
Guillaume.
  http://blog.guillaume.lelarge.info
  http://www.dalibo.com

  28. Re: Exclude pg_largeobject form pg_dump

    Amul Sul <sulamul@gmail.com> — 2016-11-07T06:06:24Z

    On Mon, Nov 7, 2016 at 2:03 AM, Guillaume Lelarge
<guillaume@lelarge.info> wrote:
>>
>> Agreed. I was afraid of that, but for some reason, didn't find that. I'll
>> fix this.
>
>
> Fixed in v4.
>

This fix is broken.

 70  -   if (dopt.include_everything && !dopt.schemaOnly)
 71 +   if (dopt.include_everything && !dopt.schemaOnly && !dopt.outputBlobs)
 72         dopt.outputBlobs = true;

dopt.outputBlobs set to FALSE when option -B is specified and this IF
condition reverts to TRUE which force to dump blobs.

>>
>>
>>>
>>> #2 :
>>> We should add note for default behaviour if --no-blobs & --blobs both
>>> are specified.
>>>
>>
>> Right. I don't know how I will handle this, but you're right that the
>> behaviour should be specified. I'll also fix this.
>>
>
> I checked other options, such as --format, and there's nothing noted as a
> default behaviour. Last one wins, which is what this patch does.
>

Such note exists for --table & --exclude-table option, see following
lines in pg_dump.sgml

 569        <para>
 570         When both <option>-t</> and <option>-T</> are given, the behavior
 571         is to dump just the tables that match at least one <option>-t</>
 572         switch but no <option>-T</> switches.  If <option>-T</> appears
 573         without <option>-t</>, then tables matching <option>-T</> are
 574         excluded from what is otherwise a normal dump.
 575        </para>
 576       </listitem>

Regards,
Amul Sul

  29. Re: Exclude pg_largeobject form pg_dump

    Guillaume Lelarge <guillaume@lelarge.info> — 2016-11-07T21:01:41Z

    2016-11-07 7:06 GMT+01:00 amul sul <sulamul@gmail.com>:

> On Mon, Nov 7, 2016 at 2:03 AM, Guillaume Lelarge
> <guillaume@lelarge.info> wrote:
> >>
> >> Agreed. I was afraid of that, but for some reason, didn't find that.
> I'll
> >> fix this.
> >
> >
> > Fixed in v4.
> >
>
> This fix is broken.
>
>  70  -   if (dopt.include_everything && !dopt.schemaOnly)
>  71 +   if (dopt.include_everything && !dopt.schemaOnly &&
> !dopt.outputBlobs)
>  72         dopt.outputBlobs = true;
>
> dopt.outputBlobs set to FALSE when option -B is specified and this IF
> condition reverts to TRUE which force to dump blobs.
>
>
I don't see what you mean. It forces dump of Blobs if we didn't use -B and
if we include everything in the dump, which seems good to me. What did you
try that didn't work as expected?

>>
> >>
> >>>
> >>> #2 :
> >>> We should add note for default behaviour if --no-blobs & --blobs both
> >>> are specified.
> >>>
> >>
> >> Right. I don't know how I will handle this, but you're right that the
> >> behaviour should be specified. I'll also fix this.
> >>
> >
> > I checked other options, such as --format, and there's nothing noted as a
> > default behaviour. Last one wins, which is what this patch does.
> >
>
> Such note exists for --table & --exclude-table option, see following
> lines in pg_dump.sgml
>
>  569        <para>
>  570         When both <option>-t</> and <option>-T</> are given, the
> behavior
>  571         is to dump just the tables that match at least one
> <option>-t</>
>  572         switch but no <option>-T</> switches.  If <option>-T</>
> appears
>  573         without <option>-t</>, then tables matching <option>-T</> are
>  574         excluded from what is otherwise a normal dump.
>  575        </para>
>  576       </listitem>
>
>
You're right on this. v5 fixes this.


-- 
Guillaume.
  http://blog.guillaume.lelarge.info
  http://www.dalibo.com

  30. Re: Exclude pg_largeobject form pg_dump

    Andreas Joseph Krogh <andreas@visena.com> — 2016-11-08T00:06:47Z

    På mandag 07. november 2016 kl. 22:01:41, skrev Guillaume Lelarge <
guillaume@lelarge.info <mailto:guillaume@lelarge.info>>:
2016-11-07 7:06 GMT+01:00 amul sul <sulamul@gmail.com 
<mailto:sulamul@gmail.com>>: On Mon, Nov 7, 2016 at 2:03 AM, Guillaume Lelarge
 <guillaume@lelarge.info <mailto:guillaume@lelarge.info>> wrote:
 >>
 >> Agreed. I was afraid of that, but for some reason, didn't find that. I'll
 >> fix this.
 >
 >
 > Fixed in v4.
 >

 This fix is broken.

  70  -   if (dopt.include_everything && !dopt.schemaOnly)
  71 +   if (dopt.include_everything && !dopt.schemaOnly && !dopt.outputBlobs)
  72         dopt.outputBlobs = true;

 dopt.outputBlobs set to FALSE when option -B is specified and this IF
 condition reverts to TRUE which force to dump blobs.
    
I don't see what you mean. It forces dump of Blobs if we didn't use -B and if 
we include everything in the dump, which seems good to me. What did you try 
that didn't work as expected?



 
I guess what he means is that if -B is given, the following code 
sets dopt.outputBlobs = false
 
+            case 'B':            /* Don't dump blobs */
 +                dopt.outputBlobs = false;
 +                break;
  
 
Then this IF sets it back to TRUE:
 
+    if (dopt.include_everything && !dopt.schemaOnly && !dopt.outputBlobs)
          dopt.outputBlobs = true;
 
 
...making it impossible to turn off dumping of blobs.
 
-- Andreas Joseph Krogh
CTO / Partner - Visena AS
Mobile: +47 909 56 963
andreas@visena.com <mailto:andreas@visena.com>
www.visena.com <https://www.visena.com>

  31. Re: Exclude pg_largeobject form pg_dump

    Amul Sul <sulamul@gmail.com> — 2016-11-08T05:01:35Z

    On Tue, Nov 8, 2016 at 5:36 AM, Andreas Joseph Krogh <andreas@visena.com> wrote:
>
>
> I don't see what you mean. It forces dump of Blobs if we didn't use -B and
> if we include everything in the dump, which seems good to me. What did you
> try that didn't work as expected?
>
>
> I guess what he means is that if -B is given, the following code sets
> dopt.outputBlobs = false
>
> +            case 'B':            /* Don't dump blobs */
> +                dopt.outputBlobs = false;
> +                break;
>
>
> Then this IF sets it back to TRUE:
>
> +    if (dopt.include_everything && !dopt.schemaOnly && !dopt.outputBlobs)
>          dopt.outputBlobs = true;
>
>
> ...making it impossible to turn off dumping of blobs.
>

Yes, thats the reason v4 patch  was not as expected.

Regards,
Amul Sul

  32. Re: Exclude pg_largeobject form pg_dump

    Guillaume Lelarge <guillaume@lelarge.info> — 2016-11-13T21:23:41Z

    2016-11-08 6:01 GMT+01:00 amul sul <sulamul@gmail.com>:

> On Tue, Nov 8, 2016 at 5:36 AM, Andreas Joseph Krogh <andreas@visena.com>
> wrote:
> >
> >
> > I don't see what you mean. It forces dump of Blobs if we didn't use -B
> and
> > if we include everything in the dump, which seems good to me. What did
> you
> > try that didn't work as expected?
> >
> >
> > I guess what he means is that if -B is given, the following code sets
> > dopt.outputBlobs = false
> >
> > +            case 'B':            /* Don't dump blobs */
> > +                dopt.outputBlobs = false;
> > +                break;
> >
> >
> > Then this IF sets it back to TRUE:
> >
> > +    if (dopt.include_everything && !dopt.schemaOnly &&
> !dopt.outputBlobs)
> >          dopt.outputBlobs = true;
> >
> >
> > ...making it impossible to turn off dumping of blobs.
> >
>
> Yes, thats the reason v4 patch  was not as expected.
>
>
It took me some time but I finally understand.

Behaviour fix in v6.


-- 
Guillaume.
  http://blog.guillaume.lelarge.info
  http://www.dalibo.com

  33. Re: Exclude pg_largeobject form pg_dump

    Amul Sul <sulamul@gmail.com> — 2016-11-17T06:15:41Z

    The following review has been posted through the commitfest application:
make installcheck-world:  tested, passed
Implements feature:       tested, passed
Spec compliant:           not tested
Documentation:            tested, passed

Patch v6 looks good to me, passing to committer.

Thanks !

The new status of this patch is: Ready for Committer

  34. Re: Exclude pg_largeobject form pg_dump

    Stephen Frost <sfrost@snowman.net> — 2016-11-29T16:12:25Z

    Guillaume, Amul,

* Amul Sul (sulamul@gmail.com) wrote:
> The following review has been posted through the commitfest application:
> make installcheck-world:  tested, passed
> Implements feature:       tested, passed
> Spec compliant:           not tested
> Documentation:            tested, passed
> 
> Patch v6 looks good to me, passing to committer.
> 
> Thanks !
> 
> The new status of this patch is: Ready for Committer

I've pushed this patch.  In the future if you are working with the
pg_dump TAP tests and add a new 'run', be sure to update the 'tests' to
indicate which tests should 'like' or 'unlike' that run.  If you have
any questions, feel free to ask.

Thanks!

Stephen

  35. Re: Exclude pg_largeobject form pg_dump

    Guillaume Lelarge <guillaume@lelarge.info> — 2016-11-29T21:31:49Z

    2016-11-29 17:12 GMT+01:00 Stephen Frost <sfrost@snowman.net>:

> Guillaume, Amul,
>
> * Amul Sul (sulamul@gmail.com) wrote:
> > The following review has been posted through the commitfest application:
> > make installcheck-world:  tested, passed
> > Implements feature:       tested, passed
> > Spec compliant:           not tested
> > Documentation:            tested, passed
> >
> > Patch v6 looks good to me, passing to committer.
> >
> > Thanks !
> >
> > The new status of this patch is: Ready for Committer
>
> I've pushed this patch.  In the future if you are working with the
> pg_dump TAP tests and add a new 'run', be sure to update the 'tests' to
> indicate which tests should 'like' or 'unlike' that run.  If you have
> any questions, feel free to ask.
>
>
Thank you. And thanks Amul for the review.


-- 
Guillaume.
  http://blog.guillaume.lelarge.info
  http://www.dalibo.com