Thread

serverless postgresql

Jeff Bowden <jlb@houseofdistraction.com> — 2004-01-13T21:38:02Z

For ease of configuration and other reasons, I would like for my 
single-user GUI app to be able to use postgresql in-process as a library 
accessing a database created in the users home directory.  I think I 
could possibly get what I want by launching a captive copy of postmaster 
with appropriate args but it seems conceptually cleaner to not have a 
seperate process at all.  Has anyone tried to do anything like this?

I've looked at sqlite and it might be workable for my project but I 
would prefer the more standard and featureful SQL of postgres.  In 
particular sqlite lacks date arithmetic and has some funny type issues 
(it claims to be typeless, except when it isn't, or something).

Re: serverless postgresql

Enrico Weigelt <weigelt@metux.de> — 2004-01-13T22:17:13Z

* Jeff Bowden <jlb@houseofdistraction.com> [2004-01-13 13:38:02 -0800]:

> For ease of configuration and other reasons, I would like for my 
> single-user GUI app to be able to use postgresql in-process as a library 
> accessing a database created in the users home directory.  I think I 
> could possibly get what I want by launching a captive copy of postmaster 
> with appropriate args but it seems conceptually cleaner to not have a 
> seperate process at all.  Has anyone tried to do anything like this?

hmm, i've also played around a little bit w/ postgresql standalone.
you could also do nice things like calling postmaster from inetd ;-)

but I dont think you really wanna have the postmaster in the same 
process w/ your application, since it relies on forks, signals,
mmap() and some other things which directly affect the process control - 
this probably infers with your application. On the application side 
you cant really control, what it does, so it sounds quite difficult.
Having the postmaster in a separate process connected by a socket or 
pty seems to be the better solution. 


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT services

  phone:     +49 36207 519931         www:       http://www.metux.de/
  fax:       +49 36207 519932         email:     contact@metux.de
  cellphone: +49 174 7066481
---------------------------------------------------------------------
 Diese Mail wurde mit UUCP versandt.      http://www.metux.de/uucp/

Re: serverless postgresql

Rick Gigger <rick@alpinenetworking.com> — 2004-01-13T22:42:29Z

I have just about the same sort of needs now and concluded that postgres
just is not suited for embedding into apps like that.  I am going with
sqlite and it is working fairly well.  We just made user defined functions
in php for sqlite to match the date functions in postgres.  Who knows what
other issues we may run into but so far sqlite is working fairly well.

I think that sqlite stores everything as a string but for searching,
sorting, ordering etc it uses the type info.

I to would absolutely love to just include a postgres dll and have postgres
in process and just store everything in a nice little file but from what I
have read hear that would involve major changes that the developers on not
interested in making.  So far sqlite has done the job for me.

Also I am considering looking into firebird for an embedded database
solution since it can you pretty much what you are talking about here with
an in proccess db that uses a single file to store it's data.  While I don't
want to switch my server stuff to it, it is probably more full featured than
sqlite (I'm sure it has date and arithmatic functions) and is made to work
well in embedded situations.

I would be intersted to see what you end up doing.

Rick

----- Original Message ----- 
From: "Jeff Bowden" <jlb@houseofdistraction.com>
To: "pgsql-general" <pgsql-general@postgresql.org>
Sent: Tuesday, January 13, 2004 2:38 PM
Subject: [GENERAL] serverless postgresql


> For ease of configuration and other reasons, I would like for my
> single-user GUI app to be able to use postgresql in-process as a library
> accessing a database created in the users home directory.  I think I
> could possibly get what I want by launching a captive copy of postmaster
> with appropriate args but it seems conceptually cleaner to not have a
> seperate process at all.  Has anyone tried to do anything like this?
>
> I've looked at sqlite and it might be workable for my project but I
> would prefer the more standard and featureful SQL of postgres.  In
> particular sqlite lacks date arithmetic and has some funny type issues
> (it claims to be typeless, except when it isn't, or something).
>
>
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
>

Re: serverless postgresql

Tom Lane <tgl@sss.pgh.pa.us> — 2004-01-13T23:58:59Z

"Rick Gigger" <rick@alpinenetworking.com> writes:
> I to would absolutely love to just include a postgres dll and have postgres
> in process and just store everything in a nice little file but from what I
> have read hear that would involve major changes that the developers on not
> interested in making.

Not only are the developers uninterested in it, the developers actively
oppose it.  We think an embedded database library cannot be reliable
enough to meet our notion of a "database", since it would be subject to
failures anytime the surrounding application has a bug.  Keeping the
client code in a separate process is a far more robust design.

			regards, tom lane

Re: serverless postgresql

Jeff Bowden <jlb@houseofdistraction.com> — 2004-01-14T00:40:19Z

Tom Lane wrote:

>"Rick Gigger" <rick@alpinenetworking.com> writes:
>  
>
>>I to would absolutely love to just include a postgres dll and have postgres
>>in process and just store everything in a nice little file but from what I
>>have read hear that would involve major changes that the developers on not
>>interested in making.
>>    
>>
>
>Not only are the developers uninterested in it, the developers actively
>oppose it.  We think an embedded database library cannot be reliable
>enough to meet our notion of a "database", since it would be subject to
>failures anytime the surrounding application has a bug.  Keeping the
>client code in a separate process is a far more robust design.
>
>  
>

That makes sense to me.  I wonder if sqlite suffers for this problem 
(e.g. app crashing and corrupting the database).

What about the notion of running postmaster on-demand as the user?   Is 
that something that anyone has experience with?  It seems like it would 
solve the complex configuration problems without compromising robustness 
or requiring any special support other than sufficient command-line 
parameters.

Oh yeah, that brings me to another question.  I was looking at the 
postmaster command-line switches and I couldn't find any that would 
allow me to point it at an arbitrary config file but then I had a look 
around and it seems that the .conf files are already used by the various 
startup scripts.  Does postmaster itself ever read the .conf files or is 
it controlled strictly by switches?

Re: serverless postgresql

Dustin Sallings <dustin@spy.net> — 2004-01-14T01:35:56Z

On Jan 13, 2004, at 13:38, Jeff Bowden wrote:

> but it seems conceptually cleaner to not have a seperate process at 
> all.

	That depends a lot on what you mean by cleaner.  A few small programs 
in their own process space dedicated to a specific set of functionality 
with well-defined interfaces can make for a much more reliable 
application (see postfix).

-- 
Dustin Sallings

Re: serverless postgresql

Tom Lane <tgl@sss.pgh.pa.us> — 2004-01-14T01:37:14Z

Jeff Bowden <jlb@houseofdistraction.com> writes:
> That makes sense to me.  I wonder if sqlite suffers for this problem 
> (e.g. app crashing and corrupting the database).

Likely.  I can tell you that Ann Harrison once told me she made a decent
amount of money as a consultant fixing broken Interbase/Firebird
database files.  It would be hard to make a living in the same game for
Postgres.  Now I don't think that Firebird is any buggier than Postgres.
But it comes in an embedded-library form; I'll bet lunch that most of
those data corruption problems were actually induced by crashes of
surrounding applications.

> What about the notion of running postmaster on-demand as the user?

Possibly.  You'd have to think carefully about what conditions the
postmaster should be shut down under, and especially what conditions
it should NOT be shut down under --- eg, a kill to the parent client
application shouldn't cause an ungraceful postmaster exit.  It could
be tricky to get the signal handling right, especially under shells that
try to deliver signals to all children of a process being signaled.
On the whole I suspect it'd be easier just to leave the postmaster
running in the background...

> Oh yeah, that brings me to another question.  I was looking at the 
> postmaster command-line switches and I couldn't find any that would 
> allow me to point it at an arbitrary config file

The config files all live in $PGDATA and so are determined by the -D
switch.  There was some talk of changing this, awhile back, but it
foundered on lack of consensus about exactly what to do instead.

			regards, tom lane

Re: serverless postgresql

Chris Browne <cbbrowne@acm.org> — 2004-01-14T05:12:45Z

jlb@houseofdistraction.com (Jeff Bowden) wrote:
> Tom Lane wrote:
>>Not only are the developers uninterested in it, the developers actively
>>oppose it.  We think an embedded database library cannot be reliable
>>enough to meet our notion of a "database", since it would be subject to
>>failures anytime the surrounding application has a bug.  Keeping the
>>client code in a separate process is a far more robust design.

> That makes sense to me.  I wonder if sqlite suffers for this problem
> (e.g. app crashing and corrupting the database).

I'm not sure its use has yet been so widespread that there is a good
feel for this.  It is doubtless _possible_; "crash patterns" likely
_don't_ lead to enormous disasters when databases are small,
localized, and the apps probably _don't_ crash all the time.

> What about the notion of running postmaster on-demand as the user?
> Is that something that anyone has experience with?  It seems like it
> would solve the complex configuration problems without compromising
> robustness or requiring any special support other than sufficient
> command-line parameters.

I haven't had call for this being controlled "by the application," but
I can't see it being vastly troublesome.  And I _don't_ see it
requiring lots of "command line" parameters; you just need to specify
the directory where the configuration is.

> Oh yeah, that brings me to another question.  I was looking at the
> postmaster command-line switches and I couldn't find any that would
> allow me to point it at an arbitrary config file but then I had a
> look around and it seems that the .conf files are already used by
> the various startup scripts.  Does postmaster itself ever read the
> .conf files or is it controlled strictly by switches?

All of the .conf files are in one directory, and that directory is
controlled by either the value of environment variable PGDATA or the
"-D" command parameter.

It seems not-overly-valuable to have the .conf files be able to be
specified in random other locations.

Is there some particular reason you have in mind why you would want to
_ignore_ the configuration in $HOME/DBDIR and instead use
configuration in some other random location?  I would think that
collecting the config into one directory, as is done right now, is a
_good_ thing.
-- 
let name="cbbrowne" and tld="ntlug.org" in name ^ "@" ^ tld;;
http://www3.sympatico.ca/cbbrowne/lsf.html
"Even in the  area of anticompetitive conduct, Microsoft  is mainly an
imitator." -- Ralph Nader (1998/11/11)

Re: serverless postgresql

Jeff Bowden <jlb@houseofdistraction.com> — 2004-01-14T05:13:00Z

Tom Lane wrote:

>Jeff Bowden <jlb@houseofdistraction.com> writes:
>  
>
>>What about the notion of running postmaster on-demand as the user?
>>    
>>
>
>Possibly.  You'd have to think carefully about what conditions the
>postmaster should be shut down under, and especially what conditions
>it should NOT be shut down under --- eg, a kill to the parent client
>application shouldn't cause an ungraceful postmaster exit.  It could
>be tricky to get the signal handling right, especially under shells that
>try to deliver signals to all children of a process being signaled.
>On the whole I suspect it'd be easier just to leave the postmaster
>running in the background...
>  
>

Details, details.... :-)

>>Oh yeah, that brings me to another question.  I was looking at the 
>>postmaster command-line switches and I couldn't find any that would 
>>allow me to point it at an arbitrary config file
>>    
>>
>
>The config files all live in $PGDATA and so are determined by the -D
>switch.  There was some talk of changing this, awhile back, but it
>foundered on lack of consensus about exactly what to do instead.
>  
>

As long as it can be done.

Re: serverless postgresql

Chris Browne <cbbrowne@acm.org> — 2004-01-14T05:18:10Z

Quoth jlb@houseofdistraction.com (Jeff Bowden):
> For ease of configuration and other reasons, I would like for my
> single-user GUI app to be able to use postgresql in-process as a
> library accessing a database created in the users home directory.  I
> think I could possibly get what I want by launching a captive copy of
> postmaster with appropriate args but it seems conceptually cleaner to
> not have a seperate process at all.  Has anyone tried to do anything
> like this?

One of the properties of Unix is that spawning extra processes to "do
their thing" allows each application to be designed more 'cleanly.'

Curiously enough, the implementors of alternatives to the Sendmail MTA
have typically concluded that it was not only conceptually cleaner to
have a set of separate cooperating processes, but that they would also
reap benefits in terms of speed and improved security.

Thus, I think you'll find that people disagree with you on this.  It
seems to me that it is conceptually cleaner to have the "application"
do "application stuff," and not muddle things up by mixing that with
the "database stuff."
-- 
If this was helpful, <http://svcs.affero.net/rm.php?r=cbbrowne> rate me
http://cbbrowne.com/info/sgml.html
Rules of the  Evil Overlord #87. "My vats  of hazardous chemicals will
be covered when not in use.  Also, I will not construct walkways above
them." <http://www.eviloverlord.com/>

planner and join type mismatches

Jim Wilson <jimw@kelcomaine.com> — 2004-01-14T14:09:14Z

Tom Lane <tgl@sss.pgh.pa.us> said:

> ---------------------------(end of broadcast)---------------------------
> TIP 9: the planner will ignore your desire to choose an index scan if your
>       joining column's datatypes do not match
> 

Re: tip 9 :-).  In 7.3.5 is it possible that the planner is finding a type
mismatch when there isn't one, resulting in a seq scan being selected?  I've
had a problem with joining tables on an int8 column (primary key column in one
of the tables).  Tested left outer and where clause syntax.

TIA,

Jim Wilson

Re: planner and join type mismatches

Tom Lane <tgl@sss.pgh.pa.us> — 2004-01-14T15:07:42Z

"Jim Wilson" <jimw@kelcomaine.com> writes:
> Re: tip 9 :-).  In 7.3.5 is it possible that the planner is finding a type
> mismatch when there isn't one, resulting in a seq scan being selected?

When you gave no details, how is anyone to know?  For questions like
this, you need to post the table definitions, the exact query, and the
results of EXPLAIN ANALYZE on that query.  BTW, pgsql-performance is a
more appropriate list than -general for such questions.

			regards, tom lane

Re: serverless postgresql

Rick Gigger <rick@alpinenetworking.com> — 2004-01-14T18:03:01Z

> Jeff Bowden <jlb@houseofdistraction.com> writes:
> > That makes sense to me.  I wonder if sqlite suffers for this problem
> > (e.g. app crashing and corrupting the database).
>
> Likely.  I can tell you that Ann Harrison once told me she made a decent
> amount of money as a consultant fixing broken Interbase/Firebird
> database files.  It would be hard to make a living in the same game for
> Postgres.  Now I don't think that Firebird is any buggier than Postgres.
> But it comes in an embedded-library form; I'll bet lunch that most of
> those data corruption problems were actually induced by crashes of
> surrounding applications.

I remember reading about how some the original commerical code from
Interbase came with some bugs that corrupted databases and that those bugs
were eventually fixed in Firebird.  Don't know if that's related to this but
it could be.

Anyway since postgres uses WAL files to verify the integrity of the database
couldn't it more or less make the same guarantee's in an embedded version?
As long as the app uses the db libs unmodified and doesn't mess with the
files it creates how does simply making it embedded increase the change of
db errors resulting in database corruption?

In addition there are times when you want the ease of a relational database
but you don't need say the same level of reliability that you would on your
server.  In these cases it would be nice to just have something that is easy
to drop into your app and have it just work.  That being said I think that
in these cases the developer isn't usually going to care if it runs in
process or not.  Maybe a good solution would be to make a postgres library
that could be included in process that would interface with the application.
All it would do is manager spawning the real postgres process.  Then you
would have the reliability of a real server but the ease of use of an
embedded database.  Couldn't that end up being pretty transparent to the
developer?  If it was made to be easy postgres could become very popular as
an "embedded" db solution but still not have it's reputation diluted by
rampant database corruption from the application.

Re: serverless postgresql

Douglas McNaught <doug@mcnaught.org> — 2004-01-14T18:12:09Z

"Rick Gigger" <rick@alpinenetworking.com> writes:

> Anyway since postgres uses WAL files to verify the integrity of the database
> couldn't it more or less make the same guarantee's in an embedded version?
> As long as the app uses the db libs unmodified and doesn't mess with the
> files it creates how does simply making it embedded increase the change of
> db errors resulting in database corruption?

If an app's internal data structures get corrupted and it goes nuts,
pretty much anything can happen to its address space.

-Doug

Re: serverless postgresql

Jonathan Bartlett <johnnyb@eskimo.com> — 2004-01-14T21:07:10Z

> Anyway since postgres uses WAL files to verify the integrity of the database
> couldn't it more or less make the same guarantee's in an embedded version?
> As long as the app uses the db libs unmodified and doesn't mess with the
> files it creates how does simply making it embedded increase the change of
> db errors resulting in database corruption?

I have a different idea.  I've been thinking about coding it, but haven't
had the time.  This could be done with no changes to Postgres itself.

Basically, you would have a library which exported functions such as

pg_instance *pg_start(char *directory);
pg_get_connection(pg_instance *);
pg_stop(pg_instance *);
pg_initdb(char *directory)

pg_start would do the following:
  1) check "directory" for an instance of the UNIX socket.
       if it is there, make a Postgres connection
       if it is not there, start the Postmaster server with "-k directory
-D directory/data" and then make a Postgres connection
  2) Create a struct to contain the directory and any other data item we
need to connect to the database
  3) Return this structure

pg_get_connection would just be a wrapper for pq_connect()

pg_stop would kill the database.

pg_initdb would simply run initdb

Does anyone see a reason why this wouldn't work?

Jon

Re: serverless postgresql

Rick Gigger <rick@alpinenetworking.com> — 2004-01-14T21:17:18Z

This is just what I recommended (I think) in my earlier post.  This would be
FANTASTIC.  Then I could scrap sqlite and any sqlite custom code that I end
up having to write (there is a little).  Plus I could have the reliability
of a multi-process postgres "embedded".

Now if they just finish the Win32 port sometime soon...

----- Original Message ----- 
From: "Jonathan Bartlett" <johnnyb@eskimo.com>
To: "Rick Gigger" <rick@alpinenetworking.com>
Cc: "Jeff Bowden" <jlb@houseofdistraction.com>; "Tom Lane"
<tgl@sss.pgh.pa.us>; <pgsql-general@postgresql.org>
Sent: Wednesday, January 14, 2004 2:07 PM
Subject: Re: [GENERAL] serverless postgresql


> > Anyway since postgres uses WAL files to verify the integrity of the
database
> > couldn't it more or less make the same guarantee's in an embedded
version?
> > As long as the app uses the db libs unmodified and doesn't mess with the
> > files it creates how does simply making it embedded increase the change
of
> > db errors resulting in database corruption?
>
> I have a different idea.  I've been thinking about coding it, but haven't
> had the time.  This could be done with no changes to Postgres itself.
>
> Basically, you would have a library which exported functions such as
>
> pg_instance *pg_start(char *directory);
> pg_get_connection(pg_instance *);
> pg_stop(pg_instance *);
> pg_initdb(char *directory)
>
> pg_start would do the following:
>   1) check "directory" for an instance of the UNIX socket.
>        if it is there, make a Postgres connection
>        if it is not there, start the Postmaster server with "-k directory
> -D directory/data" and then make a Postgres connection
>   2) Create a struct to contain the directory and any other data item we
> need to connect to the database
>   3) Return this structure
>
> pg_get_connection would just be a wrapper for pq_connect()
>
> pg_stop would kill the database.
>
> pg_initdb would simply run initdb
>
> Does anyone see a reason why this wouldn't work?
>
> Jon
>
>

embedded/"serverless" (Re: serverless postgresql)

David Garamond <lists@zara.6.isreserved.com> — 2004-01-15T06:09:02Z

Jeff Bowden wrote:
> For ease of configuration and other reasons, I would like for my 
> single-user GUI app to be able to use postgresql in-process as a library 
> accessing a database created in the users home directory.  I think I 
> could possibly get what I want by launching a captive copy of postmaster 
> with appropriate args but it seems conceptually cleaner to not have a 
> seperate process at all.  Has anyone tried to do anything like this?

[Sorry for not actually answering this question]

I believe the demands for embedded/"serverless" version of PostgreSQL to 
increase significantly once PostgreSQL is natively available on Windows. 
So I would expect that official embedded support to follow quite shortly 
after win32 port has stabilized. :-)

-- 
dave

Re: serverless postgresql

David Garamond <lists@zara.6.isreserved.com> — 2004-01-15T06:14:32Z

Rick Gigger wrote:
> I have just about the same sort of needs now and concluded that postgres
> just is not suited for embedding into apps like that.  

Why not? It's not that the PostgreSQL backend is a mammoth like Oracle. 
The Firebird embedded version is pretty much the same as their server, 
but without network and client authentication layer.

However, embedded usually demands that the backend be threaded. 
Otherwise it will be pretty useless/very inconvenient to use in many 
apps. Perhaps this is the major change that's hard to do?

-- 
dave

Re: serverless postgresql

David Garamond <lists@zara.6.isreserved.com> — 2004-01-15T06:18:43Z

Tom Lane wrote:
> Jeff Bowden <jlb@houseofdistraction.com> writes:
> 
>>That makes sense to me.  I wonder if sqlite suffers for this problem 
>>(e.g. app crashing and corrupting the database).
> 
> Likely.  I can tell you that Ann Harrison once told me she made a decent
> amount of money as a consultant fixing broken Interbase/Firebird
> database files.  It would be hard to make a living in the same game for
> Postgres.  Now I don't think that Firebird is any buggier than Postgres.
> But it comes in an embedded-library form; I'll bet lunch that most of
> those data corruption problems were actually induced by crashes of
> surrounding applications.

Do the developers generally oppose the idea of a threaded (but 
non-embedded) backend as well? If the backend is thread-safe, then users 
can still choose to run multiprocess or multithreaded right?

-- 
dave

Re: serverless postgresql

Jonathan Bartlett <johnnyb@eskimo.com> — 2004-01-15T15:36:52Z

> Do the developers generally oppose the idea of a threaded (but
> non-embedded) backend as well? If the backend is thread-safe, then users
> can still choose to run multiprocess or multithreaded right?

I've been under the impression that the developers were opposed to a
threaded server because of the complete lack of consistency in threading
behavior across platforms.  However, I don't see how doing it as a
multiprocess server as apposed to a multithreaded server affects embedded
use - as long as it can simply be called from a library, why would a user
or developer care?

Jon

Re: serverless postgresql

Paul Ganainm <paulsnewsgroups@hotmail.com> — 2004-01-15T17:56:43Z


tgl@sss.pgh.pa.us says...


> > That makes sense to me.  I wonder if sqlite suffers for this problem 
> > (e.g. app crashing and corrupting the database).

 
> Likely.  I can tell you that Ann Harrison once told me she made a decent
> amount of money as a consultant fixing broken Interbase/Firebird
> database files.  It would be hard to make a living in the same game for
> Postgres.  Now I don't think that Firebird is any buggier than Postgres.
> But it comes in an embedded-library form; I'll bet lunch that most of
> those data corruption problems were actually induced by crashes of
> surrounding applications.


The reason IMHO for "server" crashes in IB/FB is quite simply due to the 
fact that newbie types who are running IB/FB will tend to run it on the 
crappiest PC in the office under Wintendo 95 on the machine used by the 
secretary who's a bit of a ditz and keeps switching it off by mistake, 
or spilling her coffee on the exposed hard drive.


No forced writes on, no UPS, no RAID, no server room - nada, zip, 
diddly. Basically, it boils down to the fact that just because IB/FB 
*_lets_* you do something, it's not necessarily a good idea. IB/FB will 
run reasonably on the crappiest of hardware, and on the crappiest of 
OS's, which is what leads to the problem.


It's unlikely that PostgreSQL faces this problem, since it will only run 
under far more robust OS's, and you're not likely to have the office 
eejit running the Unix/Linux/BSD box with the db server on it.


Paul...


-- 
plinehan  y_a_h_o_o  and d_o_t  com
C++ Builder 5 SP1, Interbase 6.0.1.6 IBX 5.04 W2K Pro
Please do not top-post.

"XML avoids the fundamental question of what we should do, 
by focusing entirely on how we should do it." 

quote from http://www.metatorial.com

Re: serverless postgresql

Rick Gigger <rick@alpinenetworking.com> — 2004-01-15T18:11:22Z

> Rick Gigger wrote:
> > I have just about the same sort of needs now and concluded that postgres
> > just is not suited for embedding into apps like that.
>
> Why not? It's not that the PostgreSQL backend is a mammoth like Oracle.
> The Firebird embedded version is pretty much the same as their server,
> but without network and client authentication layer.
>
> However, embedded usually demands that the backend be threaded.
> Otherwise it will be pretty useless/very inconvenient to use in many
> apps. Perhaps this is the major change that's hard to do?

I should have said that postgres isn't suited for running in-proccess with
the rest of your app.  Or at least the developers don't think (and I guess I
agree now) that it's a good idea.  They could however make a library that
would start up the server in multiprocess mode and run it in the background.

Re: serverless postgresql

Rick Gigger <rick@alpinenetworking.com> — 2004-01-15T18:14:14Z

> Tom Lane wrote:
> > Jeff Bowden <jlb@houseofdistraction.com> writes:
> >
> >>That makes sense to me.  I wonder if sqlite suffers for this problem
> >>(e.g. app crashing and corrupting the database).
> >
> > Likely.  I can tell you that Ann Harrison once told me she made a decent
> > amount of money as a consultant fixing broken Interbase/Firebird
> > database files.  It would be hard to make a living in the same game for
> > Postgres.  Now I don't think that Firebird is any buggier than Postgres.
> > But it comes in an embedded-library form; I'll bet lunch that most of
> > those data corruption problems were actually induced by crashes of
> > surrounding applications.
>
> Do the developers generally oppose the idea of a threaded (but
> non-embedded) backend as well? If the backend is thread-safe, then users
> can still choose to run multiprocess or multithreaded right?

My impression is that they don't want to go multithreaded.  There was some
talk of this for the windows port but I belive that Bruce is simply
modifying the multi-proccess code so that it will work on windows.

Re: serverless postgresql

Tom Lane <tgl@sss.pgh.pa.us> — 2004-01-15T21:21:19Z

David Garamond <lists@zara.6.isreserved.com> writes:
> Do the developers generally oppose the idea of a threaded (but 
> non-embedded) backend as well? If the backend is thread-safe, then users 
> can still choose to run multiprocess or multithreaded right?

The backend isn't thread-safe.  There have been repeated discussions
about using threading in the backend (see the pgsql-hackers archives),
but so far no one has made a convincing case for it.

BTW, this whole discussion is getting pretty off-topic for -general;
I'd suggest pursuing it on -hackers.

			regards, tom lane

Re: embedded/"serverless" (Re: serverless postgresql)

Chris Travers <chris@travelamericas.com> — 2004-01-16T01:45:30Z

From: "David Garamond" <lists@zara.6.isreserved.com>
> I believe the demands for embedded/"serverless" version of PostgreSQL to
> increase significantly once PostgreSQL is natively available on Windows.
> So I would expect that official embedded support to follow quite shortly
> after win32 port has stabilized. :-)

People are always asking for embedded dbms's  without really considering the
consequences.  For example, if you need to share data, you end up with all
the MS Access sorts of issues.  I know because I used to work at Microsoft
in the department that provided support both for Access and the developer
products.  I suspect the support nightmares may be part of the reason for
pushing MSDE, and hence *get away* from the embedded dbms model ;-)

For example, do you realize that the marketing info said that Access
supported 256 *concurrent* users to a database?  At the same time
troubleshooting corruption was nearly unsupported for a while.

I agree with the approach of a wrapper library which would wrap the
startup/shutdown of a postgresql server so that the programmer doesn't have
to worry about the details, but I would add another idea-- namely that the
library should be able to determine whether the server is running remotely,
and simply pass the connection to libpq.  This would also create a
conceptually cleaner framework for configuration of software which may need
to access a local or remote data store.

Best Wishes,
Chris Travers

Re: embedded/"serverless" (Re: serverless postgresql)

Tom Lane <tgl@sss.pgh.pa.us> — 2004-01-16T04:16:39Z

"Chris Travers" <chris@travelamericas.com> writes:
> I agree with the approach of a wrapper library which would wrap the
> startup/shutdown of a postgresql server so that the programmer doesn't have
> to worry about the details,

The reason that the client programmer doesn't have to worry about
starting/stopping the database is that it's not his responsibility.
I don't think that having the client control this is a good idea at all.
David conveniently ignored the points I made before, but they are
real issues --- if the client is in charge of starting or stopping the
DB, it just adds potential for mucking things up.  I can see the bug
reports now: "I decided I'd make the shutdown routine 'kill -9' the
postmaster because I didn't like the multi-second delay for a normal
shutdown.  Now my database is corrupt."

Another set of objections to this center around the fact that with this
sort of arrangement, the database files would necessarily belong to the
client user, since there's no way to launch the postmaster as a
different userid.  (Unless the client is running as root, which I
sincerely hope he is not.)  That means there's no filesystem protection
between the client and the database, which is another recipe for
trouble.  Not much point in keeping an address-space firewall between
client and server when the client can scribble on the database anyway.

			regards, tom lane

Re: embedded/"serverless" (Re: serverless postgresql)

Rick Gigger <rick@alpinenetworking.com> — 2004-01-16T18:15:35Z

> "Chris Travers" <chris@travelamericas.com> writes:
> > I agree with the approach of a wrapper library which would wrap the
> > startup/shutdown of a postgresql server so that the programmer doesn't
have
> > to worry about the details,
>
> The reason that the client programmer doesn't have to worry about
> starting/stopping the database is that it's not his responsibility.
> I don't think that having the client control this is a good idea at all.
> David conveniently ignored the points I made before, but they are
> real issues --- if the client is in charge of starting or stopping the
> DB, it just adds potential for mucking things up.  I can see the bug
> reports now: "I decided I'd make the shutdown routine 'kill -9' the
> postmaster because I didn't like the multi-second delay for a normal
> shutdown.  Now my database is corrupt."

I recall a discussion a while back where people were complaining that some
os (I think it was mac os x) would just kill all proccesses after something
like 30 seconds on system shutdown if they didn't quit fast enough on their
own.  The response was that because of how postgres uses wal files to
prevent db corruption this was not an issue.  Why does this not apply in
this situation?

> Another set of objections to this center around the fact that with this
> sort of arrangement, the database files would necessarily belong to the
> client user, since there's no way to launch the postmaster as a
> different userid.  (Unless the client is running as root, which I
> sincerely hope he is not.)  That means there's no filesystem protection
> between the client and the database, which is another recipe for
> trouble.  Not much point in keeping an address-space firewall between
> client and server when the client can scribble on the database anyway.

By saying "the client can scribble on the database anyway" do you do you
mean the client app actually writing over the db files on disk?  It seems
like this would only be a problem with an exceptionally stupid programmer.
How could that happen on accident?

Re: embedded/"serverless" (Re: serverless postgresql)

Rick Gigger <rick@alpinenetworking.com> — 2004-01-16T18:33:59Z

> From: "David Garamond" <lists@zara.6.isreserved.com>
> > I believe the demands for embedded/"serverless" version of PostgreSQL to
> > increase significantly once PostgreSQL is natively available on Windows.
> > So I would expect that official embedded support to follow quite shortly
> > after win32 port has stabilized. :-)
>
> People are always asking for embedded dbms's  without really considering
the
> consequences.  For example, if you need to share data, you end up with all
> the MS Access sorts of issues.  I know because I used to work at Microsoft
> in the department that provided support both for Access and the developer
> products.  I suspect the support nightmares may be part of the reason for
> pushing MSDE, and hence *get away* from the embedded dbms model ;-)

My argument of having an embedded or pseudo-embedded postgres is this:

I am never, ever going to trust any really, really important data to an
embedded database.  If the user has it on his system the moron might delete
the database files in which case it really doesn't matter how your
protecting it.  The user can potentially screw everything up.  The reason I
need an embedded database is that sometime my clients need to take a copy of
the app off line and look the info, make reports etc.  They also may
eventually be filling out small amounts of data that can then be synced with
the live database once they are back on line.  If a clients thinkpad lights
on fire while he's on the plane and he loses 6 hours of data entry well
that's not my problem.  But I've ALWAYS got most of the data in a nice
consistent postgres database on my server which is backed up constantly to
multiple locations.

So there are cases where reliability requirements just go down and what I
want is something that acts exactly like postgres so that I don't have to
write custom code for 2 databases.  Since I end up just running sqlite
anyway I would prefer to just have an embedded (or pseudo-embedded)
postgres.  I'm not going to get all up in arms if something bad happens
because I understand the risks of running it embedded.  But those risks will
be the same with any embedded app.  I just don't care.

Does this apply to the vast majority of embedded users or is it just me?

Re: embedded/"serverless" (Re: serverless postgresql)

Tom Lane <tgl@sss.pgh.pa.us> — 2004-01-16T21:14:06Z

"Rick Gigger" <rick@alpinenetworking.com> writes:
>> ... just adds potential for mucking things up.  I can see the bug
>> reports now: "I decided I'd make the shutdown routine 'kill -9' the
>> postmaster because I didn't like the multi-second delay for a normal
>> shutdown.  Now my database is corrupt."

> I recall a discussion a while back where people were complaining that some
> os (I think it was mac os x) would just kill all proccesses after something
> like 30 seconds on system shutdown if they didn't quit fast enough on their
> own.  The response was that because of how postgres uses wal files to
> prevent db corruption this was not an issue.  Why does this not apply in
> this situation?

Because a system shutdown doesn't leave anything else running.  kill -9
on the postmaster leaves orphaned backends still running, and orphaned
shared memory segments still in existence.  The latter could prevent you
from starting a fresh postmaster (because two shmem segments could
exceed the kernel's SHMMAX) until you do manual cleanup, which most
people don't know how to do.  If you do succeed in starting another
postmaster with a fresh shared memory segment, you will have two
independent sets of backends modifying the database files with no
interlocking, which is a sure recipe for data corruption.

There is an interlock that is supposed to prevent this catastrophic
scenario, but I do not trust it unreservedly (and not at all on
non-Unix-derived platforms, because it depends on some rather arcane
features of the SysV shared memory API, which might not be implemented
fully on other platforms).

This is why the "don't kill -9 the postmaster" TIP is still around, BTW.
It has nothing to do with crash safety.

> By saying "the client can scribble on the database anyway" do you do you
> mean the client app actually writing over the db files on disk?  It seems
> like this would only be a problem with an exceptionally stupid programmer.
> How could that happen on accident?

Scribbling on memory that doesn't belong to you isn't something one does
intentionally, either --- but it happens.  However, my real concern here
is not so much with program bugs as clueless users.  We see enough cases
already of people removing lock files or "unnecessary" log files; I fear
it'd be a lot worse if those files were sitting in the user's home
directory.

			regards, tom lane

Re: embedded/"serverless" (Re: serverless postgresql)

Jeff Bowden <jlb@houseofdistraction.com> — 2004-01-16T21:39:12Z

Tom Lane wrote:

>"Chris Travers" <chris@travelamericas.com> writes:
>  
>
>>I agree with the approach of a wrapper library which would wrap the
>>startup/shutdown of a postgresql server so that the programmer doesn't have
>>to worry about the details,
>>    
>>
>
>The reason that the client programmer doesn't have to worry about
>starting/stopping the database is that it's not his responsibility.
>I don't think that having the client control this is a good idea at all.
>David conveniently ignored the points I made before, but they are
>real issues --- if the client is in charge of starting or stopping the
>DB, it just adds potential for mucking things up.  I can see the bug
>reports now: "I decided I'd make the shutdown routine 'kill -9' the
>postmaster because I didn't like the multi-second delay for a normal
>shutdown.  Now my database is corrupt."
>
>Another set of objections to this center around the fact that with this
>sort of arrangement, the database files would necessarily belong to the
>client user, since there's no way to launch the postmaster as a
>different userid.  (Unless the client is running as root, which I
>sincerely hope he is not.)  That means there's no filesystem protection
>between the client and the database, which is another recipe for
>trouble.  Not much point in keeping an address-space firewall between
>client and server when the client can scribble on the database anyway.
>
>  
>

Still, the main problem I, and I suspect others, would like to solve is 
installation/configuration.  For my app I don't want the user to have to 
understand anything about how keeping data in a shared 
system-administered database is different from keeping data in local 
files.  Everything should "just work".  There is no requirement for 
concurrent access.

So kill -9 on postmaster can lead to database corruption?  What happens 
in a power failure?

Re: embedded/"serverless" (Re: serverless postgresql)

Chris Ochs <chris@paymentonline.com> — 2004-01-16T21:50:15Z

Personally, I would use something like berkelydb for an embedded database
engine.  As a general rule, if it needs to be embedded, you don't need sql.
I'm sure you could think of areas where you could use a full blown sql
server, but in practice that generally isn't the case.  And like  others
have already mentioned, if you are embedding a database in software that a
user has control of, you want it as SIMPLE as possible.

Chris



----- Original Message ----- 
From: "Tom Lane" <tgl@sss.pgh.pa.us>
To: "Rick Gigger" <rick@alpinenetworking.com>
Cc: "Chris Travers" <chris@travelamericas.com>; "David Garamond"
<lists@zara.6.isreserved.com>; "Jeff Bowden" <jlb@houseofdistraction.com>;
"pgsql-general" <pgsql-general@postgresql.org>
Sent: Friday, January 16, 2004 1:14 PM
Subject: Re: embedded/"serverless" (Re: [GENERAL] serverless postgresql)


> "Rick Gigger" <rick@alpinenetworking.com> writes:
> >> ... just adds potential for mucking things up.  I can see the bug
> >> reports now: "I decided I'd make the shutdown routine 'kill -9' the
> >> postmaster because I didn't like the multi-second delay for a normal
> >> shutdown.  Now my database is corrupt."
>
> > I recall a discussion a while back where people were complaining that
some
> > os (I think it was mac os x) would just kill all proccesses after
something
> > like 30 seconds on system shutdown if they didn't quit fast enough on
their
> > own.  The response was that because of how postgres uses wal files to
> > prevent db corruption this was not an issue.  Why does this not apply in
> > this situation?
>
> Because a system shutdown doesn't leave anything else running.  kill -9
> on the postmaster leaves orphaned backends still running, and orphaned
> shared memory segments still in existence.  The latter could prevent you
> from starting a fresh postmaster (because two shmem segments could
> exceed the kernel's SHMMAX) until you do manual cleanup, which most
> people don't know how to do.  If you do succeed in starting another
> postmaster with a fresh shared memory segment, you will have two
> independent sets of backends modifying the database files with no
> interlocking, which is a sure recipe for data corruption.
>
> There is an interlock that is supposed to prevent this catastrophic
> scenario, but I do not trust it unreservedly (and not at all on
> non-Unix-derived platforms, because it depends on some rather arcane
> features of the SysV shared memory API, which might not be implemented
> fully on other platforms).
>
> This is why the "don't kill -9 the postmaster" TIP is still around, BTW.
> It has nothing to do with crash safety.
>
> > By saying "the client can scribble on the database anyway" do you do you
> > mean the client app actually writing over the db files on disk?  It
seems
> > like this would only be a problem with an exceptionally stupid
programmer.
> > How could that happen on accident?
>
> Scribbling on memory that doesn't belong to you isn't something one does
> intentionally, either --- but it happens.  However, my real concern here
> is not so much with program bugs as clueless users.  We see enough cases
> already of people removing lock files or "unnecessary" log files; I fear
> it'd be a lot worse if those files were sitting in the user's home
> directory.
>
> regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
>

Re: embedded/"serverless" (Re: serverless postgresql)

Jonathan Bartlett <johnnyb@eskimo.com> — 2004-01-16T22:00:35Z

> Personally, I would use something like berkelydb for an embedded database
> engine.  As a general rule, if it needs to be embedded, you don't need sql.
> I'm sure you could think of areas where you could use a full blown sql
> server, but in practice that generally isn't the case.  And like  others
> have already mentioned, if you are embedding a database in software that a
> user has control of, you want it as SIMPLE as possible.

But the user isn't the one accessing the database directly.  Have you ever
used a program like Delphi to create applications?  Using a relational
database, you can create applications using nothing but drag-and-drop and
SQL.  No real code.  If your application is a simple end-user app (like a
wedding planner, for example),  embedded databases are your dream come
true.  Linux has no real embedded database.  PG would be an excellent
embedded database.  I have already proposed some functions that would help
PG as an embedded database, and, if I hve time, I will implement them
(can't right now because I'm finishing a book).

Anyway, I get the feeling that those people who don't like the idea of
having PG as an embedded database have never used one.  I think Tom, for
example, has been focusing on enterprise-level applications too much to
think about the personal applications, and Chris, I'm going to guess
you've never built an application w/ an embedded DB.

Anyway, this thread is going nowhere, I'll come back if/when I have some
code.  Perhaps with something concrete we can all see better where this is
going.

Jon

>
> Chris
>
>
>
> ----- Original Message -----
> From: "Tom Lane" <tgl@sss.pgh.pa.us>
> To: "Rick Gigger" <rick@alpinenetworking.com>
> Cc: "Chris Travers" <chris@travelamericas.com>; "David Garamond"
> <lists@zara.6.isreserved.com>; "Jeff Bowden" <jlb@houseofdistraction.com>;
> "pgsql-general" <pgsql-general@postgresql.org>
> Sent: Friday, January 16, 2004 1:14 PM
> Subject: Re: embedded/"serverless" (Re: [GENERAL] serverless postgresql)
>
>
> > "Rick Gigger" <rick@alpinenetworking.com> writes:
> > >> ... just adds potential for mucking things up.  I can see the bug
> > >> reports now: "I decided I'd make the shutdown routine 'kill -9' the
> > >> postmaster because I didn't like the multi-second delay for a normal
> > >> shutdown.  Now my database is corrupt."
> >
> > > I recall a discussion a while back where people were complaining that
> some
> > > os (I think it was mac os x) would just kill all proccesses after
> something
> > > like 30 seconds on system shutdown if they didn't quit fast enough on
> their
> > > own.  The response was that because of how postgres uses wal files to
> > > prevent db corruption this was not an issue.  Why does this not apply in
> > > this situation?
> >
> > Because a system shutdown doesn't leave anything else running.  kill -9
> > on the postmaster leaves orphaned backends still running, and orphaned
> > shared memory segments still in existence.  The latter could prevent you
> > from starting a fresh postmaster (because two shmem segments could
> > exceed the kernel's SHMMAX) until you do manual cleanup, which most
> > people don't know how to do.  If you do succeed in starting another
> > postmaster with a fresh shared memory segment, you will have two
> > independent sets of backends modifying the database files with no
> > interlocking, which is a sure recipe for data corruption.
> >
> > There is an interlock that is supposed to prevent this catastrophic
> > scenario, but I do not trust it unreservedly (and not at all on
> > non-Unix-derived platforms, because it depends on some rather arcane
> > features of the SysV shared memory API, which might not be implemented
> > fully on other platforms).
> >
> > This is why the "don't kill -9 the postmaster" TIP is still around, BTW.
> > It has nothing to do with crash safety.
> >
> > > By saying "the client can scribble on the database anyway" do you do you
> > > mean the client app actually writing over the db files on disk?  It
> seems
> > > like this would only be a problem with an exceptionally stupid
> programmer.
> > > How could that happen on accident?
> >
> > Scribbling on memory that doesn't belong to you isn't something one does
> > intentionally, either --- but it happens.  However, my real concern here
> > is not so much with program bugs as clueless users.  We see enough cases
> > already of people removing lock files or "unnecessary" log files; I fear
> > it'd be a lot worse if those files were sitting in the user's home
> > directory.
> >
> > regards, tom lane
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
> >
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster
>

Re: embedded/"serverless" (Re: serverless postgresql)

Tom Lane <tgl@sss.pgh.pa.us> — 2004-01-16T22:01:45Z

Jeff Bowden <jlb@houseofdistraction.com> writes:
> Still, the main problem I, and I suspect others, would like to solve is 
> installation/configuration.  For my app I don't want the user to have to 
> understand anything about how keeping data in a shared 
> system-administered database is different from keeping data in local 
> files.  Everything should "just work".

Sure, but most of the existing packagings of PG already try to make this
automatic (at least Lamar's RPMs and Oliver's Debian package do).  No
doubt further work could be invested to make it even smoother, but that
doesn't mean we need a client-started database.

> So kill -9 on postmaster can lead to database corruption?  What happens 
> in a power failure?

See my last response to Rick Gigger.  kill -9 is a lot different from a
power failure.

			regards, tom lane

Re: embedded/"serverless" (Re: serverless postgresql)

Andrew Sullivan <ajs@crankycanuck.ca> — 2004-01-16T22:15:35Z

On Fri, Jan 16, 2004 at 02:00:35PM -0800, Jonathan Bartlett wrote:

> having PG as an embedded database have never used one.  I think Tom, for
> example, has been focusing on enterprise-level applications too much to
> think about the personal applications, and Chris, I'm going to guess

This seems to suggest that Postgres should focus on something other
than enterprise-level applications, and (presumably) on something
else.

Frankly, I am _mighty glad_ that the developers are focussed on
enterprise-level applications.  That's what makes Postgres eligible
to back enterprise-level software.  Applications that try to do
everything for everyone tend to end up doing nothing very well.  I
don't think one needs to look too hard to discover examples of that
phenomenon.

A

-- 
Andrew Sullivan  | ajs@crankycanuck.ca
This work was visionary and imaginative, and goes to show that visionary
and imaginative work need not end up well. 
		--Dennis Ritchie

Re: embedded/"serverless" (Re: serverless postgresql)

Jeff Bowden <jlb@houseofdistraction.com> — 2004-01-16T22:25:47Z

Tom Lane wrote:

>Jeff Bowden <jlb@houseofdistraction.com> writes:
>  
>
>>Still, the main problem I, and I suspect others, would like to solve is 
>>installation/configuration.  For my app I don't want the user to have to 
>>understand anything about how keeping data in a shared 
>>system-administered database is different from keeping data in local 
>>files.  Everything should "just work".
>>    
>>
>
>Sure, but most of the existing packagings of PG already try to make this
>automatic (at least Lamar's RPMs and Oliver's Debian package do).  No
>doubt further work could be invested to make it even smoother, but that
>doesn't mean we need a client-started database.
>

So maybe this is a packaging issue.  On Debian when I install postgres 
it is necessary to do root shit in order to enable non-priveledged users 
to create and destroy databases.  My understanding has alwasy been that 
these operations are restricted because it can allow users to 
accidentally or intentionally interefere with each other's use of the 
server.  If there is some way for this to be set up sanely by default to 
allow users to successfully use createdb and dropdb on only their own 
data, then I will file a wishlist bug against the Debian packages to add 
this.  Or is this all a pipe dream?

Re: embedded/"serverless" (Re: serverless postgresql)

Jonathan Bartlett <johnnyb@eskimo.com> — 2004-01-16T22:43:44Z

> This seems to suggest that Postgres should focus on something other
> than enterprise-level applications, and (presumably) on something
> else.

Nope.

>
> Frankly, I am _mighty glad_ that the developers are focussed on
> enterprise-level applications.

Me too.  But I think we can get it for small-scale stuff with no impact on
the rest.  Just an additional function or 6 to make embedded developer's
lives easier.

Jon

Re: embedded/"serverless" (Re: serverless postgresql)

Nigel J. Andrews <nandrews@investsystems.co.uk> — 2004-01-16T22:51:05Z

On Fri, 16 Jan 2004, Jeff Bowden wrote:

> Tom Lane wrote:
> 
> >Jeff Bowden <jlb@houseofdistraction.com> writes:
> >  
> >
> >>Still, the main problem I, and I suspect others, would like to solve is 
> >>installation/configuration.  For my app I don't want the user to have to 
> >>understand anything about how keeping data in a shared 
> >>system-administered database is different from keeping data in local 
> >>files.  Everything should "just work".
> >>    
> >>
> >
> >Sure, but most of the existing packagings of PG already try to make this
> >automatic (at least Lamar's RPMs and Oliver's Debian package do).  No
> >doubt further work could be invested to make it even smoother, but that
> >doesn't mean we need a client-started database.
> >
> 
> So maybe this is a packaging issue.  On Debian when I install postgres 
> it is necessary to do root shit in order to enable non-priveledged users 
> to create and destroy databases.  My understanding has alwasy been that 
> these operations are restricted because it can allow users to 
> accidentally or intentionally interefere with each other's use of the 
> server.  If there is some way for this to be set up sanely by default to 
> allow users to successfully use createdb and dropdb on only their own 
> data, then I will file a wishlist bug against the Debian packages to add 
> this.  Or is this all a pipe dream?

I don't see what the problem is. PostgreSQL is perfectly able to be installed
and run as any old user. Even if one is distributing via a package system, such
as RPM perhaps, which has problems installing as non-root user if one sees the
package already installed then there's no need to try to install it
again. Even with the package installations there's bugger all stopping any old
user doing an initdb -D myownplace.

The standard postgres user on a system is only special in that it is generally
the user the postmaster is run as. If you have other non-priviledged users and
there's no need for them to share a cluster why try to make them?

--
Nigel Andrews

Re: embedded/"serverless" (Re: serverless postgresql)

Jeff Bowden <jlb@houseofdistraction.com> — 2004-01-16T23:00:15Z

Nigel J. Andrews wrote:

>On Fri, 16 Jan 2004, Jeff Bowden wrote:
>  
>
>>So maybe this is a packaging issue.  On Debian when I install postgres 
>>it is necessary to do root shit in order to enable non-priveledged users 
>>to create and destroy databases.  My understanding has alwasy been that 
>>these operations are restricted because it can allow users to 
>>accidentally or intentionally interefere with each other's use of the 
>>server.  If there is some way for this to be set up sanely by default to 
>>allow users to successfully use createdb and dropdb on only their own 
>>data, then I will file a wishlist bug against the Debian packages to add 
>>this.  Or is this all a pipe dream?
>>    
>>
>
>I don't see what the problem is. PostgreSQL is perfectly able to be installed
>and run as any old user. Even if one is distributing via a package system, such
>as RPM perhaps, which has problems installing as non-root user if one sees the
>package already installed then there's no need to try to install it
>again. Even with the package installations there's bugger all stopping any old
>user doing an initdb -D myownplace.
>
>The standard postgres user on a system is only special in that it is generally
>the user the postmaster is run as. If you have other non-priviledged users and
>there's no need for them to share a cluster why try to make them?
>  
>

I'm sorry, maybe I didn't state my ideas clearly enough.  I was 
following on to Tom Lane's suggestion that the Debian and RPM packages 
make configuration "automatic".  I was trying to explain that even after 
installation, some administrator configuration is required to make the 
server usable for non-privelidged users.  Either in the form of creating 
and dropping databases or in giving the users rights to do it themselves.

The way I'd like to make it work for my app is to run a seperate 
postmaster as the user to avoid involving the machine administrator or 
interfering with other users.  If a shared install can be made to work 
in an equivalent way, then I would rather go with that.   I'm not clear 
on whether that's possible though.  My current understanding is that it 
is not and that my original notion is the only way for my app.

Re: embedded/"serverless" (Re: serverless postgresql)

Chris Ochs <chris@paymentonline.com> — 2004-01-17T01:17:25Z

I still have to respectfully disagree.  Postgresql is IMO just the wrong
software for the job, and given that there are still a number of really
important things that postgresql lacks, it should concentrate on those.    I
am not against it however for technical reasons, because those things can
always be overcome.  I just wouldn't want postgresql to start branching out
in different directions at this point, it makes no sense if the project
wants to keep focused and one day become comparable side by side to oracle.
IMO that should be it's main goal, and embedded functionality would be a
detour that has more chances of doing harm then good.

Chris






> >
> > Frankly, I am _mighty glad_ that the developers are focussed on
> > enterprise-level applications.
>
> Me too.  But I think we can get it for small-scale stuff with no impact on
> the rest.  Just an additional function or 6 to make embedded developer's
> lives easier.
>
> Jon
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster
>

Re: embedded/"serverless" (Re: serverless postgresql)

David Garamond <lists@zara.6.isreserved.com> — 2004-01-17T05:11:43Z

Chris Ochs wrote:
> I still have to respectfully disagree.  Postgresql is IMO just the wrong
> software for the job, and given that there are still a number of really
> important things that postgresql lacks, it should concentrate on those.    I
> am not against it however for technical reasons, because those things can
> always be overcome.  I just wouldn't want postgresql to start branching out
> in different directions at this point, it makes no sense if the project
> wants to keep focused and one day become comparable side by side to oracle.
> IMO that should be it's main goal, and embedded functionality would be a
> detour that has more chances of doing harm then good.

I believe the basic mission of Postgres will stay the same [for a long 
time]: providing a 1) reliable database; 2) with advanced; 3) and 
standard compliant features.

However, venturing into win32 world _will_ generate demands like 
embedded and other desktop app-oriented features since I expect that's 
what many people will using Postgres for in win32 (currently people are 
using IB/FB for this and not MySQL/Postgre; MySQL's embedded version is 
GPL/commercial). And we all know it's all a matter of what itches the 
developers the most. If enough people are bugging them about something, 
they will do it... :-)

-- 
dave

Re: embedded/"serverless" (Re: serverless postgresql)

Bill Gribble <grib@billgribble.com> — 2004-01-17T15:58:11Z

On Fri, 2004-01-16 at 16:00, Jonathan Bartlett wrote:
> Linux has no real embedded database.  PG would be an excellent
> embedded database.  

SQLite works great for these kinds of applications, FWIW.  

b.g.

Re: embedded/"serverless" (Re: serverless postgresql)

Chris Travers <chris@travelamericas.com> — 2004-01-18T04:52:28Z

Hi Rick;

I have been looking for a few years at a similar situation with my CRM
software and soon (once I have the SOAP interfaces done, I will start on a
stand-alone client.  I have eventually decided on a very different structure
than you have, though the result is more or less the same.

Rather than using PostgreSQL as an embedded database manager, I have decided
that it makes more sense for my app to "cache objects" in XML documents and
then use those on trips to customers' sites.  The XML documents can then be
checked and the objects re-sync'd when the app goes online again.  The
cacheing would be automatic and cached objects would expire unless
explicitly saved.

Why not use XML for this sort of problem?  Or berkeley DB?  Is there any
reason that PostgreSQL would be better?  I am not aware of any functional
replication systems which would work in this way, so I am guessing your app
will have to handle all the logic for the syncronization anyway.

Best Wishes,
Chris Travers

Re: embedded/"serverless" (Re: serverless postgresql)

Chris Travers <chris@travelamericas.com> — 2004-01-18T04:59:10Z

Hi Tom and others;

I agree with your reasoning insofar as it pertains to dedicating core
PostgreSQL time into developing such a library.  I also agree insofar as it
pertains to the PostgreSQL core distribution.

I do think, however, that given the current demand for a serverless solution
to PostgreSQL (whether or not it is the right tool for the job), I would
suggest that those who are interested and have the necessary skills set up a
gborg project.  Perhaps at some point if PostgreSQL distributions become
more commonplace, we will see it used, and this will open up new
oportunities for PostgreSQL.  It should be remembered that more options in
critical areas (such as Win32) may mean more community, and hopefully in the
long run, more developers.

Best Wishes,
Chris Travers

Re: embedded/"serverless" (Re: serverless postgresql)

Nigel J. Andrews <nandrews@investsystems.co.uk> — 2004-01-18T17:29:37Z

On Fri, 16 Jan 2004, Jeff Bowden wrote:

> Nigel J. Andrews wrote:
> >I don't see what the problem is. PostgreSQL is perfectly able to be installed
> >and run as any old user. Even if one is distributing via a package system, such
> >as RPM perhaps, which has problems installing as non-root user if one sees the
> >package already installed then there's no need to try to install it
> >again. Even with the package installations there's bugger all stopping any old
> >user doing an initdb -D myownplace.
> >
> >The standard postgres user on a system is only special in that it is generally
> >the user the postmaster is run as. If you have other non-priviledged users and
> >there's no need for them to share a cluster why try to make them?
> >  
> >
> 
> I'm sorry, maybe I didn't state my ideas clearly enough.  I was 
> following on to Tom Lane's suggestion that the Debian and RPM packages 
> make configuration "automatic".  I was trying to explain that even after 
> installation, some administrator configuration is required to make the 
> server usable for non-privelidged users.  Either in the form of creating 
> and dropping databases or in giving the users rights to do it themselves.
> 
> The way I'd like to make it work for my app is to run a seperate 
> postmaster as the user to avoid involving the machine administrator or 
> interfering with other users.  If a shared install can be made to work 
> in an equivalent way, then I would rather go with that.   I'm not clear 
> on whether that's possible though.  My current understanding is that it 
> is not and that my original notion is the only way for my app.

I think I probably didn't do the explanation clearly enough. This does depend
on what you mean by shared install. I was assuming you just wanted to manage
data independently only using the shared binaries installed from the
package. If that's not the case then obviously this isn't what you want. If
that is the case then you could run the following as any old user, provided the
environment has been set appropiately of course:

initdb -D $HOME/pgdata 	
sed -e 's/^#port = 5432/port = 12345/' $HOME/pgdata/postgresql.conf > /tmp/p
mv /tmp/p $HOME/pgdata/postgresql.conf
pg_ctl -D $HOME/pgdata start

then access the cluster using:

pgsql -P 12345 template1

That's it. That's access as the db superuser for that cluster. Sure if the
system admin upgrades the shared software then you could have problems.

My apologies if I've got the wrong end of things again, I've got a lot of
unread email and I keep reading things in a very haphazard order.


-- 
Nigel J. Andrews

Re: serverless postgresql

Jeff Bowden <jlb@houseofdistraction.com> — 2004-01-19T00:01:32Z

Christopher Browne wrote:

>jlb@houseofdistraction.com (Jeff Bowden) wrote:
>  
>
>>Oh yeah, that brings me to another question.  I was looking at the
>>postmaster command-line switches and I couldn't find any that would
>>allow me to point it at an arbitrary config file but then I had a
>>look around and it seems that the .conf files are already used by
>>the various startup scripts.  Does postmaster itself ever read the
>>.conf files or is it controlled strictly by switches?
>>    
>>
>
>All of the .conf files are in one directory, and that directory is
>controlled by either the value of environment variable PGDATA or the
>"-D" command parameter.
>
>It seems not-overly-valuable to have the .conf files be able to be
>specified in random other locations.
>
>Is there some particular reason you have in mind why you would want to
>_ignore_ the configuration in $HOME/DBDIR and instead use
>configuration in some other random location?  I would think that
>collecting the config into one directory, as is done right now, is a
>_good_ thing.
>  
>

If the existing user configuration could be reliably determined to be 
sane and have sufficient permissions or could be made so, that would be 
fine.

I am writing an app for users who are not necessarily knowledgeable 
about databases.  If they don't have an existing useable postgres 
configuration, I would like for my app to be able to create one without 
forcing the user to have to be root or learn about postgres 
configuration.  The data for this app should remain private to the user 
who created it and there is no need for concurrent access.

Re: embedded/"serverless" (Re: serverless postgresql)

Rick Gigger <rick@alpinenetworking.com> — 2004-01-21T18:23:40Z

> On Fri, Jan 16, 2004 at 02:00:35PM -0800, Jonathan Bartlett wrote:
>
> > having PG as an embedded database have never used one.  I think Tom, for
> > example, has been focusing on enterprise-level applications too much to
> > think about the personal applications, and Chris, I'm going to guess
>
> This seems to suggest that Postgres should focus on something other
> than enterprise-level applications, and (presumably) on something
> else.

Yes but sometimes an enterprise level application may need to be put on a
laptop and taken off-line.  Having an embedded database that is compatible
with the one on the server makes this a bit easier to do.

> Frankly, I am _mighty glad_ that the developers are focussed on
> enterprise-level applications.  That's what makes Postgres eligible
> to back enterprise-level software.  Applications that try to do
> everything for everyone tend to end up doing nothing very well.  I
> don't think one needs to look too hard to discover examples of that
> phenomenon.

Well, this very well may be true.  It does seem as if many of the features
that add the reliability and performance necessary for an enterprise server
application are what makes it difficult to make a good embedded solution.

Re: embedded/"serverless" (Re: serverless postgresql)

Rick Gigger <rick@alpinenetworking.com> — 2004-01-21T18:29:08Z

> Hi Rick;
>
> I have been looking for a few years at a similar situation with my CRM
> software and soon (once I have the SOAP interfaces done, I will start on a
> stand-alone client.  I have eventually decided on a very different
structure
> than you have, though the result is more or less the same.
>
> Rather than using PostgreSQL as an embedded database manager, I have
decided
> that it makes more sense for my app to "cache objects" in XML documents
and
> then use those on trips to customers' sites.  The XML documents can then
be
> checked and the objects re-sync'd when the app goes online again.  The
> cacheing would be automatic and cached objects would expire unless
> explicitly saved.
>
> Why not use XML for this sort of problem?  Or berkeley DB?  Is there any
> reason that PostgreSQL would be better?  I am not aware of any functional
> replication systems which would work in this way, so I am guessing your
app
> will have to handle all the logic for the syncronization anyway.

Well since the desktop and server app are using almost exactly the same code
it is nice to not have to recode everything that accesses the database.
While data syncronization code will have to be written either way I want the
existing code to "just work" once taken off line.

Re: embedded/"serverless" (Re: serverless postgresql)

Andrew Sullivan <ajs@crankycanuck.ca> — 2004-01-21T22:31:12Z

On Wed, Jan 21, 2004 at 11:23:40AM -0700, Rick Gigger wrote:
> 
> Yes but sometimes an enterprise level application may need to be put on a
> laptop and taken off-line.  Having an embedded database that is compatible
> with the one on the server makes this a bit easier to do.

Why can't you just run a postgres instance for this?  What is magic
about "embedded" for this sort of application?  Sounds like a clever
wrapper script is all that's necessary for something like that, no?

A

-- 
Andrew Sullivan  | ajs@crankycanuck.ca
My friend, you can put wheels on your mama but that doesn't make her a bus.
                --Joel Spolsky

Re: embedded/"serverless" (Re: serverless postgresql)

Nigel J. Andrews <nandrews@investsystems.co.uk> — 2004-01-21T23:52:37Z

On Wed, 21 Jan 2004, Andrew Sullivan wrote:

> On Wed, Jan 21, 2004 at 11:23:40AM -0700, Rick Gigger wrote:
> > 
> > Yes but sometimes an enterprise level application may need to be put on a
> > laptop and taken off-line.  Having an embedded database that is compatible
> > with the one on the server makes this a bit easier to do.
> 
> Why can't you just run a postgres instance for this?  What is magic
> about "embedded" for this sort of application?  Sounds like a clever
> wrapper script is all that's necessary for something like that, no?

That's what I still don't get. Embedded means something running on [and
probably running] your wrist watch _not_ something running on a full blown
system.

I understand in this thread's context that embedded is used to mean embedded
within an application on an ordinary system, took me a while to realise that
though, but just means I can't see why it is wanted like that. Imagine if there
was a DB around that was used by direct library calls from an application. What
would be one of the first things that would be programmed using it? A server
perhaps?

-- 
Nigel Andrews

Re: embedded/"serverless" (Re: serverless postgresql)

Keith C. Perry <netadmin@vcsn.com> — 2004-01-22T15:42:46Z

Quoting Rick Gigger <rick@alpinenetworking.com>:

> > Hi Rick;
> >
> > I have been looking for a few years at a similar situation with my CRM
> > software and soon (once I have the SOAP interfaces done, I will start on a
> > stand-alone client.  I have eventually decided on a very different
> structure
> > than you have, though the result is more or less the same.
> >
> > Rather than using PostgreSQL as an embedded database manager, I have
> decided
> > that it makes more sense for my app to "cache objects" in XML documents
> and
> > then use those on trips to customers' sites.  The XML documents can then
> be
> > checked and the objects re-sync'd when the app goes online again.  The
> > cacheing would be automatic and cached objects would expire unless
> > explicitly saved.
> >
> > Why not use XML for this sort of problem?  Or berkeley DB?  Is there any
> > reason that PostgreSQL would be better?  I am not aware of any functional
> > replication systems which would work in this way, so I am guessing your
> app
> > will have to handle all the logic for the syncronization anyway.
> 
> Well since the desktop and server app are using almost exactly the same code
> it is nice to not have to recode everything that accesses the database.
> While data syncronization code will have to be written either way I want the
> existing code to "just work" once taken off line.

Rick,

I'm not sure what OS you're using (I might have missed it in the thread) but
I've never even thought about an embedded database because PG runs very nice on
my laptop (P III - 850Mhz but I did NOT it new).  Now, I'm running Linux so
maybe that is the difference.  In fact, I'm found it to be invaluable to take it
to a client meeting and show not only the high level application gloss but I can
also show the guts of a application/web site.

Now I do understand the sync'ning issue with the database and I haven't had that
issue yet but since you'd have two Pg server's, I would think this issue could
be handled one of the replication project that is available or by something you
custom build (like a perl script that connects to both systems and syncs by
checking for missing pkey's in both systems).

If you have the ability, you might want to invest in building a laptop with
Linux on it until the features you want are available in the methods you want to
use them in.

-$0.02



-- 
Keith C. Perry, MS E.E.
Director of Networks & Applications
VCSN, Inc.
http://vcsn.com
 
____________________________________
This email account is being host by:
VCSN, Inc : http://vcsn.com

Re: embedded/"serverless" (Re: serverless postgresql)

David Garamond <lists@zara.6.isreserved.com> — 2004-01-22T16:02:29Z

Andrew Sullivan wrote:
> On Wed, Jan 21, 2004 at 11:23:40AM -0700, Rick Gigger wrote:
> 
>>Yes but sometimes an enterprise level application may need to be put on a
>>laptop and taken off-line.  Having an embedded database that is compatible
>>with the one on the server makes this a bit easier to do.
> 
> Why can't you just run a postgres instance for this?  What is magic
> about "embedded" for this sort of application?  Sounds like a clever
> wrapper script is all that's necessary for something like that, no?

The "magic" seems to be that the application appears to be managing the 
database by its own, without the help of any separate database product. 
I've seen commercial product that uses embedded Firebird and changes the 
default extension of the database file from *.fdb into *.dat or something.

Of course, with the current cluster/database directory layout, it's 
still easy to spot PostgreSQL footprints all over the place 
(pg_hba.conf, pg_xlog/, PG_VERSION, etc).

-- 
dave

Re: embedded/"serverless" (Re: serverless postgresql)

Jeff Bowden <jlb@houseofdistraction.com> — 2004-01-22T17:03:13Z

David Garamond wrote:

> Andrew Sullivan wrote:
>
>> On Wed, Jan 21, 2004 at 11:23:40AM -0700, Rick Gigger wrote:
>>
>>> Yes but sometimes an enterprise level application may need to be put 
>>> on a
>>> laptop and taken off-line.  Having an embedded database that is 
>>> compatible
>>> with the one on the server makes this a bit easier to do.
>>
>>
>> Why can't you just run a postgres instance for this?  What is magic
>> about "embedded" for this sort of application?  Sounds like a clever
>> wrapper script is all that's necessary for something like that, no?
>
>
> The "magic" seems to be that the application appears to be managing 
> the database by its own, without the help of any separate database 
> product. I've seen commercial product that uses embedded Firebird and 
> changes the default extension of the database file from *.fdb into 
> *.dat or something.
>
> Of course, with the current cluster/database directory layout, it's 
> still easy to spot PostgreSQL footprints all over the place 
> (pg_hba.conf, pg_xlog/, PG_VERSION, etc).
>

It's not so important to hide the fact that postgres is being used as it 
is to spare the user from having to know anything about general database 
administration when the all the app really needs from the db is for it 
to be a more powerful way of storing and retreiving information from 
user-owned files.

Re: embedded/"serverless" (Re: serverless postgresql)

Steve Atkins <steve@blighty.com> — 2004-01-22T17:21:27Z

On Thu, Jan 22, 2004 at 09:03:13AM -0800, Jeff Bowden wrote:
> >>Why can't you just run a postgres instance for this?  What is magic
> >>about "embedded" for this sort of application?  Sounds like a clever
> >>wrapper script is all that's necessary for something like that, no?
> >
> >
> >The "magic" seems to be that the application appears to be managing 
> >the database by its own, without the help of any separate database 
> >product. I've seen commercial product that uses embedded Firebird and 
> >changes the default extension of the database file from *.fdb into 
> >*.dat or something.
> >
> >Of course, with the current cluster/database directory layout, it's 
> >still easy to spot PostgreSQL footprints all over the place 
> >(pg_hba.conf, pg_xlog/, PG_VERSION, etc).
> >
> 
> It's not so important to hide the fact that postgres is being used as it 
> is to spare the user from having to know anything about general database 
> administration when the all the app really needs from the db is for it 
> to be a more powerful way of storing and retreiving information from 
> user-owned files.

Installation and upgrades are another issue.

To 'install' a typical embedded database requires no action at all -
all the infrastructure is embedded within the application, or at most
a single external dynamica library.

I deploy and support applications that use PostgreSQL as a backend
database and a huge amount of support effort is consumed by PostgreSQL
issues. (We're just upgrading from 7.2.* to 7.4.1, and walking even
very unix literate users through the pain of a dump, upgrade, restore
process is very painful).

Other differences between an embedded database and a typical RDBMS is
that the embedded database is likely to be between twice as fast and
ten times as fast, depending on the details of the application. One
tradeoff, of course, is that the embedded database is likely to be
single-user, or at most single-writer, multiple-reader. The other
advantage that the embedded DB likely has is that it probably accepts
queries and returns results as language native types via direct
calls, avoiding all the overhead of translating queries into SQL
and translating the result sets back again.

Another difference between the two is that an RDBMS like PostgreSQL
is that PostgreSQL will make more requirements of the underlying OS
(for instance shared-memory and semaphores) - meaning that the kernel
may need to be reconfigured and the machine rebooted.

If what you need is a single-user database then, nice as it is to
have a full RDBMS, you really don't need all that infrastructure,
and the users will (likely) be far happier without it. If it'd
be nice to have SQL support, then there are embedded, single-user,
databases with some level of SQL support.

All of this explains why an embedded PostgreSQL isn't a great idea. It
being a true multi-user database means that even if you went though
all the work needed to turn it into an embedded database you wouldn't
get most of the advantages.

Short term, if you want embedded SQL, use something like SQLite that's
designed to work that way. If you want to share code between an
embedded DB and PostgreSQL then use a database agnostic API (such as
DBI or libdbi) and restrict yourself to fairly vanilla SQL. If you
must have PostgreSQL used by a non DB aware user then you'll have to
do what I've done, which is to build some automated maintenance code
around PostgreSQL to handle installation and day-to-day maintenance
and monitoring automagically, or patch something similar together
based on the vacuum daemon and so on.

Now, while I think that an embedded fork of PostgreSQL is completely
missing the point I do think that a low maintenance fork or
configuration option would be a very useful feature. I'd love to
be able to ship an application that would

  o Have a private installation of PostgreSQL

  o That would run semi-persistently - if the DB isn't running, the
    application will transparently start it, and if the DB is idle
    for some length of time it gracefully shuts down

  o Is zero-maintenance - all vacuuming, analysing etc is handled
    automatically. So are database version upgrades.

  o That runs under the permissions of the user running the application

  o And that could, by tweaking an application configuration variable
    could swap out the private PostgreSQL installation and instead
    access a standard installation

But (apart from the easy DB version upgrade bit) I'd far rather have
point-in-time or incremental backups, easy replication, richer SQL
support, easier reporting and easier to use replication. So, much as
I'd like it, a low maintenance build is a long way down my list of
priorities.

Cheers,
  Steve

Re: embedded/"serverless" (Re: serverless postgresql)

Rick Gigger <rick@alpinenetworking.com> — 2004-01-22T18:26:03Z

> I'm not sure what OS you're using (I might have missed it in the thread)
but
> I've never even thought about an embedded database because PG runs very
nice on
> my laptop (P III - 850Mhz but I did NOT it new).  Now, I'm running Linux
so
> maybe that is the difference.  In fact, I'm found it to be invaluable to
take it
> to a client meeting and show not only the high level application gloss but
I can
> also show the guts of a application/web site.

If it were just my laptop that would be fine.  But I just can't really
install a full on database server (nor would I feel comfortable doing so) on
every machine that I would need it on.

> Now I do understand the sync'ning issue with the database and I haven't
had that
> issue yet but since you'd have two Pg server's, I would think this issue
could
> be handled one of the replication project that is available or by
something you
> custom build (like a perl script that connects to both systems and syncs
by
> checking for missing pkey's in both systems).

This is not an issue for postgres at this point.  I will most likely just
handle the syncing of specific parts of the database on my own.

Re: embedded/"serverless" (Re: serverless postgresql)

Rick Gigger <rick@alpinenetworking.com> — 2004-01-22T19:02:03Z

> > It's not so important to hide the fact that postgres is being used as it
> > is to spare the user from having to know anything about general database
> > administration when the all the app really needs from the db is for it
> > to be a more powerful way of storing and retreiving information from
> > user-owned files.
>
> Installation and upgrades are another issue.
>
> To 'install' a typical embedded database requires no action at all -
> all the infrastructure is embedded within the application, or at most
> a single external dynamica library.

This is by far (other than reliability and compatibility) the most important
thing for me.

> All of this explains why an embedded PostgreSQL isn't a great idea. It
> being a true multi-user database means that even if you went though
> all the work needed to turn it into an embedded database you wouldn't
> get most of the advantages.

Is it true that postgres is not suited for this and should not be used as
such or is it just a matter of spending the time to allow you maybe compile
an embedded version?

> Short term, if you want embedded SQL, use something like SQLite that's
> designed to work that way. If you want to share code between an
> embedded DB and PostgreSQL then use a database agnostic API (such as
> DBI or libdbi) and restrict yourself to fairly vanilla SQL.

This is exactly the approach that I have been using and it has worked very,
very well.  The only problems that I have come across have occured when my
sql just wasn't quite "vanilla" enough.  For instance sqlite doesn't have
date functions.  We have not come across any show stoppers for this approach
but we do have to always consider that we are coding for 2 different dbs and
sometimes we do have to write db specific sql when there are differences and
test on both dbs.  I understand that if that's what I have to do then that's
what I have to do.  It would just be very, very nice to have postgres in
both situations, if it were possible.

> Now, while I think that an embedded fork of PostgreSQL is completely
> missing the point I do think that a low maintenance fork or
> configuration option would be a very useful feature. I'd love to
> be able to ship an application that would
>
>   o Have a private installation of PostgreSQL
>
>   o That would run semi-persistently - if the DB isn't running, the
>     application will transparently start it, and if the DB is idle
>     for some length of time it gracefully shuts down
>
>   o Is zero-maintenance - all vacuuming, analysing etc is handled
>     automatically. So are database version upgrades.
>
>   o That runs under the permissions of the user running the application
>
>   o And that could, by tweaking an application configuration variable
>     could swap out the private PostgreSQL installation and instead
>     access a standard installation

This just might work fine for me.  :-)

> But (apart from the easy DB version upgrade bit) I'd far rather have
> point-in-time or incremental backups, easy replication, richer SQL
> support, easier reporting and easier to use replication. So, much as
> I'd like it, a low maintenance build is a long way down my list of
> priorities.

point-in-time, incremental backups, easy replication would be big for me as
well.  Especially point-in-time and incremental backups

Re: embedded/"serverless" (Re: serverless postgresql)

Tom Lane <tgl@sss.pgh.pa.us> — 2004-01-22T19:34:47Z

"Rick Gigger" <rick@alpinenetworking.com> writes:
>> All of this explains why an embedded PostgreSQL isn't a great idea. It
>> being a true multi-user database means that even if you went though
>> all the work needed to turn it into an embedded database you wouldn't
>> get most of the advantages.

> Is it true that postgres is not suited for this and should not be used as
> such or is it just a matter of spending the time to allow you maybe compile
> an embedded version?

I think that Steve has it exactly right here.  Postgres isn't designed
to be an embedded database in that sense, and none of the developers are
interested in moving it in that direction.  It would require too many
compromises versus the full-fledged-server situation.

This is definitely a case where one size does not fit all.  Rather
than trying to force-fit Postgres to an application it's not suited for,
you should use another product that is designed for that application.
In short: your time would be better spent on upgrading SQLite to do what
you need.

			regards, tom lane

Re: embedded/"serverless" (Re: serverless postgresql)

Rick Gigger <rick@alpinenetworking.com> — 2004-01-22T20:04:29Z

> "Rick Gigger" <rick@alpinenetworking.com> writes:
> >> All of this explains why an embedded PostgreSQL isn't a great idea. It
> >> being a true multi-user database means that even if you went though
> >> all the work needed to turn it into an embedded database you wouldn't
> >> get most of the advantages.
>
> > Is it true that postgres is not suited for this and should not be used
as
> > such or is it just a matter of spending the time to allow you maybe
compile
> > an embedded version?
>
> I think that Steve has it exactly right here.  Postgres isn't designed
> to be an embedded database in that sense, and none of the developers are
> interested in moving it in that direction.  It would require too many
> compromises versus the full-fledged-server situation.
>
> This is definitely a case where one size does not fit all.  Rather
> than trying to force-fit Postgres to an application it's not suited for,
> you should use another product that is designed for that application.
> In short: your time would be better spent on upgrading SQLite to do what
> you need.

How about the following comment from an earlier post:

> Now, while I think that an embedded fork of PostgreSQL is completely
> missing the point I do think that a low maintenance fork or
> configuration option would be a very useful feature. I'd love to
> be able to ship an application that would
>
>   o Have a private installation of PostgreSQL
>
>   o That would run semi-persistently - if the DB isn't running, the
>     application will transparently start it, and if the DB is idle
>     for some length of time it gracefully shuts down
>
>   o Is zero-maintenance - all vacuuming, analysing etc is handled
>     automatically. So are database version upgrades.
>
>   o That runs under the permissions of the user running the application
>
>   o And that could, by tweaking an application configuration variable
>     could swap out the private PostgreSQL installation and instead
>     access a standard installation

Is this something that could make sense for postgres?

Re: embedded/"serverless" (Re: serverless postgresql)

Steve Atkins <steve@blighty.com> — 2004-01-23T01:16:35Z

On Thu, Jan 22, 2004 at 01:04:29PM -0700, Rick Gigger wrote:

[PostgreSQL ill-suited to embedded use]

> How about the following comment from an earlier post:
> 
> > Now, while I think that an embedded fork of PostgreSQL is completely
> > missing the point I do think that a low maintenance fork or
> > configuration option would be a very useful feature. I'd love to
> > be able to ship an application that would
> >
> >   o Have a private installation of PostgreSQL
> >
> >   o That would run semi-persistently - if the DB isn't running, the
> >     application will transparently start it, and if the DB is idle
> >     for some length of time it gracefully shuts down
> >
> >   o Is zero-maintenance - all vacuuming, analysing etc is handled
> >     automatically. So are database version upgrades.
> >
> >   o That runs under the permissions of the user running the application
> >
> >   o And that could, by tweaking an application configuration variable
> >     could swap out the private PostgreSQL installation and instead
> >     access a standard installation
> 
> Is this something that could make sense for postgres?

It is something that could be done without touching the PostgreSQL
core code - for instance, by dropping a transparent shim in front of
libpg that maintains a little state and intercepts some of the calls
to libpg, yet presents the same API to the application.

Add a flag to the connect string to identify an attempt to connect to
a private installation (passing the top-level path so the shim can
find the binaries, database directory and socket). On a failed attempt
to connect to the private installation try and fork/exec the postmaster,
wait for it to spin up and then continue. Once connected, kill -TERM
the postmaster (so that it'll shutdown gracefully once everyone is
disconnected).

That'll provide everything needed for a private installation, optional
connection to a real installation, automatic spin-up and spin-down.

Vacuuming and analysing can be handled using the algorithms (and the code,
come to that) from pg_autovacuum.

That provides everything on my wish-list apart from automated version
upgrades with only a trivial amount of coding needed - probably no more
than a hundred lines of new code for a minimalist version.

Cheers,
  Steve

Re: embedded/"serverless" (Re: serverless postgresql)

Tom Lane <tgl@sss.pgh.pa.us> — 2004-01-23T03:51:27Z

Steve Atkins <steve@blighty.com> writes:
> [ much snipped ]
> Vacuuming and analysing can be handled using the algorithms (and the code,
> come to that) from pg_autovacuum.

Autovacuum will undoubtedly migrate into the core.  I'm not sure how
soon, or whether the end result will look much like the present
contrib code, but everybody agrees that this is something we need for
main-line development.  Accordingly, I don't see the point in expending
extra effort on this in a (hypothetical) embedded-like wrapper for PG.

			regards, tom lane