Thread

Postgresql and multithreading

Anuradha Ratnaweera <anuradha@lklug.pdn.ac.lk> — 2002-10-16T04:56:53Z

Is there any plans to make postgresql multithreading?

Thanks in advance (and also for all who commented to my question
regarding replication.)

	Anuradha

NB: please don't open fire to declare war on whether multithreading is
needed for PGSql or not.  I am just expecting a black and white answer
from the `authorities' ;)

-- 

Debian GNU/Linux (kernel 2.4.18-xfs-1.1)

If you can survive death, you can probably survive anything.

Re: Postgresql and multithreading

Bruce Momjian <pgman@candle.pha.pa.us> — 2002-10-16T04:59:57Z

Anuradha Ratnaweera wrote:
> 
> Is there any plans to make postgresql multithreading?
> 
> Thanks in advance (and also for all who commented to my question
> regarding replication.)
> 
> 	Anuradha
> 
> NB: please don't open fire to declare war on whether multithreading is
> needed for PGSql or not.  I am just expecting a black and white answer
> from the `authorities' ;)

We don't think it is needed, except perhaps for Win32 and Solaris, which
have slow process creation times.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Re: Postgresql and multithreading

Anuradha Ratnaweera <anuradha@lklug.pdn.ac.lk> — 2002-10-16T05:21:22Z

On Wed, Oct 16, 2002 at 12:59:57AM -0400, Bruce Momjian wrote:
> Anuradha Ratnaweera wrote:
> > 
> > Is there any plans to make postgresql multithreading?
> 
> We don't think it is needed, except perhaps for Win32 and Solaris, which
> have slow process creation times.

Thanks, Bruce.  But what I want to know is whether multithreading is
likely to get into in postgresql, say somewhere in 8.x, or even in 9.x?
(as they did with Apache).  Are there any plans to do so, or is postgres
going to remain rather a multi-process application?

	Anuradha

-- 

Debian GNU/Linux (kernel 2.4.18-xfs-1.1)

One nice thing about egotists: they don't talk about other people.

Re: Postgresql and multithreading

Gavin Sherry <swm@linuxworld.com.au> — 2002-10-16T05:24:03Z

On Wed, 16 Oct 2002, Anuradha Ratnaweera wrote:

> 
> Is there any plans to make postgresql multithreading?
> 
> Thanks in advance (and also for all who commented to my question
> regarding replication.)
> 
> 	Anuradha
> 
> NB: please don't open fire to declare war on whether multithreading is
> needed for PGSql or not.  I am just expecting a black and white answer
> from the `authorities' ;)

This has been discussed at length in the past. The answer has always been
no. Consult the archives for extensive heated discussion :-).

Gavin

Re: Postgresql and multithreading

Bruce Momjian <pgman@candle.pha.pa.us> — 2002-10-16T05:25:23Z

Anuradha Ratnaweera wrote:
> On Wed, Oct 16, 2002 at 12:59:57AM -0400, Bruce Momjian wrote:
> > Anuradha Ratnaweera wrote:
> > > 
> > > Is there any plans to make postgresql multithreading?
> > 
> > We don't think it is needed, except perhaps for Win32 and Solaris, which
> > have slow process creation times.
> 
> Thanks, Bruce.  But what I want to know is whether multithreading is
> likely to get into in postgresql, say somewhere in 8.x, or even in 9.x?
> (as they did with Apache).  Are there any plans to do so, or is postgres
> going to remain rather a multi-process application?

It may be optional some day, most likely for Win32 at first, but we see
little value to it on most other platforms;  of course, we may be wrong.
I am also not sure if it is a big win on Apache either;  I think the
jury is still out on that one, hence the slow adoption of 2.X, and we
don't want to add threads and make a mess of the code or slow it down,
which does often happen.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Re: Postgresql and multithreading

Anuradha Ratnaweera <anuradha@lklug.pdn.ac.lk> — 2002-10-16T05:34:43Z

On Wed, Oct 16, 2002 at 01:25:23AM -0400, Bruce Momjian wrote:
> Anuradha Ratnaweera wrote:
>
> > ... what I want to know is whether multithreading is likely to get
> > into in postgresql, say somewhere in 8.x, or even in 9.x?
> 
> It may be optional some day, most likely for Win32 at first, but we see
> little value to it on most other platforms;  of course, we may be wrong.

In that case, I wonder if it is worth folking a new project to add
threading support to the backend?  Of course, keeping in sync with the
original would be lot of work.

In that way, one should be able to test the hypothesis (whether threads
improve things, or the other way round - if one likes it it that way :))
without messing around with stable postgres code, as they did and do
with postgresql-R.

And a minor question is wheter it is legal to keep the _changes_ in such
a project GPL?

> I am also not sure if it is a big win on Apache either;  I think the
> jury is still out on that one, hence the slow adoption of 2.X,

As far as we are concened, it is the stability, rather than speed which
still keeps us in 1.3.

> and we don't want to add threads and make a mess of the code or slow
> it down, which does often happen.

Fully agreed.

	Anuradha

-- 

Debian GNU/Linux (kernel 2.4.18-xfs-1.1)

	Equality is not when a female Einstein gets promoted to assistant
professor; equality is when a female schlemiel moves ahead as fast as a
male schlemiel.
		-- Ewald Nyquist

Re: Postgresql and multithreading

Bruce Momjian <pgman@candle.pha.pa.us> — 2002-10-16T05:38:59Z

Anuradha Ratnaweera wrote:
> On Wed, Oct 16, 2002 at 01:25:23AM -0400, Bruce Momjian wrote:
> > Anuradha Ratnaweera wrote:
> >
> > > ... what I want to know is whether multithreading is likely to get
> > > into in postgresql, say somewhere in 8.x, or even in 9.x?
> > 
> > It may be optional some day, most likely for Win32 at first, but we see
> > little value to it on most other platforms;  of course, we may be wrong.
> 
> In that case, I wonder if it is worth folking a new project to add
> threading support to the backend?  Of course, keeping in sync with the
> original would be lot of work.

Probably not, but you can try.

> In that way, one should be able to test the hypothesis (whether threads
> improve things, or the other way round - if one likes it it that way :))
> without messing around with stable postgres code, as they did and do
> with postgresql-R.

I guess.

> And a minor question is wheter it is legal to keep the _changes_ in such
> a project GPL?

We don't think we change the license, and we are happy with BSD.  It
certainly will never be merged in with a GPL, I can say that for sure.

> > I am also not sure if it is a big win on Apache either;  I think the
> > jury is still out on that one, hence the slow adoption of 2.X,
> 
> As far as we are concened, it is the stability, rather than speed which
> still keeps us in 1.3.

You could easily lose stability with threads -- don't think they are a
free ride --- they aren't, and no, I don't feel like regurgitating what
is already a 'thread' link on the TODO list.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Re: Postgresql and multithreading

Gavin Sherry <swm@linuxworld.com.au> — 2002-10-16T05:40:47Z

On Wed, 16 Oct 2002, Anuradha Ratnaweera wrote:

> On Wed, Oct 16, 2002 at 01:25:23AM -0400, Bruce Momjian wrote:
> > Anuradha Ratnaweera wrote:
> >
> > > ... what I want to know is whether multithreading is likely to get
> > > into in postgresql, say somewhere in 8.x, or even in 9.x?
> > 
> > It may be optional some day, most likely for Win32 at first, but we see
> > little value to it on most other platforms;  of course, we may be wrong.
> 
> In that case, I wonder if it is worth folking a new project to add
> threading support to the backend?  Of course, keeping in sync with the
> original would be lot of work.

http://sourceforge.net/projects/mtpgsql

> And a minor question is wheter it is legal to keep the _changes_ in such
> a project GPL?

Do you mean 'relicence the forked copy'?

Gavin

Re: Postgresql and multithreading

Anuradha Ratnaweera <anuradha@lklug.pdn.ac.lk> — 2002-10-16T05:45:11Z

On Wed, Oct 16, 2002 at 03:40:47PM +1000, Gavin Sherry wrote:
> On Wed, 16 Oct 2002, Anuradha Ratnaweera wrote:
> 
> > And a minor question is wheter it is legal to keep the _changes_ in such
> > a project GPL?
> 
> Do you mean 'relicence the forked copy'?

Nope.  To keep the `original' code licence as it is and to release the
changes GPL?  Is the question sane at first place?

	Anuradha

-- 

Debian GNU/Linux (kernel 2.4.18-xfs-1.1)

You got to be very careful if you don't know where you're going,
because you might not get there.
		-- Yogi Berra

Re: Postgresql and multithreading

Bruce Momjian <pgman@candle.pha.pa.us> — 2002-10-16T05:46:53Z

Anuradha Ratnaweera wrote:
> On Wed, Oct 16, 2002 at 03:40:47PM +1000, Gavin Sherry wrote:
> > On Wed, 16 Oct 2002, Anuradha Ratnaweera wrote:
> > 
> > > And a minor question is wheter it is legal to keep the _changes_ in such
> > > a project GPL?
> > 
> > Do you mean 'relicence the forked copy'?
> 
> Nope.  To keep the `original' code licence as it is and to release the
> changes GPL?  Is the question sane at first place?

That would be a pretty big mess, I think.  People would add your patch
to our BSD code and it would be GPL.  It could be done, of course.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Re: Postgresql and multithreading

Bruce Momjian <pgman@candle.pha.pa.us> — 2002-10-16T05:51:28Z

Let me add one more thing on this "thread".  This is one email in a long
list of "Oh, gee, you aren't using that wizz-bang new
sync/thread/aio/raid/raw feature" discussion where someone shows up and
wants to know why.  Does anyone know how to address these, efficiently?

If we discuss it, it ends up causing a lot of effort on our part for the
requestor to finally say, "Oh, gee, I didn't realize that."  It isn't
our job to explain that wizz-bang isn't always great.  Maybe that should
be our reply, "Wizz-bang isn't always great" and leave it at that.  Of
course, some will leave thinking we are just idiots, but then again, it
takes one to know one.  It is sort of like walking into a chess match
and asking Bobby Fisher why he didn't move that pawn.  :-)

---------------------------------------------------------------------------

Anuradha Ratnaweera wrote:
> On Wed, Oct 16, 2002 at 03:40:47PM +1000, Gavin Sherry wrote:
> > On Wed, 16 Oct 2002, Anuradha Ratnaweera wrote:
> > 
> > > And a minor question is wheter it is legal to keep the _changes_ in such
> > > a project GPL?
> > 
> > Do you mean 'relicence the forked copy'?
> 
> Nope.  To keep the `original' code licence as it is and to release the
> changes GPL?  Is the question sane at first place?
> 
> 	Anuradha
> 
> -- 
> 
> Debian GNU/Linux (kernel 2.4.18-xfs-1.1)
> 
> You got to be very careful if you don't know where you're going,
> because you might not get there.
> 		-- Yogi Berra
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 6: Have you searched our list archives?
> 
> http://archives.postgresql.org
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Re: Postgresql and multithreading

Anuradha Ratnaweera <anuradha@lklug.pdn.ac.lk> — 2002-10-16T06:13:15Z

On Wed, Oct 16, 2002 at 01:51:28AM -0400, Bruce Momjian wrote:
> 
> Let me add one more thing on this "thread".  This is one email in a
> long list of "Oh, gee, you aren't using that wizz-bang new
> sync/thread/aio/raid/raw feature" discussion where someone shows up
> and wants to know why.  Does anyone know how to address these,
> efficiently?

If somebody pops up asks such dumb questions without even looking at the
FAQ, it is bad, if not idiotic, because it takes useful time away from
the developers.

But my question was not `why don't you implement this feature?`, but `do
you have plans to implement this feature in the future?', and in the
open source spirit of `if something is not there, go implement it
yourself - without troubling developers' ;)

Also, I have read the section 1.9 of the developers FAQ (Why don't we
use threads in the backend?) long, long ago.

> If we discuss it, it ends up causing a lot of effort on our part for
> the requestor to finally say, "Oh, gee, I didn't realize that."

Please don't.  See the "NB" at end of my first mail of this thread.

	Anuradha

-- 

Debian GNU/Linux (kernel 2.4.18-xfs-1.1)

QOTD:
	"I'll listen to reason when it comes out on CD."

Re: Postgresql and multithreading

Shridhar Daithankar <shridhar_daithankar@persistent.co.in> — 2002-10-16T06:27:13Z

On 16 Oct 2002 at 1:25, Bruce Momjian wrote:

> Anuradha Ratnaweera wrote:
> > Thanks, Bruce.  But what I want to know is whether multithreading is
> > likely to get into in postgresql, say somewhere in 8.x, or even in 9.x?
> > (as they did with Apache).  Are there any plans to do so, or is postgres
> > going to remain rather a multi-process application?
> It may be optional some day, most likely for Win32 at first, but we see
> little value to it on most other platforms;  of course, we may be wrong.
> I am also not sure if it is a big win on Apache either;  I think the

Well, I have done some stress testing on 1.3.26 and 2.0.39. Under same hardware 
and network setup and same test case, 1.3.26 maxed at  475-500 requests/sec and 
2.0.39 gave flat 800 requests/sec.

Yes, under light load, there is hardly any difference. But Apache2 series is 
definitely an improvement.

> jury is still out on that one, hence the slow adoption of 2.X, and we
> don't want to add threads and make a mess of the code or slow it down,
> which does often happen.

Well, slow adoption rate is attributed to 'apache 1.3.x is good enough for us' 
syndrome, as far as I can see from news. Once linux distros start shipping with 
apache 2.x series *only*, the upgrade cycle will start rolling, I guess.

Bye
 Shridhar

--
Programming Department:	Mistakes made while you wait.

Re: Postgresql and multithreading

Shridhar Daithankar <shridhar_daithankar@persistent.co.in> — 2002-10-16T06:30:18Z

On 16 Oct 2002 at 15:40, Gavin Sherry wrote:

> > In that case, I wonder if it is worth folking a new project to add
> > threading support to the backend?  Of course, keeping in sync with the
> > original would be lot of work.
> 
> http://sourceforge.net/projects/mtpgsql

Last discussion that happened for multithreading was not to add threads per 
connection like mysql, but to split tasks between threads so that IO blocking 
and efficiently using SMP abilities would be taken care off IIRC..

One thread per connection isn't going to change much, at least for mainstream 
postgresql. For embedded, it might be a necessity..

Bye
 Shridhar

--
QOTD:	"You're so dumb you don't even have wisdom teeth."

Re: Postgresql and multithreading

Justin Clift <justin@postgresql.org> — 2002-10-16T08:34:52Z

Bruce Momjian wrote:
> Anuradha Ratnaweera wrote:
<snip>
> > Nope.  To keep the `original' code licence as it is and to release the
> > changes GPL?  Is the question sane at first place?
> 
> That would be a pretty big mess, I think.  People would add your patch
> to our BSD code and it would be GPL.  It could be done, of course.

Don't think so.  The patches would be "derived code" that only exist
because of the BSD licensed PostgreSQL base.

Being "derived code" they'd have to be released as BSD and GPL wouldn't
enter the picture, regardless if they're released separately as add-on
patches or not.

:-)

Regards and best wishes,

Justin Clift

 
> --
>   Bruce Momjian                        |  http://candle.pha.pa.us
>   pgman@candle.pha.pa.us               |  (610) 359-1001
>   +  If your life is a hard drive,     |  13 Roberts Road
>   +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

-- 
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
   - Indira Gandhi

Re: Postgresql and multithreading

Greg Copeland <greg@copelandconsulting.net> — 2002-10-16T14:21:54Z

On Wed, 2002-10-16 at 01:27, Shridhar Daithankar wrote:
> Well, slow adoption rate is attributed to 'apache 1.3.x is good enough for us' 
> syndrome, as far as I can see from news. Once linux distros start shipping with 
> apache 2.x series *only*, the upgrade cycle will start rolling, I guess.

I think that's part of it.  I think the other part is that by the time
you're getting to huge r/s numbers, typical web site bandwidth is
already used up.  So, what's the point in adding more breathing room
when you don't have the bandwidth to use it anyways.

Greg

Re: Postgresql and multithreading

Robert Treat <rtreat@webmd.net> — 2002-10-16T14:37:02Z

On Wed, 2002-10-16 at 04:34, Justin Clift wrote:
> Bruce Momjian wrote:
> > Anuradha Ratnaweera wrote:
> <snip>
> > > Nope.  To keep the `original' code licence as it is and to release the
> > > changes GPL?  Is the question sane at first place?
> > 
> > That would be a pretty big mess, I think.  People would add your patch
> > to our BSD code and it would be GPL.  It could be done, of course.
> 
> Don't think so.  The patches would be "derived code" that only exist
> because of the BSD licensed PostgreSQL base.
> 
> Being "derived code" they'd have to be released as BSD and GPL wouldn't
> enter the picture, regardless if they're released separately as add-on
> patches or not.
> 

I'm pretty sure BSD allows you to relicense derived code as you see fit.
However, any derived project that was released GPL would have a hell of
a time ever getting put back into the main source (short of
relicensing).

Robert Treat

Re: Postgresql and multithreading

tycho@fruru.com — 2002-10-16T14:50:35Z

On Wed, 2002-10-16 at 16:37, Robert Treat wrote:

> I'm pretty sure BSD allows you to relicense derived code as you see fit.
> However, any derived project that was released GPL would have a hell of
> a time ever getting put back into the main source (short of
> relicensing).

Exactly.  This is one of the reasons why BSD license is very much liked
by proprietary software vendors (think MSFT), unlike the GPL which
doesn't allow someone to change and redistribute their work with
restrictive licenses.

Cheers,
Tycho

(BTW: I'm not asking to change the license of Postgresql, I know the -
dogmatic -  answer to that one.  So please don't misunderstand my mail)

-- 
Tycho Fruru			                tycho@fruru.com
"Prediction is extremely difficult. Especially about the future."
  - Niels Bohr

Re: Postgresql and multithreading

Curtis Faith <curtis@galtair.com> — 2002-10-16T18:08:21Z

Bruce Momjian wrote:
> Let me add one more thing on this "thread".  This is one email in a long
> list of "Oh, gee, you aren't using that wizz-bang new
> sync/thread/aio/raid/raw feature" discussion where someone shows up and
> wants to know why.  Does anyone know how to address these, efficiently?

You've brought up good points here. I'm sure that you consider me guilty of
this with regard to my aio discussions. So I might offer a few specific
suggestions.

1) Someone's taking the time to generate a summary of the current thinking
with regard to a particular "whiz-bang" feature. - I can do this as a first
pass  for aio, if you think it's a good idea.

2) Including the pros and cons of the feature/implementation and how close
the group is to deciding whether something would be worth doing. - I can
also do this.

3) A set of criteria that would need to be met or demonstrated before a
feature would be considered good enough for inclusion into the main code.

If there was a separate section of the document "Developer's FAQ" that
handled this, this would help.

4) A development/feature philosophy section. Maybe three or four paragraphs
with one specific example: perhaps multi-threading.

5) I'd also suggest changing some specfics in the FAQ Section 1.2 where
there is currently:

* The usual process for source additions is:
*
* Review the TODO list.
* Discuss hackers the desirability of the fix/feature.
* How should it behave in complex circumstances?

Add here that a check should be made to the new section in the FAQ
described above. Also, section 1.1 has:

* Discussion on the patch typically happens here. If the patch adds a
* major feature, it would be a good idea to talk about it first on
* the HACKERS list, in order to increase the chances of it being
* accepted, as well as toavoid duplication of effort.

We should perhaps add here a section describing the phenomenon you describe
regarding "whiz-bang" features.

I tried to prepare as best I could before bringing anything forward to
HACKERS. In particular, I read the last two years of archives with anything
to do with the WAL log and looked at the current code, read the TODOs, read
a fair amount of discussions about aio. etc. So I was attempting to comply
with my interpretation of the process. Yet, despite these efforts, you no
doubt consider me guilty of creating unnecessary work, an outcome neither
of us desired.

I'm undeterred in my desire to come up with something meaningful and am
working on some interesting tests. I do, however, now know that the level
of scepticism and the standards of architectural purity are high. I
consider this good all around.

- Curtis

Re: Postgresql and multithreading

Hannu Krosing <hannu@tm.ee> — 2002-10-16T19:25:37Z

On Wed, 2002-10-16 at 23:08, Curtis Faith wrote:
> Bruce Momjian wrote:
> I tried to prepare as best I could before bringing anything forward to
> HACKERS. In particular, I read the last two years of archives with anything
> to do with the WAL log and looked at the current code, read the TODOs, read
> a fair amount of discussions about aio. etc. So I was attempting to comply
> with my interpretation of the process. Yet, despite these efforts, you no
> doubt consider me guilty of creating unnecessary work, an outcome neither
> of us desired.

But that "unneccessary work" resulted in Tom finding and fixing an
unintended behaviour (aka a performance bug) that prevented postgres
from ever doing more than 1 commit per disk revolution on non-lieing
SCSI disks ;)

> I'm undeterred in my desire to come up with something meaningful and am
> working on some interesting tests. I do, however, now know that the level
> of scepticism and the standards of architectural purity are high. I
> consider this good all around.

I still have big expectations for use of aio, especially considering
that at least for free OSes one is not forced to stop at the DB/OS
boundary, but are free to go and improve the os side implementation as
well if it is needed.

But still some empirical tests are probably needed - if we can keep IO
occupied for 99% in a meaningful way withou aio, then the time is
probably better spent on something else ;)

--------------------
Hannu

Re: Postgresql and multithreading

Anuradha Ratnaweera <anuradha@lklug.pdn.ac.lk> — 2002-10-17T06:39:37Z

On Wed, Oct 16, 2002 at 02:08:21PM -0400, Curtis Faith wrote:
> 
> 2) Including the pros and cons of the feature/implementation and how close
> the group is to deciding whether something would be worth doing. - I can
> also do this.

The pros and cons of many such features have been discussed over the
lists as well as on the FAQs. But the second matter, the group's
likehood their implementation cannot always be deduced from those
communications or from docs.

Therefore suggested material into the FAQs are going to be extremely
useful to like-to-be developers. They also would hopefully reduce
unnecessary traffic on the list.

	Anuradha

-- 

Debian GNU/Linux (kernel 2.4.18-xfs-1.1)

I have found little that is good about human beings.  In my experience
most of them are trash.
		-- Sigmund Freud

Re: Postgresql and multithreading

Marc G. Fournier <scrappy@hub.org> — 2002-10-18T03:18:19Z

On Wed, 16 Oct 2002, Bruce Momjian wrote:

> It may be optional some day, most likely for Win32 at first, but we see
> little value to it on most other platforms;  of course, we may be wrong.
> I am also not sure if it is a big win on Apache either;  I think the
> jury is still out on that one, hence the slow adoption of 2.X, and we
> don't want to add threads and make a mess of the code or slow it down,
> which does often happen.

Actually, Apache2 is both multi-process and multi-thread, even in the
'multi-thread' model ... I've played a bit with it, and it works  at a
sort of 'half way point' taking advantage of advantages of both ... in
fact, I really wish someone would look at it seriously, since it mimics
alot of what we do to start with ...

old apache - one parent process (ie. postmaster) with child process (ie.
             postgres) actually handling the work

new apache - one parent process (ie. postmaster) with child processes (ie.
             postgres) actually handling the work, with a twist .. each
             child process can handle x threaded processes

so, in a heavily loaded web server, you might see 10 httpd processes
running, each of which handling 15 threaded connections ...

even getting away from multiple db connections per child process, I could
see some other areas where multi-threading could be useful, assuming that
my limited knowleddge of threading is remotely correct ... a big/cool one
could be:

distributed/clustered databases

	a database could be setup on multiple servers, where the tables
are created as 'CREATE TABLE newtable ON SERVER serverb', so that when you
connect to that table, the child process knows to auto-matically establish
connections to the remote servers to pull data in

	this would also work for inter-database queries, that several ppl
in the past have asked for

Re: Postgresql and multithreading

Tom Lane <tgl@sss.pgh.pa.us> — 2002-10-18T03:20:07Z

Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Let me add one more thing on this "thread".  This is one email in a long
> list of "Oh, gee, you aren't using that wizz-bang new
> sync/thread/aio/raid/raw feature" discussion where someone shows up and
> wants to know why.  Does anyone know how to address these, efficiently?

Simple: respond to 'em all with a one-line answer: "convince us why we
should use it".  The burden of proof always seems to fall on the wrong
end in these discussions.

			regards, tom lane

Re: Postgresql and multithreading

Marc G. Fournier <scrappy@hub.org> — 2002-10-18T03:21:59Z

On Wed, 16 Oct 2002, Anuradha Ratnaweera wrote:

> On Wed, Oct 16, 2002 at 01:25:23AM -0400, Bruce Momjian wrote:
> > Anuradha Ratnaweera wrote:
> >
> > > ... what I want to know is whether multithreading is likely to get
> > > into in postgresql, say somewhere in 8.x, or even in 9.x?
> >
> > It may be optional some day, most likely for Win32 at first, but we see
> > little value to it on most other platforms;  of course, we may be wrong.
>
> In that case, I wonder if it is worth folking a new project to add
> threading support to the backend?  Of course, keeping in sync with the
> original would be lot of work.

Actually, if you go through the archives, there has been talk about what
would have to be done to the main source tree towards getting threading
included ... as well as a lengthy discussion about the steps involved.

the first and foremost issue that needs to be addressed is cleaning up the
global variables that are used throughout ...

Re: Postgresql and multithreading

Greg Copeland <greg@copelandconsulting.net> — 2002-10-18T13:47:56Z

On Thu, 2002-10-17 at 22:20, Tom Lane wrote:
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > Let me add one more thing on this "thread".  This is one email in a long
> > list of "Oh, gee, you aren't using that wizz-bang new
> > sync/thread/aio/raid/raw feature" discussion where someone shows up and
> > wants to know why.  Does anyone know how to address these, efficiently?
> 
> Simple: respond to 'em all with a one-line answer: "convince us why we
> should use it".  The burden of proof always seems to fall on the wrong
> end in these discussions.
> 
> 			regards, tom lane

That may be easier said that done.  If you don't know what the
objections are, it's hard to argue your case.  If you do know and
understand the objections, chances are you already know the code very
well and/or have the mailing lists for a very long time.  This basically
means, you don't want to hear from anyone unless they are "one" with the
code.  That seems and sounds very anti-open source.

After it's all said and done, I think you guys are barking up the wrong
tree.  Open Source is all about sharing ideas.  Many times I've seen
ideas expressed here that were not exact hits yet help facilitate
discussion, understanding on the topics in general and in some cases may
even spur other ideas or associated code fixes/improvements.  When I
first started on this list, I was scolded rather harshly for not asking
all of my questions on the list.  Originally, I was told to ask
reasonable questions so that everyone can learn.  Now, it seems, that
people don't want to answer questions at all as it's bothering the
developers.

Commonly asked items, such as threading, seems like they are being
addressed rather well without core developer participation.  Right now,
I'm not seeing any down sides to what's currently in place.  If the core
developers still feel like they are spending more time then they like,
then perhaps those that following the mailing list can step forward a
little more to address general questions and defer when needed.  The
topic, such as threading, was previously addressed yet people still
followed up on the topic.  Perhaps those that don't want to be bothered
should allow more time for others to address the topic and leave it
alone once it has been addressed.  That alone seems like it would be a
huge time saver for the developers and a better use of resources.

Greg

Re: Postgresql and multithreading

Tom Lane <tgl@sss.pgh.pa.us> — 2002-10-18T14:28:38Z

Greg Copeland <greg@copelandconsulting.net> writes:
> On Thu, 2002-10-17 at 22:20, Tom Lane wrote:
>> Simple: respond to 'em all with a one-line answer: "convince us why we
>> should use it".  The burden of proof always seems to fall on the wrong
>> end in these discussions.

> ... Now, it seems, that
> people don't want to answer questions at all as it's bothering the
> developers.

Not at all.  But rehashing issues that have been talked out repeatedly
is starting to bug some of us ;-).  Perhaps the correct "standard
answer" is more like "this has been discussed before, please read the
list archives".

			regards, tom lane

Re: Postgresql and multithreading

Bruce Momjian <pgman@candle.pha.pa.us> — 2002-10-18T15:39:18Z

Tom Lane wrote:
> Greg Copeland <greg@copelandconsulting.net> writes:
> > On Thu, 2002-10-17 at 22:20, Tom Lane wrote:
> >> Simple: respond to 'em all with a one-line answer: "convince us why we
> >> should use it".  The burden of proof always seems to fall on the wrong
> >> end in these discussions.
> 
> > ... Now, it seems, that
> > people don't want to answer questions at all as it's bothering the
> > developers.
> 
> Not at all.  But rehashing issues that have been talked out repeatedly
> is starting to bug some of us ;-).  Perhaps the correct "standard
> answer" is more like "this has been discussed before, please read the
> list archives".

I need to add something to the developers FAQ on this.  I will do it
soon.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Re: Postgresql and multithreading

Greg Copeland <greg@copelandconsulting.net> — 2002-10-18T15:51:08Z

On Fri, 2002-10-18 at 09:28, Tom Lane wrote:
> Greg Copeland <greg@copelandconsulting.net> writes:
> > On Thu, 2002-10-17 at 22:20, Tom Lane wrote:
> >> Simple: respond to 'em all with a one-line answer: "convince us why we
> >> should use it".  The burden of proof always seems to fall on the wrong
> >> end in these discussions.
> 
> > ... Now, it seems, that
> > people don't want to answer questions at all as it's bothering the
> > developers.
> 
> Not at all.  But rehashing issues that have been talked out repeatedly
> is starting to bug some of us ;-).  Perhaps the correct "standard
> answer" is more like "this has been discussed before, please read the
> list archives".

I agree.  That sounds like a much more reasonable response.  In fact, if
you were to simply let the fledglings respond, it would completely take
you guys out of the loop.

Perhaps something like a Wiki or FAQ-O-Matic can be added whereby, the
user base can help maintain it?  That would seemingly help take some
load off of Bruce too.

Greg

Re: Postgresql and multithreading

Steve Wolfe <nw@codon.com> — 2002-10-18T19:11:16Z

  On the recurring debate of threading vs. forking, I was giving it a fwe
thoughts a few days ago, particularly with concern to Linux's memory model.

  On IA32 platforms with over 4 gigs of memory, any one process can only
"see" up to 3 or 4 gigs of that.  Having each postmaster fork off as a new
process obviously would allow a person to utilize very copious quantities of
memory, assuming that (a) they were dealing with concurrent PG sessions, and
(b) PG had reason to use the memory.

  I'm not entirely clear on threading in Linux - would it provide the same
benefits, or would it suddenly lock you into a 3-gig memory space?

steve

Re: Postgresql and multithreading

Anuradha Ratnaweera <anuradha@lklug.pdn.ac.lk> — 2002-10-19T06:58:44Z

On Fri, Oct 18, 2002 at 10:28:38AM -0400, Tom Lane wrote:
> Greg Copeland <greg@copelandconsulting.net> writes:
> > On Thu, 2002-10-17 at 22:20, Tom Lane wrote:
> >> Simple: respond to 'em all with a one-line answer: "convince us why we
> >> should use it".  The burden of proof always seems to fall on the wrong
> >> end in these discussions.
> 
> > ... Now, it seems, that
> > people don't want to answer questions at all as it's bothering the
> > developers.
> 
> Not at all.  But rehashing issues that have been talked out repeatedly
> is starting to bug some of us ;-).  Perhaps the correct "standard
> answer" is more like "this has been discussed before, please read the
> list archives".

Let me explain my posting which started this `thread':

- The developer's FAQ section 1.9 explains why PostgreSQL doesn't use
  threads (and many times it has been discussed on the list).

- The TODO list has an item `Experiment with multi-threaded backend' and
  points to a mailing list discussion about the implementation by Myron
  Scott.  His final comment is that he didn't `gain much performance'
  and `ended up with some pretty unmanagable code'.  He also says that
  he wouldn't `personally try this again ... but there probably was a
  better way'.

- I was going through the TODO list, and was wondering if I should try
  on this. But before doing that, naturally, I wanted to figure out if
  any of the core developers themselves have any plans of doing it.

Now, I am trying hard to figure out why this `are you going to do this?
otherwise I can try it', type posting was not differentiated from
numerous `why don't YOU implement this feature' type postings ;)

	Anuradha

-- 

Debian GNU/Linux (kernel 2.4.18-xfs-1.1)

"Life is too important to take seriously."
		-- Corky Siegel

Re: Postgresql and multithreading

Tom Lane <tgl@sss.pgh.pa.us> — 2002-10-19T16:30:03Z

Anuradha Ratnaweera <anuradha@lklug.pdn.ac.lk> writes:
> Let me explain my posting which started this `thread':

> - The developer's FAQ section 1.9 explains why PostgreSQL doesn't use
>   threads (and many times it has been discussed on the list).

> - The TODO list has an item `Experiment with multi-threaded backend' and
>   points to a mailing list discussion about the implementation by Myron
>   Scott.  His final comment is that he didn't `gain much performance'
>   and `ended up with some pretty unmanagable code'.  He also says that
>   he wouldn't `personally try this again ... but there probably was a
>   better way'.

> - I was going through the TODO list, and was wondering if I should try
>   on this. But before doing that, naturally, I wanted to figure out if
>   any of the core developers themselves have any plans of doing it.

> Now, I am trying hard to figure out why this `are you going to do this?
> otherwise I can try it', type posting was not differentiated from
> numerous `why don't YOU implement this feature' type postings ;)

Well, if you'd actually said the above, we'd probably have replied to
the effect of "we still think it's an unpromising project, but try it
if you like".  By my reading, your earliest postings in this thread
showed no sign of any familiarity at all with the history:

http://archives.postgresql.org/pgsql-hackers/2002-10/msg00704.php
http://archives.postgresql.org/pgsql-hackers/2002-10/msg00707.php
http://archives.postgresql.org/pgsql-hackers/2002-10/msg00711.php

and so you got the sort of response that's usually given to clueless
newbies...

			regards, tom lane

Re: Postgresql and multithreading

Shridhar Daithankar <shridhar_daithankar@persistent.co.in> — 2002-10-21T13:30:08Z

On 18 Oct 2002 at 13:11, Steve Wolfe wrote:

> 
>   On the recurring debate of threading vs. forking, I was giving it a fwe
> thoughts a few days ago, particularly with concern to Linux's memory model.
> 
>   On IA32 platforms with over 4 gigs of memory, any one process can only
> "see" up to 3 or 4 gigs of that.  Having each postmaster fork off as a new
> process obviously would allow a person to utilize very copious quantities of
> memory, assuming that (a) they were dealing with concurrent PG sessions, and
> (b) PG had reason to use the memory.

Well IIRC PG can not use more than 2Gigs of memory or 250K shared buffers 
(Unless you alter the buffer size itself). This does not become an issue in 
itself.

 
>   I'm not entirely clear on threading in Linux - would it provide the same
> benefits, or would it suddenly lock you into a 3-gig memory space?

Well, if you need to allocate 3Gig of memory to single process like postgresql, 
it's time to get a 64bit CPU. IIRC linux run on quite a few of them.

HTH
Bye
 Shridhar

--
QOTD:	"Oh, no, no...  I'm not beautiful.  Just very, very pretty."

Re: Postgresql and multithreading

Doug McNaught <doug@wireboard.com> — 2002-10-21T15:27:19Z

"Steve Wolfe" <nw@codon.com> writes:

>   On the recurring debate of threading vs. forking, I was giving it a fwe
> thoughts a few days ago, particularly with concern to Linux's memory model.
> 
>   On IA32 platforms with over 4 gigs of memory, any one process can only
> "see" up to 3 or 4 gigs of that.  Having each postmaster fork off as a new
> process obviously would allow a person to utilize very copious quantities of
> memory, assuming that (a) they were dealing with concurrent PG sessions, and
> (b) PG had reason to use the memory.
> 
>   I'm not entirely clear on threading in Linux - would it provide the same
> benefits, or would it suddenly lock you into a 3-gig memory space?

Linux threads are basically processes that share the same VM space, so
you'd be limited to 3GB or whatever, since that's what a VM space can
"see".

-Doug

Re: Postgresql and multithreading

D. Hageman <dhageman@dracken.com> — 2002-10-21T16:02:16Z

This in many ways is a bogus argument in that 1) postgresql runs on more 
then just Linux and 2) amount of memmory that can be addressed by a 
process is tunable up to the point that it reaches a hardware limitation.

It also should be noted that when a process reaches such a size that it 
better have a good reason.  Now let us do a gedanken experiment and say 
you do have a good reason - fork a couple of these and your machine will 
thrash like nothing else ... also that whole hardware limitation will come 
into play more sooner then later ... 

On 21 Oct 2002, Doug McNaught wrote:

> "Steve Wolfe" <nw@codon.com> writes:
> 
> >   On the recurring debate of threading vs. forking, I was giving it a fwe
> > thoughts a few days ago, particularly with concern to Linux's memory model.
> > 
> >   On IA32 platforms with over 4 gigs of memory, any one process can only
> > "see" up to 3 or 4 gigs of that.  Having each postmaster fork off as a new
> > process obviously would allow a person to utilize very copious quantities of
> > memory, assuming that (a) they were dealing with concurrent PG sessions, and
> > (b) PG had reason to use the memory.
> > 
> >   I'm not entirely clear on threading in Linux - would it provide the same
> > benefits, or would it suddenly lock you into a 3-gig memory space?
> 
> Linux threads are basically processes that share the same VM space, so
> you'd be limited to 3GB or whatever, since that's what a VM space can
> "see".
> 
> -Doug
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo@postgresql.org so that your
> message can get through to the mailing list cleanly
> 

-- 
//========================================================\\
||  D. Hageman                    <dhageman@dracken.com>  ||
\\========================================================//

Re: Postgresql and multithreading

Doug McNaught <doug@wireboard.com> — 2002-10-21T16:25:16Z

"D. Hageman" <dhageman@dracken.com> writes:

> This in many ways is a bogus argument in that 1) postgresql runs on more 
> then just Linux and 2) amount of memmory that can be addressed by a 
> process is tunable up to the point that it reaches a hardware limitation.

1) The OP specifically asked about Linux threads.
2) True up to a point--Linux (and most other Unices) reserve some
   part of the VM address space for the kernel.  On 64-bit this is a
   non-issue, on 32-bit it's quite important now that you can put 4+GB
   in a machine. 

> It also should be noted that when a process reaches such a size that it 
> better have a good reason.  Now let us do a gedanken experiment and say 
> you do have a good reason - fork a couple of these and your machine will 
> thrash like nothing else ... also that whole hardware limitation will come 
> into play more sooner then later ... 

True enough.  The only real use I can see for gobs of memory on a
32-bit PAE machine with PG is to give each process its own big hunk of
'sortmem' for doing large sorts.  If you have 64 GB in the machine
setting 'sortmem' to 1GB or so starts to look reasonable...

-Doug

Re: Postgresql and multithreading

Bruce Momjian <pgman@candle.pha.pa.us> — 2002-11-05T22:11:05Z

I have updated the developers FAQ item 1.9 to address this:

	http://developer.postgresql.org/readtext.php?src/FAQ/FAQ_DEV.html+Developers-FAQ

---------------------------------------------------------------------------

Anuradha Ratnaweera wrote:
> On Wed, Oct 16, 2002 at 01:51:28AM -0400, Bruce Momjian wrote:
> > 
> > Let me add one more thing on this "thread".  This is one email in a
> > long list of "Oh, gee, you aren't using that wizz-bang new
> > sync/thread/aio/raid/raw feature" discussion where someone shows up
> > and wants to know why.  Does anyone know how to address these,
> > efficiently?
> 
> If somebody pops up asks such dumb questions without even looking at the
> FAQ, it is bad, if not idiotic, because it takes useful time away from
> the developers.
> 
> But my question was not `why don't you implement this feature?`, but `do
> you have plans to implement this feature in the future?', and in the
> open source spirit of `if something is not there, go implement it
> yourself - without troubling developers' ;)
> 
> Also, I have read the section 1.9 of the developers FAQ (Why don't we
> use threads in the backend?) long, long ago.
> 
> > If we discuss it, it ends up causing a lot of effort on our part for
> > the requestor to finally say, "Oh, gee, I didn't realize that."
> 
> Please don't.  See the "NB" at end of my first mail of this thread.
> 
> 	Anuradha
> 
> -- 
> 
> Debian GNU/Linux (kernel 2.4.18-xfs-1.1)
> 
> QOTD:
> 	"I'll listen to reason when it comes out on CD."
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
> 

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073