Thread

RE: Allowing WAL fsync to be done via O_SYNC

Mikheev, Vadim <vmikheev@sectorbase.com> — 2001-03-15T21:28:08Z

> I believe that we don't know enough yet to nail down a hard-wired
> decision.  Vadim's idea of preferring O_DSYNC if it appears to be
> different from O_SYNC is a good first cut, but I think we'd 
> better make it possible to override that, at least for testing purposes.

So let's leave fsync as default and add option to open log files
with O_DSYNC/O_SYNC.

Vadim

Re: Allowing WAL fsync to be done via O_SYNC

Alfred Perlstein <bright@wintelcom.net> — 2001-03-15T22:51:00Z

* Mikheev, Vadim <vmikheev@SECTORBASE.COM> [010315 13:52] wrote:
> > I believe that we don't know enough yet to nail down a hard-wired
> > decision.  Vadim's idea of preferring O_DSYNC if it appears to be
> > different from O_SYNC is a good first cut, but I think we'd 
> > better make it possible to override that, at least for testing purposes.
> 
> So let's leave fsync as default and add option to open log files
> with O_DSYNC/O_SYNC.

I have a weird and untested suggestion:

How many files need to be fsync'd?

If it's more than one, what might work is using mmap() to map the
files in adjacent areas, then calling msync() on the entire range,
this would allow you to batch fsync the data.

The only problem is that I'm not sure:

1) how portable msync() is.
2) if msync garauntees metadata consistancy.

Another benifit of mmap() is the 'zero' copy nature of it.

-- 
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]

Re: Allowing WAL fsync to be done via O_SYNC

Tom Lane <tgl@sss.pgh.pa.us> — 2001-03-15T22:54:22Z

Alfred Perlstein <bright@wintelcom.net> writes:
> How many files need to be fsync'd?

Only one.

> If it's more than one, what might work is using mmap() to map the
> files in adjacent areas, then calling msync() on the entire range,
> this would allow you to batch fsync the data.

Interesting thought, but mmap to a prespecified address is most
definitely not portable, whether or not you want to assume that
plain mmap is ...

			regards, tom lane

Re: Allowing WAL fsync to be done via O_SYNC

Alfred Perlstein <bright@wintelcom.net> — 2001-03-15T23:02:20Z

* Tom Lane <tgl@sss.pgh.pa.us> [010315 14:54] wrote:
> Alfred Perlstein <bright@wintelcom.net> writes:
> > How many files need to be fsync'd?
> 
> Only one.
> 
> > If it's more than one, what might work is using mmap() to map the
> > files in adjacent areas, then calling msync() on the entire range,
> > this would allow you to batch fsync the data.
> 
> Interesting thought, but mmap to a prespecified address is most
> definitely not portable, whether or not you want to assume that
> plain mmap is ...

Yeah... :(

Evil thought though (for reference):

mmap(anon memory) returns addr1
addr2 = addr1 + maplen
split addr1<->addr2 on points A B and C
mmap(file1 over addr1 to A)
mmap(file2 over A to B)
mmap(file3 over B to C)
mmap(file4 over C to addr2)

It _should_ work, but there's probably some corner cases where it
doesn't.

-- 
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]

Re[2]: Allowing WAL fsync to be done via O_SYNC

Xu Yifeng <jamexu@telekbird.com.cn> — 2001-03-16T06:26:00Z

Hello Tom,

Friday, March 16, 2001, 6:54:22 AM, you wrote:

TL> Alfred Perlstein <bright@wintelcom.net> writes:
>> How many files need to be fsync'd?

TL> Only one.

>> If it's more than one, what might work is using mmap() to map the
>> files in adjacent areas, then calling msync() on the entire range,
>> this would allow you to batch fsync the data.

TL> Interesting thought, but mmap to a prespecified address is most
TL> definitely not portable, whether or not you want to assume that
TL> plain mmap is ...

TL>                         regards, tom lane

Could anyone consider fork a syncer process to sync data to disk ?
build a shared sync queue, when a daemon process want to do sync after
write() is called, just put a sync request to the queue. this can release
process from blocked on writing as soon as possible. multipile sync
request for one file can be merged when the request is been inserting to
the queue.

-- 
Regards,
Xu Yifeng

Re: Re[2]: Allowing WAL fsync to be done via O_SYNC

Alfred Perlstein <bright@wintelcom.net> — 2001-03-16T07:21:09Z

* Xu Yifeng <jamexu@telekbird.com.cn> [010315 22:25] wrote:
> Hello Tom,
> 
> Friday, March 16, 2001, 6:54:22 AM, you wrote:
> 
> TL> Alfred Perlstein <bright@wintelcom.net> writes:
> >> How many files need to be fsync'd?
> 
> TL> Only one.
> 
> >> If it's more than one, what might work is using mmap() to map the
> >> files in adjacent areas, then calling msync() on the entire range,
> >> this would allow you to batch fsync the data.
> 
> TL> Interesting thought, but mmap to a prespecified address is most
> TL> definitely not portable, whether or not you want to assume that
> TL> plain mmap is ...
> 
> TL>                         regards, tom lane
> 
> Could anyone consider fork a syncer process to sync data to disk ?
> build a shared sync queue, when a daemon process want to do sync after
> write() is called, just put a sync request to the queue. this can release
> process from blocked on writing as soon as possible. multipile sync
> request for one file can be merged when the request is been inserting to
> the queue.

I suggested this about a year ago. :)

The problem is that you need that process to potentially open and close
many files over and over.

I still think it's somewhat of a good idea.

-- 
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]

Re[4]: Allowing WAL fsync to be done via O_SYNC

Xu Yifeng <jamexu@telekbird.com.cn> — 2001-03-16T08:53:12Z

Hello Alfred,

Friday, March 16, 2001, 3:21:09 PM, you wrote:

AP> * Xu Yifeng <jamexu@telekbird.com.cn> [010315 22:25] wrote:
>>
>> Could anyone consider fork a syncer process to sync data to disk ?
>> build a shared sync queue, when a daemon process want to do sync after
>> write() is called, just put a sync request to the queue. this can release
>> process from blocked on writing as soon as possible. multipile sync
>> request for one file can be merged when the request is been inserting to
>> the queue.

AP> I suggested this about a year ago. :)

AP> The problem is that you need that process to potentially open and close
AP> many files over and over.

AP> I still think it's somewhat of a good idea.

I am not a DBMS guru.
couldn't the syncer process cache opened files? is there any problem I
didn't consider ?

-- 
Best regards,
Xu Yifeng

Re: Re[4]: Allowing WAL fsync to be done via O_SYNC

Alfred Perlstein <bright@wintelcom.net> — 2001-03-16T12:45:35Z

* Xu Yifeng <jamexu@telekbird.com.cn> [010316 01:15] wrote:
> Hello Alfred,
> 
> Friday, March 16, 2001, 3:21:09 PM, you wrote:
> 
> AP> * Xu Yifeng <jamexu@telekbird.com.cn> [010315 22:25] wrote:
> >>
> >> Could anyone consider fork a syncer process to sync data to disk ?
> >> build a shared sync queue, when a daemon process want to do sync after
> >> write() is called, just put a sync request to the queue. this can release
> >> process from blocked on writing as soon as possible. multipile sync
> >> request for one file can be merged when the request is been inserting to
> >> the queue.
> 
> AP> I suggested this about a year ago. :)
> 
> AP> The problem is that you need that process to potentially open and close
> AP> many files over and over.
> 
> AP> I still think it's somewhat of a good idea.
> 
> I am not a DBMS guru.

Hah, same here. :)

> couldn't the syncer process cache opened files? is there any problem I
> didn't consider ?

1) IPC latency, the amount of time it takes to call fsync will
   increase by at least two context switches.

2) a working set (number of files needed to be fsync'd) that
   is larger than the amount of files you wish to keep open.

-- 
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]

Re: Re[2]: Allowing WAL fsync to be done via O_SYNC

Bruce Momjian <pgman@candle.pha.pa.us> — 2001-03-16T15:11:30Z

> > Could anyone consider fork a syncer process to sync data to disk ?
> > build a shared sync queue, when a daemon process want to do sync after
> > write() is called, just put a sync request to the queue. this can release
> > process from blocked on writing as soon as possible. multipile sync
> > request for one file can be merged when the request is been inserting to
> > the queue.
> 
> I suggested this about a year ago. :)
> 
> The problem is that you need that process to potentially open and close
> many files over and over.
> 
> I still think it's somewhat of a good idea.

I like the idea too, but people want the transaction to return COMMIT
only after data has been fsync'ed so I don't see a big win.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Re: Re[2]: Allowing WAL fsync to be done via O_SYNC

Alfred Perlstein <bright@wintelcom.net> — 2001-03-16T15:43:24Z

* Bruce Momjian <pgman@candle.pha.pa.us> [010316 07:11] wrote:
> > > Could anyone consider fork a syncer process to sync data to disk ?
> > > build a shared sync queue, when a daemon process want to do sync after
> > > write() is called, just put a sync request to the queue. this can release
> > > process from blocked on writing as soon as possible. multipile sync
> > > request for one file can be merged when the request is been inserting to
> > > the queue.
> > 
> > I suggested this about a year ago. :)
> > 
> > The problem is that you need that process to potentially open and close
> > many files over and over.
> > 
> > I still think it's somewhat of a good idea.
> 
> I like the idea too, but people want the transaction to return COMMIT
> only after data has been fsync'ed so I don't see a big win.

This isn't simply handing off the sync to this other process, it requires
an ack from the syncer before returning 'COMMIT'.

-- 
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]

Re: Allowing WAL fsync to be done via O_SYNC

Ken Hirsch <kahirsch@bellsouth.net> — 2001-03-16T15:44:49Z

From: "Bruce Momjian" <pgman@candle.pha.pa.us>
> > > Could anyone consider fork a syncer process to sync data to disk ?
> > > build a shared sync queue, when a daemon process want to do sync after
> > > write() is called, just put a sync request to the queue. this can
release
> > > process from blocked on writing as soon as possible. multipile sync
> > > request for one file can be merged when the request is been inserting
to
> > > the queue.
> >
> > I suggested this about a year ago. :)
> >
> > The problem is that you need that process to potentially open and close
> > many files over and over.
> >
> > I still think it's somewhat of a good idea.
>
> I like the idea too, but people want the transaction to return COMMIT
> only after data has been fsync'ed so I don't see a big win.

For a log file on a busy system, this could improve throughput a lot--batch
commit.  You end up with fewer than one fsync() per transaction.

Re: Re[4]: Allowing WAL fsync to be done via O_SYNC

Tom Lane <tgl@sss.pgh.pa.us> — 2001-03-16T16:03:21Z

Alfred Perlstein <bright@wintelcom.net> writes:
>> couldn't the syncer process cache opened files? is there any problem I
>> didn't consider ?

> 1) IPC latency, the amount of time it takes to call fsync will
>    increase by at least two context switches.

> 2) a working set (number of files needed to be fsync'd) that
>    is larger than the amount of files you wish to keep open.

These days we're really only interested in fsync'ing the current WAL
log file, so working set doesn't seem like a problem anymore.  However
context-switch latency is likely to be a big problem.  One thing we'd
definitely need before considering this is to replace the existing
spinlock mechanism with something more efficient.

Vadim has designed the WAL stuff in such a way that a separate
writer/syncer process would be easy to add; in fact it's almost that way
already, in that any backend can write or sync data that's been added
to the queue by any other backend.  The question is whether it'd
actually buy anything to have another process.  Good stuff to experiment
with for 7.2.

			regards, tom lane

Re: Re[4]: Allowing WAL fsync to be done via O_SYNC

Alfred Perlstein <bright@wintelcom.net> — 2001-03-16T16:18:26Z

* Tom Lane <tgl@sss.pgh.pa.us> [010316 08:16] wrote:
> Alfred Perlstein <bright@wintelcom.net> writes:
> >> couldn't the syncer process cache opened files? is there any problem I
> >> didn't consider ?
> 
> > 1) IPC latency, the amount of time it takes to call fsync will
> >    increase by at least two context switches.
> 
> > 2) a working set (number of files needed to be fsync'd) that
> >    is larger than the amount of files you wish to keep open.
> 
> These days we're really only interested in fsync'ing the current WAL
> log file, so working set doesn't seem like a problem anymore.  However
> context-switch latency is likely to be a big problem.  One thing we'd
> definitely need before considering this is to replace the existing
> spinlock mechanism with something more efficient.

What sort of problems are you seeing with the spinlock code?

> Vadim has designed the WAL stuff in such a way that a separate
> writer/syncer process would be easy to add; in fact it's almost that way
> already, in that any backend can write or sync data that's been added
> to the queue by any other backend.  The question is whether it'd
> actually buy anything to have another process.  Good stuff to experiment
> with for 7.2.

The delayed/coallecesed (sp?) fsync looked interesting.

-- 
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]

Re: Re[4]: Allowing WAL fsync to be done via O_SYNC

Tom Lane <tgl@sss.pgh.pa.us> — 2001-03-16T16:23:39Z

Alfred Perlstein <bright@wintelcom.net> writes:
>> definitely need before considering this is to replace the existing
>> spinlock mechanism with something more efficient.

> What sort of problems are you seeing with the spinlock code?

It's great as long as you never block, but it sucks for making things
wait, because the wait interval will be some multiple of 10 msec rather
than just the time till the lock comes free.

We've speculated about using Posix semaphores instead, on platforms
where those are available.  I think Bruce was concerned about the
possible overhead of pulling in a whole thread-support library just to
get semaphores, however.

			regards, tom lane

Re: Re[4]: Allowing WAL fsync to be done via O_SYNC

Marc G. Fournier <scrappy@hub.org> — 2001-03-16T17:10:34Z

On Fri, 16 Mar 2001, Tom Lane wrote:

> Alfred Perlstein <bright@wintelcom.net> writes:
> >> definitely need before considering this is to replace the existing
> >> spinlock mechanism with something more efficient.
>
> > What sort of problems are you seeing with the spinlock code?
>
> It's great as long as you never block, but it sucks for making things
> wait, because the wait interval will be some multiple of 10 msec rather
> than just the time till the lock comes free.
>
> We've speculated about using Posix semaphores instead, on platforms
> where those are available.  I think Bruce was concerned about the
> possible overhead of pulling in a whole thread-support library just to
> get semaphores, however.

But, with shared libraries, are you really pulling in a "whole
thread-support library"?  My understanding of shared libraries (altho it
may be totally off) was that instead of pulling in a whole library, you
pulled in the bits that you needed, pretty much as you needed them ...

Re: Re[4]: Allowing WAL fsync to be done via O_SYNC

Doug McNaught <doug@wireboard.com> — 2001-03-16T17:17:38Z

Tom Lane <tgl@sss.pgh.pa.us> writes:

> Alfred Perlstein <bright@wintelcom.net> writes:
> >> definitely need before considering this is to replace the existing
> >> spinlock mechanism with something more efficient.
> 
> > What sort of problems are you seeing with the spinlock code?
> 
> It's great as long as you never block, but it sucks for making things
> wait, because the wait interval will be some multiple of 10 msec rather
> than just the time till the lock comes free.

Plus, using select() for the timeout is putting you into the kernel
multiple times in a short period, and causing a reschedule everytime,
which is a big lose.  This was discussed in the linux-kernel thread
that was referred to a few days ago.

> We've speculated about using Posix semaphores instead, on platforms
> where those are available.  I think Bruce was concerned about the
> possible overhead of pulling in a whole thread-support library just to
> get semaphores, however.

Are Posix semaphores faster by definition than SysV semaphores (which
are described as "slow" in the source comments)?  I can't see how
they'd be much faster unless locking/unlocking an uncontended
semaphore avoids a system call, in which case you might run into the
same problems with userland backoff...

Just looked, and on Linux pthreads and POSIX semaphores are both
already in the C library.  Unfortunately, the Linux C library doesn't
support the PROCESS_SHARED attribute for either pthreads mutexes or
POSIX semaphores.  Grumble.  What's the point then?

Just some ignorant ramblings, thanks for listening...

-Doug

Re: Re[4]: Allowing WAL fsync to be done via O_SYNC

Larry Rosenman <ler@lerctr.org> — 2001-03-16T17:23:48Z

Yes, you are.  On UnixWare, you need to add -Kthread, which CHANGES a LOT 
of primitives to go through threads wrappers and scheduling.

See the doc on the http://UW7DOC.SCO.COM or http://www.lerctr.org:457/ 
web pages.

Also, some functions are NOT available without the -Kthread or -Kpthread 
directives. 

LER

>>>>>>>>>>>>>>>>>> Original Message <<<<<<<<<<<<<<<<<<

On 3/16/01, 11:10:34 AM, The Hermit Hacker <scrappy@hub.org> wrote 
regarding Re: Re[4]: [HACKERS] Allowing WAL fsync to be done via O_SYNC :


> On Fri, 16 Mar 2001, Tom Lane wrote:

> > Alfred Perlstein <bright@wintelcom.net> writes:
> > >> definitely need before considering this is to replace the existing
> > >> spinlock mechanism with something more efficient.
> >
> > > What sort of problems are you seeing with the spinlock code?
> >
> > It's great as long as you never block, but it sucks for making things
> > wait, because the wait interval will be some multiple of 10 msec rather
> > than just the time till the lock comes free.
> >
> > We've speculated about using Posix semaphores instead, on platforms
> > where those are available.  I think Bruce was concerned about the
> > possible overhead of pulling in a whole thread-support library just to
> > get semaphores, however.

> But, with shared libraries, are you really pulling in a "whole
> thread-support library"?  My understanding of shared libraries (altho it
> may be totally off) was that instead of pulling in a whole library, you
> pulled in the bits that you needed, pretty much as you needed them ...




> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
>     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)

Re: Re[4]: Allowing WAL fsync to be done via O_SYNC

Bruce Momjian <pgman@candle.pha.pa.us> — 2001-03-16T17:34:27Z

[ Charset ISO-8859-1 unsupported, converting... ]
> Yes, you are.  On UnixWare, you need to add -Kthread, which CHANGES a LOT 
> of primitives to go through threads wrappers and scheduling.

This was my concern;  the change that happens on startup and lib calls
when thread support comes in through a library.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Re: Re[4]: Allowing WAL fsync to be done via O_SYNC

Tom Lane <tgl@sss.pgh.pa.us> — 2001-03-16T17:36:12Z

Larry Rosenman <ler@lerctr.org> writes:
>> But, with shared libraries, are you really pulling in a "whole
>> thread-support library"?

> Yes, you are.  On UnixWare, you need to add -Kthread, which CHANGES a LOT 
> of primitives to go through threads wrappers and scheduling.

Right, it's not so much that we care about referencing another shlib,
it's that -lpthreads may cause you to get a whole new thread-aware
version of libc, with attendant overhead that we don't need or want.

			regards, tom lane

Re: Re[4]: Allowing WAL fsync to be done via O_SYNC

Alfred Perlstein <bright@wintelcom.net> — 2001-03-16T20:15:58Z

> On 3/16/01, 11:10:34 AM, The Hermit Hacker <scrappy@hub.org> wrote 
> regarding Re: Re[4]: [HACKERS] Allowing WAL fsync to be done via O_SYNC :
> 
> > But, with shared libraries, are you really pulling in a "whole
> > thread-support library"?  My understanding of shared libraries (altho it
> > may be totally off) was that instead of pulling in a whole library, you
> > pulled in the bits that you needed, pretty much as you needed them ...


* Larry Rosenman <ler@lerctr.org> [010316 10:02] wrote:
> Yes, you are.  On UnixWare, you need to add -Kthread, which CHANGES a LOT 
> of primitives to go through threads wrappers and scheduling.
> 
> See the doc on the http://UW7DOC.SCO.COM or http://www.lerctr.org:457/ 
> web pages.
> 
> Also, some functions are NOT available without the -Kthread or -Kpthread 
> directives. 

This is true on FreeBSD as well.

-- 
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]

Re: Allowing WAL fsync to be done via O_SYNC

William K. Volkman <wkv@hiscorp.net> — 2001-03-17T01:01:39Z

The Hermit Hacker wrote:
>> 
> But, with shared libraries, are you really pulling in a "whole
> thread-support library"?  My understanding of shared libraries (altho it
> may be totally off) was that instead of pulling in a whole library, you
> pulled in the bits that you needed, pretty much as you needed them ...

Just by making a thread call libc changes personality to use thread
safe routines (I.E. add mutex locking).  Use one thread feature, get
the whole set...which may not be that bad.
-- 
William K. Volkman.
CIO - H.I.S. Financial Services Corporation.
102 S. Tejon, Ste. 920, Colorado Springs, CO 80903
Phone: 719-633-6942  Fax: 719-633-7006  Cell: 719-330-8423

Re: Allowing WAL fsync to be done via O_SYNC

Alfred Perlstein <bright@wintelcom.net> — 2001-03-18T20:03:28Z

* William K. Volkman <wkv@hiscorp.net> [010318 11:56] wrote:
> The Hermit Hacker wrote:
> >> 
> > But, with shared libraries, are you really pulling in a "whole
> > thread-support library"?  My understanding of shared libraries (altho it
> > may be totally off) was that instead of pulling in a whole library, you
> > pulled in the bits that you needed, pretty much as you needed them ...
> 
> Just by making a thread call libc changes personality to use thread
> safe routines (I.E. add mutex locking).  Use one thread feature, get
> the whole set...which may not be that bad.

Actually it can be pretty bad.  Locked bus cycles needed for mutex
operations are very, very expensive, not something you want to do
unless you really really need to do it.

-- 
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]

Re: Allowing WAL fsync to be done via O_SYNC

Tom Lane <tgl@sss.pgh.pa.us> — 2001-03-18T20:52:03Z

Alfred Perlstein <bright@wintelcom.net> writes:
>> Just by making a thread call libc changes personality to use thread
>> safe routines (I.E. add mutex locking).  Use one thread feature, get
>> the whole set...which may not be that bad.

> Actually it can be pretty bad.  Locked bus cycles needed for mutex
> operations are very, very expensive, not something you want to do
> unless you really really need to do it.

It'd be interesting to try to get some numbers about the actual cost
of using a thread-aware libc, on platforms where there's a difference.
Shouldn't be that hard to build a postgres executable with the proper
library and run some benchmarks ... anyone care to try?

			regards, tom lane

Re: Allowing WAL fsync to be done via O_SYNC

Larry Rosenman <ler@lerctr.org> — 2001-03-18T22:15:06Z

* Tom Lane <tgl@sss.pgh.pa.us> [010318 14:55]:
> Alfred Perlstein <bright@wintelcom.net> writes:
> >> Just by making a thread call libc changes personality to use thread
> >> safe routines (I.E. add mutex locking).  Use one thread feature, get
> >> the whole set...which may not be that bad.
> 
> > Actually it can be pretty bad.  Locked bus cycles needed for mutex
> > operations are very, very expensive, not something you want to do
> > unless you really really need to do it.
> 
> It'd be interesting to try to get some numbers about the actual cost
> of using a thread-aware libc, on platforms where there's a difference.
> Shouldn't be that hard to build a postgres executable with the proper
> library and run some benchmarks ... anyone care to try?
I can get the code compiled, but don't have the skills to generate
a test case worthy of anything....

LER

> 
> 			regards, tom lane
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749

Re: Allowing WAL fsync to be done via O_SYNC

Alfred Perlstein <bright@wintelcom.net> — 2001-03-18T22:48:31Z

* Larry Rosenman <ler@lerctr.org> [010318 14:17] wrote:
> * Tom Lane <tgl@sss.pgh.pa.us> [010318 14:55]:
> > Alfred Perlstein <bright@wintelcom.net> writes:
> > >> Just by making a thread call libc changes personality to use thread
> > >> safe routines (I.E. add mutex locking).  Use one thread feature, get
> > >> the whole set...which may not be that bad.
> > 
> > > Actually it can be pretty bad.  Locked bus cycles needed for mutex
> > > operations are very, very expensive, not something you want to do
> > > unless you really really need to do it.
> > 
> > It'd be interesting to try to get some numbers about the actual cost
> > of using a thread-aware libc, on platforms where there's a difference.
> > Shouldn't be that hard to build a postgres executable with the proper
> > library and run some benchmarks ... anyone care to try?
> I can get the code compiled, but don't have the skills to generate
> a test case worthy of anything....

There's a 'make test' or something ('regression' maybe?) target that
runs a suite of tests on the database, you could use that as a
bench/timer, you could also try mysql's "crashme" script.

-- 
-Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org]

Re: Allowing WAL fsync to be done via O_SYNC

Tom Lane <tgl@sss.pgh.pa.us> — 2001-03-18T23:47:22Z

Larry Rosenman <ler@lerctr.org> writes:
> I can get the code compiled, but don't have the skills to generate
> a test case worthy of anything....

contrib/pgbench would do as a first cut.

			regards, tom lane

Re: Allowing WAL fsync to be done via O_SYNC

Bruce Momjian <pgman@candle.pha.pa.us> — 2001-03-19T14:29:01Z

> * William K. Volkman <wkv@hiscorp.net> [010318 11:56] wrote:
> > The Hermit Hacker wrote:
> > >> 
> > > But, with shared libraries, are you really pulling in a "whole
> > > thread-support library"?  My understanding of shared libraries (altho it
> > > may be totally off) was that instead of pulling in a whole library, you
> > > pulled in the bits that you needed, pretty much as you needed them ...
> > 
> > Just by making a thread call libc changes personality to use thread
> > safe routines (I.E. add mutex locking).  Use one thread feature, get
> > the whole set...which may not be that bad.
> 
> Actually it can be pretty bad.  Locked bus cycles needed for mutex
> operations are very, very expensive, not something you want to do
> unless you really really need to do it.

And don't forget buggy implementations.

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026