Re: hackers-digest V1 #771 (safe/fast I/O)

mmap() is cool since it avoids copying data between kernel and user address
spaces.  However, mmap() is going to be either synchronous ("won't return 'til
it has set up the page table stuff and maybe allocated backing store") or not
("will return immediately but your process will silently block if you try to
access the address range before the back office work is done for the region").
There is no callback facility and no way to poll for region readiness.

aio_*() is cool since you can queue a read or write and then either get a
callback when it's complete or poll it.  However, there's no way to allocate
the backing store before you start scribbling, so there is always a copy on
aio_write().  And there's no page flipping in aio_read()'s definition, so
unless you allocate your read buffers in page boundaries and unless your 
kernel is really smart, you're always going to see a copy in aio_read().

O_ASYNC and select() are only useful for externally synchronized I/O like
TTY and network.  select() always returns both readable and writable for
either files in a file system or for block or character special disk files.

As far as I know, other than on the MASSCOMP (which more or less did what
VMS did and what Win/NT now does in this area), no UNIX system, especially
including POSIX.1B systems, has quite what's wanted for high performance
transactional I/O.

True asynchrony means having the ability to choose when to block, and to
parallize computation with I/O, and to get more total work done per unit time
by doing application level seek ordering and write buffering (avoiding excess
mechanical movement).  In the last I/O intensive system I helped build here,
we decided that mmap(), even with its periodic time losses, gave us better
total throughput due to the lack of copy overhead.  It helps if you both mmap
things with a lot of regionality, and access them with high locality of
reference.  But it was the savings of memory bus bandwidth that bought us
the most.

#ifndef BUFFER_H
#define BUFFER_H

#include <stdio.h>
#include "misc.h"

#define	BUF_SIZE		4096

typedef struct buffer {
	void *			opaque;
} buffer;

typedef enum bufprot {
	buf_ro,
	buf_rw
	/* Note that there is no buf_wo since RMW is the processor standard. */
} bufprot;

int		buf_init(int nmax, int grow);
int		buf_shutdown(FILE *);
int		buf_get(buffer *);
int		buf_mget(buffer *, int, off_t, bufprot);
int		buf_refcount(buffer);
void		buf_ref(buffer);
void		buf_unref(buffer);
void		buf_clear(buffer);
void		buf_add(buffer, size_t);
void		buf_sub(buffer, size_t);
void		buf_shift(buffer, size_t);
size_t		buf_used(buffer);
size_t		buf_avail(buffer);
void *		buf_used_ptr(buffer);
void *		buf_avail_ptr(buffer);
struct iovec	buf_used_iov(buffer);
struct iovec	buf_avail_iov(buffer);
region		buf_used_reg(buffer);
region		buf_avail_reg(buffer);
int		buf_printf(buffer, const char *, ...);

#endif /* !BUFFER_H */