Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

On Fri, Jun 18, 2021 at 12:18 AM Andres Freund <andres@anarazel.de> wrote:

> Hi,
>
> On 2021-06-17 14:22:52 -0400, Robert Haas wrote:
> > On Thu, Jun 17, 2021 at 2:17 PM Andres Freund <andres@anarazel.de>
> wrote:
> > > Adding a hacky special case implementation for cross-database relation
> > > accesses that violates all kinds of assumptions (like holding a lock on
> > > a relation when accessing it / pinning pages, processing relcache
> > > invals, ...) doesn't seem like a good plan.
> >
> > I agree that we don't want hacky code that violates assumptions, but
> > bypassing shared_buffers is a bit hacky, too. Can't we lock the
> > relations as we're copying them? We know pg_class's OID a fortiori,
> > and we can find out all the other OIDs as we go.


> We possibly can - but I'm not sure that won't end up violating some
> other assumptions.
>

Yeah, we can surely lock the relation as described by Robert, but IMHO,
while creating the database we are already holding the exclusive lock on
the database and there is no one else allowed to be connected to the
database, so do we actually need to bother about the lock for the
correctness?


> > I'm just thinking that the hackiness of going around shared_buffers
> > feels irreducible, but maybe the hackiness in the patch is something
> > that can be solved with more engineering.
>
> Which bypassing of shared buffers are you talking about here? We'd still
> have to solve a subset of the issues around locking (at least on the
> source side), but I don't think we need to read pg_class contents to be
> able to go through shared_buffers? As I suggested, we can use the init
> fork presence to infer relpersistence?
>

I believe we want to avoid scanning pg_class for identifying the relation
list so that we can avoid this special-purpose code?  IMHO, scanning the
disk, such as going through all the tablespaces and then finding the source
database directory and identifying each relfilenode, also appears to be a
special-purpose code, unless I am missing what you mean by special-purpose
code.

Or do you mean that looking at the filesystem at all is bypassing shared
> buffers?
>

I think we already have such a code in multiple places where we bypass the
shared buffers for copying the relation
e.g. index_copy_data(), heapam_relation_copy_data().  So the current system
as it stands, we can not claim that we are designing something for the
first time where we are bypassing the shared buffers.   So if we are
thinking that bypassing the shared buffers is hackish and we don't want to
use it for the new patches then lets avoid it completely even while
identifying the relfilenodes to be copied.

-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com