Re: [WIP]Vertical Clustered Index (columnar store extension) - take2

Peter Smith <smithpb2250@gmail.com>

From: Peter Smith <smithpb2250@gmail.com>
To: Tomas Vondra <tomas@vondra.me>
Cc: "Aya Iwata (Fujitsu)" <iwata.aya@fujitsu.com>, "pgsql-hackers@lists.postgresql.org" <pgsql-hackers@lists.postgresql.org>
Date: 2025-05-29T08:08:20Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →
  1. Mark ItemPointer arguments as const throughout

  2. Reorganize GUC structs

  3. Eliminate XLOG_HEAP2_VISIBLE from vacuum phase III

  4. Update various forward declarations to use typedef

  5. Pathify RHS unique-ification for semijoin planning

  6. Revert "Don't lock partitions pruned by initial pruning"

  7. For inplace update, send nontransactional invalidations.

  8. Remove nearly-unused SizeOfIptrData macro.

Attachments

Hi Tomas.

> What I really miss in the current patch is some sort of READMEs with
>
> - high-level design of the VCI indexes
>
> - description of the consistency/visibility model (does it behave the
> same way as querying the heap, can it be out of sync for, ...)
>
> - WOS/ROS architecture (when are rows inserted into WOS, promoted into
> ROS, what triggers that, ...)
>
> - what's the in-memory / on-disk format
>
> - some places in the patch mention "internal VCI tables" but I have no
> idea what that is
>
> - how does the execution work? compression? crucial points to consider
> for optimal performance, etc.
>
> - limitations (temporary - can be relaxed in the future, permanent -
> inherent to the columnar design) and trade-offs
>
> - what are the various custom scan executor nodes
>
> - what "background" processes happen (custom workers, ...)
>
> - anything else substantial for understanding the design
>
> Maybe there's some of this in the 0002 patch, but I haven't stumbled
> over it so far.

Acknowledged. We are working now on creating a draft README and will
post it here ASAP.

~~~

For patch 0001 you reported some items as being "unused". You were
correct on all counts; those are all removed in the latest patch set.

> gen_node_support.c
> ------------------
> - What's SmcAllocSetContext for? Doesn't seem to be mentioned anywhere.

Removed.

> params.c
> ------------------
> - ExecInitParam_hook seems unused, no?

Removed.

> lwlock.c
> ------------------
> - Vci_lwlock_kind seems unused.

Removed.

> memnodes.h
> ------------------
> - What's SmcAllocSetContext? It's not defined anywhere.

Removed.

> params.h
> ------------------
> - Adds noNeedLoadFromMain, loadedFromMain to ParamExecData, but it
> doesn't seem to be used anywhere (in either patch).

Removed.

> lwlock.h
> ------------------
> - LWLockKind / Vci_lwlock_kind seems unused

Removed

======
Kind Regards,
Peter Smith.
Fujitsu Australia.