Horton

Re: [WIP]Vertical Clustered Index (columnar store extension) - take2

Japin Li <japinli@hotmail.com>

View thread
From: Japin Li <japinli@hotmail.com>
To: Peter Smith <smithpb2250@gmail.com>
Cc: "pgsql-hackers@lists.postgresql.org" <pgsql-hackers@lists.postgresql.org>, Tomas Vondra <tomas@vondra.me>, "Aya Iwata (Fujitsu)" <iwata.aya@fujitsu.com>, Timur Magomedov <t.magomedov@postgrespro.ru>, shveta malik <shveta.malik@gmail.com>
Date: 2025-07-25T03:57:52Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Mark ItemPointer arguments as const throughout

  2. Reorganize GUC structs

  3. Eliminate XLOG_HEAP2_VISIBLE from vacuum phase III

  4. Update various forward declarations to use typedef

  5. Pathify RHS unique-ification for semijoin planning

  6. Revert "Don't lock partitions pruned by initial pruning"

  7. For inplace update, send nontransactional invalidations.

  8. Remove nearly-unused SizeOfIptrData macro.

On Wed, 23 Jul 2025 at 14:07, Peter Smith <smithpb2250@gmail.com> wrote:
> On Tue, Jul 22, 2025 at 8:12 PM Japin Li <japinli@hotmail.com> wrote:
> ...
>> 1.
>> I encountered another crash while checking VCI's internal relations.
>>
>> rm -rf demo
>> initdb -D demo
>> cat <<EOF >demo/postgresql.auto.conf
>> shared_preload_libraries = 'vci'
>> max_worker_processes = '20'
>> EOF
>>
>> pg_ctl -D demo start
>>
>> cat <<EOF | psql postgres
>> CREATE EXTENSION vci;
>> CREATE TABLE t (id int, info text);
>> CREATE INDEX ON t USING vci (id);
>> SELECT relname FROM pg_class WHERE relname ~ '^vci_*' LIMIT 1 \gset
>> SELECT * FROM :relname;
>> \d+ :relname
>> REFRESH MATERIALIZED VIEW :relname;
>> EOF
>>
>> VCI's definition of internal relations as materialized views lacks the
>> corresponding view bodies.
>>
>> +    /*
>> +     * @see
>> +     * https://www.postgresql.jp/document/9.4/html/catalog-pg-rewrite.html
>> +     */
>> +    new_rel_reltup->relhasrules = true;
>> +
>> +    new_rel->rd_att->tdtypeid = new_type_oid;
>> +
>> +    InsertPgClassTuple(pg_class, new_rel, new_oid, (Datum) 0, (Datum) 0);
>>
>> Given that VCI's internal relations are materialized views, are VCI workers
>> responsible for their periodic refreshment?
>>
>> Or is it by design that users are unable to read the internal relations?
>>
>
> IIUC, those VCI internal relations (implemented as materialized views)
> are entirely managed by VCI logic. Users are not required to be aware
> of them, and they definitely are not meant to tamper with them.
>

Thanks for your explanation!

> The REFRESH that you attempted should have caused a more graceful error, like:
> ERROR:  extension "vci" prohibits this operation on view
> "vci_0000016482_00000_d"
> So, thanks for reporting that the ERROR failed. Investigating...

I'm considering storing this metadata in heap tables, as Citus Columnar [1]
and TimescaleDB [2] also utilize them for some metadata. Is this a sound
approach? I'm wondering if this is a suitable strategy for VCI?

[1] https://github.com/citusdata/citus/blob/main/src/backend/columnar/columnar_metadata.c#L174
[2] https://github.com/timescale/timescaledb/blob/main/src/chunk.c#L151

-- 
Regards,
Japin Li