Horton

Re: Add SPLIT PARTITION/MERGE PARTITIONS commands

Robert Haas <robertmhaas@gmail.com>

View thread
From: Robert Haas <robertmhaas@gmail.com>
To: Dmitry Koval <d.koval@postgrespro.ru>
Cc: Alexander Korotkov <aekorotkov@gmail.com>, pgsql-hackers@lists.postgresql.org
Date: 2024-08-28T13:45:36Z
Lists: pgsql-hackers

Commits

Same data as JSON: GET /api/v1/messages/:b64id/commits the thread's linked commits as JSON, with link sources. API reference →

  1. Adjust errcode in checkPartition()

  2. Fix usage of palloc() in MERGE/SPLIT PARTITION(s) code

  3. Implement ALTER TABLE ... SPLIT PARTITION ... command

  4. Implement ALTER TABLE ... MERGE PARTITIONS ... command

  5. Calculate agglevelsup correctly when Aggref contains a CTE.

  6. Use PqMsg_* macros in applyparallelworker.c.

  7. Restrict psql meta-commands in plain-text dumps.

  8. Ensure we have a snapshot when updating various system catalogs.

  9. Use specific collation where needed in new test

  10. Expand virtual generated columns in the planner

  11. Virtual generated columns

  12. Define PG_LOGICAL_DIR for path pg_logical/ in data folder

  13. Revert support for ALTER TABLE ... MERGE/SPLIT PARTITION(S) commands

  14. Avoid repeated table name lookups in createPartitionTable()

  15. Provide deterministic order for catalog queries in partition_split.sql

  16. Don't copy extended statistics during MERGE/SPLIT partition operations

  17. Fix the name collision detection in MERGE/SPLIT partition operations

  18. Fix regression tests conflict in 3ca43dbbb6

  19. Add permission check for MERGE/SPLIT partition operations

  20. Fix one more portability shortcoming in new test_pg_dump test.

  21. Inherit parent's AM for partition MERGE/SPLIT operations

  22. Add tab completion for partition MERGE/SPLIT operations

  23. Rename tables in tests of partition MERGE/SPLIT operations

  24. Make new partitions with parent's persistence during MERGE/SPLIT

  25. Document the way partition MERGE/SPLIT operations create new partitions

  26. Change the way ATExecMergePartitions() handles the name collision

  27. Grammar fixes for split/merge partitions code

  28. Checks for ALTER TABLE ... SPLIT/MERGE PARTITIONS ... commands

  29. Fix some grammer errors from error messages and codes comments

  30. Support TZ and OF format codes in to_timestamp().

  31. Support identity columns in partitioned tables

  32. Fix indentation in twophase.c

  33. Fix corner-case planner failure for MERGE.

  34. Doc: fix documentation example for bytea hex output format.

  35. Avoid repeated name lookups during table and index DDL.

On Tue, Aug 27, 2024 at 2:24 PM Dmitry Koval <d.koval@postgrespro.ru> wrote:
> They contains changes from reverted commits 1adf16b8fb, 87c21bb941, and
> subsequent fixes and improvements including df64c81ca9, c99ef1811a,
> 9dfcac8e15, 885742b9f8, 842c9b2705, fcf80c5d5f, 96c7381c4c, f4fc7cb54b,
> 60ae37a8bc, 259c96fa8f, 449cdcd486, 3ca43dbbb6, 2a679ae94e, 3a82c689fd,
> fbd4321fd5, d53a4286d7, c086896625, 4e5d6c4091.
> I didn't include fix 04158e7fa3 into patches because Robert Haas
> objected to its use.

To be clear, I'm not against 04158e7fa3. I just don't think it fixes everything.

> 1. Function createPartitionTable() should be rewritten using partitioned
> table OID (not name) and without using ProcessUtility().

Agree.

> 2. Should it be considered an error when we split a partition owned by
> another user and get partitions that owned by our user?
> (I think this is not a problem. Perhaps disallow merging other users'
> partitions would be too strict a restriction.)
>
> 3. About the functional index "create index on foo (run_me(a));".
> (Should we disallow merging of another user's partitions when
> partitioned table has functional indexes? SECURITY_RESTRICTED_OPERATION?)
>
> 4. Need to decide what is correct in case there are per-partition
> constraints or triggers on a split partition. They not duplicated to the
> new partitions now. (But might be in this case we should have an error
> or warning?)

I think we want to avoid giving errors or warnings.  For all of these
cases, and others, we need to consider what the expected behavior is,
and have test cases and documentation as appropriate. But we shouldn't
think of it as "let's make it fail if the user does something that's
not safe" but rather "let's figure out how to make it safe."

> 5. "If we're merging partitions, wouldn't it be better to adjust the
> constraints on the first partition - or perhaps the largest partition if
> we want to be clever -- and insert the data from all of the others into
> it? Maybe that would even have syntax that puts the user in control of
> which partition survives, e.g. ALTER TABLE tab1 MERGE PARTITION part1
> WITH part2, part3, .... That would also make it really obvious to the
> user what all of the properties of part1 will be after the merge: they
> will be exactly the same as they were before the merge, except that the
> partition constraint will have been adjusted."
> (Similar optimization was proposed in [3] but was rejected [4]).

Interesting. Maybe it would be a good idea to set up some test cases
to see which approach is better in different cases. Like try moving
data from foo1 to foo2 with DELETE..INSERT vs. creating a new table
with CTAS from foo1 UNION ALL foo2 and then indexing it. I think
Alexander has a good point there, but I think my point is good too so
I'm not sure which way wins.

-- 
Robert Haas
EDB: http://www.enterprisedb.com