Horton
Menu

Thread

  1. Alter table fast

    veem v <veema0000@gmail.com> — 2025-01-09T16:25:09Z

    Hello,
It's postgres version 16.1, we want to convert an existing column data type
from integer to numeric and it's taking a long time. The size of the table
is ~50GB and the table has ~150million rows in it and it's not partitioned.
We tried running the direct alter and it's going beyond hours, so wanted to
understand from experts what is the best way to achieve this?

1)Should we go with below
Alter table <table_name> alter column <column_name> type numeric(15,0)
USING <column_name>::NUMERIC(15,0);
OR
We should add a new not null column.
update the data in that column from the existing column.
drop the old column
rename the new column to the old column.

Regards
Veem

  2. Re: Alter table fast

    Ron Johnson <ronljohnsonjr@gmail.com> — 2025-01-09T16:27:20Z

    On Thu, Jan 9, 2025 at 11:25 AM veem v <veema0000@gmail.com> wrote:

> Hello,
> It's postgres version 16.1, we want to convert an existing column data
> type from integer to numeric and it's taking a long time. The size of the
> table is ~50GB and the table has ~150million rows in it and it's not
> partitioned. We tried running the direct alter and it's going beyond hours,
> so wanted to understand from experts what is the best way to achieve this?
>
>
Out of curiosity, why NUMERIC(15,0) instead of BIGINT?

-- 
Death to <Redacted>, and butter sauce.
Don't boil me, I'm still alive.
<Redacted> lobster!

  3. Re: Alter table fast

    veem v <veema0000@gmail.com> — 2025-01-09T19:17:14Z

    On Thu, 9 Jan 2025 at 21:57, Ron Johnson <ronljohnsonjr@gmail.com> wrote:

> On Thu, Jan 9, 2025 at 11:25 AM veem v <veema0000@gmail.com> wrote:
>
>> Hello,
>> It's postgres version 16.1, we want to convert an existing column data
>> type from integer to numeric and it's taking a long time. The size of the
>> table is ~50GB and the table has ~150million rows in it and it's not
>> partitioned. We tried running the direct alter and it's going beyond hours,
>> so wanted to understand from experts what is the best way to achieve this?
>>
>>
> Out of curiosity, why NUMERIC(15,0) instead of BIGINT?
>
>
>
It's for aligning the database column types to the data model and it's
happening across all the upstream downstream systems.
I was thinking if this can be made  faster with the single line alter
statement "Alter table <table_name> alter column <column_name> type
numeric(15,0) USING <column_name>::NUMERIC(15,0);" or through the UPDATE
column rename column strategy. Additionally if this can be further improved
using any session level parameter like "max_parallel_workers_per_gather",
"max_parallel_workers", "maintenance_work_mem", "work_mem"?

  4. Re: Alter table fast

    shammat@gmx.net — 2025-01-09T19:52:27Z

    Am 09.01.25 um 20:17 schrieb veem v:
>> Out of curiosity, why NUMERIC(15,0) instead of BIGINT?
>
> It's for aligning the database column types to the data model and
> it's happening across all the upstream downstream systems. I was
> thinking if this can be made  faster with the single line alter
> statement "Alter table <table_name> alter column <column_name> type
> numeric(15,0) USING <column_name>::NUMERIC(15,0);"

Hmm, I would rather change numeric(15,0) to bigint if I had to "align" types across systems.

  5. Re: Alter table fast

    Peter J. Holzer <hjp-pgsql@hjp.at> — 2025-01-12T21:19:01Z

    On 2025-01-09 20:52:27 +0100, shammat@gmx.net wrote:
> Am 09.01.25 um 20:17 schrieb veem v:
> > > Out of curiosity, why NUMERIC(15,0) instead of BIGINT?
> > 
> > It's for aligning the database column types to the data model and
> > it's happening across all the upstream downstream systems. I was
> > thinking if this can be made  faster with the single line alter
> > statement "Alter table <table_name> alter column <column_name> type
> > numeric(15,0) USING <column_name>::NUMERIC(15,0);"
> 
> Hmm, I would rather change numeric(15,0) to bigint if I had to "align" types across systems.

I'm also wondering what "the data model" is.

If I have numeric(15,0) in an abstract data model, that means that I
expect values larger than 99,999,999,999,999 but at most
999,999,999,999,999. That seems to be oddly specific and also somewhat
at odds with reality when until now there apparently haven't been any
values larger than 2,147,483,647. What kind of real world value could
suddenly jump by more than 5 orders of magnitude but certainly not by 7?

A bigint is much less precise (more than 2,147,483,647 but not more
than 9,223,372,036,854,775,807) and therefore more suitable for values
where you don't really know the range.

However, for the problem at hand, I doubt it makes any difference.
Surely converting a few million values takes much less time than
rewriting a 50 GB table and all its indexes.

So there isn't really a faster way to do what Veem wants. There may
however be less disruptive way: He could create a new column with the
new values (which takes at least as long but can be done in the
background) and then switch it over and drop the old column.

        hp

-- 
   _  | Peter J. Holzer    | Story must make more sense than reality.
|_|_) |                    |
| |   | hjp@hjp.at         |    -- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |       challenge!"

  6. Re: Alter table fast

    Marco Torres <mtors25@gmail.com> — 2025-01-12T21:36:10Z

    This is the right approach, Peter J. Holzer, from a well season DBA
perspective "ALTER TABLE working_table
ADD COLUMN B INTEGER ; UPDATE working_table
SET B = A;"

Bare in mind the indexes or existing references to an from other tables and
act accordingly-- define the new and drop the old.

Good luck.

On Sun, Jan 12, 2025, 2:20 PM Peter J. Holzer <hjp-pgsql@hjp.at> wrote:

> On 2025-01-09 20:52:27 +0100, shammat@gmx.net wrote:
> > Am 09.01.25 um 20:17 schrieb veem v:
> > > > Out of curiosity, why NUMERIC(15,0) instead of BIGINT?
> > >
> > > It's for aligning the database column types to the data model and
> > > it's happening across all the upstream downstream systems. I was
> > > thinking if this can be made  faster with the single line alter
> > > statement "Alter table <table_name> alter column <column_name> type
> > > numeric(15,0) USING <column_name>::NUMERIC(15,0);"
> >
> > Hmm, I would rather change numeric(15,0) to bigint if I had to "align"
> types across systems.
>
> I'm also wondering what "the data model" is.
>
> If I have numeric(15,0) in an abstract data model, that means that I
> expect values larger than 99,999,999,999,999 but at most
> 999,999,999,999,999. That seems to be oddly specific and also somewhat
> at odds with reality when until now there apparently haven't been any
> values larger than 2,147,483,647. What kind of real world value could
> suddenly jump by more than 5 orders of magnitude but certainly not by 7?
>
> A bigint is much less precise (more than 2,147,483,647 but not more
> than 9,223,372,036,854,775,807) and therefore more suitable for values
> where you don't really know the range.
>
> However, for the problem at hand, I doubt it makes any difference.
> Surely converting a few million values takes much less time than
> rewriting a 50 GB table and all its indexes.
>
> So there isn't really a faster way to do what Veem wants. There may
> however be less disruptive way: He could create a new column with the
> new values (which takes at least as long but can be done in the
> background) and then switch it over and drop the old column.
>
>         hp
>
> --
>    _  | Peter J. Holzer    | Story must make more sense than reality.
> |_|_) |                    |
> | |   | hjp@hjp.at         |    -- Charles Stross, "Creative writing
> __/   | http://www.hjp.at/ |       challenge!"
>

  7. Re: Alter table fast

    Gus Spier <gus.spier@gmail.com> — 2025-01-28T02:29:24Z

    I don't think I qualify as an expert, but I can nominate a course of action
for you to consider.

Rather than convert an existing column from one data type to another, might
it not be easier to simply recreate the table with the correct data type.
There are  caveats!

You have to consider any referential integrity constraints.
You have to have a LOT of available disk space.
You really want to do this in batches.

If you can swing those issues, you might have a shot.

 1. CREATE TABLE <table_name_new> LIKE <original_table_name>;
2. ALTER <table_name_new> ALTER COLUMN <column_name> TYPE <new_data_type>;
3. INSERT INTO <table_name_new> SELECT <column list w. the target
column_name cast as the new type>
-- recommend you do this in batches
4. DROP TABLE <original_table_name>;
5. ALTER TABLE <table_name_new> RENAME to <original_table_name>

Regards,
Gus

On Thu, Jan 9, 2025 at 11:27 AM Ron Johnson <ronljohnsonjr@gmail.com> wrote:

> On Thu, Jan 9, 2025 at 11:25 AM veem v <veema0000@gmail.com> wrote:
>
>> Hello,
>> It's postgres version 16.1, we want to convert an existing column data
>> type from integer to numeric and it's taking a long time. The size of the
>> table is ~50GB and the table has ~150million rows in it and it's not
>> partitioned. We tried running the direct alter and it's going beyond hours,
>> so wanted to understand from experts what is the best way to achieve this?
>>
>>
> Out of curiosity, why NUMERIC(15,0) instead of BIGINT?
>
> --
> Death to <Redacted>, and butter sauce.
> Don't boil me, I'm still alive.
> <Redacted> lobster!
>