Thread

Insert records in the tavke only if they are not exist

Igor Korot <ikorot01@gmail.com> — 2024-12-07T03:36:40Z

Hi, All,

When my application starts up, ot creates some tables and insert records in
them.

When the app starts for the second time it should check if the tables and
the records in them are exist and skip the process.

Everything is good, except what if I have a connection from 2 different
users?

I can run this inside transaction, but will this be enough? Will stating
transaction lock the DB and the second user will wait for transaction to
complete?

Thank you.

Re: Insert records in the tavke only if they are not exist

David G. Johnston <david.g.johnston@gmail.com> — 2024-12-07T03:55:00Z

On Friday, December 6, 2024, Igor Korot <ikorot01@gmail.com> wrote:
>
> When my application starts up, ot creates some tables and insert records
> in them.
>
> When the app starts for the second time it should check if the tables and
> the records in them are exist and skip the process.
>
> Everything is good, except what if I have a connection from 2 different
> users?
>
> I can run this inside transaction, but will this be enough? Will stating
> transaction lock the DB and the second user will wait for transaction to
> complete?
>
>
Seems like letting the create table fail would be a reliable way to
determine what is happening.  But this procedure overall just seems better
avoided.  Can’t you just run an installer/updater separate from running the
application?

David J.

Re: Insert records in the tavke only if they are not exist

Igor Korot <ikorot01@gmail.com> — 2024-12-07T04:02:22Z

Hi, Dvid,

On Fri, Dec 6, 2024, 9:55 PM David G. Johnston <david.g.johnston@gmail.com>
wrote:

> On Friday, December 6, 2024, Igor Korot <ikorot01@gmail.com> wrote:
>>
>> When my application starts up, ot creates some tables and insert records
>> in them.
>>
>> When the app starts for the second time it should check if the tables and
>> the records in them are exist and skip the process.
>>
>> Everything is good, except what if I have a connection from 2 different
>> users?
>>
>> I can run this inside transaction, but will this be enough? Will stating
>> transaction lock the DB and the second user will wait for transaction to
>> complete?
>>
>>
> Seems like letting the create table fail would be a reliable way to
> determine what is happening.  But this procedure overall just seems better
> avoided.  Can’t you just run an installer/updater separate from running the
> application?
>

Interesting idea bout the installer.
Except those tables will contain some additional info about the schema and
the application.

So if I create/remove new table, either in my app or from psql the record
will need to be inserted/deleted.

Thank you.


> David J.
>
>

Re: Insert records in the tavke only if they are not exist

Adrian Klaver <adrian.klaver@aklaver.com> — 2024-12-07T17:18:48Z

On 12/6/24 19:36, Igor Korot wrote:
> Hi, All,
> 
> When my application starts up, ot creates some tables and insert records 
> in them.
> 
> When the app starts for the second time it should check if the tables 
> and the records in them are exist and skip the process.

Does this also apply to starts after the second time?

Is the data expected to change over time?

> 
> Everything is good, except what if I have a connection from 2 different 
> users?
Is it possible to have the app start up without external users connecting?

> 
> I can run this inside transaction, but will this be enough? Will stating 
> transaction lock the DB and the second user will wait for transaction to 
> complete?
> 
> Thank you.
> 

-- 
Adrian Klaver
adrian.klaver@aklaver.com

Re: Insert records in the tavke only if they are not exist

Igor Korot <ikorot01@gmail.com> — 2024-12-07T17:59:56Z

Hi, Adrian,

On Sat, Dec 7, 2024 at 11:18 AM Adrian Klaver <adrian.klaver@aklaver.com> wrote:
>
> On 12/6/24 19:36, Igor Korot wrote:
> > Hi, All,
> >
> > When my application starts up, ot creates some tables and insert records
> > in them.
> >
> > When the app starts for the second time it should check if the tables
> > and the records in them are exist and skip the process.
>
> Does this also apply to starts after the second time?

Yes, it does.

However, if the new table is created in the meantime - the new record
should be created, because it is not there yet.

>
> Is the data expected to change over time?

Data change is possible.

This is one of the tables:

    "CREATE TABLE IF NOT EXISTS \"sys.abcatfmt\"(\"abf_name\" char(30)
NOT NULL, \"abf_frmt\" char(254), \"abf_type\" smallint, \"abf_cntr\"
integer" ));";
    "CREATE UNIQUE INDEX IF NOT EXISTS pbcatf_x ON
\"sys.abcatfmt\"(\"abf_name\" ASC);";

My understanding is that "INSERT OR IGNORE" will check the unique
index and will not do anything if the recrd
with such data on the index already exists.

>
> >
> > Everything is good, except what if I have a connection from 2 different
> > users?
> Is it possible to have the app start up without external users connecting?

Not sure what you mean here.

Are you asking if the computer can run the app?
Then the answer is NO. App is always started by the user.
Please clarify.

Thank you.

>
> >
> > I can run this inside transaction, but will this be enough? Will stating
> > transaction lock the DB and the second user will wait for transaction to
> > complete?
> >
> > Thank you.
> >
>
> --
> Adrian Klaver
> adrian.klaver@aklaver.com
>

Re: Insert records in the tavke only if they are not exist

Adrian Klaver <adrian.klaver@aklaver.com> — 2024-12-07T18:32:23Z

On 12/7/24 09:59, Igor Korot wrote:
> Hi, Adrian,
> 
> On Sat, Dec 7, 2024 at 11:18 AM Adrian Klaver <adrian.klaver@aklaver.com> wrote:
>>
>> On 12/6/24 19:36, Igor Korot wrote:
>>> Hi, All,
>>>
>>> When my application starts up, ot creates some tables and insert records
>>> in them.
>>>
>>> When the app starts for the second time it should check if the tables
>>> and the records in them are exist and skip the process.
>>
>> Does this also apply to starts after the second time?
> 
> Yes, it does.
> 
> However, if the new table is created in the meantime - the new record
> should be created, because it is not there yet.

What determines that a new table needs to be created and populated?

> 
>>
>> Is the data expected to change over time?
> 
> Data change is possible.
> 
> This is one of the tables:
> 
>      "CREATE TABLE IF NOT EXISTS \"sys.abcatfmt\"(\"abf_name\" char(30)
> NOT NULL, \"abf_frmt\" char(254), \"abf_type\" smallint, \"abf_cntr\"
> integer" ));";
>      "CREATE UNIQUE INDEX IF NOT EXISTS pbcatf_x ON
> \"sys.abcatfmt\"(\"abf_name\" ASC);";
> 
> My understanding is that "INSERT OR IGNORE" will check the unique

It is INSERT ... ON CONFLICT DO NOTHING | UPDATE.

> index and will not do anything if the recrd
> with such data on the index already exists.

So you have the option of either skipping the insert or updating 
selected fields in the row.

> 
>>
>>>
>>> Everything is good, except what if I have a connection from 2 different
>>> users?
>> Is it possible to have the app start up without external users connecting?
> 
> Not sure what you mean here.
> 
> Are you asking if the computer can run the app?
> Then the answer is NO. App is always started by the user.
> Please clarify.

This is going to need a more detailed explanation of what 'app' means.

Are you talking about the front end that the user launches or the 
backend that runs the database or something else?


> 
> Thank you.
> 
>>
>>>
>>> I can run this inside transaction, but will this be enough? Will stating
>>> transaction lock the DB and the second user will wait for transaction to
>>> complete?
>>>
>>> Thank you.
>>>
>>
>> --
>> Adrian Klaver
>> adrian.klaver@aklaver.com
>>

-- 
Adrian Klaver
adrian.klaver@aklaver.com

Re: Insert records in the tavke only if they are not exist

Igor Korot <ikorot01@gmail.com> — 2024-12-07T20:17:35Z

Hi, Adrian,

On Sat, Dec 7, 2024 at 12:32 PM Adrian Klaver <adrian.klaver@aklaver.com> wrote:
>
> On 12/7/24 09:59, Igor Korot wrote:
> > Hi, Adrian,
> >
> > On Sat, Dec 7, 2024 at 11:18 AM Adrian Klaver <adrian.klaver@aklaver.com> wrote:
> >>
> >> On 12/6/24 19:36, Igor Korot wrote:
> >>> Hi, All,
> >>>
> >>> When my application starts up, ot creates some tables and insert records
> >>> in them.
> >>>
> >>> When the app starts for the second time it should check if the tables
> >>> and the records in them are exist and skip the process.
> >>
> >> Does this also apply to starts after the second time?
> >
> > Yes, it does.
> >
> > However, if the new table is created in the meantime - the new record
> > should be created, because it is not there yet.
>
> What determines that a new table needs to be created and populated?

Consider the following scenario:

1. Program is installed.
2. Program is started for the first time.
3. My tables are created and populated
4. Program is executed.
5. User closes the program.
6.. Later on the user decides that there is a need
for another table (inside psql or any other client)
7. Then the program starts for the second time.

At this time all my tables that were created will stay (courtesy
of CREATE TABLE IF NOT EXIST).
All records that were there are staying unchanged.
However, for the table that was made in between the runs
will be added

Now if the program is installed on 2 different machines
and started simultaneously on both - I want to ensure that
only 1 set of tables is made and only 1 set of records in them
is available

Now, the creation/population is done inside a transaction.

>
> >
> >>
> >> Is the data expected to change over time?
> >
> > Data change is possible.
> >
> > This is one of the tables:
> >
> >      "CREATE TABLE IF NOT EXISTS \"sys.abcatfmt\"(\"abf_name\" char(30)
> > NOT NULL, \"abf_frmt\" char(254), \"abf_type\" smallint, \"abf_cntr\"
> > integer" ));";
> >      "CREATE UNIQUE INDEX IF NOT EXISTS pbcatf_x ON
> > \"sys.abcatfmt\"(\"abf_name\" ASC);";
> >
> > My understanding is that "INSERT OR IGNORE" will check the unique
>
> It is INSERT ... ON CONFLICT DO NOTHING | UPDATE.
>
> > index and will not do anything if the recrd
> > with such data on the index already exists.

Good.
I'm using INSERT ON CONFLICT DO NOTHING.

>
> So you have the option of either skipping the insert or updating
> selected fields in the row.
>
> >
> >>
> >>>
> >>> Everything is good, except what if I have a connection from 2 different
> >>> users?
> >> Is it possible to have the app start up without external users connecting?
> >
> > Not sure what you mean here.
> >
> > Are you asking if the computer can run the app?
> > Then the answer is NO. App is always started by the user.
> > Please clarify.
>
> This is going to need a more detailed explanation of what 'app' means.
>
> Are you talking about the front end that the user launches or the
> backend that runs the database or something else?

Front-end.
This is a C++ app.

>
>
> >
> > Thank you.
> >
> >>
> >>>
> >>> I can run this inside transaction, but will this be enough? Will stating
> >>> transaction lock the DB and the second user will wait for transaction to
> >>> complete?
> >>>
> >>> Thank you.
> >>>
> >>
> >> --
> >> Adrian Klaver
> >> adrian.klaver@aklaver.com
> >>
>
> --
> Adrian Klaver
> adrian.klaver@aklaver.com
>

Re: Insert records in the tavke only if they are not exist

Adrian Klaver <adrian.klaver@aklaver.com> — 2024-12-07T23:07:22Z

On 12/7/24 12:17, Igor Korot wrote:
> Hi, Adrian,
> 
> On Sat, Dec 7, 2024 at 12:32 PM Adrian Klaver <adrian.klaver@aklaver.com> wrote:
>>
>> On 12/7/24 09:59, Igor Korot wrote:
>>> Hi, Adrian,
>>>
>>> On Sat, Dec 7, 2024 at 11:18 AM Adrian Klaver <adrian.klaver@aklaver.com> wrote:
>>>>
>>>> On 12/6/24 19:36, Igor Korot wrote:
>>>>> Hi, All,
>>>>>
>>>>> When my application starts up, ot creates some tables and insert records
>>>>> in them.
>>>>>
>>>>> When the app starts for the second time it should check if the tables
>>>>> and the records in them are exist and skip the process.
>>>>
>>>> Does this also apply to starts after the second time?
>>>
>>> Yes, it does.
>>>
>>> However, if the new table is created in the meantime - the new record
>>> should be created, because it is not there yet.
>>
>> What determines that a new table needs to be created and populated?
> 
> Consider the following scenario:

I did and it made me start twitching.

> 
> 1. Program is installed.
> 2. Program is started for the first time.
> 3. My tables are created and populated

What distinguishes your tables from other users tables?

> 4. Program is executed.
> 5. User closes the program.
> 6.. Later on the user decides that there is a need
> for another table (inside psql or any other client)

The above is where I started twitching.

How do you keep them out of your tables?

How do you get these changes to play nice with the existing structure?

> 7. Then the program starts for the second time.
> 
> At this time all my tables that were created will stay (courtesy
> of CREATE TABLE IF NOT EXIST).

You are depending on folks not knowing about DROP TABLE and/or you 
having thought out the permissions for access thoroughly.

> All records that were there are staying unchanged.

INSERT/UPDATE against your tables is not a possibility?

> However, for the table that was made in between the runs
> will be added
> 
> Now if the program is installed on 2 different machines
> and started simultaneously on both - I want to ensure that
> only 1 set of tables is made and only 1 set of records in them
> is available

I assume this means they are both pointing at the same instance of a 
database?

This is the part that confuses me.

If you are going to allow ad hoc and at will changes how do you know 
what is actually the correct change?

>  
> Now, the creation/population is done inside a transaction.

I'm not sure that a transaction is going to solve the issue I raised 
above, it will just make one thing happen with no guarantee that it is 
the correct outcome.

>> Are you talking about the front end that the user launches or the
>> backend that runs the database or something else?
> 
> Front-end.
> This is a C++ app.

To me this is the tail wagging the dog. The thought of allowing users to 
change the database structure and you dealing with it after the fact is 
just disturbing to me.

> 
>>
>>
>>>
>>> Thank you.

-- 
Adrian Klaver
adrian.klaver@aklaver.com

Re: Insert records in the tavke only if they are not exist

Igor Korot <ikorot01@gmail.com> — 2024-12-07T23:58:39Z

Hi, Adrian,

On Sat, Dec 7, 2024 at 5:07 PM Adrian Klaver <adrian.klaver@aklaver.com> wrote:
>
> On 12/7/24 12:17, Igor Korot wrote:
> > Hi, Adrian,
> >
> > On Sat, Dec 7, 2024 at 12:32 PM Adrian Klaver <adrian.klaver@aklaver.com> wrote:
> >>
> >> On 12/7/24 09:59, Igor Korot wrote:
> >>> Hi, Adrian,
> >>>
> >>> On Sat, Dec 7, 2024 at 11:18 AM Adrian Klaver <adrian.klaver@aklaver.com> wrote:
> >>>>
> >>>> On 12/6/24 19:36, Igor Korot wrote:
> >>>>> Hi, All,
> >>>>>
> >>>>> When my application starts up, ot creates some tables and insert records
> >>>>> in them.
> >>>>>
> >>>>> When the app starts for the second time it should check if the tables
> >>>>> and the records in them are exist and skip the process.
> >>>>
> >>>> Does this also apply to starts after the second time?
> >>>
> >>> Yes, it does.
> >>>
> >>> However, if the new table is created in the meantime - the new record
> >>> should be created, because it is not there yet.
> >>
> >> What determines that a new table needs to be created and populated?
> >
> > Consider the following scenario:
>
> I did and it made me start twitching.

Good..

>
> >
> > 1. Program is installed.
> > 2. Program is started for the first time.
> > 3. My tables are created and populated
>
> What distinguishes your tables from other users' tables?

I am making the tables and naming them with a specific pattern.

Now I'm curious - if I start psql and will want to create a table
named pg_am, what will happen?
I presume psql will produce error saying that the system table with
that name exists and the user can't create ome.
Am I right?

>
> > 4. Program is executed.
> > 5. User closes the program.
> > 6.. Later on the user decides that there is a need
> > for another table (inside psql or any other client)
>
> The above is where I started twitching.
>
> How do you keep them out of your tables?

My tables are named with the specific pattern.

Also - see above.

>
> How do you get these changes to play nice with the existing structure?

Again - not sure what you mean here....

>
> > 7. Then the program starts for the second time.
> >
> > At this time all my tables that were created will stay (courtesy
> > of CREATE TABLE IF NOT EXIST).
>
> You are depending on folks not knowing about DROP TABLE and/or you
> having thought out the permissions for access thoroughly.

If one of my tables  will be dropped - it will be re-created.
I'm using CREATE TABLE IF NOT EXIST.

>
>
> > All records that were there are staying unchanged.
>
> INSERT/UPDATE against your tables is not a possibility?

It is..
All I;m saying that when the app starts-up, this is done automatically

And if the user decides to insert some data - that's on him

>
> > However, for the table that was made in between the runs
> > will be added
> >
> > Now if the program is installed on 2 different machines
> > and started simultaneously on both - I want to ensure that
> > only 1 set of tables is made and only 1 set of records in them
> > is available
>
> I assume this means they are both pointing at the same instance of a
> database?

Correct.

>
> This is the part that confuses me.
>
> If you are going to allow ad hoc and at will changes how do you know
> what is actually the correct change?

I don't.
Think about MS ACCESS-like applications.
ACCESS creates the internal tables to keep track of some
internals.
Is there a possibility of a clash with the user table? Ofc there is.
But MS is still doing it nevertheless, because chances of this occurring
are really slim.
And if a user acquire access to those internal tables and start modify
them and in the process screw something up, well he uses MS product
as a developer and so should know better as a developer
And if MS can do it - why can't I?


>
> >
> > Now, the creation/population is done inside a transaction.
>
> I'm not sure that a transaction is going to solve the issue I raised
> above, it will just make one thing happen with no guarantee that it is
> the correct outcome.

What is considered correct outcome is strongly on user if we are talking
about my app.

>
>
> >> Are you talking about the front end that the user launches or the
> >> backend that runs the database or something else?
> >
> > Front-end.
> > This is a C++ app.
>
> To me this is the tail wagging the dog. The thought of allowing users to
> change the database structure and you dealing with it after the fact is
> just disturbing to me.

But not to me.
My application targets developers, not end users.

Thank you.

>
> >
> >>
> >>
> >>>
> >>> Thank you.
>
> --
> Adrian Klaver
> adrian.klaver@aklaver.com
>

Re: Insert records in the tavke only if they are not exist

Rob Sargent <robjsargent@gmail.com> — 2024-12-08T00:23:57Z


> But not to me.
> My application targets developers, not end users.
>
> Thank you.

What does your app enable developers to do?

Re: Insert records in the tavke only if they are not exist

David G. Johnston <david.g.johnston@gmail.com> — 2024-12-08T00:40:19Z

On Sat, Dec 7, 2024 at 4:59 PM Igor Korot <ikorot01@gmail.com> wrote:

>
> I am making the tables and naming them with a specific pattern.
>
> Now I'm curious - if I start psql and will want to create a table
> named pg_am, what will happen?
> I presume psql will produce error saying that the system table with
> that name exists and the user can't create ome.
> Am I right?
>
>
You are wasting people's time asking a question like this when it takes but
seconds to experiment.

Most of what you are doing should be considered something to test out and
observe the behavior yourself.  Formalize it as a test if possible if you
rely heavily on it so you can be aware if a bug were to be introduced that
broke such a behavior, or even just to get lots of exposure to concurrency
if you are designing a system especially reliant on specific concurrent
behavior.

As it stands the methodology you describe is quite non-traditional for a
relational client-server database system.

David J.

Re: Insert records in the tavke only if they are not exist

Adrian Klaver <adrian.klaver@aklaver.com> — 2024-12-08T00:49:50Z

On 12/7/24 15:58, Igor Korot wrote:
> Hi, Adrian,
> 

>> What distinguishes your tables from other users' tables?
> 
> I am making the tables and naming them with a specific pattern.
> 
> Now I'm curious - if I start psql and will want to create a table
> named pg_am, what will happen?
> I presume psql will produce error saying that the system table with
> that name exists and the user can't create ome.
> Am I right?

When you refer to psql are you talking about the CLI program or the 
Postgres server in general?

As to table  names it depends. A schema is a namespace so if you put the 
table in it's own schema then the server will not complain:

create table test_sch.pg_am(id integer);
CREATE TABLE

  \d *.pg_am
                Table "pg_catalog.pg_am"
   Column   |  Type   | Collation | Nullable | Default
-----------+---------+-----------+----------+---------
  oid       | oid     |           | not null |
  amname    | name    |           | not null |
  amhandler | regproc |           | not null |
  amtype    | "char"  |           | not null |
Indexes:
     "pg_am_oid_index" PRIMARY KEY, btree (oid)
     "pg_am_name_index" UNIQUE CONSTRAINT, btree (amname)

               Table "test_sch.pg_am"
  Column |  Type   | Collation | Nullable | Default
--------+---------+-----------+----------+---------
  id     | integer |           |          |


I would strongly advise against this naming schema for the above reason.


> 
>>
>>> 4. Program is executed.
>>> 5. User closes the program.
>>> 6.. Later on the user decides that there is a need
>>> for another table (inside psql or any other client)
>>
>> The above is where I started twitching.
>>
>> How do you keep them out of your tables?
> 
> My tables are named with the specific pattern.
> 
> Also - see above.

Yes, but if you allow users into the database:

a) They can figure out what that pattern is.

b) Mess with the tables.

> 
>>
>> How do you get these changes to play nice with the existing structure?
> 
> Again - not sure what you mean here....

Examples:

1) A user creates table that creates a FK relationship to one of 'your' 
tables and you then change 'your' table and invalidate that relationship.

2) A user changes the structure of 'your' tables.


> If one of my tables  will be dropped - it will be re-created.
> I'm using CREATE TABLE IF NOT EXIST.

This means every time the app is run it could potentially run one or 
more schema/data migrations.

How are you going to track the state of the database in order to get it 
back to what it was when the change or changes occurred?

> 
>>
>>
>>> All records that were there are staying unchanged.
>>
>> INSERT/UPDATE against your tables is not a possibility?
> 
> It is..
> All I;m saying that when the app starts-up, this is done automatically

Again, using what point in time state?

> 
> And if the user decides to insert some data - that's on him

No the point of this thread is it's on you or you would not be asking 
how to restore to a known state.


>>
>> This is the part that confuses me.
>>
>> If you are going to allow ad hoc and at will changes how do you know
>> what is actually the correct change?
> 
> I don't.
> Think about MS ACCESS-like applications.
> ACCESS creates the internal tables to keep track of some
> internals.
> Is there a possibility of a clash with the user table? Ofc there is.
> But MS is still doing it nevertheless, because chances of this occurring
> are really slim.
> And if a user acquire access to those internal tables and start modify
> them and in the process screw something up, well he uses MS product
> as a developer and so should know better as a developer
> And if MS can do it - why can't I?

Because MS has the lawyers to keep at bay anyone complaining that Access 
blew up their application.

> 
> 
>>
>>>
>>> Now, the creation/population is done inside a transaction.
>>
>> I'm not sure that a transaction is going to solve the issue I raised
>> above, it will just make one thing happen with no guarantee that it is
>> the correct outcome.
> 
> What is considered correct outcome is strongly on user if we are talking
> about my app.

Except you are asking how to deal with changes and so you are taking 
ownership of them. And in the case you describe, two different instances 
of the app making changes, how do you decide which one wins?


>> To me this is the tail wagging the dog. The thought of allowing users to
>> change the database structure and you dealing with it after the fact is
>> just disturbing to me.
> 
> But not to me.
> My application targets developers, not end users.

Who are users in your application.

> 
> Thank you.
> 
>>
>>>
>>>>
>>>>
>>>>>
>>>>> Thank you.
>>
>> --
>> Adrian Klaver
>> adrian.klaver@aklaver.com
>>

-- 
Adrian Klaver
adrian.klaver@aklaver.com