Horton

Thread

  1. how to make query

    Sachin Kumar <sachinkumaras@gmail.com> — 2021-01-07T10:15:02Z

    Hi Team,

we are using Python Django and Postgresql. Our requirement is to search 10
Million Card data and bring specific records as a result after checking the
record every time.

We have to find data from a 2nd table that is related to the first one
which we are checking using the loop.

For Ex:-
10000 data - loop 10000 times
to find data from the 2nd table.

This makes our query too slower and it gets time out.

-- 

Best Regards,
Sachin Kumar

  2. Re: how to make query

    Philip Semanchuk <philip@americanefficient.com> — 2021-01-07T13:58:56Z

    
> On Jan 7, 2021, at 5:15 AM, Sachin Kumar <sachinkumaras@gmail.com> wrote:
> 
> Hi Team,
> 
> we are using Python Django and Postgresql. Our requirement is to search 10 Million Card data and bring specific records as a result after checking the record every time.
> 
> We have to find data from a 2nd table that is related to the first one which we are checking using the loop.
> 
> For Ex:- 
> 10000 data - loop 10000 times 
> to find data from the 2nd table. 
> 
> This makes our query too slower and it gets time out.

Hi Sachin,
I’m not sure based on your description, but it sounds like you need to tell Django to select related objects all in one query rather than one at a time. If that’s the case, then your problem is a Django issue, not a Postgres one. Check out select_related() --

https://docs.djangoproject.com/en/3.1/ref/models/querysets/#django.db.models.query.QuerySet.select_related


Cheers
Philip

  3. Re: how to make query

    Sachin Kumar <sachinkumaras@gmail.com> — 2021-01-07T14:25:14Z

    Hi Philip,

I am using Python Django and PostgreSQL to build our report app.

I am using below mention query to find some details. the query mention
below is running fine in PostgreSQL command line and giving result in 2 sc
for 2 million cards but when i i am try the same in Django application it
is taking too much of time and the web page is expired while running this
query.

can anyone help how to make query faster in Django? Or help me in
correcting this query to be used in Django

rows = M_Assign.objects.raw(''' select C.id
<http://s.bl-1.com/h/dck3nM6g?url=http://c.id/>,M."merchant_name",C."MERCHANT_ID",S."store_id"
Owning_Store_ID ,S."store_name" Owning_Store_Name,F."franchisee_name"
Owning_Group,C."ACCOUNT_NUMBER",C."STORE_ID",C."GIFT_LIST_REF" FROM
vdaccount_card_assign C INNER JOIN vd_merchant_master M ON
C."MERCHANT_ID"=M."merchant_id" AND C."MERCHANT_ID"='003561002966107' INNER
JOIN vd_store_master S ON S."store_id"=C."GIFT_LIST_REF" OR
C."GIFT_LIST_REF"='' INNER JOIN vd_franchisee F ON
S."franchisee_id"=F."franchisee_id" '''

On Thu, Jan 7, 2021 at 7:28 PM Philip Semanchuk <
philip@americanefficient.com> wrote:

>
>
> > On Jan 7, 2021, at 5:15 AM, Sachin Kumar <sachinkumaras@gmail.com>
> wrote:
> >
> > Hi Team,
> >
> > we are using Python Django and Postgresql. Our requirement is to search
> 10 Million Card data and bring specific records as a result after checking
> the record every time.
> >
> > We have to find data from a 2nd table that is related to the first one
> which we are checking using the loop.
> >
> > For Ex:-
> > 10000 data - loop 10000 times
> > to find data from the 2nd table.
> >
> > This makes our query too slower and it gets time out.
>
> Hi Sachin,
> I’m not sure based on your description, but it sounds like you need to
> tell Django to select related objects all in one query rather than one at a
> time. If that’s the case, then your problem is a Django issue, not a
> Postgres one. Check out select_related() --
>
>
> https://docs.djangoproject.com/en/3.1/ref/models/querysets/#django.db.models.query.QuerySet.select_related
>
>
> Cheers
> Philip



-- 

Best Regards,
Sachin Kumar

  4. Re: how to make query

    Steve Midgley <science@misuse.org> — 2021-01-07T16:50:13Z

    On Thu, Jan 7, 2021 at 6:26 AM Sachin Kumar <sachinkumaras@gmail.com> wrote:

> Hi Philip,
>
> I am using Python Django and PostgreSQL to build our report app.
>
> I am using below mention query to find some details. the query mention
> below is running fine in PostgreSQL command line and giving result in 2 sc
> for 2 million cards but when i i am try the same in Django application it
> is taking too much of time and the web page is expired while running this
> query.
>
> can anyone help how to make query faster in Django? Or help me in
> correcting this query to be used in Django
>
> rows = M_Assign.objects.raw(''' select C.id
> <http://s.bl-1.com/h/dck3nM6g?url=http://c.id/>,M."merchant_name",C."MERCHANT_ID",S."store_id"
> Owning_Store_ID ,S."store_name" Owning_Store_Name,F."franchisee_name"
> Owning_Group,C."ACCOUNT_NUMBER",C."STORE_ID",C."GIFT_LIST_REF" FROM
> vdaccount_card_assign C INNER JOIN vd_merchant_master M ON
> C."MERCHANT_ID"=M."merchant_id" AND C."MERCHANT_ID"='003561002966107' INNER
> JOIN vd_store_master S ON S."store_id"=C."GIFT_LIST_REF" OR
> C."GIFT_LIST_REF"='' INNER JOIN vd_franchisee F ON
> S."franchisee_id"=F."franchisee_id" '''
>
> On Thu, Jan 7, 2021 at 7:28 PM Philip Semanchuk <
> philip@americanefficient.com> wrote:
>
>>
>>
>> > On Jan 7, 2021, at 5:15 AM, Sachin Kumar <sachinkumaras@gmail.com>
>> wrote:
>> >
>> > Hi Team,
>> >
>> > we are using Python Django and Postgresql. Our requirement is to search
>> 10 Million Card data and bring specific records as a result after checking
>> the record every time.
>> >
>> > We have to find data from a 2nd table that is related to the first one
>> which we are checking using the loop.
>> >
>> > For Ex:-
>> > 10000 data - loop 10000 times
>> > to find data from the 2nd table.
>> >
>> > This makes our query too slower and it gets time out.
>>
>> Hi Sachin,
>> I’m not sure based on your description, but it sounds like you need to
>> tell Django to select related objects all in one query rather than one at a
>> time. If that’s the case, then your problem is a Django issue, not a
>> Postgres one. Check out select_related() --
>>
>>
>> https://docs.djangoproject.com/en/3.1/ref/models/querysets/#django.db.models.query.QuerySet.select_related
>>
>>
>> Cheers
>> Philip
>
>
>
Hi Sachin,

As a general matter, people on this list will be able to help you more
effectively if you provide some DDL, ideally some sample data, and an
"explain analyze" report from your current setup, along with the query SQL.
If you can simplify your table structures in the DDL to focus only on the
place where you are having performance problems, that's even better.

Best,
Steve

  5. Re: how to make query

    Philip Semanchuk <philip@americanefficient.com> — 2021-01-07T17:05:29Z

    
> On Jan 7, 2021, at 9:25 AM, Sachin Kumar <sachinkumaras@gmail.com> wrote:
> 
> Hi Philip,
> 
> I am using Python Django and PostgreSQL to build our report app.
> 
> I am using below mention query to find some details. the query mention below is running fine in PostgreSQL command line and giving result in 2 sc for 2 million cards but when i i am try the same in Django application it is taking too much of time and the web page is expired while running this query.
> 
> can anyone help how to make query faster in Django? Or help me in correcting this query to be used in Django 


Sachin,
If your query runs well in Postgres, then your problem is with Django, not Postgres. You would be better off asking on a Django mailing list how to get Django to construct the query you want.




> 
> On Thu, Jan 7, 2021 at 7:28 PM Philip Semanchuk <philip@americanefficient.com> wrote:
> 
> 
> > On Jan 7, 2021, at 5:15 AM, Sachin Kumar <sachinkumaras@gmail.com> wrote:
> > 
> > Hi Team,
> > 
> > we are using Python Django and Postgresql. Our requirement is to search 10 Million Card data and bring specific records as a result after checking the record every time.
> > 
> > We have to find data from a 2nd table that is related to the first one which we are checking using the loop.
> > 
> > For Ex:- 
> > 10000 data - loop 10000 times 
> > to find data from the 2nd table. 
> > 
> > This makes our query too slower and it gets time out.
> 
> Hi Sachin,
> I’m not sure based on your description, but it sounds like you need to tell Django to select related objects all in one query rather than one at a time. If that’s the case, then your problem is a Django issue, not a Postgres one. Check out select_related() --
> 
> https://docs.djangoproject.com/en/3.1/ref/models/querysets/#django.db.models.query.QuerySet.select_related
> 
> 
> Cheers
> Philip
> 
> 
> -- 
> 
> Best Regards, 
> Sachin Kumar