ORDER BY logic in PostgreSQL source code

I'm currently working on a GiST extension for PostgreSQL and I ran into strange ORDER BY results during my queries.

Because I can't find the problem in my source code, I want to investigate the issue by looking at the PostgreSQL source,
maybe inserting extra log messages.
While trying to do this, I came to the conclusion that I need developer help, because I can't find the part of the source code
where the ORDER BY logic is implemented.

Could you please help, by telling me which files should I take a look at?

Thanks in advance!