Split psql's lexer into two separate .l files for SQL and backslash cases.
Tom Lane <tgl@sss.pgh.pa.us>
Split psql's lexer into two separate .l files for SQL and backslash cases. This gets us to a point where psqlscan.l can be used by other frontend programs for the same purpose psql uses it for, ie to detect when it's collected a complete SQL command from input that is divided across line boundaries. Moreover, other programs can supply their own lexers for backslash commands of their own choosing. A follow-on patch will use this in pgbench. The end result here is roughly the same as in Kyotaro Horiguchi's 0001-Make-SQL-parser-part-of-psqlscan-independent-from-ps.patch, although the details of the method for switching between lexers are quite different. Basically, in this patch we share the entire PsqlScanState, YY_BUFFER_STATE stack, *and* yyscan_t between different lexers. The only thing we need to do to switch to a different lexer is to make sure the start_state is valid for the new lexer. This works because flex doesn't keep any other persistent state that depends on the specific lexing tables generated for a particular .l file. (We are assuming that both lexers are built with the same flex version, or at least versions that are compatible with respect to the contents of yyscan_t; but that doesn't seem likely to be a big problem in practice, considering how slowly flex changes.) Aside from being more efficient than Horiguchi-san's original solution, this avoids possible corner-case changes in semantics: the original code was capable of popping the input buffer stack while still staying in backslash-related parsing states. I'm not sure that that equates to any useful user-visible behaviors, but I'm not sure it doesn't either, so I'm loath to assume that we only need to consider the topmost buffer when parsing a backslash command. I've attempted to update the MSVC build scripts for the added .l file, but will rely on the buildfarm to see if I missed anything. Kyotaro Horiguchi and Tom Lane
Files
| Path | Change | +/− |
|---|---|---|
| src/bin/psql/command.c | modified | +1 −1 |
| src/bin/psql/.gitignore | modified | +1 −0 |
| src/bin/psql/Makefile | modified | +10 −5 |
| src/bin/psql/nls.mk | modified | +2 −1 |
| src/bin/psql/psqlscan.h | modified | +2 −20 |
| src/bin/psql/psqlscan_int.h | added | +129 −0 |
| src/bin/psql/psqlscan.l | modified | +87 −726 |
| src/bin/psql/psqlscanslash.h | added | +35 −0 |
| src/bin/psql/psqlscanslash.l | added | +735 −0 |
| src/bin/psql/variables.c | modified | +1 −1 |
| src/tools/msvc/clean.bat | modified | +1 −0 |
| src/tools/msvc/Mkvcbuild.pm | modified | +1 −1 |