Time the Rewrite JSQLParser

[manticore-projects]

Greetings. Hoping to have earned enough credit for pointing out some weak spots of the current implementation I would like to suggest starting a rewrite of the library and to freeze the existing code:

• we still have problems with deep nested structures and have only patched that up, but never solved it
• some of the production (e. g. CREATE TABLE and ALTER TABLE ... ALTER ...) need massive refactoring
• we got a large area of common SQL covered and number of issues has reduced significantly
• we go a very good Test Suite and Documentation, serving as benchmark for any new implementation

So, I would like to propose:

1. freeze the current JSQLParser (except real bug fixes)
2. open a new branch based on a clean SQL:2016 compliant grammar
3. optional: drop support for Maven
4. optional: drop support for the Validation Framework (unless someone volunteers to migrate it)
5. optional: drop support for the redundant toString() and DeParser implementation
6. optional: switch to Java 11 (or even 17 for using multi-line strings)

The shown Grammar has been provided by https://github.com/prestodb/sql and is impressive:

i) it is high performance and does not suffer from deep-nested syntax (I have challenged it with all our performance related benchmarks and it just worked fine). So all our hacks and work-arounds won't be needed and the code becomes so much cleaner
ii) it is SQL:2016 compliant
iii) it has a very elegant and slim Deparser included, preserving Comments and also works out of the box
iv) creates the full AST and Nodes and so allows for simple XSLT based parsing thus reducing boiler plate code
v) I have added already the proven Test and Documentation tools to the build system

Of course, we would need to start adding

• the whole Java Beans Logic again, which will be a lot of work
• the RDBMS specific Grammar

I think the current JSQL Parser is great but also has reached a maturity and end-of-life. It got more and more messy to add features since it is clearly a "grown" solution based on learning by doing.
Although I do not want to create a fork or competing solution but instead continue the teamwork and lift JSQLParser to the next level, where we can combine the best out of both worlds: the experience and samples and references from the current JSQLParser with a modern and high performance approach of a SQL2016 Grammar.

Do let me know what you think and don't hold back please. I can stand the heat.

[wumpz]

1. since we have no real idea how fast a new version is up and running this seems not realistic
2. this seems like a good start, however, since we want to introduce some non-SQL 2016 compliant stuff the grammar first needs to inspected to verify its extensibility, e.g. both bracket variants
   2.1 first adapt base changes, like complete identifier character set (UTF-8 stuff)
   2.2 read your last points just after writing this
3. Drop support for gradle. I don't know why you want to continually drop maven support. My experience with gradle is quite the opposite. Maybe I have not your gradle experience but you got the point.
4. its a pity, that it was not accepted, it was a good idea, but you are right
5. My point of view is still: streamline toString() impl with Deparsers #133.
6. JDK 17 or even JDK 21

Since JSqlParser took willingly all kinds of features its state is getting worse. A more drastic even automatic quality check should be established to fight the entropy. But I have at the moment no idea, how this could be realized.

[manticore-projects]

Thank you a lot for response and consideration.

1. since we have no real idea how fast a new version is up and running this seems not realistic

Actually, I do share your concern.
However, without a freeze we will chase a moving target and I do not know how to enforce full commitment on the new version. Any ideas?

2. this seems like a good start, however, since we want to introduce some non-SQL 2016 compliant stuff  the grammar first needs to inspected to verify its extensibility, e.g. both bracket variants
   2.1 first adapt base changes, like complete identifier character set (UTF-8 stuff)
   2.2 read your last points just after writing this

The Grammar is a Java CC grammar and so ever dirty trick applied to JSQLParser would work there also.
I have added the Unicode stuff already. SquareBracket quotation and Escape token rewrite would be exactly the same.

3. Drop support for gradle. I don't know why you want to continually drop maven support. My experience with gradle is quite the opposite. Maybe I have not your gradle experience but you got the point.

Gradle update version numbers of dependencies and release numbers based on GIT Tags automatically. Furthermore it allows for direct execution of scripts in Java/Groovy DSL.

I find it extremely annoying to do that every time manually in Maven -- but if you want to keep doing that, please be my guest.

4. its a pity, that it was not accepted, it was a good idea, but you are right

I share your sentiment and would like to salvage as much as possible. Maybe we should convert it into its own project, where it can depend on a fixed version of JSQLParser and can be maintained/updated by however has a need or interest.

if you agree, I will create a JSQLValidator project in GitHub (as a draft), which you can fork from.

5. My point of view is still: [streamline toString() impl  with Deparsers #133](https://github.com/JSQLParser/JSqlParser/issues/133).

There is a Deparser implementation, which deparses automatically based on the AST/Nodes.
If you insist in implementing the toString() separately, this can be done of course but it will almost double the effort of the rewrite.

Although I am not against it: very often I read the toString() method for understanding, what the statement or expression is actually supposed to do.

6. JDK 17 or even JDK 21

JDK 21 is not released yet and it does not have any JSQLParser relevant improvements, while MultiLine Strings are big for us.
So JDK17 it should be.

Since JSqlParser took willingly all kinds of features its state is getting worse. A more drastic even automatic quality check should be established to fight the entropy. But I have at the moment no idea, how this could be realized.

Agreed. I started on a very basic Scanner to enforce some API conventions. I do wonder, if there was no established tool around but did not find anything.

Another question was, how we deal with RDBMS specific, not SQL standard compliant stuff.

[wumpz]

Thank you a lot for response and consideration.

1. since we have no real idea how fast a new version is up and running this seems not realistic

Actually, I do share your concern. However, without a freeze we will chase a moving target and I do not know how to enforce full commitment on the new version. Any ideas?

Do you have some kind of estimate? We have to compromise here. So small changes should still go into JSqlParser 4.x.

2. this seems like a good start, however, since we want to introduce some non-SQL 2016 compliant stuff  the grammar first needs to inspected to verify its extensibility, e.g. both bracket variants
   2.1 first adapt base changes, like complete identifier character set (UTF-8 stuff)
   2.2 read your last points just after writing this

The Grammar is a Java CC grammar and so ever dirty trick applied to JSQLParser would work there also. I have added the Unicode stuff already. SquareBracket quotation and Escape token rewrite would be exactly the same.

And we have a lot of those tricks in there. Still, I hope that some improvements are coming into JavaCC (ideas are all over the place).

3. Drop support for gradle. I don't know why you want to continually drop maven support. My experience with gradle is quite the opposite. Maybe I have not your gradle experience but you got the point.

Gradle update version numbers of dependencies and release numbers based on GIT Tags automatically. Furthermore it allows for direct execution of scripts in Java/Groovy DSL.

What does this version number thing do? Why should dependency and release numbers chaged based on GIT Tags and not definined in the build.gradle file? The execution of those scripts is maybe a tool for small projects but not for large multimodule projects. The build will be so hard to maintain.

I find it extremely annoying to do that every time manually in Maven -- but if you want to keep doing that, please be my guest.

4. its a pity, that it was not accepted, it was a good idea, but you are right

I share your sentiment and would like to salvage as much as possible. Maybe we should convert it into its own project, where it can depend on a fixed version of JSQLParser and can be maintained/updated by however has a need or interest.

if you agree, I will create a JSQLValidator project in GitHub (as a draft), which you can fork from.

That seems to be a good idea. Since we are in JSQLParsers own organization (https://github.com/JSQLParser), the new repository should be here. I am thinking right now about the pros and cons of making JSQLParser a multimodule project, but since we do not want to maintain this stuff, I think a separate project should be better.

5. My point of view is still: [streamline toString() impl  with Deparsers #133](https://github.com/JSQLParser/JSqlParser/issues/133).

There is a Deparser implementation, which deparses automatically based on the AST/Nodes. If you insist in implementing the toString() separately, this can be done of course but it will almost double the effort of the rewrite.

Although I am not against it: very often I read the toString() method for understanding, what the statement or expression is actually supposed to do.

6. JDK 17 or even JDK 21

JDK 21 is not released yet and it does not have any JSQLParser relevant improvements, while MultiLine Strings are big for us. So JDK17 it should be.

Since JSqlParser took willingly all kinds of features its state is getting worse. A more drastic even automatic quality check should be established to fight the entropy. But I have at the moment no idea, how this could be realized.

Agreed. I started on a very basic Scanner to enforce some API conventions. I do wonder, if there was no established tool around but did not find anything.

What kind of enforcement are you talking about? Tools to maintain some architectural integrity are, e.g. https://www.archunit.org/ or even the heavy gun https://jqassistant.org/.

Another question was, how we deal with RDBMS specific, not SQL standard compliant stuff.

[manticore-projects]

Hallo Tobias, hoffentlich geht es Dir gut? Noch habe ich Dein Repository nicht kaputt gemacht 😄 Aber wie wollen wir Snapshots pushen? Aus meiner Sicht sollte immer ein Snapshot nach Sonar Type gepusht werden, wenn es einen Commit nach MASTER mit bestandenen Tests/QA checks gegeben hat. Das ist ziemlich einfach per GitHub Actions moeglich, jedoch muesstest Du "Repository Secrets" hinterlegen. Mir fehlen dazu die Rechte, ich brauche es auch nicht wirklich -- es reicht voellig wenn diese Secrets vorhanden sind. Damit kann ich die Publish/Deploy Action bauen. Ich brauche nur den Secret KEY (den Namen des Secrets), nicht den Wert oder das Password, also z. B. ossrhUsername=xxxxx ossrhPassword=xxxxx Sonst haben wir den Nachteil, dass Aenderungen nur als Source, nicht als Binary zur Verfuegung stehen. Viele Gruesse Andreas

…

--=-iy1s+qO0k+yW7iIb0w22 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable <html><head><style>pre,code,address { margin: 0px; } h1,h2,h3,h4,h5,h6 { margin-top: 0.2em; margin-bottom: 0.2em; } ol,ul { margin-top: 0em; margin-bottom: 0em; } blockquote { margin-top: 0em; margin-bottom: 0em; } </style></head><body><div>Hallo Tobias,</div><div><br></div><div>hoffentlic= h geht es Dir gut?</div><div><br></div><div>Noch habe ich Dein Repository n= icht kaputt gemacht =F0=9F=98=84</div><div>Aber wie wollen wir Snapshots pu= shen? Aus meiner Sicht sollte immer ein Snapshot nach Sonar Type gepusht we= rden, wenn es einen Commit nach MASTER mit bestandenen Tests/QA checks gege= ben hat. Das ist ziemlich einfach per GitHub Actions moeglich, jedoch muess= test Du "Repository Secrets" hinterlegen. Mir fehlen dazu die Rechte, ich b= rauche es auch nicht wirklich -- es reicht voellig wenn diese Secrets vorha= nden sind. Damit kann ich die Publish/Deploy Action bauen.</div><div><br></= div><div>Ich brauche nur den Secret KEY (den Namen des Secrets), nicht den = Wert oder das Password, also z. B.</div><div><br></div><div><font color=3D"= #ff0000">ossrhUsername</font>=3Dxxxxx</div><div><font color=3D"#ff0000">oss= rhPassword</font>=3Dxxxxx</div><div><br></div><div>Sonst haben wir den Nach= teil, dass Aenderungen nur als Source, nicht als Binary zur Verfuegung steh= en.</div><div><br></div><div>Viele Gruesse</div><div>Andreas</div><div><spa= n></span></div></body></html>

--=-iy1s+qO0k+yW7iIb0w22--

[wumpz]

Hallo Andreas,   grüß Dich. Hoffe bei Dir läuft soweit alles gut. Ich habe, wie Du sicherlich bemerkt hast, für meine "Freizeit"Repos kaum Zeit. Außer der beruflichen Belastung habe ich hier zwei Pubertiere und noch weiter Baustellen, die mich von anderen Sachen abhalten.    Schön, dass das Repository noch läuft ;).    Ich habe keine Ahnung, wie solche Secrets hinterlegt werden. Muss ich mir mal ansehen. Damit sind aber nur Snapshot - Deployments möglich.   ...   So hab nach Deiner Vorgabe die Secrets hinterlegt: ossrhUsername, ossrhPassword.   Hoffe so geht es weiter.     Gruß   Tobias      Gesendet: Mittwoch, 28. Juni 2023 um 05:32 Uhr Von: "manticore-projects" ***@***.***> An: "JSQLParser/JSqlParser" ***@***.***> Cc: "Tobias" ***@***.***>, "Comment" ***@***.***> Betreff: Re: [JSQLParser/JSqlParser] Time the Rewrite JSQLParser (Discussion #1744)   Hallo Tobias, hoffentlich geht es Dir gut? Noch habe ich Dein Repository nicht kaputt gemacht 😄 Aber wie wollen wir Snapshots pushen? Aus meiner Sicht sollte immer ein Snapshot nach Sonar Type gepusht werden, wenn es einen Commit nach MASTER mit bestandenen Tests/QA checks gegeben hat. Das ist ziemlich einfach per GitHub Actions moeglich, jedoch muesstest Du "Repository Secrets" hinterlegen. Mir fehlen dazu die Rechte, ich brauche es auch nicht wirklich -- es reicht voellig wenn diese Secrets vorhanden sind. Damit kann ich die Publish/Deploy Action bauen. Ich brauche nur den Secret KEY (den Namen des Secrets), nicht den Wert oder das Password, also z. B. ossrhUsername=xxxxx ossrhPassword=xxxxx Sonst haben wir den Nachteil, dass Aenderungen nur als Source, nicht als Binary zur Verfuegung stehen. Viele Gruesse Andreas

…

--=-iy1s+qO0k+yW7iIb0w22 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable <html><head><style>pre,code,address { margin: 0px; } h1,h2,h3,h4,h5,h6 { margin-top: 0.2em; margin-bottom: 0.2em; } ol,ul { margin-top: 0em; margin-bottom: 0em; } blockquote { margin-top: 0em; margin-bottom: 0em; } </style></head><body><div>Hallo Tobias,</div><div><br></div><div>hoffentlic= h geht es Dir gut?</div><div><br></div><div>Noch habe ich Dein Repository n= icht kaputt gemacht =F0=9F=98=84</div><div>Aber wie wollen wir Snapshots pu= shen? Aus meiner Sicht sollte immer ein Snapshot nach Sonar Type gepusht we= rden, wenn es einen Commit nach MASTER mit bestandenen Tests/QA checks gege= ben hat. Das ist ziemlich einfach per GitHub Actions moeglich, jedoch muess= test Du "Repository Secrets" hinterlegen. Mir fehlen dazu die Rechte, ich b= rauche es auch nicht wirklich -- es reicht voellig wenn diese Secrets vorha= nden sind. Damit kann ich die Publish/Deploy Action bauen.</div><div><br></= div><div>Ich brauche nur den Secret KEY (den Namen des Secrets), nicht den = Wert oder das Password, also z. B.</div><div><br></div><div><font color=3D"= #ff0000">ossrhUsername</font>=3Dxxxxx</div><div><font color=3D"#ff0000">oss= rhPassword</font>=3Dxxxxx</div><div><br></div><div>Sonst haben wir den Nach= teil, dass Aenderungen nur als Source, nicht als Binary zur Verfuegung steh= en.</div><div><br></div><div>Viele Gruesse</div><div>Andreas</div><div><spa= n></span></div></body></html>

--=-iy1s+qO0k+yW7iIb0w22-- — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

[manticore-projects]

On Sun, 2023-07-02 at 15:09 -0700, Tobias wrote: So hab nach Deiner Vorgabe die Secrets hinterlegt: ossrhUsername, ossrhPassword. Hoffe so geht es weiter.

Hallo Tobias,  das hat geklappt, danke! Aber ich habe vergessen, dass die Publication auch gezeichnet werden muss. Kannst Du bitte noch 2 weiterer Secrets PGP_SECRET und PGP_PASSPHRASE anlegen. The PGP_PASSPHRASE is the password of the GPG/PGP secret key you created for the ‘publish-java’ demo. You need to use the following to get the value for the PGP_SECRET.  The id is the LONG_ID of the GPG/PGP secret key. When setting up the secret, paste in the whole private key block including the BEGIN and END lines. LONG_ID=4F99EE30B6804AC7516E3EA3DB7CF378BD5A1A0B gpg --armor --export-secret-keys $LONG_ID

-----BEGIN PGP PRIVATE KEY BLOCK ...... -----END PGP PRIVATE KEY BLOCK-----

--=-inK1I74muTjrXFfqjA8b Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable <html><head><style>pre,code,address { margin: 0px; } h1,h2,h3,h4,h5,h6 { margin-top: 0.2em; margin-bottom: 0.2em; } ol,ul { margin-top: 0em; margin-bottom: 0em; } blockquote { margin-top: 0em; margin-bottom: 0em; } </style></head><body><div>On Sun, 2023-07-02 at 15:09 -0700, Tobias wrote:<= /div><blockquote type=3D"cite" style=3D"margin:0 0 0 .8ex; border-left:2px = #729fcf solid;padding-left:1ex"><div>So hab nach Deiner Vorgabe die Secrets= hinterlegt:&nbsp;ossrhUsername,&nbsp;ossrhPassword.</div><div>Hoffe so geh= t es weiter.</div></blockquote><div><br></div><div>Hallo Tobias,&nbsp;</div=

<div><br></div><div>das hat geklappt, danke!</div><div><br></div><div>Aber=

ich habe vergessen, dass die Publication auch gezeichnet werden muss.</div=

<div>Kannst Du bitte noch 2 weiterer Secrets <b>PGP_SECRET</b> und <b>PGP_=

PASSPHRASE</b> anlegen.</div><div><br></div><div>The PGP_PASSPHRASE is the = password of the GPG/PGP secret key you created for the =E2=80=98publish-jav= a=E2=80=99 demo.</div><div><br></div><div>You need to use the following to get the value for the PGP_SECRET.&nbsp;</div>= <div>The id=20 is the LONG_ID of the GPG/PGP secret key. <font color=3D"#ff0000">When sett= ing up the secret,=20 paste in the whole private key block including the BEGIN and END lines.</fo= nt></div><pre class=3D"pn po pp pq pr ps pt pu pv ax pw bj"><span class=3D"= px ol gq pt b iq py pz l jf qa" data-selectable-paragraph=3D""><br></span><= /pre><pre class=3D"pn po pp pq pr ps pt pu pv ax pw bj"><span id=3D"d4a5" c= lass=3D"px ol gq pt b iq py pz l jf qa" data-selectable-paragraph=3D""></sp= an><span id=3D"d4a5" class=3D"px ol gq pt b iq py pz l jf qa" data-selectab= le-paragraph=3D"">LONG_ID=3D4F99EE30B6804AC7516E3EA3DB7CF378BD5A1A0B<br><fo= nt color=3D"#ff0000">gpg --armor --export-secret-keys $LONG_ID</font></span=

</pre><pre class=3D"pn po pp pq pr ps pt pu pv ax pw bj"><span class=3D"px=

ol gq pt b iq qb pz l jf qa" data-selectable-paragraph=3D""><br></span></p= re><pre class=3D"pn po pp pq pr ps pt pu pv ax pw bj"><span id=3D"3d49" cla= ss=3D"px ol gq pt b iq qb pz l jf qa" data-selectable-paragraph=3D""><font = color=3D"#80"><i>-----BEGIN PGP PRIVATE KEY BLOCK</i></font></span></pr= e><pre class=3D"pn po pp pq pr ps pt pu pv ax pw bj"><font color=3D"#80= "><i>......</i></font></pre><pre class=3D"pn po pp pq pr ps pt pu pv ax pw = bj"><span id=3D"f1f1" class=3D"px ol gq pt b iq qb pz l jf qa" data-selecta= ble-paragraph=3D""><font color=3D"#80"><i>-----END PGP PRIVATE KEY BLOC= K-----</i></font></span><span id=3D"f1f1" class=3D"px ol gq pt b iq qb pz l= jf qa" data-selectable-paragraph=3D""></span></pre><div><br></div><div><sp= an></span></div></body></html>

…

--=-inK1I74muTjrXFfqjA8b--