Modernize Xpect

[cdietrich]

Xpect still uses the old xtext workflow (see what new Xtext project wizard generates) and the old xtext generator.(xpand based)
this is cause of
https://github.com/eclipse/Xpect/blob/018800b3af074c402b2aa53174e2457ea6f48046/org.eclipse.xpect/src/org/eclipse/xpect/GenerateXpect.mwe2#L124C6-L124C6
which uses
https://github.com/meysholdt/xtext_stateful_lexer/blob/48a4201ea3e0e52743af553ffaea89ba2d79144c/de.itemis.statefullexer/src/de/itemis/statefullexer/StatefulLexerFragment.xtend#L22

• this needs to be rewritten on base of
  https://github.com/eclipse/xtext/blob/main/org.eclipse.xtext.xtext.generator/src/org/eclipse/xtext/xtext/generator/parser/antlr/ex/ExternalAntlrLexerFragment.java or by an adjusted version of that

we should move the new fragment to a new plugin in the xpect project.
we also need to make sure content assist and highlighting lexer get put to the ide project.

once this is done we can move in xpect

• to a newer minimal Xtext version (ExternalAntlrLexerFragment above requires at lease 2.20), i propose a much newer version
• the new xtext workflow (also create an ide project), compare generated code, specially the runtime and ui moduels for diffs
• thus get rid of xpand and dependency to xtexts for removal old generator
• then we can change the pom to get rid of the fornax plugin to execute the workflow which is also no longer maintained (see also what new Xtext project wizard generates)
• we can and should also use newer tycho version (and get rid of outdated tycho 1.7/2.x stuff)
• and with that have better separation between compile time and runtime deps (put dep. to xtext.xtext.generator to build.properties, e.g. as new xtext project wizard generates it)
• simplifiy target file setup in tycho as in xtext
• Optional: repair oomph
• Check if we can fix some Xtext and Java deprecations

also the workflows in the examples need to be modernized, but that should be straight forward.
we also should get of deprecated xtext junit4 there

maybe we also should create a 0.2.0 branch with the current state and bump master to 0.3.0 then

in an alternative approach we would update target platform, xtext+java version, workflow, tycho and pom first and use fragment adapter (see history of /org.eclipse.xtext.ui.codetemplates/src/org/eclipse/xtext/ui/codetemplates/GenerateCodetemplateLanguages.mwe2) to call old workflow and then create the new fragment.
in this case we could make sure the "new dependencies" work first before introducing the changed code.

regardings versions:

• Java 11 + Eclipse 2022-03 + Xtext 2.31
• Java 17 + Eclipse 2023-06 + Xtext 2.31 (or 2.32 when it is already out) (as we will have to drop 11 Support in Xtext anyway one day i propose to go this way)

[merks]

Wow, this doesn't sound fun at all. 😱

[merks]

These are the dependencies I see:

[cdietrich]

yes. and xpand is not maintained since 2016

[cdietrich]

see also
eclipse-xtext/xtext#2676

[cdietrich]

This will also fix #300

[cdietrich]

with merging the complete logic from stateful lexer to modernizeXpect branch (see #322)
all dependencies to xpand, stateful lexer, icu and org.eclipse.xtext.generator (the old one) should be removable

[merks]

That sounds very promising!

[cdietrich]

many thanks for your contribution @tjeske

[cdietrich]

open tasks

• do another integ test round at sponsor of this feature
• wait for community feedback (asked in xtext forum and on mailing lists)
• suppresswarnings for generated lexer generate supresswarnings for stateful lexers #326
• fix the adapter warnings and use the common pattern of <T> T getAdapter( ... Class<T>) and someclass.cast(someObject) fix adapter warnings #327
• check other warnings

[cdietrich]

will close this for now. open points can be handled separately
many thanks for your efforts @tjeske