Improve type-safety of the TokenBuilder's options

[aabounegm]

It now optionally accepts a generic parameter that is a type union of the names of the available terminals of the given language. It should generally be keyof typeof MyLanguageTerminals imported from './generated/ast.js'.

Usage:

import { MyLanguageTerminals } from './generated/ast.js';

type MyTerminals = keyof typeof MyLanguageTerminals;

export const MyLanguageModule: Module<MyLanguageServices, PartialLangiumServices & MyLanguageAddedServices> = {
    parser: {
        TokenBuilder: () => new IndentationAwareTokenBuilder<MyTerminals>({
            indentTokenName: '' // <-- Now this will require that the given string is actually the name of a token defined in the grammar
                                // and IntelliSense can suggest real token names
        }),
        Lexer: (services) => new IndentationAwareLexer(services),
    },
};

If the generic parameter is not passed, if falls back to the previous behaviour of simply using strings.

[aabounegm]

Hmm, I see that NoInfer was added in TS 5.4, but Langium currently uses 5.1.6. Is there a chance it will be updated soon or should I remove NoInfer for now?
Note that the IntelliSense suggestions can become misleading without it. For example:

new IndentationAwareTokenBuilder({
     indentTokenName: 'INDENT',
     dedentTokenName: '' // <-- Here it will suggest only "INDENT"
                         // (though it wouldn't mark other inputs as erroneous)
})

[msujew]

Feel free to update to TypeScript 5.4 as part of this PR. Aside from that the change looks great, thank you!

[aabounegm]

Great, done. I'd like to note that I attempted updating to the latest version (5.5), but it gave the following errors (since it now does some RegExp validation), so I thought I'd use 5.4 instead for now and this issue can be addressed separately:

packages/langium/test/documentation/jsdoc.test.ts:173:78 - error TS1501: This regular expression flag is only available when targeting 'es2018' or later.

173                     const [paramName, description] = contentMd.split(/\s(.*)/s);
                                                                                 ~

packages/langium/test/parser/token-builder.test.ts:181:44 - error TS1501: This regular expression flag is only available when targeting 'es2018' or later.

181         expect(tokenA.PATTERN).toEqual(/A/is);
                                               ~

[aabounegm]

I'd also like to add that it could be useful to expose the alias type MyTerminals = keyof typeof MyLanguageTerminals; directly from ./generated/ast.js. I didn't explore the generator source code, but from a quick look I assume it would just be a matter of adding:

export type ${config.projectName}TerminalNames = keyof typeof ${config.projectName}Terminals;

to packages/langium-cli/src/generator/ast-generator.ts:243, correct?
Is that something desirable?

[msujew]

I think that reasonable, yes. Since this is only a type definition, it doesn't even affect bundle size :)

[aabounegm]

Hmm, I did not expect the side-effect this had on the tests. Any suggestions for how to best fix it?

[aabounegm]

@msujew I'm not sure if you would count that as a fix or a hack, but I noticed that all terminal rules use the name "A" so I just added it to the prefix and it worked 😃

[msujew]

I've updated the tests a bit. This now looks good to me, thank you!