Remove Groovy from MAL and LAL kernel.

[wu-sheng]

Background

This proposal has been considered for months, starting from @yswdqz 's when the https://github.com/apache/skywalking-graalvm-distro project was established. Groovy script kernel was chosen from MAL design by @hanahmily, mostly considering its convenience and flexibility to change.
But we saw its limitations when we tried to get OAP running Groovy in the Graal.
More importantly, there are 4 years after Groovy kernel was picked and LAL made the same choice considering aligning the tech stack. With the cloud-native becoming the mainstream, we always need to build a new image which means a re-compiling. Groovy's script-based feature is not very useful. Personally, I never saw a real-world use case, to just change the MAL/LAL directly on a product/test environment. Also, this feature is limited by OAP running in a cluster mode, it will never be able to hot-reload as the distributed consistent will be a disaster.

Another Groovy Replacement Story

Another history I could share is that, @yswdqz was trying hard to reimplement MAL from antlr to keep the script unchanged as a transparent switch. But it doesn't go well. From a tech perspective, I think we have a chance to implement this to get rid of Groovy.

But being honest, from the beginning, I am super concerned about @yanbw 's plan about this, because the maintenance of this new engine could be another bad experience. We would never know how users would write the script, a lot of unexpected errors would be reported and hard to debug.

New Proposal

Right now, I am thinking of the necessity to keep the script engine, rather than directly using programming languages.

1. OAP re-compiling is always required because images are preferred, no code or config should be changed to affect the logic. Things like MAL and LAL seem too heavy and too dangerous to change in that way.
2. The time cost of re-compiling is only ~2 mins mostly, it is not a big thing.
3. Native language would provide more clear debugging information and easier to address the bugs.

The only side effect is, that all existing MAL/LAL scripts would have to be rewritten. But it is not hard I think.

About the specific language, we should consider Java or Kotlin. Kotlin could provide a better Lamda style, but it will be a new tech stack.

---

If this proposal gets passed, we could consider this in 11.0 which will make our next 10.0 will be like the old 7.0. No further feature-related feature.

@kezhenxu94 @hanahmily @yswdqz @wankai123 and other @apache/skywalking-committers I want to hear from you about this.

[kezhenxu94]

It seems the new proposal is not very clear to me, what is the new proposal actually? Build the LAL/MAL script into the OAP codebase? What's the proposed usage of the LAL/MAL?

[wu-sheng]

It means MAL and LAL are still pipelines to process metrics and logs. Nothing is changed from the design perspective.
But they are going to implement it in the OAP codebase, rather than a script engine like today.

The simple answer is, to use Kotlin/Java to write MAL/LAL and remove the Groovy engine.
Of course, with that, current scripts are not able to use anymore.

[wu-sheng]

With consideration, we are going(trying) to add DEBUG capabilities without breaking change and tech stack replacement for the short term.

Still, the performance concern, especially for LAL, exists. We may try PoC to verify the difference for the potential replacement by a native Java/Kotlin based log process.