Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add simple Module System for "include" #1076

Merged
merged 21 commits into from
Aug 11, 2019

Conversation

madoar
Copy link
Collaborator

@madoar madoar commented Jul 29, 2019

This PR changes a few scripts to support the new module system for include.
This PR functions as a preview for discussion purposes.

This PR requires PhoenicisOrg/phoenicis#2050

@@ -1,5 +1,5 @@
include("engines.wine.quick_script.steam_script");
include("utils.functions.filesystem.files");
const {ls, mkdir, fileExists, cat, cp, getFileSize, fileName, lns, remove, touch, writeToFile, createTempFile, createTempDir, chmod, Checksum} = include("utils.functions.filesystem.files");
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This allows us to only "include" the functions we actually require.
Currently I include all defined functionality for simplicity reasons but we can change that later on.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For me, it's not obvious why anybody should know that functions are exported in this order. Also: what happens if you add a function in the middle?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The order is not important. You can add the "included" functions in any order you want to

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't you have to change all includes then?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you mean? What I'm doing here is simple destructuring (see also https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Destructuring_assignment).

So according to my understanding it is ok to use

const {x, y} = include("a");

in script "X" and

const {y, z} = include("a");

in script "Y"

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my understanding, x is the first exported function, y the second etc. So if you add a new function between x and y to the module "a", y will suddenly be the new function.

Copy link
Collaborator Author

@madoar madoar Jul 30, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you are confusing this with array destructuring. What I'm doing here is object destructuring. In object destructuring the keys are matched not the indices.

You can test this quite easy yourself in node. Try executing the following code in node:

const x = {a: "a", b: "b", c: "c"};
// destructure "x"
const {c, a} = x;
// print c and a
console.log(a);
console.log(c);

The result will be:

a
c

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, then it's fine.

}
};

module.ls = ls;
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternatively we could export the functions via:

module.ls = function ls(...) {...};

I didn't do this here because some functions depend on each other which is problematic because if they are defined using module.x = function x(...) {...}; the method also need to be accessed as module.x from inside the same script.

@madoar
Copy link
Collaborator Author

madoar commented Jul 29, 2019

Nice, Codacy helps us now finding unused/unrequired includes!

@plata
Copy link
Collaborator

plata commented Jul 31, 2019

@qparis we need your input here before we take any decision.

@@ -1,5 +1,5 @@
include("engines.wine.quick_script.local_installer_script");
include("engines.wine.engine.object");
const Wine = include("engines.wine.engine.object");
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to restructure the engines.wine.engine.object script. The reason being that the script not only exports the engine Wine but also the corresponding version variables like LATEST_STAGING_VERSION. This becomes a problem when accessing an engine from phoenicis (i.e. the Java side), because you need to know the concrete name of the exported engine if you export multiple values at once.

I see two solutions for this:

  1. we move all exported variables that are not the engine to a new script. This allows us to export the engine using a default export
  2. we use a fixed name for all engine exports (i.e. Engine). This solution will lead to a harder to understand code because every script, independent of the corresponding engine type, will use Engine instead of the concrete engine name, i.e. Wine

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the variables even be global variables? Logically, the belong to the Wine "class".

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question. What we could do is change the variables to be accessible through static methods of class Wine:

class Wine {
   static LATEST_STAGING_VERSION() {
      return "version";
   }
}

This would then require us to use the following to access the most current version:

const Wine = include(...);

Wine.LATEST_STAGING_VERSION()

Sadly it is not yet possible to define static fields or accessor methods.

The benefit of returning the variables as a first class member of an object is that we are able to replace them with accessor methods that automatically fetch the most current version and return it:

Object.defineProperty(module, "LATEST_STAGING_VERSION", {
   get: function() { return version; }
});

This should then make it possible to use:

const {LATEST_STAGING_VERSION} = include(...);

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can LATEST_STAGING_VERSION be a string module export?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you mean by a string module export?
For now I thought to move LATEST_STAGING_VERSION to a new module. This new module would then look like the following:

/* exported LATEST_STABLE_VERSION */
module.LATEST_STABLE_VERSION = "4.0.1";
/* exported LATEST_DEVELOPMENT_VERSION */
module.LATEST_DEVELOPMENT_VERSION = "4.11";
/* exported LATEST_STAGING_VERSION */
module.LATEST_STAGING_VERSION = "4.11";
/* exported LATEST_DOS_SUPPORT_VERSION */
module.LATEST_DOS_SUPPORT_VERSION = "4.0";

Later on we could then think about replacing the hard-coded version strings with more dynamic functions that automatically fetch the corresponding string. To retain the field access syntax for the version strings this would require the usage of

Object.defineProperty(module, "LATEST_STAGING_VERSION", {
   get: function() { return version; }
});

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, this sounds to be the good approach

@qparis
Copy link
Member

qparis commented Aug 6, 2019

Overall, I am 100% in phase with this approach. I think it is more reliable than what we used to do and more up to the standards.

@plata
Copy link
Collaborator

plata commented Aug 7, 2019

The only thing I'm a bit worried about:
Assuming someone would like to replace our Java part with her own software, is it clear how the module system needs to be implemented?

Or to put it differently:
Do we have a chance to be closer to the ES6 module syntax (export)?

@qparis
Copy link
Member

qparis commented Aug 7, 2019

We could use transpilation in the worst case scenario

@madoar
Copy link
Collaborator Author

madoar commented Aug 7, 2019

Assuming someone would like to replace our Java part with her own software, is it clear how the module system needs to be implemented?

To ensure that other people know how the module system should behave and be used I think we should add a short description to the documentation. I think the PR description in PhoenicisOrg/phoenicis#2050 makes a good first version of this explanation.

Do we have a chance to be closer to the ES6 module syntax (export)?

I don't think so. export and import are protected keywords in ES6 that are provided by the programming language and not by a library. This makes it impossible for us to provide custom implementations for these two keywords. The only approach that seems feasible is to "extend" the ES6 module system with a custom loader, but as we have discovered in PhoenicisOrg/phoenicis#2042 this seems to not be possible for now.

We could use transpilation in the worst case scenario

To what should we transpile the code?

@qparis
Copy link
Member

qparis commented Aug 7, 2019

ES6 (scripts) to ES5 modules

@madoar
Copy link
Collaborator Author

madoar commented Aug 7, 2019

It seems like we also need to move:

var WINE_PREFIX_DIR = "wineprefix";

Any ideas where to move the variable? I think we use either my newly added versions script:

/* exported LATEST_STABLE_VERSION */
module.LATEST_STABLE_VERSION = "4.0.1";
/* exported LATEST_DEVELOPMENT_VERSION */
module.LATEST_DEVELOPMENT_VERSION = "4.11";
/* exported LATEST_STAGING_VERSION */
module.LATEST_STAGING_VERSION = "4.11";
/* exported LATEST_DOS_SUPPORT_VERSION */
module.LATEST_DOS_SUPPORT_VERSION = "4.0";

or we add another script for wine engine constants/variables?
@plata @qparis what do you think?

@madoar
Copy link
Collaborator Author

madoar commented Aug 7, 2019

A short status update:
I'm almost done with the apparent changes (i.e. modifying all currently existing includes). The issue is that there were some values that were indirectly included through other scripts. E.g. script A includes script B which includes script C and script A uses variable c defined in script C. This won't work anymore because we now require script A to also include script C if it wants to use its exports.

I'm not sure yet whether I can fix all such occurrences, because they are really hard to detect without running all scripts. A possible solution here is to cleanup the globals in:

scripts/.eslintrc.yml

Lines 41 to 89 in ecd735e

# files
ls: false
mkdir: false
fileExists: false
cat: false
cp: false
getFileSize: false
fileName: false
lns: false
remove: false
touch: false
tr: false
writeToFile: false
createTempFile: false
createTempDir: false
chmod: false
# classes
AppResource: false
CabExtract: false
Checksum: false
Downloader: false
WineEngine: false
Extractor: false
InstallationType: false
Resource: false
SetupWizard: false
Wine: false
WineShortcut: false
# quick scripts
PlainInstaller: false
CustomInstallerScript: false
GogScript: false
InstallerScript: false
LocalInstallerScript: false
OnlineInstallerScript: false
OriginScript: false
QuickScript: false
SteamScript: false
UplayScript: false
ZipScript: false
# wine
LATEST_DEVELOPMENT_VERSION: false
LATEST_STABLE_VERSION: false
LATEST_STAGING_VERSION: false
LATEST_DOS_SUPPORT_VERSION: false
WINE_PREFIX_DIR: false

I'm not sure whether they are still required with explicit exports.

@plata
Copy link
Collaborator

plata commented Aug 8, 2019

We should not put the WINE_PREFIX_DIR into the versions. About ESLint I'm not sure. You have to try.

@madoar
Copy link
Collaborator Author

madoar commented Aug 10, 2019

I have good news! I've finished updating the scripts to work with the new module system! In addition I've added some missing includes that eslint mentioned after I removed the corresponding globals from the .eslintrc file.

@plata @qparis @ImperatorS79 @Zemogiter can you please test if this PR works for you and whether you have any issues with it? If everything works as intended I think we're ready to have this merged (at least from my side...)

@Zemogiter
Copy link
Contributor

I could but I'm not sure about the POL5 side of this PR because it says it's 2 commits behind master branch. Does it matter?

@madoar
Copy link
Collaborator Author

madoar commented Aug 10, 2019

No the two commits are only dependency updates, but I'll merge the master on the Phoenicis side in a moment.

@Zemogiter
Copy link
Contributor

Just launched your branch of POL5 and currently have only 2 repos on the list - local copy of your script branch and classpath. Wine version tab aka engines tab loads fine, but when right after I clicked install button on subnautica script I got this error:

ReferenceError: Wine is not defined
	at com.oracle.truffle.js.runtime.JSException.create(JSException.java:100)
	at com.oracle.truffle.js.runtime.Errors.createReferenceError(Errors.java:188)
	at com.oracle.truffle.js.runtime.Errors.createReferenceErrorNotDefined(Errors.java:350)
	at com.oracle.truffle.js.nodes.access.PropertyGetNode$UndefinedPropertyErrorNode.getValue(PropertyGetNode.java:663)
	at com.oracle.truffle.js.nodes.access.PropertyGetNode.getValue(PropertyGetNode.java:199)
	at com.oracle.truffle.js.nodes.access.PropertyGetNode.getValue(PropertyGetNode.java:160)
	at com.oracle.truffle.js.nodes.access.GlobalPropertyNode.execute(GlobalPropertyNode.java:151)
	at com.oracle.truffle.js.nodes.access.GlobalScopeVarWrapperNode.execute(GlobalScopeVarWrapperNode.java:106)
	at com.oracle.truffle.js.nodes.access.PropertyNode.evaluateTarget(PropertyNode.java:169)
	at com.oracle.truffle.js.nodes.access.PropertyNode.execute(PropertyNode.java:122)
	at com.oracle.truffle.js.nodes.access.WritePropertyNode.evaluateTarget(WritePropertyNode.java:276)
	at com.oracle.truffle.js.nodes.access.WritePropertyNode.executeVoid(WritePropertyNode.java:211)
	at com.oracle.truffle.js.nodes.control.AbstractBlockNode.executeVoid(AbstractBlockNode.java:74)
	at com.oracle.truffle.js.nodes.control.BlockNode.execute(BlockNode.java:62)
	at com.oracle.truffle.js.nodes.function.FunctionBodyNode.execute(FunctionBodyNode.java:73)
	at com.oracle.truffle.js.nodes.function.FunctionRootNode.executeInRealm(FunctionRootNode.java:147)
	at com.oracle.truffle.js.runtime.JavaScriptRealmBoundaryRootNode.execute(JavaScriptRealmBoundaryRootNode.java:93)
	at <js> :program(Unknown)
	at org.graalvm.sdk/org.graalvm.polyglot.Value.execute(Value.java:338)
	at org.phoenicis.scripts.engine.injectors.IncludeInjector.lambda$injectInto$0(IncludeInjector.java:43)
	at org.graalvm.sdk/org.graalvm.polyglot.Context.eval(Context.java:368)
	at org.phoenicis.scripts.engine.implementation.PolyglotScriptEngine.evalAndReturn(PolyglotScriptEngine.java:73)
	at org.phoenicis.scripts.session.PhoenicisInteractiveScriptSession.eval(PhoenicisInteractiveScriptSession.java:35)
	at org.phoenicis.scripts.interpreter.BackgroundScriptInterpreter.lambda$createInteractiveSession$1(BackgroundScriptInterpreter.java:45)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by host exception: com.oracle.truffle.js.runtime.JSException: ReferenceError: Wine is not defined

And got this in the terminal log:

[ERROR] org.phoenicis.multithreading.ControlledThreadPoolExecutorService (l.64) - java.lang.ClassCastException: class java.lang.String cannot be cast to class org.graalvm.polyglot.Value (java.lang.String is in module java.base of loader 'bootstrap'; org.graalvm.polyglot.Value is in module org.graalvm.sdk of loader 'app')
	at org.phoenicis.javafx.components.application.skin.ApplicationInformationPanelSkin.lambda$installScript$7(ApplicationInformationPanelSkin.java:235)
	at org.phoenicis.scripts.session.PhoenicisInteractiveScriptSession.eval(PhoenicisInteractiveScriptSession.java:35)
	at org.phoenicis.scripts.interpreter.BackgroundScriptInterpreter.lambda$createInteractiveSession$1(BackgroundScriptInterpreter.java:45)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

Exception in thread "pool-3-thread-4" java.lang.ClassCastException: class java.lang.String cannot be cast to class org.graalvm.polyglot.Value (java.lang.String is in module java.base of loader 'bootstrap'; org.graalvm.polyglot.Value is in module org.graalvm.sdk of loader 'app')
	at org.phoenicis.javafx.components.application.skin.ApplicationInformationPanelSkin.lambda$installScript$7(ApplicationInformationPanelSkin.java:235)
	at org.phoenicis.scripts.session.PhoenicisInteractiveScriptSession.eval(PhoenicisInteractiveScriptSession.java:35)
	at org.phoenicis.scripts.interpreter.BackgroundScriptInterpreter.lambda$createInteractiveSession$1(BackgroundScriptInterpreter.java:45)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

@plata
Copy link
Collaborator

plata commented Aug 10, 2019

Just as reference, the main topics which should be tested are:

  • app installation
  • Engine plugins, verbs, settings

@madoar
Copy link
Collaborator Author

madoar commented Aug 10, 2019

@Zemogiter can you test another installer script? Just to have a comparison?

@Zemogiter
Copy link
Contributor

Zemogiter commented Aug 10, 2019

I've tried Mount&Blade steam script but after I enter the verification code it stays for some time in system monitor then the steam.exe process disappears. But I had this problem on master branches as well.
UPDATE:Tried Anno 2070 uplay script and it behaves the same as Subnautica. The both use virtual desktop and I pasted my PR to the local repo.

@madoar
Copy link
Collaborator Author

madoar commented Aug 10, 2019

For Mount&Blade can you share the content of the wine.log file in the container folder?

@madoar
Copy link
Collaborator Author

madoar commented Aug 10, 2019

When installing Warlock - Master of the Arcane the installation crashes too with the following content inside the wine.log file:

wine: Fehlerhaftes EXE-Format für C:\windows\system32\wineboot.exe.
0025:err:process:start_wineboot failed to start wineboot, err 193
wine: Fehlerhaftes EXE-Format für C:\windows\system32\regedit.exe.

The german text means: Wrong EXE-format for .... Any ideas?

@Zemogiter
Copy link
Contributor

Zemogiter commented Aug 10, 2019

wine.log:

wine: cannot find L"C:\\windows\\system32\\winemenubuilder.exe"
000b:err:wineboot:ProcessRunKeys Error running cmd L"C:\\windows\\system32\\winemenubuilder.exe -a -r" (2)
[0810/122738:ERROR:network_change_notifier_win.cc(170)] WSALookupServiceBegin failed with: 0
004c:err:module:load_builtin_dll failed to load .so lib for builtin L"winepulse.drv": libpulsecommon-10.0.so: nie można otworzyć pliku obiektu dzielonego: Nie ma takiego pliku ani katalogu
ALSA lib conf.c:3558:(snd_config_hooks_call) Cannot open shared library libasound_module_conf_pulse.so (/usr/lib/i386-linux-gnu/alsa-lib/libasound_module_conf_pulse.so: libasound_module_conf_pulse.so: nie można otworzyć pliku obiektu dzielonego: Nie ma takiego pliku ani katalogu)
ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM default
wine: Zły format EXE dla C:\Program Files\Steam\bin\gldriverquery64.exe.

The text in polish means Can't open shared object file: No file or directory and Bad EXE format for ..., respectivly.

@madoar
Copy link
Collaborator Author

madoar commented Aug 10, 2019

This looks unrelated to this PR to me. Could this be a problem with the newest Wine version build @plata @qparis @ImperatorS79?

@Zemogiter
Copy link
Contributor

Tested Starcraft 2 script and while the installation works upon launching I get the shortcut error:

ReferenceError: ShortcutReader is not defined
	at <js> :program(Unnamed:1:4-17)
	at org.graalvm.sdk/org.graalvm.polyglot.Context.eval(Context.java:368)
	at org.phoenicis.scripts.engine.implementation.PolyglotScriptEngine.evalAndReturn(PolyglotScriptEngine.java:73)
	at org.phoenicis.scripts.session.PhoenicisInteractiveScriptSession.eval(PhoenicisInteractiveScriptSession.java:35)
	at org.phoenicis.scripts.interpreter.BackgroundScriptInterpreter.lambda$createInteractiveSession$1(BackgroundScriptInterpreter.java:45)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

@madoar
Copy link
Collaborator Author

madoar commented Aug 10, 2019

@Zemogiter I just fixed the shortcut bug. The reason were missing const ShortcutReader = ... strings in the execution code in Phoenicis. If we had PhoenicisOrg/phoenicis#2025 merged this would not be necessary...

@madoar
Copy link
Collaborator Author

madoar commented Aug 10, 2019

The ReferenceError: Wine is not defined is hard for me to fix because I can't tell where it is missing from the provided log. Can you test a bit to find out which script is missing the Wine include?

@Zemogiter
Copy link
Contributor

ReferenceError: Wine is not defined was caused by lack of Wine include in virtual desktop plugin.

@madoar
Copy link
Collaborator Author

madoar commented Aug 10, 2019

@Zemogiter but the include is there, see:

const Wine = include("engines.wine.engine.object");

@Zemogiter
Copy link
Contributor

Yes but when I copied the plugin from my PR I accidentaly removed it.

@madoar
Copy link
Collaborator Author

madoar commented Aug 10, 2019

So the scripts are working except the problems shown in the wine.log files?

@Zemogiter
Copy link
Contributor

Yes. Engine tab works and so is the engine tools.

Copy link
Collaborator

@plata plata left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Go for it. Looks good and tested by two guys should be enough.

@madoar madoar merged commit 920c415 into PhoenicisOrg:master Aug 11, 2019
@madoar madoar deleted the add-module-system branch August 11, 2019 09:43
Zemogiter added a commit to Zemogiter/Scripts that referenced this pull request Aug 11, 2019
Added an include in order to be compatible with PhoenicisOrg#1076
petermetz pushed a commit to petermetz/scripts that referenced this pull request Jun 7, 2020
- change all scripts to use the new module system
- move wine version strings to new script
- add missing explicit includes
- move WINE_PREFIX_DIR to new constants script
- remove now unneeded globals from .eslintrc
- remove unused includes from scripts
- update documentation pages
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants