Keep temporary files in memory

[arthuro555]

Description

GDevelop produces two kinds of temporary files when making previews, code files for events-based extensions and code files for scene event sheets. This is not a bad thing in itself, but it causes problems in the context of the web app, where there is no filesystem. Right now the solution is to upload them to an s3, but this comes with a few issues:

1. S3s are not free, it increases the cost of GDevelop.
2. It makes the web app unusable offline as a PWA.
3. Someone smart could probably exploit the system to send many/big files on there, making GDevelop vulnerable.
4. It introduces a lot of web app-only code. If possible, it is better to have polymorphic code everywhere to not double the maintenance cost.

My solution

My solution would be to keep files in memory. To do so, I think it would be better to rework the preview system to just... not use the filesystem. My idea is to use a service worker to intercept any traffic to the preview "site", to handle it from the IDE. There, we could return the temporary files that have been kept in-memory. Additionally, if we want the web app to work offline, we could pre-download the gdjs runtime and "serve" it via the service worker alongside the temporary files.

For events-based extension though, this would not suffice, as we still need an include path to give to the gd::PlatformExtension. For that, we could introduce a "marker" to tell the platform extension to fetch it from the in-memory filesystem (for example, if the include path begins with memory::, get the path that follows it from the in-memory filesystem).

Additional notes

• This kind of system could also be used as a base for storing user-provided resources in-memory for the web app in the future
• Since that in-memory filesystem would be likely used by the c++ exporter code, it would probably need to be made in C++ and to be attached to the gd::Project or gd::Platform.
• This system is polymorphic enough to be usable both for the web and desktop app, lowering the maintenance cost.

[arthuro555]

I made a simple In-Memory file system implementation (untested) that can link an in-memory directory to another AbstractFileSystem's directory, that delays all destructive actions to those other file systems to when a Persist method is called. This could be used for multiple effects in GDevelop:

• Make embedded tools like piskel not destructive anymore (only persist file changes when the project is saved)
• Keep temporary files in-memory by simple not linking the /tmp directory to a real filesystem
• Combine multiple sources (for example, we could have the gdjs sources inside an AbstractFileSystem for zip files mounted to /gdjs, the GDevelop project cloud storage at /project_home, a LocalStorage powered extensions cache mounted at /extesions_cache, a filesystem shared with the service worker for it to allow routing at /preview...), and easily do operations like copying from one to the other.

---

Regardless of this piece of code (that I think is not fully usable in GDevelop as it is), I really think that this enhancement idea is very important. The cloud is nice and all, but it is just a very very bad idea to use it for temporary files in the web app. The problems it causes include:

• The Web IDE PWA cannot be used offline because using events extensions or making a preview forces you to upload files to the cloud
• Making a preview makes a publicly accessible build of the game instead of a local private preview
• Making a preview uses a lot of bandwidth which is costly for many users (especially mobile ones that are likely to have capped mobile data)
• The S3 costs the GDevelop company money that there is no need to spend in the first place
• Making a preview is super slow (because we have to wait for it to upload the files)

Additionally the electron app relies way too much on the local file system for no good reason:

• The electron app generates a ton of junk temporary files that could have just been kept in memory
• Previewing on the electron app is slow too because all assets need to be copied next to the temporary index.html file

If we just keep the auto-generated files in memory and use a service worker to allow accessing those in-memory files as if they were hosted (take the example of StackBlitz), we immediately get rid of:

• A ton of code complexity due to supporting different file systems for previewing/loading events extensions/etc
• A barrier that prevents users with low quality/capped/irregular connections to the internet from using GDevelop's web version
• The agonizing slowness of previews on bigger projects which ruins user experience & the flow
• Junk files piling up
• Reliance on diverse external APIs (Electron & AWS), that make us do a ton only to work for single platforms, in favor of the web platform, which would work everywhere out of the box, including on mobile platform if we want to make an IDE build for it in the future.