Server: Resolves #9355: Add sync debug page to Joplin Server #9368

personalizedrefrigerator · 2023-11-25T02:00:52Z

Summary

Adds a new /sync_debug route to Joplin Server that provides the following information:

whether an item is on the server
whether an item shows up in the initial diff

To take advantage of existing rate-limiting code, information is fetched from a client-side script rather than on the server.

Resolves #9355.

Notes

Joplin Server doesn't seem to have a bundler or other workflow set up for converting TypeScript to client-side JavaScript. As such, most of the new code is in JavaScript, rather than TypeScript.

Screenshot

Testing plan

Start Joplin Server in development mode
Sync a client to Joplin Server
Copy the markdown link to a note
Open http://localhost:22300/sync_debug in a browser
Paste the markdown link into the Check if an item is present on the server input box and press Submit
Paste the markdown link into the Check if an item is in the diff used for an initial sync input box and press Submit
Repeat for a resource with a file URL, the external link to a notebook, and a note uuid
Repeat with a UUID that does not correspond to an item

packages/server/package.json

personalizedrefrigerator · 2023-11-28T16:20:55Z

packages/utils/package.json

@@ -20,6 +20,7 @@
  "scripts": {
    "tsc": "tsc --project tsconfig.json",
    "watch": "tsc --watch --preserveWatchOutput --project tsconfig.json",
+    "build": "yarn run tsc",


This is currently needed because yarn run tsc is run after yarn workspaces foreach --verbose --interlaced --topological run build. A script run during yarn ... run build, however, now relies on utils being compiled.

A similar line is added in #9360.

I'd prefer we keep the existing order - install, build, tsc to keeps things more predictable. So far we managed to do this, so we can probably find a solution here too? In some cases I think we use ts-node to run TypeScript files before they've been built, which would be fine.

This is currently required to use execCommand in gulpfile.js.

There does seem to be a way to run gulpfiles with ts-node, but if we switch to a different build method, this shouldn't be necessary (see comment below).

It also might not be sufficient — build.ts is run with ts-node in #9360 but still seems to be unable to import execCommand on a first build. I think it's related to @joplin/utils compiling JavaScript into a dist/ directory and not exporting the TypeScript?

It looks like we're also running tsc during build for packages/react-native-saf-x and packages/fork-htmlparser2.

I'm not sure using ts-node would help here — #9360 uses ts-node and is experiencing the same issue.

laurent22 · 2023-11-29T14:21:25Z

packages/server/package.json

    "test": "jest --verbose=false",
    "test-ci": "yarn test",
    "test-debug": "node --inspect node_modules/.bin/jest -- --verbose=false",
    "clean": "gulp clean",
    "populateDatabase": "JOPLIN_TESTS_SERVER_DB=pg node dist/utils/testing/populateDatabase",
    "stripeListen": "stripe listen --forward-to http://joplincloud.local:22300/stripe/webhook",
-    "watch": "tsc --watch --preserveWatchOutput --project tsconfig.json"
+    "watch": "gulp watch"


I'm not a fan of moving this to gulp because that's yet another level of indirection. And we used to use gulp for watching and it caused various issues so I'd prefer if we stay as close as possible to the tsc binary for this.

Here are some options:

Do compilation with TypeScript project references, as was done in an earlier commit.

Benefits:

Calling tsc for watching/building only requires one call (e.g. tsc --build --watch --preserveWatchOutput or tsc --build).

Drawbacks:

Requires building .d.ts files in the referenced project.

There doesn't seem to be a way to specify the root tsconfig.json file (other than by changing the current working directory). As the server package currently specifies a tsconfig.json for build/watch commands, I suspect that specifying the directory with the root tsconfig.json is (or was) important.

Current method — call tsc with execCommand from gulp (or similar method: Call tsc from a compile-typescript.ts file run by ts-node).

Benefits:

Can watch both projects at the same time

Runs tsc for both projects during the tsc step (rather than during build)

Works on non-bash-like shells (as opposed to tsc --watch --project project1 & ; tsc --watch --project project2)

Drawbacks:

Requires modifying @joplin/utils to make execCommand available during the build step or modifying @joplin/utils to also export TypeScript files and moving to a TypeScript gulp configuration file. (Alternatively, re-implementing execCommand.)

Create a separate watch-public command for the public/ directory and compile both with tsc --project ./tsconfig.jsoon && tsc --project ./public/tsconfig.json.

Benefits:

Simple — doesn't require running tsc from JavaScript or significantly changing build flags.

Drawbacks:

yarn run watch won't watch the public directory.

Use webpack or rollup and a build-bundled-js command that runs during yarn build.

Benefits:

Allows import statements in TypeScript that will be built.

Similar to how JavaScript for WebViews, etc. are bundled in other packages (e.g. app-mobile/)

Drawbacks:

More complicated to set up

Easier to unintentionally bundle large amounts of code

laurent22 · 2023-11-29T14:25:21Z

packages/utils/package.json

@@ -20,6 +20,7 @@
  "scripts": {
    "tsc": "tsc --project tsconfig.json",
    "watch": "tsc --watch --preserveWatchOutput --project tsconfig.json",
+    "build": "yarn run tsc",


I'd prefer we keep the existing order - install, build, tsc to keeps things more predictable. So far we managed to do this, so we can probably find a solution here too? In some cases I think we use ts-node to run TypeScript files before they've been built, which would be fine.

packages/server/public/css/index/sync_debug.css

…ug-report

laurent22 · 2023-12-13T19:30:58Z

@personalizedrefrigerator, I was looking at this PR again. Most of the implementation is on client-side however that doesn't really seem necessary? For example, checking if an item is on the server, or part of that initial diff, can be done server-side. In fact in general we mostly do things server-side in Joplin Server/Cloud as it's easier to manage and test. Is there any reason why we should do all this client-side?

personalizedrefrigerator · 2023-12-13T19:34:42Z

@personalizedrefrigerator, I was looking at this PR again. Most of the implementation is on client-side however that doesn't really seem necessary? For example, checking if an item is on the server, or part of that initial diff, can be done server-side. In fact in general we mostly do things server-side in Joplin Server/Cloud as it's easier to manage and test. Is there any reason why we should do all this client-side?

It was originally done server-side and generated a single, long debug report. However, such checks can take a long time, particularly with a large number of notes. (The code was similar to the disabled changes route).

Doing this client-side takes advantage of existing protections against too many requests (and thus makes denial of service attacks more difficult).

personalizedrefrigerator · 2024-01-05T02:47:14Z

I think this pull request would be helpful in resolving this sync issue.

Would it make sense to implement the sync debug page in JavaScript instead of TypeScript? Doing so would simplify this pull request.

personalizedrefrigerator · 2024-07-10T06:22:41Z

For now, I'm closing this as not planned. If necessary, this pull request can be re-opened in the future.

personalizedrefrigerator and others added 11 commits November 21, 2023 08:25

Draft: Create debug report route on joplin server

97e3a51

Switch to seperate sync debug page

d084136

Move new JS to separate .js file

77cef1b

Update labels

7ca0ab2

Disable submit button while processing

f4dcbbf

Merge branch 'dev' into pr/joplin-server-debug-report

f864283

Set up TypeScript building for the public/js directory

21ea1cb

Fix .gitignore

e73d8c1

Simplify tsconfig.json

d127495

Remove debug logging

fdbe833

Fix indentation

7cc313b

personalizedrefrigerator commented Nov 28, 2023

View reviewed changes

packages/server/package.json Outdated Show resolved Hide resolved

personalizedrefrigerator added 3 commits November 28, 2023 07:37

eslintignore: Ignore generated .d.ts files in server/public/js

bff383b

Compile TypeScript using Gulp

0cf24fe

Fix build

1c26a35

personalizedrefrigerator commented Nov 28, 2023

View reviewed changes

laurent22 reviewed Nov 29, 2023

View reviewed changes

personalizedrefrigerator commented Nov 29, 2023

View reviewed changes

packages/server/public/css/index/sync_debug.css Outdated Show resolved Hide resolved

personalizedrefrigerator added 5 commits November 29, 2023 12:27

Adjust CSS naming to be closer to RCSS

a0ab036

Merge remote-tracking branch 'upstream/dev' into pr/joplin-server-deb…

b10f79f

…ug-report

Move tsc to package.json

48c5e02

Additional documentation

aa2dd7d

Simplify additional documentation

117db82

personalizedrefrigerator closed this Jul 10, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Server: Resolves #9355: Add sync debug page to Joplin Server #9368

Server: Resolves #9355: Add sync debug page to Joplin Server #9368

personalizedrefrigerator commented Nov 25, 2023 •

edited

Loading

personalizedrefrigerator Nov 28, 2023 •

edited

Loading

laurent22 Nov 29, 2023

personalizedrefrigerator Nov 29, 2023 •

edited

Loading

personalizedrefrigerator Dec 2, 2023

laurent22 Nov 29, 2023

personalizedrefrigerator Nov 29, 2023 •

edited

Loading

laurent22 Nov 29, 2023

laurent22 commented Dec 13, 2023

personalizedrefrigerator commented Dec 13, 2023 •

edited

Loading

personalizedrefrigerator commented Jan 5, 2024 •

edited

Loading

personalizedrefrigerator commented Jul 10, 2024

Server: Resolves #9355: Add sync debug page to Joplin Server #9368

Server: Resolves #9355: Add sync debug page to Joplin Server #9368

Conversation

personalizedrefrigerator commented Nov 25, 2023 • edited Loading

Summary

Notes

Screenshot

Testing plan

personalizedrefrigerator Nov 28, 2023 • edited Loading

Choose a reason for hiding this comment

laurent22 Nov 29, 2023

Choose a reason for hiding this comment

personalizedrefrigerator Nov 29, 2023 • edited Loading

Choose a reason for hiding this comment

personalizedrefrigerator Dec 2, 2023

Choose a reason for hiding this comment

laurent22 Nov 29, 2023

Choose a reason for hiding this comment

personalizedrefrigerator Nov 29, 2023 • edited Loading

Choose a reason for hiding this comment

laurent22 Nov 29, 2023

Choose a reason for hiding this comment

laurent22 commented Dec 13, 2023

personalizedrefrigerator commented Dec 13, 2023 • edited Loading

personalizedrefrigerator commented Jan 5, 2024 • edited Loading

personalizedrefrigerator commented Jul 10, 2024

personalizedrefrigerator commented Nov 25, 2023 •

edited

Loading

personalizedrefrigerator Nov 28, 2023 •

edited

Loading

personalizedrefrigerator Nov 29, 2023 •

edited

Loading

personalizedrefrigerator Nov 29, 2023 •

edited

Loading

personalizedrefrigerator commented Dec 13, 2023 •

edited

Loading

personalizedrefrigerator commented Jan 5, 2024 •

edited

Loading