Tags corrupted when importing notes with special chars

[twx99]

Describe the bug
Importing an exported (unencrypted or encrypted) backup-file containing certain notes with certain characters result in corrupted tag-data.

To Reproduce
I've created a standard notes database export with example data, attached as backup.zip.

1. Download the database backup file (backup.zip) generated by the Windows client 3.5.11 from a good working Standard Notes installation. Exporting from the latest IOS app to date results in the same problem described below as well.
2. Register an new, empty account somewhere. I use a own docker server.
3. Import the backup
4. Data corrupts, tags seem to be the main issue. See screenshot bug.png. A lot of tags are not added for example.

Expected behavior
See screenshot expected.png.

Desktop:

• OS: Windows 10
• Browser: N/A
• Version 3.5.11

Server

• OS: Raspian 10 Buster (Raspberry PI 4)
• Version; I've looked for this but cannot find the info. I've used the docker-compose installation on 2020-12-17 as described here: https://github.com/standardnotes/syncing-server . Worked flawless except for this bug encountered today.

Additional context

• Trying to use "Batch Manager"-extension to remove the corrupted notes after the failed import result in Batch Manager bugging: It's not possible to select notes or tags. The red delete-button flicker for a millisecond and notes are being unmarked again. When trying to reproduce this, it happens some times, and some times not, even though I'm importing the same data every time.
• I've isolated the problem to the note called "Raspberry Pi - App". This note contained alot of special characters. I have obfuscated this note to keep it in the database for reference. Removing it from the source database, creating a new backup, and importing it does not reproduce the problem for me.
• Found this workaround: Importing the backup file to an empty Standard Notes installation - before registering and connecting to an empty account - does work flawless. The import works as expected. I can then, with the data already imported, register for an account, and sign in to it. That gives a question if I want to merge the imported notes with the empty account, which work.

Thanks for an excellent app. Love the simplicity, speed, and extension based approach. I use it completely without extensions, which fits my needs perfect. This is the first time I ever produce an bug report like this and I hope it will help your team, and maybe other users.

backup.zip

[JaspalSuri]

Hi @twx99,

Thank you so much for this detailed bug report and for sharing the backup file. We really appreciate the time that you've put into this.

While I could reproduce the issue (and duplicate tags were produced 😞 ) removing the Raspberry Pi - App note and reimporting the backup didn't work as expected. Could you try reproducing this again or provide us with another backup that doesn't contain that note?

We're received a similar report about how certain characters or words might trigger the import issue, but haven't been able to reproduce that, either. Perhaps we can find a connection between the two.

Thanks for an excellent app. Love the simplicity, speed, and extension based approach.

Thanks so much for the kind words and thanks again for taking the time to report this. We hope to find a fix soon.

[twx99]

You are correct, I can not produce a working workaround with the database I provided here. But I could if I removed it from the orginal database and created a backup from there. I've done alot of more tests; It seems that if the database have been corrupted once and then restored, it will poison also future backups taken. I went as far as deleting all notes except 4 in that database, and replacing the text and heading in them with only [a-z] chars, yet the backup became corrupted, suggesting the problem is now lingering in metadata or other non visual data on the account/server.

While laborating with accounts on sync.standardnotes.org I found that if I once put a document containing certain chars on an account, it seem to corrupt account metadata for that user on the server. Meaning all future backups will be corrupted, even if that note have been deleted. Reproduce:

1. Login to sync.standardnotes.org using credentials: [email protected]:temptemp
2. Take a backup
3. Register a new account and restore the database. It will corrupt.

I also found a way to reproduce the error in full, on a new account. Do this:

1. Register a new account on sync.standardnotes.org
2. Create 3 notes, set tags to like #linux #tag1 #tag2 randomly on the notes, 2-3 tags per note. Add any text.
3. Create 1 note containing attached file (chars.txt) tag it #linux
4. Create 2 more notes like in step 2
5. Take a backup, and try to restore it on a another newly registered account

[JaspalSuri]

Hi @twx99,

Thanks for taking the time to do that! I'll look into the steps that you've provided.

So far I've been able to reproduce an issue where duplicate tags are created if a backup (decrypted or encrypted) from one new account is imported into another new account, regardless of the text contained within the notes (I know that I'm skipping a lot of steps here). I'll be sure to reach out to you once we've learned more about this issue and hopefully we can find a fix for it soon!

I also noticed that importing the backup into the web app when you're not signed in will import it properly. Signing in after doing so should inherit the data.

[myreli]

We are triaging our older issues, and it seems like this has been resolved (v3.148.0).

Let us know if you run into any problems.