I can't get this working with LevelDB

[ShayBox]

I'm trying to parse the JSON from LevelDB, I'm pretty sure it's lz-string compressed, it comes from Electron. There are only a few characters that are 2 or 3 bytes long, it looks like UTF-16-BE to me, but it doesn't parse correctly using UTF-16-BE or LE, and I found this blog that says chrome uses lz-string.

I included value.txt which is the raw bytes from LevelDB, example.txt which is the expected JSON from Electron developer tools, and my test project with LevelDB included.

use std::{fs::File, io::Write};

use anyhow::bail;
use lz_str::IntoWideIter;
use rusty_leveldb::{Options, DB};

fn main() -> anyhow::Result<()> {
    println!("Hello, world!");

    let mut db = DB::open(
        "leveldb",
        Options {
            create_if_missing: false,
            paranoid_checks: true,
            ..Default::default()
        },
    )?;

    let key = "_file://\0\u{1}persist:root";
    let Some(mut value) = db.get(key.as_bytes()) else {
        bail!("None - Failed to get value from key: '{key}'");
    };

    println!("Value Length: {}", value.len());

    if let Ok(mut file) = File::create("value.txt") {
        file.write_all(&value)?;
    }

    if value.len() & 1 == 1 {
        println!("Length not even, removing the last element");
        value.pop();
    }

    let buffer = value
        .chunks(2)
        .map(|slice| {
            // The slice is always guaranteed to be 2 here.
            // We check to see if the length is a multiple of 2 earlier.
            u16::from_be_bytes(slice.try_into().unwrap())
        })
        .collect::<Vec<u16>>()
        .into_wide_iter();

    if let Some(decompressed_value) = lz_str::decompress_internal(buffer, 8) {
        println!("1: Decompressed Value Length: {}", decompressed_value.len());
    }

    if let Some(decompressed_value) = lz_str::decompress_from_uint8_array(&value) {
        println!("2: Decompressed Value Length: {}", decompressed_value.len());
    }

    println!("Goodbye, world!");

    Ok(())
}

$ cargo run
Hello, world!
Value Length: 165591
Length not even, removing the last element
1: Decompressed Value Length: 0
Goodbye, world!

value.txt using ImHex in Hexadecimal (8 bits)

value.txt using ImHex in Hexadecimal (16 bits)

example.txt using ImHex in Hexadecimal (8 bits)

example.txt using ImHex in Hexadecimal (16 bits)

Here's my test project with the LevelDB database included
test-project-with-leveldb.zip

[adumbidiot]

Hey, thanks for the interest. First of all, I don't believe any of this is an issue with this library, but I'll try to help out the best I can.

I found this blog that says chrome uses lz-string.

You've misunderstood the blog post. The blog says that instead of saving raw strings to localstorage, some websites compress them with lz-string before saving. This is a technique implemented by the websites themselves, not Chrome; the same compression with lz-string would also be performed with Firefox.

Its been a while since I've messed around with Chrome's localstorage; last I remember, they didn't use any compression. However, I just did a quick check, and they now seem to use Snappy compression, which is the standard compression for leveldb databases. The library you are trying to use actually has support for it built in: https://docs.rs/rusty-leveldb/1.0.8/rusty_leveldb/enum.CompressionType.html.

and my test project with LevelDB included.

From my quick testing, your database is probably corrupted. Hope it wasn't too important. When iterating keys I see sane results, but retrieving a single key gives the strange UTF-16 like behavior. The likely culprit is your leveldb library: https://docs.rs/rusty-leveldb/1.0.8/rusty_leveldb/struct.Options.html#structfield.compression_type. Kind of strange they made a release like that, I would think there are many cases where you can't remember what compression you used to create the database.

As for the parts relating to this library, I don't want to make decompress_from_utf16 take a raw byte buffer since:

1. I would not be able to choose the correct endianess without a BOM.
2. Any valid result produced from this function comes from valid unicode, aka an &str like it currently takes.

I want to force the user to decode into an &str before feeding it into this function. This reasoning should be documented on the function itself.

Also, I think you've misunderstood the binary data example so it should be reworked a bit. I think its too easy to misunderstand and misuse in its current state as a go-to for forcibly trying to decode binary with lz-string. I should update it to remove references to "padding" the data since that is usually the wrong answer and instead focus on the capabilities of compress_to_uint8_array and decompress_to_uint8_array. That way, it will hopefully be more obvious that a user needs to track down which exact function compressed their data.

I'll also put large warning signs next to the internal functions, 99.99% of users shouldn't need to touch them.

Overall, I think the main issues WRT lz-str here are with documentation and clarity; it is functionally ok.

Hope all of this helps, let me know if you have any questions.
I'll work on the documentation/example issues sometime in the next few weeks most likely.

[ShayBox]

I see, well thanks for helping anyway

I don't think my database is corrupt, I've re-created it a few times and tried with and without Snappy compression after creating and copying a fresh database (it's the default created on startup of Wootility)

I had extracted parts of internal functions to try and find the problem but it wasn't related to the library

I'm able to completely decode it using UTF-8 BE, except for a hand full of special characters that decode as two or three characters in Rust but are only one character (multiple bytes), such as ⌘

And it doesn't seem to be an issue with the library, using a LevelDB explorer and extracting the bytes manually gives the same value.txt

EDIT: I'm also able to re-use the LevelDB in the original Electron program without corruption or issue

[ShayBox]

I figured out the encoding! UTF-16 LE starting at every 00 (null byte) and ending at the start of the next null byte

[adumbidiot]

I don't think my database is corrupt, I've re-created it a few times and tried with and without Snappy compression after creating and copying a fresh database (it's the default created on startup of Wootility)

I was incorrect, that database seems fine. I was using an old binding of leveldb that I made a while ago, and I was accidentally converting from UTF-16 when printing.

UTF-8 BE

You mean UTF-16 BE? UTF-8 BE isn't a thing.

I figured out the encoding! UTF-16 LE starting at every 00 (null byte) and ending at the start of the next null byte

If you want to read LocalStorage from Chrome, you really should just look at the source instead of guessing: https://chromium.googlesource.com/chromium/src/+/839226052b486f1ad604f69b0d3d9b2aa0c7746d/content/renderer/dom_storage/local_storage_cached_area.cc.

As you can see, the strings can take 1 of two different formats, encoded in the first byte. If everything is ascii, it is encoded in single-byte form. If not, it uses UTF-16. Looks like it uses native endian encoding, but maybe I'm wrong, haven't dug into that string16 type too deeply. You need to handle both cases to correctly read/write values.

Edit:
Actually, I'm fairly certain UTF-16 is wrong here too. LocalStorage needs to support JS Strings, which are more like UCS-2 since they can have unpaired surrogates. Basically, this is UTF16 that is allowed to have unpaired surrogates, so Rust's from_utf16 is worthless here.

[ShayBox]

Yes I did mean 16 BE, that's a lot of helpful info, I never thought to look at the chromium source code, it always seemed impossible to navigate, but that helps me a lot, thanks 👍