Command Line Interface (CLI) mode

[makemeunsee]

Hello,
I'm wondering if there's an obvious, easy way to reuse 10ten within a CLI tool, the translation feature in particular.
Something like an export of jpdict.ts to a Node module, but I'm not much familiar with Typescript, and so how much of a pipe dream this is.
Thanks for your input!

[birtles]

Hi @makemeunsee! The dictionary lookup feature is published as a separate Node module: @birchill/jpdict-idb. 10ten does a little pre- and post-processing (some of which is published in @birchill/normal-jp) but most of the hard work happens in @birchill/jpdict-idb.

However, it relies on using IndexedDB to store the data and look it up. You might be able to use something like fake-indexeddb in Node but I believe that only operates in-memory meaning a CLI tool would end up downloading the data each time. There might be some other kind of IndexedDB shim for node that can persist to disk that would work. Maybe bigeasy/indexeddb?

In the coming months I plan to obsolete @birchill/jpdict-idb and replace it with a version that runs using SQLite. Once that's available I imagine that would be more suitable for a Node project. I'm sure that module will be open source too so perhaps you could help me shape the API in a way that would suit your use case (e.g. allowing substituting in a different SQLite-compatible library).

[makemeunsee]

Hi @birtles

Thanks for the precious input, it helps a lot!
IndexedDB is a critical point to know about indeed.

I'll try to come up with a PoC soon, based on your suggestion, and report here.

Cheers!

[birtles]

No problem @makemeunsee!

I was just thinking about your question this morning and I was wondering what the CLI will do. You mentioned wanting to use the translation feature. If the CLI tool is parsing whole sentences then you'll probably want to use a tokenizer like MeCab.

Also, the other part of 10ten that might be useful is the deinflection code. That's currently part of 10ten but it might make sense to turn that into a library some day.

[makemeunsee]

what the CLI will do

The ambitious answer is: what 10ten does, but as text :)
Which means, as I see it, given a string, it would return structured data, e.g. for each of words, kanji, names, the best match(es), having applied deinflection, normalization, etc., and the alt forms, pronunciations, translations.

Combined with OCR & notifications, that's 10ten for any visual Japanese content on the desktop.
See this thread for inspiration on the OCR+notification bit (so the easy bit): https://news.ycombinator.com/item?id=39713063

About tokenization and whole sentences parsing, I have considered it, and played with lindera to some extent in this regard, but it'd be a lot of work to properly cover all (or even just some beyond the most basic ones) morphological cases, to know what to meaningfully translate and what to use as grammatical hints.
So I'd keep that for later, and focus on making what exists and works well (kudos for that btw) available in the CLI environment.

[makemeunsee]

Hi @birtles

I've had time to dive into this, and have a proof of concept working, see below for a sample.
bigeasy/indexeddb is dead but fake-indexeddb is indeed a convenient, drop in replacement of the IndexedDB API for Node.
But as you said, the lack of persistence is a big issue, as DB initialization is much too slow (minutes...) even when loading the uncompressed jsonl files from disk.

I cant think of a way to get around this but to use another underlying DB, and having looked at the jpdict-idb code, it is a significant endeavor.
So I'm curious, as you mentioned working on an sqlite solution, how far along are you on this front?
What architecture do you have in mind, a single, external DB instance exposed via http?
Asking before I fork jpdict-idb, and start gutting it and frankensteining it 😅

Sample run; the fetches take the overwhelming bulk of the time (and I dont load the names on purpose yet):

❯ time node dist/index.js -t 行きましょう
行きましょう
waiting DB ready
DB ready
loading words
Requesting download for 'words' series with current version undefined
fetching ./data/version-en.json
fetching ./data/words/en/2.0.258-1.jsonl
fetching ./data/words/en/2.0.258-2.jsonl
fetching ./data/words/en/2.0.258-3.jsonl
fetching ./data/words/en/2.0.258-4.jsonl
fetching ./data/words/en/2.0.258-5.jsonl
fetching ./data/words/en/2.0.258-6.jsonl
fetching ./data/words/en/2.0.258-7.jsonl
fetching ./data/words/en/2.0.258-8.jsonl
fetching ./data/words/en/2.0.258-9.jsonl
fetching ./data/words/en/2.0.258-10.jsonl
fetching ./data/words/en/2.0.258-11.jsonl
fetching ./data/words/en/2.0.258-12.jsonl
fetching ./data/words/en/2.0.258-13.jsonl
fetching ./data/words/en/2.0.258-14.jsonl
fetching ./data/words/en/2.0.258-15.jsonl
fetching ./data/words/en/2.0.258-16.jsonl
fetching ./data/words/en/2.0.258-17.jsonl
fetching ./data/words/en/2.0.258-18.jsonl
fetching ./data/words/en/2.0.258-19.jsonl
fetching ./data/words/en/2.0.258-20.jsonl
fetching ./data/words/en/2.0.258-21.jsonl
fetching ./data/words/en/2.0.258-22.jsonl
fetching ./data/words/en/2.0.258-23.jsonl
fetching ./data/words/en/2.0.258-24.jsonl
fetching ./data/words/en/2.0.258-25.jsonl
fetching ./data/words/en/2.0.258-26.jsonl
fetching ./data/words/en/2.0.258-27.jsonl
fetching ./data/words/en/2.0.258-28.jsonl
fetching ./data/words/en/2.0.258-29.jsonl
fetching ./data/words/en/2.0.258-30.jsonl
fetching ./data/words/en/2.0.258-31.jsonl
fetching ./data/words/en/2.0.258-32.jsonl
fetching ./data/words/en/2.0.258-33.jsonl
fetching ./data/words/en/2.0.258-34.jsonl
fetching ./data/words/en/2.0.258-35.jsonl
fetching ./data/words/en/2.0.258-36.jsonl
fetching ./data/words/en/2.0.258-37.jsonl
fetching ./data/words/en/2.0.258-38.jsonl
fetching ./data/words/en/2.0.258-39.jsonl
fetching ./data/words/en/2.0.258-40.jsonl
fetching ./data/words/en/2.0.258-41.jsonl
words loaded
loading kanji
Requesting download for 'kanji' series with current version undefined
fetching ./data/kanji/en/4.1.47.jsonl
Requesting download for 'radicals' series with current version undefined
fetching ./data/radicals/en/4.1.1.jsonl
kanji loaded
translating
{
  type: 'translate',
  data: [
    {
      id: 1578850,
      k: [
        {
          ent: '行く',
          p: [ 'i1' ],
          match: true,
          wk: 5,
          bv: { l: 5 },
          matchRange: [ 0, 2 ]
        },
        { ent: '往く', i: [ 'rK' ], match: false }
      ],
      r: [
        { ent: 'いく', p: [ 'i1' ], a: 0, match: true },
        { ent: 'ゆく', p: [ 'i1' ], a: 0, match: true },
        { ent: 'イク', i: [ 'sk' ], a: 0, match: true }
      ],
      s: [
        {
          g: [
            { str: 'to go' },
            { str: 'to move (towards)' },
            { str: 'to head (towards)' },
            { str: 'to leave (for)' }
          ],
          pos: [ 'v5k-s', 'vi' ],
          xref: [ { k: '来る', sense: 1 } ],
          match: true
        },
        {
          g: [
            { str: 'to move through' },
            { str: 'to travel across' },
            { str: 'to walk along (e.g. a road)' }
          ],
          pos: [ 'v5k-s', 'vi' ],
          match: true
        },
        {
          g: [
            { str: 'to go (well, badly, etc.)' },
            { str: 'to proceed' },
            { str: 'to turn out' },
            { str: 'to get along' }
          ],
          pos: [ 'v5k-s', 'vi' ],
          match: true
        },
        {
          g: [
            { str: 'to do (in a particular way)' },
            { str: 'to go (with; a choice)' },
            { str: 'to try' }
          ],
          pos: [ 'v5k-s', 'vi' ],
          inf: 'oft. as で〜',
          match: true
        },
        {
          g: [
            { str: 'to pass (of time, seasons, etc.)' },
            { str: 'to go by' }
          ],
          pos: [ 'v5k-s', 'vi' ],
          match: true
        },
        {
          g: [ { str: 'to stream' }, { str: 'to flow' } ],
          pos: [ 'v5k-s', 'vi' ],
          match: true
        },
        {
          g: [ { str: 'to die' }, { str: 'to pass away' } ],
          pos: [ 'v5k-s', 'vi' ],
          inf: 'usu. written as 逝く',
          match: true
        },
        {
          g: [
            { str: 'to reach (a stage, extent, age, etc.)' },
            { str: 'to get to' },
            { str: 'to go (so far as ...)' }
          ],
          pos: [ 'v5k-s', 'vi' ],
          match: true
        },
        {
          g: [
            {
              str: 'to reach (of information, instructions, wind, etc.)'
            },
            { str: 'to arrive' }
          ],
          pos: [ 'v5k-s', 'vi' ],
          match: true
        },
        {
          g: [
            { str: 'to continue ...' },
            { str: 'to go on ...' },
            { str: 'to (progress) steadily' },
            { str: 'to gradually ...' },
            { str: 'to progressively ...' }
          ],
          pos: [ 'aux-v', 'v5k-s' ],
          misc: [ 'uk' ],
          inf: 'after the -te form of a verb; い sometimes omitted in casual speech',
          match: true
        },
        {
          g: [
            { str: 'to have an orgasm' },
            { str: 'to come' },
            { str: 'to cum' }
          ],
          pos: [ 'v5k-s', 'vi' ],
          misc: [ 'uk', 'col' ],
          inf: 'oft. written as イク',
          match: true
        },
        {
          g: [
            { str: 'to trip' },
            { str: 'to get high' },
            { str: 'to have a drug-induced hallucination' }
          ],
          pos: [ 'v5k-s', 'vi' ],
          misc: [ 'uk', 'sl' ],
          match: true
        }
      ],
      reason: '< deinflect_polite_volitional'
    }
  ],
  textLen: 6,
  more: false
}

real	1m33.096s
user	1m35.751s
sys	0m0.499s

[birtles]

Hi @makemeunsee

I cant think of a way to get around this but to use another underlying DB, and having looked at the jpdict-idb code, it is a significant endeavor. So I'm curious, as you mentioned working on an sqlite solution, how far along are you on this front? What architecture do you have in mind, a single, external DB instance exposed via http? Asking before I fork jpdict-idb, and start gutting it and frankensteining it 😅

I was thinking about this and I remembered another option.

When you first install 10ten it comes with a snapshot of the word database and an index into that (see here). The whole database snapshot is then loaded into memory and looked up from there. Internally it's called the "flat file database" and the code for it is here: https://github.com/birchill/10ten-ja-reader/blob/main/src/background/flat-file.ts

That code could be made a lot simpler by replacing the async and retry parts with Node's sync file reading functions.

In 10ten we only use the flat file snapshot until we have download the database into IndexedDB because it occupies a lot of memory and we can't easily update the snapshot. For a CLI tool, however, I think it wouldn't matter too much to use a lot of memory. (And if you really wanted, you could directly read the necessary parts of the file from disk to avoid loading it all into memory.)

Regarding updating, the code to generate the snapshot is here: https://github.com/birchill/10ten-ja-reader/blob/main/scripts/update-snapshot.ts You could possibly set up a regular task to run code similar to that, generate a new snapshot, and then publish a new version of your tool every few weeks to keep the data fresh.

Regarding the SQLite version, I have a prototype for benchmarking purposes but I haven't started on the actual library yet. I expect it will follow pretty closely jpdict-idb with regards to downloading patches to update a local database which would mean that for a CLI tool you'd still pay the initial download cost unless you shipped a snapshot to begin with.

[makemeunsee]

Yes, I spotted the 'flat file DB' but didnt think too much of it. I'll look further into it, thanks!

Noted for the updating part, though it'll come after making the CLI actually usable :)

Regarding the SQLite version
a local database

That's the part I'm curious about, I dont know really what's possible to do within a browser extension.
How can it be a 'local' DB? The wasm build of SQLite?
How do you manage persistence then?
Imagining how the CLI could be based on the SQLite version, then maybe just the persistence layer would need to differ from the browser version.

[birtles]

Regarding the SQLite version
a local database

That's the part I'm curious about, I dont know really what's possible to do within a browser extension. How can it be a 'local' DB? The wasm build of SQLite? How do you manage persistence then? Imagining how the CLI could be based on the SQLite version, then maybe just the persistence layer would need to differ from the browser version.

Right, using the WASM build of SQLite (or actually a variant that allows substituting IndexedDB as the persistence layer since it's much faster for bulk writes).

[makemeunsee]

It works!

10ten_as_notif.mp4

Thanks for your guidance, it made the difference between me giving up and pushing further!

It's quite crude really but should suit my needs for the time being; I'm open to improving it later on if there's an interest.

Cheers!

[birtles]

Well done! It looks great! One day I hope to have 10ten work on images in web pages but what you have should allow it to work in many more situations too.