Offer for contribution: Automatic category field, existing part notice and Digikey barcode decoding

[Treeed]

Hi,

I implemented some features that I was missing in Partdb. Namely these are:

• When adding a part from an info provider, the category field wouldn't be filled with anything useful. This often led me to accidentally add parts in the wrong category and meant unnecessary clicks when adding part. I changed the entity manager to allow fuzzy searching for hierarchical structures and modified the dto converter to automatically look for the given category/create it if it doesn't exist. The new part dialog displays a warning when a new category is proposed.
• Some people organize their parts differently than the info providers and the info providers have varying organizations too. To mitigate this I implemented a config system that allows a config file to specify successive replacements in the category structure. This allows moving one category to a different place or rebasing an entire branch (or the whole tree) on a different entry.
• I often order the same part more than once. When adding the new part via the info provider system, the system would make a new part, regardless of the identical old one. I changed the info provider service to look for each part that is listed in the info provider search results and add a button to view the existing part if one exists.
• All of the digikey bags already have barcodes with all the information needed to find a part, so registering the barcodes individually doesn't seem so useful. I added a decoder for the digikey barcodes which allows directly jumping to the part without registering the barcode. I noticed however, that the barcode scanner applet is hardly able to decipher datamatrix codes and that there's no good open source alternative, so this will require some more work.

Please take a look at my implementation at https://github.com/Treeed/Part-DB-server/tree/autoCat

@jbtronics would you like to pull these features into the main repo? If so, I would write some docstrings and do some housekeeping and we could have some discussion on how you would like to have the functionality live across the project and implementation details. Also I would need some help on some of the technology specific things, like how to add translated strings.

Regards
Jona

[jbtronics]

Most of it sounds pretty useful, and looks good implementation wise from a quick look through the code. It would be best if you could split up that into different pull requests so I can review these more in detail.

The thing which I do not really like (and I think will hopefully become obsolete anyway soon), is the thing with the configuration of the info providers. First in my opinion the base class for the info providers should stay an interface and not contain any implementation code, as it should just be a minimal interface description. If you want to provide any easy to use implementations, that should be either done via a trait, that developers can import for their implementation, or by providing a seperate abastract class, that implements the interface and provides additional functionality (but normally everything should just require the interface).

Secondly the configuration via these yaml files is neither very user friendly (especially on docker container), and you might run into weird behavior due to caching (or you have the performance penality from needing it to check everytime).
I am working for some time now on a better configuration system for Part-DB, for which I have written an own settings bundle for symfony with which it should be possible to make parts of Part-DB (like the info providers) configurable via UI.
The base things are basically finished already, but that still needs some polishing and work for which i had no time yet. But then you basically just need to add new properties and a attribute to a class, to get a new configuration option, which users can change from a UI. See for example https://github.com/Part-DB/Part-DB-server/blob/settings-bundle/src/Settings/InfoProviderSystem/TMESettings.php or the other files in the Settings folder of the "settings-bundle" branch.

[Treeed]

Okay, cool, I'll go a bit more into detail then.

The category thing:

I think the refactoring of the StructuralDBElementRepository is generally useful, even if it's not used right away. This raises the question whether you would prefer to keep the existing function signatures or rather generalize them and deal with changing them everywhere else as I have done.
Inserting the category without some form of translation is properly not generally useful, since it is unlikely to match the local structure, however it might at least be better than having a random category in there.
Originally I planned to have two stages of conversion: One from the provider specific system to a system of unified categories via a set of config files that are distributed with partdb and a second stage where the user can assign the unified categories to their own structure. The idea would have been to have a set of parts that are common among providers and known to be in the different categories and then generate the config for the first conversion stage from the results from requesting the parts from the different providers. However, when I found that Mouser doesn't really provide an API I lost interest in that, since I found Mouser and Digikey to be the only two reliable databases.
I know the config file as an input method doesn't really match the concept of partdb. Since I'm not exactly a UI wiz I felt it was the easiest way to implement that for me. I have already found some funny caching effects since the dtos are cached and the conversion is done before the data is saved to the dto. I'll take a look at your branch, that definitely sounds interesting!
I didn't like the concept of changing that interface either. I didn't know about traits. Would a trait also be the place where the conversion code should live if the config is implemented via your settings bundles?

Existing parts:

There's also a couple of open questions about this, but I think I'll just open a pull request, so we can continue the discussion there

Digikey Barcodes:

Same here

Scanner performance

After some digging in the strange unmaintained monoculture of open source barcode readers I found there is a wasm wrapper for zxing-cpp which is a bit ahead of zxing-js which is what html5-qrcode uses. I have made a fork of html5-qrcode and integrated that into partdb and it improves the performance on data digikey matrices from terrible (takes 30s to minutes of slowly waving around the camera) to usable (takes some 1-2s and reasonably good alignment).
Unfortunately that wrapper loads its wasm build from an external source and building that wrapper into a self contained npm module is a bit of a nightmare. Due to the way that wasm is loaded its csp policy seems to always be "default-src" and the only way to change the loading location is by overwriting some method and passing a string in.

import { getZXingModule } from "zxing-wasm";

getZXingModule({
  locateFile: (path, prefix) => {
    if (path.endsWith(".wasm")) {
      return `https://unpkg.com/zxing-wasm@1/dist/full/${path}`;
    }
    return prefix + path;
  },
});

I guess one could

• live with the limitations of webpack and manually build and include the custom html5-qrcode and zxing wasm files
• wrestle with webpack until it works
• fork zxing wasm aswell
• live with fastly.jsdelivr.net as a trusted host

[Treeed]

I looked a bit further into the zxing situation and it appears the loading behaviour is not zxing-wasm related, but governed by the wasm toolchain (emscripten) itself. Furthermore, it appears wasm loading generally can't be controlled by the CSP; it can only be turned on or off entirely (WebAssembly/content-security-policy#37).

[jbtronics]

I found out how you can use the local WASM file (you just need to configure a webpack rule, that it should just copy the wasm file without trying to parse it in any way).

I am using the barcode-detector Polyfill which uses the Zxing Wasm to emulate the native barcode detector library on browsers where it is not available (almost everywhere beside Chrome for Android). I would assume that the native version of this API is better and faster than the Zxing lib, so the zxing lib only needs to be loaded when a native version is not available.

I created a fork of html5-qrcode which removes the zxing-js lib completly and only uses the native detector API (or the polyfill provided via WASM). This should basically achieve the same results as your fork.

I adjusted the CSP to allow for the loading of the wasm (via wasm-unsafe-eval).

[Treeed]

Oh, I hadn't realised the native API was able to do Datamatrix codes. I had previously seen that the html5-qrcode library supported using that API, but since it never actually decoded Datamatrix codes I thought it was only meant for 1D barcodes. I played around with that decoder some more and it turns out it only works well on my Galaxy S8 if I increase the canvas size significantly.

With a small canvas (whatever the default size in partdb turns out as), zxing-wasm works alright and the native scanner hardly works at all. With a large canvas (1080x1920) zxing-wasm works really well and the native scanner works alright.
I have enabled both scanners in my variant of html5-qrcode and added some alerts to see which one manages to decode a view first and zxing gets about 9 out of 10 of the views that only one of them decodes. The native scanner seems to be a bit better on 1D barcodes though.

For some reason, some of my digikey bags have a visually identical barcode which both decoders really struggle with. It works fine on some scanning app by cognex though.

Neither decodes this
ZXing decodes this, but the native one doesn't

EDIT: I looked into the ZXing code a bit and it turns out the issue with those "unreadable" bags is that some rows of the data matrix are slightly taller than others. There's a "clocking pattern" on the side of the Datamatrix to compensate for this kind of distortion, but ZXing doesn't use that and instead just reads the code in a regular grid. There's an issue about it here: zxing-cpp/zxing-cpp#669