🚧 This repo is a near-complete re-write of the legacy mothertongues code and is set to replace it. It is under development however and is not complete yet. For a list of planned improvements see this section 🚧
MTD is an open-source tool that allow language communities and developers to quickly and inexpensively make their dictionary data digitally accessible. MTD is a tool that parses and prepares your data for being used with an MTD User Interface.
Please visit the website or docs for more information.
This project started as just a single dictionary for Gitxsan - a language spoken in Northern British Columbia, but it became quickly apparent that many communities also had the same problem. That is, they had some dictionary data but all of the options for sharing that data online were prohibitively expensive. MTD aims to make it easier to create online digital dictionary resources.
Note - Just because you can make an online dictionary does not mean you should. Before making a dictionary, you must have clear consent from the language community in order to publish a dictionary. For some background on why this is important, please read sections 1 and 2.1 here
It is recommended to install mothertongues using pip. The package name is mothertongues, and as of version 1.0.x it is imported with import mothertongues. The CLI can be run using mothertongues --help.
pip install mothertongues
If you just want to try something out you can use the mothertongues command line to create a configuration and some sample data:
poetry run python3 cli.py new-project- Then run your dictionary:
poetry run python3 cli.py build-and-run <YourDictionaryConfigDirPath>/config.mtd.json
To install locally you will have to have Git, Python 3.8+, poetry and Node 16+ on your machine. You can then follow these steps:
- Clone repo and UI submodule
git clone https://github.com/MotherTongues/mothertongues.git --recursive - Build the UI:
cd mothertongues/mothertongues-UI && npm install - Build the Python Development version of the UI:
npx nx build mtd-mobile-ui --configuration=pydev - Install the Python package:
cd .. && poetry install
In order to create a Mother Tongues Dictionary you will need at least two things:
- A configuration file for your language/dictionary
- A configuration file for each source of data
You can find out more about how to create these files against the MTD configuration schema by visiting the guides
Once you have those files, you can either create a dictionary using the command line interface.
The basic workflow for creating a dictionary is as follows:
- Fork and clone the mtd-starter
- Edit and prepare the repo using your own data
- Export your data to a format readable by the Mother Tongues User Interfaces
- Add your exported data (
dictionary_data.json) from step 3 and then publish your dictionary! 🎉
There are a variety of improvements from the legacy mothertongues library. Here are a few of them:
- The CLI now builds an inverted index over any of your dictionary entry keys and search is conducted on the terms of that index.
- Results are now ranked by a combination of Edit Distance and OkapiBM25 meaning you're likely to get better results. We sort results by first sorting them based on edit distance - for results with the same edit distance, we then sort them based on their OkapiBM25 score.
- We use strongly typed configurations to reduce errors when building your dictionary
- We support two search algorithms out-of-the-box: an unweighted Levenstein search over Levenstein automata (very fast) or a quadratic weighted Levenstein search over the terms of the index (flexible and slower, but still pretty fast).
- The search algorithm is now optimized for multi-term searches
- There is a unified normalization strategy between the UI and CLI that allows for performing case normalization, Unicode normalization, removal of punctuation (customizable), arbitrary replace rules, and removal of combining diacritics/accents.
- There is a built in web-server in the CLI for quickly spinning up a development version of your app
mothertongues build-and-run <path_to_language_config> - There is an API that serves the MTD JSON schemas and validates your configuration files and dictionary data.
- The sorter is improved to be able to handle Out of Vocabulary (OOV) characters
- Parsing JSON is many times faster thanks to @dhdaines
If something is not working, or you'd like to see another feature added, feel free to dive in! We please ask that you read the contributing guidelines before submitting any pull requests. Open an issue or submit PRs. Help writing and clarifying documentation is also very welcome.
This repo follows the Contributor Covenant Code of Conduct.
Thank you to both Patrick Littell & Mark Turin for their contributions, guidance and support as well as institutional support from the First Peoples' Cultural Council and SSHRC Insight Grant 435-2016-1694, ‘Enhancing Lexical Resources for BC First Nations Languages’.
Thank you to all other contributors for support with improving MotherTongues, finding bugs and writing documentation.
This project exists thanks to all the people who contribute.