An exploratory research project to generate new Bliss vocabulary using machine learning techniques.
The Bliss language is an Augmentative and Alternative Communication (AAC) language used by individuals with severe speech and physical impairments around the world, but also by others for language learning and support, or just for the fascination and joy of this unique language representation. It is a semantic graphical language that is currently composed of more than 5000 authorized symbols - Bliss-characters and Bliss-words. It is a generative language that allows its users to create new Bliss-words as needed.
We are exploring the generation of new Bliss vocabulary using emerging AI techniques, including Large Language Models (LLM), OCR, and other models for text generation and completion.
- Python 3
- Version 3.9+. On Mac, Homebrew is the easiest way to install.
- Clone the project from GitHub. Create a fork
with your GitHub account, then run the following in your command line (make sure to replace
your-username
with your username):
git clone https://github.com/your-username/baby-bliss-bot
cd baby-bliss-bot
Always activate and use the python virtual environment to maintain an isolated environment for project's dependencies.
-
Create the virtual environment (one time setup):
python -m venv .venv
-
Activate (every command-line session):
- Windows:
.\.venv\Scripts\activate
- Mac/Linux:
source .venv/bin/activate
- Windows:
Run in the baby-bliss-bot directory:
pip install -r requirements.txt
Run the following command to lint all python scripts:
flake8
We performed experiments with a number of existing models listed below to understand how useful they are in helping with generating new Bliss symbols etc.
Conclusion: useful
See the Llama2FineTuning.md in the documentation folder for details on how to fine tune, evaluation results and the conclusion about how useful it is.
Conclusion: not useful
See the TrainStyleGAN3Model.md in the documentation folder for details on how to train this model, training results and the conclusion about how useful it is.
Conclusion: shows promise
See the StyleGAN2-ADATraining.md in the documentation folder for details on how to train this model and training results.
Conclusion: not useful
See the Texture Inversion documentation for details.
Conclusion: useful
RAG (Retrieval-augmented generation) technique is explored to resolve ambiguities by retrieving relevant contextual information from external sources, enabling the language model to generate more accurate and reliable responses.
See RAG.md for more details.
Conclusion: useful
When users have a back-and-forth conversation, the application requires a form of "memory" to retain and incorporate past interactions into its current processing. Two methods are explored to achieve this:
- Summarizing the chat history and providing it as contextual input.
- Using prompt engineering to instruct the language model to consider the past conversation.
The second method, prompt engineering, yields more desired responses than summarizing chat history.
See ReflectChatHistory.md for more details.
/notebooks
directory contains all notebooks used for training or fine-tuning various models.
Each notebook usually comes with a accompanying dockerfile.yml
to elaborate the environment that the notebook was
running in.
/jobs
directory contains all jobs and scripts used for training or fine-tuning various models, as well
as other explorations with RAG (Retrieval-augmented generation) and preserving chat history.
All utility functions are in the utils
directory.