A Python module for running the Mixtral-8x7B language model with customisable precision and attention mechanisms. Mixtral-8x7B, being a pretrained base model, lacks moderation mechanisms. Read more here.
pip/pip3 install mixtral.py
import mixtral
mixtral = Mixtral()
print("Running the model:")
print(mixtral.generate_text("..."))
import mixtral
print("Running the model in half precision:")
mixtral_half_precision = Mixtral(use_half_precision=True)
print(mixtral_half_precision.generate_text("..."))
import mixtral
print("Running the model in lower precision using (8-bit & 4-bit) using bitsandbytes:")
mixtral_4bit = Mixtral(load_in_4bit=True)
print(mixtral_4bit.generate_text("..."))
import mixtral
print("Load the model with Flash Attention 2:")
mixtral_flash_attention_2 = Mixtral(use_flash_attention_2=True)
print(mixtral_flash_attention_2.generate_text("..."))
- minimum 100GB GPU RAM (Mistral AI)
- Hardware requirements after fine-tuning can be found in the discussion here
- mixtral.py is under the GNU General Public License v3
- Mixtral 8x7b is under the Apache 2.0 License
I welcome contributions from the community and appreciate the time and effort put into making mixtral.py better. To contribute, please follow the guidelines and steps outlined below:
Note: Your pull request will be closed if you do not specify the changes you've made.
Start by forking this repository. You can do this by clicking on the "Fork" button located at the top right corner of the GitHub page. This will create a personal copy of the repository under your own GitHub account.
Next, clone the forked repository to your local machine using the following command:
$ git clone https://github.com/yourusername/mixtral.py.git
Navigate to the cloned directory:
$ cd mixtral.py
Before making any changes, it's recommended to create a new branch. This ensures that your changes won't interfere with other contributions and keeps the main branch clean. Use the following command to create and switch to a new branch:
$ git checkout -b branch-name
Now, you can proceed to make your desired changes to the project. Whether it's fixing bugs, adding new features, improving documentation, or optimizing code, your efforts will be instrumental in enhancing the project.
Once you have made the necessary changes, commit your work using the following commands:
$ git add .
$ git commit -m "Your commit message"
Push the changes to your forked repository:
$ git push origin branch-name
Head over to the original repository on GitHub and go to the "Pull requests" tab.
- Click on the "New pull request" button.
- Select your forked repository and the branch containing your changes.
- Provide a clear and informative title for your pull request, and use the description box to explain the modifications you have made. Your pull request will be closed if you do not specify the changes you've made.
- Finally, click on the "Create pull request" button to submit your changes.
gpg --keyserver 185.125.188.27 --recv-keys 20247EC023F2769E66181C0F581B4A2A862BBADE
or
wget https://github.com/ibnaleem/ibnaleem/blob/main/public_key.asc
The signatures (.asc and .sig) can be found in the /sig directory. Download either of them. Open an issue with the title "invalid signature/wrong signature/corrupt signature" for any issues regarding my signatures.
gpg --sign-key 20247EC023F2769E66181C0F581B4A2A862BBADE
gpg --send-keys 20247EC023F2769E66181C0F581B4A2A862BBADE
If you cannot upload your signature, see "gpg: keyserver receive failed: No route to host"
gpg --verify mixtral.py.asc mixtral.py
Desired output:
gpg: Signature made Tue 6 Feb 10:27:34 2024 GMT
gpg: using RSA key 20247EC023F2769E66181C0F581B4A2A862BBADE
gpg: Good signature from "Ibn Aleem <[email protected]>" [ultimate]