Install dependencies with uv #81
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
edsu
force-pushed
the
uv
branch
3 times, most recently
from
6946ed7 to
e2553aa
Compare
2 tasks
edsu
force-pushed
the
uv
branch
15 times, most recently
from
5b0e19e to
49950f8
Compare
edsu
commented
Jan 31, 2025
.autoupdate/preupdate
Outdated
There was a problem hiding this comment.
This is for Jenkins when it runs dependency updates.
edsu
commented
Jan 31, 2025
.github/workflows/test.yml
Outdated
There was a problem hiding this comment.
Need to run tests using uv now.
edsu
commented
Jan 31, 2025
Dockerfile
Outdated
There was a problem hiding this comment.
Update to running with uv, and a notable change to the base Docker image to get Python3.10 so we can manage Whisper dependencies with uv.
edsu
commented
Jan 31, 2025
There was a problem hiding this comment.
This documentation for the benchmarking process. You can ignore the output files below.
edsu
commented
Feb 1, 2025
docs/report.py
Outdated
There was a problem hiding this comment.
This is a program for generating a report using the baseline transcripts and the latest transcripts in the SDR.
jmartin-sul
approved these changes
Feb 1, 2025
We've been installing Python dependencies with Pip, and not tracking their versions. Since we've started using uv in some other infrastructure team Python projects it makes sense to add here so that speech-to-text can be tracked by infra-team's weekly dependency update process. Unlike pip, uv always installs system specific Python wheels. There wasn't a wheel available for triton (an openai-whisper's dependency) under Python3.8 so the installation of dependencies failed. So, in addition to adding uv this PR also upgrades our base Docker image from `nvidia/cuda:12.1.0-devel-ubuntu20.04` to `nvidia/cuda:12.8.0-cudnn-devel-ubuntu22.04` which allows us to install python3.10 when we `apt install python3`. Since this significantly change Whisper's behavior I wanted to be able to compare the VTT transcript output before and after the Docker image change. I added the start of a benchmarking system that will allow us to compare the output of a set of 22 SDR items, with a previous benchmark. Ideally this benchmark would be human vetted, and actually represent a ground truth for what we believe the transcript should be. But for the time being it is simply a snapshot in time of what the transcript looked like today. See the benchmark/README.md file for details. Closes #80 Refs #65
jmartin-sul
added a commit
to sul-dlss/access-update-scripts
that referenced
this pull request
Feb 7, 2025
it's a python project, but the update hook was already added in sul-dlss/speech-to-text#81 closes sul-dlss/speech-to-text#65
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note: it looks like A LOT of files have changed, but you can ignore the majority of them in
docs/reportsanddocs/baselinewhich are the output of running a benchmark.We've been installing Python dependencies with Pip, and not tracking their versions. Since we've started using uv in some other infrastructure team Python projects it makes sense to add here so that speech-to-text can be tracked by infra-team's weekly dependency update process.
Unlike pip, uv always installs system specific Python wheels. There wasn't a wheel available for triton (an openai-whisper's dependency) under Python3.8 so the installation of dependencies failed.
So, in addition to adding uv this PR also upgrades our base Docker image from
nvidia/cuda:12.1.0-devel-ubuntu20.04tonvidia/cuda:12.8.0-cudnn-devel-ubuntu22.04which allows us to install python3.10 when weapt install python3.Since this could significantly change Whisper's behavior I wanted to be able to compare the VTT transcript output before and after the Docker image change. I added the start of a benchmarking system that will allow us to compare the output of a set of 22 SDR items, with a previous benchmark. Ideally this benchmark would be human vetted, and actually represent a ground truth for what we believe the transcript should be. But for the time being it is simply a snapshot in time of what the transcript looked like today. See the benchmark/README.md file for details.
Closes #80
Refs #65