Our project, AutoChart, is inspired by the critical challenges faced by the Emergency and Family Medicine Departments, characterized by a severe manpower shortage leading to prolonged patient wait times and doctor burnout. AutoChart aims to alleviate these pressures by streamlining medical documentation processes.
The heartbreak over manpower shortages in critical healthcare departments motivated us. These shortages result in patients enduring long wait times in discomfort and doctors experiencing extreme burnout, primarily due to the overwhelming burden of charting and medical documentation.
Our goal is to lighten doctors' workloads and expedite patient treatment with our innovative solution.
AutoChart transforms doctor-patient conversations into transcribed, well-organized, medically precise charts. This process involves:
- Transcribing the conversation using OpenAI's Whisper model.
- Organizing the transcription into a structured medical chart with the help of OpenAI's GPT-4 model.
Our development process included:
- Leveraging OpenAI's Whisper Model: Fine-tuned for Singaporean English using the Singapore ASR dataset.
- Utilizing OpenAI's GPT-4: For formatting the conversation into a medical chart.
- The size of the ASR dataset was formidable at 1.5TB, leading us to use only a small sample.
- Data transformation required converting pure audio data into an amplitude array at a 16000Hz sampling rate, demanding significant computational resources.
- Extended training time for the Whisper model due to its complexity and the high computational resources required.
- Successful Application Implementation: Hosted on Huggingface Spaces.
- Effective Training: Achieved with the Whisper model.
Our journey taught us valuable lessons in audio data processing, including:
- Transforming audio data into an amplitude array.
- Utilizing the log-mel spectrogram to reduce audio data dimensionality.
- Fine-tuning the Whisper model for enhanced audio data interpretation.
Moving forward, we aim to:
- Collaborate with Medical Professionals: To refine the document's precision.
- Explore Integration Opportunities: In hospitals and clinical settings.
- Pursue Regulatory Approvals: Such as FDA/ISO13485 certifications, to facilitate medical industry adoption.
AutoChart represents a significant step forward in addressing the pressing challenges within the healthcare documentation process. By leveraging advanced AI technologies, we envision a future where healthcare professionals can focus more on patient care and less on administrative tasks.
The link to the huggingface repo is here, it is set to private as it is using my personal openAI api so if you would like to try the app for yourself please contact me.
Sample conversation used in the demo
This project aims to fine-tune OpenAI's Whisper model to better understand and transcribe Singaporean English. We utilize the National Speech Corpus provided by IMDA for our dataset.
The preprocessing stage involves downloading files from a Dropbox link specified in the code. This link will vary depending on the specific section of the dataset being used. To initiate the preprocessing, run the main file under the preprocessing-pipeline
directory. The details of this process can be viewed in the DataProcessingPipeline.ipynb
notebook in the repository.
After preprocessing, we move on to the training pipeline. The training process is conducted on a fraction of the dataset for demonstration purposes. The details of this process can be viewed in the Training_pipeline.ipynb
notebook in the repository.
The evaluations of the model and detailed training metrics can be found on our Hugging Face repository: Hugging Face Repository - whisperfinetune_modelcheckpoints. This includes performance metrics, model checkpoints, and other relevant evaluation data.
- Ensure you have Python installed on your system.
- The code is designed to run with Google Drive mounted on macOS. If you're using a different operating system, modifications might be necessary.
Before running the project, you'll need to install the necessary dependencies.
-
Clone the repository to your local machine.
-
Navigate to the repository's preprocessing-pipeline directory in the command line.
-
Run the main script with your access token:
python main.py 'YOUR_ACCESS_TOKEN'
Replace
'YOUR_ACCESS_TOKEN'
with your actual access token for the Dropbox API.
The current implementation is tailored for macOS with Google Drive integration. If you are using a different operating system or cloud storage solution, you will need to adjust the file paths and storage configurations accordingly.
WhisperFineTune Transcription App
Feel free to test the app with your own audio samples to see how effectively the model transcribes Singaporean English.