Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
get preds off gpu to improve perf (#269)
We were leaving the label predictions tensors on the GPU, which was leading to lots of expensive GPU calls to read the data. In particular, `tt profile` revealed the following: ![Screen Shot 2023-07-19 at 11 41 06 AM](https://github.com/allenai/mmda/assets/1287054/9f6736d2-51be-4d65-8bfb-fe93cca9eed7) Scalene's GPU time reporting is generally full of false attribution, but it was a hint in the right direction in this case. Based on my interpretation of the code, we were being forced to access this tensor off the GPU three times for _every_ input word in the document. After pulling the label preds into system memory: ![Screen Shot 2023-07-19 at 11 44 24 AM](https://github.com/allenai/mmda/assets/1287054/5853924a-824b-4818-aeba-fe121d48c9af) I confirmed the remaining high GPU items reported by the profiler have nothing to do with the GPU, and the code is too scary to futz about with anyway.
- Loading branch information