XLA_TARGET=rocm compilation failed with "crosstool_wrapper_driver_is_not_gcc failed" #68
Comments
|
Hey @Awlexus, this could be an issue with the build environment. To be sure, you can alternatively use the Docker scripts ( In case your GPU uses gfx1100 (7900 XTX), you may need to use a more recent XLA revision as per #63 (comment) (either by setting |
|
Thanks @jonatanklosko, I was able to compile it by using a a more recently xla git ref, but I could not get it to start GPU. I tried again by using the docker script to build it (which took a long time) and experienced the same error. It was able to allocate the memory, but the program would soon after be stopped by the operating system. Not sure where exactly this error comes from. Error log
|
|
Hmm, do you do any Nx stuff on boot? Does the error happen every time? I assume it doesn't happen if you use CPU only? You can also try |
|
Sorry for the late reply, I was away for a bit. I'm not sure what changed since then, but now I'm getting a different error message. I already tried to write out a reply, before I noticed the change, so I added it at the end in case it could be helpful. I now ran into the error message Now I'm running into the following error that soon afterwards causes the OS to send a Old Reply
I've added a serving of openai/whisper to my application's supervision tree, but that should be all {:ok, model_info} = Bumblebee.load_model({:hf, @whisper_model})
{:ok, featurizer} = Bumblebee.load_featurizer({:hf, @whisper_model})
{:ok, tokenizer} = Bumblebee.load_tokenizer({:hf, @whisper_model})
{:ok, generation_config} = Bumblebee.load_generation_config({:hf, @whisper_model})
generation_config = Bumblebee.configure(generation_config, max_new_tokens: 100)
serving =
Bumblebee.Audio.speech_to_text_whisper(
model_info,
featurizer,
tokenizer,
generation_config,
compile: [batch_size: 4],
chunk_num_seconds: 30,
stream: true,
defn_options: [compiler: EXLA]
)
Yes, it happens every time, before the serving is able to complete a single run |
|
Hmm, this looks like |
|
I'm running Arch Linux and rely on the packages provided there, so I'm not sure what I could be missing. I have installed every package that pops up when I search for rocm, but just to be sure I've provided a list of the installed packages below.
$ ls -lah /opt
drwxr-xr-x 34 root root 4.0K Dec 31 18:53 rocm/Installed rocm packages
|
|
I see. It must be something environment related, given that others managed to run it with that revision, but I don't have any more guesses right now. One alternative would be running stuff inside Docker, though that's not exactly convenient. Or you could try building with the latest openxla revision to see if it's something fixed upstream, but note that this usually requires some adjustments in the build file or/and in exla (depending on how much the xla APIs changed). |
|
We just had a new release, see #82 (comment). You can try it with ROCm 6.0, and if there are issues, leave a comment on #82 :) |
Hi, I've been trying to get GPU support running, but I keep running into this issue. I was first looking at this issue to get it running. I added the dependencies like this:
I made sure to install the dependencies mentioned in this comment (adjusted for arch linux):
$ sudo pacman -S miopen-hip hipfft rocrand \ hipsparse hipsolver hipsparse rccl hip-runtime-amd \ rocfft roctracer hipblas rocm-device-libs rocsolver rocblasAnd then tried to compile it with
$ XLA_BUILD=true XLA_TARGET=rocm mix compileCompilation logs
The text was updated successfully, but these errors were encountered: