[Tracking] ROCm packages #197885

Madouura · 2022-10-26T10:38:47Z

Madouura · 2022-10-30T15:11:45Z

Updating to 5.3.1, marking all WIP until pushed to their respective PRs and verified.

Madouura · 2022-10-30T17:33:26Z

~~If anyone is interested in helping me debug rocBLAS, here's the current derivation~~
Already fixed.

Flakebi · 2022-10-31T09:46:25Z

Hi, thanks a lot for your work on ROCm packages!

So far, the updates where all aggregated in a single rocm: 5.a.b -> 5.x.y pr. I think that makes more sense than splitting the package updates into single prs for a couple of reasons:

Often, packages have backward- (and forward-) incompatible changes, i.e. a the 5.3.0 version of rocm-runtime only works with 5.3.0 of rocm-comgr, but not with 5.2.0 or 5.4.0 (made up example).
Nobody tests a mixture of versions, i.e. only all packages at the same version are known to work.
If I want to test hip, OpenCL and other things for an update, it’s easier to do it one time (and compile everything a single time), rather than 10 times.

tl;dr, do you mind merging all your 5.3.1 updates into a single PR?

PS: Not sure how you did the update, I usually do it with for f in rocm-smi rocm-cmake rocm-thunk rocm-runtime rocm-opencl-runtime rocm-device-libs rocm-comgr rocclr rocminfo llvmPackages_rocm.llvm hip; nix-shell maintainers/scripts/update.nix --argstr commit true --argstr package $f; end.

Madouura · 2022-10-31T09:50:00Z

I was actually afraid of the opposite being true so I split them up.
Got it, I'll aggregate them.
Thanks for the tip on the update script, that would have saved me a lot of time.

Madouura · 2022-10-31T09:51:08Z

Hip I think should stay separate though, since there are other changes.
Actually never mind it's just an extra dependency so should be fine to split it.

Madouura · 2023-12-17T09:49:38Z

ROCm 6.0.0 has been released.
rocmPackages_5 is now in maintenance-mode.
I will eventually backport the changes I am making with rocmPackages_6 to rocmPackages_5, however it is not a high priority.

kurnevsky · 2023-12-17T22:26:52Z

By setting environment variable manually

Interesting - now pytorch works for me, but it doesn't seem to work correctly. I'm trying to generate an image from sdxl+lora with diffusers, and it generates an incorrect image...

I tried identical code and model with manually defined seeds in google colab with cuda - it works there. Also seems to work locally on cpu with f32 types.

(or it might be some problem in one of the libs, since locally I use all python libs from nix)

sersorrel · 2024-02-07T23:28:44Z

The export LD_LIBRARY_PATH=/nix/store/...-clr-5.7.1/lib solution fixed the same torchWithRocm problem for me, also with a 7900 XTX. I couldn't see how you got that path – it's returned by nix build --print-out-paths nixpkgs#rocmPackages.clr, right?

ScatteredRay · 2024-02-16T02:09:18Z

Hey, giving this a try. Still very much WIP, but it's working so far for my current project.

dwf · 2024-03-21T23:16:19Z

@Madouura First, thanks for all your work on this front.

You left a comment to the effect that rocBLASLt is "Very broken with Tensile at the moment, only supports GFX9". It looks like other platforms might be supported now, but I wondered if you might be able to elaborate with the "very broken with Tensile" part. I notice that they ship a vendored "Tensilelite", was that what you were trying to use?

Any pointers you have on how I might manage to build this would be useful. I'm currently eyeing the rocBLAS derivation as a potentially good starting point.

Edit: no longer a priority for me

yshui · 2024-04-01T16:13:59Z

pytorch now fails to build after 5 -> 6 transition, because it depends on miopengemm which was removed.

SomeoneSerge · 2024-04-09T23:06:31Z

I edited the description to add an entry for rocblaslt. It's, apparently, a dependency for zluda

jalil-salame · 2024-05-19T17:53:14Z

Apparently pytorch now requires hipBLASLt:

python3.11-torch> CMake Error at cmake/public/LoadHIP.cmake:37 (find_package):
python3.11-torch>   By not providing "Findhipblaslt.cmake" in CMAKE_MODULE_PATH this project
python3.11-torch>   has asked CMake to find a package configuration file provided by
python3.11-torch>   "hipblaslt", but CMake did not find one.
python3.11-torch>   Could not find a package configuration file provided by "hipblaslt" with
python3.11-torch>   any of the following names:
python3.11-torch>     hipblasltConfig.cmake
python3.11-torch>     hipblaslt-config.cmake
python3.11-torch>   Add the installation prefix of "hipblaslt" to CMAKE_PREFIX_PATH or set
python3.11-torch>   "hipblaslt_DIR" to a directory containing one of the above files.  If
python3.11-torch>   "hipblaslt" provides a separate development package or SDK, be sure it has
python3.11-torch>   been installed.
python3.11-torch> Call Stack (most recent call first):
python3.11-torch>   cmake/public/LoadHIP.cmake:160 (find_package_and_print_version)
python3.11-torch>   cmake/Dependencies.cmake:1258 (include)
python3.11-torch>   CMakeLists.txt:754 (include)
python3.11-torch>
python3.11-torch> -- Configuring incomplete, errors occurred!

ony · 2024-06-16T06:34:30Z

As per pytorch/pytorch#119081 (comment) in 2.4.0+ (future release) it should be possible to use something like:

  pythonPackagesExtensions = prev.pythonPackagesExtensions ++ [
    (python-final: python-prev: {
      torch = python-prev.torch.overrideDerivation (oldAttrs: {
        TORCH_BLAS_PREFER_HIPBLASLT = 0;  # not yet in nixpkgs
      });
    })
  ];

AngryLoki · 2024-06-20T01:31:40Z

@ony , TORCH_BLAS_PREFER_HIPBLASLT is environment variable for runtime; pytorch still links and requires hipblaslt, even when unused. pytorch/pytorch#120551 should help, but I have no idea whether and when it could be accepted.

By the way, hipblaslt is not difficult to build. Just don't build 6.0 release, skip directly to 6.1. When I tried, bundled TensileLine in 6.0 generated wall of unreadable errors, while 6.1 worked from first attempt.

nixos-discourse · 2024-07-11T23:15:50Z

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/testing-gpu-compute-on-amd-apu-nixos/47060/4

DerDennisOP · 2024-10-16T14:22:18Z

I'm not able to build rocmlir-rock-6.0.2, when trying to install zluda.

FAILED: mlir/lib/Dialect/Rock/Transforms/CMakeFiles/obj.MLIRRockTransforms.dir/ViewToTransform.cpp.o
/nix/store/16pvlpl13g06f1rqxp7z0il9i4l9mlww-rocm-llvm-clang-wrapper-6.0.2/bin/clang++ -DGTEST_HAS_RTTI=0 -D_DEBUG -D_GLIBCXX_ASSERTIONS -D_LIBCPP_ENABLE_ASSERTIONS -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/
build/source/build/mlir/lib/Dialect/Rock/Transforms -I/build/source/mlir/lib/Dialect/Rock/Transforms -I/build/source/external/llvm-project/llvm/include -I/build/source/build/external/llvm-project/llvm/include -I/build/source/external/llv
m-project/mlir/include -I/build/source/build/external/llvm-project/llvm/tools/mlir/include -I/build/source/external/mlir-hal/mlir/include -I/build/source/build/external/mlir-hal/include -I/build/source/external/mlir-hal/include -I/build/
source/mlir/include -I/build/source/build/mlir/include -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wm
issing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmislead
ing-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused
-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsugg
est-override -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -Werror=global-constructors -O3 -DNDEBUG -std=gnu++17 -fPIC   -D_DEBUG -D_GLIBCXX_ASSERTIONS -D_LI
BCPP_ENABLE_ASSERTIONS -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -D_DEBUG -D_GLIBCXX_ASSERTIONS -D_LIBCPP_ENABLE_ASSERTIONS -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS  -fno-exception
s -funwind-tables -fno-rtti -UNDEBUG -MD -MT mlir/lib/Dialect/Rock/Transforms/CMakeFiles/obj.MLIRRockTransforms.dir/ViewToTransform.cpp.o -MF mlir/lib/Dialect/Rock/Transforms/CMakeFiles/obj.MLIRRockTransforms.dir/ViewToTransform.cpp.o.d
-o mlir/lib/Dialect/Rock/Transforms/CMakeFiles/obj.MLIRRockTransforms.dir/ViewToTransform.cpp.o -c /build/source/mlir/lib/Dialect/Rock/Transforms/ViewToTransform.cpp
In file included from /build/source/mlir/lib/Dialect/Rock/Transforms/ViewToTransform.cpp:14:
/build/source/mlir/include/mlir/Conversion/TosaToRock/TosaToRock.h:21:10: fatal error: 'mlir/Conversion/RocMLIRPasses.h.inc' file not found
#include "mlir/Conversion/RocMLIRPasses.h.inc"
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Is there an easy fix for it?

AngryLoki · 2024-10-26T20:39:06Z

@DerDennisOP , it was addressed in pull-request ROCm/rocMLIR#1640 (issue ROCm/rocMLIR#1620), you may want use it.

ilylily · 2024-10-26T20:42:06Z

@DerDennisOP @AngryLoki i think you'll actually also need ROCm/rocMLIR#1542 (closes ROCm/rocMLIR#1500). similar patch in a nearby file

arunoruto · 2024-12-11T21:58:12Z

@DerDennisOP , it was addressed in pull-request ROCm/rocMLIR#1640 (issue ROCm/rocMLIR#1620), you may want use it.

@DerDennisOP @AngryLoki i think you'll actually also need ROCm/rocMLIR#1542 (closes ROCm/rocMLIR#1500). similar patch in a nearby file

Is there a plan to patch these things in upstream too? As far as I can see, the hydro logs show the same error as @DerDennisOP.
Would a general update to a newer version resolve this problem? Maybe updating everything to 6.2.4 or even 6.3.0 would be feasible.

mschwaig · 2024-12-11T23:01:57Z

Right now I do not have the time to update ROCm, but I could help out as a reviewer.
I think updating to a newer version would be a great idea.

arunoruto · 2024-12-11T23:14:00Z

Right now I do not have the time to update ROCm, but I could help out as a reviewer. I think updating to a newer version would be a great idea.

While I do not have that much experience with ROCm, I could try it.
I was also wondering, there are a lot of PRs from the autoupdater @r-ryantm, for example #358391. If the nixpkgs-review succeeds on these PRs, would they be directly viable for a marge? Do we need to bring all the packages up to date at the same time? Are there more things to consider?

bgamari · 2024-12-12T00:19:51Z

Would a general update to a newer version resolve this problem? Maybe updating everything to 6.2.4 or even 6.3.0 would be feasible.

FWIW, I have a branch where I have tried updating things to 6.2.4. Unfortunately, I am seeing linking failures in the clang (specifically libcxx) bootstrap that I doubt I will have time to fix in the near-term.

I was also wondering, there are a lot of PRs from the autoupdater @r-ryantm,

I don't think any of those PRs are viable. Apart from the MRs themselves not building, the auto-updater generally doesn't seem to respect the fact that ROCM components expect to be upgraded in lock-step.

bgamari · 2024-12-12T02:05:20Z

FWIW, I have opened a draft MR to record the state of my attempt: #364423

LunNova · 2024-12-12T15:27:35Z

I have a mix of 6.3 and 6.2 working here with pytorch nightly but in no state to upstream. Might be helpful for someone with more time trying to fix it in nixpkgs.

https://github.com/LunNova/ml.nix

Madouura added 9.needs: package (new) This needs a new package to be added 6.topic: hardware labels Oct 26, 2022

Madouura self-assigned this Oct 26, 2022

Madouura changed the title ~~[Tracking] ROCM packages~~ [Tracking] ROCm packages Oct 26, 2022

Madouura mentioned this issue Oct 31, 2022

rocm-related: 5.3.0 -> 5.3.1 & aggregate #198770

Merged

13 tasks

Madouura mentioned this issue Dec 17, 2023

rocmPackages_5: disallow further version upgrades, fixup links #274980

Merged

13 tasks

SomeoneSerge mentioned this issue Dec 26, 2023

rocmPackages: drop symlinks & split outputs #276846

Open

ca5ua1 mentioned this issue Jan 31, 2024

Update request: rocmPackages.hipblas 5.7.1 → 6.0.0 #285279

Closed

1 task

mschwaig mentioned this issue Feb 10, 2024

rocmPackages.* 5.7.1→ 6.0.2 #287846

Merged

13 tasks

ScatteredRay mentioned this issue Feb 16, 2024

rocmPackages_6: init at 6.0.2(?) #289187

Closed

13 tasks

dwf mentioned this issue Mar 30, 2024

rocmPackages_[56].miopen tests fail to build, rocmPackages_6.miopen is broken #299156

Open

dwf mentioned this issue Apr 8, 2024

Update request: jaxlib with ROCm support #302676

Open

errnoh mentioned this issue Apr 10, 2024

ZLUDA: init at 3 #288644

Merged

13 tasks

samueldr added the 5. scope: tracking Long-lived issue tracking long-term fixes or multiple sub-problems label Apr 23, 2024

SomeoneSerge mentioned this issue Oct 16, 2024

comfyui: init #268378

Open

13 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Tracking] ROCm packages #197885

[Tracking] ROCm packages #197885

Madouura commented Oct 26, 2022 •

edited by SomeoneSerge

Loading

Madouura commented Oct 30, 2022 •

edited

Loading

Madouura commented Oct 30, 2022 •

edited

Loading

Flakebi commented Oct 31, 2022

Madouura commented Oct 31, 2022

Madouura commented Oct 31, 2022 •

edited

Loading

Madouura commented Dec 17, 2023 •

edited

Loading

kurnevsky commented Dec 17, 2023 •

edited

Loading

sersorrel commented Feb 7, 2024

ScatteredRay commented Feb 16, 2024

dwf commented Mar 21, 2024 •

edited

Loading

yshui commented Apr 1, 2024

SomeoneSerge commented Apr 9, 2024

jalil-salame commented May 19, 2024

ony commented Jun 16, 2024

AngryLoki commented Jun 20, 2024

nixos-discourse commented Jul 11, 2024

DerDennisOP commented Oct 16, 2024

AngryLoki commented Oct 26, 2024

ilylily commented Oct 26, 2024

arunoruto commented Dec 11, 2024

mschwaig commented Dec 11, 2024

arunoruto commented Dec 11, 2024 •

edited

Loading

bgamari commented Dec 12, 2024 •

edited

Loading

bgamari commented Dec 12, 2024

LunNova commented Dec 12, 2024

[Tracking] ROCm packages #197885

[Tracking] ROCm packages #197885

Comments

Madouura commented Oct 26, 2022 • edited by SomeoneSerge Loading

Key

WIP

Ready

TODO

Merged

ROCm-related

Notes

Won't implement

Madouura commented Oct 30, 2022 • edited Loading

Madouura commented Oct 30, 2022 • edited Loading

Flakebi commented Oct 31, 2022

Madouura commented Oct 31, 2022

Madouura commented Oct 31, 2022 • edited Loading

Madouura commented Dec 17, 2023 • edited Loading

kurnevsky commented Dec 17, 2023 • edited Loading

sersorrel commented Feb 7, 2024

ScatteredRay commented Feb 16, 2024

dwf commented Mar 21, 2024 • edited Loading

yshui commented Apr 1, 2024

SomeoneSerge commented Apr 9, 2024

jalil-salame commented May 19, 2024

ony commented Jun 16, 2024

AngryLoki commented Jun 20, 2024

nixos-discourse commented Jul 11, 2024

DerDennisOP commented Oct 16, 2024

AngryLoki commented Oct 26, 2024

ilylily commented Oct 26, 2024

arunoruto commented Dec 11, 2024

mschwaig commented Dec 11, 2024

arunoruto commented Dec 11, 2024 • edited Loading

bgamari commented Dec 12, 2024 • edited Loading

bgamari commented Dec 12, 2024

LunNova commented Dec 12, 2024

Madouura commented Oct 26, 2022 •

edited by SomeoneSerge

Loading

Madouura commented Oct 30, 2022 •

edited

Loading

Madouura commented Oct 30, 2022 •

edited

Loading

Madouura commented Oct 31, 2022 •

edited

Loading

Madouura commented Dec 17, 2023 •

edited

Loading

kurnevsky commented Dec 17, 2023 •

edited

Loading

dwf commented Mar 21, 2024 •

edited

Loading

arunoruto commented Dec 11, 2024 •

edited

Loading

bgamari commented Dec 12, 2024 •

edited

Loading