Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gemma 2 LoRA support #2606

Open
Aquasar11 opened this issue Dec 21, 2024 · 0 comments
Open

Gemma 2 LoRA support #2606

Aquasar11 opened this issue Dec 21, 2024 · 0 comments
Assignees
Labels
Investigating Lora/P-tuning triaged Issue has been triaged by maintainers

Comments

@Aquasar11
Copy link

Please add Gemma 2 9b LoRA support. I have tried to build the Tensorrt-llm engine, and when I set the --lora_plugin auto or any other type, it gives me this error

Traceback (most recent call last):
  File "/usr/local/bin/trtllm-build", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/commands/build.py", line 602, in main
    parallel_build(model_config, ckpt_dir, build_config, args.output_dir,
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/commands/build.py", line 425, in parallel_build
    passed = build_and_save(rank, rank % workers, ckpt_dir,
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/commands/build.py", line 390, in build_and_save
    engine = build_model(build_config,
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/commands/build.py", line 383, in build_model
    return build(model, build_config)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/builder.py", line 1096, in build
    model = optimize_model_with_config(model, build_config)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/builder.py", line 828, in optimize_model_with_config
    model.use_lora(build_config.lora_config)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/top_model_mixin.py", line 54, in use_lora
    raise NotImplementedError("Subclass shall override this")
NotImplementedError: Subclass shall override this
@github-actions github-actions bot added triaged Issue has been triaged by maintainers Investigating labels Dec 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Investigating Lora/P-tuning triaged Issue has been triaged by maintainers
Projects
None yet
Development

No branches or pull requests

3 participants