Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Start llama.cpp server #18

Open
wants to merge 3 commits into
base: main_v2_2
Choose a base branch
from

Conversation

dnhkng
Copy link
Contributor

@dnhkng dnhkng commented Mar 8, 2024

This PR adds starting the llama.cpp server with parameters taken from the config file.

As the parameters are in the same position as those from the ooba process, the name has been refactored from 'ooba_params' to 'server_params'.

The server is started and stopped using subprocess, but tries to follow a similar style as the code for ooba.

Note: This requires you pull and compile llama.cpp first!

@dnhkng dnhkng marked this pull request as ready for review March 10, 2024 11:26
@sam-paech
Copy link
Contributor

Thanks for this! I want to make a few changes to this before merging:

  • resolve relative paths (~)
  • add same functionality as the ooba class (download model etc)
  • set --ctx-size 1024+, throw error if user sets this arg less than 1024 in config

I'll let you know when I have a chance to start working on this, or if you want to do this please feel free.

@dnhkng
Copy link
Contributor Author

dnhkng commented Mar 12, 2024

- set --ctx-size 1024+, throw error if user sets this arg less than 1024 in config
Seems easy, I can add this.

resolve relative paths (~)
To the model? to the executable?

add same functionality as the ooba class (download model etc)
Probably best if you take care of this, as I'm not sure of the details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants