Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: DIA-1362: Add custom LLM endpoint testing scripts #210

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

niklub
Copy link
Contributor

@niklub niklub commented Sep 20, 2024

This PR adds tests to verify Custom LLM endpoint connection.

How to create a custom LLM endpoint

  1. Download and create ollama server
  2. Run local server with ollama run llama3.1
  3. Connect your instance using the following parameters (note /v1 suffix):
    • endpoint: http://localhost:11434/v1
    • deployment name: llama3.1
    • api_key: ollama
  4. (optional) To connect remotely (i.e. when localhost is not available to use), use ngrok or any other proxy server
  5. (optional) To enforce authentication header, use one of the following option:
    • specifying Basic Auth with ngrok. Not that after specifying user:pass string, it must be provided as base64 encoded parameter in the corresponding custom LLM endpoint field. For example, run echo -n "user:pass" | base64, and then use the string: Basic <base64 encoded credentials>
    • use auth_proxy_server.py script launching it with: TARGET_URL=http://localhost:11434 EXPECTED_HEADER=secret uvicorn auth_proxy_server:app and provide secret as Authentication header field in Custom LLM endpoint connection

"""
app = FastAPI()

TARGET_URL = os.getenv('TARGET_URL')
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a check to ensure TARGET_URL is set. If not, raise an exception or log an error to prevent runtime issues.

df = pd.DataFrame([["I'm happy"], ["I'm sad"], ["I'm neutral"]], columns=["input"])

results = asyncio.run(agent.arun(input=df))
print(results)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add assertions to verify the expected behavior of the agent. Currently, the test only prints results without checking for correctness.

@robot-ci-heartex robot-ci-heartex temporarily deployed to fb-dia-1362 September 20, 2024 10:10 Destroyed
@robot-ci-heartex
Copy link
Collaborator

@robot-ci-heartex robot-ci-heartex marked this pull request as draft September 21, 2024 00:21
@robot-ci-heartex
Copy link
Collaborator

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants