feat: Add ability to configure OpenAI base URL in ChatGPTAgentConfig

[celmore25]

Motivation

To allow for extremely low latency interactions over voice with LLMs, Groq currently has the highest throughput models which are publicly available. By allowing for a custom OpenAI API compatible endpoint, users can take advantage of a broader set of hardware and model providers.

Changes

• Added base_url parameter to ChatGPTAgentConfig to allow customization of the OpenAI API base URL.
• Updated instantiate_openai_client function to use the base_url parameter from the configuration.
• Modified ChatGPTAgent to utilize the updated instantiate_openai_client function.
• Modified the get_tokenizer_info function to allow for llama model usage with custom base URLs. This estimates the token usage and is not exact. Future feature expansion would be needed here to allow more models with certain token counting.
• Added tests to verify the new base_url functionality in tests/streaming/agent/test_base_agent.py.

This enhancement allows users to specify a custom OpenAI API base URL, providing greater flexibility in agent configuration.

Usage

Example end to end example:

OPENAI_MODEL_NAME="llama3-70b-8192"
OPENAI_BASE_URL="https://api.groq.com/openai/v1"
OPENAI_API_KEY="<key here>"
ELEVENLABS_API_KEY="<key here>"
ELEVENLABS_VOICE_ID="<ID here>"
DEEPGRAM_API_KEY="<key here>"

import asyncio
import signal
import os

from pydantic_settings import BaseSettings, SettingsConfigDict

from vocode.helpers import create_streaming_microphone_input_and_speaker_output
from vocode.logging import configure_pretty_logging
from vocode.streaming.agent.chat_gpt_agent import ChatGPTAgent
from vocode.streaming.models.agent import ChatGPTAgentConfig
from vocode.streaming.models.message import BaseMessage
from vocode.streaming.models.synthesizer import ElevenLabsSynthesizerConfig
from vocode.streaming.models.transcriber import (
    DeepgramTranscriberConfig,
    PunctuationEndpointingConfig,
)
from vocode.streaming.streaming_conversation import StreamingConversation
from vocode.streaming.synthesizer.eleven_labs_synthesizer import ElevenLabsSynthesizer
from vocode.streaming.transcriber.deepgram_transcriber import DeepgramTranscriber
from vocode.streaming.models.audio import AudioEncoding

configure_pretty_logging()

class Settings(BaseSettings):
    """
    Settings for the streaming conversation quickstart.
    These parameters can be configured with environment variables.
    """

    openai_api_key: str = os.environ.get('OPENAI_API_KEY', '')
    openai_model_name: str = os.environ.get('OPENAI_MODEL_NAME', '')
    openai_base_url: str = os.environ.get('OPENAI_BASE_URL', '')
    elevenlabs_api_key: str = os.environ.get('ELEVENLABS_API_KEY', '')
    elevenlabs_voice_id: str = os.environ.get('ELEVENLABS_VOICE_ID', '')
    deepgram_api_key: str = os.environ.get('DEEPGRAM_API_KEY', '')

    # This means a .env file can be used to overload these settings
    # ex: "OPENAI_API_KEY=my_key" will set openai_api_key over the default above
    model_config = SettingsConfigDict(
        env_file=".env",
        env_file_encoding="utf-8",
        extra="ignore"
    )


settings = Settings()


async def main():
    (
        microphone_input,
        speaker_output,
    ) = create_streaming_microphone_input_and_speaker_output(
        use_default_devices=False,
        speaker_sampling_rate=16000,
    )

    conversation = StreamingConversation(
        output_device=speaker_output,
        transcriber=DeepgramTranscriber(
            DeepgramTranscriberConfig.from_input_device(
                microphone_input,
                endpointing_config=PunctuationEndpointingConfig(),
                api_key=settings.deepgram_api_key,
            ),
        ),
        agent=ChatGPTAgent(
            ChatGPTAgentConfig(
                openai_api_key=settings.openai_api_key,
                model_name=settings.openai_model_name,
                base_url_override=settings.openai_base_url,
                initial_message=BaseMessage(text="What up"),
                prompt_preamble="""The AI is having a pleasant conversation about life""",
            )
        ),
        synthesizer=ElevenLabsSynthesizer(
            ElevenLabsSynthesizerConfig(
                api_key=settings.elevenlabs_api_key,
                voice_id=settings.elevenlabs_voice_id,
                sampling_rate=16000,
                audio_encoding=AudioEncoding.LINEAR16
            )
        ),
    )
    await conversation.start()
    print("Conversation started, press Ctrl+C to end")
    signal.signal(signal.SIGINT, lambda _0, _1: asyncio.create_task(conversation.terminate()))
    while conversation.is_active():
        chunk = await microphone_input.get_audio()
        conversation.receive_audio(chunk)


if __name__ == "__main__":
    asyncio.run(main())

[ajar98]

@celmore25 this is an awesome change! should be simple to get in my suggestion and then let's get this in

[ajar98]

let's either name this base_url_override or default it to "https://api.openai.com/v1" — i'd prefer the former since it changes the code less

[celmore25]

Hi @ajar98 nice to meet you! Thanks for looking over this.

I just pushed some changes to go to the override option. Let me know if anything else needs to get changed!

[ajar98]

thanks @celmore25 ! Just had to fix the test

BTW - we have inbuilt support for groq - see

vocode-core/vocode/streaming/agent/groq_agent.py

Line 29 in f1d688b

class GroqAgent(RespondAgent[GroqAgentConfig]):

still think it's quite useful to support any vLLM compatible API with this change!

[arpagon]

Great, this will allow you to use Ollama, for example.