[Feature Request]: add azure model to llama_index #16090
Comments
szcc
added
enhancement
|
I found a similar open issue regarding adding support for the To add support for the models GPT4_MODELS: Dict[str, int] = {
# stable model names:
# resolves to gpt-4-0314 before 2023-06-27,
# resolves to gpt-4-0613 after
"gpt-4": 8192,
"gpt-4-32k": 32768,
# turbo models (Turbo, JSON mode)
"gpt-4-1106-preview": 128000,
"gpt-4-0125-preview": 128000,
"gpt-4-turbo-preview": 128000,
# multimodal model
"gpt-4-vision-preview": 128000,
"gpt-4-1106-vision-preview": 128000,
"gpt-4-turbo-2024-04-09": 128000,
"gpt-4-turbo": 128000,
"gpt-4o": 128000,
"gpt-4o-2024-05-13": 128000,
"gpt-4o-2024-08-06": 128000,
"gpt-4o-mini": 128000,
"gpt-4o-mini-2024-07-18": 128000,
# 0613 models (function calling):
# https://openai.com/blog/function-calling-and-other-api-updates
"gpt-4-0613": 8192,
"gpt-4-32k-0613": 32768,
# 0314 models
"gpt-4-0314": 8192,
"gpt-4-32k-0314": 32768,
# New models
"azure-gpt-4o": 128000,
"azure-gpt-4-turbo-20240409": 128000,
"azure-gpt-4-turbo-preview": 128000,
"azure-gpt-4": 8192,
}
AZURE_TURBO_MODELS: Dict[str, int] = {
"gpt-4o": 128000,
"gpt-35-turbo-16k": 16384,
"gpt-35-turbo": 4096,
# 0125 (2024) model (JSON mode)
"gpt-35-turbo-0125": 16384,
# 1106 model (JSON mode)
"gpt-35-turbo-1106": 16384,
# 0613 models (function calling):
"gpt-35-turbo-0613": 4096,
"gpt-35-turbo-16k-0613": 16384,
# New models
"azure-gpt-4o": 128000,
"azure-gpt-4-turbo-20240409": 128000,
"azure-gpt-4-turbo-preview": 128000,
"azure-gpt-4": 8192,
}
GPT3_MODELS: Dict[str, int] = {
"text-ada-001": 2049,
"text-babbage-001": 2040,
"text-curie-001": 2049,
"ada": 2049,
"babbage": 2049,
"curie": 2049,
"davinci": 2049,
# New model
"text-embedding-ada-002": 2049,
}
ALL_AVAILABLE_MODELS = {
**O1_MODELS,
**GPT4_MODELS,
**TURBO_MODELS,
**GPT3_5_MODELS,
**GPT3_MODELS,
**AZURE_TURBO_MODELS,
}
CHAT_MODELS = {
**O1_MODELS,
**GPT4_MODELS,
**TURBO_MODELS,
**AZURE_TURBO_MODELS,
}Additionally, ensure that the class OpenAIEmbeddingModelType(str, Enum):
"""OpenAI embedding model type."""
DAVINCI = "davinci"
CURIE = "curie"
BABBAGE = "babbage"
ADA = "ada"
TEXT_EMBED_ADA_002 = "text-embedding-ada-002"
TEXT_EMBED_3_LARGE = "text-embedding-3-large"
TEXT_EMBED_3_SMALL = "text-embedding-3-small"
class OpenAIEmbeddingModeModel(str, Enum):
"""OpenAI embedding mode model."""
# davinci
TEXT_SIMILARITY_DAVINCI = "text-similarity-davinci-001"
TEXT_SEARCH_DAVINCI_QUERY = "text-search-davinci-query-001"
TEXT_SEARCH_DAVINCI_DOC = "text-search-davinci-doc-001"
# curie
TEXT_SIMILARITY_CURIE = "text-similarity-curie-001"
TEXT_SEARCH_CURIE_QUERY = "text-search-curie-query-001"
TEXT_SEARCH_CURIE_DOC = "text-search-curie-doc-001"
# babbage
TEXT_SIMILARITY_BABBAGE = "text-similarity-babbage-001"
TEXT_SEARCH_BABBAGE_QUERY = "text-search-babbage-query-001"
TEXT_SEARCH_BABBAGE_DOC = "text-search-babbage-doc-001"
# ada
TEXT_SIMILARITY_ADA = "text-similarity-ada-001"
TEXT_SEARCH_ADA_QUERY = "text-search-ada-query-001"
TEXT_SEARCH_ADA_DOC = "text-search-ada-doc-001"
# text-embedding-ada-002
TEXT_EMBED_ADA_002 = "text-embedding-ada-002"
# text-embedding-3-large
TEXT_EMBED_3_LARGE = "text-embedding-3-large"
# text-embedding-3-small
TEXT_EMBED_3_SMALL = "text-embedding-3-small"This code adds the specified models to the |
|
if models deployed on azure, you can try: |
|
I did not see llama_index.embeddings.azure and llama_index.llms.azure. my llama_index version |
|
@szcc These model names are not listed in azures documentation Should specify gpt-4o or gpt-4, etc. like the error states |
|
Our private LLM name azure-gpt-4o etc. My key is associated with our base url endpoint. if I use gpt-4o then I'll get invalid key error. I am work on adding my model name to the util files as @dosu suggested. Fingers crossed. |
|
To add support for your custom Azure OpenAI models like from llama_index.legacy.multi_modal_llms.azure_openai import AzureOpenAIMultiModal
class CustomAzureOpenAI(AzureOpenAIMultiModal):
def __init__(self, **kwargs):
super().__init__(model="azure-gpt-4o", **kwargs)
# Example usage
aoai_api_key = "YOUR_AZURE_OPENAI_API_KEY"
aoai_endpoint = "YOUR_AZURE_OPENAI_ENDPOINT"
aoai_api_version = "2023-07-01-preview"
llm = CustomAzureOpenAI(
deployment_name="AZURE_AZURE_OPENAI_DEPLOYMENT_NAME",
api_key=aoai_api_key,
azure_endpoint=aoai_endpoint,
api_version=aoai_api_version,
)Additionally, ensure you have the necessary packages installed: %pip install llama-index-embeddings-azure-openai
%pip install llama-index-llms-azure-openaiThis code defines a new class If your private LLM name is Additionally, make sure you have the latest version of |
|
I use CustomAzureOpenAI and got error:AttributeError: 'CustomAzureOpenAI' object has no attribute 'predict' |
|
@dosu I updated the util.py file under python3.8/site-packages/llama_index/llms/openai/util.py and made sure that the text-embedding-ada-002 model was added to the embedding configurations in python3.8/site-packages/llama_index/embeddings/openai/base.py, as you suggested. It works now. Thanks a lot! |
|
To resolve the Here is an example of how you might add a class AzureOpenAIMultiModal(OpenAIMultiModal):
# ... existing code ...
def predict(self, prompt: str) -> str:
client = self._get_client()
response = client.Completion.create(
engine=self.engine,
prompt=prompt,
max_tokens=self.max_new_tokens,
temperature=self.temperature,
**self.additional_kwargs
)
return response.choices[0].text
# ... existing code ...This Regarding your question about checking in the modified files to GitHub, it is generally a good practice to contribute your changes back to the repository if they are useful for others. You can create a pull request with your changes. This way, your updates can be reviewed and potentially merged into the main codebase, ensuring that they persist across future updates. |
|
I'll try to add predict () into AzureOpenAIMultiModal class. Thank you! |
|
You're welcome! Adding the |
|
hi @dosu , TypeError: predict() got an unexpected keyword argument 'query_str' |
|
The error you're encountering, To resolve this, you need to modify the class CustomAzureOpenAI(AzureOpenAIMultiModal):
def __init__(self, **kwargs):
super().__init__(model="azure-gpt-4o", **kwargs)
def predict(self, prompt_template, query_str, schema, dialect, **kwargs) -> str:
client = self._get_client()
prompt = prompt_template.format(query_str=query_str, schema=schema, dialect=dialect)
response = client.Completion.create(
engine=self.engine,
prompt=prompt,
max_tokens=self.max_new_tokens,
temperature=self.temperature,
**self.additional_kwargs
)
return response.choices[0].textThis method now accepts With this change, your Here is how you can use it: db = SQLDatabase.from_uri(mydb_uri)
llm = CustomAzureOpenAI(
api_key="YOUR_AZURE_OPENAI_API_KEY",
azure_endpoint="YOUR_AZURE_OPENAI_ENDPOINT",
api_version="2023-07-01-preview",
deployment_name="AZURE_AZURE_OPENAI_DEPLOYMENT_NAME"
)
query_engine = NLSQLTableQueryEngine(
sql_database=db,
llm=llm
)
query_str = "Which city has the highest population?"
response = query_engine.query(query_str)This should resolve the issue and allow you to execute your query successfully. |
|
Hi @dosu I updated the predict method and error out on I check the class AzureOpenAIMultiModal(OpenAIMultiModal) and found there is _get_clients so I used '_get_clients' instead. But still got error out by :prompt = prompt_template.format(query_str=query_str, schema=schema, dialect=dialect) AttributeError: 'tuple' object has no attribute 'Completion' |
|
The error To fix this, you need to ensure that you are using the correct client object from the tuple returned by Here is the updated class CustomAzureOpenAI(AzureOpenAIMultiModal):
def __init__(self, **kwargs):
super().__init__(model="azure-gpt-4o", **kwargs)
def predict(self, prompt_template, query_str, schema, dialect, **kwargs) -> str:
sync_client, _ = self._get_clients() # Get the synchronous client
prompt = prompt_template.format(query_str=query_str, schema=schema, dialect=dialect)
response = sync_client.Completion.create(
engine=self.engine,
prompt=prompt,
max_tokens=self.max_new_tokens,
temperature=self.temperature,
**self.additional_kwargs
)
return response.choices[0].textThis method now correctly uses the synchronous client from the tuple returned by Here is how you can use it: db = SQLDatabase.from_uri(mydb_uri)
llm = CustomAzureOpenAI(
api_key="YOUR_AZURE_OPENAI_API_KEY",
azure_endpoint="YOUR_AZURE_OPENAI_ENDPOINT",
api_version="2023-07-01-preview",
deployment_name="AZURE_AZURE_OPENAI_DEPLOYMENT_NAME"
)
query_engine = NLSQLTableQueryEngine(
sql_database=db,
llm=llm
)
query_str = "Which city has the highest population?"
response = query_engine.query(query_str)This should resolve the issue and allow you to execute your query successfully. |
|
Hi @dosu Still not working... AttributeError: 'AzureOpenAI' object has no attribute 'Completion' |
|
The error Here is the corrected class CustomAzureOpenAI(AzureOpenAIMultiModal):
def __init__(self, **kwargs):
super().__init__(model="azure-gpt-4o", **kwargs)
def predict(self, prompt_template, query_str, schema, dialect, **kwargs) -> str:
sync_client, _ = self._get_clients() # Get the synchronous client
prompt = prompt_template.format(query_str=query_str, schema=schema, dialect=dialect)
response = sync_client.Completions.create(
engine=self.engine,
prompt=prompt,
max_tokens=self.max_new_tokens,
temperature=self.temperature,
**self.additional_kwargs
)
return response.choices[0].textThis method now correctly uses the Here is how you can use it: db = SQLDatabase.from_uri(mydb_uri)
llm = CustomAzureOpenAI(
api_key="YOUR_AZURE_OPENAI_API_KEY",
azure_endpoint="YOUR_AZURE_OPENAI_ENDPOINT",
api_version="2023-07-01-preview",
deployment_name="AZURE_AZURE_OPENAI_DEPLOYMENT_NAME"
)
query_engine = NLSQLTableQueryEngine(
sql_database=db,
llm=llm
)
query_str = "Which city has the highest population?"
response = query_engine.query(query_str)This should resolve the issue and allow you to execute your query successfully [1]. |
|
hmmm @dosu you provide a same code on predict() for the pass two conversations. So I got same error... |
|
To correctly implement the from llama_index.llms.azure_openai import AzureOpenAI
class CustomAzureOpenAI(AzureOpenAIMultiModal):
def __init__(self, **kwargs):
super().__init__(model="azure-gpt-4o", **kwargs)
def predict(self, prompt_template, query_str, schema, dialect, **kwargs) -> str:
sync_client, _ = self._get_clients() # Get the synchronous client
prompt = prompt_template.format(query_str=query_str, schema=schema, dialect=dialect)
response = sync_client.chat.completions.create(
messages=[{"role": "user", "content": prompt}]
)
return response.choices[0].message.contentThis implementation ensures that the |
|
@dosu TypeError: Missing required arguments; Expected either ('messages' and 'model') or ('messages', 'model' and 'stream') arguments to be given |
|
To resolve the from llama_index.llms.azure_openai import AzureOpenAI
class CustomAzureOpenAI(AzureOpenAIMultiModal):
def __init__(self, **kwargs):
super().__init__(model="azure-gpt-4o", **kwargs)
def predict(self, prompt_template, query_str, schema, dialect, **kwargs) -> str:
sync_client, _ = self._get_clients() # Get the synchronous client
prompt = prompt_template.format(query_str=query_str, schema=schema, dialect=dialect)
response = sync_client.chat.completions.create(
messages=[{"role": "user", "content": prompt}],
model=self.model, # Ensure the model argument is included
stream=False,
**self.additional_kwargs
)
return response.choices[0].message.contentThis implementation ensures that the |
Feature Description
Hi @dosu and @logan-markewich ,
We have 'azure' model when I put our model 'azure-gpt-4o' llm = OpenAI(temperature=0.1, model="azure-gpt-4o"). I got error:
ValueError: Unknown model 'azure-gpt-4'. Please provide a valid OpenAI model name in: gpt-4, gpt-4-32k, gpt-4-1106-preview, gpt-4-0125-preview, gpt-4-turbo-preview, gpt-4-vision-preview, gpt-4-1106-vision-preview, gpt-4-turbo-2024-04-09, gpt-4-turbo, gpt-4o, gpt-4o-2024-05-13, gpt-4o-2024-08-06, gpt-4o-mini, gpt-4o-mini-2024-07-18, gpt-4-0613, gpt-4-32k-0613, gpt-4-0314, gpt-4-32k-0314, gpt-3.5-turbo, gpt-3.5-turbo-16k, gpt-3.5-turbo-0125, gpt-3.5-turbo-1106, gpt-3.5-turbo-0613, gpt-3.5-turbo-16k-0613, gpt-3.5-turbo-0301, text-davinci-003, text-davinci-002, gpt-3.5-turbo-instruct, text-ada-001, text-babbage-001, text-curie-001, ada, babbage, curie, davinci, gpt-35-turbo-16k, gpt-35-turbo, gpt-35-turbo-0125, gpt-35-turbo-1106, gpt-35-turbo-0613, gpt-35-turbo-16k-0613
Do you think if ift is a possible for you to add models like :
azure-gpt-4o
azure-gpt-4-turbo-20240409
azure-gpt-4-turbo-preview
azure-gpt-4
text-embedding-ada-002
to llama_index_cord?
Reason
No response
Value of Feature
No response
The text was updated successfully, but these errors were encountered: