[Question]: How can I run a correctness evaluator using Azure OpenAI?

[nshern]

Question Validation

• I have searched both the documentation and discord for an answer.

Question

The documentation only states how to do this using a regular OpenAI instance, but how can I recreate this using Azure OpenAI?

[dosubot]

To run a correctness evaluator using Azure OpenAI, you can adapt the example provided for the regular OpenAI instance by using the AzureOpenAI class instead. Here's how you can set it up:

1. Install the necessary packages:

   %pip install llama-index-llms-azure-openai
   !pip install llama-index

2. Set up environment variables:

   import os
   
   os.environ["AZURE_OPENAI_API_KEY"] = "<your-api-key>"
   os.environ["AZURE_OPENAI_ENDPOINT"] = "https://<your-resource-name>.openai.azure.com/"
   os.environ["OPENAI_API_VERSION"] = "2023-07-01-preview"

3. Use the AzureOpenAI class:

   from llama_index.llms.azure_openai import AzureOpenAI
   from llama_index.core.evaluation import CorrectnessEvaluator
   
   llm = AzureOpenAI(
       engine="your-engine-name",
       model="gpt-35-turbo-16k",
       azure_endpoint="https://<your-resource-name>.openai.azure.com/",
       api_key="<your-api-key>",
       api_version="2023-07-01-preview",
   )
   
   evaluator = CorrectnessEvaluator(llm=llm)
   
   query = (
       "Can you explain the theory of relativity proposed by Albert Einstein in"
       " detail?"
   )
   
   reference = """
   Certainly! Albert Einstein's theory of relativity consists of two main components: special relativity and general relativity. Special relativity, published in 1905, introduced the concept that the laws of physics are the same for all non-accelerating observers and that the speed of light in a vacuum is a constant, regardless of the motion of the source or observer. It also gave rise to the famous equation E=mc², which relates energy (E) and mass (m).
   
   General relativity, published in 1915, extended these ideas to include the effects of gravity. According to general relativity, gravity is not a force between masses, as described by Newton's theory of gravity, but rather the result of the warping of space and time by mass and energy. Massive objects, such as planets and stars, cause a curvature in spacetime, and smaller objects follow curved paths in response to this curvature. This concept is often illustrated using the analogy of a heavy ball placed on a rubber sheet, causing it to create a depression that other objects (representing smaller masses) naturally move towards.
   
   In essence, general relativity provided a new understanding of gravity, explaining phenomena like the bending of light by gravity (gravitational lensing) and the precession of the orbit of Mercury. It has been confirmed through numerous experiments and observations and has become a fundamental theory in modern physics.
   """
   
   response = """
   Certainly! Albert Einstein's theory of relativity consists of two main components: special relativity and general relativity. Special relativity, published in 1905, introduced the concept that the laws of physics are the same for all non-accelerating observers and that the speed of light in a vacuum is a constant, regardless of the motion of the source or observer. It also gave rise to the famous equation E=mc², which relates energy (E) and mass (m).
   However, general relativity, published in 1915, extended these ideas to include the effects of magnetism. According to general relativity, gravity is not a force between masses but rather the result of the warping of space and time by magnetic fields generated by massive objects. Massive objects, such as planets and stars, create magnetic fields that cause a curvature in spacetime, and smaller objects follow curved paths in response to this magnetic curvature. This concept is often illustrated using the analogy of a heavy ball placed on a rubber sheet with magnets underneath, causing it to create a depression that other objects (representing smaller masses) naturally move towards due to magnetic attraction.
   """
   
   result = evaluator.evaluate(
       query=query,
       response=response,
       reference=reference,
   )
   
   print(result.score)
   print(result.feedback)

This setup uses the AzureOpenAI class to initialize the language model with Azure-specific configurations and then evaluates the correctness of a response using the CorrectnessEvaluator class.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[nshern]

To run a correctness evaluator using Azure OpenAI, you can adapt the example provided for the regular OpenAI instance by using the AzureOpenAI class instead. Here's how you can set it up:

1. **Install the necessary packages:**
   %pip install llama-index-llms-azure-openai
   !pip install llama-index

2. **Set up environment variables:**
   import os
   
   os.environ["AZURE_OPENAI_API_KEY"] = "<your-api-key>"
   os.environ["AZURE_OPENAI_ENDPOINT"] = "https://<your-resource-name>.openai.azure.com/"
   os.environ["OPENAI_API_VERSION"] = "2023-07-01-preview"

3. **Use the `AzureOpenAI` class:**
   from llama_index.llms.azure_openai import AzureOpenAI
   from llama_index.core.evaluation import CorrectnessEvaluator
   
   llm = AzureOpenAI(
       engine="your-engine-name",
       model="gpt-35-turbo-16k",
       azure_endpoint="https://<your-resource-name>.openai.azure.com/",
       api_key="<your-api-key>",
       api_version="2023-07-01-preview",
   )
   
   evaluator = CorrectnessEvaluator(llm=llm)
   
   query = (
       "Can you explain the theory of relativity proposed by Albert Einstein in"
       " detail?"
   )
   
   reference = """
   Certainly! Albert Einstein's theory of relativity consists of two main components: special relativity and general relativity. Special relativity, published in 1905, introduced the concept that the laws of physics are the same for all non-accelerating observers and that the speed of light in a vacuum is a constant, regardless of the motion of the source or observer. It also gave rise to the famous equation E=mc², which relates energy (E) and mass (m).
   
   General relativity, published in 1915, extended these ideas to include the effects of gravity. According to general relativity, gravity is not a force between masses, as described by Newton's theory of gravity, but rather the result of the warping of space and time by mass and energy. Massive objects, such as planets and stars, cause a curvature in spacetime, and smaller objects follow curved paths in response to this curvature. This concept is often illustrated using the analogy of a heavy ball placed on a rubber sheet, causing it to create a depression that other objects (representing smaller masses) naturally move towards.
   
   In essence, general relativity provided a new understanding of gravity, explaining phenomena like the bending of light by gravity (gravitational lensing) and the precession of the orbit of Mercury. It has been confirmed through numerous experiments and observations and has become a fundamental theory in modern physics.
   """
   
   response = """
   Certainly! Albert Einstein's theory of relativity consists of two main components: special relativity and general relativity. Special relativity, published in 1905, introduced the concept that the laws of physics are the same for all non-accelerating observers and that the speed of light in a vacuum is a constant, regardless of the motion of the source or observer. It also gave rise to the famous equation E=mc², which relates energy (E) and mass (m).
   However, general relativity, published in 1915, extended these ideas to include the effects of magnetism. According to general relativity, gravity is not a force between masses but rather the result of the warping of space and time by magnetic fields generated by massive objects. Massive objects, such as planets and stars, create magnetic fields that cause a curvature in spacetime, and smaller objects follow curved paths in response to this magnetic curvature. This concept is often illustrated using the analogy of a heavy ball placed on a rubber sheet with magnets underneath, causing it to create a depression that other objects (representing smaller masses) naturally move towards due to magnetic attraction.
   """
   
   result = evaluator.evaluate(
       query=query,
       response=response,
       reference=reference,
   )
   
   print(result.score)
   print(result.feedback)

This setup uses the AzureOpenAI class to initialize the language model with Azure-specific configurations and then evaluates the correctness of a response using the CorrectnessEvaluator class.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

The example using openai in the documentation does not require using an engine as a parameter. What do you recommend I put in here, as it is required in your example.

[dosubot]

For the "engine" parameter in the AzureOpenAI class, you should use the "deployment name" of your model in Azure OpenAI Studio. This is necessary for specifying which model deployment to use. Here's an example of how to set it up:

llm = AzureOpenAI(
    engine="your-deployment-name",  # Use your actual deployment name here
    model="gpt-35-turbo-16k",
    azure_endpoint="https://<your-resource-name>.openai.azure.com/",
    api_key="<your-api-key>",
    api_version="2023-07-01-preview",
)

The "engine" is essentially the name of the model deployment you have configured in Azure ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}