From 10a45344d2d26d2026ac969dab5cbe93cc7eaa54 Mon Sep 17 00:00:00 2001
From: Stig <65776111+stkillen@users.noreply.github.com>
Date: Tue, 5 Nov 2024 15:42:50 +0100
Subject: [PATCH] [AI Test Tool] Add README (#2294)

<!-- Thank you for submitting a Pull Request. If you're new to
contributing to BCApps please read our pull request guideline below
* https://github.com/microsoft/BCApps/Contributing.md
-->
#### Summary <!-- Provide a general summary of your changes -->
Add README.

#### Work Item(s) <!-- Add the issue number here after the #. The issue
needs to be open and approved. Submitting PRs with no linked issues or
unapproved issues is highly discouraged. -->
Fixes
[AB#556533](https://dynamicssmb2.visualstudio.com/Dynamics%20SMB/_workitems/edit/556533)
---
 src/Tools/AI Test Toolkit/README.md | 91 +++++++++++++++++++++++++++++
 1 file changed, 91 insertions(+)
 create mode 100644 src/Tools/AI Test Toolkit/README.md

diff --git a/src/Tools/AI Test Toolkit/README.md b/src/Tools/AI Test Toolkit/README.md
new file mode 100644
index 0000000000..f5052226e3
--- /dev/null
+++ b/src/Tools/AI Test Toolkit/README.md	
@@ -0,0 +1,91 @@
+# BC Copilot Test Toolkit
+The BC Copilot Test Toolkit lets developers write and run automated tests for copilot features. The toolkit supports running data driven tests and the ability to get the output generated from the tests.
+
+## Running AI tests in BC Copilot Test Toolkit
+
+### Prerequisite
+1. The BC Copilot Test Toolkit is installed
+1. Datasets and tests are written (see [Writing data-driven AI tests](#writing-data-driven-ai-tests))
+
+### Setup
+1. In _Business Central_, open the _AI Test Suite_ page
+1. Upload the required datasets for the tests
+1. Create a test suite
+1. In the test suite, define a line for each codeunit
+
+### Execute
+1. Run the AI Test Suite from the header, or one line at a time
+1. The test method will be executed for each dataset line
+    1. If the test evaluates in AL, it will either fail or succeed based on the condition
+    1. If the test output is set, it must be generated for all the tests which needs to be evaluated externally
+1. Results are logged in AI 'Log Entries'
+
+### Inspect the results
+1. Open _Log Entries_ to result of each execution
+1. Download the test output which generates the `.jsonl` file or export the logs to Excel
+1. You can also use the API (page 149038 "AIT Log Entry API") to get the result for a suite
+1. Open AL Test Tool and switch to the created test suite to execute each test manually
+
+
+## Writing data-driven AI tests
+
+### Defining test codeunit
+A data-driven AI test is defined like any normal AL test, except that it:
+1. Is executed through the BC Copilot Test Toolkit
+2. Utilizes the `AIT Test Context` codeunit to get input for the test
+3. Optionally, sets the output using the `AIT Test Context` codeunit
+
+See the `AIT Test Context` for the full API.
+
+#### Example
+An example test for an AI feature that returns an integer.
+
+```
+    [Test]
+    procedure TestCopilotFeature()
+    var
+        AITestContext: Codeunit "AIT Test Context";
+        Question: Text;
+        Output: Integer;
+        ExpectedOutput: Integer;
+    begin
+        // [Scenario] AI Test
+
+        // Get the input from the dataset
+        TestInput := AITestContext.GetQuestion().ValueAsText();
+
+        // Call the LLM to get an output
+        Output := CopilotFeature.CallLLM(Question);
+
+        // Get the expected output from the dataset
+        ExpectedOutput := AITestContext.GetExpectedData().ValueAsInteger();
+
+        // Assert the result
+        Assert.AreEqual(ExpectedOutput, Output, '');
+    end;
+```
+In this example
+1. This test procedure will be called with each input from the dataset
+1. We get the `question` and `expected_data` from the input dataset using `AITestContext.GetQuestion()` and `AITestContext.GetExpectedData()` respectively
+1. Alternatively, we could use `AITestContext.GetInput()` and get the line as `json` 
+
+
+### Defining Datasets
+Datasets are provided as `.jsonl` files where each line represents a test case.
+
+There's no fixed structure required for each line, but using certain formatting will allow easier access to data and name and description definition.
+
+See the `AIT Test Context` for the full API.
+
+#### Example
+
+```
+{"name": "Test01", "question": "A question", "expected_data": 5}
+{"name": "Test02", "question": "A second question", "expected_data": 2}
+{"name": "Test03", "question": "A third question", "expected_data": 2}
+```
+
+In this example
+1. Setting `name` for each line, that will be used in the BC Copilot Test Toolkit when uploading the dataset
+1. Setting `question` for each line, we can use `AITestContext.GetQuestion()` in the test codeunit, to get the question directly
+1. Setting `expected_data` for each line, we can use the `AITestContext.GetExpectedData()` in the test codeunit, to get the question directly
\ No newline at end of file