Built site for gh-pages

UKGovernmentBEIS · Aug 25, 2024 · 135ba06 · 135ba06
1 parent de13b27
commit 135ba06
Show file tree

Hide file tree

Showing 25 changed files with 760 additions and 819 deletions.
diff --git a/.nojekyll b/.nojekyll
@@ -1 +1 @@
-b9bc64c4
+de1dcad4
diff --git a/agents-api.html b/agents-api.html
diff --git a/agents.html b/agents.html
diff --git a/caching.html b/caching.html
@@ -2,12 +2,12 @@
 <html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en"><head>
 
 <meta charset="utf-8">
-<meta name="generator" content="quarto-1.5.54">
+<meta name="generator" content="quarto-1.5.32">
 
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 
 
-<title>Caching – Inspect</title>
+<title>Inspect – Caching</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -262,6 +262,11 @@
   <div class="sidebar-item-container"> 
   <a href="./agents-api.html" class="sidebar-item-text sidebar-link"><span class="chapter-title">Agents API</span></a>
   </div>
+</li>
+          <li class="sidebar-item">
+  <div class="sidebar-item-container"> 
+  <a href="./interactivity.html" class="sidebar-item-text sidebar-link"><span class="chapter-title">Interactivity</span></a>
+  </div>
 </li>
           <li class="sidebar-item">
   <div class="sidebar-item-container"> 
@@ -422,27 +427,13 @@ <h3 class="anchored" data-anchor-id="cache-directory">Cache Directory</h3>
 </section>
 <section id="sec-provider-caching" class="level2">
 <h2 class="anchored" data-anchor-id="sec-provider-caching">Provider Caching</h2>
-<div class="callout callout-style-default callout-note callout-titled">
-<div class="callout-header d-flex align-content-center">
-<div class="callout-icon-container">
-<i class="callout-icon"></i>
-</div>
-<div class="callout-title-container flex-fill">
-Note
-</div>
-</div>
-<div class="callout-body-container callout-body">
-<p>The provider caching feature described below is currently available only in the development version of Inspect. You can install the development version with:</p>
-<div class="sourceCode" id="cb11"><pre class="sourceCode bash code-overflow-wrap code-with-copy"><code class="sourceCode bash"><span id="cb11-1"><a href="#cb11-1" aria-hidden="true" tabindex="-1"></a><span class="ex">pip</span> install git+https://github.com/ukgovernmentbeis/inspect_ai</span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
-</div>
-</div>
 <p>Model providers may also provide prompt caching features to optimise cost and performance for multi-turn conversations. Currently, Inspect includes support for <a href="https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching">Anthropic Prompt Caching</a> and will extend this support to other providers over time as they add caching to their APIs.</p>
 <p>Provider prompt caching is controlled by the <code>cache-prompt</code> generation config option. The default value for <code>cache-prompt</code> is <code>"auto"</code>, which enables prompt caching automatically if tool definitions are included in the request. Use <code>true</code> and <code>false</code> to force caching on or off. For example:</p>
-<div class="sourceCode" id="cb12"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb12-1"><a href="#cb12-1" aria-hidden="true" tabindex="-1"></a><span class="ex">inspect</span> eval ctf.py <span class="at">--cache-prompt</span><span class="op">=</span>auto  <span class="co"># enable if tools defined</span></span>
-<span id="cb12-2"><a href="#cb12-2" aria-hidden="true" tabindex="-1"></a><span class="ex">inspect</span> eval ctf.py <span class="at">--cache-prompt</span><span class="op">=</span>true  <span class="co"># force caching on</span></span>
-<span id="cb12-3"><a href="#cb12-3" aria-hidden="true" tabindex="-1"></a><span class="ex">inspect</span> eval ctf.py <span class="at">--cache-prompt</span><span class="op">=</span>false <span class="co"># force caching off</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+<div class="sourceCode" id="cb11"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb11-1"><a href="#cb11-1" aria-hidden="true" tabindex="-1"></a><span class="ex">inspect</span> eval ctf.py <span class="at">--cache-prompt</span><span class="op">=</span>auto  <span class="co"># enable if tools defined</span></span>
+<span id="cb11-2"><a href="#cb11-2" aria-hidden="true" tabindex="-1"></a><span class="ex">inspect</span> eval ctf.py <span class="at">--cache-prompt</span><span class="op">=</span>true  <span class="co"># force caching on</span></span>
+<span id="cb11-3"><a href="#cb11-3" aria-hidden="true" tabindex="-1"></a><span class="ex">inspect</span> eval ctf.py <span class="at">--cache-prompt</span><span class="op">=</span>false <span class="co"># force caching off</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 <p>Or with the <code>eval()</code> function:</p>
-<div class="sourceCode" id="cb13"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb13-1"><a href="#cb13-1" aria-hidden="true" tabindex="-1"></a><span class="bu">eval</span>(<span class="st">"ctf.py"</span>, cache_prompt<span class="op">=</span><span class="va">True</span>)</span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+<div class="sourceCode" id="cb12"><pre class="sourceCode python code-with-copy"><code class="sourceCode python"><span id="cb12-1"><a href="#cb12-1" aria-hidden="true" tabindex="-1"></a><span class="bu">eval</span>(<span class="st">"ctf.py"</span>, cache_prompt<span class="op">=</span><span class="va">True</span>)</span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 <section id="cache-scope" class="level3">
 <h3 class="anchored" data-anchor-id="cache-scope">Cache Scope</h3>
 <p>Providers will typically provide various means of customising the scope of cache usage. The Inspect <code>cache-prompt</code> option will by default attempt to make maximum use of provider caches (in the Anthropic implementation system messages, tool definitions, and all messages up to the last user message are included in the cache).</p>
@@ -451,9 +442,9 @@ <h3 class="anchored" data-anchor-id="cache-scope">Cache Scope</h3>
 <section id="usage-reporting" class="level3">
 <h3 class="anchored" data-anchor-id="usage-reporting">Usage Reporting</h3>
 <p>When using provider caching, model token usage will be reported with 4 distinct values rather than the normal input and output. For example:</p>
-<div class="sourceCode" id="cb14"><pre class="sourceCode default code-with-copy"><code class="sourceCode default"><span id="cb14-1"><a href="#cb14-1" aria-hidden="true" tabindex="-1"></a>13,684 tokens [I: 22, CW: 1,711, CR: 11,442, O: 509]</span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
+<div class="sourceCode" id="cb13"><pre class="sourceCode default code-with-copy"><code class="sourceCode default"><span id="cb13-1"><a href="#cb13-1" aria-hidden="true" tabindex="-1"></a>13,684 tokens [I: 22, CW: 1,711, CR: 11,442, O: 509]</span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
 <p>Where the prefixes on reported token counts stand for:</p>
-<table class="caption-top table">
+<table class="table">
 <tbody>
 <tr class="odd">
 <td><strong>I</strong></td>

diff --git a/datasets.html b/datasets.html
@@ -2,12 +2,12 @@
 <html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en"><head>
 
 <meta charset="utf-8">
-<meta name="generator" content="quarto-1.5.54">
+<meta name="generator" content="quarto-1.5.32">
 
 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
 
 
-<title>Datasets – Inspect</title>
+<title>Inspect – Datasets</title>
 <style>
 code{white-space: pre-wrap;}
 span.smallcaps{font-variant: small-caps;}
@@ -262,6 +262,11 @@
   <div class="sidebar-item-container"> 
   <a href="./agents-api.html" class="sidebar-item-text sidebar-link"><span class="chapter-title">Agents API</span></a>
   </div>
+</li>
+          <li class="sidebar-item">
+  <div class="sidebar-item-container"> 
+  <a href="./interactivity.html" class="sidebar-item-text sidebar-link"><span class="chapter-title">Interactivity</span></a>
+  </div>
 </li>
           <li class="sidebar-item">
   <div class="sidebar-item-container"> 
@@ -337,7 +342,7 @@ <h2 class="anchored" data-anchor-id="overview">Overview</h2>
 <h2 class="anchored" data-anchor-id="dataset-samples">Dataset Samples</h2>
 <p>The core data type underlying the use of datasets with Inspect is the <code>Sample</code>, which consists of a required <code>input</code> field and several other optional fields:</p>
 <p><strong>Class</strong> <code>inspect_ai.dataset.Sample</code></p>
-<table class="caption-top table">
+<table class="table">
 <colgroup>
 <col style="width: 20%">
 <col style="width: 40%">
@@ -377,19 +382,24 @@ <h2 class="anchored" data-anchor-id="dataset-samples">Dataset Samples</h2>
 <td>Optional. Arbitrary metadata associated with the sample.</td>
 </tr>
 <tr class="even">
+<td><code>sandbox</code></td>
+<td><code>str | tuple[str,str]</code></td>
+<td>Optional. Sandbox environment type (or optionally a tuple with type and config file)</td>
+</tr>
+<tr class="odd">
 <td><code>files</code></td>
 <td><code>dict[str | str] | None</code></td>
 <td>Optional. Files that go along with the sample (copied to sandbox environments).</td>
 </tr>
-<tr class="odd">
+<tr class="even">
 <td><code>setup</code></td>
 <td><code>str | None</code></td>
 <td>Optional. Setup script to run for sample (executed within default sandbox environment).</td>
 </tr>
 </tbody>
 </table>
 <p>So a CSV dataset with the following structure:</p>
-<table class="caption-top table">
+<table class="table">
 <colgroup>
 <col style="width: 56%">
 <col style="width: 43%">
@@ -497,7 +507,7 @@ <h2 class="anchored" data-anchor-id="amazon-s3">Amazon S3</h2>
 <h2 class="anchored" data-anchor-id="chat-messages">Chat Messages</h2>
 <p>The most important data structure within <code>Sample</code> is the <code>ChatMessage</code>. Note that often datasets will contain a simple string as their input (which is then internally converted to a <code>ChatMessageUser</code>). However, it is possible to include a full message history as the input via <code>ChatMessage</code>. Another useful application of <code>ChatMessage</code> is providing multi-modal input (e.g.&nbsp;images).</p>
 <p><strong>Class</strong> <code>inspect_ai.model.ChatMessage</code></p>
-<table class="caption-top table">
+<table class="table">
 <colgroup>
 <col style="width: 10%">
 <col style="width: 35%">