Skip to content

Commit

Permalink
fix: variable case
Browse files Browse the repository at this point in the history
  • Loading branch information
linonetwo committed Nov 20, 2024
1 parent d44726c commit 373a7e5
Show file tree
Hide file tree
Showing 6 changed files with 23 additions and 16 deletions.
18 changes: 12 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,22 +23,22 @@ Prompts are WikiText tiddlers with variable transclusion, in the [wiki's 'prompt
Each tiddler's wikified body will be put to review platform. And they can use following variables from pipeline.

- `InputTiddler` `.tid` file content with metadata part, read from TW core or other plugins.
- `InputWikiText` Text part of tiddler, extracted by pipeline
- `AIOutput` GPT generated output, generated by pipeline
- `inputWikiText` Text part of tiddler, extracted by pipeline
- `aIOutput` GPT generated output, generated by pipeline

### Pipeline

1. Use pure JS to get `InputTiddler` and `InputWikiText`
1. Use pure JS to get `InputTiddler` and `inputWikiText`
1. These variables will be available in tiddlers in the [pipeline's 'prompts](./wiki/tiddlers/prompts/pipeline) folder.
1. We process "input" X "Data prompt tiddler" matrix member one by one, we can get `prompt` field of data tiddler, as `DataPrompt` variable
1. Compose the variables using WikiText, and we use wikified tiddler text as AI input, to get the content for `AIOutput` variable.
1. Compose the variables using WikiText, and we use wikified tiddler text as AI input, to get the content for `aIOutput` variable.

```mermaid
graph TD
A[TW core or other plugins] --> B[InputTiddler and InputWikiText]
A[TW core or other plugins] --> B[InputTiddler and inputWikiText]
B --> D[Pipeline Template]
C[Data Prompt Templates' Metadata] --> D
D --> E[AIOutput]
D --> E[aIOutput]
E --> F[Data Prompt Templates' Body]
B --> F
F --> G[Text to Review]
Expand All @@ -51,3 +51,9 @@ In the review platform, there are "original language" and "translation" areas, b
Original WikiText + AI prompt that generate the material will be the "original language", and AI generated chat material can will put in the "translation" area, open for human to review.

You can join the project in [Paratranz](https://paratranz.cn/projects/12129), and review each material, edit them to be correct (AI will have mistake, they don't understand WikiText well, because there was never good learning material before like we are creating).

## How to run

1. Have [Deno](https://deno.com/) installed
1. Get a [DeepSeek API key](https://platform.deepseek.com/api_keys) or OpenAI API key, put it in a `.env` file, copied from [.env.template](./.env.template) file.
1. Run `deno task dev`
7 changes: 4 additions & 3 deletions generateChatML.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ const wikiPath = join(Deno.cwd(), "wiki");
wikiInstance.boot.argv = [wikiPath, "--version"];
wikiInstance.boot.startup({});

export async function generateChatML(tidContent: string): Promise<string> {
export async function generateChatML(inputWikiText: string): Promise<string> {
const dataPromptTiddlers = wikiInstance.wiki.filterTiddlers(
`[!is[system]tag[Prompt]tag[Data]]`,
)
Expand All @@ -24,14 +24,15 @@ export async function generateChatML(tidContent: string): Promise<string> {
const chatmlContents = await Promise.all(
dataPromptTiddlers.map(async (tiddler) => {
const prompt = tiddler.fields.prompt as string;
const AIOutput = await callLLM(prompt, tidContent);
const aIOutput = await callLLM(prompt, inputWikiText);
const chatmlContent = wikiInstance.wiki.renderTiddler(
"text/plain",
tiddler.fields.title,
{
variables: {
prompt,
AIOutput,
aIOutput,
inputWikiText,
},
},
);
Expand Down
4 changes: 2 additions & 2 deletions wiki/tiddlers/prompts/data/AddComment.tid
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ prompt: Add comments to explain the wikitext line by line
{{!!prompt}}
<|im_end|>
<|im_start|>user
<<InputWikiText>>
<<inputWikiText>>
<|im_end|>
<|im_start|>assistant
<<AIOutput>>
<<aIOutput>>
<|im_end|>
4 changes: 2 additions & 2 deletions wiki/tiddlers/prompts/data/ExplainInNaturalLanguage.tid
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@ prompt: Explain the wikitext in natural language for user new to WikiText.
<|im_start|>user
What does this mean?
```tid
<<InputWikiText>>
<<inputWikiText>>
```
<|im_end|>
<|im_start|>assistant
<<AIOutput>>
<<aIOutput>>
<|im_end|>
4 changes: 2 additions & 2 deletions wiki/tiddlers/prompts/data/GenerateWikiText.tid
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ prompt: What will user ask to generate this WikiText (not including the metadata
You are a helpful TiddlyWiki community member, you want to help new users write WikiText for their wiki or TW plugins.
<|im_end|>
<|im_start|>user
<<AIOutput>>
<<aIOutput>>
<|im_end|>
<|im_start|>assistant
<<InputWikiText>>
<<inputWikiText>>
<|im_end|>
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ prompt: What will user ask to generate this WikiText with metadata on the top?
You are a helpful TiddlyWiki community member, you want to help new users write WikiText and tiddler fields for their wiki or TW plugins.
<|im_end|>
<|im_start|>user
<<AIOutput>>, with metadata
<<aIOutput>>, with metadata
<|im_end|>
<|im_start|>assistant
<<InputTiddler>>
Expand Down

0 comments on commit 373a7e5

Please sign in to comment.