Skip to content

Commit

Permalink
add concurrency test to 00_core nb
Browse files Browse the repository at this point in the history
  • Loading branch information
@algal
algal committed Oct 26, 2024
1 parent 8841006 commit 429ccdc
Showing 1 changed file with 214 additions and 13 deletions.
227 changes: 214 additions & 13 deletions nbs/00_core.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -179,7 +179,16 @@
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/algal/miniconda/envs/jup3/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
" from .autonotebook import tqdm as notebook_tqdm\n"
]
}
],
"source": [
"from datasets import load_dataset"
]
Expand All @@ -189,6 +198,13 @@
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Generating train split: 100%|█| 200000/200000 [00:00<00:00, 3852934.04 examples/\n"
]
},
{
"data": {
"text/markdown": [
Expand Down Expand Up @@ -271,7 +287,7 @@
"name": "stderr",
"output_type": "stream",
"text": [
"100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:01<00:00, 1.59it/s]\n"
"100%|█████████████████████████████████████████████| 3/3 [00:01<00:00, 1.64it/s]\n"
]
}
],
Expand All @@ -294,9 +310,9 @@
{
"data": {
"text/markdown": [
"- Postpartum complications can be life-threatening. New mothers need support and resources to stay healthy after giving birth. ➡ *Las complicaciones posparto pueden ser mortales. Las nuevas madres necesitan apoyo y recursos para mantener su salud después del parto.*\n",
"- Incomplete or inaccurate project documentation can have serious legal consequences. It's important to ensure all project details are properly recorded and communicated. ➡ *La documentación de proyectos incompleta o inexacta puede tener graves consecuencias legales. Es importante asegurarse de que todos los detalles del proyecto se registren y se comuniquen adecuadamente.*\n",
"- El Salvador's former president Nayib Bukele has consolidated power and cracked down on opposition, drawing criticism from international observers. However, his populist policies remain popular among many Salvadorans frustrated with high crime rates and economic stagnation. ➡ *El expresidente de El Salvador, Nayib Bukele, ha consolidado el poder y reprimido a la oposición, lo que ha generado críticas de observadores internacionales. Sin embargo, sus políticas populistas siguen siendo populares entre muchos salvadoreños frustrados por las altas tasas de delincuencia y el estancamiento económico.*"
"- The political situation in El Salvador continues to be complex, with ongoing tensions between the ruling party and opposition groups. President Nayib Bukele has consolidated significant power, raising concerns about the state of democracy in the country. ➡ *La situación política en El Salvador sigue siendo compleja, con tensiones persistentes entre el partido gobernante y los grupos de oposición. El presidente Nayib Bukele ha consolidado un poder significativo, lo que genera preocupaciones sobre el estado de la democracia en el país.*\n",
"- Thorough documentation is critical for any legal proceedings. Incomplete or inaccurate records can have serious consequences. ➡ *La documentación exhaustiva es fundamental para cualquier proceso legal. Los registros incompletos o inexactos pueden tener consecuencias graves.*\n",
"- Postpartum complications can be life-threatening, but with proper care and support, new mothers can recover and thrive. Let's work together to ensure all women have access to the resources they need during this crucial time. ➡ *Las complicaciones posparto pueden poner en riesgo la vida, pero con la atención y el apoyo adecuados, las nuevas madres pueden recuperarse y prosperar. Trabajemos juntos para garantizar que todas las mujeres tengan acceso a los recursos que necesitan durante este momento crucial.*"
],
"text/plain": [
"<IPython.core.display.Markdown object>"
Expand Down Expand Up @@ -357,7 +373,7 @@
"name": "stderr",
"output_type": "stream",
"text": [
"100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:03<00:00, 1.33s/it]\n"
"100%|█████████████████████████████████████████████| 3/3 [00:03<00:00, 1.27s/it]\n"
]
}
],
Expand All @@ -379,17 +395,17 @@
{
"data": {
"text/markdown": [
"- Postpartum complications can be life-threatening. New mothers need support and resources to stay healthy after giving birth. ➡ *Las complicaciones posparto pueden ser mortales. Las nuevas madres necesitan apoyo y recursos para mantener su salud después del parto.*\n",
"- The political situation in El Salvador continues to be complex, with ongoing tensions between the ruling party and opposition groups. President Nayib Bukele has consolidated significant power, raising concerns about the state of democracy in the country. ➡ *La situación política en El Salvador sigue siendo compleja, con tensiones persistentes entre el partido gobernante y los grupos de oposición. El presidente Nayib Bukele ha consolidado un poder significativo, lo que genera preocupaciones sobre el estado de la democracia en el país.*\n",
"\n",
"\t- **Critique:** This translation demonstrates a high level of accuracy and fluency. It effectively conveys the original message, maintaining the tone and emphasis on the importance of proper documentation. The translator has chosen appropriate terminology (e.g., \"consecuencias legales\" for \"legal consequences\") and has accurately rendered the passive voice construction. The translation reads naturally in Spanish and captures the nuances of the original text. While it's a strong translation, there's a slight opportunity for improvement in terms of conciseness, which prevents it from reaching the highest level of excellence.\n",
"\t- **Score:** 4\n",
"- Incomplete or inaccurate project documentation can have serious legal consequences. It's important to ensure all project details are properly recorded and communicated. ➡ *La documentación de proyectos incompleta o inexacta puede tener graves consecuencias legales. Es importante asegurarse de que todos los detalles del proyecto se registren y se comuniquen adecuadamente.*\n",
"\t- **Critique:** The translation accurately conveys the meaning of the source text, capturing the complexity of El Salvador's political situation. It effectively translates key terms and maintains the tone of the original. The translator demonstrates a strong grasp of both languages, rendering the content naturally in Spanish. The phrasing is appropriate and flows well, preserving the nuances of the English version. There are no noticeable errors or awkward constructions. The translation skillfully handles the political terminology and concepts, making it suitable for professional use. It reads as if it were originally written in Spanish, indicating the translator's expertise.\n",
"\t- **Score:** 5\n",
"- Thorough documentation is critical for any legal proceedings. Incomplete or inaccurate records can have serious consequences. ➡ *La documentación exhaustiva es fundamental para cualquier proceso legal. Los registros incompletos o inexactos pueden tener consecuencias graves.*\n",
"\n",
"\t- **Critique:** The translation accurately conveys the main ideas of the source text, maintaining the structure and key information. It correctly translates complex terms like \"consolidated power\" and \"cracked down on opposition.\" The translator effectively renders \"populist policies\" and \"economic stagnation.\" The text reads naturally in Spanish and maintains the original tone. There are no significant errors or awkward phrasings. The translation demonstrates a high level of competence, capturing nuances and providing a faithful rendering of the content. It's suitable for professional use, though it doesn't reach the level of exceptional mastery that would warrant a perfect score.\n",
"\t- **Critique:** The translation accurately conveys the main message of the source text, maintaining both the meaning and tone. It effectively captures the importance of thorough documentation in legal proceedings and the potential consequences of incomplete or inaccurate records. The translator has chosen appropriate Spanish equivalents for key terms, such as \"exhaustiva\" for \"thorough\" and \"proceso legal\" for \"legal proceedings\". The sentence structure is natural in Spanish, and the translation maintains the formal register suitable for legal contexts. There are no errors in grammar or vocabulary. The translation demonstrates a high level of competence, accurately conveying complex concepts while reading naturally in the target language. It could be considered the work of an experienced translator.\n",
"\t- **Score:** 4\n",
"- El Salvador's former president Nayib Bukele has consolidated power and cracked down on opposition, drawing criticism from international observers. However, his populist policies remain popular among many Salvadorans frustrated with high crime rates and economic stagnation. ➡ *El expresidente de El Salvador, Nayib Bukele, ha consolidado el poder y reprimido a la oposición, lo que ha generado críticas de observadores internacionales. Sin embargo, sus políticas populistas siguen siendo populares entre muchos salvadoreños frustrados por las altas tasas de delincuencia y el estancamiento económico.*\n",
"- Postpartum complications can be life-threatening, but with proper care and support, new mothers can recover and thrive. Let's work together to ensure all women have access to the resources they need during this crucial time. ➡ *Las complicaciones posparto pueden poner en riesgo la vida, pero con la atención y el apoyo adecuados, las nuevas madres pueden recuperarse y prosperar. Trabajemos juntos para garantizar que todas las mujeres tengan acceso a los recursos que necesitan durante este momento crucial.*\n",
"\n",
"\t- **Critique:** The translation accurately conveys the main message and maintains the tone of the original text. It demonstrates a good understanding of both languages and uses appropriate medical terminology (e.g., \"posparto\" for \"postpartum\"). The Spanish version reads naturally and maintains the concise style of the source. It effectively captures the urgency of the situation with \"pueden ser mortales\" for \"life-threatening\". The translator made a good choice in translating \"stay healthy\" to \"mantener su salud\", which sounds more natural in Spanish. Overall, it's a high-quality translation suitable for professional use, with no significant errors or awkward phrasing.\n",
"\t- **Critique:** The translation accurately conveys the meaning of the original text, maintaining both the informative and encouraging tone. It correctly translates key terms like \"postpartum complications\" and \"life-threatening.\" The Spanish version flows naturally and captures the nuances of the original, including the call to action. The translator has made excellent choices in vocabulary and structure, resulting in a text that reads as if it were originally written in Spanish. The translation demonstrates a high level of proficiency in both languages and would be suitable for professional use in healthcare communications.\n",
"\t- **Score:** 5"
],
"text/plain": [
Expand All @@ -405,6 +421,191 @@
"show(f'{t}\\n\\n{c}' for t, c in zip(translations, critiques))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Test that generate outputs align with inputs"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's verify that the `FastData.generate` returns results in the same order as the inputs it was passed.\n",
"\n",
"To show this, we will define a new prompt template, where the model is asked only to echo a piece of data\n",
"from the input. Then we will verify that the values in the inputs matches the values in the outputs, in order and in value."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"sp=\"You will help with accurate and faithful data processing.\"\n",
"prompt_template = \"\"\"\\\n",
"Below you find an item of data, a datum, which is an alphanumeric string:\n",
"\n",
"<datum>{datum}</datum>\n",
"\n",
"After reviewing this datum, please echo is back exactly, without any preamble:\n",
"\"\"\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"class Datum:\n",
" \"A data value\"\n",
" def __init__(self, datum: str): store_attr()\n",
" def __repr__(self): return f\"{self.datum}\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"First we'll test that the prompt and schema class work as execpted."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Below you find an item of data, a datum, which is an alphanumeric string:\n",
"\n",
"<datum>b9121446-e46c-47c0-9e6d-b4df35c0974b</datum>\n",
"\n",
"After reviewing this datum, please echo is back exactly, without any preamble:\n",
"\n"
]
}
],
"source": [
"import uuid\n",
"str(uuid.uuid4())\n",
"print(prompt_template.format(**dict(datum=str(uuid.uuid4()))))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"04da7de4-cc39-4699-9d25-5a476e366732"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"Datum(str(uuid.uuid4()))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we ask the model to \"generate\" (i.e., echo) 100 of these values."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|█████████████████████████████████████████| 100/100 [00:04<00:00, 24.17it/s]\n"
]
}
],
"source": [
"in_vals = [{\"datum\":str(uuid.uuid4())} for _ in range(100)]\n",
"out_vals = fast_data.generate(\n",
" prompt_template=prompt_template,\n",
" inputs=in_vals,\n",
" schema=Datum,\n",
" sp=sp\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we will see if the inputs and outputs are aligned.\n",
"\n",
"If they are aligned, then that shows two things. First it shows that the model is echoing the values faithfully. Second, it shows _either_ that the model itself returned outputs in the order in which they were submitted, or else that `generate` has returned outputs in submission order.\n",
"\n",
"We are submitting a large enough quantity of items, that we _asssume_ the model will return some results out of submission order. If you want confidence which does not depend on this assumption, then could modify the test above to increase the number and complexity of the generation task, or simply inspect the implementation.\n",
"\n",
"Let's start by spot checking the first item:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"('f42ea0db-24ce-4e09-a50d-edf74d0eb611',\n",
" 'f42ea0db-24ce-4e09-a50d-edf74d0eb611')"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"in_list = [x['datum'] for x in in_vals]\n",
"out_list = [x.datum for x in out_vals]\n",
"(in_list[0],out_list[0])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Success: output items are aligned with input items\n"
]
}
],
"source": [
"for (idx,(in_item,out_item)) in enumerate(zip(in_list,out_list)):\n",
" if in_item != out_item:\n",
" print(\"Failure: output items were not aligned with input items!\")\n",
" print(f\"\\titem {idx} had in={in_item} and out={out_item}\")\n",
" break\n",
"else:\n",
" print(\"Success: output items are aligned with input items\") "
]
},
{
"cell_type": "code",
"execution_count": null,
Expand Down

0 comments on commit 429ccdc

Please sign in to comment.