If we want to generate output that is structured in a specific way, we can use various techniques to
-To enable constrained decoding, we will use one of the most popular packages for this task instructor. It is built on pydantic and can leverage function calling and JSON-mode of the OpenAI API as well as other constrained sampling approaches.
from pydantic import BaseModel, Field
-from typing import List, Optional, Literal
-import erdantic as erd
-import instructor
-from IPython.display import SVG
-from openai import OpenAI
-from dotenv import load_dotenv
-load_dotenv('../.env', override=True)TrueFor most constrained generation tasks, we need to define a data schema in a programmatic way. The most common way to do so is to use pydantic data classes. Here is an example of a simple data schema for a recipe:
from pydantic import BaseModel
-
-class Recipe(BaseModel):
- title: str
- ingredients: List[str]
- instructions: List[str]This schema can also be extended to include descriptions of different fields or to only allow certain values for specific fields. For example, we could add a field for the number of servings and only allow positive integers.
-from pydantic import BaseModel, Field
-from typing import Literal, List
-
-class Recipe(BaseModel):
- title: str
- ingredients: List[str]
- instructions: List[str]
- servings: int = Field(..., gt=0, description="The number of servings for this recipe")
- rating: Literal["easy", "medium", "hard"] = Field("easy", description="The difficulty level of this recipe")If we want to extract copolymerization reactions a data schema could look like the following.
-We can now use instructor to “patch” the OpenAI API client to ensure that our output fulfils the schema.
client = instructor.patch(OpenAI(), mode=instructor.Mode.MD_JSON)
-
-
-class Monomer(BaseModel):
- name: str = Field(..., title="Name", description="Name of the monomer.")
- reactivity_constant: Optional[float] = Field(
- None,
- title="Reactivity constant",
- description="Reactivity constant of the monomer. r1 for monomer 1 and r2 for monomer 2. Must be greater or equal 0.",
- ge=0,
- )
- reactivity_constant_error: Optional[float] = Field(
- None,
- title="Reactivity constant error",
- description="Error in the reactivity constant. Often indicated with +/-. Must be greater or equal 0",
- ge=0,
- )
- q_parameter: Optional[float] = Field(
- None,
- title="Q parameter",
- description="Q parameter of the monomer. Q1 for monomer 1 and Q2 for monomer 2. Must be greater or equal 0",
- ge=0,
- )
- e_parameter: Optional[float] = Field(
- None,
- title="e parameter",
- description="e parameter of the monomer. e1 for monomer 1 and e2 for monomer 2.",
- )
-
-
-class CopolymerizationReaction(BaseModel):
- temperature: Optional[float] = Field(
- ...,
- title="Temperature",
- description="Temperature at which the reaction is carried out",
- )
- temperature_unit: Optional[Literal["C", "K"]] = Field(
- ..., title="Temperature unit", description="Unit of temperature"
- )
- solvent: Optional[str] = Field(
- None,
- title="Solvent",
- description="Solvent used in the reaction. If bulk polymerization was performed, this field should be left empty",
- )
- initiator: Optional[str] = Field(
- None, title="Initiator", description="Initiator used in the reaction"
- )
- monomers: Optional[List[Monomer]] = Field(
- ...,
- title="Monomers",
- description="Monomers used in the reaction. Ensure that the reactivity ratios are not confused with other numbers (such as Q and e). The two monomers MUST be used in the same reaction and mentioned in the same context.",
- min_items=2,
- max_items=2,
- )
- polymerization_type: Optional[str] = Field(
- ...,
- title="Polymerization type",
- description="Type of polymerization (e.g., bulk, solution, suspension, emulsion)",
- )
- determination_method: Optional[str] = Field(
- ...,
- title="Determination method",
- description="Method used to determine the reactivity ratios (e.g. Kelen Tudor, Fineman-Ross, Mayo-Lewis).",
- )
- diagram = erd.create(CopolymerizationReaction)
-diagram.draw("diagram.svg")
-SVG("diagram.svg")
In this case, we will use PDF files in the form as images as input for the model. To perform this conversion, we import some utilities.
-from pdf2image import convert_from_path
-from utils import process_image, get_prompt_vision_modelThe code below only converts each page of the PDF into an image and then generates dictionary objects in a format that can be used by the OpenAI API.
-filepath = 'paper01.pdf'
-pdf_images = convert_from_path(filepath)
-
-images_base64 = [process_image(image, 2048, 'images', filepath, j)[0] for j, image in enumerate(pdf_images)]
-images = get_prompt_vision_model(images_base64=images_base64)Armed with the images, we can now use the OpenAI API to extract the text from the images. For this, we just call the API with our prompts and the images.
-completion = client.chat.completions.create(
- model="gpt-4-turbo",
- response_model=List[CopolymerizationReaction],
- max_retries=2,
- messages=[
- {
- "role": "system",
- "content": """You are a scientific assistant, extracting accurate information about co-polymerization reactions from scientific papers.
-Do not use data that was reproduced from other sources.
-If you confuse the reactivity ratios with other numbers, you will be penalized.
-Monomer names might be quite similar, if you confuse them, you will be penalized.
-NEVER combine data from different reactions, otherwise you will be penalized.
-If you are unsure, return no data. Quality is more important than quantity.
-""",
- },
- {
- "role": "user",
- "content": """Extract the data from the paper into the provided data schema. We want an iterable of reaction objects and each reaction will be its own object. You can find each page of the paper as an image below.
-The relationship between monomers and parameters is typically indicated by subscripts that can be a number or an abbreviation of the monomer.
-Never return data that you are not absolutely sure about! You will be penalized for incorrect data.""",
- },
- {"role": "user", "content": [*images]},
- ],
- temperature=0,
-)completion[CopolymerizationReaction(temperature=60.0, temperature_unit='C', solvent='carbon tetrachloride', initiator='AIBN', monomers=[Monomer(name='methacrylic acid', reactivity_constant=0.54, reactivity_constant_error=0.01, q_parameter=None, e_parameter=None), Monomer(name='styrene', reactivity_constant=0.06, reactivity_constant_error=0.03, q_parameter=None, e_parameter=None)], polymerization_type='solution', determination_method='Kelen-Tudos'),
- CopolymerizationReaction(temperature=60.0, temperature_unit='C', solvent='chloroform', initiator='AIBN', monomers=[Monomer(name='methacrylic acid', reactivity_constant=0.51, reactivity_constant_error=0.01, q_parameter=None, e_parameter=None), Monomer(name='styrene', reactivity_constant=0.08, reactivity_constant_error=0.03, q_parameter=None, e_parameter=None)], polymerization_type='solution', determination_method='Kelen-Tudos'),
- CopolymerizationReaction(temperature=60.0, temperature_unit='C', solvent='acetone', initiator='AIBN', monomers=[Monomer(name='methacrylic acid', reactivity_constant=0.43, reactivity_constant_error=0.0, q_parameter=None, e_parameter=None), Monomer(name='styrene', reactivity_constant=0.65, reactivity_constant_error=0.02, q_parameter=None, e_parameter=None)], polymerization_type='solution', determination_method='Kelen-Tudos'),
- CopolymerizationReaction(temperature=60.0, temperature_unit='C', solvent='1,4-dioxane', initiator='AIBN', monomers=[Monomer(name='methacrylic acid', reactivity_constant=0.41, reactivity_constant_error=0.02, q_parameter=None, e_parameter=None), Monomer(name='styrene', reactivity_constant=0.59, reactivity_constant_error=0.05, q_parameter=None, e_parameter=None)], polymerization_type='solution', determination_method='Kelen-Tudos'),
- CopolymerizationReaction(temperature=60.0, temperature_unit='C', solvent='acetonitrile', initiator='AIBN', monomers=[Monomer(name='methacrylic acid', reactivity_constant=0.12, reactivity_constant_error=0.0, q_parameter=None, e_parameter=None), Monomer(name='styrene', reactivity_constant=0.29, reactivity_constant_error=0.0, q_parameter=None, e_parameter=None)], polymerization_type='solution', determination_method='Kelen-Tudos')]\n",
- " American Chemical Society\n",
- "
\n",
- " Recently, chemical modifications of chitosan (CS) have attracted the attention of scientific researchers due to its wide range of applications.\n",
- " In this research, chitin (CH) was extracted from the scales of \n",
- " Cyprinus carpio fish and converted to CS by three chemical steps: (i) demineralization, (ii) deprotonation, and (iii) deacetylation.The degree (measured as a percentage) of deacetylation (DD %) was calculated utilizing the acid–base titration method.\n",
- " The structure of CS was characterized by Fourier transform infrared (FT-IR) spectroscopy and thermogravimetric analysis (TGA).\n",
- " Three new CS Schiff bases (CSSBs) (CS-P1, CS-P2, and CS-P3) were synthesized via coupling of CS with 2-chloroquinoline-3-carbaldehyde, quinazoline-6-carbaldehyde, and oxazole-4-carbaldehyde, respectively.\n",
- " The newly prepared derivatives were verified, structurally, by nuclear magnetic resonance ( \n",
- " 1 H and 13 C NMR) and FT-IR spectroscopy.Antimicrobial activity was evaluated for the prepared compounds against both “Gram-negative” and “Gram-positive” bacteria, namely, \n",
- " Cytotoxicity of the synthesized CSSBs was evaluated via a MTT screening test.\n",
- " The results indicated a critical activity increase of the synthesized compound rather than CS generally tested bacteria and fungi and the absence of cytotoxic activity.\n",
- " These findings suggested that these new CSSBs are novel biomaterial candidates with enhanced antibacterial and nontoxic characteristics for applications in areas of both biology and medicine.\n",
- "
\n",
- " \n",
- " \n",
- " \n",
- "
\n",
- " \n",
- " \n",
- " \n",
- "
\n",
- " CS has a chemical structure of α(1→4)-2-amino-2-deoxy-β- \n",
- " d -glucopyranose derived from the N-deacetylation of CH, a typical auxiliary biopolymer found in the exoskeletons of roaches and crustaceans and fungi cell walls ( Figure \n",
- " ).The main source of CH and CS is shellfish, for example, crabs, shrimp, and lobsters, and fish scales.\n",
- " It has a few receptive amino groups, which offer further chemical modifications for the development of an incredible assorted variety of valuable derivatives that are cost-effective.\n",
- " \n",
- " CS, deacetylated CH, is an extremely useful, readily available bioactive polymer, which is a renewable, natural, nontoxic, edible, and biodegradable polymer characterized by biocompatibility.\n",
- " \n",
- " \n",
- " \n",
- " It has versatile applications ranging from biomedical designing, pharmaceuticals, drug transport, restorative materials, metal particle chelation, and absorptivity to water treatment and plant security.\n",
- " \n",
- " \n",
- " Recently, the reaction of CS with the rings of aromatic and heterocyclic aldehydes resulted in the efficient production of stable Schiff bases (SBs), which are exceptional compounds in many application areas, particularly in pharmacology and medicine, e.g., as antimicrobial and cancer prevention agents.\n",
- " \n",
- " \n",
- " \n",
- " \n",
- " \n",
- "
\n",
- " Structure of CH and \n",
- " Cyprinus carpio fish.
\n",
- " Furthermore, quinoline and quinazoline compounds are present in several natural products and in manufactured pharmacologically significant heterocyclic materials.\n",
- " Quinoline and quinazoline derivatives are called antimalarial, \n",
- " \n",
- " \n",
- " \n",
- " \n",
- " \n",
- "
\n",
- " In this study, based on the above facts, first, CH is extracted from the local \n",
- " Cyprinus carpio fish scales by demineralization and deprotonation followed by deacetylation to produce CS using fresh reagents, thereby reducing the time for the overall procedure to obtain CS with a degree of deacylation (DD) percentage of at least 60.Second, this study aims to determine the DD % of the products; samples of CS are obtained in different steps of the procedure by the acid–base titration method.\n",
- " Third, this study also aims to characterize and verify the physicochemical properties of the products employing FT-IR and TGA.\n",
- " Finally, this study attempts to develop a synthesis method for three new CSSB derivatives CS-P1, CS-P2, and CS-P3 through coupling CS with 2-chloroquinoline-3-carbaldehyde, quinazoline-6-carbaldehyde, and oxazole-4-carbaldehyde, respectively.\n",
- " The structures of the prepared derivatives were verified via FT-IR and \n",
- " 1 H and 13 C NMR spectroscopy.The antimicrobial potential of CS and the new derivatives was tested against four kinds of bacteria, \n",
- " The cytotoxicity of the newly prepared compounds was evaluated through the MTT assay.\n",
- " The results were compared with those of CS.\n",
- "
\n",
- " Drug resistance of microorganisms to antibiotics has encouraged researchers to search for new antibiotics to challenge this dangerous phenomenon.\n",
- " Many CS derivatives were synthesized to improve the antimicrobial effectiveness of CS, for example, SBs, via methylation, amination, etc. \n",
- " The structures of the prepared CSSBs were confirmed using 1H NMR, \n",
- " 13 C NMR, and FT-IR techniques.
\n",
- " The details of the treatment of C \n",
- " 1 , C 2 , C 3 , and C 4 samples are presented in Table S1 in the Supporting Information (SI).
\n",
- " The DD % for sample C \n",
- " 4 was determined by the acid–base titration method.The DD % was calculated using the endpoint of the acid–base titration when the indicator changed into blue-green colors.\n",
- " The DD % values of the deacetylated products were calculated according to the following equations \n",
- " C\n",
- " 1 and V\n",
- " 1 are the concentration and the volume of HCl used, respectively, C\n",
- " 2 and V\n",
- " 2 are the concentration and the amount of NaOH used for titration, respectively, W is the weight of samples used for acid–base titration.The calculated DD for each sample is tabulated in Table S2 and Figure S1 in the SI.\n",
- "
\n",
- " FT-IR spectroscopy was used to study the structures of CH and the related derivatives.\n",
- " The FT-IR spectra of CS, CH, C \n",
- " 3 , and C 4 are shown in Figures S2–S5 in the SI, respectively.By comparing the spectra for sample C \n",
- " 3 and CH, and sample C 4 and CS, several bands in the range of 4000–400 cm –1 were noted in the spectra.As shown in Table S3 in the SI, bands of the synthesized CS match with those of CH.\n",
- " The primary amines showed various bands in the range 3425–2881 cm \n",
- " –1 correlated with ν(N–H) and ν(NH 2 ) vibrations.The band at 2921–2879 cm \n",
- " –1 was observed corresponding to the methyl group in pyranose.The appearance of a band at 1597 cm \n",
- " –1 and the disappearance of a band at 1655 cm –1 indicate the deacetylation of CH.The doublet form of the amide band is attributed to the existence of intermolecular and intramolecular hydrogen bonds (CO···HN, CO···HOCH \n",
- " 2 ).\n",
- " \n",
- " –1 are related to the vibrations of NHCO.The bands in the range of 1000–1158 cm \n",
- " –1 were related to the vibrations of C–O–C, C–OH, and C–C ring bonds.\n",
- " The characteristic band at 1600–1400 cm \n",
- " –1 due to the N–H bending of CS was observed, in addition to the bands at 1600.9 and 1647.19 cm –1 corresponding to the bending of −NH 2 .\n",
- " \n",
- " 3 , CH 2 , and CH groups was confirmed by the appearance of vibration peaks within the 1422–603 cm –1 range.
\n",
- " TGA of CS, CH, C3, and C4 was performed, as shown in Figures S6–S9 in the SI, respectively.\n",
- " Physicochemical properties of C3 and C4 were compared with those of CH and CS.\n",
- " The thermal stability of C3 was similar to that of CH.\n",
- "
\n",
- " Three CSSBs were prepared, as shown in the schematic diagram in Scheme \n",
- " .\n",
- " Extracted CS was dissolved in 2.0% aqueous acetic acid, followed by addition of an equimolar amount of the carbonyl compounds dissolved in ethanol to obtain CSSBs.\n",
- " The resultant compounds obtained from the reaction of CS with 2-chloroquinoline-3-carbaldehyde, quinazoline-6-carbaldehyde, and oxazole-4-carbaldehyde were labeled CS-P1, CS-P2, and CS-P3, respectively.\n",
- "
\n",
- " The FT-IR spectra of CSSBs (CS-P1, CS-P2, and CS-P3) are shown in Figures S10–S12 in the SI, respectively.\n",
- " All spectra of the new derivatives displayed a vibration band at 1633–1655 cm \n",
- " –1 corresponding to the (−CN) group.The aromatic ring showed a stretching vibration band ranging from 1400 to 1500 cm \n",
- " –1 related to the C–C bond, while the absorption band at 1057 cm –1 corresponded to the aromatic in-plane C–H bending.\n",
- " No band was observed in the region 1660–1730 cm \n",
- " –1 , which proved the absence of the carbonyl group, which in turn indicated that no residue of free aldehydes remained.The vibration bands at 2921 and 2883 cm \n",
- " –1 were related to the C–H stretching of methyl and methylene groups, respectively.\n",
- " The glycosidic bonds showed bands at 1155 and 900 cm \n",
- " –1 .The vibration bands at 1205–975 cm \n",
- " –1 were related to the C–O, C–C, and C–O–C stretching of glycosidic bonds and the pyranose ring.\n",
- " \n",
- " \n",
- "
\n",
- " The structure of the prepared CSSBs was confirmed by \n",
- " 1 H and 13 C NMR.The \n",
- " 1 H NMR spectra of the synthesized derivatives CS-P1, CS-P2, and CS-P3 are shown in Figures S13–S15 in the SI, respectively.The \n",
- " 13 C NMR spectra for CS-P1, CS-P2, and CS-P3 are shown in Figures S16–S18 in the SI, respectively.The \n",
- " 1 H and 13 C NMR data of all SBs are shown below.
\n",
- " δ \n",
- " H (500 MHz, d\n",
- " 6 -DMSO) 7.33 (1H, d, J 0.8), 6.84 (1H, dd, J 7.5, 1.4), 6.78 (1H, td, J 7.5, 1.4), 6.38 (1H, td, J 7.5, 1.4), 6.33 (1H, dd, J 7.5, 1.5), 6.12 (1H, s), 5.83 (1H, d, J 2.5), 5.04 (1H, d, J 0.8), 4.93 (1H, s), 4.25 (1H, s), 3.92 (1H, s), 3.62 (1H, td, J 7.7, 4.0), 3.46 (1H, dd, J 12.2, 6.0), 3.35 (1H, ddd, J 9.4, 5.9, 3.5), 3.21 (1H, dd, J 12.3, 5.9), 3.00 (1H, dd, J 4.0, 2.5), 1.80 (1H, ddd, J 12.3, 7.7, 3.5), 1.31 (1H, ddd, J 12.3, 7.7, 3.5).
\n",
- " δ \n",
- " C (125 MHz, d\n",
- " 6 -DMSO)153.38,143.12, 134.23, 131.24, 129.33, 127.75, 122.07, 120.77, 114.44, 95.68, 74.70, 73.70, 69.45, 66.03, 37.86.\n",
- "
\n",
- " δ \n",
- " H (500 MHz, d\n",
- " 6 -DMSO) 7.84 (1H, s), 7.64 (1H, s), 7.30 (1H, s), 7.25 (1H, dd, J 7.5, 1.5), 6.45 (1H, d, J 7.5), 6.14 (1H, d, J 3.1), 5.67 (1H, s), 5.53 (1H, s), 5.26 (2H, s), 3.88 (1H, td, J 7.8, 5.0), 3.64 (1H, s), 3.37 (1H, dd, J 12.4, 1.3), 3.25 (1H, ddd, J 4.8, 3.4, 1.3), 3.11 (1H, dd, J 12.4, 1.3), 2.89 (1H, dd, J 5.0, 3.1), 1.70 (1H, ddd, J 12.3, 7.8, 3.5), 1.30 (1H, ddd, J 12.3, 7.8, 3.5).
\n",
- " δ \n",
- " C (125 MHz, d\n",
- " 6 -DMSO) 164.71, 147.34, 147.27, 139.42, 132.73, 125.45, 121.55, 112.39, 96.94, 75.14, 74.96, 70.71, 67.29, 63.16, 39.52, 39.12.
\n",
- " δ \n",
- " H (500 MHz, d\n",
- " 6 -DMSO) 7.36 (1H, d, J 0.8), 6.38 (1H, d, J 2.5), 5.72 (1H, s), 5.26 (1H, s), 4.56 (1H, d, J 0.8), 4.33 (1H, s), 4.18–3.99(2H, m), 3.89 (1H, s), 3.63 (1H, dd, J 12.4, 1.3), 3.52 (1H, ddd, J 4.9, 3.5, 1.3), 3.38 (1H, dd, J 12.4, 1.3), 3.16 (1H, dd, J 4.4, 2.5), 1.96 (1H, ddd, J 12.3, 7.8, 3.6), 1.54 (1H, ddd, J 12.3, 7.8, 3.6).\n",
- "
\n",
- " δ \n",
- " C (125 MHz, d\n",
- " 6 -DMSO) 158.92, 152.49, 134.20, 97.34, 86.33, 75.36, 71.11, 67.70, 64.24, 39.52, 34.06.
\n",
- " The synthesized new CSSBs presented typical peaks of the Cs and SB parts.\n",
- "
\n",
- " Different organic solvents were used to test the solubility of the synthesized compounds.\n",
- " \n",
- " Table \n",
- " shows the results.\n",
- " The prepared compounds dissolve in dimethyl sulfoxide and mixtures of equal proportions of dimethyl sulfoxide and trifluoroacetic acid.\n",
- " Partial dissolution or swelling was observed in some solvents such as dilute hydrochloric acid and acetic acid at 70 °C.\n",
- " In contrast, the products are not soluble in most inorganic solvents.\n",
- "
\n",
- " The cytotoxicity assay for the synthesized compounds was carried out based on MTT [3-(4,5-dimethylthiazol-2-yl)-2,5-diphenyltetrazolium bromide].\n",
- " MTT assay is a colorimetric assay used for assessing cell viability and to measure cytotoxicity.\n",
- " As illustrated in Table \n",
- " , the results of the tested compounds CS-P1, CS-P2, and CS-P3 show a small variation between samples in comparison with the control.\n",
- " Several earlier research studies have demonstrated that CS and CSSB derivatives have little cellular toxicity.\n",
- " Consequently, CS has many applications in the medical field.\n",
- " \n",
- " \n",
- " Conversely, a smaller amount (25 mg) of the tested compound showed a slight cytotoxicity of up to 2.5% compared with high concentrations, which is extremely suitable in medical applications.\n",
- " According to reported studies, compounds with higher than 75% cell viability are considered noncytotoxic.\n",
- " \n",
- " The cell viability assessment proved that the selection of these compound may depend on the antibacterial activity for their use in medical applications.\n",
- "
\n",
- " The inhibition zone technique was used to assess the antibacterial activity of the CSSB derivatives.\n",
- " The results are shown in Table \n",
- " .\n",
- " From the results, all of the CSSBs and CS have a similar impact on strains of both \n",
- " The reported study demonstrated that CS could terminate the cell formation of \n",
- " \n",
- " CSSBs showed antibacterial action against \n",
- " The results also revealed that CS-P1, CS-P2, and CS-P3 have antibacterial action against \n",
- " Antifungal activity tests against two strains of fungi were carried out, and all verified CSSBs presented good results.\n",
- " The difference in inhibition between the two strains may result from the variances in the cell wall structures.\n",
- " From the result, it may be confirmed that the antibacterial action is through the breakage of the cell wall rather than the mechanism of interaction of CS derivatives with the DNA in the microorganism.\n",
- " The reaction of the active group on the CSSBs with the cell wall reduces permeability, which causes a shortage of substances in the cell, for example, amino acids, proteins, electrolytes, and lactate dehydrogenase.\n",
- " Therefore, the synthesized compounds lead to the inhibition of the metabolism of the bacteria and cause death.\n",
- " \n",
- " \n",
- " \n",
- "
\n",
- " Based on reported research, many mechanisms are proposed to clarify the action path of CS on microorganisms, which differs depending on the metabolic procedure and the structure of the cell wall.\n",
- " The first suggestion is the interruption of the cell wall of the organism because of the electrostatic attraction between the positively charged amine groups in CS and the negative residue group in the bacterial cell wall, such as −COO \n",
- " – or PO 4\n",
- " 3– .The second mechanism suggested is the interaction of bacterial DNA with CS, which causes the inhibition of protein synthesis and mRNA by permeation of CS into the bacterial cell and then the nuclei.\n",
- " Another suggestion is based on the ability of CS to form a complex with metals, for instance, Zn \n",
- " 2+ , Mg 2+ , and Ca 2+ ; these metals are important for bacterial metabolic processes and growth.
\n",
- " The search for new antibiotics has increased parallelly to the increase in the number of antibiotic-resistant microbes known as superbugs.\n",
- " CS might be a promising material in this field.\n",
- " CS with DD = 89% was extracted from the scales of \n",
- " Cyprinus carpio fish obtained from the local market.The structure of CS was characterized, and the DD % was determined by the acid–base titration method.\n",
- " Three novel CSSB derivatives with different parts as branches were synthesized, and their configurations were proved by FT-IR and \n",
- " 1 H and 13 C NMR spectroscopy.The new SB structure showed antibacterial activity against most microorganisms and fungi that were tested.\n",
- " The prepared CSSBs demonstrated, almost, no cytotoxic effect on mouse fibroblast cell lines.\n",
- " In light of the above findings, it is safe to say that the prepared CSSBs might be used in different biomedical fields with a high degree of safety and efficacy.\n",
- "
\n",
- " \n",
- " \n",
- " Cyprinus carpio fish scales were collected from the local market in Kirkuk city, Iraq.Its DD was 89%, which was confirmed via a titration procedure.\n",
- " \n",
- " Ethanol (99.7%), hydrochloric acid (37%), 2-chloroquinoline-3-carbaldehyde, methyl orange, quinazoline-6-carbaldehyde, oxazole-4-carbaldehyde, and acetic acid were obtained from Merck (Germany).\n",
- " Sodium hydroxide pellets, sodium bicarbonate, hexamethylentetramine, magnesium sulfate, aqueous ammonia, sodium borohydride, tetrachloromethane, and trifluoroacetic were purchased from R&M Chemicals Pvt. Ltd., India.\n",
- " Aniline blue was purchased from Alpha Chemika, India.\n",
- " All reagents were of analytical grade and directly used without any further purification.\n",
- "
\n",
- " FT-IR spectra were obtained using a PerkinElmer System 2000 FT-IR spectrometer.\n",
- " A PerkinElmer TGA 7 thermogravimetric analyzer was applied.\n",
- " A PerkinElmer 2400 Series II analyzer was used to determine the percentage of N in the samples.\n",
- " A Bruker AC-400 NMR spectrometer was used to record the \n",
- " 1 H and 13 C NMR spectra.Deuterated dimethyl sulfoxide ( \n",
- " d\n",
- " 6 -DMSO) was used as the solvent.
\n",
- " Two eukaryote and four bacterial strains were used for assessing the antimicrobial effectiveness of CS and the newly prepared derivatives.\n",
- " The examined microorganisms included four species of bacteria, \n",
- " These organisms were collected from the biology department of Kirkuk University from patients who had a liver transplant.\n",
- " Nutrient agar was used as a culture medium.\n",
- " The strains were refreshed through inoculating Luria-Bertani (LB) culture medium, \n",
- " The inhibition ability of CS and SB derivatives was assessed.\n",
- " A 48-well plate was coated with samples, and 250 μL of the organism suspension (100 CFU/mL) was kept at 37 °C with LB culture.\n",
- " The latter was incubated at 37 °C for 1 day.\n",
- " Dimethyl sulfoxide (DMSO) was employed as a solvent and served as a control sample.\n",
- "
\n",
- " The cytotoxicity evaluation of CS and the new SB derivatives was carried out using MTT assay.\n",
- " A change in the reagent color to yellow was an indication of cell viability.\n",
- " \n",
- " Laminar flow cabinet biosafety class III was used to carry out all tests.\n",
- " A mouse fibroblast cell line (NIH3T3) was used in the test, which was grown in Dulbecco’s modified Eagle’s medium (DMEM), completed with trypsin/EDTA (100 μg/mL), and enriched with 10% FBS at 37 °C.\n",
- " Different amounts (25, 50, 100, 150, and 200 mg) of CS and the SB derivatives were used to assess their cytotoxicity.\n",
- " For the assessment of cytotoxicity of the newly prepared compounds, a microtiter plastic plate with 96 wells was coated with cells at a concentration of 10 × 10 \n",
- " 3 cells per well and incubated overnight.The cells were incubated for 48 h with a Schiff base sample concentration of 90 μg/mL and alone as a negative control.\n",
- " At the end of the incubation time MTT (40 μL, 3 μg/mL) was added and kept at 37 °C for another 4 h.\n",
- " After the formation of crystals, sodium dodecyl sulfate (SDS) in distilled water (15%, 250 μL) was added to end the reaction.\n",
- " Doxorubicin (100 μg/mL) was used as a positive control under the same conditions.\n",
- " The absorbance was recorded at 595 and 620 nm as standard wavelengths.\n",
- " The tests were repeated three times ( \n",
- " n = 3), and the mean value with mean ± SD was calculated.The SPSS 11 program was used to calculate the statistical significance between samples and the negative control.\n",
- " The below equation was used to calculate the percentage of change in cell viability \n",
- "
\n",
- " Approximately 100 g of fish scales was weighed and then washed and sun-dried for 3 days.\n",
- " The fish scales were crushed using a high-speed Hamilton beach blender.\n",
- " The scales were then immersed in hot ethanol (50–60 °C) for an hour to kill the germs and remove the pungent odor.\n",
- " The resulting sample (designated as C \n",
- " 1 ) was washed with an ample amount of water and subjected to freeze-drying.C \n",
- " 1 was treated with 5% HCl at 25 °C for 24 h to eliminate the CaCO 3 component through a demineralization process.Then, 10.0 g of the sample was collected and labeled C \n",
- " 2 .The remaining sample was processed with a deproteinization step; 10% NaOH was used to remove the protein component at 60 °C.\n",
- " Then, 10.0 g of the sample was collected and labeled C \n",
- " 3 .The remaining sample was subjected to a deacetylation process by treating with NaOH (50%) under refluxing conditions to produce CS.\n",
- " Then, 10.0 g of the sample was collected and labeled C \n",
- " 4 .\n",
- " \n",
- " \n",
- "
\n",
- " The new CSSB derivatives were synthesized according to a reported method.\n",
- " \n",
- " A gram of CS in 50 mL of glacial acetic acid (2%) was stirred at 25 °C for 7 h.\n",
- " A mixture of the aldehyde was dissolved in ethanol (1:1, molar ratio aldehyde to CS) that was added to the mixture gradually.\n",
- " The mixture was stirred and heated in a water bath at 50 °C for 10 h.\n",
- " Aqueous sodium hydroxide 6% was added to the reaction mixture until precipitation of the desired compound.\n",
- " The precipitate was collected and washed several times with distilled water and ethanol to remove any remaining materials.\n",
- " The products were filtered and dried in a vacuum oven at 60 °C overnight.\n",
- " The schematic diagram shown in Scheme \n",
- " illustrates the synthesis routes to the new CSSBs.\n",
- "
\n",
- " A solution of methyl orange and aniline blue (0.10 g of each, 100 mL of distilled water) was added to a beaker, separately.\n",
- " The mixture was then transferred into a volumetric flask (100 mL).\n",
- "
\n",
- " In four clean and dry conical flasks, 0.30 g of each sample, C \n",
- " 3 , C 4 , CH, and CS, was added.Then, 30 mL of 0.1 M HC was added to each flask.\n",
- " Equivalent amounts of both methyl orange and aniline blue solutions were added to each set; the mixture was mixed with a glass rod until the color of the mixture became stable.\n",
- " The titration method was carried out for each sample set against NaOH (0.1 M) until the indicator color changed.\n",
- "
\n",
- " FT-IR spectra of CH, CS, C \n",
- " 3 , and C 4 and the CSSB derivatives (CS-P1, CS-P2, and CS-P3) were recorded via a FT-IR spectroscope (Model 8400, Shimadzu).KBr was mixed with 2 mg of the sample and pressed to form a homogeneous disc with ≈0.5 mm thickness.\n",
- " The spectral region between 4000 and 400 cm \n",
- " –1 was scanned.
\n",
- " The thermal stability of deacetylated CS was evaluated; approximately 5 mg of CS was analyzed using a thermogravimetric analyzer (model 50/50H, Shimadzu).\n",
- " The heating was done as described from 60 to 750 °C at a rate of of 5 °C/min under 30 mL/min flow rate of nitrogen.\n",
- " \n",
- " \n",
- " \n",
- "
\n",
- " \n",
- " The Supporting Information is available free of charge at https://pubs.acs.org/doi/10.1021/acsomega.0c01342 .\n",
- " \n",
- "
\n",
- " General procedure and details of the treatment of CS and CH; tables show the results of the acid–base titration method and the characteristic absorption bands in the FT-IR spectra of standard and experimentally prepared CS; figures display the DD % of samples; TG and DTG plots; and \n",
- " 1 H NMR, 13 C NMR, and FT-IR spectra for all prepared compounds ( PDF )
\n",
- " All authors contributed to the writing of the manuscript.\n",
- " All authors have approved the final version of the manuscript.\n",
- " The photo in Figure \n",
- " was taken by the authors.\n",
- "
\n",
- " The authors declare no competing financial interest.\n",
- "
\n",
- " M.O.M. wishes to thank the Ministry of Higher Education and Scientific Research, Iraq, for the award of a grant, and the authors acknowledge the Department of Biology, College of Science, Kirkuk University, Iraq, for their help with carrying out the biological tests.\n",
- "
As a second step of the data extraction one should process the obtained data to: 1. obtain a format like markup language that can be used as input for text-based models (Parsing with NOUAGAT) 2. remove irrelevant information from the raw data to make the system more robust and cheaper (Data cleaning)
- - - -To run this Notebook, you will need access to at least one GPU. The results that are printed were obtained using a single A100 graphic card with 80 GB of memory. Note that even using such a powerful GPU took the notebook more than 10 hours to complete.
-This book aims to illustrate with a practical example how to decide which learning paradigm is better for each application. To demonstrate the process, we will extract some information about chemical reactions from paragraphs of text.
-Choosing the learning paradigm should begin by trying some leading general-purpose LLM. For this practical case, the first model to test is the recent Llama-3 8B model with zero and one-shot prompts.
-We will start by importing all the packages needed.
-import json
-from dotenv import load_dotenv
-
-import torch
-from datasets import (
- load_dataset,
- Dataset,
-)
-from transformers import (
- AutoModelForCausalLM,
- AutoTokenizer,
- BitsAndBytesConfig,
- TrainingArguments,
- pipeline,
-)
-from transformers.pipelines.pt_utils import KeyDataset
-from peft import (
- LoraConfig,
-)
-from trl import (
- SFTTrainer,
- DataCollatorForCompletionOnlyLM,
-)
-from evaluate import load
-import litellm
-from litellm import completion
-from litellm.caching import Cache
-from statistics import mean
-import numpy as np
-import matplotlib.pyplot as plt To continue, we will allow LiteLLM to cache requests made to LLM-APIs. Additionally, we will import all environment variables.
Note that using the environment variables is the safest way of keeping personal API keys secret.
-litellm.cache = Cache()
-load_dotenv(".env", override=True)As starting model, we will try the Llama-3 8B model. We will call this model through the Groq API, which allows performing fast inference with several open-source models.
- -Groq provides some of the most popular open-source models, such as Llama or Mixtral models, with a high inference speed. To use the Groq API, the .env file must also contain the GROQ_API_KEY.
base_model = "groq/llama3-8b-8192"The dataset used in this tutorial is the one used in Ai et al.’s (Ai et al. 2024) recent work, which contains data about chemical reactions. The dataset contains 100K reaction procedure—ORD JSON pairs.
- -ORD stands for Open Reaction Database, a comprehensive data structure specially designed to describe all the elements involved in chemical reactions.
-The best way to obtain the data is to install the GitHub repository of the Ai et al. work and take it from there. To do so, run the following commands:
-!git clone https://github.com/qai222/LLM_organic_synthesis.git
-!cp LLM_organic_synthesis/workplace_data/datasets/USPTO-n100k-t2048_exp1.7z .
-!7za x USPTO-n100k-t2048_exp1.7z
-!cp USPTO-n100k-t2048_exp1/*.json .
-!rm -rf USPTO-n100k-t2048_exp1/ USPTO-n100k-t2048_exp1.7z LLM_organic_synthesisThis will leave four .json files in your current directory that contain all the data used here.
test_ds_path = "test.json"
-test_dataset = load_dataset("json", data_files=test_ds_path, split="train")
-test_dataset = test_dataset.shuffle(seed=42).select(range(100))
-test_datasetDataset({
- features: ['output', 'instruction'],
- num_rows: 100
-})Note that we only selected 100 samples from the test set. This will be enough for the demo shown here.
-test_dataset[0]{'output': '{"inputs": {"m1": {"components": [{"identifiers": [{"type": "NAME", "value": "4-nitrobenzyl (1R,3R,5R,6S)-6-((1R)-1-hydroxyethyl)-1-methyl-2-oxo-1-carbapenam-3-carboxylate"}], "amount": {"mass": {"value": 743.0, "units": "MILLIGRAM"}}, "reaction_role": "REACTANT"}]}, "m2": {"components": [{"identifiers": [{"type": "NAME", "value": "2-(tri-n-butylstannyl)-7-trifluoromethylthioimidazo[5,1-b]thiazole"}], "amount": {"mass": {"value": 1.06, "units": "GRAM"}}, "reaction_role": "REACTANT"}]}}, "conditions": {"conditions_are_dynamic": true}, "outcomes": [{"products": [{"identifiers": [{"type": "NAME", "value": "4-nitrobenzyl (1S,5R,6S)-6-((1R)-1-hydroxyethyl)-1-methyl-2-(7-trifluoromethylthioimidazo[5,1-b]thiazol-2-yl)-1-carbapen-2-em-3-carboxylate"}], "measurements": [{"type": "AMOUNT", "details": "MASS", "amount": {"mass": {"value": 172.0, "units": "MILLIGRAM"}}}], "reaction_role": "PRODUCT"}]}]}',
- 'instruction': 'Below is a description of an organic reaction. Extract information from it to an ORD JSON record.\n\n### Procedure:\nThe procedure of Example 1a) was repeated, except that 743 mg of 4-nitrobenzyl (1R,3R,5R,6S)-6-((1R)-1-hydroxyethyl)-1-methyl-2-oxo-1-carbapenam-3-carboxylate and 1.06 g 2-(tri-n-butylstannyl)-7-trifluoromethylthioimidazo[5,1-b]thiazole were used as the starting compounds. Thus, 172 mg of 4-nitrobenzyl (1S,5R,6S)-6-((1R)-1-hydroxyethyl)-1-methyl-2-(7-trifluoromethylthioimidazo[5,1-b]thiazol-2-yl)-1-carbapen-2-em-3-carboxylate was prepared.\n\n### ORD JSON:\n'}Note that the output comes in JSON format. For this simple example, we will not constrain the output to JSON format. However, to ensure good results, this constraining must be done. The chapter [“Constrained decoding and enforcing valid outputs”] (./file-you-link-to.qmd) provides explanations and examples about the options and how to constrain the output.
-This dataset is very big. Therefore, we will only take 100 samples from the test set used in the article mentioned above for our test set.
-We define a simple prompt template. The prompt contains a simple system part (named PREFIX) where the role and task of the model are defined, as well as the example used only for the one-shot prompt. Additionally, the prompt has a user prompt where the reaction instruction will be provided.
-PREFIX = """You are a helpful scientific assistant. Your task is to extract information about organic reactions. {shot}"""
-SUFFIX = """\n\n{sample}\n\n"""
-SHOT = """
-One example is provided to you to show how to perform the task:
-
-### Procedure:\nA suspension of 8 g of the product of Example 7 and 0.4 g of DABCO in 90 ml of xylenes were heated under N2 at 130\u00b0-135\u00b0 C. while 1.8 ml of phosgene was added portionwise at a rate to maintain a reflux temperature of about 130\u00b0-135\u00b0 C. The mixture was refluxed an additional two hours, cooled under N2 to room temperature, filtered, and the filtrate was concentrated in vacuo to yield 6.9 g of the subject compound as a crude oil.\n\n
-### ORD JSON:\n{\"inputs\": {\"m1_m2_m4\": {\"components\": [{\"identifiers\": [{\"type\": \"NAME\", \"value\": \"product\"}], \"amount\": {\"mass\": {\"value\": 8.0, \"units\": \"GRAM\"}}, \"reaction_role\": \"REACTANT\"}, {\"identifiers\": [{\"type\": \"NAME\", \"value\": \"DABCO\"}], \"amount\": {\"mass\": {\"value\": 0.4, \"units\": \"GRAM\"}}, \"reaction_role\": \"REACTANT\"}, {\"identifiers\": [{\"type\": \"NAME\", \"value\": \"xylenes\"}], \"amount\": {\"volume\": {\"value\": 90.0, \"units\": \"MILLILITER\"}}, \"reaction_role\": \"SOLVENT\"}]}, \"m3\": {\"components\": [{\"identifiers\": [{\"type\": \"NAME\", \"value\": \"phosgene\"}], \"amount\": {\"volume\": {\"value\": 1.8, \"units\": \"MILLILITER\"}}, \"reaction_role\": \"REACTANT\"}]}}, \"conditions\": {\"temperature\": {\"control\": {\"type\": \"AMBIENT\"}}, \"conditions_are_dynamic\": true}, \"workups\": [{\"type\": \"ADDITION\", \"details\": \"was added portionwise at a rate\"}, {\"type\": \"TEMPERATURE\", \"details\": \"to maintain a reflux temperature of about 130\\u00b0-135\\u00b0 C\"}, {\"type\": \"TEMPERATURE\", \"details\": \"The mixture was refluxed an additional two hours\", \"duration\": {\"value\": 2.0, \"units\": \"HOUR\"}}, {\"type\": \"FILTRATION\", \"details\": \"filtered\"}, {\"type\": \"CONCENTRATION\", \"details\": \"the filtrate was concentrated in vacuo\"}], \"outcomes\": [{\"products\": [{\"identifiers\": [{\"type\": \"NAME\", \"value\": \"subject compound\"}], \"measurements\": [{\"type\": \"AMOUNT\", \"details\": \"MASS\", \"amount\": {\"mass\": {\"value\": 6.9, \"units\": \"GRAM\"}}}], \"reaction_role\": \"PRODUCT\"}]}]}
-\n
-"""PREFIX is supposed to be the content of the system prompt. SUFFIX is the user prompt. SHOT is the 1-shot prompt that will be added to the system prompt when used.
-To continue, we loop all over the dataset two times, one for each type of prompt (zero- and one-shot). For each dataset sample, we format the prompt to include the procedure-output schema pairs using the template defined in the previous cell. In addition, we also predict using the model and store those predictions for future evaluation.
-shots = ["0-shot", "1-shot"]
-results_llama = {}
-
-# Start by looping over the shots
-for s in shots:
- predictions = []
- references = []
-
-# Loop over all the samples of the dataset
- for t in test_dataset:
- instruction = t["instruction"]
- output = t["output"]
- # Format the prompt
- if s == "0-shot":
- shot = ""
- else:
- shot = SHOT
- system = PREFIX.format(shot=shot)
- user = SUFFIX.format(sample=instruction)
- prompt = [
- {"role": "system", "content": system},
- {"role": "user", "content": user},
- ]
- # Do the completion using Groq API through LiteLLM
- pred = (
- completion(
- model=base_model,
- messages=prompt,
- caching=True,
- temperature=0,
- )
- .choices[0]
- .message.content
- )
- # Save the predictions and the references for later evaluation
- references.append(output)
- predictions.append(pred)
-
- results_llama[s] = {
- "predictions": predictions,
- "references": references,
- }The beauty of LiteLLM is that it allows obtaining completions from models by different providers using the OpenAI prompt completions’ schema for all providers:
{"role": "system", "content": system_message}, {"role": "user", "content": user_message}
After generating the predictions, it’s essential to evaluate them. We will initially use the BERTScore for a simple evaluation, as it provides precision, recall, and F\(_1\) scores based on similarity measures. However, for a complex schema like the one we are predicting, more robust evaluation methods should be utilized. (REF EVALUATION CHAPTER)
-BERTScore (zhang2020bertscore?) is an evaluation method that proceeds by calculating the similarity of the candidate text with the reference. This similarity is calculated as a sum of cosine similarities token by token. To produce the embeddings this similarity calculation, in the original article, they used the BERT model’s embeddings. However, for our case we will be using the embeddings from the DistilBERT model (sanh2020distilbert?) which achieves 97% of the original BERT model language understanding while only being 40% the size of BERT original model.
-bertscore = load("bertscore")
-shots = ['0-shot', '1-shot']
-
-# Start by looping over the shots
-for s in shots:
- predictions = results_llama[s]["predictions"]
- references = results_llama[s]["references"]
-
- results_ = bertscore.compute(
- predictions=predictions,
- references=references,
- model_type="distilbert-base-uncased",
- )
-
- results_llama[s].update(
- {
- "precision": mean(results_["precision"]),
- "recall": mean(results_["recall"]),
- "f1_scores": mean(results_["f1"]),
- }
- )Results for the 0-shot prompt
- Precision: 0.865
- Recall: 0.8918
- F1-Score: 0.8781
-
-Results for the 1-shot prompt
- Precision: 0.9392
- Recall: 0.9553
- F1-Score: 0.9471
-The results are excellent, especially with the one-shot prompt. However, we are going to try a different model, a closed-source model, to compare.
-The second model we will use is the newer OpenAI, GPT-4o. Doing this allows us to compare open- and closed-source models.
-The procedure and code are exactly the same as for the previous case; the only difference is to define a different model.
-base_model = "gpt-4o"OpenAI models are also supported by the LiteLLM package.
And we obtain the completions using both prompts for all the test samples.
-results_openai = {}
-shots = ["0-shot", "1-shot"]
-
-# Start by looping over the shots
-for s in shots:
- predictions = []
- references = []
-
-# Loop over all the samples of the dataset
- for t in test_dataset:
- instruction = t["instruction"]
- output = t["output"]
- # Format the prompt following OpenAI's prompting guidelines
- if s == "0-shot":
- shot = ""
- else:
- shot = SHOT
- system = PREFIX.format(shot=shot)
- user = SUFFIX.format(sample=instruction)
- prompt = [
- {"role": "system", "content": system},
- {"role": "user", "content": user},
- ]
- # Do the completion using Groq API through LiteLLM
- pred = (
- completion(
- model=base_model,
- messages=prompt,
- caching=True,
- temperature=0,
- )
- .choices[0]
- .message.content
- )
- # Remove some residual stuff in the json output by the model.
- if "```json" in pred:
- pred = pred.replace("```json\n", "")
- pred = pred.replace("```", "")
-
- # Save the predictions and the references for later evaluation
- references.append(output)
- predictions.append(pred)
-
- results_openai[s] = {
- "predictions": predictions,
- "references": references,
- }Finally, we evaluate again using BERTScore.
-for s in shots:
- predictions = results_openai[s]["predictions"]
- references = results_openai[s]["references"]
-
- results_ = bertscore.compute(predictions=predictions, references=references, model_type="distilbert-base-uncased")
-
- results_openai[s].update({
- "precision": mean(results_["precision"]),
- "recall": mean(results_["recall"]),
- "f1_scores": mean(results_["f1"]),
- })Results for the 0-shot prompt
- Precision: 0.8949
- Recall: 0.9093
- F1-Score: 0.9019
-
-Results for the 1-shot prompt
- Precision: 0.9545
- Recall: 0.9619
- F1-Score: 0.9581
-The results with this GPT-4o model are excellent, improving slightly on the ones obtained with the Llama-3 8B base model. However, we are going to try to improve these results further by fine-tuning the Llama-3 8B model.
-As the final step, we will fine-tune the Llama-3 8B using data similar to the one we used above.
-We will use packages built by HuggingFace to do the fine-tuning.
-First, we define the base model we will use and the path of the dataset.
-# Model
-base_model = "meta-llama/Meta-Llama-3-8B-Instruct"It is important to include the HF_token in the .env file. When we created this notebook, the model we will fine-tune (Llama3-8B) was only available after an access request.
The next step is to load the dataset for the fine-tuning. For that, similar to the testing of the previous models, we will use the dataset used by Ai et al., but for this case, we will use their train dataset. Since this is a quick demonstration, we will only take 5000 samples.
-dataset = load_dataset("json", data_files="train.json", split="train")
-dataset = dataset.shuffle(seed=42).select(range(5000)) # Only use 5000 samples for quick demo
-dataset = dataset.train_test_split(test_size=0.1, seed=42) # We define 90-10 % training-evaluation splits.
-datasetDatasetDict({
- train: Dataset({
- features: ['output', 'instruction'],
- num_rows: 4500
- })
- test: Dataset({
- features: ['output', 'instruction'],
- num_rows: 500
- })
-})Then, we define the method to fine-tune the model. For this fine-tuning, we will use the popular QLoRA method. QLoRA (Dettmers et al. 2023) is an efficient approach that reduces memory usage during fine-tuning while preserving full fine-tuning task performance.
-load_in_4bit=True: loads the model using the 4-bit quantization.bnb_4bit_quant_type="nf4": quantizes following the nf4 method.(Dettmers et al. 2022)bnb_4bit_use_double_quant=True: activate nested quantization for 4-bit base models.bnb_4bit_compute_dtype=torch.bfloat16: Compute dtype for 4-bit base models.# QLoRA configuration
-bnb_config = BitsAndBytesConfig(
- load_in_4bit=True,
- bnb_4bit_quant_type="nf4", # fp4 or nf4
- bnb_4bit_use_double_quant=True,
- bnb_4bit_compute_dtype=torch.bfloat16
-)r: The rank of the updated matrices, expressed as integer, meaning that the adaptor that is build in top of the model to improved will be made by matrices of rank 32. Lower rank results in smaller update matrices with fewer trainable parameters that can not be enough to capture the diverse data during the training. On the other hand, higher ranks may lead to overfitting. This rank is a hyperparameter that needs to be optimized.
-lora_alpha: LoRA scaling factor. It changes how the adaptation layer’s weights affect the base model’s.lora_dropout: Dropout is a regularization technique where a proportion of neurons (or parameters) are randomly “dropped out” or turned off during training to prevent overfitting.bias: Specifies if the bias parameters should be trained. Can be ‘none’, ‘all’ or ‘lora_only’.task_type: Task to perform, “Causal LM”: Causal language modeling.peft_config = LoraConfig(
- r=32,
- lora_alpha=64,
- lora_dropout=0.1,
- bias="none",
- task_type="CAUSAL_LM",
-)Before training, we define the tokenizer and the model for fine-tuning, set the training arguments, and initialize the trainer.
-# Load tokenizer
-tokenizer = AutoTokenizer.from_pretrained(base_model) # Define the tokenizer
-tokenizer.pad_token = tokenizer.eos_token
-tokenizer.padding_side = "left" # Where the "pad_token" is placed
-
-# Model config
-model = AutoModelForCausalLM.from_pretrained(
- base_model, # Model that we are going to fine-tune
- quantization_config=bnb_config, # QLoRA config defined above
- device_map="auto", # Where the model is trained, set device_map="auto" loads a model onto available GPUs first.
-)The pad_token is a special token used to make arrays of tokens the same size for batching purpose. The typical is to use the eos_token which is a special token representing the end of a sentence.
learning_rate: the learning rate is a hyperparameter that sets how the training algorithm updates the values of the weights.per_device_train_batch_size: batch size for the training.per_device_eval_batch_size: batch size for the evaluation.gradient_accumulation_steps: number of accumulated gradients over each batch. Gradient accumulation is a technique that simulates a larger batch size and it is very related to the batch size, since it also allows reducing the computation time. However, contrary to the batch size, during the gradient accumulation the weights of the model are not updated during each forward and backward pass, but gradients are accumulated from multiple small batches before performing the update. Thus, setting a higher gradient_accumulation_steps can help to accelerate the training when increasing the batch size is not possible due to VRAM impediments.optim: optimizer used. The main role of the optimizer is to minimize the loss function. The paged_adamw_32bit is the well-known AdamW optimizer. AdamW optimization is a stochastic gradient descent method.(Loshchilov and Hutter 2019)num_train_epochs: number of times that the model goes through each sample during the training. A larger number might lead to the best training results or to overfitting. A lower number might give a model that does not work as expected at all.fp16 and bf16: these parameters help to achieve mixed precision training, which is a technique that aims to optimize the computational efficiency of training models by utilizing lower-precision numerical formats for certain variables. However, the choice of both parameters depends on the architecture of the GPUs that are being used during the training.logging_steps: when the logging is done.evaluation_strategy: the evaluation strategy to adopt during training. The most used is ‘steps’ meaning that the evaluation is done after a certain number of training steps.eval_steps: define in which steps the evaluation is done.max_grad_norm: maximum gradient norm (for gradient clipping). Gradient Clipping is a method where the error derivative is changed or clipped to a threshold during backward propagation through the network, and using the clipped gradients to update the weights.warmup_steps: number of steps used for a linear warm-up from 0 to learning_rate. The warm-up helps to stabilize the optimization process and prevent divergence.warmup_ratio: ratio of total training steps used for the linear warm-up.group_by_length: whether to group together samples of roughly the same length in the training dataset (to minimize padding applied and be more efficient). Only useful if applying dynamic padding.lr_scheduler_type: describes the decay of the learning rate during the training.output_dir: directory to safe the report of the training process.save_strategy: what we want to save during the training. Set it to “no” to only safe the final model.
-lr_scheduler_type option.
-# Define the different hyperparameters and arguments for the fine-tuning
-training_arguments = TrainingArguments(
- learning_rate=6e-5,
- per_device_train_batch_size=2,
- per_device_eval_batch_size=2,
- gradient_accumulation_steps=8,
- optim="paged_adamw_32bit",
- num_train_epochs=10,
- fp16=False,
- bf16=True, #bf16 to True with an A100, False otherwise
- logging_steps=1, # Logging is done every step.
- evaluation_strategy="steps",
- eval_steps=0.01,
- max_grad_norm=0.3,
- warmup_steps=100,
- warmup_ratio=0.03,
- group_by_length=True,
- lr_scheduler_type="cosine",
- output_dir="./results/",
- save_strategy='no',
-)response_template = " ### Answer:"
-collator = DataCollatorForCompletionOnlyLM(response_template, tokenizer=tokenizer)Data collators are objects that will form a batch by using a list of dataset elements as input. There is one data collator for each task, here we use the one for completion-only.
-The completion-only training instead of training the model on the whole input (prompt + answer) make the training more efficient by training only the model on completion. This can sometimes increase the performance, especially for situations like ours in which we want to use the model only for completions, and not to generate more instructions.
-def formatting_prompts_func(example):
- output_texts = []
- for i in range(len(example["instruction"])):
- text = f"### Question: {example["instruction"][i]}\n ### Answer: {example["output"][i]}"
- output_texts.append(text)
- return output_textsThe formatting function is intended for cases where the prompt is constructed from more than one feature of the dataset. Using the formatting functions allows us to join them optimally.
-trainer = SFTTrainer(
- model=model, # Model to fine-tune
- max_seq_length=2048, # Max number of tokens of the completion
- args=training_arguments, # Training arguments to use
- train_dataset=dataset["train"], # Set of the dataset used for the training
- eval_dataset=dataset["test"], # Set of the dataset used for the evaluations
- peft_config=peft_config, # Configuration and PEFT method to use
- tokenizer=tokenizer, # Tokenizer used
- packing=False,
- formatting_func=formatting_prompts_func, # Prompt formatting function
- data_collator=collator,
-)Some of the prompts used are bigger than the default value of 1024 tokens, so we must set a max_seq_length bigger than that.
When packing is set to True during training, multiple short examples fill in the input sequence length instead of padding them to increase training efficiency. However, when using a collator, it must be False.
We set a very small eval_steps variable so the training has a lot of evals, to obtain more visual loss curves. Ideally, that much evals steps are not needed and will make the training process slower.
And finally when everything is ready we train the model.
-trainer.train()| Step | -Training Loss | -Validation Loss | -
|---|---|---|
| 29 | -0.630700 | -0.593230 | -
| 58 | -0.280600 | -0.287035 | -
| 87 | -0.094700 | -0.088160 | -
| 116 | -0.075500 | -0.072126 | -
| 145 | -0.071100 | -0.066134 | -
| 174 | -0.053100 | -0.055799 | -
| 203 | -0.065700 | -0.054912 | -
| 232 | -0.045500 | -0.048931 | -
| 261 | -0.047300 | -0.050089 | -
| 290 | -0.041400 | -0.049855 | -
| 319 | -0.044300 | -0.043511 | -
| 348 | -0.041200 | -0.042149 | -
| 377 | -0.042800 | -0.042795 | -
| 406 | -0.038100 | -0.039256 | -
| 435 | -0.045000 | -0.041538 | -
| 464 | -0.039600 | -0.037373 | -
| 493 | -0.045400 | -0.038947 | -
| 522 | -0.036700 | -0.036142 | -
| 551 | -0.037700 | -0.035618 | -
| 580 | -0.020200 | -0.034974 | -
| 609 | -0.023400 | -0.035457 | -
| 638 | -0.023100 | -0.033618 | -
| 667 | -0.034600 | -0.035628 | -
| 696 | -0.031500 | -0.032545 | -
| 725 | -0.025500 | -0.033716 | -
| 754 | -0.027200 | -0.032997 | -
| 783 | -0.035700 | -0.032624 | -
| 812 | -0.023700 | -0.033319 | -
| 841 | -0.030000 | -0.032736 | -
| 870 | -0.023900 | -0.031512 | -
| 899 | -0.026400 | -0.031785 | -
| 928 | -0.017500 | -0.030995 | -
| 957 | -0.023000 | -0.031565 | -
| 986 | -0.024500 | -0.030332 | -
| 1015 | -0.017600 | -0.030359 | -
| 1044 | -0.020000 | -0.034667 | -
| 1073 | -0.026300 | -0.029186 | -
| 1102 | -0.025200 | -0.030255 | -
| 1131 | -0.021600 | -0.029377 | -
| 1160 | -0.017400 | -0.028646 | -
| 1189 | -0.023600 | -0.029402 | -
| 1218 | -0.026700 | -0.028909 | -
| 1247 | -0.015600 | -0.028068 | -
| 1276 | -0.033300 | -0.029490 | -
| 1305 | -0.020000 | -0.028139 | -
| 1334 | -0.029600 | -0.029012 | -
| 1363 | -0.022500 | -0.027676 | -
| 1392 | -0.014100 | -0.028040 | -
| 1421 | -0.016300 | -0.028978 | -
| 1450 | -0.015900 | -0.027864 | -
| 1479 | -0.017700 | -0.027962 | -
| 1508 | -0.024100 | -0.028795 | -
| 1537 | -0.014900 | -0.027778 | -
| 1566 | -0.020100 | -0.028307 | -
| 1595 | -0.016400 | -0.027065 | -
| 1624 | -0.022100 | -0.027976 | -
| 1653 | -0.009900 | -0.027239 | -
| 1682 | -0.017600 | -0.026706 | -
| 1711 | -0.020000 | -0.027776 | -
| 1740 | -0.020400 | -0.028023 | -
| 1769 | -0.014200 | -0.027051 | -
| 1798 | -0.020500 | -0.028300 | -
| 1827 | -0.012600 | -0.027237 | -
| 1856 | -0.019300 | -0.027134 | -
| 1885 | -0.014500 | -0.027291 | -
| 1914 | -0.014800 | -0.027198 | -
| 1943 | -0.018400 | -0.027487 | -
| 1972 | -0.016500 | -0.026561 | -
| 2001 | -0.013400 | -0.027307 | -
| 2030 | -0.018600 | -0.028109 | -
| 2059 | -0.017900 | -0.028044 | -
| 2088 | -0.019900 | -0.027581 | -
| 2117 | -0.007100 | -0.027354 | -
| 2146 | -0.016400 | -0.027352 | -
| 2175 | -0.020500 | -0.027349 | -
| 2204 | -0.013000 | -0.027181 | -
| 2233 | -0.017700 | -0.027262 | -
| 2262 | -0.019200 | -0.027305 | -
| 2291 | -0.014000 | -0.027452 | -
| 2320 | -0.014300 | -0.027622 | -
| 2349 | -0.004600 | -0.027471 | -
| 2378 | -0.010000 | -0.027483 | -
| 2407 | -0.024700 | -0.027695 | -
| 2436 | -0.011400 | -0.027572 | -
| 2465 | -0.012000 | -0.027545 | -
| 2494 | -0.007100 | -0.027586 | -
| 2523 | -0.010500 | -0.027584 | -
| 2552 | -0.015100 | -0.027578 | -
| 2581 | -0.004900 | -0.027581 | -
| 2610 | -0.016500 | -0.027570 | -
| 2639 | -0.016800 | -0.027609 | -
| 2668 | -0.016900 | -0.027610 | -
| 2697 | -0.019100 | -0.027594 | -
| 2726 | -0.012600 | -0.027575 | -
| 2755 | -0.016100 | -0.027580 | -
| 2784 | -0.022600 | -0.027586 | -
-
TrainOutput(global_step=2810, training_loss=0.03654618244585034, metrics={'train_runtime': 22218.9233, 'train_samples_per_second': 2.025, 'train_steps_per_second': 0.126, 'total_flos': 1.8010605645245645e+18, 'train_loss': 0.03654618244585034, 'epoch': 9.991111111111111})To better visualize how the fine-tuning went, the best option is to plot the loss curves for the training and for the evaluation. The ideal loss curve depicts the model’s loss values over time. At first, the loss is high, but it gradually declines, signifying that the model’s performance is improving.
-
-
-The loss curves produced during the fine-tuning of our model are not far from the ideal behavior meaning that the training proceeded correctly.
- -The loss curves shown here were automatically created by reporting the training to WandB. This is a helpful possibility to easily obtain the loss curves for the fine-tuning.
-The easiest way to evaluate the fine-tuned model and perform inference is to use the trained model directly. To do that, we have to define a pipeline for text generation, do the inference using that pipeline, and evaluate similarly as for the previous models.
-# Define the pipeline that will do the inference
-sft_pipe = pipeline(
- "text-generation",
- do_sample=False, # This allows to set Temperature to 0 (or None for this case)
- temperature=None,
- model=trainer.model, # We do the inference with the trained model.
- tokenizer=tokenizer,
-)The temperature defines the degrees of freedom that are allowed to the model when it predicts the next word. For data extraction, the best value is 0 because the model do not need to make up the data, only to extract it from the corresponding text. This is because at temperature equal to 0, the model is going to pick always the most probable token.
-# Create the 0 and 1-shot prompts.
-results_sft = {}
-prompts_ = {}
-shots = ["0-shot", "1-shot"]
-
-# Start by looping over the shots
-for s in shots:
- references = []
- prompts = []
-
- # Loop over all the samples of the dataset
- for t in test_dataset:
- instruction = t["instruction"]
- output = t["output"]
- references.append(output)
- if s == "0-shot":
- shot = ''
- else:
- shot = SHOT
- # Format the prompt
- system = PREFIX.format(shot=shot)
- user = SUFFIX.format(sample=instruction)
- prompt = system + user
- prompts.append(prompt)
-
- # Save the prompts and the references.
- prompts_[s] = {
- "prompts": prompts,
- }
- results_sft[s] = {
- "references": references,
- }# Do the inference using batching.
-for s in shots:
- # Create a tmp dataset to make easier the batching
- ds = Dataset.from_dict(prompts_[s])
- predictions_sft = []
- # Inference time!
- with torch.cuda.amp.autocast():
- for out in sft_pipe(KeyDataset(ds, "prompts"), batch_size=16):
- # Clean the output.
- for i, sample in enumerate(out):
- sample["generated_text"] = sample["generated_text"].replace(prompts_[s]["prompts"][i], "")
- predictions_sft.append(sample["generated_text"])
-
- # Save the results.
- results_sft[s].update({
- "predictions": predictions_sft,
- })Note that we do not use LiteLLM for this particular case. This is because this package does not yet support completion-only tasks with HuggingFace models. Thus, the prompt is not written according to the OpenAI completions guide.
We perform the inference using the torch.cuda.amp.autocast function. This allows us to avoid errors related with different data types.
Note that after completion, we replace the prompt with an empty string. We do this because the model was always instructed to complete the instructions with the prompt at the beginning. Above, for the case where we used this same Llama model, this was not done because the API already provided the completion without the prompt.
-The limiting factor for the inference when batching is the GPU memory. This is because during inference the GPU will contain not only the model but an amount of prompts equals to the batch size.
-Finally, we calculate the metrics to evaluate this last model’s results.
-for s in shots:
- predictions_sft = results_sft[s]["predictions"]
- references = results_sft[s]["references"]
-
- results = bertscore.compute(predictions=predictions_sft, references=references, model_type="distilbert-base-uncased")
-
- results_sft[s].update({
- "precision": mean(results["precision"]),
- "recall": mean(results["recall"]),
- "f1_scores": mean(results["f1"]),
- })Results for the 0-shot prompt
- Precision: 0.8383
- Recall: 0.906
- F1-Score: 0.8704
-
-Results for the 1-shot prompt
- Precision: 0.8666
- Recall: 0.9126
- F1-Score: 0.8889
-The results using the 0-shot prompt are similar as the other models.
-On the other hand, for this fine-tuned model, the 1-shot results do not show an improvement as big as for the other models. This is because when fine-tuning is done, the model gets used to a very robust prompt-completion format, that for the case of the 1-shot prompt is broken resulting in worse results than expected.
-To study the results more graphically, we can plot all the results in several bar plots.
-# set width of bar
-barWidth = 0.2
-fig = plt.subplots(figsize=(20, 10))
-
-plt_models = ["Llama3-8B", "Llama3-8B Fine-tuned", "GPT-4o"]
-plt_metrics = ["Precision", "Recall", "F1-Score"]
-plt_data = {}
-for index, model in enumerate(plt_models):
- plt_data[model] = metrics_0_shot[index]
-
-# Set position of bar on X axis
-br1 = np.arange(len(metrics_0_shot[0]))
-br2 = [x + barWidth for x in br1]
-br3 = [x + barWidth for x in br2]
-
-# Make the plot
-plt.bar(
- br1,
- metrics_0_shot[0],
- color="b",
- width=barWidth,
- edgecolor="black",
- label=plt_models[0],
-)
-plt.bar(
- br2,
- metrics_0_shot[1],
- color="skyblue",
- width=barWidth,
- edgecolor="black",
- label=plt_models[1],
-)
-plt.bar(
- br3,
- metrics_0_shot[2],
- color="mediumseagreen",
- width=barWidth,
- edgecolor="black",
- label=plt_models[2],
-)
-
-# Adding Xticks
-plt.xlabel("Metric", fontweight="bold", fontsize=15)
-plt.ylabel("Results", fontweight="bold", fontsize=15)
-plt.xticks(
- [r + barWidth for r in range(len(metrics_0_shot[0]))], plt_metrics, fontsize=12
-)
-plt.ylim(0, 1)
-plt.title("Different metrics with 0 shot prompt for the models", fontsize=20)
-
-plt.legend()
-plt.savefig("bars0.png")
-plt.show()
For the 0-shot prompt is possible to see that the best results for all the metrics are the ones obtained when using the closed-source model that is presumably the bigger one of the three. Attending to both Llama3-8B models the results are very similar. The fact that all the results are that similar between them is probably because of the evaluation used.
-# set width of bar
-barWidth = 0.2
-fig = plt.subplots(figsize=(20, 10))
-
-plt_data = {}
-for index, model in enumerate(plt_models):
- plt_data[model] = metrics_1_shot[index]
-
-# Set position of bar on X axis
-br1 = np.arange(len(metrics_1_shot[0]))
-br2 = [x + barWidth for x in br1]
-br3 = [x + barWidth for x in br2]
-
-# Make the plot
-plt.bar(
- br1,
- metrics_1_shot[0],
- color="b",
- width=barWidth,
- edgecolor="black",
- label=plt_models[0],
-)
-plt.bar(
- br2,
- metrics_1_shot[1],
- color="skyblue",
- width=barWidth,
- edgecolor="black",
- label=plt_models[1],
-)
-plt.bar(
- br3,
- metrics_1_shot[2],
- color="mediumseagreen",
- width=barWidth,
- edgecolor="black",
- label=plt_models[2],
-)
-
-# Adding Xticks
-plt.xlabel("Metric", fontweight="bold", fontsize=15)
-plt.ylabel("Results", fontweight="bold", fontsize=15)
-plt.xticks(
- [r + barWidth for r in range(len(metrics_1_shot[0]))], plt_metrics, fontsize=12
-)
-plt.ylim(0, 1)
-plt.title("Different metrics with 1-shot prompt for the models", fontsize=20)
-
-plt.legend()
-plt.savefig("bars1.png")
-plt.show()
When using the 1-shot prompt it is possible to see that the results are slightly better for the closed-source model.
-Additionally, is possible to see that the fine-tuned model perform worse than the vanilla Llama3-8B model. As pointed above, this is because the fine-tuned model is seeing a format that is not the one that saw during training. Because of that, and having these results to prove it, we recommend avoiding the use of few-shot prompts with fine-tuned models.
-# set width of bar
-barWidth = 0.15
-fig = plt.subplots(figsize=(20, 10))
-
-plt_models = ["Llama3-8B 0-shot", "Llama3-8B 1-shot", "Llama3-8B Fine-tuned 0-shot", "Llama3-8B Fine-tuned 1-shot", "GPT-4o 0-shot", "GPT-4o 1-shot"]
-plt_metrics = ["Precision", "Recall", "F1-Score"]
-plt_data = {}
-for index, model in enumerate(plt_models):
- plt_data[model] = metrics_[index]
-
-# Set position of bar on X axis
-br1 = np.arange(len(metrics_[0]))
-br2 = [x + barWidth for x in br1]
-br3 = [x + barWidth for x in br2]
-br4 = [x + barWidth for x in br3]
-br5 = [x + barWidth for x in br4]
-br6 = [x + barWidth for x in br5]
-
-# Make the plot
-plt.bar(
- br1,
- metrics_[0],
- color="blue",
- width=barWidth,
- edgecolor="black",
- label=plt_models[0],
-)
-plt.bar(
- br2,
- metrics_[1],
- color="darkblue",
- width=barWidth,
- edgecolor="black",
- label=plt_models[1],
-)
-plt.bar(
- br3,
- metrics_[2],
- color="lightskyblue",
- width=barWidth,
- edgecolor="black",
- label=plt_models[2],
-)
-plt.bar(
- br4,
- metrics_[3],
- color="deepskyblue",
- width=barWidth,
- edgecolor="black",
- label=plt_models[3],
-)
-plt.bar(
- br5,
- metrics_[4],
- color="springgreen",
- width=barWidth,
- edgecolor="black",
- label=plt_models[4],
-)
-plt.bar(
- br6,
- metrics_[5],
- color="seagreen",
- width=barWidth,
- edgecolor="black",
- label=plt_models[5],
-)
-
-# Adding Xticks
-plt.xlabel("Metric", fontweight="bold", fontsize=15)
-plt.ylabel("Results", fontweight="bold", fontsize=15)
-plt.xticks(
- [r + 2.5 * barWidth for r in range(len(metrics_[0]))], plt_metrics, fontsize=12
-)
-plt.ylim(0, 1)
-plt.title("Different metrics using both prompts for the models", fontsize=20)
-plt.legend(loc='lower right')
-
-plt.savefig("another.png")
-plt.show()
Comparing all the results together allow us to see that the results with the 1-shot prompt are better for the three models compared to when using the 0-shot prompt.
-Also, it is possible to observe that the closed-source model with the 1-shot prompt achieves the best result overall with an F1-score higher than 0.95.
-In addition, it is important to point again the not so good results of the fine-tuned model when using the 1-shot prompt. Again this is because during the fine-tuning the model gets very used to a robust format, and that format is broken with the 1-shot. Commonly, when fine-tuning few-shot prompts are not used with the fine-tuned model due to this reason.
-Finally, claim that the differences are that small probably because of the method used to evaluate. Since the BERTScore method measures the similarity token by token, when applied to a JSON schema in which a lot of tokens are curly brackets and other tokens related with the schema, the results of these metrics are not very reliable. For more robust evaluations, consult the evaluations notebook[REF].
-Structured data is at the heart of machine learning. LLMs offer a convenient way to generate structured data based on unstructured inputs. This book gives hands-on examples of the different steps in the extraction workflow using LLMs.
- - -{#sec-annotation}
-For the manual data annotation one should use an automatically processable file format like JSON format, which provides the data in key-value pairs.
-{
- "key": "value",
- "key2": "value 2"
-}Therefore one could store the desired annotated data in an structured form. For example a JSON file of a Buchwald-Hartwig reaction could look like this:
-{
- "reaction_name": "Buchwald-Hartwig reaction",
- "reactants": [
- "5-Bromo-m-xylene",
- "Benzylmethylamine"
- ],
- "catalyst": [
- "Bis(dibenzylideneacetone)palladium(0)",
- "Tri(o-tolyl)phosphine"
- ],
- "base": "Sodium tert-butoxide",
- "solvent": "Toluene",
- "temperature": 65,
- "temperature_unit": "°C",
- "product": "N-Benzyl-N-methyl(3,5-xylyl)amine",
- "yield": 88
-}To make the process faster and error-free one could use an annotation tool like doccano.
- - - -At the start of the data extraction process you have to collect a set of potentially relevant data sources. Therefore, you could collect a dataset manually or use a tool to help to automate and speed up this process.
-The Crossref API is a very useful tool to collect the metadata of relevant articles. Besides the API there are multiple Python libraries available that make access to the API easier. One of these libraries is crossrefapi. As an example, 100 sources including metadata on the topic ‘buchwald-hartwig coupling’ are extracted and saved into a JSON file.
-from crossref.restful import Works
-import json
-
-works = Works(timeout=60)
-
-# Performing the search for sources on the topic of buchwald-hartwig coupling for 10 papers
-query_result = works.query(bibliographic='buchwald-hartwig coupling').select('DOI', 'title', 'author', 'type', 'publisher', 'issued').sample(10)
-
-results = [item for item in query_result]
-
-# Save 100 results including their metadata in a json file
-with open('buchwald-hartwig_coupling_results.json', 'w') as file:
- json.dump(results, file)
-
-print(results)With the obtained metadata one could afterwards try to filter for relevant or available data sources which could be downloaded through an API provided by the publishers or obtain from a data dump.
-An example for using such an article downloading API is provided in the chapter about data mining.
- - - -There are multiple datasets available which are open for data mining.
-To download full text documents from open access databases the paperscraper tool can be used.
As an example, we will download full text articles from ChemRxiv on the topic of ‘buchwald-hartwig coupling’.
-from paperscraper.get_dumps import chemrxiv
-
-# Download of the ChemRxiv paper dump
-chemrxiv(save_path='chemrxiv_2020-11-10.jsonl')from paperscraper.xrxiv.xrxiv_query import XRXivQuery
-from paperscraper.pdf import save_pdf_from_dump
-import pandas as pd
-
-df = pd.read_json('./chemrxiv_2020-11-10.jsonl', lines=True)
-
-# define keywords for the paper search
-synthesis = ['synthesis']
-reaction = ['buchwald-hartwig']
-
-# combine keywords using "AND" logic, i.e. search for papers that contain both keywords
-query = [synthesis, reaction]
-
-# start searching for relevant papers in the ChemRxiv dump
-querier = XRXivQuery('./chemrxiv_2020-11-10.jsonl')
-querier.search_keywords(query, output_filepath='buchwald-hartwig_coupling_ChemRxiv.jsonl')
-
-# Save PDFs in current folder and name the files by their DOI
-save_pdf_from_dump('./buchwald-hartwig_coupling_ChemRxiv.jsonl', pdf_path='./PDFs', key_to_save='doi')For further steps in the data extraction process annotated data in needed to evaluate the extraction pipeline. For this, one could use an annotation tool like doccano which is shown in the following example.
- - - -At the start of each data extraction process one has to first collect a dataset containing all data sources relevant for the extraction topic. Various Python packages are available to make this process easier and more efficient.
-To run machine learning models on papers, we need to convert them into plain text. This might require parsing PDFs, and sometimes optical character recognition (OCR).
-This is a difficult problem as not only the content of the paper needs to be extracted, but also the layout can be important for the interpretation of the paper. In addition, scientific papers often contain mathematical formulas, which are not easy to parse.
-To address those issues, researchers from Meta have trained the Nougat system, which takes PDF as input and produces Markdown as output.
-For certain applications it might be necessary to clean the data before using it for other downstream tasks.
- - -${missingFields[0]} field.`,
- message: `The items being returned for this search do not include all the required fields. Please ensure that your index items include the ${missingFields[0]} field or use index-fields in your _quarto.yml file to specify the field names.`,
- };
- } else if (missingFields.length > 1) {
- const missingFieldList = missingFields
- .map((field) => {
- return `${field}`;
- })
- .join(", ");
-
- throw {
- name: `Error: Search index is missing the following fields: ${missingFieldList}.`,
- message: `The items being returned for this search do not include all the required fields. Please ensure that your index items includes the following fields: ${missingFieldList}, or use index-fields in your _quarto.yml file to specify the field names.`,
- };
- }
- }
-}
-
-let lastQuery = null;
-function showCopyLink(query, options) {
- const language = options.language;
- lastQuery = query;
- // Insert share icon
- const inputSuffixEl = window.document.body.querySelector(
- ".aa-Form .aa-InputWrapperSuffix"
- );
-
- if (inputSuffixEl) {
- let copyButtonEl = window.document.body.querySelector(
- ".aa-Form .aa-InputWrapperSuffix .aa-CopyButton"
- );
-
- if (copyButtonEl === null) {
- copyButtonEl = window.document.createElement("button");
- copyButtonEl.setAttribute("class", "aa-CopyButton");
- copyButtonEl.setAttribute("type", "button");
- copyButtonEl.setAttribute("title", language["search-copy-link-title"]);
- copyButtonEl.onmousedown = (e) => {
- e.preventDefault();
- e.stopPropagation();
- };
-
- const linkIcon = "bi-clipboard";
- const checkIcon = "bi-check2";
-
- const shareIconEl = window.document.createElement("i");
- shareIconEl.setAttribute("class", `bi ${linkIcon}`);
- copyButtonEl.appendChild(shareIconEl);
- inputSuffixEl.prepend(copyButtonEl);
-
- const clipboard = new window.ClipboardJS(".aa-CopyButton", {
- text: function (_trigger) {
- const copyUrl = new URL(window.location);
- copyUrl.searchParams.set(kQueryArg, lastQuery);
- copyUrl.searchParams.set(kResultsArg, "1");
- return copyUrl.toString();
- },
- });
- clipboard.on("success", function (e) {
- // Focus the input
-
- // button target
- const button = e.trigger;
- const icon = button.querySelector("i.bi");
-
- // flash "checked"
- icon.classList.add(checkIcon);
- icon.classList.remove(linkIcon);
- setTimeout(function () {
- icon.classList.remove(checkIcon);
- icon.classList.add(linkIcon);
- }, 1000);
- });
- }
-
- // If there is a query, show the link icon
- if (copyButtonEl) {
- if (lastQuery && options["copy-button"]) {
- copyButtonEl.style.display = "flex";
- } else {
- copyButtonEl.style.display = "none";
- }
- }
- }
-}
-
-/* Search Index Handling */
-// create the index
-var fuseIndex = undefined;
-var shownWarning = false;
-
-// fuse index options
-const kFuseIndexOptions = {
- keys: [
- { name: "title", weight: 20 },
- { name: "section", weight: 20 },
- { name: "text", weight: 10 },
- ],
- ignoreLocation: true,
- threshold: 0.1,
-};
-
-async function readSearchData() {
- // Initialize the search index on demand
- if (fuseIndex === undefined) {
- if (window.location.protocol === "file:" && !shownWarning) {
- window.alert(
- "Search requires JavaScript features disabled when running in file://... URLs. In order to use search, please run this document in a web server."
- );
- shownWarning = true;
- return;
- }
- const fuse = new window.Fuse([], kFuseIndexOptions);
-
- // fetch the main search.json
- const response = await fetch(offsetURL("search.json"));
- if (response.status == 200) {
- return response.json().then(function (searchDocs) {
- searchDocs.forEach(function (searchDoc) {
- fuse.add(searchDoc);
- });
- fuseIndex = fuse;
- return fuseIndex;
- });
- } else {
- return Promise.reject(
- new Error(
- "Unexpected status from search index request: " + response.status
- )
- );
- }
- }
-
- return fuseIndex;
-}
-
-function inputElement() {
- return window.document.body.querySelector(".aa-Form .aa-Input");
-}
-
-function focusSearchInput() {
- setTimeout(() => {
- const inputEl = inputElement();
- if (inputEl) {
- inputEl.focus();
- }
- }, 50);
-}
-
-/* Panels */
-const kItemTypeDoc = "document";
-const kItemTypeMore = "document-more";
-const kItemTypeItem = "document-item";
-const kItemTypeError = "error";
-
-function renderItem(
- item,
- createElement,
- state,
- setActiveItemId,
- setContext,
- refresh,
- quartoSearchOptions
-) {
- switch (item.type) {
- case kItemTypeDoc:
- return createDocumentCard(
- createElement,
- "file-richtext",
- item.title,
- item.section,
- item.text,
- item.href,
- item.crumbs,
- quartoSearchOptions
- );
- case kItemTypeMore:
- return createMoreCard(
- createElement,
- item,
- state,
- setActiveItemId,
- setContext,
- refresh
- );
- case kItemTypeItem:
- return createSectionCard(
- createElement,
- item.section,
- item.text,
- item.href
- );
- case kItemTypeError:
- return createErrorCard(createElement, item.title, item.text);
- default:
- return undefined;
- }
-}
-
-function createDocumentCard(
- createElement,
- icon,
- title,
- section,
- text,
- href,
- crumbs,
- quartoSearchOptions
-) {
- const iconEl = createElement("i", {
- class: `bi bi-${icon} search-result-icon`,
- });
- const titleEl = createElement("p", { class: "search-result-title" }, title);
- const titleContents = [iconEl, titleEl];
- const showParent = quartoSearchOptions["show-item-context"];
- if (crumbs && showParent) {
- let crumbsOut = undefined;
- const crumbClz = ["search-result-crumbs"];
- if (showParent === "root") {
- crumbsOut = crumbs.length > 1 ? crumbs[0] : undefined;
- } else if (showParent === "parent") {
- crumbsOut = crumbs.length > 1 ? crumbs[crumbs.length - 2] : undefined;
- } else {
- crumbsOut = crumbs.length > 1 ? crumbs.join(" > ") : undefined;
- crumbClz.push("search-result-crumbs-wrap");
- }
-
- const crumbEl = createElement(
- "p",
- { class: crumbClz.join(" ") },
- crumbsOut
- );
- titleContents.push(crumbEl);
- }
-
- const titleContainerEl = createElement(
- "div",
- { class: "search-result-title-container" },
- titleContents
- );
-
- const textEls = [];
- if (section) {
- const sectionEl = createElement(
- "p",
- { class: "search-result-section" },
- section
- );
- textEls.push(sectionEl);
- }
- const descEl = createElement("p", {
- class: "search-result-text",
- dangerouslySetInnerHTML: {
- __html: text,
- },
- });
- textEls.push(descEl);
-
- const textContainerEl = createElement(
- "div",
- { class: "search-result-text-container" },
- textEls
- );
-
- const containerEl = createElement(
- "div",
- {
- class: "search-result-container",
- },
- [titleContainerEl, textContainerEl]
- );
-
- const linkEl = createElement(
- "a",
- {
- href: offsetURL(href),
- class: "search-result-link",
- },
- containerEl
- );
-
- const classes = ["search-result-doc", "search-item"];
- if (!section) {
- classes.push("document-selectable");
- }
-
- return createElement(
- "div",
- {
- class: classes.join(" "),
- },
- linkEl
- );
-}
-
-function createMoreCard(
- createElement,
- item,
- state,
- setActiveItemId,
- setContext,
- refresh
-) {
- const moreCardEl = createElement(
- "div",
- {
- class: "search-result-more search-item",
- onClick: (e) => {
- // Handle expanding the sections by adding the expanded
- // section to the list of expanded sections
- toggleExpanded(item, state, setContext, setActiveItemId, refresh);
- e.stopPropagation();
- },
- },
- item.title
- );
-
- return moreCardEl;
-}
-
-function toggleExpanded(item, state, setContext, setActiveItemId, refresh) {
- const expanded = state.context.expanded || [];
- if (expanded.includes(item.target)) {
- setContext({
- expanded: expanded.filter((target) => target !== item.target),
- });
- } else {
- setContext({ expanded: [...expanded, item.target] });
- }
-
- refresh();
- setActiveItemId(item.__autocomplete_id);
-}
-
-function createSectionCard(createElement, section, text, href) {
- const sectionEl = createSection(createElement, section, text, href);
- return createElement(
- "div",
- {
- class: "search-result-doc-section search-item",
- },
- sectionEl
- );
-}
-
-function createSection(createElement, title, text, href) {
- const descEl = createElement("p", {
- class: "search-result-text",
- dangerouslySetInnerHTML: {
- __html: text,
- },
- });
-
- const titleEl = createElement("p", { class: "search-result-section" }, title);
- const linkEl = createElement(
- "a",
- {
- href: offsetURL(href),
- class: "search-result-link",
- },
- [titleEl, descEl]
- );
- return linkEl;
-}
-
-function createErrorCard(createElement, title, text) {
- const descEl = createElement("p", {
- class: "search-error-text",
- dangerouslySetInnerHTML: {
- __html: text,
- },
- });
-
- const titleEl = createElement("p", {
- class: "search-error-title",
- dangerouslySetInnerHTML: {
- __html: ` ${title}`,
- },
- });
- const errorEl = createElement("div", { class: "search-error" }, [
- titleEl,
- descEl,
- ]);
- return errorEl;
-}
-
-function positionPanel(pos) {
- const panelEl = window.document.querySelector(
- "#quarto-search-results .aa-Panel"
- );
- const inputEl = window.document.querySelector(
- "#quarto-search .aa-Autocomplete"
- );
-
- if (panelEl && inputEl) {
- panelEl.style.top = `${Math.round(panelEl.offsetTop)}px`;
- if (pos === "start") {
- panelEl.style.left = `${Math.round(inputEl.left)}px`;
- } else {
- panelEl.style.right = `${Math.round(inputEl.offsetRight)}px`;
- }
- }
-}
-
-/* Highlighting */
-// highlighting functions
-function highlightMatch(query, text) {
- if (text) {
- const start = text.toLowerCase().indexOf(query.toLowerCase());
- if (start !== -1) {
- const startMark = "";
- const endMark = "";
-
- const end = start + query.length;
- text =
- text.slice(0, start) +
- startMark +
- text.slice(start, end) +
- endMark +
- text.slice(end);
- const startInfo = clipStart(text, start);
- const endInfo = clipEnd(
- text,
- startInfo.position + startMark.length + endMark.length
- );
- text =
- startInfo.prefix +
- text.slice(startInfo.position, endInfo.position) +
- endInfo.suffix;
-
- return text;
- } else {
- return text;
- }
- } else {
- return text;
- }
-}
-
-function clipStart(text, pos) {
- const clipStart = pos - 50;
- if (clipStart < 0) {
- // This will just return the start of the string
- return {
- position: 0,
- prefix: "",
- };
- } else {
- // We're clipping before the start of the string, walk backwards to the first space.
- const spacePos = findSpace(text, pos, -1);
- return {
- position: spacePos.position,
- prefix: "",
- };
- }
-}
-
-function clipEnd(text, pos) {
- const clipEnd = pos + 200;
- if (clipEnd > text.length) {
- return {
- position: text.length,
- suffix: "",
- };
- } else {
- const spacePos = findSpace(text, clipEnd, 1);
- return {
- position: spacePos.position,
- suffix: spacePos.clipped ? "…" : "",
- };
- }
-}
-
-function findSpace(text, start, step) {
- let stepPos = start;
- while (stepPos > -1 && stepPos < text.length) {
- const char = text[stepPos];
- if (char === " " || char === "," || char === ":") {
- return {
- position: step === 1 ? stepPos : stepPos - step,
- clipped: stepPos > 1 && stepPos < text.length,
- };
- }
- stepPos = stepPos + step;
- }
-
- return {
- position: stepPos - step,
- clipped: false,
- };
-}
-
-// removes highlighting as implemented by the mark tag
-function clearHighlight(searchterm, el) {
- const childNodes = el.childNodes;
- for (let i = childNodes.length - 1; i >= 0; i--) {
- const node = childNodes[i];
- if (node.nodeType === Node.ELEMENT_NODE) {
- if (
- node.tagName === "MARK" &&
- node.innerText.toLowerCase() === searchterm.toLowerCase()
- ) {
- el.replaceChild(document.createTextNode(node.innerText), node);
- } else {
- clearHighlight(searchterm, node);
- }
- }
- }
-}
-
-function escapeRegExp(string) {
- return string.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); // $& means the whole matched string
-}
-
-// highlight matches
-function highlight(term, el) {
- const termRegex = new RegExp(term, "ig");
- const childNodes = el.childNodes;
-
- // walk back to front avoid mutating elements in front of us
- for (let i = childNodes.length - 1; i >= 0; i--) {
- const node = childNodes[i];
-
- if (node.nodeType === Node.TEXT_NODE) {
- // Search text nodes for text to highlight
- const text = node.nodeValue;
-
- let startIndex = 0;
- let matchIndex = text.search(termRegex);
- if (matchIndex > -1) {
- const markFragment = document.createDocumentFragment();
- while (matchIndex > -1) {
- const prefix = text.slice(startIndex, matchIndex);
- markFragment.appendChild(document.createTextNode(prefix));
-
- const mark = document.createElement("mark");
- mark.appendChild(
- document.createTextNode(
- text.slice(matchIndex, matchIndex + term.length)
- )
- );
- markFragment.appendChild(mark);
-
- startIndex = matchIndex + term.length;
- matchIndex = text.slice(startIndex).search(new RegExp(term, "ig"));
- if (matchIndex > -1) {
- matchIndex = startIndex + matchIndex;
- }
- }
- if (startIndex < text.length) {
- markFragment.appendChild(
- document.createTextNode(text.slice(startIndex, text.length))
- );
- }
-
- el.replaceChild(markFragment, node);
- }
- } else if (node.nodeType === Node.ELEMENT_NODE) {
- // recurse through elements
- highlight(term, node);
- }
- }
-}
-
-/* Link Handling */
-// get the offset from this page for a given site root relative url
-function offsetURL(url) {
- var offset = getMeta("quarto:offset");
- return offset ? offset + url : url;
-}
-
-// read a meta tag value
-function getMeta(metaName) {
- var metas = window.document.getElementsByTagName("meta");
- for (let i = 0; i < metas.length; i++) {
- if (metas[i].getAttribute("name") === metaName) {
- return metas[i].getAttribute("content");
- }
- }
- return "";
-}
-
-function algoliaSearch(query, limit, algoliaOptions) {
- const { getAlgoliaResults } = window["@algolia/autocomplete-preset-algolia"];
-
- const applicationId = algoliaOptions["application-id"];
- const searchOnlyApiKey = algoliaOptions["search-only-api-key"];
- const indexName = algoliaOptions["index-name"];
- const indexFields = algoliaOptions["index-fields"];
- const searchClient = window.algoliasearch(applicationId, searchOnlyApiKey);
- const searchParams = algoliaOptions["params"];
- const searchAnalytics = !!algoliaOptions["analytics-events"];
-
- return getAlgoliaResults({
- searchClient,
- queries: [
- {
- indexName: indexName,
- query,
- params: {
- hitsPerPage: limit,
- clickAnalytics: searchAnalytics,
- ...searchParams,
- },
- },
- ],
- transformResponse: (response) => {
- if (!indexFields) {
- return response.hits.map((hit) => {
- return hit.map((item) => {
- return {
- ...item,
- text: highlightMatch(query, item.text),
- };
- });
- });
- } else {
- const remappedHits = response.hits.map((hit) => {
- return hit.map((item) => {
- const newItem = { ...item };
- ["href", "section", "title", "text", "crumbs"].forEach(
- (keyName) => {
- const mappedName = indexFields[keyName];
- if (
- mappedName &&
- item[mappedName] !== undefined &&
- mappedName !== keyName
- ) {
- newItem[keyName] = item[mappedName];
- delete newItem[mappedName];
- }
- }
- );
- newItem.text = highlightMatch(query, newItem.text);
- return newItem;
- });
- });
- return remappedHits;
- }
- },
- });
-}
-
-let subSearchTerm = undefined;
-let subSearchFuse = undefined;
-const kFuseMaxWait = 125;
-
-async function fuseSearch(query, fuse, fuseOptions) {
- let index = fuse;
- // Fuse.js using the Bitap algorithm for text matching which runs in
- // O(nm) time (no matter the structure of the text). In our case this
- // means that long search terms mixed with large index gets very slow
- //
- // This injects a subIndex that will be used once the terms get long enough
- // Usually making this subindex is cheap since there will typically be
- // a subset of results matching the existing query
- if (subSearchFuse !== undefined && query.startsWith(subSearchTerm)) {
- // Use the existing subSearchFuse
- index = subSearchFuse;
- } else if (subSearchFuse !== undefined) {
- // The term changed, discard the existing fuse
- subSearchFuse = undefined;
- subSearchTerm = undefined;
- }
-
- // Search using the active fuse
- const then = performance.now();
- const resultsRaw = await index.search(query, fuseOptions);
- const now = performance.now();
-
- const results = resultsRaw.map((result) => {
- const addParam = (url, name, value) => {
- const anchorParts = url.split("#");
- const baseUrl = anchorParts[0];
- const sep = baseUrl.search("\\?") > 0 ? "&" : "?";
- anchorParts[0] = baseUrl + sep + name + "=" + value;
- return anchorParts.join("#");
- };
-
- return {
- title: result.item.title,
- section: result.item.section,
- href: addParam(result.item.href, kQueryArg, query),
- text: highlightMatch(query, result.item.text),
- crumbs: result.item.crumbs,
- };
- });
-
- // If we don't have a subfuse and the query is long enough, go ahead
- // and create a subfuse to use for subsequent queries
- if (now - then > kFuseMaxWait && subSearchFuse === undefined) {
- subSearchTerm = query;
- subSearchFuse = new window.Fuse([], kFuseIndexOptions);
- resultsRaw.forEach((rr) => {
- subSearchFuse.add(rr.item);
- });
- }
- return results;
-}