Lm eval documentation #538
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Closed
eturner24
requested changes
Nov 15, 2024
| [role='_abstract'] | ||
| LM-Eval service defines a new Custom Resource Definition (CRD) called `LMEvalJob`. An `LMEvalJob` object represents an evaluation job. `LMEvalJob` objects are monitored by the TrustyAI Kubernetes operator. | ||
|
|
||
| Therefore, to run an evaluation job, you first need to create an `LMEvalJob` object with the following information: `model`, `model arguments`, `task`, and `secret`. |
There was a problem hiding this comment.
Suggested change
| Therefore, to run an evaluation job, you first need to create an `LMEvalJob` object with the following information: `model`, `model arguments`, `task`, and `secret`. | |
| To run an evaluation job, you create an `LMEvalJob` object with the following information: `model`, `model arguments`, `task`, and `secret`. |
|
|
||
| Therefore, to run an evaluation job, you first need to create an `LMEvalJob` object with the following information: `model`, `model arguments`, `task`, and `secret`. | ||
|
|
||
| Once the `LMEvalJob` is created, the LM-Eval service will run the evaluation job and update the status and results to the `LMEvalJob` object when the information is available. |
There was a problem hiding this comment.
Suggested change
| Once the `LMEvalJob` is created, the LM-Eval service will run the evaluation job and update the status and results to the `LMEvalJob` object when the information is available. | |
| After the `LMEvalJob` is created, the LM-Eval service runs the evaluation job. The status and results of the `LMEvalJob` object update when the information is available. |
Using "once" to mean "after" can be confusing to non-English speakers - using "after" is more precise.
Reference: https://www.ibm.com/docs/en/ibm-style?topic=word-usage#o
|
|
||
| [NOTE] | ||
| -- | ||
| It is not recommended to deploy the TrustyAI custom resource (CR) in any namespace that contains non-tabular models. Such models are unsupported by TrustyAI, and may cause errors within the TrustyAI service. |
There was a problem hiding this comment.
Suggested change
| It is not recommended to deploy the TrustyAI custom resource (CR) in any namespace that contains non-tabular models. Such models are unsupported by TrustyAI, and may cause errors within the TrustyAI service. | |
| TrustyAI does not support non-tabular models. Deploying the TrustyAI custom resource (CR) in a namespace that contains non-tabular models can cause errors within the TrustyAI service. |
Rewriting for passive voice. Also better to state the recommendation rather that saying "it is not recommended"
References:
|
|
||
| .Sample LMEvalJob object | ||
|
|
||
| Below is an example of an `LMEvalJob` object. |
There was a problem hiding this comment.
Suggested change
| Below is an example of an `LMEvalJob` object. | |
| The sample `LMEvalJob` object contains the following features: |
Avoid use of "below"
Do not use to indicate a relative location in a document, as in “the information below”.
From: https://www.ibm.com/docs/en/ibm-style?topic=word-usage#b
There was a problem hiding this comment.
I wonder if this list might also be better as callouts in the code.
|
|
||
| Below is an example of an `LMEvalJob` object. | ||
|
|
||
| * It uses the `google/flan-t5-base` model from link:https://huggingface.co/google/flan-t5-base[Hugging Face]. |
There was a problem hiding this comment.
Suggested change
| * It uses the `google/flan-t5-base` model from link:https://huggingface.co/google/flan-t5-base[Hugging Face]. | |
| * The `google/flan-t5-base` model from link:https://huggingface.co/google/flan-t5-base[Hugging Face]. |
| ** `env`: Specify environment variables. It uses the `EnvVar` data structure of kubernetes. | ||
| ** `volumeMounts`: Mount the volumes into the lm-eval container | ||
| ** `resources`: Specify the resources for the lm-eval container. | ||
| * `volumes`: Specify the volume information for the lm-eval and other containers. It uses the `Volume` data structure of kubernetes. |
There was a problem hiding this comment.
Suggested change
| * `volumes`: Specify the volume information for the lm-eval and other containers. It uses the `Volume` data structure of kubernetes. | |
| * `volumes`: Specifies the volume information for the `lm-eval` and other containers. This parameter uses the `Volume` data structure of Kubernetes. |
| ** `volumeMounts`: Mount the volumes into the lm-eval container | ||
| ** `resources`: Specify the resources for the lm-eval container. | ||
| * `volumes`: Specify the volume information for the lm-eval and other containers. It uses the `Volume` data structure of kubernetes. | ||
| * `sideCars`: A list of containers that run along with the lm-eval container. It uses the `Container` data structure of kubernetes. |
There was a problem hiding this comment.
Suggested change
| * `sideCars`: A list of containers that run along with the lm-eval container. It uses the `Container` data structure of kubernetes. | |
| * `sideCars`: A list of containers that run along with the `lm-eval` container. It uses the `Container` data structure of Kubernetes. |
|
|
||
|
|
||
| | `outputs` | ||
| | This section defines custom output locations for the evaluation results storage. At the moment only Persistent Volume Claims (PVC) are supported. |
There was a problem hiding this comment.
Suggested change
| | This section defines custom output locations for the evaluation results storage. At the moment only Persistent Volume Claims (PVC) are supported. | |
| | This parameter defines a custom output location to store the the evaluation results. Only Persistent Volume Claims (PVC) are supported. |
Not sure if parameter is the right word
| | This section defines custom output locations for the evaluation results storage. At the moment only Persistent Volume Claims (PVC) are supported. | ||
|
|
||
| | `outputs.pvcManaged` | ||
| | Create an operator-managed PVC to store this job's results. The PVC will be named `<job-name>-pvc` and will be owned by the `LMEvalJob`. After job completion, the PVC will still be available, but it will be deleted upon deleting the `LMEvalJob`. Supports the following fields: |
There was a problem hiding this comment.
Suggested change
| | Create an operator-managed PVC to store this job's results. The PVC will be named `<job-name>-pvc` and will be owned by the `LMEvalJob`. After job completion, the PVC will still be available, but it will be deleted upon deleting the `LMEvalJob`. Supports the following fields: | |
| | Creates an operator-managed PVC to store this job's results. The PVC is named `<job-name>-pvc` and is owned by the `LMEvalJob`. After the job finishes, the PVC is still be available, but it is deleted with the `LMEvalJob`. Supports the following fields: |
Can we find a different wording than "is owned by" - in addition to being passive voice, it anthropomorphizes the job. More info: https://www.ibm.com/docs/en/ibm-style?topic=grammar-anthropomorphism
| = LM-Eval evaluation job | ||
|
|
||
| [role='_abstract'] | ||
| LM-Eval service defines a new Custom Resource Definition (CRD) called `LMEvalJob`. An `LMEvalJob` object represents an evaluation job. `LMEvalJob` objects are monitored by the TrustyAI Kubernetes operator. |
There was a problem hiding this comment.
Where is the LM-Eval service? RHOAI or Openshift? Should it be code formatted?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
LM-Eval upstream contents x4 modules (incl assembly) and updated TrustyAI book to include the LMEval assembly. Conditionalized the bias tutorial and explainability assembly to only appear upstream.