Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to save probabilities #289

Closed
wants to merge 6 commits into from
Closed

Add option to save probabilities #289

wants to merge 6 commits into from

Conversation

AlexTMallen
Copy link
Collaborator

  • modifies prepare_data to return a LayerData object
  • adds an option to save the raw probabilities, alongside the texts, to the results dir
  • adds train_lr_eval.csv to results dir

@@ -31,7 +31,7 @@ def execute(self, highlight_color: Color = "cyan"):
@torch.inference_mode()
def apply_to_layer(
self, layer: int, devices: list[str], world_size: int
) -> dict[str, pd.DataFrame]:
) -> tuple[dict[str, pd.DataFrame], dict]:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe you could do something similar to here: https://github.com/EleutherAI/elk/pull/259/files#diff-d13b83b80dc8fe2ae73e22669dd7a1a3167a1ae731d341fa96f03a766d877933R37
🟢

Instead of having tuple[dict[str, pd.DataFrame], dict]

But we can also leave it for now, and once we merge our pull-request it will be changed anyway

{
**meta,
"ensembling": mode,
"inlp_iter": i,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is inlp here?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Iterated nullspace projection iteration for the logistic regression model.

@@ -11,7 +11,7 @@
from simple_parsing import subgroups
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function apply_to_layer was already a bit long. I think we should refactor it a bit in a second pull-request

get_logprobs(val_lr_credences, mode).detach().cpu()
)

row_bufs["train_lr_eval"].append(
Copy link
Collaborator

@lauritowal lauritowal Aug 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the names are getting a bit confusing. Maybe we should have a subfolder

"evals" containing

  • trainset_lr.csv
  • validationset_lr.csv
  • trainset_ccs.csv
  • etc.

Copy link
Collaborator

@lauritowal lauritowal Aug 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Could be also included in a second pull-request) 🟢

Copy link
Collaborator

@lauritowal lauritowal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the function apply_to_layer is getting a bit long and maybe a bit confusing, so we might want to create a refactor pull-request in a second step. Everything seems to work fine, though.

@norabelrose
Copy link
Member

included in #292

@AlexTMallen AlexTMallen deleted the save_preds branch November 2, 2023 18:23
@AlexTMallen AlexTMallen restored the save_preds branch November 2, 2023 18:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants