Code

Upstream checkpoint 1#5

Code

Open

sedrick-keh-tri wants to merge 31 commits intobased-fork-3TRI-ML/lm-evaluation-harness:based-fork-3from upstream-checkpoint-1TRI-ML/lm-evaluation-harness:upstream-checkpoint-1Copy head branch name to clipboard

+11,000-1,261

Commits

Commits on Feb 15, 2024

• Update README.md (EleutherAI#1430)

  davidbhoffmann

  authored
  f3b7917

  Copy full SHA for f3b7917

Commits on Feb 18, 2024

• improve hf_hub activation (EleutherAI#1438)

  michaelfeil

  authored
  a604f05

  Copy full SHA for a604f05

Commits on Feb 19, 2024

• Correct typo in task name (EleutherAI#1443)

  larekrow

  authored
  19cbb29

  Copy full SHA for 19cbb29

• update bbh, gsm8k, mmlu parsing logic and prompts (Orca2 bbh_cot_zeroshot 0% -> 42%) (EleutherAI#1356)

  Show description for 89deeea
  

  thnkinbtfly

  and

  haileyschoelkopf

  authored
  89deeea

  Copy full SHA for 89deeea

Commits on Feb 20, 2024

• Add a new task HaeRae-Bench (EleutherAI#1445)

  Show description for 8680e93
  h-albert-lee

  authored
  8680e93

  Copy full SHA for 8680e93

• Group reqs by context (EleutherAI#1425)

  Show description for 45941c6
  

  baberabb

  and

  haileyschoelkopf

  authored
  45941c6

  Copy full SHA for 45941c6

• Add a new task GPQA (the part without CoT) (EleutherAI#1434)

  Show description for 5ab295c
  

  uanu2002

  and

  haileyschoelkopf

  authored
  5ab295c

  Copy full SHA for 5ab295c

Commits on Feb 21, 2024

• Added KMMLU evaluation method and changed ReadMe (EleutherAI#1447)

  Show description for c26a6ac
  

  3 peopleauthored
  c26a6ac

  Copy full SHA for c26a6ac

Commits on Feb 22, 2024

• Add TemplateLM boilerplate LM class (EleutherAI#1279)

  Show description for ba5cdf0
  

  16 peopleauthored
  ba5cdf0

  Copy full SHA for ba5cdf0

• Log which subtasks were called with which groups (EleutherAI#1456)

  Show description for 00dc996
  haileyschoelkopf

  authored
  00dc996

  Copy full SHA for 00dc996

• PR fixing the issue EleutherAI#1391 (wrong contexts in the mgsm task) (EleutherAI#1440)

  Show description for a72babb
  

  leocnj

  and

  haileyschoelkopf

  authored
  a72babb

  Copy full SHA for a72babb

• feat: Add Weights and Biases support (EleutherAI#1339)

  Show description for 2683fbb
  

  ayulockin

  and

  lintangsutawika

  authored
  2683fbb

  Copy full SHA for 2683fbb

• Fixed generation args issue affection OpenAI completion model (EleutherAI#1458)

  Show description for 75ac1f4
  

  Am1n3e

  and

  haileyschoelkopf

  authored
  75ac1f4

  Copy full SHA for 75ac1f4

Commits on Feb 23, 2024

• update parsing logic of mgsm following gsm8k (EleutherAI#1462)

  thnkinbtfly

  authored
  8371662

  Copy full SHA for 8371662

• Adding documentation for Weights and Biases CLI interface (EleutherAI#1466)

  Show description for eacb74e
  veekaybee

  authored
  eacb74e

  Copy full SHA for eacb74e

Commits on Feb 24, 2024

• Add environment and transformers version logging in results dump (EleutherAI#1464)

  Show description for f78e2da
  LSinev

  authored
  f78e2da

  Copy full SHA for f78e2da

Commits on Feb 26, 2024

• Apply code autoformatting with Ruff to tasks/*.py an *__init__.py (EleutherAI#1469)

  LSinev

  authored
  d27c0c0

  Copy full SHA for d27c0c0

• setting trust_remote_code (EleutherAI#1467)

  veekaybee

  authored
  c1145df

  Copy full SHA for c1145df

• add arabic mmlu (EleutherAI#1402)

  Show description for 7de7b27
  khalil-Hennara

  authored
  7de7b27

  Copy full SHA for 7de7b27

• Add Gemma support (Add flag to control BOS token usage) (EleutherAI#1465)

  Show description for 4c51111
  

  haileyschoelkopf

  and

  lintangsutawika

  authored
  4c51111

  Copy full SHA for 4c51111

• Revert "setting trust_remote_code (EleutherAI#1467)" (EleutherAI#1474)

  Show description for f6befdb
  haileyschoelkopf

  authored
  f6befdb

  Copy full SHA for f6befdb

• Create a means for caching task registration and request building. Ad… (EleutherAI#1372)

  Show description for 1e6c927
  

  inf3rnus

  and

  haileyschoelkopf

  authored
  1e6c927

  Copy full SHA for 1e6c927

• Cont metrics (EleutherAI#1475)

  Show description for 96d185f
  

  lintangsutawika

  and

  haileyschoelkopf

  authored
  96d185f

  Copy full SHA for 96d185f

Commits on Feb 27, 2024

• Refactor evaluater.evaluate (EleutherAI#1441)

  Show description for 5ccd65d
  

  baberabb

  and

  haileyschoelkopf

  authored
  5ccd65d

  Copy full SHA for 5ccd65d

• add multilingual mmlu eval (EleutherAI#1484)

  jordane95

  authored
  7cd004c

  Copy full SHA for 7cd004c

• update name of val split in truthfulqa multilingual (EleutherAI#1488)

  haileyschoelkopf

  authored
  a08eb87

  Copy full SHA for a08eb87

• Fix AttributeError in huggingface.py When 'model_type' is Missing (EleutherAI#1489)

  Show description for cc771ec
  

  richwardle

  and

  haileyschoelkopf

  authored
  cc771ec

  Copy full SHA for cc771ec

Commits on Feb 28, 2024

• fix duplicated kwargs in some model init (EleutherAI#1495)

  lchu6

  authored
  b177c82

  Copy full SHA for b177c82

Commits on Mar 1, 2024

• Add multilingual truthfulqa targets (EleutherAI#1499)

  jordane95

  authored
  d272c19

  Copy full SHA for d272c19

• always include EOS token in stopsequences if possible (EleutherAI#1480)

  haileyschoelkopf

  authored
  284dd80

  Copy full SHA for 284dd80

Commits on Mar 24, 2024

• Merge branch 'based-fork-3' into upstream-checkpoint-1

  sedrick-keh-tri

  authored
  58b49c5

  Copy full SHA for 58b49c5