Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for VertexAI, update dependencies, update question sets and add results for some new experiments #29

Merged
merged 82 commits into from
Aug 28, 2024
Merged

Add support for VertexAI, update dependencies, update question sets and add results for some new experiments #29

merged 82 commits into from
Aug 28, 2024

Conversation

semio
Copy link
Collaborator

@semio semio commented May 12, 2024 •
edited
Loading

main points:

  • update mypy version because the previous one is very old and caused github action issues
  • the code changes in ai_eval_spreadsheet schema and wrapper were due to mypy update.
  • added VERTEXAI_* settings in config and support querying VertexAI
  • update question sets and results from new experiments
  • add some instruction for setting environment variables in README
  • add support for using Claude for evaluation

semio added 30 commits February 20, 2024 11:19
@semio
handle empty question set
bdb7f30
@semio
fetch latest questions from contentful
308a404
@semio
use gpt4 turbo
c4b9501
@semio
configuration for gpt4
8fed9b6
@semio
only keep 30 questions for next eval
364f8a2
@semio
add gpt-4 result
8c12646
@semio
config for gemini pro v1
681c042
@semio
handle Gemini Pro Errors
08170ea 
Some times Gemini Pro will not response because of "Recitation Reasons",
and litellm will raise an Error. Instead of raising error let's create
an response stating the situation.
@semio
fix response is not declared when there is error
6721bca
@semio
chinese questions
c87a7df
@semio
add experiment config for chinese questions
00ed2a2
@semio
add more waiting time for alibaba models
358d7d2
@semio
gemini pro and alibaba results
93982ce
@semio
update results
e01d42e
@semio
force question_id to be string
0bb6c7d
@semio
make archive for 60 prompts test
d7de6ff
@semio
new experiment
947cea7
@semio
update the evaluator
0f1219a
@semio
add experiment config
f71c4e5
@semio
add no_option_letter
cb7c166
@semio
add results for gemini
633717b
@semio
new chinese questions and config for qwen-max
48162c1
@semio
result for qwen
d220720
@semio
config for gpt-4
21741e9
@semio
results for gpt4
a32b100
@semio
generate results.xlsx
06cb90e
@semio
one of chinese prompt translation is not correct, fix and add new config
61105dc
@semio
more qwen data
d8212df
@semio
update results
6d751ae
@semio
update notebook
fa91aa6
semio added 27 commits May 18, 2024 10:54
@semio
create archive for experiment 202405012311
bd8a3dd
@semio
experiment for 4 missing climate study questions
3112b6b
@semio
update dependencies
650bb90
@semio
update notebook
7352218
@semio
include all questions again
6967421
@semio
use bigger max_tokens for evaluators
4f902a6
@semio
update evaluators
2feba0f 
- move common functions into one file
- add llama3 based evaluator
@semio
add llama3 evaluator in config generation
2c85d47
@semio
add more experiment archives
7e12e41
@semio
update notebooks
5b41c01
@semio
misc
dc17205
@semio
new experiment for gpt4o
7033684
@semio
dependency
7088b4b
@semio
session result sheet is not in use
6348b39
@semio
improve the evaluator prompt
0dd9229
@semio
new experiment and results for llama3 and claude3 opus
a636c39
@semio
update notebooks
a89d5f5
@semio
update notebook
1c49fb8
@semio
add more columns to the results.xlsx
cb13f5a 
- auto marked correctness
- human rating scores
@semio
also update a few archived results
3120772
@semio
experiment and result for qwen-max-0428
e7c32d7
@semio
notebooks update
e5ad78d
@semio
results.xlsx for latest experiment, with human rating
b2ca8f5
@semio
update notebooks
a0208e6
@semio
update results with human rating
6e62b18
@semio
update notebook
e0df9bd
@semio
remove cli.py because we don't use it any more
3b13abe
@semio semio merged commit 3fe7eba into main Aug 28, 2024
2 checks passed
@semio semio deleted the update_questions branch August 28, 2024 08:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@motin motin motin approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@semio @motin