Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add GGML Benchmarks #129

Closed
wants to merge 1 commit into from
Closed

Add GGML Benchmarks #129

wants to merge 1 commit into from

Conversation

mariecwhite
Copy link
Collaborator

@mariecwhite mariecwhite commented Sep 4, 2023 •
edited
Loading

Adds GPT2 GGML Benchmarks for FP32, FP16, INT4. Also runs under 1, 8 and 16 threads.

name device data_type num_threads first_prediction_ms loop_prediction_ms
GPT2LMHEAD_FP32_GGML c2-standard-16 fp32 1 69.8 40.2
GPT2LMHEAD_FP32_GGML c2-standard-16 fp32 8 12.9 7.9
GPT2LMHEAD_FP32_GGML c2-standard-16 fp32 16 11.5 7.7
GPT2LMHEAD_FP16_GGML c2-standard-16 fp16 1 50.6 23.5
GPT2LMHEAD_FP16_GGML c2-standard-16 fp16 8 8.4 4.9
GPT2LMHEAD_FP16_GGML c2-standard-16 fp16 16 7.7 4.8
GPT2LMHEAD_INT4_GGML c2-standard-16 int4 1 43.4 14.5
GPT2LMHEAD_INT4_GGML c2-standard-16 int4 8 7.2 3.2
GPT2LMHEAD_INT4_GGML c2-standard-16 int4 16 6.7 3.3

@mariecwhite mariecwhite force-pushed the ggml branch 7 times, most recently from eb5e652 to 35f79f2 Compare September 4, 2023 04:58
mariecwhite
mariecwhite commented Sep 4, 2023
View reviewed changes
.github/workflows/run_ggml_benchmark.yml

on:
workflow_dispatch:
pull_request:
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reminder to remove before merge.

@mariecwhite mariecwhite requested a review from pzread September 4, 2023 05:25
@mariecwhite mariecwhite marked this pull request as ready for review September 4, 2023 05:25
@mariecwhite mariecwhite deleted the ggml branch September 17, 2023 22:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pzread pzread Awaiting requested review from pzread

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@mariecwhite