Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: high-level implementations for Positional Embeddings #1237

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

avik-pal
Copy link
Member

@avik-pal avik-pal commented Feb 11, 2025

  • Move Embedding into a different File
  • Rotary Position Embedding (RoPE)
    • Implementation
    • Tests
    • Documentation
  • Sinusoidal Positional Embedding
    • Implementation
    • Tests
    • Documentation

Copy link
Contributor

github-actions bot commented Feb 11, 2025

Benchmark Results (ASV)

main 9c7c15c... main/9c7c15caaf91e0...
basics/overhead 0.129 ± 0.0015 μs 0.14 ± 0.0024 μs 0.925
time_to_load 0.914 ± 0.015 s 0.913 ± 0.0086 s 1

Benchmark Plots

A plot of the benchmark results have been uploaded as an artifact to the workflow run for this PR.
Go to "Actions"->"Benchmark a pull request"->[the most recent run]->"Artifacts" (at the bottom).

@avik-pal avik-pal force-pushed the ap/embedding branch 2 times, most recently from 4cf03a1 to 9dd4d1f Compare February 12, 2025 13:52
Copy link

codecov bot commented Feb 12, 2025

Codecov Report

Attention: Patch coverage is 44.92754% with 38 lines in your changes missing coverage. Please review.

Project coverage is 74.37%. Comparing base (73816b6) to head (9c7c15c).

Files with missing lines Patch % Lines
src/layers/embedding.jl 44.92% 38 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (73816b6) and HEAD (9c7c15c). Click for more details.

HEAD has 27 uploads less than BASE
Flag BASE (73816b6) HEAD (9c7c15c)
51 24
Additional details and impacted files
@@             Coverage Diff             @@
##             main    #1237       +/-   ##
===========================================
- Coverage   85.30%   74.37%   -10.94%     
===========================================
  Files         153      152        -1     
  Lines        6317     6330       +13     
===========================================
- Hits         5389     4708      -681     
- Misses        928     1622      +694     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant