Skip to content

Data Processing benchmark featuring Rust, Go, Swift, Zig etc.

License

Notifications You must be signed in to change notification settings

DoLooP/related_post_gen

 
 

Repository files navigation

Problem:

Given a list of posts, compute the top 5 related posts for each post based on the number of shared tags.

Steps:

  • Read the posts JSON file.
  • Iterate over the posts and populate a map containing: tag -> List<int>, with the int representing the post index of each post with that tag.
  • Iterate over the posts and for each post:
    • Create a map: PostIndex -> int to track the number of shared tags
    • For each tag, Iterate over the posts that have that tag
    • For each post, increment the shared tag count in the map.
  • Sort the related posts by the number of shared tags.
  • Write the top 5 related posts for each post to a new JSON file.

Run Benchmark

./run.sh go | rust | python | all
Rules

No:

  • FFI (including assembly inlining)
  • Unsafe code blocks
  • Custom benchmarking
  • Disabling runtime checks (bounds etc)

Must:

  • Parse json at runtime
  • Not hardcore number of posts
  • Support up to 100 tags
  • Use a stable release of the compiler/runtime

Updated Results from github workflow (raw data)

Language Processing Time Total (PT + I/O)
Go Concurrent 16.69 ms 52.3 ms
Rust Concurrent 19.28 ms 39.3 ms
Go 32.28 ms 68.3 ms
Rust 34.12 ms 55.8 ms
Java (GraalVM) 35.15 ms 65.5 ms
Swift Concurrent 41.23 ms 445.0 ms
Fsharp Concurrent 42.57 ms 906.2 ms
Fsharp 49.00 ms 900.4 ms
Crystal 56.56 ms 116.5 ms
Vlang 63.38 ms 413.3 ms
Swift 66.49 ms 456.4 ms
Odin 70.88 ms 321.7 ms
Nim 91.67 ms 122.5 ms
Dart VM 108.13 ms 623.4 ms
LuaJIT 123.92 ms 435.7 ms
Dart AOT 142.00 ms 292.1 ms
Zig 178.00 ms 221.2 ms
JS (Deno) 184.80 ms 290.7 ms
JS (Node) 208.20 ms 292.5 ms
Java (JIT) 278.54 ms 567.3 ms
Numpy 0.41 s 669.3 ms
Julia v2 666.17 ms 5.523 s
JS (Bun) 743.80 ms 831.4 ms
Lua 2375.23 ms 3.095 s
Python 2.71 s 2.789 s
Old Results with details (on my machine)
Language Processing Time Total (+ I/O) Details
Rust - 4.5s Initial
Rust v2 - 2.60s Replace std HashMap with fxHashMap by phazer99
Rust v3 - 1.28s Preallocate and reuse map and unstable sort by vdrmn and Darksonn
Rust v4 - 0.13s Use Post index as key instead of Pointer and Binary Heap by RB5009
Rust v5 38ms 52ms Rm hashing from loop and use vec[count] instead of map[index]count by RB5009
Rust v6 23ms 36ms Optimized Binary Heap Ops by scottlamb
Rust Rayon 9ms 22ms Parallelize by masmullin2000
Rust Rayon 8ms 22ms Remove comparison out of hot loop
Go - 1.5s Initial
Go v2 - 80ms Add rust optimizations
Go v3 56ms 70ms Use goccy/go-json
Go v3 34ms 55ms Use generic binaryheap by DrBlury
Go v4 26ms 50ms Replace binary heap with custom priority queue
Go v5 20ms 43ms Remove comparison out of hot loop
Go Con 10ms 33ms Go concurrency by tirprox and DrBlury
Go Con v2 5ms 29ms Use arena, use waitgroup, rm binheap by DrBlury
Python - 7.81s Initial
Python v2 1.35s 1.53s Add rust optimizations by dave-andersen
Numpy 0.57s 0.85s Numpy implementation by Copper280z
Crystal 50ms 96ms Inital w/ previous optimizations
Crystal v2 33ms 72ms Replace binary heap with custom priority queue
Odin 110ms 397ms Ported from golang code
Odin v2 104ms 404ms Remove comparison out of hot loop
Dart VM 125ms 530ms Ported frog golang code
Dart bin 274ms 360ms Compiled executable
Vlang 339ms 560ms Ported from golang code
Zig 80ms 110ms Provided by akhildevelops

About

Data Processing benchmark featuring Rust, Go, Swift, Zig etc.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Shell 16.6%
  • Lua 13.7%
  • Python 10.1%
  • Rust 7.7%
  • Dart 6.8%
  • Swift 6.8%
  • Other 38.3%