How do LLMs perform compared to human?

[findmyway]

This is how I implemented the benchmark solutions:

1. First extract solutions from the Python version
2. Rewrite the Python code into the Julia version.
3. Rewrite test cases based on my understanding if possible.

Once finished, I evaluated my first attempts all at once. And the pass rate in average is slight above 0.5. So I would say, many LLMs already did it better then me.

You might be surprised that my pass rate is way too low. Honestly speaking, I was also quite surprised, especially given that I've been programming in Julia for years and I had peeked the Python version solution. I found that the failed ones were mainly due to corner cases, incorrect grammar and misunderstanding of problems. Actually the Python version solutions sometimes were quite misleading.