Roadmap 🚀

[waghanza]

Hi,

This project has a learning / promotion purpose in the first place.

The idea is to have :

• a list of awesome frameworks
• a list of production indicators (performance at first glance, but others will come)
• a as fair as possible comparison between them

Next steps are :

• Add concept on engines (in addition to actual one) => Multiple engines #3958
  • falcon, iodine in addition to puma for ruby
  • hypercorn, bjoern, uwsgi in addition to gunicorn in python
  • swoole, road runner, workerman in addition to php-fpm in php
  • uwebsockets, deno in addition to node in javascript
• Having a more real-life workload representation => Most of real-life frameworks use I/O layers. Shouldn't they be here as well? #3803
• Run on cloud
• Having a clean UI to expose results => https://github.com/the-benchmarker/website
• Having a clean backend to push the results => Probably with https://hasura.io/cloud rather than an in-house solution

Regards

---

(Open to discussion)

Regards,

[septatrix]

I would also suggest pypy for Python

[waghanza]

Thanks for your suggestion. As a global reflexion, I'm not sure it is very useful to check those results for all VMs since it increase a lot the complexity (maintenance) for this project.

For example, we have 28 frameworks in python, and checking for :

• python
• ironpython
• pypi
• graalpython

would add 28*3 (+ in case of multiple engines [hypercorn, uvicorn, gunicorn ...])

I think for our sake that we need to stick on a restricted list (what is mainly used)

wdyt @ahopkins @the-benchmarker/web-frameworks ?

[ahopkins]

I would say test against the latest version of python that all frameworks support. The bigger question is not how fast a framework can perform. Benchmarks for those numbers are highly suspect and cannot really tell you that since there are too many factors. What benchmarks can give you are an idea of relative performance. For that, we need one baseline version, which should (IMO) be the latest version which all tested frameworks support.

[waghanza]

With your comment @ahopkins, and as long as I work on #3958, I think that using variants is over-engineering.

I mean, in my mind variants could be :

• an engine, for example puma
• a vm, for example truffleruby
• a dialect, for example jruby

We have better to introduce only engines (the more relevant example is node in javascript), and introduce vm later

[SuspiciousLookingOwl]

Having a clean UI to expose results (/cc @doanguyen I'll ask for you help when the time will come)

Sorry if this sounds like a self promotion, but I made a web app (using React) to show the data in Chart (filterable by frameworks) and Table (filterable by language and sortable).

Demo: https://web-frameworks-benchmark.netlify.app/
Repo: https://github.com/SuspiciousLookingOwl/web-frameworks-benchmark

[waghanza]

should be promoted in the README @SuspiciousLookingOwl

what do you think @the-benchmarker/web-frameworks ?

[dalisoft]

@waghanza I sure it should be in README, looks nice and works good

[waghanza]

@SuspiciousLookingOwl your idea seems awesome. I see however that you are parsing README.md file directly from github. The README was not made for that, then I can break your parsing.

If you prefer, I can export raw data as json, and push them (on next result update)

[SuspiciousLookingOwl]

I agree that parsing the md file isn't an ideal solution. My target was having the web app to always shows the latest data from this repo without too much hassle on my end.

If you prefer, I can export raw data as json, and push them (on next result update)

That sounds awesome! It will be great if you can do that 😁

[waghanza]

I've created a discussion @SuspiciousLookingOwl
#4165

[SuspiciousLookingOwl]

IMO #3803 should also be in the roadmap.

[waghanza]

Agree @SuspiciousLookingOwl, I've updated the roadmap

• I change flavors to engines. Initially I wanted to run all frameworks on various things (like jruby, deno, rhino) but seems too complicated.

[bung87]

hey, I have some thoughts after adding my project to this project and https://github.com/TechEmpower/FrameworkBenchmarks

1. Compare to TechEmpower/FrameworkBenchmarks this project does not constrain the http headers.
2. Newer added framework may take high priority and within its language and platform etc.
3. Compiled language and non-compiled language may results in different categories , generally compiled language will be n x faster than non-compiled language

[waghanza]

hey, I have some thoughts after adding my project to this project and https://github.com/TechEmpower/FrameworkBenchmarks

thank you for that ❤️

1. Compare to TechEmpower/FrameworkBenchmarks this project does not constrain the http headers.

not yet, it is planned, but some work have to be done before

2. Newer added framework may take high priority and within its language and platform etc.

could you specify you though ? I don't get you here

3. Compiled language and non-compiled language may results in different categories , generally compiled language will be n x faster than non-compiled language

the idea is NOT to show only performances, but all valuable insights (resources consumptions, performances, latencies ...) about frameworks

[bung87]

about 2. eg. my project scorper just added may run it first then run other frameworks written in Nim then run rest frameworks. if any error occurs the author does not need wait for long.

[cyrusmsk]

Any news on the Roadmap?

[waghanza]

I plan to rewrite readme and to integrate it in a more clear way in README.md

[cyrusmsk]

I plan to rewrite readme and to integrate it in a more clear way in README.md

Maybe create a task with sub-tasks and somebody also will make a PR for tests improvements?