Clearly incorrect test results

[phphleb]

Hello!
In the metrics group I see that the test results look very strange and the results are incorrect, for example:
Here is an empty section.
https://web-frameworks-benchmark.netlify.app/compare?f=slim,lumen,yii,laminas,codeigniter4,spiral,laravel,symfony#maximumLatency
Here is incorrect
https://web-frameworks-benchmark.netlify.app/compare?f=slim,lumen,yii,laminas,codeigniter4,spiral,laravel,symfony#minimumLatency
Version 2025-09-03 (3c20ceb)

[phphleb]

I wrote about this earlier here

[waghanza]

Hi @phphleb,

The previous version was using keep alive and all frameworks were not competing in the the same area.

Could you please explain what is incorrect for you ? and what (and why ?) do you expect some results ?

Regards,

[waghanza]

PS : The link above was using wrk and not oha (previous rounds)

[phphleb]

The fact that the mode was switched, I could have missed the message. However, it is a pity that the comparison is now less visually informative, even though it may be technically correct.

Intuitively, I feel that the values of the frameworks are not distributed in a logical manner. For example, yii-swoole and yii should not display such close values, because in reality the difference is much greater. I mean real web applications. And before that the difference was more or less natural.

Perhaps I bothered you in vain, but doubts still remain.

[waghanza]

Perhaps I bothered you in vain, but doubts still remain.

Not at all.

However, to make sure that I understand what you say : yii numbers should vary from phpleb.

Do you have any ratio / clue. For example, in javascript, node / deno / bun are battle tested, that mean that we know the ratio of optimization we could have between those, and then it help veryify results.

Let's check results that will be published later today

[phphleb]

What was meant in this case is that from the outside it may seem that there are serious problems for the yii team or for the swoole developers, but with the previous settings the spread was more like what it should be.

[phphleb]

It's also possible that project owners need to tweak their configurations to meet the new requirements, but the tests will also show who responded to the notification about the mode change and who didn't. For example, my notification seems to have gotten lost.

For instance, webman (based on workerman) didn’t lose its position, while many other frameworks with the suffix -workerman and even workerman itself dropped to unrealistically low results, so it’s more likely that the old configuration doesn't fit the new conditions.

[jknack]

@waghanza what has changed? Previous round jooby was on Java top 5 and now it is one of the bottom.

[waghanza]

@jknack keep alive was disabled + arch is not the same (x64 instead of M1)

---

EDIT

3 things changed :

• oha used instead of wrk
• Keep alive is now disabled
• Architecture is more reliable (x64 used instead of m1)

PS : The long term goal for 3rd point is to use real life provider to consider this project as stable

• VM (like AWS ec2)
• PODS (like Kubernetes)
• Bare metal (like a standalone server)

[cyrusmsk]

It's also possible that project owners need to tweak their configurations to meet the new requirements, but the tests will also show who responded to the notification about the mode change and who didn't. For example, my notification seems to have gotten lost.

For instance, webman (based on workerman) didn’t lose its position, while many other frameworks with the suffix -workerman and even workerman itself dropped to unrealistically low results, so it’s more likely that the old configuration doesn't fit the new conditions.

What do you mean by “configured properly”?

The server may receiving different types of requests.. with “keep-alive” or with “connection: close” requirements. They should be able to handle all kind of requests.

The idea of this benchmark is to use normal configurations that developers will use in real-world scenario cases, not tweaked and hacked configurations just to show higher results in meaningless benchmarks - for that type of results is better to go to TechEmpower.

Even this benchmark has some solutions, that are very far from production ready. But the repo owner at least is open to improvement of the quality of tests and maybe in future when more sophisticated checks were enabled - some frameworks will be just removed as non-compliant real-world solution.

[waghanza]

Thanks for your return @cyrusmsk . Indeed, their is clearly some space to improvement.

The main purpose of this project is to compare framework in real-world scenarios (more decision making purpose, than tech improvment). The idea is then not to push further any frameworks limits, like TechEmpower do.

What I mean behind that ?

I known that framework could be tweaked ... scaling could be done with an architectural design, but we want to skip all of this so you can factually compare frameworks with a more "fair" comparison.

[jknack]

@waghanza there should be something wrong... I'm sure jooby supports keep-alive header. Can you point me to your test command? I'll try locally

[waghanza]

I mean the test command mark explictly that we do not want keep alive

oha --wait-ongoing-requests-after-deadline --no-tui --disable-keepalive --latency-correction -c 512 -z 15s  http://127.0.0.1:3000

[phphleb]

What do you mean by “configured properly”?

It turned out very simple there, as in reality, the project owners were given a certain server and a certain type of incoming requests and they set up the configuration for this. If this is not the case, then we will never be able to prove that the increasingly popular web server is slower than Laravel.

[phphleb]

The server may receiving different types of requests

If you submit more diverse types of requests, then perhaps the settings will be averaged, like in a real working web application. For example, with alternation.

PS I mean that in projects the settings will be changed to more realistic ones in this case.