Server Response Header Change

[NateBrady23]

Currently, we have this rule in the general requirements:

All test types require Server and Date HTTP response headers. We expect the Server header to be whatever is normal for the platform or framework. If the framework does not normally provide a Server response header, we nevertheless require that you provide one as this roughly normalizes network load across all implementations. For Date, we expect that the rendered date be accurate. However, it does not need to be rendered from the system clock to a byte buffer for each request. Re-rendering once per second is an acceptable tactical optimization (and is an optimization baked into many frameworks).

We are thinking about making a change here in an effort to continue normalizing across all implementations for Round 20 and beyond. With the current requirement, we have some frameworks that leave their default Server header (maybe express) and some that are meeting the minimum requirements using 1 character.

We've come up with a few options to mitigate this, though we have not reached a consensus internally and would like to hear your thoughts.

1. Remove the requirement altogether.
2. Require some minimum length.
3. Require the header to be Server: TFB for all implementations
4. Remove the requirement for just the plaintext test and apply one of the above options to all other tests.

Let's chat!

[NateBrady23]

Shamelessly pinging some folks @benaadams @sebastienros @joanhey @jcheron @zloster @bgrainger @an-tao @lpereira @amichair

[benaadams]

Initially I thought (1) was a bad idea; as people will just remove the server header and some implementations may not be able to so be disadvantaged. However, in production the server header is often removed and in RFC 7231 it changed to a MAY rather than a MUST (which I think it was in the earlier HTTP specs?)

RFC 7231 HTTP/1.1 Semantics and Content

The "Server" header field contains information about the software
used by the origin server to handle the request, which is often used
by clients to help identify the scope of reported interoperability
problems, to work around or tailor requests to avoid particular
server limitations, and for analytics regarding server or operating
system use. An origin server MAY generate a Server field in its
responses.

(3) likewise may cause issues for people that can't change their server header; which would be more problematic than removing the requirement as they will just fail; rather than being disadvantaged.

It's mostly a major factor for plaintext as it hits bandwidth limits; and if the server isn't in the top bunch, might not be a factor as they aren't saturating the bandwidth?

So I'd err to (1) or (2)

[sebastienros]

These are band-aids, and I am sure we are all very sad that we have arrived at these solutions, but also maybe super glad that frameworks are now so fast that we have to talk about these kind of changes.

Looking at the original issue, we are measuring the network performance with Plaintext, just that. So I really see three options right now:

1. Fix the network
   It means make it faster so it's not the bottleneck anymore. However once you have 40GB/s network cards you will see that the client CPU will become the bottleneck and we'll hit the same issues again, at a higher rate though.

2. Fix Plaintext
   Mission accomplished, lots of frameworks are so fast with pipelining that not much work needs to be done there. Maybe it would be time to start deprecating it and adding some other flavor that would make it more challenging. Could be with simply adding https, or removing pipelining (which would show the actual cost of json serialization), or something else that would prevent the client from being a bottleneck.

3. Add more clients
   Assuming the network bandwidth will be fixed eventually (I heard there is very recent progress on this), having more clients would allow the server to be a bottleneck again, but it's not trivial to manage.

These options might not match with your schedule. My opinion about your options then would be to not have any requirement. If the framework can remove the header, or optimize it, then it's all good for the performance.

[NateBrady23]

@sebastienros I agree with everything you said, except that I'm trying to focus more on normalization across all implementations in this particular issue rather than the issue of the network being the bottleneck for the top tier implementations. I think that's a separate issue we'd also like to tackle.

[lpereira]

I particularly never cared too much about the plaintext benchmark because it doesn't really reflect reality that well (very few clients support pipelining in HTTP/1.1, so it never gets used in practice); the fact that many of the top frameworks "cheat" in that test by not using the idiomatic mechanisms available in the framework doesn't really make it look good, either. (I know it helped find a lot of the overhead in e.g. HTTP parsers, but it's beyond the point, IMO.) I wouldn't mind if this benchmark went away -- or at the very least, its pipelined requirement -- for round 20+, and something that reflected more a real workload (something along the lines I suggested in #5506) were put in its place.

Having said that, I agree with @benaadams's points. I also think that either (1) or (2) would be fine for me.

(FWIW, as it is today, you can't override/remove the Server header from Lwan without modifying it. Easy to fix if this becomes a requirement, though, but I'd rather not change it just for the sake of the benchmark.)

As @sebastienros mentioned, the clients are often the bottleneck with these tests, especially when constrained by network I/O. Writing a new one (or modifying e.g. this weighttp fork) to take advantage of things like io_uring, that validated only every few N every responses (so no need to fully parse every response, but important to catch cheaters), etc. would be an interesting exercise, too. (If the plaintext benchmark ends up staying with the pipeline requirement, not having to depend on a Lua script for pipelining tests would be an interesting plus, as it takes a good amount of CPU time that could be spent shoving coal into the network card.)

[NateBrady23]

@lpereira

I particularly never cared too much about the plaintext benchmark because it doesn't really reflect reality that well

I think we might be doing a poor job communicating that this benchmark isn't meant to reflect reality. We elude to this in the test requirements for it, but we could do a better job.

as it takes a good amount of CPU time that could be spent shoving coal into the network card

We're discussing more options for this, like using the db server as another wrk client during plaintext and json tests

Having said that, I agree with @benaadams's points. I also think that either (1) or (2) would be fine for me.

Thanks for your input! Seems we definitely have some options in the lead.

[lpereira]

I think we might be doing a poor job communicating that this benchmark isn't meant to reflect reality

Oh, no, it's very clear the plaintext benchmark is a "go nuts" thing. I'm just grumpy because my toy isn't in the top 10 with the cool kids. :)

[lpereira]

And, BTW, I like the idea of using the db server as an additional client for the benchmarks that do not require database access.

[benaadams]

I think we might be doing a poor job communicating that this benchmark isn't meant to reflect reality. We elude to this in the test requirements for it, but we could do a better job.

I have been impressed that; whether by design or accident, each benchmark, including plaintext, has managed to push on a different weak point for improvement for the frameworks. I assume by design 😉

[an-tao]

I agree with everyone above, I'd like to choose option 1 or 2.
And considering headers of responses, I think we should pay a little attention to the content-length header, I see some frameworks using hard coded content length, this is too far from reality. Should we consider prohibiting this practice?
About the bottleneck problem, can it be solved by simply improving the performance (such as the number of CPU cores) of the client and db machines and improving the network bandwidth?
I mean, let the tested frameworks run on a relatively slow machine.

[helyho]

I think no matter which option, it has an impact on all tests. I think it should be applied to all tests, at least when / JSON and / plaintext.

by the way I'd like to choose option 1 or 2.

[msmith-techempower]

I particularly never cared too much about the plaintext benchmark because it doesn't really reflect reality that well

This argument gets thrown around a lot, and over the years I have never really been able to communicate why I think plaintext is an important test. @bhauer has written about it on the blog posts that coincide with the round releases in the past, but it often gets lost.

plaintext is a silly test. It's meant to be. It illustrates a high-water mark for the platform/framework. It basically shows that if you are doing no real work in your business logic and just doing the simplest stuff (routing and returning a plain text string) then you can pump out X requests per second. A higher plaintext result, without gamifying the test and I really don't want to get into that here, means that your stack has more "room" for business logic.

It also serves as a valuable piece of data when looking at the whole - when you look at progressively more and more business logic (json, db, queries, updates, fortunes) one gains a sense of the cost of such operations. Something at the core of the project and one of the founding questions we had when we started this project is "what does one line of code cost in performance?"

I think @sebastienros makes a good point about the value of the plaintext test perhaps being lost in just how performant the top 5-10% are - 7M RPS approaches what my PC can do locally with no network. There may be value in having a more real-world plaintext test that does not include pipelining, and then the "silly" plaintext with pipelining we have today for giggles.

I am personally in favor of options 1 and 3, and find that 2 is just setting a standard to which the players who are gamifying the plaintext test shall adhere, but maybe that's the point.

[bhauer]

Of the options presented, option 1 is my preference. For the Plaintext test, and this test only, removing this header seems reasonable. It eliminates some potential bandwidth variability where available bandwidth has proven to be a significant driver of results, at least among the highest performance tier.

[talawahtech]

I am in favor of removing the Server header as a requirement for the plaintext test as well. That is the simplest way to ensure the uniformity of the response across frameworks.

I would also like to make another suggestion. Since there is still resistance to fully removing the pipelining aspect of the plaintext test, what do you guys think about reducing the number of pipelined requests sent by wrk? That will allow us to actually get a true view of what top 10 performers look like relative to each other and the rest of the field.

It is currently set to 16 but it could be reduced to 12, 10, 8 or whatever number works such that only the absolute fastest test can approach the bandwidth limits. If tests get even faster, we can drop the number even further, if the available bandwidth increases, we can increase it again.

[joanhey]

The length of the headers and url, impact in all tests. Especially in the plaintext.

I think that the minimum length server header is the correct (option 2), if we want a realistic approach.
Almost no header is a MUST, they all come with an IF.

Why ?

• In some servers - frameworks is not easy to disable. Ex Nginx server.
• Every day we send more headers in our apps (CORS, auth, cookies, cache, ...)
  so the server header will be a good minimum for benchmark an app.
• A lot of the frameworks will be behind a proxy (Nginx, Varnish, HaProxy, ATS, ...) that send the server header
• In a server load balancing, it will be necessary in some situations.
• ...

About the HTTP pipelining the problem is that the results are mixed with and without pipeline. So you can't compare one with another.
It's like the db benchmark, we want to compare with the same database.
We can't filter by pipeline enabled or not, to compare frameworks. And perhaps a fw is really fast with pipeline and not without (rare). But we can't compare both without pipelining.
When we have an http2 test we can compare with a more realistic pipeline.

The worst is the people that arrive to see the bench, and they don't know they are mixed. They only see a big difference.
And perhaps they don't know that all the browser and majority of proxies will not take advantage of it.
Below the plaintext benchmark in the Requirements summary only say: HTTP pipelining is enabled. Perhaps will be good to add an small note about the pipelining.

[benaadams]

In some servers - frameworks is not easy to disable. Ex Nginx server.

NGNIX violates the http spec; it should add itself to Via not change the Server header if present.

RFC 7230 in 2.3

A "gateway" (a.k.a. "reverse proxy") is an intermediary that acts as an origin server for the outbound connection but translates received requests and forwards them inbound to another server or servers. Gateways are often used to encapsulate legacy or untrusted information services, to improve server performance through "accelerator" caching, and to enable partitioning or load balancing of HTTP services across multiple machines.

RFC 7230 in 5.7.1

An HTTP-to-HTTP gateway MUST send an appropriate Via header field in each inbound request message and MAY send a Via header field in forwarded response messages.

It allows for overriding fields as an option but not by default: RFC 7230 in 5.7.2

An HTTP-to-HTTP proxy is called a "transforming proxy" if it is designed or configured to modify messages in a semantically meaningful way (i.e., modifications, beyond those required by normal HTTP processing, that change the message in a way that would be significant to the original sender or potentially significant to downstream recipients). For example, a transforming proxy might be acting as a shared annotation server (modifying responses to include references to a local annotation database), a malware filter, a format transcoder, or a privacy filter. Such transformations are presumed to be desired by whichever client (or client organization) selected the proxy.

And NGNIX isn't normally used as a transforming proxy.

Other reverse proxies:

• Varnish cache: Adds Via:1.1 varnish
• Apache Traffic Server: Adds Via: http/1.1 ...
• IIS Application Request Routing: Doesn't add anything
• Squid cache: Doesn't add anything
• AWS Cloudfront: Adds Via:1.1 ...cloudfront.net
• Google HTTP/HTTPS Load Balancing: Adds: Via: 1.1 google
• HAProxy: Doesn't add anything by default (instructions on how to add X-Via)

So it's quite an anomaly in doing this.

This is aside from RFC 7231 in 7.4.2 saying you no longer have to bother with the server header

The "Server" header field contains information about the software used by the origin server to handle the request, which is often used by clients to help identify the scope of reported interoperability problems, to work around or tailor requests to avoid particular server limitations, and for analytics regarding server or operating system use. An origin server MAY generate a Server field in its responses.

[sumeetchhetri]

I would be in favour of option 3, which would enforce a guideline for all frameworks to follow, or else just enforce a minimum length via option 2.

[Fenny]

I think we all want a fair playground and normalize the tests between different frameworks.
The logical choice would be to set a default server header / minimum length for all tests.

[talawahtech]

Is a minimum going to be enforced for round 20? Or are we all going to end with a bunch of single letter server names?

[NateBrady23]

@talawahtech a min length requirement for the routes exists in the toolset now. A test should not pass verification unless they meet that minimum.

[talawahtech]

@nbrady-techempower right, the requirement is clear for routes, but it is unclear for the "Server" HTTP header, will a minimum be enforced for that as well?

[NateBrady23]

@talawahtech Those changes have not been made yet. There's still time before Round 20 closes, which frameworks do you think are getting a competitive edge from single letter server names? If you can open up an issue with listing them, I'll take a look when I can.

[talawahtech]

@nbrady-techempower well as an example, the Server header for the dotnet platform implementation is currently set to "K"[1] (and that is technically allowed). I assume the full header would otherwise be "Kestrel" which is 7 letters, and removing those 6 letters probably allows an extra 1-2% worth of requests to get though with the current bandwidth limit.

But my question is, if you are looking at it on a cases by case basis, what is the unofficial requirement going to be? Can/Should I reduce the Server name for the libreactor implementation to 7 letters as well?

1. FrameworkBenchmarks/frameworks/CSharp/aspnetcore/PlatformBenchmarks/BenchmarkApplication.cs

   Line 23 in 875969e

   private readonly static AsciiString _headerServer = "Server: K";

[sebastienros]

Exactly. We raised the issue for the routes, and it has been defined since then. For Plaintext every char matters, so we optimized it for the Platform scenarios, not the other more complex ones (MVC and Middleware).

[NateBrady23]

@talawahtech This RFC is still open, so you're free to do that. It's not being enforced yet; the routes are.

[sebastienros]

My other favorite, Actix, does it too with a single A: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Rust/actix/src/main.rs#L30

[talawahtech]

Ok, I guess it will be "L" for libreactor until the requirement changes.

[NateBrady23]

After round 20, I think we're settling on a min 3 with a recommendation of Server: TFB. Seems like most here and internally are fine with that.

[benaadams]

I think pico.v was first to use single letter server header V which moved them significantly enough through the 7M bandwidth barrier on plaintext; it's only 10k-20k rps; but psychologically significant since it was definitely above 7M and it introduced a gap; which then other servers followed.

Am fine with Server: TFB; though it's mostly a factor on plaintext and where the bandwidth limit is being hit (> 6.8M rps?)