Why We're Moving From IIS + Nginx to Hyper

There’s usually a disparity between the best reason and the “real” reason.

We’re rewriting Azure IMDS from scratch as a part of investments into Azure Boost. As a part of that rewrite, we’re migrating away from IIS Hostable Web Core to the popular Rust crate Hyper. Other services that are currently Ngnix based are also migrating to Hyper.

Background on Azure Boost

Azure Boost is a new product offering that offloads work traditionally done by the host onto a purpose built add-in card. If you’re unfamiliar with the concept, it’s kind of like a graphics card. Or if you’re a bit older, a sound card, network card, etc. All of these are “add-in cards” meaning they have a standardized connector that can be connected to the computer.

The difference here is that instead of adding a single purpose card, we’re actually slotting in an entire second computer:

To avoid accidentally leaking anything, I’ll stick with what you can see in the image + what’s disclosed in the Azure blog post. At the top, there are two high speed network interfaces. Below we have two heat sinks. While heat sinks cover multiple components, it’s obvious based on the context that one is for the FPGA that underlies Azure MANA and the other is for the System on Chip (SoC) running Linux that is performing the offloaded work.

The “real” reason

If you read through the marketing material for Azure Boost, you’ll see two core themes: Security and Performance. The project offers a chance to rewrite your product, which is a rare opportunity in an older mega-corporation. Offloading to the add-in card doesn’t really communicate how dramatic this shift is. As a service owner, I’m migrating from a massive, server class CPU running Windows to a tiny little SoC running Linux. Even if we could recompile all our software and all our dependencies for this new target it’s still not going to work. We need to re-architect the services, not lift and shift them.

Microsoft has also embraced more open source over the years and distanced itself some from “Not Invented Here” thinking. There are people dragging their feet in protest of course. It’s a huge company, not a monolith. However the momentum is clear and the people making policy decisions have made this transition. This all means we get to ask what the best tool for the job is.

Our web server choice needs to meet the following requirements:

• Secure
• Lightweight
• Active community + development
• Easy to customize
• Easy to patch (the blast radiuses of these services are massive. In the worse case scenario, it needs to be practical to fork and fix the project ourselves if we aren’t getting a patch fast enough)

We can and will consider each point in detail, but first lets step back and acknowledge the reality of the situation. IIS is out of consideration because we’re using Linux. What about Ngnix? Overall, it meets many of these requirements. Where it doesn’t a nuanced debate can be had. But here’s the thing, the direction from the CTO of Azure has been quite explicit, even controversial:

I’ve actually worked directly with Mark on another project that will be publicly announced in early 2025. While I won’t go as far as putting words in his mouth, I will say that based on my time collaborating with him I don’t believe he’s hamming it up for the online audience here.

While there are good arguments for Hyper and good argument’s for Mark’s view here (I fully support his conclusion), this is the official policy for Azure Boost development. The truest answer for why we’re using Hyper? Because we’re doing non garbage collected systems programming. That means we need to use Rust and Hyper is the best Rust web server for our needs.

So when it comes to Ngnix, bringing over ~160K lines of C with a history of memory vulnerabilities into our brand new project that’s otherwise written in Rust is simply anathema to what we’re doing.

Second order “decisions”

This is a reality that I don’t think gets enough attention and the main reason I wrote this post. I will discuss why Hyper is the best choice later on, but you’re not going to see anything novel in there. We all are intuitively aware that a person’s stated reasons for holding a position are not necessarily reflective of their actual motivations.

To give a trite example, consider politicians. As an elected official, your job isn’t to have a clear, consistent policy. Your goal is to secure your next election. People are fickle, especially in groups. Opinion can shift quickly. Even with reasoned change over time, politicians are faced with the same problem. The best message is to be extreme, confident, and convicted about the zeitgeist of the moment. When the winds change, so too will their message and they’ll make up bogus reasons to rationalize their movement because the truth (that they’re patronizing you) turns people off.

In my experience people are quite attuned to this phenomena on interpersonal topics, but less so in the work place or other professional contexts. Some real examples I’ve encountered in my career:

It’s not our bug

Another team’s product has a major bug in it. The bug was poorly managed and the risk was known for years. Worse yet, there’s a lot of debt that makes it way too risky to try to promptly fix it the right way. Instead of leveling with us, owning the issue, and asking for help to mitigate customer pain, that team pitches an obviously bad workaround as the “objectively correct design”. After much contentious back and forth, they partially concede this point.

It’s pretty obvious what happened here. In a panic, the other team is trying to save face and deflect from their mistakes. That’s not something people want to admit, so instead a series of ad-hoc arguments and confabulations are made to attempt to find an alternative, acceptable justification for their goal.

Toe the line

Think Jeff Bezo’s “disagree, but commit”. Once a battle is lost or a decision overruled, that person will have to take that decision elsewhere and invariably will be forced to defend it. These conversations don’t go well because the person often isn’t going to be able to make the best case or give full enthusiasm, as they’re advancing beliefs they don’t hold. On the other side, it’s very uncomfortable to voice this predicament.

My advice: Be an adult, be honest. If that’s going to backfire on you, well then it isn’t an environment you should be staying in anyway. If you’re pushing something just say that you preferred a different approach, but the group (or person) decided X for Y reasons. When I raised Z concern, they were not worried because of Q. As long as your tone is fine, it will go over fine. Everyone will be on a better footing and you’ll avoid the song and dance where they shadow box against decisions you don’t even agree with.

Higher stakes politics

As you move up an organization, the fact of the matter matters less and less. This is partly just a reality of systems and human nature, but it’s also important to remember they’re not playing the same game you are. Yeah, executives have ego, pride, and face saving of their own to do. They also have a wildly different job. If you’re pitching a new feature, your job is to pitch the best design for the feature. They’re looking to balance that against time to market, the next quarterly earnings report, competing priorities for finite funding, etc.

Even things that might seem more negative at first are still just neutral realities. You want to do X project that requires some small investment from another division in the company. Why are they resisting? It’s so obviously good for us to do! Well, from their perspective that’s an unfunded mandate. They already have a full slate of commitments. Opportunity cost is omnipresent. To do your ask, no matter how small, something else needs to be punted.

This is all to say that when you’re hashing out the details at a high enough level in a company, you’ll quickly come to realize that the engineering facts of the specific topic are but a small portion of what leads to the decisions that are made.

The case for Hyper

For completeness then, what’s the argument for Hyper if you’re not subject to our specific policies and prior decisions? I was skeptical of Hyper because I’d never heard of it and it was pretty new. At the time, it didn’t even have a stable release yet.

Luddite concerns

My concern was primarily rooted in an outdated form of thinking. When I first learned backend web development, I learned with scripting languages like PHP. It was also just a different world back then with fewer options. A lot of what dominates the web today didn’t exist until 2009-2012 (Golang, Typescript, NodeJS) or did exist but wasn’t as popular yet / had a weaker ecosystem. When I was in high school self teaching web development, I wasn’t going to gravitate to the shiny, new, more complex offerings that were emerging.

In general, infrastructure and parsing are some of the most obvious places you shouldn’t be rolling your own code. Speed and security matter a lot here. It’s super easy to mess up, and you’re unlikely to do better than people who spend their careers focused on that one topic. In my mind, you didn’t use some random web server for the same reason you didn’t roll your own database.

That said, it wasn’t like language specific web servers are a new thing. See Java and C#. I just wasn’t particularly exposed to them, and they had downsides related to coupling as well.

The more legitimate concern then was over some shiny new unknown entity. This too stemmed from just being unaware of the project.

Hyper’s strengths

Hyper has really grown on me. Here’s why:

It’s written in Rust

I know that’s become tiring to hear, but it matters a lot here. Web servers are exposed to the public internet by design. The product category itself is tautologically an attack surface. It implements complex, optimized binary protocols. It often runs on the machine with high privilege. To reject the relevancy of memory safe languages here is to reject the concept entirely.

Fortunately, we don’t need to debate theory. All 9 of Ngnix’s vulnerabilities from 2020 through 2024 were memory safety related, meaning they wouldn’t happen in safe Rust.

As I’ve discussed before, security benefits alone are missing half the point. Rust isn’t just secure, it’s a remarkably well tooled language that also promotes correctness. You’re going to be far more efficient than if working with C/C++, and while it’s a little slower up front to write than say python what you deploy to production will generally work first try. We all know it’s far cheaper to fix bugs before check-in than after release.

Hyper is not a hobby project

Yes Hyper is new, but it’s well established and recently made its first stable release. It’s largely written by one person, but so what? Linux and git were too. What matters is that a community has formed around it. Hyper has many individual and corporate contributors today.

It’s widely used and depended on with ~150M downloads at time of writing and is used for mission critical, high-scale services in: Microsoft, AWS, Cloudflare, Discord, etc. Even curl is experimenting with switching its backend to Hyper for improved security.

Performance & Correctness

Hyper is blazingly fast. It’s so fast that when comparing it to Ngnix, much of any disparity is going to come down to how your test is set up. But you can comfortably say it’s equal or even faster in performance. Ngnix’s worker processor model was a game changing improvement over thread based models in IIS and Apache. Particularly for proxy and static file workloads, the uplift was immense.

Today though, seriously performant web servers use asynchronous architectures. Cloudflare’s post on pingora isn’t talking about Hyper the whole time but the architectural performance comparison applies to both competing web servers. The same post comments on Hyper’s correctness, saying it was so high that it ended up causing issues where they wanted to support legacy, non-compliant web servers behind their reverse proxy. That’s a funny, esoteric problem to hit for them and great news for us that want to deploy it in more typical workloads.

Tokio based

The entire tokio stack is highly layered, composable, and just generally well designed. The runtime periodically gets large performance improvements and the whole stack is truly a la carte. Look no further than the tracing crate which is technically a part of the tokio ecosystem but has become so prolific that it’s the defacto, quasi-standard library implementation for application and library tracing.

Some people will say Hyper is a library, not a web server. They also release full server implementations, so I think that’s getting a little lost in the sauce. In fact it demonstrates a strength not a weakness. You can pick and choose what you want. Don’t like the ergonomics of the axum web framework? You can pick another without abandoning the rock solid underlying HTTP implementation. Want to reuse middleware? So long as you alternative web framework is tower compatible, you’re off to the races.

Best of all, in Rust there’s no performance penalty for these abstractions. The static dispatched generics can be tedious to learn, but they allow the compiler to inline to its heart’s content.

Ngnix is past its golden age

In every transitionary period, a gray area exists. Yeah, Rust is more secure all else equal, but battle tested code is more secure too. Sure Hyper is used at scale, but not even close to the scale Ngnix is. This is all true, but ask yourself where are the two projects trending?

Hyper is stabilizing, picking up momentum, and is already better in many ways. It’s got an active community and is easy to work with or change (both in the main tree or by taking advantage of its layered design). Ngnix is slowing down, increasing difficult to get changes into, and harder to integrate with or customize.

Today, you can still justify either choice. The question to ask yourself is are the growing pains of Hyper so large that you want to build your new project on the declining platform rather than the rising one?