What is Envoy?

Rate-limiting is an effective and simple way to mitigate cascading failure and shared resource exhaustion. Envoy is a feature rich proxy which allows for the easy addition of rate limiting, to any service. This post walks through configuring envoy to enforce rate limiting without changing any application level configuration.

Have you ever had the problem of a resource (api, db, etc) being overwhelmed or saturated with requests? In microservice architectures, resources without constraints on their usage can easily become overwhelmed by the number of clients making requests. There may be any number of clients, each implementing a wide variety of retry/backoff or rate-limiting policies. Greedy clients can easily starve resources from other clients by saturating a service. Worst of all greedy clients can make requests until it completely brings down a service.

A common way to enforce usage constraints is to enable rate limiting. Envoy allows for quick, performant and reliable global rate limiting at the HTTP layer. This is in contrast to IP based rate limiting or an application level rate limiting like may web frameworks provide.

This tutorial will be using the architecture illustrated in the image above. The Service Client on the left represents a client with particularly high usage. During its execution it can completely saturate all the service instances and cause other higher priority clients to have their requests dropped.

Envoy enables the ability to rate limit any service at the network level, without making any modifications to an application. Additionally, since envoy is application level 7 aware it can inspect and rate limit on HTTP header information.

During simulated background jobs the API /slow resource spikes to 3500 requests per second, impacting other endpoints and clients:

The solution configured below uses Envoy to enforce a 500 requests / second limit. But first…

Envoy is a light weight proxy server that natively handles TCP/IP/HTTP/GRPC/HTTP2 Connections. It is highly configurable and supports many different plugins. It also makes observability a first class citizen.

Before Envoy, application level retries, latency injection, rate limiting, and circuit breaking were pretty expensive application level initiatives. Envoy outsources these and empowers ops minded people to configure and enable these organizational capabilities without any application level changes. Envoy is completely upending service resiliency and microservice operations.

(The resources that I read in order to write this are linked at the end of this article.)

The solution outlined below will:

We need a way to limit the # of requests being made in order to insulate the API from spiking and ensure that other clients can make progress when these batch jobs (simulated by vegeta) are being executed. In order to do this we’ll place Envoy proxies to the fake api locally on the batch/vegeta client.

Running Envoy as a sidecar to the batch job client allows for rate limiting requests before even hitting the load balancer! Envoy is a good candidate for this because it’s highly configurable, performant and can handle HTTP balancing. Additionally, it is a cloud native project and is highly observable.

The first step is to configure Envoy to sit in between the batch job and the API. All requests that the service will make to the API will be done through envoy. First we’ll configure Envoy to know about the api. Then we’ll update the Batch job to make requests to Envoy instead of directly to the API. The final state of this step will look like the following:

This step only routes API traffic through Envoy, it doesn’t yet perform any rate limiting. In order to configure Envoy we need a couple of things:

A cluster Represents the upstream resource that Envoy will connect to (in this case it is the API load balancer). The configuration for this is pretty simple:

In our example we’re running a fake api on localhost:8080 to simulate the load balancer in the chart above. The above configuration says any requests through Envoy to the API should be made to localhost:8080.

This section establishes that all requests are routed to the api cluster defined above.

The rest of the configuration file establishes which address Envoy itself will listen on and the rules for connections.

The above graphs show that Envoy is now receiving ALL requests to the API, and is sending them upstream to the Load Balancer!

The final step is to configure Envoy to use the rate limiting service in order to enforce rate limits and slow the rate of requests to the API. This makes it so that Envoy will check the rate limits on each incoming connection and filter down requests according to the configuration above (max 500 request/second):

The Envoy configuration file to use the rate limiter looks like:

We then run the load test at 1000 requests / second (2x the allowed limit):

Looking at Lyft’s ratelimiter logs shows it accepting requests and doing rate limit lookups:

Stopping our load test prints the vegeta report showing that 1/2 of the requests are getting rate limited by Envoy resulting in a status code of 429!!!

Visualizing the # of requests that our api service actually sees shows that it hovers around ~500 requests / second, exactly what we were hoping for!

Once again, looking at envoys outgoing api connections shows that requests hover around ~500 requests per second as well!

Success!

I hope that the above illustrates how simple it is to configure Envoy to mitigate API resource exhaustion from a greedy client. I have found this pattern extremely useful because resiliency is often on the back burner to feature development. Before Envoy application level retries, latency injection, rate limiting, and circuit breaking were pretty expensive application level initiatives. Envoy outsources these and empowers ops minded people to configure and enable these organizational capabilities without any application level changes. Envoy is completely upending service resiliency and I hope you had as much fun reading this article as I did writing it!!

Related posts:

The OnePlus 6T has just been announced, starting at £499, it’s definitely the best value for money phone you can buy. OnePlus are very well known for their great value for money and the brilliant…

O que foi 2017 para mim?! Com certeza um ano cheio de realizações. Sim, realizações. Posso me certificar que a palavra “Realização” com certeza tornou-se a (palavra resumo) de meu ano. Creio que…

Looking back on 2017, we can safely say that this has been the most turbulent year in the short history of blockchain and cryptocurrencies so far. Of course we’re still only at the beginning, but…