During Public Preview you can use the Team plan for free, and get an early-adopter discount at GA. Sign in
Chaos Engineering
We've all been there - the status page is all green ticks, but something in your application isn't right. A retry that didn't fire, a timeout that wasn't long enough, an error code nobody planned for.
You can build the greatest application in the world, but at some point you're inevitably going to encounter issues - what matters is having thought through the error cases up front, and having a tested scenario to prove they're handled.
That's what chaos engineering is for - deliberately injecting failures into a system so you can verify your application handles them the way you intended. Locally has built-in support for chaos engineering, and allows you to emulate different failure scenarios across different components. This lets you see how your application behaves when something goes wrong, and then prove that it's fixed - be that adding exponential backoff to requests, or making sure you fail over to the multi-region setup you expect.
Faults can be injected into the Control Plane, the Directory, sign-in and tokens, and every Data Plane Emulator - over HTTP, and over the protocols they serve beyond it: AMQP, MQTT, WebSockets, gRPC, the MongoDB wire protocol and database connections. Each fault answers with the service's own errors, so you're testing the retry handling your client really has.
To see this end to end, locallybuild/example-app-service-multi-region deploys an orders API across two regions - which you can then use to see what happens when the primary region goes offline.
Enabling Chaos
Chaos is enabled and configured per Subscription - so enabling it for one Subscription doesn't affect any others. You can do this through the Chaos tab in the Locally Dashboard:
You can also enable Chaos for a Subscription using the CLI:
$
locally chaos enable --subscription <subscription-id>
To disable chaos when you are done testing:
$
locally chaos disable --subscription <subscription-id>
Disabling keeps every fault's settings, so enabling it again picks up where you left off - and setting a fault switches Chaos on for you. The Directory and Identity belong to a Tenant rather than a Subscription, so they have their own switch, locally chaos directory enable.
locally chaos show prints what's configured, and locally chaos <surface> reset puts a surface back to normal.
Configuring a fault
Each fault is configured on its own, with a percentage of requests it applies to - so you can simulate anything from the occasional hiccup to a sustained outage. For example, to fail a quarter of requests to Storage with a 429:
$
locally chaos storage 429 --percentage 25
You can also add rules, to apply a different percentage to some requests:
--location- one region, such aslondon.--name- one resource, such as a single storage account (acct1, orrg1/acct1for one Resource Group's). Add an endpoint to narrow it further, such asacct1:blob. On the Directory and Identity,--nameis a user or application instead.--plugin- on the Control Plane, one Resource Provider, such asMicrosoft.Compute.
The most specific rule wins, so you can fail requests in a single region, or everywhere except one - a rate of 100 plus a rule at 0:
$
locally chaos storage 429 --percentage 100
$
locally chaos storage 429 --percentage 0 --location london
Some faults take more settings:
--millisor--seconds- how long a delay or a scheduled fault lasts. One value picks anything up to it each time; a range, such as--seconds 5-20, picks within it.--retry-after- the retry hint a throttled client is given, such as5s. Without it, the service's usual hint is sent.--interval- how often a scheduled fault starts, such as10m. Onlatencyanddrop-connectionit also delays or cuts connections that are already open.
$
locally chaos directory 429 --percentage 50 --retry-after 5s
You can also configure Chaos for a single region from the map on the Regions page - right-click a region and choose the fault, the percentage of requests it applies to and, optionally, a single plugin. Turning on a fault this way enables Chaos for the Subscription, and the map shows which regions are affected.
Scheduled faults
Most faults act on single requests. Scheduled faults act on running services instead: every --interval, each resource in scope rolls against the percentage, and on a hit the fault lasts for the duration you set. For example, every 30 seconds, maybe freeze a container group for 5 to 20 seconds:
$
locally chaos containerinstances freeze --percentage 50 --seconds 5-20 --interval 30s
Each resource rolls on its own clock, with its first roll at a random point within the first interval - so resources armed together, such as every container group in a region, don't all fail at the same moment.
| Fault | What it does |
|---|---|
| incident | Takes each resource down for a while, then brings it back. Use outage for downtime that only ends when you turn it off. |
| failover | Makes the primary unavailable for a while: a secondary serves reads only, or the SDKs fail over to the next region, then the primary returns. |
| freeze | Pauses an app, container group or VM, then carries on where it left off. Nothing is closed, so open connections go quiet. |
| exit | Kills an app, container group or VM outright, as a crash would. |
| restart | Restarts an app, container group, VM or database server, as its own restart action does, dropping its connections. |
| cold-start | Holds a site or app in its warm-up response, as if it had just started. |
What can be faulted?
Each of these can be configured within the Chaos tab of the Locally Dashboard or using the Locally CLI, as locally chaos <surface> <fault>. "All six" is the six faults every HTTP service takes: 403, 429, 500, drop-connection, latency and outage.
| Surface | CLI | Faults |
|---|---|---|
| Control Plane (Resource Manager) | control-plane | all six, incident, lro-delay, lro-failure, out-of-capacity, quota-exceeded, recase-names |
| Directory (Microsoft Graph) | directory | all six, eventual-consistency, incident, stale-read |
| Identity (sign-in and tokens) | identity | all six, clock-skew, discovery-unavailable, incident, invalid-grant, jwks-unavailable, msi-unavailable, slow-down |
| API Management | apimanagement | all six, 502, 504, disconnect, incident, partial-response, slow-body |
| App Configuration | appconfiguration | all six, failover, incident, stale-read |
| Container Apps | containerapps | all six, 502, 504, cold-start, disconnect, exit, freeze, partial-response, restart, slow-body |
| Container Instances | containerinstances | drop-connection, latency, outage, disconnect, exit, freeze, incident, restart |
| Container Registry | containerregistry | all six, incident, partial-response, slow-body |
| Cosmos DB | cosmos | all six, disconnect, failover, incident, partial-response, partition-split, request-timeout, retry-with, session-not-available, stale-read, write-forbidden |
| Durable Task Scheduler | durabletask | all six, disconnect, stream-reset |
| Event Grid | eventgrid | all six, connack-refuse, delivery-delay, delivery-drop, disconnect, duplicate-delivery, handshake-failure, lock-lost, reorder |
| Event Hubs | eventhub | all six, delivery-delay, disconnect, duplicate-delivery, entity-disabled, link-stolen, quota-exceeded, reorder |
| Function Apps | functions | all six, 502, 504, cold-start, delivery-delay, delivery-drop, duplicate-delivery, exit, freeze, partial-response, restart, slow-body |
| HorizonDB | horizondb | drop-connection, latency, outage, disconnect, incident |
| IoT Hub | iothub | all six, connack-refuse, delivery-delay, delivery-drop, disconnect, entity-disabled, quota-exceeded |
| Key Vault | keyvault | all six, incident, object-being-deleted |
| Managed HSM | managedhsm | all six, incident, object-being-deleted |
| Monitor (queries and metrics) | monitor | all six, delivery-delay, delivery-drop, duplicate-delivery, incident, request-timeout, stale-read |
| Monitor (ingestion) | monitoringestion | all six, incident, partial-success, quota-exceeded |
| MySQL | mysql | all six, disconnect, failover, restart, too-many-connections |
| PostgreSQL | postgresql | all six, disconnect, failover, restart, too-many-connections |
| Redis | redis | all six, busy, disconnect, failover, loading, moved, restart, too-many-connections |
| Service Bus | servicebus | all six, delivery-delay, disconnect, duplicate-delivery, entity-disabled, link-stolen, lock-lost, quota-exceeded, reorder |
| SQL Database | sql | all six, disconnect, failover, restart, too-many-connections |
| Storage | storage | all six, delivery-delay, delivery-drop, duplicate-delivery, failover, incident, lock-lost, partial-response, request-timeout, slow-body, stale-read |
| Virtual Machines | virtualmachines | drop-connection, latency, outage, disconnect, exit, freeze, restart |
| Web Apps | webapps | all six, 502, 504, cold-start, disconnect, exit, freeze, partial-response, restart, slow-body, stream-reset |
functions covers Function Apps only: Web Apps and Container Apps have their own surfaces. Likewise monitor covers queries and metrics, and monitoringestion covers sending telemetry and logs.
Faults
Every service
| Fault | What it does |
|---|---|
| 403 | Answers with 403, as when a role assignment hasn't replicated yet (on the Control Plane, AuthorizationFailed). |
| 429 | Answers with 429 Too Many Requests and a retry hint. Databases send their engine's own throttling error instead. |
| 500 | Answers with 500 Internal Server Error. |
| drop-connection | Closes the connection without answering, so the client sees it reset with no status code. |
| latency | Holds requests for a random delay, up to the duration you set, then handles them as normal. |
| outage | Fails every matching request while it's on, as a region or service that's down does - usually with a 503 - and closes the live connections that were open when it started. |
Connections and responses
| Fault | What it does |
|---|---|
| disconnect | Closes live connections cleanly, in the protocol's own terms, so the client has to reconnect. |
| partial-response | Sends the status, headers and part of the body, then cuts the connection. |
| slow-body | Slows the response body down. |
| 502 / 504 | Answers from the service's front end with 502 Bad Gateway or 504 Gateway Timeout, as when it can't reach the app behind it, or the app takes too long. |
| request-timeout | Fails requests with the service's own server-side timeout, such as Cosmos DB's 408 or Storage's OperationTimedOut. |
| stream-reset | Resets live gRPC calls with RST_STREAM. |
| too-many-connections | Refuses new connections with the database engine's own connection-limit error. |
| stale-read | Serves stale reads after a write. |
| quota-exceeded | Refuses requests because a quota is full. On the Control Plane, resource creation fails with 409 QuotaExceeded. |
| entity-disabled | Refuses requests because the entity (such as a queue or event hub) is disabled. |
| object-being-deleted | Keeps a deleted name busy for a while, so recreating or recovering it fails until it's free. |
Messages and events
| Fault | What it does |
|---|---|
| delivery-delay | Holds messages, events or triggers for a while before delivering them. |
| delivery-drop | Never delivers some messages, events or triggers. |
| duplicate-delivery | Delivers some messages, events or triggers twice. |
| reorder | Swaps messages or events with the next one in their batch, so they arrive out of order. |
| lock-lost | Reports that a lock or lease the client holds is gone, such as a message lock, a blob lease or a queue message's pop receipt. |
| link-stolen | Detaches live AMQP receiver links as stolen, as when another receiver takes over the entity or partition. |
| connack-refuse | Refuses MQTT connections. |
| handshake-failure | Fails Event Grid's webhook validation handshakes. |
Specific to one service
| Fault | What it does |
|---|---|
| recase-names | Control Plane - changes the casing of Resource Group names in responses (for example sEaRcH-aPi-rEsOuRcEs), to find code that compares resource IDs case-sensitively. |
| out-of-capacity | Control Plane - fails resource creation, and updates that change a SKU or zone, with a capacity error such as AllocationFailed. |
| lro-failure / lro-delay | Control Plane - makes Long-Running Operations finish in a failed state, or take longer to finish. |
| eventual-consistency | Directory - returns a not-yet-replicated view of what was just written, as Microsoft Graph does. |
| write-forbidden | Cosmos DB - refuses writes with 403 substatus 3, as a region that isn't accepting writes does. |
| partition-split | Cosmos DB - answers with 410 Gone substatus 1002, as Cosmos DB does while a partition splits. |
| session-not-available | Cosmos DB - answers reads with 404 substatus 1002, as when a replica hasn't caught up with the client's session. |
| retry-with | Cosmos DB - answers writes with 449 Retry With, as on a write conflict the client is expected to retry. |
| busy / loading / moved | Redis - answers commands with -BUSY, -LOADING or -MOVED, as a server running a long script, still loading its data, or a cluster whose slot has moved does. |
| partial-success | Monitor ingestion - accepts telemetry batches with 206, rejecting some of the items in each. |
| clock-skew | Identity - issues tokens that aren't valid for another 5 minutes, as an authority with a skewed clock does. |
| invalid-grant | Identity - refuses refresh tokens with invalid_grant (AADSTS700082), as if they'd expired. |
| slow-down | Identity - answers device code polls with slow_down, so the client has to poll less often. |
| discovery-unavailable / jwks-unavailable | Identity - answers the OpenID Connect metadata or signing keys endpoint with 503, so tokens can't be validated. |
| msi-unavailable | Identity - fails managed identity token requests with 500 or 503 from the endpoint the app calls. |
During Public Preview you can use the Team plan for free, and get an early-adopter discount at GA. Sign in