Chaos Engineering

Preview

During Public Preview you can use the Team plan for free, and get an early-adopter discount at GA. Sign in

Chaos Engineering

Available on Starter Standard Team Compare plans →

We've all been there - the status page is all green ticks, but something in your application isn't right. A retry that didn't fire, a timeout that wasn't long enough, an error code nobody planned for.

You can build the greatest application in the world, but at some point you're inevitably going to encounter issues - what matters is having thought through the error cases up front, and having a tested scenario to prove they're handled.

That's what chaos engineering is for - deliberately injecting failures into a system so you can verify your application handles them the way you intended. Locally has built-in support for chaos engineering, and allows you to emulate different failure scenarios across different components. This lets you see how your application behaves when something goes wrong, and then prove that it's fixed - be that adding exponential backoff to requests, or making sure you fail over to the multi-region setup you expect.

Faults can be injected into the Control Plane, the Directory, sign-in and tokens, and every Data Plane Emulator - over HTTP, and over the protocols they serve beyond it: AMQP, MQTT, WebSockets, gRPC, the MongoDB wire protocol and database connections. Each fault answers with the service's own errors, so you're testing the retry handling your client really has.

To see this end to end, locallybuild/example-app-service-multi-region deploys an orders API across two regions - which you can then use to see what happens when the primary region goes offline.

Enabling Chaos

Chaos is enabled and configured per Subscription - so enabling it for one Subscription doesn't affect any others. You can do this through the Chaos tab in the Locally Dashboard:

Screenshot of the Chaos section within the Locally Dashboard

You can also enable Chaos for a Subscription using the CLI:

$ locally chaos enable --subscription <subscription-id>

To disable chaos when you are done testing:

$ locally chaos disable --subscription <subscription-id>

Disabling keeps every fault's settings, so enabling it again picks up where you left off - and setting a fault switches Chaos on for you. The Directory and Identity belong to a Tenant rather than a Subscription, so they have their own switch, locally chaos directory enable.

locally chaos show prints what's configured, and locally chaos <surface> reset puts a surface back to normal.

Configuring a fault

Each fault is configured on its own, with a percentage of requests it applies to - so you can simulate anything from the occasional hiccup to a sustained outage. For example, to fail a quarter of requests to Storage with a 429:

$ locally chaos storage 429 --percentage 25

You can also add rules, to apply a different percentage to some requests:

  • --location - one region, such as london.
  • --name - one resource, such as a single storage account (acct1, or rg1/acct1 for one Resource Group's). Add an endpoint to narrow it further, such as acct1:blob. On the Directory and Identity, --name is a user or application instead.
  • --plugin - on the Control Plane, one Resource Provider, such as Microsoft.Compute.

The most specific rule wins, so you can fail requests in a single region, or everywhere except one - a rate of 100 plus a rule at 0:

$ locally chaos storage 429 --percentage 100 $ locally chaos storage 429 --percentage 0 --location london

Some faults take more settings:

  • --millis or --seconds - how long a delay or a scheduled fault lasts. One value picks anything up to it each time; a range, such as --seconds 5-20, picks within it.
  • --retry-after - the retry hint a throttled client is given, such as 5s. Without it, the service's usual hint is sent.
  • --interval - how often a scheduled fault starts, such as 10m. On latency and drop-connection it also delays or cuts connections that are already open.
$ locally chaos directory 429 --percentage 50 --retry-after 5s

You can also configure Chaos for a single region from the map on the Regions page - right-click a region and choose the fault, the percentage of requests it applies to and, optionally, a single plugin. Turning on a fault this way enables Chaos for the Subscription, and the map shows which regions are affected.

Scheduled faults

Most faults act on single requests. Scheduled faults act on running services instead: every --interval, each resource in scope rolls against the percentage, and on a hit the fault lasts for the duration you set. For example, every 30 seconds, maybe freeze a container group for 5 to 20 seconds:

$ locally chaos containerinstances freeze --percentage 50 --seconds 5-20 --interval 30s

Each resource rolls on its own clock, with its first roll at a random point within the first interval - so resources armed together, such as every container group in a region, don't all fail at the same moment.

Fault What it does
incident Takes each resource down for a while, then brings it back. Use outage for downtime that only ends when you turn it off.
failover Makes the primary unavailable for a while: a secondary serves reads only, or the SDKs fail over to the next region, then the primary returns.
freeze Pauses an app, container group or VM, then carries on where it left off. Nothing is closed, so open connections go quiet.
exit Kills an app, container group or VM outright, as a crash would.
restart Restarts an app, container group, VM or database server, as its own restart action does, dropping its connections.
cold-start Holds a site or app in its warm-up response, as if it had just started.

What can be faulted?

Each of these can be configured within the Chaos tab of the Locally Dashboard or using the Locally CLI, as locally chaos <surface> <fault>. "All six" is the six faults every HTTP service takes: 403, 429, 500, drop-connection, latency and outage.

Surface CLI Faults
Control Plane (Resource Manager) control-plane all six, incident, lro-delay, lro-failure, out-of-capacity, quota-exceeded, recase-names
Directory (Microsoft Graph) directory all six, eventual-consistency, incident, stale-read
Identity (sign-in and tokens) identity all six, clock-skew, discovery-unavailable, incident, invalid-grant, jwks-unavailable, msi-unavailable, slow-down
API Management apimanagement all six, 502, 504, disconnect, incident, partial-response, slow-body
App Configuration appconfiguration all six, failover, incident, stale-read
Container Apps containerapps all six, 502, 504, cold-start, disconnect, exit, freeze, partial-response, restart, slow-body
Container Instances containerinstances drop-connection, latency, outage, disconnect, exit, freeze, incident, restart
Container Registry containerregistry all six, incident, partial-response, slow-body
Cosmos DB cosmos all six, disconnect, failover, incident, partial-response, partition-split, request-timeout, retry-with, session-not-available, stale-read, write-forbidden
Durable Task Scheduler durabletask all six, disconnect, stream-reset
Event Grid eventgrid all six, connack-refuse, delivery-delay, delivery-drop, disconnect, duplicate-delivery, handshake-failure, lock-lost, reorder
Event Hubs eventhub all six, delivery-delay, disconnect, duplicate-delivery, entity-disabled, link-stolen, quota-exceeded, reorder
Function Apps functions all six, 502, 504, cold-start, delivery-delay, delivery-drop, duplicate-delivery, exit, freeze, partial-response, restart, slow-body
HorizonDB horizondb drop-connection, latency, outage, disconnect, incident
IoT Hub iothub all six, connack-refuse, delivery-delay, delivery-drop, disconnect, entity-disabled, quota-exceeded
Key Vault keyvault all six, incident, object-being-deleted
Managed HSM managedhsm all six, incident, object-being-deleted
Monitor (queries and metrics) monitor all six, delivery-delay, delivery-drop, duplicate-delivery, incident, request-timeout, stale-read
Monitor (ingestion) monitoringestion all six, incident, partial-success, quota-exceeded
MySQL mysql all six, disconnect, failover, restart, too-many-connections
PostgreSQL postgresql all six, disconnect, failover, restart, too-many-connections
Redis redis all six, busy, disconnect, failover, loading, moved, restart, too-many-connections
Service Bus servicebus all six, delivery-delay, disconnect, duplicate-delivery, entity-disabled, link-stolen, lock-lost, quota-exceeded, reorder
SQL Database sql all six, disconnect, failover, restart, too-many-connections
Storage storage all six, delivery-delay, delivery-drop, duplicate-delivery, failover, incident, lock-lost, partial-response, request-timeout, slow-body, stale-read
Virtual Machines virtualmachines drop-connection, latency, outage, disconnect, exit, freeze, restart
Web Apps webapps all six, 502, 504, cold-start, disconnect, exit, freeze, partial-response, restart, slow-body, stream-reset

functions covers Function Apps only: Web Apps and Container Apps have their own surfaces. Likewise monitor covers queries and metrics, and monitoringestion covers sending telemetry and logs.

Faults

Every service

Fault What it does
403 Answers with 403, as when a role assignment hasn't replicated yet (on the Control Plane, AuthorizationFailed).
429 Answers with 429 Too Many Requests and a retry hint. Databases send their engine's own throttling error instead.
500 Answers with 500 Internal Server Error.
drop-connection Closes the connection without answering, so the client sees it reset with no status code.
latency Holds requests for a random delay, up to the duration you set, then handles them as normal.
outage Fails every matching request while it's on, as a region or service that's down does - usually with a 503 - and closes the live connections that were open when it started.

Connections and responses

Fault What it does
disconnect Closes live connections cleanly, in the protocol's own terms, so the client has to reconnect.
partial-response Sends the status, headers and part of the body, then cuts the connection.
slow-body Slows the response body down.
502 / 504 Answers from the service's front end with 502 Bad Gateway or 504 Gateway Timeout, as when it can't reach the app behind it, or the app takes too long.
request-timeout Fails requests with the service's own server-side timeout, such as Cosmos DB's 408 or Storage's OperationTimedOut.
stream-reset Resets live gRPC calls with RST_STREAM.
too-many-connections Refuses new connections with the database engine's own connection-limit error.
stale-read Serves stale reads after a write.
quota-exceeded Refuses requests because a quota is full. On the Control Plane, resource creation fails with 409 QuotaExceeded.
entity-disabled Refuses requests because the entity (such as a queue or event hub) is disabled.
object-being-deleted Keeps a deleted name busy for a while, so recreating or recovering it fails until it's free.

Messages and events

Fault What it does
delivery-delay Holds messages, events or triggers for a while before delivering them.
delivery-drop Never delivers some messages, events or triggers.
duplicate-delivery Delivers some messages, events or triggers twice.
reorder Swaps messages or events with the next one in their batch, so they arrive out of order.
lock-lost Reports that a lock or lease the client holds is gone, such as a message lock, a blob lease or a queue message's pop receipt.
link-stolen Detaches live AMQP receiver links as stolen, as when another receiver takes over the entity or partition.
connack-refuse Refuses MQTT connections.
handshake-failure Fails Event Grid's webhook validation handshakes.

Specific to one service

Fault What it does
recase-names Control Plane - changes the casing of Resource Group names in responses (for example sEaRcH-aPi-rEsOuRcEs), to find code that compares resource IDs case-sensitively.
out-of-capacity Control Plane - fails resource creation, and updates that change a SKU or zone, with a capacity error such as AllocationFailed.
lro-failure / lro-delay Control Plane - makes Long-Running Operations finish in a failed state, or take longer to finish.
eventual-consistency Directory - returns a not-yet-replicated view of what was just written, as Microsoft Graph does.
write-forbidden Cosmos DB - refuses writes with 403 substatus 3, as a region that isn't accepting writes does.
partition-split Cosmos DB - answers with 410 Gone substatus 1002, as Cosmos DB does while a partition splits.
session-not-available Cosmos DB - answers reads with 404 substatus 1002, as when a replica hasn't caught up with the client's session.
retry-with Cosmos DB - answers writes with 449 Retry With, as on a write conflict the client is expected to retry.
busy / loading / moved Redis - answers commands with -BUSY, -LOADING or -MOVED, as a server running a long script, still loading its data, or a cluster whose slot has moved does.
partial-success Monitor ingestion - accepts telemetry batches with 206, rejecting some of the items in each.
clock-skew Identity - issues tokens that aren't valid for another 5 minutes, as an authority with a skewed clock does.
invalid-grant Identity - refuses refresh tokens with invalid_grant (AADSTS700082), as if they'd expired.
slow-down Identity - answers device code polls with slow_down, so the client has to poll less often.
discovery-unavailable / jwks-unavailable Identity - answers the OpenID Connect metadata or signing keys endpoint with 503, so tokens can't be validated.
msi-unavailable Identity - fails managed identity token requests with 500 or 503 from the endpoint the app calls.
Preview

During Public Preview you can use the Team plan for free, and get an early-adopter discount at GA. Sign in

A local cloud for you and your AI agents.

Your Azure infrastructure, running on your machine. Deploy in seconds, break things freely, and ship to Azure when you're ready.