Distributed Systems for .NET Developers
On this page
Learning Objective
Reason about what changes when a system spans several machines, apply the CAP and consistency vocabulary to real design choices, avoid unsafe distributed locks and clock assumptions, and handle stale reads and delayed revocation deliberately.
Concept Explanation
A single process and a single database hide most hard problems: calls succeed or throw, operations happen in one order, and there is one clock. Once you have multiple servers, replicas, caches and services, those guarantees vanish, and the classic list of fallacies of distributed computing, attributed to Peter Deutsch and colleagues at Sun, describes the false assumptions: the network is reliable, latency is zero, bandwidth is infinite, the network is secure, topology does not change, there is one administrator, transport cost is zero, and the network is homogeneous. Every remote call can time out without telling you whether the work happened, which is why idempotency and retries dominate the design of these systems.
The CAP theorem, proposed by Eric Brewer and proved by Seth Gilbert and Nancy Lynch, says that during a network partition a system must choose between consistency and availability. It is often oversimplified. The practical points are that partitions do happen, that the choice is per operation and not per system, and that even without a partition there is a trade-off between latency and consistency, which the PACELC formulation captures. Consistency comes in degrees. Strong consistency means every read sees the latest write, eventual consistency means replicas converge given time, and useful middle levels include read-your-writes, where a user always sees their own changes, and monotonic reads, where time does not appear to go backward. A React user who saves a form and then sees the old value on the next screen has hit a read replica lagging behind the primary, and the fix is deliberate: read from the primary after a write, pass a version or ETag from the write response, or keep the optimistic result until the server catches up.
Time is the next trap. Clocks on different machines disagree, so never use wall-clock timestamps from two machines to decide which event came first, and never trust a client's clock for anything important. Use database-generated values, version numbers, or logical ordering for that. Clock skew also affects security, since expiry checks on tokens and signed requests allow a small tolerance, and a token lifetime is only as reliable as the clocks of the systems checking it.
Distributed locks are tempting and dangerous. A lock acquired with a timeout can expire while its holder is paused by a garbage collection or a network delay, and then two processes both believe they hold it. Martin Kleppmann's 2016 analysis of the Redlock algorithm popularized the point that a lock needs a fencing token, a number that increases with each acquisition and that the protected resource checks, rejecting requests carrying an older number. Prefer designs that avoid locks: atomic conditional updates, unique constraints, or a single writer per partition. When you truly need mutual exclusion, use the database's facilities, such as the application locks in SQL Server or a blob lease, and combine them with fencing. Security-relevant state has the same issue in a milder form: a revoked permission or a disabled account remains effective in every cache and token until it expires, so choose lifetimes and invalidation deliberately and accept a bounded delay knowingly.
| Problem | Symptom | Practical response |
|---|---|---|
| Timeouts and lost replies | Unknown whether the work happened | Idempotency keys and safe retries |
| Replica lag | User does not see their own write | Read from primary after write, or use version tokens |
| Clock skew | Wrong ordering, early or late expiry | Database-assigned order, tolerances on tokens |
| Lock expiry during a pause | Two holders at once | Fencing tokens, or avoid locks with atomic updates |
| Network partition | Split brain, divergent state | Quorums, a single writer, explicit conflict rules |
| Cached revocation | A removed user keeps access briefly | Short lifetimes, event-driven invalidation, documented bound |
Worked Example: Read-Your-Writes, Fencing Tokens and Retries with Jitter
The first block handles a lagging replica in the API and React. The C# then shows a fenced lock and a retry policy.
// After a write, return the new version. The client passes it back so any server can tell whether its data is fresh enough.
app.MapPut("/api/tasks/{id:int}", async (int id, UpdateTask body, AppDb db, CancellationToken ct) =>
{
var task = await db.Tasks.FindAsync([id], ct);
if (task is null) return Results.NotFound();
task.Title = body.Title;
await db.SaveChangesAsync(ct);
return Results.Ok(new { task.Id, task.Title, version = task.RowVersionAsLong }); // the write response is authoritative
});
// Reads that follow a write ask for at least that version; if the replica is behind, use the primary.
app.MapGet("/api/tasks/{id:int}", async (int id, [FromQuery] long? minVersion, ReadWriteDbFactory dbs, CancellationToken ct) =>
{
var replicaRead = await dbs.Replica.Tasks.AsNoTracking().SingleOrDefaultAsync(t => t.Id == id, ct);
if (replicaRead is not null && (minVersion is null || replicaRead.RowVersionAsLong >= minVersion)) return Results.Ok(replicaRead);
var fresh = await dbs.Primary.Tasks.AsNoTracking().SingleOrDefaultAsync(t => t.Id == id, ct); // fall back to the primary
return fresh is null ? Results.NotFound() : Results.Ok(fresh);
});// React: keep the server's write response in the cache, so the next screen shows it even if a replica lags.
const update = useMutation({
mutationFn: (input: UpdateTask) => request(`/api/tasks/${input.id}`, TaskSchema, { method: 'PUT', body: input }),
onSuccess: (saved) => queryClient.setQueryData(taskKeys.detail(userId, saved.id), saved), // no immediate refetch racing the replica
});-- A lock with a fencing token: the token increases on every acquisition, and the resource rejects older tokens.
CREATE TABLE dbo.JobLease (JobName nvarchar(100) PRIMARY KEY, HolderId uniqueidentifier NOT NULL, FenceToken bigint NOT NULL, ExpiresAtUtc datetime2 NOT NULL);
-- Acquire: succeeds only if the lease is free or expired, and returns the new token.
UPDATE dbo.JobLease
SET HolderId = @me, FenceToken = FenceToken + 1, ExpiresAtUtc = DATEADD(second, 30, SYSUTCDATETIME())
OUTPUT inserted.FenceToken
WHERE JobName = @job AND (ExpiresAtUtc < SYSUTCDATETIME() OR HolderId = @me);
-- Every write by the job includes its token. A holder that paused past its lease carries an old token and is rejected.
UPDATE dbo.Reports SET Status = 'done' WHERE Id = @id AND @token >= (SELECT FenceToken FROM dbo.JobLease WHERE JobName = @job);// Retries: only for safe or idempotent operations, with exponential backoff, jitter, a cap, and a total budget.
var pipeline = new ResiliencePipelineBuilder<HttpResponseMessage>()
.AddRetry(new RetryStrategyOptions<HttpResponseMessage>
{
MaxRetryAttempts = 3,
BackoffType = DelayBackoffType.Exponential,
UseJitter = true, // avoids many clients retrying in lockstep, a retry storm
Delay = TimeSpan.FromMilliseconds(200),
ShouldHandle = args => ValueTask.FromResult(args.Outcome.Result?.StatusCode is HttpStatusCode.ServiceUnavailable or HttpStatusCode.RequestTimeout),
})
.AddTimeout(TimeSpan.FromSeconds(5))
.Build();Jitter matters more than it first seems. Without it, every client that failed at the same moment retries at the same moment, which turns a brief outage into a sustained overload.
Real-World Examples
- In October 2018 GitHub suffered a network partition lasting 43 seconds between its US East Coast network hub and its primary data center, which triggered automated database failover to a different region. The cluster's replicas ended up with writes that had not been replicated, and the resulting reconciliation kept parts of the service degraded for about a day. It is a well-documented case of how a very brief partition can create divergent state.
- On 28 February 2017 an operator running a command to remove a small number of servers from an Amazon S3 billing system in the US-East-1 region entered a larger set than intended, and the resulting removal of capacity caused an outage of several hours that affected many dependent services. The incident report led to safeguards in tooling and illustrates how failure spreads through dependencies that teams did not know they had.
Review Questions & Answers
When a network partition occurs, a system cannot be both fully consistent and fully available, so it must choose which to give up for the affected operations. Partitions do occur, and the choice can differ per operation, so it is a design tool, not a label for a whole database.
The read may be served by a replica that has not yet received the write. Returning the new version from the write, caching the response on the client, or reading from the primary until the version is visible restores read-your-writes behavior.
The holder can pause or be delayed past the timeout, after which another process acquires the lock, and both then act as if they hold it. A fencing token that the protected resource checks rejects stale holders.
When many clients fail simultaneously and retry on the same schedule, their combined retries overload the recovering service again. Adding random jitter to the backoff spreads the retries across time and gives the service room to recover.