Catalog My learning Blog
Full-Stack Developer Course: React, .NET and SQL Server · Phase 13 · Lesson 123 · 7 min read

Distributed Systems for .NET Developers

On this page
  1. Learning Objective
  2. Concept Explanation
  3. Worked Example: Read-Your-Writes, Fencing Tokens and Retries with Jitter
  4. Real-World Examples
  5. Review Questions & Answers

Learning Objective

Reason about what changes when a system spans several machines, apply the CAP and consistency vocabulary to real design choices, avoid unsafe distributed locks and clock assumptions, and handle stale reads and delayed revocation deliberately.

Concept Explanation

A single process and a single database hide most hard problems: calls succeed or throw, operations happen in one order, and there is one clock. Once you have multiple servers, replicas, caches and services, those guarantees vanish, and the classic list of fallacies of distributed computing, attributed to Peter Deutsch and colleagues at Sun, describes the false assumptions: the network is reliable, latency is zero, bandwidth is infinite, the network is secure, topology does not change, there is one administrator, transport cost is zero, and the network is homogeneous. Every remote call can time out without telling you whether the work happened, which is why idempotency and retries dominate the design of these systems.

The CAP theorem, proposed by Eric Brewer and proved by Seth Gilbert and Nancy Lynch, says that during a network partition a system must choose between consistency and availability. It is often oversimplified. The practical points are that partitions do happen, that the choice is per operation and not per system, and that even without a partition there is a trade-off between latency and consistency, which the PACELC formulation captures. Consistency comes in degrees. Strong consistency means every read sees the latest write, eventual consistency means replicas converge given time, and useful middle levels include read-your-writes, where a user always sees their own changes, and monotonic reads, where time does not appear to go backward. A React user who saves a form and then sees the old value on the next screen has hit a read replica lagging behind the primary, and the fix is deliberate: read from the primary after a write, pass a version or ETag from the write response, or keep the optimistic result until the server catches up.

Time is the next trap. Clocks on different machines disagree, so never use wall-clock timestamps from two machines to decide which event came first, and never trust a client's clock for anything important. Use database-generated values, version numbers, or logical ordering for that. Clock skew also affects security, since expiry checks on tokens and signed requests allow a small tolerance, and a token lifetime is only as reliable as the clocks of the systems checking it.

Distributed locks are tempting and dangerous. A lock acquired with a timeout can expire while its holder is paused by a garbage collection or a network delay, and then two processes both believe they hold it. Martin Kleppmann's 2016 analysis of the Redlock algorithm popularized the point that a lock needs a fencing token, a number that increases with each acquisition and that the protected resource checks, rejecting requests carrying an older number. Prefer designs that avoid locks: atomic conditional updates, unique constraints, or a single writer per partition. When you truly need mutual exclusion, use the database's facilities, such as the application locks in SQL Server or a blob lease, and combine them with fencing. Security-relevant state has the same issue in a milder form: a revoked permission or a disabled account remains effective in every cache and token until it expires, so choose lifetimes and invalidation deliberately and accept a bounded delay knowingly.

ProblemSymptomPractical response
Timeouts and lost repliesUnknown whether the work happenedIdempotency keys and safe retries
Replica lagUser does not see their own writeRead from primary after write, or use version tokens
Clock skewWrong ordering, early or late expiryDatabase-assigned order, tolerances on tokens
Lock expiry during a pauseTwo holders at onceFencing tokens, or avoid locks with atomic updates
Network partitionSplit brain, divergent stateQuorums, a single writer, explicit conflict rules
Cached revocationA removed user keeps access brieflyShort lifetimes, event-driven invalidation, documented bound

Worked Example: Read-Your-Writes, Fencing Tokens and Retries with Jitter

The first block handles a lagging replica in the API and React. The C# then shows a fenced lock and a retry policy.

C#
// After a write, return the new version. The client passes it back so any server can tell whether its data is fresh enough.
app.MapPut("/api/tasks/{id:int}", async (int id, UpdateTask body, AppDb db, CancellationToken ct) =>
{
    var task = await db.Tasks.FindAsync([id], ct);
    if (task is null) return Results.NotFound();
    task.Title = body.Title;
    await db.SaveChangesAsync(ct);
    return Results.Ok(new { task.Id, task.Title, version = task.RowVersionAsLong });   // the write response is authoritative
});

// Reads that follow a write ask for at least that version; if the replica is behind, use the primary.
app.MapGet("/api/tasks/{id:int}", async (int id, [FromQuery] long? minVersion, ReadWriteDbFactory dbs, CancellationToken ct) =>
{
    var replicaRead = await dbs.Replica.Tasks.AsNoTracking().SingleOrDefaultAsync(t => t.Id == id, ct);
    if (replicaRead is not null && (minVersion is null || replicaRead.RowVersionAsLong >= minVersion)) return Results.Ok(replicaRead);
    var fresh = await dbs.Primary.Tasks.AsNoTracking().SingleOrDefaultAsync(t => t.Id == id, ct);   // fall back to the primary
    return fresh is null ? Results.NotFound() : Results.Ok(fresh);
});
JavaScript
// React: keep the server's write response in the cache, so the next screen shows it even if a replica lags.
const update = useMutation({
  mutationFn: (input: UpdateTask) => request(`/api/tasks/${input.id}`, TaskSchema, { method: 'PUT', body: input }),
  onSuccess: (saved) => queryClient.setQueryData(taskKeys.detail(userId, saved.id), saved),   // no immediate refetch racing the replica
});
SQL
-- A lock with a fencing token: the token increases on every acquisition, and the resource rejects older tokens.
CREATE TABLE dbo.JobLease (JobName nvarchar(100) PRIMARY KEY, HolderId uniqueidentifier NOT NULL, FenceToken bigint NOT NULL, ExpiresAtUtc datetime2 NOT NULL);

-- Acquire: succeeds only if the lease is free or expired, and returns the new token.
UPDATE dbo.JobLease
SET HolderId = @me, FenceToken = FenceToken + 1, ExpiresAtUtc = DATEADD(second, 30, SYSUTCDATETIME())
OUTPUT inserted.FenceToken
WHERE JobName = @job AND (ExpiresAtUtc < SYSUTCDATETIME() OR HolderId = @me);

-- Every write by the job includes its token. A holder that paused past its lease carries an old token and is rejected.
UPDATE dbo.Reports SET Status = 'done' WHERE Id = @id AND @token >= (SELECT FenceToken FROM dbo.JobLease WHERE JobName = @job);
C#
// Retries: only for safe or idempotent operations, with exponential backoff, jitter, a cap, and a total budget.
var pipeline = new ResiliencePipelineBuilder<HttpResponseMessage>()
    .AddRetry(new RetryStrategyOptions<HttpResponseMessage>
    {
        MaxRetryAttempts = 3,
        BackoffType = DelayBackoffType.Exponential,
        UseJitter = true,                                  // avoids many clients retrying in lockstep, a retry storm
        Delay = TimeSpan.FromMilliseconds(200),
        ShouldHandle = args => ValueTask.FromResult(args.Outcome.Result?.StatusCode is HttpStatusCode.ServiceUnavailable or HttpStatusCode.RequestTimeout),
    })
    .AddTimeout(TimeSpan.FromSeconds(5))
    .Build();

Jitter matters more than it first seems. Without it, every client that failed at the same moment retries at the same moment, which turns a brief outage into a sustained overload.

Every remote call can time out without saying whether the work happened, machine clocks disagree, replicas lag, locks can expire during a pause, and permission changes take time to reach every cache. Each layer has a deliberate response.
Every remote call can time out without saying whether the work happened, machine clocks disagree, replicas lag, locks can expire during a pause, and permission changes take time to reach every cache. Each layer has a deliberate response.

Real-World Examples

  • In October 2018 GitHub suffered a network partition lasting 43 seconds between its US East Coast network hub and its primary data center, which triggered automated database failover to a different region. The cluster's replicas ended up with writes that had not been replicated, and the resulting reconciliation kept parts of the service degraded for about a day. It is a well-documented case of how a very brief partition can create divergent state.
  • On 28 February 2017 an operator running a command to remove a small number of servers from an Amazon S3 billing system in the US-East-1 region entered a larger set than intended, and the resulting removal of capacity caused an outage of several hours that affected many dependent services. The incident report led to safeguards in tooling and illustrates how failure spreads through dependencies that teams did not know they had.
Key TakeawaysIn distributed systems every call can fail without telling you whether it took effect, so design for idempotency, retries with jitter and timeouts. Choose consistency per operation, give users read-your-writes with version tokens, do not order events by machine clocks, avoid distributed locks or use fencing tokens, and treat cached permissions and revocation as having a deliberate maximum delay.

Review Questions & Answers

When a network partition occurs, a system cannot be both fully consistent and fully available, so it must choose which to give up for the affected operations. Partitions do occur, and the choice can differ per operation, so it is a design tool, not a label for a whole database.

The read may be served by a replica that has not yet received the write. Returning the new version from the write, caching the response on the client, or reading from the primary until the version is visible restores read-your-writes behavior.

The holder can pause or be delayed past the timeout, after which another process acquires the lock, and both then act as if they hold it. A fencing token that the protected resource checks rejects stale holders.

When many clients fail simultaneously and retry on the same schedule, their combined retries overload the recovering service again. Adding random jitter to the backoff spreads the retries across time and gives the service room to recover.