Redis Transactions, Pipelining and Lua Scripts
On this page
Pipelining vs MULTI/EXEC vs Lua scripts
Moving points from one user to another needs three things: read the sender's balance, decide whether there's enough, then update both balances, with no other client changing anything in between. Redis has three features that sound like they might do this. Only one does it cleanly.
| Tool | Saves round trips | Atomic | Can decide based on a value it read |
|---|---|---|---|
| Pipelining | Yes | No | No |
MULTI / EXEC (with WATCH) | Partly | Yes, for the queued commands | No, the app decides and retries |
| Lua script or Function | Yes, one call | Yes, the whole script | Yes |
Pipelining: fewer round trips, no atomicity
Normally a client sends a command and waits for the reply before sending the next. With 1 ms of network latency, 10,000 commands take at least 10 seconds of waiting. A pipeline sends many commands in one go and reads all the replies together.
with r.pipeline(transaction=False) as pipe:
for i in range(10_000):
pipe.set(f"k:{i}", i)
pipe.execute()Against a local Redis, 10,000 SETs took 1.00 second one by one and 0.05 seconds pipelined. Over a real network the gap is larger. But other clients' commands can still run between yours: pipelining is purely about latency.
Redis transactions: MULTI, EXEC and WATCH
Commands between MULTI and EXEC are queued and then run back to back, with no other client's command in between. Two things surprise people coming from SQL:
- No rollback. If one command fails at run time, the others still run. An error Redis catches while queuing, such as a wrong number of arguments, is different:
EXECthen returnsEXECABORTand runs nothing.
SET name sam
MULTI
SET a 1
INCR name # will fail: "sam" isn't a number
SET b 2
EXEC
# 1) OK
# 2) ERR value is not an integer or out of range
# 3) OK <- a and b were both set anyway- No decisions inside. Every command is queued before any runs, so you can't read the balance and branch on it within the transaction.
WATCH adds optimistic locking. Watch the key, read it, decide in your app, then MULTI/EXEC. If anyone changed the watched key in the meantime, EXEC does nothing and returns null, and you retry:
pipe = r.pipeline()
pipe.watch("user:42:points")
balance = int(pipe.get("user:42:points"))
# ... another client changes user:42:points here ...
pipe.multi()
pipe.decrby("user:42:points", 30)
pipe.incrby("user:99:points", 30)
pipe.execute() # raises redis.WatchError: nothing applied, retry from watch()It works, but under contention on a busy key you retry repeatedly, and every attempt costs several round trips.
In .NET, StackExchange.Redis doesn't expose WATCH directly, because one connection is shared by many callers. Its transactions take conditions instead, such as tran.AddCondition(Condition.StringEqual(key, expected)), and Execute returns false when a condition fails (StackExchange.Redis docs).
Lua scripts and Redis Functions
A Lua script runs on the server as one uninterruptible step. It can read, branch and write, so the transfer needs no retry loop. You saw this on Day 10 for locks and rate limits. Since Redis 7.0, Functions are the maintained way to do it: named, loaded once, stored with the data and replicated, instead of sending script text with every call.
#!lua name=points
local function transfer(keys, args)
local amount = tonumber(args[1])
local balance = tonumber(redis.call('GET', keys[1]) or '0')
if balance < amount then
return 0
end
redis.call('DECRBY', keys[1], amount)
redis.call('INCRBY', keys[2], amount)
return 1
end
redis.register_function('transfer', transfer)cat points.lua | redis-cli -x FUNCTION LOAD REPLACE # "points"
SET user:42:points 100
FCALL transfer 2 user:42:points user:99:points 30 # (integer) 1
FCALL transfer 2 user:42:points user:99:points 500 # (integer) 0, not enough points
MGET user:42:points user:99:points # "70" "30"Why a slow script blocks Redis
The property that makes scripts atomic, that nothing else runs meanwhile, also makes a slow script block every client (Day 1). Keep scripts short and bounded: no loops over large collections. If a script runs past busy-reply-threshold (5 seconds by default), Redis doesn't stop it, since that would break atomicity. Other clients get BUSY errors. You can SCRIPT KILL or FUNCTION KILL it only if it hasn't written anything yet; after a write, the only way out is SHUTDOWN NOSAVE, which loses every change since the last save (Redis docs). Pass every key the script touches in KEYS, not built inside the script, so Redis Cluster can route it; all of them must live in the same slot (Day 8).
Check yourself
The other queued commands still ran. Redis transactions don't roll back on run-time errors, so validate inputs or use a script that checks before writing.
Pipeline them in batches of a few hundred to a few thousand. The time is mostly network round trips, not Redis work.
When the decision logic must run in the application, such as calling another service, and contention on the key is low, so retries are rare.
Scripts run atomically on the single command thread, so nothing else runs until the script finishes. A script that takes two seconds blocks every client for two seconds. Keep scripts short and bounded, and move bulk work into batches outside a script.
Pipelining sends many commands without waiting for each reply, which saves round trips, but other clients' commands can run in between. MULTI/EXEC queues commands and runs them together without interleaving, but doesn't roll back on errors. You can pipeline a transaction to get both benefits.