Large events and overload

A busy start, a mass restart or a decoder replaying its stored memory can push far more passings per second at AMMC than a normal lap. This page explains what happens inside when that occurs, what is worth configuring and — just as usefully — what is not.

How much it takes

Measured on a developer laptop, decoder streaming as fast as TCP would carry it:

Path	Result
Decoder → console / WebSocket	~48 000 passings/s, nothing lost
Decoder → MariaDB (in a container)	~1 000 passings/s

For scale, a decoder crossing 200 transponders every 20 seconds produces 10 passings/s. Even a decoder emptying its whole 64 000-passing memory after a reconnect stays inside what the socket and the queue absorb. The database is the slow part of the chain, and it is the only part worth thinking about.

The pipeline

Everything the decoder sends is parsed once, then handed to a queue that holds 64 000 passings — the same size as decoder memory, deliberately, so a full replay fits. Each destination you enabled reads that queue independently:

decoder ──TCP──> parser ──> queue (64 000) ─┬─> database
                                            ├─> WebSocket clients
                                            ├─> UDP broadcast
                                            └─> console log

Because they are independent, a slow destination cannot slow the others down, and it cannot slow the decoder down either. What it does instead is fall behind and start missing passings — see below.

Sockets: nothing to configure

AMMC sets its own socket options on connect: a 1 MB receive buffer, a 64 KB send buffer, and TCP_NODELAY so requests are not delayed by Nagle's algorithm. If the operating system refuses a size it logs a warning and carries on with the default.

There is no tuning parameter here, and none is needed. TCP applies backpressure: if AMMC were ever slower than the decoder, the decoder would simply be told to wait, and nothing would be lost. In practice the parser is thousands of times faster than any decoder.

The database is the bottleneck

Passings are written by a single writer, one insert at a time. That is what sets the ~1 000/s figure above, and it has one consequence that surprises people:

-y does not make inserts faster. With the pool at 1, 10 and 50 the measured rate was 1 047, 953 and 992 passings/s — the same number three times. The connection pool is not what limits throughput; the single writer is.

So leave -y at its default of 10 unless your database administrator asks otherwise. -c (connection timeout, default 1000 ms) only affects how long AMMC waits when establishing a connection; raising it helps a database that is slow to accept connections, not one that is slow to insert.

What actually makes inserts faster is the database:

put it on the same machine, or on the same switch — round-trip latency is paid once per passing
give the passings table an index on passing_number if you query it during the event; AMMC creates the table but adds no indexes
avoid a database on a laptop that is also running the timing software, a screen recorder and a browser

When the database goes away

If an insert fails, the passing goes into a local retry queue and AMMC keeps retrying it every 200 ms until it succeeds — nothing is thrown away. But while it is retrying, the writer is not reading new passings from the queue. They pile up in the 64 000-slot queue behind it.

That gives you a comfortable but finite margin. At 10 passings/s the database can be down for well over an hour before the queue is full. Once it does fill, the oldest passings are dropped to make room and you will see

[WARN  ammc_db] DB consumer lagged by 128 messages, continuing

Those passings are lost from the database. They were still converted and still went to your WebSocket and UDP clients — only the database missed them. Recover them afterwards by restarting with -r, or -p to re-download everything the decoder still holds.

WebSocket clients

Each connected client is served by its own task with its own view of the queue, so a client on a bad Wi-Fi link falls behind on its own without affecting anyone else. If it falls more than 64 000 passings behind it skips ahead and logs

[WARN  ws] Websocket passing consumer lagged by 512 messages, continuing

The client stays connected and keeps receiving current data — it just has a gap. This is the right trade for a live scoreboard, which wants now rather than a backlog. If your application cannot tolerate a gap, write to a database as well and reconcile from there.

UDP broadcast

UDP is fire-and-forget by design: no acknowledgement, no retransmission and no backpressure. Under load it is the cheapest destination AMMC has, and also the one with the fewest guarantees. Do not use it as your only record of an event — see UDP broadcast.

Configuring a large event

For a large event, in order of how much difference it makes:

Write to a database, on a machine you control. It is the only destination that keeps every passing across a restart. Local or same-switch beats anything remote.
Hide the connection string with --de so a shared machine does not expose the password — see Storing data.
Use -q. Verbose console logging at thousands of passings per second costs real time, especially on Windows where the console is slow. -q prints errors and warnings only, which is what you want to see anyway.
Set the track fields (--trackId, --trackName, …) so passings from several decoders or several days can be told apart later — Track info.
Leave the buffers and the pool alone. They are already sized for a full decoder replay.
Start it as a service so it restarts on its own, and add AMMC_STARTUP_DELAY_SECS if it comes up before the network does — see Introduction.

A configuration for a busy day, entirely through the environment so nothing sensitive is on the command line:

set AMMC_INTERFACE=10.0.11.10
set AMMC_DE=bXlzcWw6Ly9hbW1jOnNlY3JldEBkYi5sb2NhbDozMzA2L3RpbWluZw==
set AMMC_WEBSOCKET=9000
set AMMC_TRACK_NAME=PragueRage1
set AMMC_QUITE=true
ammc-amb.exe

Restarting mid-event

If AMMC is stopped and restarted during an event, use -r (AMMC_REQUERY_DB). It looks up the last passing number already stored and asks the decoder to resend only from there — the gap is filled, nothing is duplicated, and nothing has to be done by hand. It needs a database to look the number up in; without one it warns and requests nothing.

-p (AMMC_REQUEST_PASSINGS) is the blunt version: it asks the decoder for everything it still holds, up to 64 000 passings. Useful after a crash with no database, noisy otherwise.

What to watch in the log

Three warnings mean "AMMC is at its limit", and all three name the destination that is behind:

DB consumer lagged by N messages
Websocket passing consumer lagged by N messages
UDP consumer lagged by N messages

Anything else — a database insert error, a reconnect — is logged as it happens and recovers on its own. If you see lag warnings on a real event, we would like to know: contact us with the log from a run with -v.