These are the failures that worry me most in trading systems: something stops happening, but nothing reports an error. The usual SRE tools still help, including the four golden signals, error budgets, and symptom-based alerts. The difficulty is deciding where aggregate metrics are enough. A low error rate offers little reassurance when a failed cancellation leaves a position open or a missed fill confirmation leaves the system with an incorrect view of exposure.

The examples below come from a production order management system (OMS) audit. They cover alert priorities, gaps in monitoring, SLOs, and how to organize dashboards for the people using them.

The Four Golden Signals, Applied to an OMS

The Google SRE book is a useful starting point here. For an OMS, the first job is choosing measurements that describe the order path.

Latency deserves particular attention. An average can look healthy while P99 reaches 2 seconds. With 10,000 orders a day, the slowest 1% still amounts to about 100 orders, each potentially needing investigation. Set alerts on P99 and P99.9, and emit latency as a histogram so you can calculate those percentiles.

Traffic is a leading signal. A sudden drop in order flow — even if error rates are flat — can mean a feed is stale or a circuit breaker silently tripped. Volume spikes above statistical norms are early warning for either legitimate load growth or a malfunctioning client flooding the system.

Saturation in a trading context often shows up in event-driven infrastructure before it surfaces in CPU or memory. Kafka consumer lag growing on the order-processing topic is a more reliable predictor of trading-path degradation than host-level CPU at 80%.

Event-Based vs. Metric-Based Alerting

Error rates, latency percentiles, and resource utilization tell you how the system is behaving overall. They can still miss a problem with an individual order.

An error rate of 0.5% can be within SLO even while thousands of requests are failing. That may be an acceptable trade-off for some operations. It needs a closer look on the trading path, where the cost depends on which request failed.

A single Cancel Reject processing failure can leave a position open without the trader knowing. Waiting for an aggregate error threshold would delay the response. An event-based alert can page as soon as that processing failure occurs.

For example, these alerts answer different questions:

Resource: DatabaseConnectionPoolExhausted
Impact:   OrderAcknowledgementLatencyP99 > 500ms

Aggregate: ErrorRateExceeds0.5%
Event:     CancelRejectReceivedWithNoMatchingCancellation

The first pair separates a resource problem from its effect on orders. A connection pool alert can help with diagnosis, while an acknowledgement latency alert catches delays from other causes too. The second pair separates an aggregate threshold from a specific order event. Both distinctions matter when deciding what should page someone.

Review each critical path for events that need an immediate response: order placement, cancellation, fill receipt, position reconciliation, and gateway connectivity. Keep metric alerts alongside them to catch changes in latency, throughput, and queue depth.

Alert Priorities in a Trading System

A common problem is that every alert has the same priority. A FIX gateway failure sits at the same level as a DNS resolution error. Business alerts may go to a shared channel with no severity or escalation path, leaving whoever reads the message to work out how urgent it is.

Classify alerts by their business impact. Three tiers cover most cases:

• P1 — trading halted or regulatory exposure. Requires immediate response, wakes up on-call. Examples: exchange gateway down, order cancellation confirmation failure, FIX session lost.
• P2 — degraded trading capability, actionable within minutes. Examples: elevated order rejection rate, ack timeout, service error rate elevated, imminent certificate expiry.
• P3 — non-urgent, informational. Examples: background sync failures, pod restarts on non-critical services, DNS warnings, certificate expiry with weeks of lead time.

A classification table for a typical OMS:

The inverse problem is noise. In most production trading systems, some alerts fire almost daily and are unactionable: background sync failures that resolve themselves, timeouts against external dependencies with automatic retry, gateway health checks that fire during normal logon sequences. None require human intervention, but all page engineers.

Traders and operations staff often share these channels with engineers. If most messages resolve themselves, people learn to skim them. That makes the next real incident easier to miss.

Missing Alerts

Changing priorities only helps with alerts that already exist. The audit also found several gaps:

Tail latency. P99 latency alerts were entirely absent. Order-entry end-to-end P99 latency — the metric most predictive of trader experience — had no alert threshold. The additions: order-entry end-to-end P99 > 2s fires P1; exchange adapter FIX round-trip P99 > 1s fires P2. Histograms, not gauges. Alert on the percentile, not the mean.

Order lifecycle state traps. An order stuck in NEW for more than 30 seconds is not a failed order — no error is thrown. It is simply stuck: the trader sees no fill, the system sees no failure, and the state will never self-resolve. PENDING_RESERVATION sitting for more than 5 minutes has the same character. These are silent failures produced by state machine edge cases, not error conditions. They need independent monitoring.

Dead man’s switches. No FIX heartbeat received from the exchange gateway in 2× the expected interval is a P1 condition. Nothing necessarily logs an error when messages stop arriving, so this needs a watchdog that checks for their absence.

absent(fix_heartbeat_received_total[2m]) → page

Without this, the failure mode is invisible until a human checks manually.

Certificate expiry. A single alert close to expiry leaves little time to act. Escalate as the deadline approaches: < 30 days (P3, informational), < 7 days (P2, action needed this week), < 24 hours (P1, imminent outage). This gives the team time to schedule a renewal before it becomes an incident.

One service in the audited system had no alerts at all: no error rates, latency, or connectivity checks. A service-by-service review helps catch omissions like this.

SLOs, Error Budgets, and Burn Rates

Priority labels are only meaningful if they are grounded in explicit reliability targets. Without SLOs, “P1” is a color. With them, P1 means: this service has 194 minutes of allowed downtime over the next 90 days, and the current burn rate will exhaust that budget in 11 days.

A practical tier structure for a trading platform:

allowed_downtime = (1 − SLA_target) × window_hours
burn_rate       = (1 − observed_SLI) / (1 − SLO_target)

Multi-window burn rate alerting catches two different failure modes. A short evaluation window (1 hour + 5 minutes) catches fast burns — a complete outage that would exhaust the budget in days. A long window (6 hours + 30 minutes) catches slow burns — a partial degradation that quietly drains the budget over weeks without ever crossing an hourly threshold. Neither window alone is sufficient.

For internal platform services without 24/7 trading requirements, scoping SLOs to business hours is a legitimate design decision. An availability SLO measured against 09:00–17:00 trading hours is a more honest contract than a 24×7 SLO that includes weekend maintenance windows. The key is labeling it explicitly.

Concrete targets from the audited OMS: HTTP success rate ≥ 99.5–99.9% when measuring 5xx failures, P99 response time < 200–500ms depending on service criticality, Kafka consumer lag < 1 minute, message processing error rate < 0.1%.

Keep the individual order alerts even after adding SLOs. A burn rate alert tells you the budget is at risk; a Cancel Reject processing alert tells you which order needs attention now.

Dashboard Structure

Traders and engineers come to dashboards with different questions. A trader needs to know whether orders are getting through and positions are current. An engineer also needs the Kubernetes and JVM details to investigate a failure. Separate dashboards let each group find what they need without sorting through unrelated panels.

The business dashboard needs a traffic-light summary at the top: is the system taking orders, is fill confirmation working, is the data feed live. If a trader has to parse a graph to answer that question, the dashboard has failed.

The Four Golden Signals tab is the engineer’s first stop during an incident. One row per critical service, consistent layout: latency by percentile, error rate, saturation signal. This is the triage surface — it should answer “where is the degradation” in under 30 seconds.

The infrastructure tab is diagnostic, not primary. Pod restarts, consumer lag per topic, database CPU and storage, network I/O — these are cause-level signals. They explain why a Golden Signals indicator is red. They should not be the thing that pages engineers.

Include orderId, traceId, and clientOrderId in order-related logs, and link the dashboards to those logs and traces. During an incident, being able to go from a rejection spike to the affected orders saves a lot of manual searching.

Keeping Alerts Useful

The audit left a fairly practical list of work: assign priorities, add the missing latency and heartbeat checks, and remove pages that nobody could act on. Counting alerts would have told us very little about any of those problems.

Start with the order path when reviewing coverage. Can an order get stuck without anyone noticing? Can position data go stale while every service still looks healthy? Infrastructure dashboards help investigate those failures, but they won’t necessarily detect them.

For each alert, ask what the on-call engineer should do when it fires. Put the business impact and a runbook link in the message, and agree on severity ahead of time. If an alert keeps firing without requiring action, fix it while people still pay attention to the channel.