← EXCOIN / INSIGHTS

Market data quality: what an application must detect and show

A fast price chart does not prove correct data. The system needs source, age, ordering and rules for outages and corrected values.

A price without origin and time is incomplete

An application may display a number to several decimal places, but users cannot tell whether it is current, delayed or corrected. Every data stream has a provider, instrument, value type, event time and receipt time. Those times can differ considerably. A product used for a trade decision must specify which value is shown and which is used for an actual instruction. When the feed is unavailable, an old price must not look fresh. Formatting precision is not the same as content quality. Define the meaning and contractual limits of each source before building a chart. Show the user when the data are not suitable for the intended action.

Record whether a value is a bid, ask, last trade, indicative quote or aggregation. Two numbers for the same instrument can differ because they represent different events. Users and engineers need consistent terms. Sources may have different trading hours, currencies, time zones and correction policies. Before combining feeds, establish a contract for instrument identifier, message type, time, currency, size, quality and acceptable delay. If a field lacks business meaning, making its technical name consistent may still produce misleading data. Keep the original provider identifier so a disputed value can be investigated.

A stream needs ordering and recovery

In a live transmission messages may be lost, duplicated or received out of order. FIX Trading Community describes snapshots and incremental updates for market data, including conventions for maintaining an order book. Each actual feed has its provider's own specification. Check whether messages have sequence numbers, how a gap is detected and whether another snapshot can be requested. Without that, an application may keep drawing a seemingly smooth chart or book while missing an important step. Recovery after interruption is part of correctness, not only network connectivity. A connection that stays open can still carry an incomplete view.

When combining a snapshot and later changes, specify precisely when updates begin to apply. If an event disappears while the snapshot loads, the final state will be wrong. Test interruption in peak traffic and reconnection. Users should see when the state is incomplete. Some screens may display the last known value with a clear warning; others must stop an action dependent on complete data. The product owner should decide according to risk, rather than leaving it to the chart component's author. Record the time and reason when the system switched from valid to degraded. Reconstruct the state after a deliberate sequence gap in a test.

Timestamps separate three kinds of delay

Time between a market event and the provider sending it, between sending and your application receiving it, and between receipt and rendering all contribute to apparent latency. One general latency number hides where a problem begins. Record event time, ingest time and perhaps display time under a clear convention. Clock synchronization has limits; when comparing systems watch their time offset too. If a feed provides only receipt time, do not promise accuracy for the original trade time. For history distinguish trading session date from actual timestamp, particularly around midnight and time-zone changes. Store enough detail to explain why a late message appeared at a different point in a chart.

For users, the age of the last verified value may matter more than average network latency. Show it where a decision is made. Once an approved limit is reached, change the screen state or availability of an action. A fast feed may suffer from ordering rather than delay. Break metrics down by instrument, source and message type. Find out whether a peak delays all values or only a subset. That directs repair toward the cause rather than a blind purchase of larger hardware. State the observation window and percentile when making a performance claim.

Duplicates, gaps and outliers need different responses

A duplicate event may leave the last price unchanged while doubling volume or trade count. A missing interval can break a chart and remain hidden after transmission resumes. A sudden jump may be the real market or a bad input. Do not repair all three with the same rule. Define checks for each category: message identifier, sequence, instrument existence, plausible ranges and comparison with another source when licensed and available. A deviation needs status and an owner so it is not ignored merely because the entire feed was not stopped. Keep evidence of why a particular value was rejected or accepted.

When correcting historical data, preserve the original value and correction time where needed to explain a user decision. A chart may later show accurate history, while the user saw something different when acting. Those views should not be conflated. For a fallback source define how different methodology and licence terms are treated. Two providers' prices need not be directly interchangeable. Source selection should be versioned and auditable, especially when values feed calculations or alerts. A correction should trigger a review of any derived report that relied on the earlier value.

Balance latency and completeness by task

A live display may favor fast updates with an uncertainty indicator. An accounting or analytical result may wait for more complete data. Both need not use an identical path. Distinguish live display, confirmed historical record and perhaps a controlled reporting batch. For each define when it is considered complete and whether later corrections are accepted. If figures differ, the interface should explain why. Accuracy and freshness are separate quality dimensions; the use case determines their balance. Users should not mistake an indicative live view for the confirmed record used for a later report.

At high volume set queue limits and a plan to catch up. When the application cannot handle the stream, it must not silently drop messages while claiming to be current. Sampling can be appropriate for a less important visualization if approved; not for an order book or commercial state calculation without a separate decision. Measure delay percentiles and age of the oldest unprocessed event. After recovery decide whether to fill history first or take a new snapshot and mark earlier gaps. Put the choice in the runbook. Test the boundary where the system deliberately degrades service to protect correctness.

Monitor quality, not merely availability

A green provider connection does not prove data are right. Monitor age of the last message, sequence gaps, decode errors, corrections, difference from expected volume and anomalies by instrument. OpenTelemetry describes metrics as aggregated signals of system behavior; use them for real quality conditions. Every alert needs a threshold, time window and person who decides what to do. Different rules may apply during and outside market hours. Do not wake on-call staff for silence that is normal for the market. Include the impact on downstream features so a data alert can be prioritized correctly.

The operational view should show problem source, scope and effect on user features. During an outage some parts of a platform can remain active while others pause. Record when quality degraded, what users saw and when it recovered. After an incident compare monitoring with reality: did it detect the issue before customers and lead to a safe action? If not, add a specific test or measure. A generic feed down alert does not cover a feed that runs but sends a wrong or stale value for one important instrument. Let support identify that instrument without reading raw network packets.

Test streams, not only static API responses

Create example sequences for a correct stream, duplicate, gap, delayed snapshot, corrected value and altered order. For each define expected internal state, screen and alert. Test loss of connection at peak volume and reconnection. With several message types, check unknown fields or versions. Test data must respect source licence restrictions. If a live feed cannot legally be stored for replay, make approved synthetic examples with the same failure characteristics. Keep their expected outcomes in a regression suite so later performance changes do not break correctness.

Ask someone outside implementation to identify from the operating view that a displayed value is no longer suitable for a decision. If they cannot, the warning is not clear enough. Before launch verify fallback rules and what happens when both sources are unavailable. After release watch real anomalies and expand the regression set. EXCOIN can design a data path, but latency and correctness cannot be promised without the specific source, licence and product. A strong result is a system that can show value origin and age, detect an incomplete stream and acknowledge uncertainty safely.

Sources

Discuss a project