Identify the source and right to use it
Architecture begins with the intended use: a human display, internal calculation, historical analysis or redistribution. The same price may carry different contractual conditions in each case. Record the provider, venue, feed type, instruments required and permitted display and retention. Engineers need these limits before designing caches and exports. Otherwise they may build a capability that the licence does not allow.
Separate contractual rights from the technical ability to retrieve data. A successful API connection is not permission to distribute its output to other customers. For history, define retention and correction rules. Rights depend on the actual contract and venue, so this page gives no general licensing conclusion. A clear source inventory lets business and legal owners check the chosen design at the right time.
Time is more than one number
An event may have a time at the venue, a provider send time, an infrastructure receive time and an application processing time. Combining them into one field makes it hard to see whether delay came from the source, network or your system. Databento's documentation distinguishes event, publisher and receive timestamps and notes that accuracy of the original venue timestamps can vary. The principle extends beyond their specific feed.
Keep the original timestamp during normalisation and add your own observation. Document time zone, precision and missing values. Cross-venue comparison requires knowing whether two sources measure the same instant. Historical analysis can infer false sequence from a wrong order. A live interface must show how old its information is. Time quality is a business property of data, not simply the type of a database column.
Normalise without losing meaning
Providers use different instrument identifiers, sizes, price units and event types. A normalisation layer helps an application work with a common model, but must not hide differences that change an outcome. Keep provenance and conversion rules for each field. Some attributes should remain venue-specific and be used only where needed. A universal structure with hundreds of empty fields can be worse than clearly versioned, specific types.
Build samples for ordinary updates, missing values, corrections to historical events and changes in instrument definition. A new mapping version should not silently change the meaning of stored history. Where possible retain a reference to the original message alongside the normalised record so a disputed calculation can be explained. If a provider changes schema, the team needs an alert and response path, not just another empty column.
Snapshots, increments and gaps
A live stream can begin with a full snapshot and continue with incremental changes. The FIX Trading Community describes full refresh and incremental market data messages. An application must know how to build a consistent view from them and what to do when an update is missing or out of order. Without that logic, a screen may look active while showing an invalid book or price.
Track sequence numbers, last-message age and connection state according to the source protocol. When a gap is detected, mark the output untrustworthy, obtain a new snapshot and resume the stream as the provider specifies. Do not silently continue calculations on incomplete state. Historical replay should apply the same ordering definition as the live path, or pilot and production outcomes may disagree.
Distribute and cache for actual use
One application may need every tick; another needs only the latest known price or an interval summary. Do not force the most demanding stream on every consumer. Define data products with age, completeness and calculation method. A cache needs a validity period and behaviour during source outages. An older value should be labelled as older rather than presented as live.
Storage follows volume, queries and licence terms. For historical work, consider separating raw input from derived aggregates so a calculation error can be repaired. On live screens, monitor concurrent consumers and the system's ability to catch up after delay. Measure compression and batching against the necessary latency; the lowest possible technical latency is not always a business goal.
Quality monitoring is part of the data product
Set expected frequency and relevant trading hours for every feed. Delay, outage, duplicate messages and unusual jumps are different problems with different responses. A dashboard should show age of the last valid message, gap count, processing rate and state of important consumers. Every alert needs an owner. Without one, a chart is decoration while people may act on stale information.
Check relationships between sources too. Two feeds may use different methods, so a difference is not by itself proof of a fault. When correcting a value, record where the new figure came from and whether historical outputs require recalculation. Incident explanation matters: when it began, who was affected, what should be checked again and how repetition can be limited.
Use a pilot to test the decisive risk
Pilot one provider, a limited instrument set and a specific consumer. Include normal trading, quiet periods, connection restart and corrected data. Compare the original message with what the application shows. If the product needs several venues, test their different timestamp and identifier semantics early. A load test on random synthetic numbers does not by itself prove that market data has been handled correctly.
This page does not claim EXCOIN has access to a paid feed, an exchange licence or client implementation results. A real design needs the provider's terms and the precise intended use. Useful early deliverables are a data contract, a freshness definition, a gap recovery test and quality measures. They show whether a simple integration is enough or a separate platform for several consumers is justified.
Include a provider or message-format change in the pilot. Verify who gets alerted, how untrustworthy values stop being published and when normal flow resumes. Consumers have different outage tolerance: an internal analysis may wait, while a live display must show data age immediately. Record the rule for each one. After an incident, replay the retained messages and compare whether a revised calculation yields the expected result. Without reproducibility, a repair remains an assumption rather than an established correction.