EXCOIN / DATA INTEGRATIONS

Data and API integrations that remain understandable after failure

Connecting two systems is not complete when the first request succeeds. Data meaning, contract ownership, retry behavior and a route to repair disagreements must be clear.

01 / ENGINEERINGCS · EN · ES
01 / 08

Integration begins with the meaning of data

Two APIs can use the same word for different things. A “customer” may be a person, contracting party or account; “active” may mean approved, paid or merely not disabled. A technical transformation of format does not necessarily preserve business meaning. The first step is a shared vocabulary and state map: where a value comes from, who changes it, when it becomes valid and how staleness can be recognized.

The map should include the person who resolves exceptions. Each important field needs a source of truth, synchronization rule and procedure for conflicts. Only then should the interface be chosen. Sometimes a synchronous query is enough; other cases suit events or batches. The choice depends on tolerated delay, partner availability, volume and cost of repair. Architecture that leaves these conditions unstated often transfers manual work to operations.

02 / 08

A contract is more than a list of fields

An interface should describe syntax and also value meaning, permitted state changes, error responses, versions, time zones, identifiers and limits. For a synchronous API, decide how long to wait, when calls may be repeated and what timeout means. For events, establish whether duplicates, reordered delivery or schema changes may occur. The owners of connected systems must agree on responsibility for such changes.

OpenAPI provides a machine-readable description for HTTP interfaces; AsyncAPI supports descriptions of message and event interfaces. A specification alone cannot explain every business rule or guarantee that a partner behaves as expected. Worked examples, contract tests and a record of decisions about field meaning are useful complements. When an API changes, the team should know which consumers are affected and how to pass safely through a transition period.

03 / 08

Synchronous, asynchronous and batch flows

A synchronous call offers a quick answer but brings the other system's downtime into the user's path. An asynchronous message decouples timing while adding queue monitoring, retries and a need to confirm the final outcome. A batch can be simple for large transfers when delayed updates are acceptable. One product may use all three approaches for different tasks. Users and operators need to understand the freshness and certainty of the state they see.

For example, a client portal may need an immediate permission check while forwarding a later change to reporting. If reporting lags, it must not appear current. A historical import may be restarted, but its result should be reproducible and free of duplicates. Choose the mechanism according to impact rather than a framework preference. The time window, processing state and repair path belong in the brief.

04 / 08

Idempotency, ordering and unknown results

When a caller receives no response, it cannot automatically know whether the other side processed the request. A retry may be necessary, but the receiver must recognize the same operation. The idempotent receiver pattern describes identifying repeated requests and returning a saved result. A business system must additionally define what counts as the same operation, how long an identifier is kept and what happens when a repeated request changes its content.

Event ordering may matter for account state, but infrastructure does not always guarantee it across all sources. A consumer needs rules for late messages, duplicate versions and gaps in a sequence. Sometimes a fresh authoritative state is safer than blindly replaying old events. Test these cases with a concrete domain example. Operators must tell whether a flow is merely delayed or needs intervention.

05 / 08

Mapping and data quality

A mapping table explains source and target fields, transformation, owner and the treatment of invalid values. Knowing that one JSON object becomes another is insufficient. Decide what happens when an identifier is missing, currency format differs, a time zone is unexpected or a historical record has no equivalent in the new system. Some entries should be rejected for repair; others can be normalized safely.

Measure quality across the flow: completeness, freshness, rejected records and share of manual corrections. A baseline allows the team to judge whether an integration improves the process. For migration, test a sample of real anonymized data and compare the outcome with its source. Quietly inventing default values can create a lasting discrepancy. Every automatic correction needs an explained rule and a way to inspect its effect.

06 / 08

Observability and operating repair

Integration health is not identical to endpoint availability. Track the age of the last successful synchronization, queue length, errors, retry count and discrepancies against the source of truth. A stable correlation identifier helps trace one case without spreading sensitive information through logs. Alarms need priority and owners. Significant errors need an instruction for what to check and whether reprocessing is safe.

During an incident, distinguish fixing technical delivery from correcting business state. Rerunning an entire import without rules may worsen the situation. An operating interface might offer a safe retry for a record, quarantine for invalid input and a record of human decisions. Not every flow needs all these features at full scale; select them by the impact and frequency of exceptions. The essential point is to include repair in the original design.

07 / 08

A first verifiable delivery

Start with one important flow and its actual source and destination. Define contract, mapping, identifier, behavior during downtime and a measure of success. The prototype should process a valid case, invalid input, repeated request and timeout. The result is more than moving data: it includes an error view and a procedure for discovering what happened. This stage reveals real API limits and data quality before a wider implementation.

Before estimating more flows, obtain partner documentation, test access, an anonymized data sample and people authorized to decide the meaning of fields. Record tolerated delay and operating ownership for each connection. The greatest risk may be technology, an unclear process or the provider contract. A delivery plan should separate these uncertainties from known work and show what the client approves after the first evidence is available.

08 / 08

Questions for long-term maintenance

A good integration should make sense to someone changing it after a year of operation. They need to know why the flow exists, where the source of truth sits, how a failure is recognized and how to recover after a bad change. Maintainability rests on a contract, tests of consequential exceptions and a short operating guide, not the number of components. If these answers are missing, each additional system multiplies the problem.

Before a change, review known consumers and check whether they use a value differently from what the technical specification suggests. A transition may need both contract versions at once. Retiring the old one should have an owner, date and evidence that no important flow still relies on it. That discipline reduces surprises during ordinary product evolution.

  • Who owns the meaning and change of each important value?
  • When is delayed state acceptable and how is it shown to users?
  • What happens to repeated, missing or late records?
  • How is semantic equivalence checked after migration?
  • Who responds to a discrepancy and repairs it without losing history?
  • How does a contract change reach every known consumer?
  • Is there a quality measure that leads to a specific decision?

Sources

Discuss a project