The most dangerous state of an integration is not when it crashes, but when it quietly stops working. The screen shows no errors, no one receives a warning, and everyone assumes data is flowing.
The problem usually surfaces days later during a reconciliation. By that point, the accumulated discrepancy has grown, and finding which records are missing requires serious effort.
In this article, we covered the error queue structure, retry logic, alert rules, and root cause analysis.
Table of Contents
Why are silent errors dangerous?
A visible error is addressed immediately, and its impact remains limited. A silent error, however, continues to grow until it is noticed.
Every untransferred record widens the gap between the two systems slightly more. A week later, this gap can reach hundreds of records.
Worse still, decisions are made using incorrect data throughout this period. Orders are taken with missing stock information, or shipments are made with missing current balances.
The cost of correction also increases over time; finding which records are missing becomes increasingly difficult.
Therefore, monitoring is not an optional add-on to integration, but a mandatory part of it.
Error types
Integration errors fall into a few groups, and each group requires a different intervention.
Connection errors are temporary and usually resolve themselves. For these, retrying is a sufficient solution.
Data errors, on the other hand, require human intervention. A card code with no matching counterpart will not fix itself.
Authorization errors are another distinct group; when access credentials change, the entire transfer halts.
Rule errors, on the other hand, arise from gaps in mapping definitions and cannot be resolved without fixing the logic.
Error queue
A queue structure is necessary to ensure failed records are not lost. This structure is the safety net of the integration.
The error reason for each record waiting in the queue must be stored. A generic failed tag does not help with fixing.
The record itself must also be saved, so it can be resent after correction.
Queue length is a metric that must be monitored. A constantly growing queue means an unresolved problem.
The queue must also be cleared regularly; accumulated old records mask the real issue.
Retry logic
Automatic retries for transient errors resolve most issues without human intervention.
It is recommended to gradually increase the interval between retries. Retrying every second unnecessarily stresses the target system.
The number of retries must be limited; infinite retries hide a real error forever.
Records that reach the limit should be moved to a list awaiting human intervention and trigger an alert.
For data errors, however, retries are meaningless; these records should go directly to the correction list.
Alerting rules
Properly setting up alerts determines whether monitoring will be effective or not.
Sending an alert for every single error quickly leads to all alerts being ignored. This is the most common failure mode of monitoring.
The correct approach is to generate alerts based on threshold values. Errors exceeding a certain count or downtime of a specific duration trigger alerts.
For critical flows, even a single error may require an alert; invoice transfer is an example of this.
It is also essential that the alert reaches the right person; notification management article.
Monitoring dashboard
A good monitoring dashboard shows the status of the integration at a glance and also allows drilling down into details.
The last successful sync time is the most practical indicator; this single piece of information instantly reveals most issues.
The number of pending records should also be visible. Normally close to zero, this number rises rapidly during an issue.
Grouping the error list by type facilitates prioritization. A single long list is unreadable.
It is also important for the dashboard to be in a place checked daily; a screen sitting on a separate page is forgotten.
Reconciliation checks
Even if the error list is empty, data consistency must be verified separately. Some issues occur without generating errors.
Record count comparison is the simplest check and can be done daily. The document count on both sides must match.
Amount totals should also be compared; a matching count with mismatched amounts indicates a field mapping error.
Balance reconciliation, on the other hand, is done periodically and reveals accumulated discrepancies.
When a discrepancy is found, its source must be investigated; making manual corrections and moving on causes the issue to recur.
Root cause analysis
Fixing errors one by one is necessary, but not sufficient. If the same error keeps repeating, the root cause must be sought.
Categorically recording error causes makes this analysis possible. Free-text descriptions cannot be grouped.
When the three most common causes are examined, a structural deficiency usually emerges.
Missing card definitions are a typical example of this; the solution is to add a control to the card creation process.
When the root cause is eliminated, the number of errors drops permanently, and the monitoring burden also decreases.
Ownership and routine
For monitoring to work, it must have an owner. A screen everyone is responsible for is looked at by no one.
A daily check routine should be established; a list checked first thing in the morning catches issues early.
How long the check will take must also be realistic; a routine of just a few minutes is sustainable.
Defining a backup owner prevents the tracking from breaking during leaves and handovers.
Who unresolved issues will be escalated to must also be determined from the start.
Points to consider
Generating excessive alerts is the most common cause of monitoring failure; ignored alerts also hide real issues.
Never cleaning the error list produces the same result; old records make new issues invisible.
Looking only at errors and not performing reconciliations leads to missing silent drifts.
Making individual corrections without looking at the root cause makes you repeat the same work forever.
Monitoring setup should be handled as an inseparable part of the sync plan; ERP sync article.
Frequently asked questions
How often should the error list be checked?
Daily check is sufficient for most businesses; instant notifications can be set up via alerts in critical flows.
Can failed records be fixed automatically?
Automatic retry is sufficient for temporary errors; data errors require human intervention.
Can reconciliation be automated?
Count and amount comparisons can be generated automatically; when a discrepancy is found, review remains with humans.
Is data lost during an outage?
If a queue structure is set up, it is not lost; on-prem agent article.
The real risk in integration is not the crashing system, but the flow that silently stops working. Without a monitoring setup, this goes unnoticed for days.
Put the last successful sync time in a place you look at every day; a single indicator instantly reveals most issues.
Tie alerts to threshold values; a setup that sends notifications for every error is quickly completely ignored.
Do not neglect reconciliation checks either; some deviations occur without producing any errors.
Get to the root cause of recurring errors; making individual corrections just makes you repeat the same work forever.
By talking to the EQLEM team you can plan your monitoring setup.