Quick Summary
Azure Logic Apps polling integrations can silently lose records even when every run shows Succeeded. The root cause is almost always a mismatch between trigger cadence and lookback window. ScriptsHub Technologies resolved this for a B2B SaaS client whose customer-onboarding pipeline was dropping leads – eliminated within a month via a three-layer defense.
Why Azure Logic Apps Polling Pipelines Silently Lose Records
Earlier this year, a mid-market B2B SaaS company brought ScriptsHub Technologies in to investigate a recurring complaint involving its Azure Logic Apps Polling integration: lead records that marketing could see in the source CRM weren’t reaching the destination data warehouse. The pipeline moved roughly 50,000 records per day, and operations opened five or six “missing record” tickets each week. The engagement ran for four weeks.
The pipeline was straightforward on paper. A Logic App configured with a Recurrence trigger fired every 30 minutes against the client’s operational CRM, queried for records modified since the last successful run, and pushed the result to a Snowflake warehouse.
It wasn’t. For months, teams dismissed the issue as user error because volumes were low. Yet the pattern remained consistent: the tickets pointed to no single failure, and a systemic flaw drove the problem. By the time we joined the engagement, marketing had made it a habit to cross-check every record against the source.
What Causes Silent Record Loss in Azure Logic Apps Polling?
Before changing a single trigger setting, our team spent a week reconstructing the failure. We pulled six months of Logic App run history, joined it against source audit logs, and compared the modification timestamp of every missing record against the watermarks (the cutoff timestamps each polling run records as “everything before this is fully processed”) of the runs that should have caught it.
The pattern was unmistakable. Every missing record had a modification timestamp that fell inside a window the Logic App had already processed. The trigger had advanced its watermark past those timestamps, but the rows themselves had never been returned by the original query. They were ghosts – visible to anyone querying the source directly, invisible to the polling job that had supposedly looked at that exact range.
Most developers design polling jobs assuming three timestamps are equivalent: when a row is logically changed, when that change becomes queryable, and when the polling job runs. In production, those three almost never align – and the gaps are where ghosts appear.
Cadence vs Lookback Window: The Configuration That Drops Records
Two parameters control every polling integration, and the difference between them matters more than most teams realize.
Cadence is how often the trigger fires – driven by freshness requirements and source-system load.
Lookback window is how far back each run looks – the WHERE clause on your query: modified_date >= @watermark. Lookback is driven by visibility lag of your source system.
The mistake most teams make is implicitly setting cadence equal to lookback. Run every 30 minutes, look back 30 minutes. It feels symmetric. It also assumes zero visibility lag, which is never true in any production database.
What Causes Visibility Lag in Polling Pipelines?
Three things create visibility lag in any production database, and all three are universal.
Commit timing. A row’s modification timestamp is stamped when the application updates the row, but the row only becomes visible to other queries after the transaction commits. A long-running transaction can leave a 30-second, 5-minute, or longer gap between logical modification time and query-visible time.
Replica or snapshot lag. If your polling job reads from a read replica or change-data-capture feed, there’s added delay between commit and replica visibility. Most teams underestimate this by an order of magnitude.
Index propagation. Even on a single-instance database, a newly committed row may not appear in a non-clustered index immediately, depending on storage engine and isolation level.
If your lookback window is shorter than your worst-case visibility lag, you will lose records – definitively, on a predictable schedule.
How Do You Prevent Azure Logic Apps Polling From Dropping Records?
A production-grade fix doesn’t rely on a single mechanism. We designed three layers, each addressing a distinct failure mode the others can’t cover.
Layer 1 – The Overlap Window (Visibility-Lag Buffer). Every poll deliberately re-examines the previous several windows. The cost is modest – a few hundred extra rows per run – and the integration now tolerates visibility lag up to the size of the overlap. For our client, this layer alone caught the bulk of the ghost records on subsequent polling cycles within the first week.
Layer 2 – The Idempotent Upsert (Deduplication Guard). Because the overlap window guarantees the same record gets processed multiple times, the destination must handle that gracefully. We replaced every INSERT with a MERGE keyed on the combination of source-system identifier and source-side primary key. The final state is identical regardless of how many times the record arrives.
Why this works:Idempotent destinations turn the overlap window from a liability into a safety net. You can size the overlap as wide as you need without worrying about duplicate downstream effects.
Layer 3 – The Reconciliation Sweep (Long-Tail Safety Net). A separate job runs daily during off-peak hours and reconciles a much wider window – typically 24 to 48 hours back. It queries the source for all records modified in that window, queries the destination for the same set, and any records present upstream but missing downstream get re-pushed through the same upsert.
When to use which: The overlap window handles the bulk of visibility lag. The reconciliation sweep handles the long tail and acts as a continuous audit. Together they make data completeness a function of the wider 24-hour window, not the cadence.
How Wide Should the Lookback Window Be in Azure Logic Apps Polling?
The fix begins with a single mental shift: cadence and lookback are independent parameters and should be configured independently. The right approach is to make the lookback window deliberately wider than the cadence – wide enough to absorb visibility lag and any reasonable processing delay, with margin to spare.
Cadence is 30 minutes. Lookback is 120 minutes. Every run re-examines the previous three windows in addition to the newest one. Records whose visibility was delayed by 5, 30, or 90 minutes still get caught – just on a later run.
Why this works: The probability that a record’s visibility lag exceeds the lookback drops sharply as the window grows. At 30 minutes of lookback, you’re betting no transaction runs longer than 30 minutes – a bet that loses regularly. At 120 minutes, the bet is against transactions over two hours, far safer on a healthy OLTP system.
When to widen further: If your source database runs periodic batch jobs that hold long transactions – overnight ETL, end-of-month closes, schema migrations – size the lookback to comfortably exceed those windows. We’ve configured lookbacks as wide as 6 hours for systems with regular long-running batch activity.
Suspect cadence-vs-lookback drift in your own pipeline? Our integration engineers can audit your Logic Apps polling configuration and pinpoint the exact gap producing your missing-record tickets. Talk to ScriptsHub →
Operating the Three-Layer Defense in Production
Building the defense is half the work; proving it keeps working six months later is the other half. Three observability hooks make the architecture auditable on day 180 as well as day one.
Reconciliation gap as a metric. The Layer 3 sweep emits a count of records re-pushed each day. Publish that count to Application Insights as a custom metric. Alert when the daily gap exceeds a threshold – start at the 95th percentile of historical gaps and tighten over time.
Lookback effectiveness ratio. Track the percentage of records caught by the overlap window versus those caught by the reconciliation sweep. If reconciliation is catching more than 5% of records, the lookback is too narrow and visibility lag is exceeding the buffer.
Runs-succeeded vs. records-delivered divergence. Build an Application Insights workbook comparing trigger success counts against destination row counts over a rolling 24-hour window. Any sustained divergence is a leading indicator – it precedes missing-record tickets by 24 to 72 hours.
Cost profile. The overlap window adds roughly 5–10% to per-run row processing on the source database. The reconciliation sweep adds one workflow execution per day plus one source-database scan. For most enterprise pipelines, this is a rounding error against the cost of a single missed record surfacing in a compliance audit or a customer complaint.
Red Flags Your Logic App Is Quietly Losing Data
Look for these symptoms – they’re the telltale signs of cadence-vs-lookback drift even when run history shows nothing wrong. Treat each row as an if-then diagnostic rule.
Mistakes to Avoid in Azure Logic Apps Polling Pipelines
Four mistakes recur across nearly every broken polling integration we audit.
1. Setting lookback equal to cadence. The most common configuration error in any polling integration. It assumes zero visibility lag and guarantees silent record loss on any source that occasionally commits late. Size lookback at 3-4× cadence as a starting point.
2. Using non-deterministic upsert keys. Pipelines that use auto-generated GUIDs as the dedup key produce a new GUID on every reprocessing – duplicates everywhere. The key must derive from the source record itself: a composite of source-system identifier and source primary key. Never the destination’s auto-ID, never a runtime GUID.
3. Storing watermarks in local time. Pipelines built on local-time watermarks lose an hour of records every spring DST and double-process an hour every fall. Use UTC at every layer – convert to local only at presentation.
4. Deferring the reconciliation sweep to “later.” Teams routinely deploy the overlap window first and treat reconciliation as a phase-two enhancement. This guarantees weeks of unmeasurable long-tail loss between phases. Every integration we’ve built since starts with the reconciliation layer as non-negotiable from day one.
The Results: From Weekly Tickets to Zero Missing Records
Recovery was visible within days, but the full benefits took the four-week engagement to materialize.
- By week one, the overlap window alone – Layer 1 – had eliminated most of the silent record loss; the ghost records were being caught on subsequent polling cycles.
- By week two, the idempotent upsert (Layer 2) ensured that the deliberate reprocessing produced no downstream duplicates.
- By week three, the reconciliation sweep (Layer 3) was running daily and the lookback-effectiveness ratio settled at 0.3% – meaning the overlap window was catching 99.7% of records and reconciliation was a true safety net rather than a primary recovery mechanism.
The most significant change wasn’t quantitative. After three weeks of zero reconciliation gaps, marketing’s habit of cross-checking every lead against the source CRM stopped. The destination warehouse became operationally trusted as authoritative – which had been the original goal of the integration when it was built two years earlier.
The pattern is consistent across industries – financial services (batches committing after the export cutoff), healthcare (clinical feeds dropping records during nightly maintenance), B2B SaaS (events lost behind replica lag). Any Logic Apps polling integration that treats cadence and lookback as the same parameter will silently lose records.
Key Takeaways
Bottom line: Azure Logic Apps polling pipelines silently lose records when trigger cadence equals lookback window. The fix is a three-layer defense – overlap window, idempotent upsert, daily reconciliation sweep – implemented in this order, with observability hooks layered on top.
- Root cause: Visibility lag – the gap between when a row is logically modified and when it becomes queryable – caused by commit timing, replica lag, and index propagation.
- Diagnostic rule: If your Logic App shows 100% Succeeded runs but operations finds missing records, you have a cadence-vs-lookback mismatch.
- Configuration rule: Lookback window should be 3-4× the trigger cadence as a minimum starting point. Widen further if the source runs long-running transactions.
- Idempotency rule: Replace every INSERT with MERGE keyed on
(source_system, source_record_id). Never use auto-generated GUIDs as dedup keys. - Watermark rule: Always store watermarks in UTC. Local time causes DST-induced losses every spring and fall.
- Observability rule: Publish reconciliation-gap counts and lookback-effectiveness ratio to Application Insights. Alert before tickets arrive.
- Recovery expectation: 80-95% of missing-record tickets eliminated within one week of widening the lookback. Reconciliation sweep closes the long tail within one month.
Is Your Integration Quietly Losing Data?
ScriptsHub Technologies has delivered integration resilience engineering for B2B SaaS, financial services, and healthcare clients across the US, UK, and India. Each engagement is tailored – cadences, lookback sizing, reconciliation strategies, observability hooks, and idempotency keys all calibrated to the operational reality of each client.
If your team is fielding sporadic missing-record tickets from a Logic Apps polling pipeline that don’t trace to a single failure, or if operations is spending hours each week reconciling source and destination, these are exactly the symptoms we resolve.
We offer a complimentary Integration Health Assessment – a structured diagnostic that maps your current polling architecture, identifies cadence-versus-lookback mismatches, surfaces visibility-lag risk, and provides a prioritized hardening roadmap.
→ Request your Integration Health Assessment: connect with us at info@scriptshub.net or visit scriptshub.net.
Azure Logic Apps Polling: FAQ
Why does my Azure Logic App show Succeeded but still miss records?
Run history reports trigger success, not data completeness. If a row’s transaction commits after the polling query runs but before its timestamp falls outside the lookback window, the row is permanently missed.
What’s the difference between Logic Apps cadence and lookback?
Cadence is how often the trigger fires. Lookback is how far back the query looks. They’re independent parameters, and treating them as equal causes silent record loss.
How wide should my Logic Apps polling lookback window be?
Start with 3-4× your cadence. If cadence is 15 minutes, set lookback to 60 minutes. Widen further if your source database runs long batch transactions or maintenance windows.
Will overlapping windows create duplicate records downstream?
Not if the destination uses idempotent upserts (MERGE) keyed on a deterministic composite of source-system and source primary key. The same record can be processed multiple times with no duplicate effect.
When should I use Azure Logic Apps polling instead of Event Grid?
Use polling when your source can’t emit events – legacy databases, third-party APIs without webhooks. Use Event Grid when push notifications are available; it eliminates the cadence-vs-lookback problem entirely by firing on actual changes.
Should I use Azure Logic Apps or Azure Functions for polling integrations?
Logic Apps is faster to build and easier to maintain. Azure Functions is cheaper at high frequencies with finer control. For most enterprise integrations, Logic Apps wins on maintainability – choose Functions only if cost or sub-minute polling matters.