[Piano Analytics] Delay in generating yesterday's data

Incident Report for Piano Software

Postmortem

Incident explanation

  • Due to inconsistencies in our delivery process, two services with incompatible versions were deployed to production during a full redelivery of all services, aimed at quickly resolving a security vulnerability
  • These services are part of our data ingestion pipeline, which became blocked. As all consolidation and aggregation jobs depend on this step, they could not be computed, delaying data availability.

‌

Action performed

  • Both services were rolled back to their last stable versions, which unblocked the process.

‌

Actions to come

  • Strengthen validation of this part of the pipeline during deliveries.
    Set up an on-call rotation for the D-1 data feed (details to be defined).
Posted Oct 30, 2026 - 11:15 UTC

Resolved

The issue has been resolved and the delay in data generation has now been caught up.
All scheduled exports have been sent.
Posted Oct 30, 2026 - 11:12 UTC

Update

We are continuing to monitor for any further issues.
Posted Oct 30, 2026 - 11:03 UTC

Monitoring

A fix has been implemented and data recovery is currently underway (already completed for around 90% of customers).
Posted Oct 30, 2026 - 09:51 UTC

Update

We are continuing to investigate this issue.
Posted Oct 30, 2026 - 09:26 UTC

Investigating

We are currently experiencing a delay in data generation for yesterday on Piano Analytics.
Daily scheduled exports may be still pending to be sent. Hourly and dataflow exports are not affected by the problem.

Our technical teams are actively working to resolve the issue.
Please excuse us for any inconvenience caused.
Posted Oct 30, 2026 - 09:25 UTC
This incident affected: Data Processing (Piano Analytics Data Processing) and Dashboards, Interfaces and Platform (Piano Analytics Exports).