[Piano Analytics] Real-time data latency

Incident explanation:

• The latency was caused by an issue on our cloud data platform provider's side. An internal infrastructure configuration change introduced resource contention that affected workloads with high ingestion concurrency, leading to elevated response times and intermittent timeouts when loading files. The change has been reverted by the provider, and ingestion performance has fully returned to its normal baseline.

Actions performed:

• Opened and tracked an incident with our data platform provider and provided detailed monitoring evidence to support their investigation.
• Deployed mitigations on our streaming ingestion stack (increased service scaling and improved handling of stalled ingestion calls) to limit customer impact during the incident.
• Temporarily tuned ingestion timeout and retry settings to keep file loading as stable as possible while the root cause was being addressed.
• Continuously monitored ingestion latency and error rates until performance returned to normal.
• Confirmed full recovery, then reverted the temporary configuration to restore standard behaviour and avoid masking any future issues.

Actions to come:

• Continued close monitoring of ingestion latency over the coming days to confirm lasting stability.
• Keep the temporary mitigation patterns configurable in the code so they can be re-applied quickly should a similar provider-side degradation occur.