Incident 1: AC1 Event Ingestion Pipeline Errors (southamerica-east1)

 

This communication covers two related incidents affecting the AC1 event ingestion pipeline in South America region:

 

• Initial Incident: Intermittent Errors on AC1 Ingestion Pipeline

We experienced intermittent errors in the LiferayCloud-AC1 event ingestion pipeline in southamerica-east1 on Friday, September 26th. The issue resulted in a brief period of data loss for events during the affected time.

• Impacted Service: LiferayCloud-AC1 Event Ingestion Pipeline (southamerica-east1)
• Incident Time (PDT):
  • Start: 2025-09-26 07:56 PDT
  • End: 2025-09-26 10:55 PDT (Approx. 3 hours)
• Impact: This resulted in an event ingestion loss of less than 1% per project during the incident window.

 

• Recurring Incident: AC1 Ingestion Pipeline Errors

The AC1 event ingestion pipeline error recurred on Monday, September 29th, with a longer duration and greater impact.

• Impacted Service: LiferayCloud-AC1 Event Ingestion Pipeline (southamerica-east1)
• Incident Time (UTC):
  • Start: 2025-09-29 11:31 UTC
  • End: 2025-09-29 17:55 UTC (Approx. 5.5 hours)
• Impact: The average event ingestion data loss per workspace during this period was approximately 7%.

 

Resolution and Next Steps

 

A fix has been successfully deployed to resolve this recurring ingestion issue. The pipeline is now operating normally. We will continue to monitor the system closely and conduct a deeper root cause analysis to prevent future recurrence. We will update this channel if any further issues arise.

 

---

 

Incident 2: Project Details Endpoint Errors Affecting AC3 (europe-west3) and AC4 (us-west1)

 

We experienced a significant error when making requests to the Project Details endpoint, which caused failures in our downstream metric processing pipeline ("Composing"). This led to a halt in daily metric calculation and subsequent data gaps for LiferayCloud-AC3 and LiferayCloud-AC4.

 

Incident Details

• Impacted Systems: Metric Processing for LiferayCloud-AC3 (europe-west3) and LiferayCloud-AC4 (us-west1)
• Affected Component: Project Details Endpoint
• Incident Window (UTC) - Processing Jobs Halted:
  • AC3: 2025-09-25 21:45 to 2025-09-30 21:58
  • AC4: 2025-09-25 21:33 to 2025-09-30 20:32
• Status: The underlying endpoint issue is fixed, and the processing jobs are now running again.

 

Customer Impact

Customers might see data gaps in specific dashboards for the period of September 25th through September 29th.

• Unaffected: The Sites dashboard is unaffected, as it primarily uses session data.
• Affected: Users will likely see certain metrics on the Asset and Page dashboards display zero values for the impacted time frame (Sept. 25 - Sept. 29).

 

Recovery and Next Steps

The system is now stable. We are actively performing backfilling (back-processing) of the data for the affected period of September 25th through September 29th for both AC3 and AC4.

 

We apologize for the disruption and are grateful for your continued trust as we complete the backfill and restore full service integrity..