Summary

On April 16th, 2024 at 11:50AM PST, Analytic Cloud SDK 1.2.0 was released. 

As part of this release, Analytics Cloud’s caching layer was updated to transition the buffering of event requests from the Local Storage API to the Cookie API. Depending on a website’s configuration and the number of requests sent on each page, the event requests buffer could grow beyond the Cookie/Request header limit allowed for most Liferay DXP installations. This resulted in invalid requests.

Thirty minutes after the 1.2.0 release, we observed issues with the release. At that point, the rollback process was immediately started. By 2:00 PM PST the SDK was successfully rolled back to the previous version.

 

Incident Timeline

 

Impact

From April 16, 11:50 AM PST through April 16, 2:00 PM PST, visitors to some of our customer’s sites may have downloaded faulty cookies. These cookies prevented users from accessing our customer’s sites due to Invalid request errors. Unfortunately those users will continue to face Invalid Request Errors until the problematic cookies are removed. 

 

Root cause

An unexpected cookie header size was being stored in the browser causing the DXP server to respond with invalid request status code - 400.

This issue was more likely to occur on websites that heavily use web contents for defining a page and send more than 5 simultaneous Analytics Cloud events when loading.

 

Mitigation Plan

Approximately 2h after the 1.2.0 release, the previous Analytics Cloud SDK was restored. Additionally, on April 18th a hotfixed version of the Analytics Cloud SDK was released with the purpose of removing the offending Cookies.

For users still observing 400 Response status code, we recommend clearing the browser's cache. This will manually remove the offending Cookies and restore access to the site. If this is not possible, please reach out to Liferay Support for further assistance. 

 

FAQ

Has the issue been resolved?

Yes, the SDK release was rolled back and should not affect any new visitors to your site. However, customers who visited your site during the incident window may have to clear their cookies in order to access your site. 

How can Liferay prevent this from happening again?

We are rethinking our approach to Cookies and will likely not use Cookies for event buffering. Additionally we are moving ahead with a canary deployment strategy to avoid any widespread issues such as this in the future.