ISSUE SUMMARY

Between 2026-05-13 07:17 UTC and 2026-05-15 14:59 UTC, users were displayed a deprecated paywall screen with a lower conversion rate. This resulted in a severe drop in daily sales. During this period of 55 hours and 42 minutes, a minimum of 4500 users were affected.

User traffic metrics remained normal throughout this period.

EVENT TIMELINE

Date & Time Event
2026-05-13 07:17 UTC A remote config payload is pushed targeting behaviour of a CTA button (unrelated to the paywall screen) in the app.
2026-05-14 Drop in sales - slow yet steady but not significant enough to trigger a reaction.
2026-05-15 Early Hours Second day of steady drop in sales - significant enough to be considered an anomaly.
2026-05-15 14:00 UTC A support email is received from a user asking how they can make the purchase in the app.
2026-05-15 14:15 UTC to 14:30 UTC Email is acknowledged, triggering an investigation. Purchase flow tested leading to the discovery of a deprecated paywall screen being displayed instead of the new one with better conversion rate.
2026-05-15 14:30 UTC to 14:45 UTC An RCA leads to a broken JSON configuration payload being fetched from the remote server.
2026-05-15 14:59 UTC A new remote payload with a proper JSON body is made available for distribution, effectively fixing the issue for new & existing users.
2026-05-15 15:07 UTC A support email replying to the user is sent.
2026-05-19 16:03 UTC The deprecated paywall screen is permanently removed from the codebase. Changes sent for production release under build 32.

RCA (ROOT CAUSE ANALYSIS)

Using the 5 Whys methodology.

Step Details
Issue Sales dropped significantly.
Why #1 Users were displayed a deprecated paywall screen with a lower conversion rate instead of the newer paywall screen with significantly better conversion performance.
Why #2 The default paywall screen (old & deprecated) was intended to be overridden using a configuration payload fetched from a remote server. The payload was successfully fetched but not applied.
Why #3 The fetched remote payload contained an invalid JSON structure.
Why #4 The JSON structure was not validated on the server side before being made available for distribution.
Why #5 The configuration payload relied on manual verification, which was skipped (human error), and there was no automated system in place to validate payload integrity prior to release.

IMPACT

Between the time the faulty payload became available for distribution and the fix was applied, around 4500 users were affected.

Daily revenue dropped by more than 90%, including at least one refund likely triggered by the deprecated purchase flow experience.

User acquisition and traffic metrics remained normal throughout the incident period, indicating that the primary impact was conversion-related rather than traffic-related.

It is also possible that the app experienced a temporary ranking decrease due to the sudden drop in sales velocity, though this remains speculative.

CONTRIBUTING FACTORS

Human Error

The configuration payload structure was not manually validated before deployment. Additionally, no post-deployment verification was performed.

Technical Limitation

The dashboard/system responsible for payload distribution did not automatically validate JSON integrity before release.

Tightly Coupled Configuration File

A single JSON payload was used to configure multiple unrelated application behaviors. As a result, a failure in one part of the payload caused unrelated configurations to fail and revert to defaults.

Had the configuration been decoupled into isolated payloads with a 1:1 mapping to features, the incident impact would likely have been limited to the intended target only.

RESOLUTION

Immediate Fix
  • The configuration payload was updated with valid JSON data, restoring proper paywall behavior.
Permanent Fix
  • The deprecated paywall screen and all related code were removed from the codebase (PR #147).
  • A ticket was created to decouple the configuration system into isolated payloads (Issue #138 - TBD).
  • A ticket was created to document release checklist for remote configuration changes (Issue #139 - TBD).

ACTION ITEMS

  • Decouple configuration payloads
  • Establish release checklist for remote configuration changes

SUPPORTING EVIDENCE

  • Revenue analytics for May

  • User support email

  • Remote payload fetch velocity

  • Remote payload versions