CASE 1/3 The mistranslated streak: rebuilding the retention vocabulary in 16 languages
The core retention mechanic of a learning app is the streak. Ours, it turned out, didn't mean anything in most of our languages.
This surfaced through the research program I ran: structured interviews with churned and retained users in Turkey, Thailand and Indonesia, 90+ survey responses, plus a localization audit my interns ran. The same class of defect kept appearing: strings shipped untranslated, machine-translated, or plain wrong. One user found a microphone permission that said we needed the mic "to speak Spanish", inside the English course, a hardcoded leftover leaking into every course.
At the center sat the word streak. In Hindi the app used a literary word closer to silsila, which no learner connects to a daily run. We were building loss aversion around a word that produced no mental image of anything to lose.
The move
A terminology pass, not a translation pass. For each of 16 languages we chose the term a learner would actually recognize, and verified each against what the market leader uses there, since their term has effectively taught the concept to millions: seri in Turkish, runtunan in Indonesian, racha in Spanish, sequência in Portuguese. For Hindi we deliberately transliterated "streak" into Devanagari, because the concept word matters more than a dictionary equivalent. For Vietnamese we kept the English word, because that's what the market knows. Then the same discipline went outward: D0/D1 notification copy re-localized and A/B tested (reverting where the new copy lost, no ego), all 48 award names, and the learn tab.
Mechanics don't retain users, meanings do. If the word at the center of your retention loop means nothing in a user's language, you don't have a retention loop in that market. You have a counter.
CASE 2/3 The poisoned metric: when "US retention" wasn't measuring US users
A reported US retention drop had been escalated to the founder. The product was fine. The denominator was sick.
It started as a routine anomaly in the weekly metrics review: a US activation drop matching nothing in the product timeline. Digging into the cohort, the US bucket was contaminated: a meaningful share of "US" users were actually Russian users on VPNs, geolocating to the US because of app-store and payment restrictions at home. They behaved nothing like US learners, and their weak retention dragged the whole series down.
The move
Separate the populations, correct the metric, then re-read history: every experiment judged against the polluted US baseline deserved a second look. The mystery that had reached the founder closed as a measurement artifact.
Before you debug the product, debug the denominator. A retention problem appearing from nowhere is sometimes a population problem wearing a product costume.
CASE 3/3 The push delivery regression nobody could see
Android push impressions had been depressed for two months. The working hypothesis across the team was that something had changed at our push vendor, which was a reasonable place to start: the drop looked like a delivery problem, and delivery is the part of the stack a vendor owns.
I picked up the investigation. The first job was to work through the candidates honestly rather than argue about them.
Ruling things out
The vendor's own theory. They pointed at a specific Android OS version. I pulled OS-version-wise retention and delivery from our side and found the opposite pattern: those users were among our best performing. That closed it.
The SDK. We shipped the recommended upgrade. No movement.
Our instrumentation. Here we did find a real defect, a deep-link regression that mis-stamped the notification-source attribute and made notification-sourced retention read roughly a quarter of its true value. Worth fixing, but it explained the measurement rather than the delivery.
Campaign configuration. Send times, audiences, chart definitions. One conversion definition was silently excluding iOS. Corrected, still not the cause.
The credential chain. We audited which messaging project our pushes actually ran through, down to comparing the app's configuration file against the credentials uploaded to the vendor. Nothing there.
The question that worked
At that point I stopped asking what broke and asked what had been different during our best period.
Two weeks in July stood out in the good direction: push reach per active user was roughly 40 percent above the surrounding months, then decayed back. Nothing had changed at the vendor in that window. What had changed were our own builds.
That reframed the problem. Something we did had improved delivery, and something we did afterwards had undone it.
I laid the delivery timeline against the release calendar and ran a falsification test on it: if the vendor pipeline had broken, day-7 delivery would cliff on the same date as day-1 delivery. Instead it declined gradually, which is the shape of something spreading through the installed base as users update. That pointed at our own code. I took the timeline and the test to the engineering manager, and engineering diffed the notification path across the two builds.
The cause
Push SDK initialisation had moved off the main thread and into a background task, as part of work to make app start faster. For a short window in July an unrelated change to the startup sequence had put it back on the main thread, which is exactly why those two weeks were our best period, entirely by accident.
The consequence only shows up on devices that restrict background work aggressively. There, initialising in the background meant the push token was failing or being delayed at registration. Users who had granted notification permission were quietly never becoming reachable. No error surfaced anywhere, which is why it survived two months of investigation and why it was equally invisible on the vendor's side.
The structural finding underneath it
Alongside this I mapped why Android push is harder for us than iOS. Our push arrives as a data-only message, which carries no notification for the OS to draw. The phone has to wake the app, and the SDK inside the app draws the notification. If the phone declines to wake it, nothing appears, the message still counts as sent, and no error is raised. iOS has none of this, because Apple draws notifications itself.
Cutting delivery by device manufacturer made it visible in one table. Half our Android base sits on manufacturers with aggressive battery management, and their day-1 push reach ran about 10 percentage points below Samsung.
The comparison that settled the framing: on Samsung, where the OS does not aggressively kill background apps, delivery sat at parity with iOS. Stock Android devices came in slightly above it. So this was never Android being a worse platform. It was specific device behaviour interacting with how our push was constructed.
That also explained a puzzle about send counts being lower on Android, not just impressions. On these devices the OS force-kills the app on day 0, the phone declines to wake it for the first send attempts, and the profile is marked unreachable and drops out of the day-1 audience the same day. The gap was same-day write-offs, not expired tokens.
Result
Two fixes shipped in September. The second one, which moved push initialisation back onto the main thread, is the one that worked.
Measured as a rate with a fixed denominator (of users who hit a campaign's trigger with notifications enabled, what share actually received that push), Android delivery increased by 10 percentage points. iOS was flat over the same window, which is what you want for an Android-only change. The step lands on the exact date the second build reached users, and the first build had already been fully rolled out the day before with no movement. Android delivery went from roughly 10 points below Samsung to parity or better, and above the accidental July peak.
A second finding from the measurement
The recovery was not uniform, and it sorted by message time-to-live. Campaigns with a one-hour TTL recovered 39 to 48 percent in relative terms. Evening campaigns with a ten-minute TTL recovered only 17 percent.
A registered token is necessary but not sufficient. If the delivery window is ten minutes and a battery-restricted device happens to be dozing, the message still expires. That points to a next lever that is a campaign configuration change rather than an app change.
The retention effect, measured the hard way
In the first days after the fixed build reached new users, day-1 retention on Android was up 1.4 points on the previous week, a 6.6 percent relative gain. Tempting number. It was also a week in which several other things shipped: a batch of retention experiments, and unrelated changes on both platforms. Signups dipped too, and smaller cohorts retain slightly better on their own. Claiming the full rise would have been easy and unfalsifiable, so I built the attribution out of controls instead, and the claim that survives is smaller.
The experiment launches washed out first. By design, a slice of users received none of that week's experiments. That slice rose as much as everyone else, slightly more in fact, and nothing moved in the days when the experiments were live but the fix was not. Whatever lifted retention did not care whether a user was in an experiment. A delivery fix does not.
Then the fingerprint test. The defect only ever existed on manufacturers that aggressively restrict background work; Samsung was never broken. If the fix drove the gain, it should appear only where the defect lived. It did. Devices on restricted manufacturers had run about 3 points of day-1 retention below Samsung; within five days of the fix they closed that gap by a full point, while Samsung itself moved only with the market. And the exposure was sharp: new installs were on the fixed build within two days of release, 96 percent of new users by day three, so the retention step lands on the same 48 hours as the rollout step.
And the prediction test. The mechanism implies markets whose Android base is dominated by unrestricted devices, such as the US, UK and Australia, should see no improvement at all. They did not, and that is the mechanism holding. The other side held too: Android-heavy markets where restricted manufacturers are common improved, India by about 20 percent in relative terms, Indonesia and Thailand by about 5. A fix that "worked everywhere" would have argued against this attribution, not for it.
What I claim: the recovery that sits where the defect lived, the point of retention gap closed against Samsung, and, because new installs adopted the fixed build almost immediately, a read that is close to the full effect rather than an early glimpse of a larger one. What I do not claim: the rest of that week's platform-level rise, which belongs to the other changes shipping alongside and to the broader market. That version would not have survived one adversarial chart.
Ruling a vendor out takes the same rigour as ruling one in, and until that work is done you cannot see your own code clearly.
The thing that actually unlocked it was not a tool or a new dataset. It was inverting the question: instead of asking what broke during the bad period, asking what had been different during the good one. The answer was a side effect nobody had designed and nobody owned, which is exactly the kind of cause that survives two months of competent people looking in the right general area.
The last lesson came after the fix shipped. The strongest evidence a fix worked is not that the metric went up; metrics go up for many reasons in the same week. It is that the gain appears precisely where the defect lived, fails to appear where it did not, and survives every control you can construct against yourself. The claim that survived three falsification tests is smaller than the one I could have made on day one, and worth more.
ALSO IN THE LOG Migrations and recoveries
- CRM migration for 5M users (CleverTap → MoEngage): redesigned event schema and SDK integration, and overrode a vendor path that would have force-logged-out every user, designing push-token regeneration on launch instead.
- Email deliverability recovery: rebuilt sender authentication, isolated campaigns to a subdomain, recovered root-domain reputation, campaign mail back in the inbox.
- Research rigor: mixed-methods program across 3 markets, 90+ survey responses, managing 3 market-based interns.