5 A/B Tests That Increased Our Book Summary App's Conversions

For 15-minute non-fiction book summaries of best-selling books, check out sumizeit.com.

TL;DR

Over the past few years, we've run a series of A/B tests on Sumizeit aimed at improving app conversion rate, onboarding conversion, and free trial vs. freemium performance — the kind of experimentation most subscription and mobile app growth teams run constantly but rarely publish with real numbers attached. This post breaks down five tests: a referral-style "gift a subscription" experiment rooted in behavioral psychology, an App Store screenshot test, a free trial vs. free-summaries pricing experiment, a paywall/freemium limits test, and a personalized onboarding flow test — each with sample sizes, statistical significance, and the reasoning behind why the winning variant won. If you want the psychology backing some of these results, Thinking, Fast and Slow and Contagious are two of the source texts we kept returning to while designing these experiments.

Test 1: Gifting a Free Subscription to a Friend Increased Conversion by 13%

A few years ago, partly out of curiosity about behavioral psychology and reciprocity, we tested giving new Premium subscribers a free subscription to gift to a friend, at no extra cost to the purchaser. The control group got the standard Premium purchase flow with no gifting mechanic.

We ran this test across roughly 9,400 users over a four-week period, split 50/50 between control and treatment, and tracked conversion from free trial to paid Premium as the primary metric. The treatment group converted at 13% higher than control, a difference that held up at a 95% confidence level (p < 0.05) and wasn't explained by any imbalance in traffic source or acquisition channel between the two groups.

The mechanism is worth naming explicitly, since it's not really about the free subscription itself — it's about reciprocity and social proof, the same psychological lever Robert Cialdini built an entire framework around and that shows up repeatedly in Contagious, Jonah Berger's research on why ideas and products spread. Giving someone the ability to gift something valuable makes the purchase feel less like a personal indulgence and more like a generous act, which lowers the psychological friction of paying in the first place. We're re-running this test now with a larger sample to see if the effect holds at scale, and we'll update this post with results in a few weeks.

Test 2: Outcome-Focused App Store Screenshots Doubled Download Rate

For our iOS app, we tested two different App Store screenshot sets: one built around features (highlighting the text/audio/video summary formats, the library size, the quiz mode) and one built around outcomes (showing a user finishing a book in 15 minutes, a "books read this year" counter, a before/after framing around reading habits).

Using Apple's App Store Connect product page experimentation tool, we split App Store visitors roughly evenly across both variants over a three-week window, with a combined sample of approximately 42,000 App Store page views. The outcome-focused screenshots converted browsers to downloads at roughly 2x the rate of the feature-focused set — a jump from a 3.1% download rate to 6.4%, a difference far outside the margin of error given the sample size.

The takeaway we keep coming back to internally: people don't download a book summary app because it has an audio format and a quiz mode, they download it because they want to become the kind of person who reads more, faster, without the guilt of an unfinished bookshelf. Selling the transformation outperformed selling the functionality, and it wasn't close.

Test 3: Free Summaries Outperformed a Credit-Card-Required Free Trial

Free 7-day trials are close to an industry standard among book summary apps, so we wanted real data before assuming that pattern applied to us too. We split website visitors 50/50: one group got the standard 7-day free trial requiring a credit card upfront, and the other got 2 free full summaries with no card required, converting to paid only when they wanted a third.

Across website visitors tracked over five weeks, the credit-card trial group technically showed higher initial "signups," but the data underneath was messy — about 60% of those signups used fake card numbers or cards that failed the charge attempt on day 7, meaning the vast majority of that group never actually converted to a real paying customer. The 2-free-summaries group had a lower top-of-funnel signup number but a meaningfully higher real revenue-per-visitor figure once we accounted for failed charges and chargebacks: roughly 22% higher net revenue per visitor over the test window.

This lines up with a pattern well documented in subscription and mobile app growth circles: a credit-card-gated trial filters for people willing to hand over a card, not people who actually intend to pay, and it can quietly inflate a vanity signup metric while hiding a much worse true conversion rate. Letting people experience real value first, with no card required, produced a smaller but far more honest and profitable funnel for us.

Test 4: One Free Daily Summary Beat Both "Too Little" and "Too Much" Free Access

After user feedback that our paywall felt too restrictive, we tested three variants across a six-week window with roughly 15,600 users split into three even groups:

• Group A: one free book summary per day, reinforced with a daily push notification
• Group B: over 20 free summaries available to try, no daily limit
• Group C: control, existing paywall with no additional free access

Group A had both the highest conversion to paid (18% higher than control) and the highest 30-day retention, with daily active usage roughly 2.4x that of Group C. Group B, despite offering by far the most free value, actually converted worse than Group A and only marginally better than the unchanged control.

Our read on this, based on the same daily-trigger mechanics covered in Indistractable: a single daily free summary functions as a habit loop — a reason to open the app every day, reinforced by a notification, that builds a routine before asking for money. Over 20 free summaries removes urgency and habit-forming structure entirely; users could binge them whenever, which meant many just... didn't, and the app never became a daily habit. Scarcity paired with a daily cue outperformed abundance, which is counterintuitive if your instinct is "more free stuff should convert better," but it tracks with how habit formation actually works.

Test 5: Personalized Onboarding Increased Conversion Before Users Even Saw the App

About a year ago, with conversion stalling, we tested a personalized onboarding flow against sending users straight to a generic Discover page. The onboarding group answered a short series of questions about their reading interests and goals, which we used to generate a personalized book recommendation list; the control group skipped straight to a non-personalized Discover page with generic bestseller recommendations.

Across new app installs over four weeks, the personalized onboarding group converted to paid Premium at a rate 24% higher than the control group — notably, before a meaningful number of users in that group had actually opened a single book summary. The effect was strongest in the first 48 hours after install, suggesting the perceived value was being created by the onboarding experience itself, not by product usage that came later.

We followed this up with two additional rounds of testing on the onboarding flow itself. Adding screens featuring recognizable public figures and the kinds of books they're known to read increased conversion by an additional 20% over the already-personalized baseline. And a smaller but real lift came from reinforcing each answer with a brief affirming response — a friendly on-screen character responding to each question with something like "great pick" before moving to the next screen — which further increased conversion beyond the personalization and celebrity-book screens alone.

The psychological read here connects to a broader pattern behavioral researchers describe as the IKEA effect and commitment-consistency bias — mechanisms discussed at length in Thinking, Fast and Slow: the moment someone actively answers questions and gets personalized output back, the product starts to feel co-created rather than generic, and people are more willing to pay for something they feel they had a hand in shaping. That effect apparently doesn't require the person to have used the core product yet — the sense of "this was built around me" is enough to move the conversion needle on its own.

How We Actually Ran These Tests (For the Statisticians in the Room)

A few methodology notes, since "we A/B tested X" means very different things depending on who's running it. For each of the five tests above, we used a fixed sample-size calculation upfront rather than peeking at results daily and stopping whenever a winner looked good — the latter is one of the most common ways teams fool themselves into false positives, since checking a test repeatedly and stopping at the first significant-looking result inflates your false-positive rate well past the nominal 5%.

We also randomized at the user level rather than the session level for anything involving pricing or paywall changes, since session-level randomization can expose the same user to both variants and contaminate the results. For the App Store screenshot test specifically, randomization happened at Apple's infrastructure level via App Store Connect's built-in experimentation tool, which handles this correctly by design. Every test above hit its pre-calculated minimum sample size before we called a winner, and every reported lift held at a 95% confidence level unless noted otherwise.

One failure mode worth naming honestly: not every test we've run has produced a clean, publishable result like the five above. We've had tests that showed no statistically significant difference at all, and at least one where a promising-looking early trend fully reversed by the time the test reached its planned sample size — which is exactly why we don't recommend stopping a test early just because day three looks exciting.

What These Five Tests Have in Common

Looking across all five results, the pattern isn't really about any single tactic — it's that removing friction and building perceived personal relevance consistently beat adding more raw value or more aggressive commitment devices. A free daily summary beat 20 free summaries. Free-with-no-card beat a credit-card-gated trial. A few onboarding questions beat a generic homepage. Outcome-focused screenshots beat a full feature list. Gifting beat a plain purchase flow.

If you're running similar experiments on a subscription app, a few practical notes from our own process: run tests for a minimum of two to four weeks to smooth out day-of-week effects, calculate statistical significance before declaring a winner rather than eyeballing a percentage difference, and watch downstream metrics (real revenue, 30-day retention) rather than just top-of-funnel signups — as Test 3 showed us, a metric that looks good at the top of the funnel can hide a much worse story underneath.

The Bottom Line

None of these five tests required a large team or an enterprise experimentation platform — just a clear hypothesis, an even split, a large enough sample, and the discipline to measure real downstream outcomes instead of vanity metrics. If there's one theme tying a "gift a subscription" test to a personalized onboarding flow, it's that people convert when something feels chosen for them, not sold to them. We'll keep running these experiments and will post updated results as new tests wrap up.

For 15-minute non-fiction book summaries of best-selling books, check out sumizeit.com.