How to measure ASO: store reports, page A/B tests and organic installs

The grocery delivery service's designer has drawn new screenshots following the plan for setting up the app page, and the owner wants to simply swap them in: "In a month we'll check AppsFlyer and see if organic went up." Organic won't tell you that. On Android it has already halved, from about 300 installs on 1-14 September to 150 on 1-14 October, with nobody touching the page, and some of those installs may have been claimed by placement 2210 from the fraud check of a new ad network. To learn what the page does, you need the stores' own reports and a test in which both versions of the page run on the same days. The service's numbers in this lesson are made up; the reports and tests are described from Apple's and Google's help pages.

What the stores count

The stores count conversion differently: Apple divides downloads by devices that saw the app in the store (in search results, on the tabs or on its page) (metric definitions), while since July 2026 Play Console counts Install taps, so the two figures aren't compared head to head (what App Store Connect and Play Console show).

Take your norms from the stores. Apple compares an app's conversion and retention with a group of similar apps: same category and business model, 25th, 50th and 75th percentiles, with retention counted over users who agreed to share analytics (Apple peer group benchmarks). Play Console has CTR benchmarks with the median, 25th and 75th percentile of peer apps (store listing report). Skip any "average store page conversion" figure from an article without a described sample, since there is no way to tell whose apps it came from.

Organic in an MMP and ASO are different things

AppsFlyer calls an install organic when it found no ad touch under its rules (AppsFlyer attribution model). Here is how five people end up in the reports:

ASO's work here is only in the first row and partly in the second: someone searching by name can also walk away from a weak page. The rest is ads, iPhone consent and someone else's clicks, so measuring the page by organic in an MMP gets you nowhere. The third row is also a reminder that Apple puts downloads from ads in search results into the App Store search source (sources in App Store Connect).

Two things can be measured. The first is page conversion, and the stores provide tests for that, covered below. The second is category search: Play Console has a Search term dimension, and September's query breakdown already showed how many people look for the service by something other than its name. The new name and keywords will ship with the next app version. After that, compare category visitors week by week for the four weeks before the release and the four after, keeping the season in mind.

Page A/B test in Google Play

In Play Console a page test is called an experiment (store listing A/B tests in Google Play). The settings:

• Type. Default graphics experiments change the icon, feature graphic and screenshots for the listing's default language. Localized experiments can also change descriptions, in up to five languages. You can run one default graphics experiment or up to five localized ones at a time.
• Number of variants - up to two against the current listing.
• Share of visitors who will see a variant instead of the current listing.
• Target metric - unique user install clicks or unique user open clicks.
• Minimum detectable effect and confidence level set how big a difference counts as a win and how sure you want to be.

After six months the experiment stops on its own.

Suppose the delivery service launches a graphics experiment on 26 October: one variant with the new screenshots, half of the visitors, install clicks as the target metric, a minimum detectable effect of 3% and a 90% confidence level. Four weeks later the picture might look like this:

Play Console shows the difference together with a confidence interval. Here it runs from +4.3% to +10.7% over the current listing: entirely above zero and above 3%, so the variant wins, and Play Console may recommend applying it. Had the interval included zero, the variant wouldn't have won, and you'd either keep waiting or keep the current listing.

October's noise doesn't hurt this test. The new TikTok setup changes who comes to the page, and placement 2210 changes whom the MMP credits with installs. But both groups in the test get the same flow of people on the same days, and MMP attribution never enters the experiment report. A "month before versus month after" comparison has no such protection.

Page test in the App Store

At Apple the same thing is called Product Page Optimization (product page optimization):

• up to three treatments against the original page, changing the icon, screenshots and previews;
• treatments are shown to a random share of people on the App Store with iOS and iPadOS 15 or later, and each person sees the same treatment until the test ends;
• a test runs for up to 90 days, only one at a time, and can't be changed once started;
• new images go through App Review; a test without a new icon can be submitted without a new app version, while alternate icons must already be in the app;
• the results show impressions, conversion rate, improvement over the original page and confidence.

Apple advises waiting until a treatment performs better or worse than the original with at least 90% confidence. Applying a treatment mid-test ends the test. The service runs the same four screenshots on iOS as a separate test: the App Store audience is different, and the Android result doesn't carry over.

Separate pages for ads and queries

In the App Store, besides the default page, you can publish up to 70 custom product pages with different screenshots, promotional text and previews (custom product pages). Each has its own URL. You can assign keywords to a page, and then it shows in search for those keywords instead of the default one. From iOS 18 you can attach a deep link to the right screen. Apple Ads can send people to these pages, and App Store Connect shows impressions, downloads, conversion and retention for each. The Apple in-app purchase revenue metrics are no use to the delivery service: groceries are paid for by card or to the courier.

Google Play allows up to 50 similar pages (custom store listings). They can change the name, icon, descriptions and graphics, and you can show them to people in a chosen country, people who arrive with chosen search terms, traffic from Google Ads, or former users (custom store listings).

Suppose the service builds an App Store page for Astana queries: screenshots with Astana addresses and a deep link into the catalogue. Comparing it with the default page as an A/B test would mislead: this page gets people who searched for доставка продуктов астана, and the default page gets everyone else. If its conversion is higher, that tells you about people with a precise query, not that the page is better. Only a test shows which page is better.

From an Install tap to a first order

The Play Console test counts taps, while the service makes money on orders. The store doesn't pass the page variant to the app, so first orders can't be counted per variant. What's left is to translate the result into money and check it after rolling out.

People from Meta and TikTok ads also land on the page, so a variant that gets 7.5% more taps lowers those networks' cost per install by about 7% at the same budget. As an example, take Meta's September figures in AppsFlyer: a CPI of 500 ₸ and 10 first orders per 100 installs. This is the calculation for the Android side; on iOS its own test will show the effect.

If the new installs order at the same rate, the page brought customers. If they're empty, the extra installs came from people who ordered nothing, and spend per first order stayed where it was. The store test can't tell these two cases apart; to the test, both are a win.

You can tell them apart in AppsFlyer. Take the Android app and, for each network, compare D7 first orders per 100 installs in the cohorts from the four weeks before applying the variant and the four weeks after. Look at TikTok separately: in October its geography and optimization event changed (in-app events). Put the share of address_out_of_zone next to it: the new screenshots name the cities, so the number of people with an out-of-zone address shouldn't grow.

To tell 7 orders per thousand installs apart from noise you need more than 20,000 installs in each period, and Meta and Google bring about 4,000 a month on Android. So before-and-after cohorts only catch a crude failure, say 8 orders per 100 instead of 10, and then the variant is rolled back.

Task: an ASO measurement plan for your app

Take an app you work with. Without access to the accounts, the plan can still be drawn up from this lesson and shown to the client as a proposal.

• Export page conversion by source for four weeks from App Store Connect and Play Console, and put the store's benchmarks next to it.
• In Play Console, split search queries into brand and category queries.
• Write down one hypothesis for a page test and its settings: experiment type, variant, share of visitors, target metric, minimum detectable effect, confidence level, duration.
• Write down which MMP cohorts you'll use to check first orders after applying the variant, and which result will mean the variant brought empty installs.
• Write the owner two sentences on why organic in an MMP is no yardstick for ASO.

The delivery service's plan fit on half a page. Hypothesis: a catalogue with the cities on the first screenshot will win over more people from Almaty and Astana and fewer from other cities. In Play Console, a test on install clicks with a 3% minimum effect and 90% confidence; in the App Store, a separate test run to 90% confidence; both for at least four weeks. After rolling out, D7 cohorts for Meta and Google for the four weeks before and after, TikTok separately, plus the share of address_out_of_zone. Failure means a noticeable drop in first orders per 100 installs.

The service's two sentences to the owner read: "Organic in AppsFlyer depends on ads and on other people's clicks: in the first half of October it halved although nobody changed the page. We'll check the new screenshots with a test in the stores themselves and look at orders by cohort before and after."

Video: an introduction to ASO

An introductory video from ASOMobile. Check what exactly each store lets you test against Apple's and Google's help pages linked above.