The Next Screen AI Learns Is in Your Pocket

1

It started at a hackathon

This one came out of an app-building hackathon in Paris.

The goal was to build an app. We took a morning routine app and tried to make it do the routine for you: open your other apps and do the steps in them.

On an iPhone, that is hard. Every app lives in its own sandbox, a closed box that cannot see the other apps. To open another app, you must list its link, its "URL scheme", before you publish. You need one for every app you might ever want to reach. That takes a long time. And the list cannot grow with the user.

So we went around it. Screen Time tells us which apps a person really uses. We built a small browser inside our app, and those apps open there in their web version. Inside that browser, our app can take a script, a prompt or plain directions, and do the task for you.

When we arrived, people told us it could not be done. By the end, they were surprised that we had found a way.

MWM, a Paris app studio, hosted the event in their offices. They looked after us very well, and they were the standout sponsor. Their technical team loved what we built and was impressed by how fast we moved. Thank you to them. We had a great time.

It also showed me a bigger problem. There is very little data on how people really use their phones.

Think about that. The phone is the screen you touch more than any other. And when you want to know how people really move through it, there is almost nothing to grab.

So I went looking. Who is teaching AI to use a phone? And who actually knows how humans use one?

Two different crowds, it turns out. They barely talk.

2

The phone is the harder screen

AI "computer use" started on the desktop. That was the easy version. Phone use is next.

On a web page, an agent can read the code and find the button. On a phone, apps are closed boxes. Often the agent gets a screenshot and a partial "accessibility tree", which is the hidden list of screen elements that screen readers use. That's it.

Then the phone adds its own fun. You don't click, you swipe, long-press, pinch and pull to refresh, and each one means something different. The buttons are a few millimeters wide and move around with every screen size. Logins, payments and security codes stop an agent in the middle of a task.

Morning routine

• Drink waterskipped
• Stretchskipped
• Make the bedskipped
• Go outsideskipped

Start my day

Tap to wake

The glass is asleep, so you can scroll past it.

And there are two worlds. Android lets an app control the screen once you give it permission. The iPhone is far more locked down.

The phone is still the bigger prize. It's where people live.

3

Who is teaching AI to use phones

A lot of people. Phone owners, funded startups, and open research labs, many of them in China.

Look at the last row next to the AGI row.

On the older test, a startup says it scores 97.4%. On a newer and harder one, the best model its authors tested fails about 4 tasks in 10.

A fake phone with 28 fake apps.

So the money is there. The talent is there. The agents still get lost.

4

Who watches how humans touch the screen

A different crowd. They sell to app teams, not to AI labs.

Each small screen is a chapter of this article. The glow is where you stayed. The rings are your taps. It lives in this tab's memory: nothing is stored, nothing is sent.

Replay my sessionWipe it

Your session

this page, this tab

Reading

0:00

Stops

0

Longest pause

0 s

Went back up

0

Taps

0

Rage taps

0

Keep reading. The heat builds where you stay.

This data is worth money. UXCam's own site says Carrefour's bank cut rage taps by 50% after a heatmap showed the cause. That is the vendor talking, so take it as a sales line. But app teams do pay for this.

Here is where these tools stop. They tell you what happened. They do not tell you what a new design will do before you ship it. And nothing they learn goes into an AI agent.

That is the hole we fell into at the hackathon.

One crowd teaches AI to use phones. The other watches humans use phones. Almost nobody connects the two.

5

Where the opening is

Five ideas I would test.

1. Fake users that tap like real ones. Train agents on real touch patterns. Run them through a new version of your app before launch. They show you the screen where a human would stall or rage-tap, before a human does.
2. UXCam, but for agents. Agents are starting to use apps for people. App makers will want to know which sessions are agents, where they get stuck, and why they quit. I have not found anyone who owns this.
3. An agent-readiness score. How easy is your app for an agent to use? Clear labels, stable layouts, no dead ends. If an agent can't finish the order in your app, it finishes it in someone else's.
4. Licensed touch data. Consented, privacy-safe data on how humans really move through apps. Agent builders need it. That is a product on its own.
5. Accessibility for free. The clean labels that help an agent also help someone with low vision or limited hand movement. Same work, two wins.

Exhibit D Two crowds, one screen

The thread is the same in all five. Whoever has the best data on how humans actually use phones makes both apps and agents better.

6

What could go wrong

Plenty.

The platforms can build it themselves. Google and Apple own the operating system. Anything they add for free can wipe out a startup. The safer bet is to serve app makers. That is the customer the platforms tend to leave to others.

The iPhone is mostly closed. AGI's app is Android only for now. The open tools that reach iOS need a developer setup. If your business needs the iPhone, your business depends on Apple.

Privacy. Session replay means recording people's screens. You need to mask personal data and you need real consent. Not a checkbox on page nine.

Trust. Gallup and Bentley University asked Americans in May 2026 if they trust businesses to use AI responsibly. 27% said yes. Now ask them to hand an agent their phone.

And one question I can't answer yet. Who pays first? The app maker who wants a better app, or the AI lab that wants a better agent?

I think it's the app maker. If you build apps, tell me if I'm wrong.