Visual Guide
How I won a cautious partner over to run the A/B experiment that proved a new content discovery feature worked, on a device base where they, not us, controlled every release.
A content discovery feature shipped to a large set-top-box base, validated by an A/B experiment the partner initially did not want to run. I worked across the build, but one part defines how I operate: I earned the organisational buy-in to test the feature properly, then let the data settle the question.
I led a new feature, the Visual Guide: a content discovery experience that brought live, upcoming and on-demand TV together so viewers could find something to watch in one place. This case study focuses on the on-demand area, built to increase on-demand plays and widen the range of brands viewers watched.
To know whether it actually moved those numbers, I planned an A/B experiment across the set-top-box user base of a major UK pay-TV operator. The complication was ownership. The boxes belonged to that operator, and they controlled the release process for anything that landed on them, so running my experiment meant they had to change how they released.
Experimentation was not part of the operator's standard release process. Teams operating at that scale rightly protect stability, and an experiment asked them to take on extra operational work and to run multiple versions of the interface across a very large device base, with the added risk that comes with it. There was also limited prior appetite for it.
Without their agreement, I had no way to prove the feature met its goals, and the work behind it risked stalling on opinion instead of evidence.
I started by separating what I could change from what I could not. The hesitation came down to three things:
- Extra operational work for their release team
- Running multiple versions of the interface across a very large device base, and the added risk that brings
- Limited conviction that the learning would be worth it
The first two were real and fixed, so I could not argue them away. The third, the perceived value of experimentation, was the one I could influence, so that is where I put my effort.
Rather than send the usual feature pack, which is a fairly transactional handover for test and support teams, I brought the operator's release lead and my counterpart there into the actual problem and walked them through:
- the unmet viewer need behind the feature
- the specific metrics we were trying to move
- what the experiment would reveal
- how that learning would feed the next round of improvements for our shared users
When the familiar scepticism came up, that interface quality does not really move commercial metrics, I reframed the conversation around retention and engagement, which experience and content discovery clearly affect. Crucially, I positioned the experiment not as a one-off favour but as a repeatable capability: the process they set up once could be reused for every future test, on both sides.
The on-demand area of the Visual Guide, released to the operator's set-top-box base and measured with a controlled A/B experiment rather than shipped on assumption.
The experiment ran on a defined slice of the user base over several weeks, against a clean control group, so the result would be a real read on viewer behaviour and not a guess.
The experiment ran on 30% of the user base for five weeks, and the results were clear:
- On-demand plays rose by 8%
- The number of new on-demand brands viewers watched rose by 7%
Those were the exact behaviours the feature was designed to drive. Beyond the numbers, both teams came away with a repeatable experimentation procedure they did not have before, and my counterpart on the operator side had a clear, evidenced story to take back to their own stakeholders.
Getting a cautious stakeholder to yes is rarely about pushing harder. It is about dropping the parts of the argument you cannot win, making the value of the thing impossible to ignore, and framing the ask as a capability they keep rather than a cost they absorb. I bring that same approach to AI work: I can build the system, and I can earn the buy-in to test it properly and prove it moved the metric before anyone bets the business on it.
Facing a similar problem in your business?
I help teams work out what is actually worth building, then ship it. Let’s talk.
Start a conversationStart a conversation