Every large language model exists because of three things: human intelligence, computation resources and content.
Human curiosity, creativity, and ingenuity invented AI. The scientists and engineers are now the leaders of the AI Age, and some of the most powerful people in the world.
Compute has an entire industry behind it: cloud providers, chip manufacturers, energy companies. These companies are the picks and axes of the AI age.
Published content represents the raw ingredients of this new era. And even more so, infrastructure is now being built to provide this content in a scalable way.
This time, publishers think different
Recently, I’ve had dozens (hundreds?) of conversations with our publisher partners, who are asking very similar questions about their content:
- “What is the value of my content?”
- How can a marketplace not lead to a race to the bottom?”
These questions are proof that publishers are thinking differently, particularly when compared to when advertising marketplaces were built 25 years ago. Publishers are not immediately outsourcing their future, but digging in and learning more.
Why have things changed?
First, global scale software engineering and infrastructure costs less than 1% than it did 25 years ago (Does anyone want to challenge me on this? I’ll debate this stat all day long…).
Second, it’s hard to imagine a new content marketplace in the same hands as those who built advertising marketplaces, will lead to a different outcome (riches for the marketplace owners, and crumbs for everyone else).
Publishers are coming to the realization that publisher-controlled infrastructure is the best way for a robust content marketplace to be built. A publisher-controlled marketplace is not only good for the publishing industry, but for the entire AI ecosystem.
This is where Ozone Labs comes in.
What we’re building
Ozone Labs is an area where we experiment with new technology, collaborate with our partners, and together invent the future. Here’s what we deliver:
Deliverable #1: Ozone Labs mentality
At Ozone we like to say:
“We build systems that build software”
That mindset is the thinking behind Ozone Labs:
- Start with a hackathon (which we’ve now developed into the Labs Live concept). Collaborate with partners and find the right areas to focus.
- Codify working processes. Agents are the experts that build and operate software, and Ozone houses the experts who build agents. At Ozone, our agents build software, analyze data, make product decisions, and traffic campaigns. Watch this space for even more agents!
- Release early, and get feedback data quickly, iterate.
With Ozone Labs, everyone is invited to apply for an Labs Live hackathon which currently occur as a quarterly interaction with Ozone
Deliverable #2: Simulation platform
As a result of hundreds of meetings with the top publishers worldwide, we’ve built an end-to-end platform simulating how we imagine published content will feed AI infrastructure in the future:
The simulation platform enables publisher-led experimentation on how their content exists in this new world. Some parts of the platform include:
- Grounding server with millions of documents across participating Ozone publishers
- Human-in-the-loop evaluator of many different LLM services
- Blind comparison arena for head-to-head metrics using ELO score evaluation
- Content pipeline that formats publisher content in the optimal way for LLM consumption
- Chatbot playground to see how publisher content looks in action for an LLM
There are dozens of experimentation services, each of which we will share with the community in detail, so that our community has the capability to help invent the future together.
Deliverable #3: Research hypotheses
We have run many hackathons, and over 50 projects have been built. Some have been already transformational at Ozone. Others have failed. Many are being tweaked and modified.
Ozone Labs is our equivalent to baseball’s “farm system”. Our goal is to shine a light on what we are building at Ozone at an early stage, to the benefit of all.
Early research: Toward measuring publisher content value
One of the first questions we’re aiming to tackle is a relatively simple one:
“Does providing an LLM access to premium publisher content improve answer quality?”
The setup is straightforward:
- Generate topical questions - obtained from current conversations happening in online platforms like social media, search queries, and Google trends
- Present the questions to LLMs with web search turned off.
- Use RAG triad scoring, and human-in-the-loop scoring, to record which answer is better.
- Repeat Steps 2-3 with Ozone publisher content access
Models from different vendors performed at totally different levels. By having this platform, it is helping us understand how LLMs consume information. In our first test, Claude outperformed, while Gemini lagged. A retrospective on this test surfaced the different ways that each LLM uses tokens. This will lead to modifications and improvements. Subsequent versions of this test will be time-series, so that we can measure drift in performance over time.
A simple experiment such as this can form the baseline to many other experiments.
Where this goes
Ozone Labs is set up simply, but it's powerful. We can solve complex, multi-faceted issues as a community. Content pricing, marketplace design, publisher attribution, quality measurement — these are hard problems, and they're better solved together than apart.
Let's build together with Ozone Labs!