The Inefficiency of Searching and Finding

We have incredibly good technology with AI, and at the same time we are tremendously inefficient when it comes to searching and finding.

Take jobs and candidates.

A candidate sends out a great many applications and gets few interviews. A company screens a great many candidates and invites only a few.

Then take products and customers.

We research products at great effort until we have found the right one, especially when we are looking for something very specific. That takes time. At the same time, sellers put a lot of effort into reaching a large number of people, of whom only a small share buys.

Or take vendors and companies.

A great many companies are contacted with cold emails, cold calls and other measures. Everyone expects a single-digit conversion rate from the outset. The companies receiving all these sales messages no longer even read them and in the end only go with the vendor they already know.

There are surely more examples. But these three alone show: searching and finding is very inefficient in both directions.

My assumptions

In the future, people will use AI or an AI agent to search for and find things.

That means the human is no longer necessarily the one doing everything themselves. AI takes over part of it.

AI understands text and semi-structured data very well.

AI can create semi-structured data very quickly.

And AI can research very quickly in many places, because it can read a great deal.

The emphasis is on many places.

A human always searches in one, two or maybe three places, because they cannot cognitively process more.

If AI is the main actor in searching and finding, there is no longer necessarily a need to create a central place.

Today, platforms are the operating system of work and private life. These platforms are necessary for humans. But are they also necessary for AI? They can even be a hindrance if the information they contain is not searchable.

An AI does not need a central place. It can search in many places. It needs the information directly and in structured form. It does not need an algorithm that pre-sorts the results for it.

What if we codified the world?

Take Schema.org as an example. I don't care which semantics it ends up being. What matters is that it is lightweight and designed so that any AI can make sense of it.

If I codify everything that exists in the world – products, services, people, interests – in semi-structured form with the help of AI, it is, as we all know, extremely fast and easy.

And if I no longer have to store this information in one central place, because AI doesn't need a central place, the data could live in many different places.

We might then no longer need the one Google index or the one Bing index. Meta is surely building its own index right now, as is ChatGPT, and in China there are several indexes too.

But many more different indices could emerge.

Perhaps we then no longer need just one shopping platform or three shopping platforms, but many different ones. And perhaps we no longer need one or a handful of job boards, but many different ones.

AI is perfectly capable of reading a great deal.

If I am technically no longer dependent on a central place, I have the mental freedom to consider: should I do this per niche, regionally or locally?

And then the whole advertising game changes massively.

Why do we even still need the big platforms?

We all know: anyone today can install a local AI agent on their own device and get a cheap cloud subscription with an open-source model, or, with suitable hardware, download the LLM locally right away. I have that. A local LLM and a local AI agent (Agent Hermes). With it you can search without using ChatGPT, Anthropic, Meta and other providers. You can even search without Google. After all, there are many meta search engines. I tried it. It works. But the search results are meagre. Because so far, the internet has been built for humans. My AI agent didn't find what I was looking for.

Good indices are missing.

The best one is still the Google index.

But if building such systems is now becoming so cheap, why don't I build other indices? Why don't I codify the world, use open-source AI on the search side, and thereby undermine the entire monetisation model of the big AI and tech companies?

Technically possible.

Where are the catches?

First: Discovery

I need some form of discovery so that an AI agent even knows: where are these indices?

Suppose there are niche indices or local indices. Then the AI agent has to be able to enter somewhere in order to work its way onward from there.

If Google is no longer the entry point, what is?

I think of the physical world as it used to be: marketplaces in towns, libraries, notice boards.

I think of Wikipedia. The best example of hyperlinks. (Hyperlinks – that's how the internet once started. Being connected. But then came platforms and rounded buttons and icons that look like children's toys.)

And I think of all the online marketplaces that once emerged but never really worked. Why shouldn't we rethink them once more? Maybe they work now.

What was the problem back then?

Back then the problem was: nothing was going on there. Maintaining it all was laborious.

The effort has now dropped.

But there is still nothing going on there. But does anything need to be going on there at all if AI agents are doing the searching?

Not really.

(Humans often need the feeling: hey, I'll scroll around LinkedIn for a bit, because something is somehow going on there.

But if your AI is now searching for jobs for you, it really doesn't damn well matter whether anything is going on there.)

So perhaps decentralised discovery could work.

Second: Trust

We need a trust layer.

Whatever you build: it must not get spammed. It needs some kind of gates, and I need to know: the jobs there are real, the applicants there are real, the products there are real, the sellers there are real and the people there are real.

I somehow have to prove authenticity.

And that, too, is not impossible.

Perhaps trust can also be organised in a decentralised way. People are actually pretty good at networking.

Third: The technical infrastructure

You have to factor in the technical infrastructure behind it. I have to constantly re-fetch, check and update the data so that search intent and result match. Anyone who has their own website knows this. Dozens of bots come by every day. They perform different actions. It's about checking what exists, what's in it, what it fits, what has changed, what you can do, and so on. You get a small glimpse of how much work goes into building an index.

A new world – but do we still need platforms?

Anyone who now builds a niche index also has to somehow use bots to collect the information and keep it up to date.

But does it necessarily have to become a platform? That's what I'm asking myself!

Think of game theory. A platform works because the forces influence each other. The more participants, the more consumers are attracted, the more the participants do on the platform, the more consumers it attracts, and so on. That is an equilibrium.

Do I still need that when AI agents are searching for something? They are not participants of the platform. They just want to collect the information and data. The information is consumed by many AI agents, and if it is consumed from many different places, the game theory of a platform no longer works, because it can be several places. It would only work if only one place had really good information and all other places had only bad information.

But if the codified world is not in one central place but in many places, and the indexes are many smaller central places, what does that mean for the game theory of a platform?

The end of the algorithm

What I'm sure of: we no longer need an algorithm. If products, services, people and intentions are available really well as data and codified, and you work with other means to match search intents with products, services, people, jobs, etc. – with a semantics? – the result set will become smaller. I no longer need an algorithm that filters and determines the search results for me. I as a human no longer consume it. My AI agent consumes. It can filter on its own.

My thesis

We can completely rethink the whole topic of searching and finding, because the parameters have changed on all sides.

We can redefine searching and finding, independently of big platforms and big tech or AI companies.

It will certainly be a lot of work. But also highly interesting!

My questions

What remains unresolved for me: would a business model – through advertising? – in the end turn the whole thing back into a platform, so that nothing would be gained? Or is there a new business model that enables quality and openness? Or will it end up as an open-source project?

What are your thoughts on this?