A ten-million-token heist

This is the story of a 10-million heist, a great comms caper.

If you followed LinkedIn properly, last Thursday Google unveiled Gemini 1.5, their latest LLM, followed a few hours later by OpenAI unveiling Sora, a text-to-video model with a mind-blowing, dreamy rendering (I’ll leave you to watch the now-famous video of the sand chair that runs away). What followed was thousands of posts about those famous videos.

Too bad for Google, because it’s over THEIR Gemini 1.5 demo videos that you should have been marveling / getting excited / hiding.

Why?

Gemini 1.5 is a multimodal model that offers an input context of 1M tokens, possibly 10M, and that posts performance equivalent to Gemini 1.0 ultra (GPT-4 equivalent). Such a context represents prompts equivalent to 1h of video, 30k lines of code, or a 1400-page document.

The most impressive part is its performance: 99.7% accuracy. That is, the model “remembers” the slightest detail that was given to it as input (look closely at the image, for 10M tokens!)

What this implies, and the outlook

Part of the RAG systems that we (the collective we) have built in blood and tears these past months is going to be competed with by 1 (ONE) call to an API.

For the uninitiated: a RAG makes it possible to provide an LLM with content in order to formulate an answer to a question. This makes it possible to work on knowledge unknown to the model, like an internal procedure.

Let’s take a concrete example (and simplify): your 1000-page internal procedure is cut into several blocks of text or of image, each block is transformed and stored in a database. When a request comes in, we look in those blocks for the elements that can provide an answer, so as to then have them formulated by an LLM.

With Gemini 1.5, you just put the content of your document in the prompt and it’s settled. An API call will, granted, maybe cost 5€ at first, but will certainly offer better results given the announced performance.

Concrete applications

Imagine, you have a set of technical and functional documentation, you want to know the impact of changing a single parameter on the whole: it’s done in 1 call to an API. And I’m not even talking about the multimodal potential!

Imagine an AI specialized in medicine with a large context like this one. In less than a minute, a medical file, the history of care, the treatments, the X-rays, the MRIs can be analyzed by a system that isn’t subject to biological vagaries like fatigue, concentration, personal worries, etc. This is no longer science fiction, it’s within the next 2 years at most.

I’m talking about medicine, but the application to the legal, consulting, and HR sectors is going to be insane (large ego shields to plan for).

Conclusion

I stand by my line: always bet on Google.