REX GenAI

While plenty of people are writing predictive posts about AI in 2024, I’m going to do a short REX on the client requests tied to GenAI that we handle / have handled at Coddity for a year now.

Document agents

Unsurprisingly, the majority is made up of requests to develop agents that use documentation / knowledge specific to the company. These requests can be addressed with homegrown RAG-type architectures, or with solutions like Google’s Search and Conversation or, more recently, OpenAI’s private GPTs. The promise of RAGs is fairly high, with performance that depends heavily on how they’re implemented and on the content they have to process. We’re a long way from the “drag and drop your pdf and chat with it” promised by many, at the scale of a company. All the more so because the cost of a high-performing RAG architecture is far from negligible (just look at the cost of managed DBs that include vector search). The challenge for 2024 is going to be evolving these architectures to take multimodality into account, opening new possibilities for RAGs.

Private conversational agents

No surprise, lots of requests to have an internal ChatGPT that no data leaves, with an internal or filtered LLM for employees. For the moment, the cost of having an internal agent (on-premise or cloud) built from an open-source LLM like Mistral or Llama is fairly high if you want a solution that’s high-performing, fast, and available, which cools a lot of enthusiasm. Using an LLM via an API is the reasonable solution, whether that’s OpenAI/Microsoft, Google’s more-than-complete Vertex platform, or, a novelty of the last few weeks, the Mistral AI API.

⚙ Business integration / AI service

This is about integrating AI directly as a software service inside existing architectures. Obviously we’re talking about production environments and their constraints, to which you add those specific to LLMs: hallucinations, latency, or control of completions. Not to mention the significant cost if you choose a solution based on an on-prem LLM.

How humans receive it

Having done a lot of training / presentations at large and small companies over these past 4 months, here’s what I take from it:

👉 Early adopters are a real driving force in the integration of genAI. I think the key is the speed/ease of testing POCs and showing a value add that makes it possible to create more substantial projects.

👉 Training teams in the use of GenAI / prompting is a real subject that must absolutely not be neglected. I “mocked” prompt engineering quite a bit, an error of judgment on my part.

👉 There are use cases for every job; you still have to go and see them. I’ve lost count of the “oh but it can do that??” (my favorite moment)

Now, some points for reflection.

👇

Perception versus reality

We see a gap between the perception of what it’s possible to do with GenAI and reality: no, you don’t need an army of AI researchers to handle the majority of use cases. Between the available APIs, the open-source models, and the infra available in 1 click, you very quickly get to a POC that unlocks the resources for a “production ready” solution. I’m talking about the majority of cases, not all use cases; this view obviously doesn’t apply to very specific cases or domains.

A very recent example of a state institution that wants a service based on generative AI. The use case is fairly classic, but the call for tenders insisted heavily on the need to have an R&D team, AI researchers, for a bidding company to be considered credible. The project was split into 3 phases: a POC with 3 companies over 2 months, a production-ready prototype with 2 companies over 4 to 6 months, and production. Given that we don’t have AI researchers or an R&D team, that the case is objectively rather simple, and that we’re going to face mastodons like CapG or Sopra, we chose the provocation angle and developed, in a few days, 80% of the POC, which we sent back as our response to the RFP. How? By using open-source models that let us reach, on this precise use case, a mind-blowing level of performance. (We’re waiting for the response to our proposal.)

Where it’s “funny” is that the perception/reality gap goes both ways: a good share of people think GenAI is magic and that with a single drag and drop of PDF documents, a conversational agent is capable of replacing a complete team of 15 people. It’s possible, but it still takes a bit of engineering 👨‍🔧 A surreal exchange with a client who tells me exactly that, and to whom I reply: “if we’re capable of replacing your whole team in 3 clicks, I think you have a problem”.

Tailor vs Ready-to-Wear

Lots of SMEs/mid-caps want to deploy GenAI to improve their processes / take the load off their teams, very often on use cases common to other companies. If creating a genAI solution remains very affordable and fast (see the first point), how can a company like Coddity sell?

The operational ROI is immediate, but selling our clients the complete solution isn’t worthwhile for them (amortization and maintenance). The alternative is to sell the setup as SaaS. It’s appealing, but it means turning ourselves, on 1 not-very-expensive product, into a software vendor and amortizing the investment by going out to find other clients. It can work on 1 product, but it isn’t scalable to others, potentially very different ones, all the more so because selling custom services or a product are not at all the same thing (and I know what I’m talking about).

A prospect tells us that his external regulatory-watch consultant is going to retire and asks if we can use AI to replace her for less money. Yes we can, but setting this solution up and running it would cost him between 10 and 15k, after digging into the subject. Obviously it becomes unprofitable for him to buy at 15k plus the maintenance. A subscription at 1000 € a year, on the other hand, is conceivable; all that’s left is to find a few more clients.

Inertia

The last point is classic in software publishing (custom vs SaaS), but in my view the cost gaps are much smaller in AI, which opens up new perspectives. Let’s take a step back. If developing a regulatory-watch AI for the industrial sector is ultimately “fairly simple”, a specialized company like Dekra can / will be able to / is going to do it. They have the resources and the knowledge to do it, if they so choose (remember Kodak?).

If this AI lets 1 regulatory consultant work like 4, what is a mastodon like Dekra going to do with the other 3? While they’re bogged down on that question, any bootstrapped startup of a few people will have a clear field to take a non-negligible market share, with a selling price and costs much lower than Dekra’s.

It’s in that sense that I bring up the Onclusive affair.

Conclusion

You want to make a fortune in 2024/2025? Look into the sectors that consume a lot of brain time. I think that’s a good vein!