Our GenAI tech stack in production
Let’s talk GenAI tech (but not LLMs)!
I was sharing my Monday observations on the GenAI market, and notably the importance of a robust software foundation.
The fundamentals of a GenAI platform
A GenAI platform rests on fundamental elements: task orchestration (often asynchronous), agents with memory management, intelligent document parsing, a high-performance vector database, and fine-grained observability of LLM generation.
If you do a Google search, sorry, Perplexity, you’ll be offered LangChain or llamaIndex, 2 “popular” frameworks that have the elements cited above. In the end we set LangChain aside; its abstraction layer turned out to be more of a hindrance than anything else. A GenAI project is above all a project of flow orchestration and API calls, nothing that requires a framework.
That led us to develop our own technical foundation at Coddity, now proven in production.
👷♀️ System
The core is in Go, to develop chains and our agents: Go’s native handling of concurrency is perfect for LLM tasks, asynchronous by nature, aside from chat streaming.
We developed our own packages for managing agent graphs and memory, as well as a library of connectors to different LLM providers, integrating automatic fallback mechanisms in case of failure.
🛠️ Vector DB
We chose Qdrant. Beyond its performance and its scalability, its API is very well designed, with graph capabilities and its hybrid search with BM42. Its native UI is also a plus for debugging and monitoring.
📕 Parser
Parsing is often the invisible key to the performance of GenAI platforms that process proprietary data. Without high-performance parsing, “garbage in, garbage out” will be confirmed.
We opted for a pooled approach with a dedicated service on our servers, capable of efficiently handling nearly all the document formats we come across.
⚙️ Infrastructure
We use Clever Cloud for the orchestrator: it’s simple, high-performing, and French.
For use cases that require privacy or fine-tuning, we rely on Runpod, and in particular on their serverless, which lets you spin up GPU on demand, perfect for cost optimization with variable demand.
For LLM observability, after using a simple homegrown platform, we’re thinking of moving to Langsmith, which will require evolving the orchestrator so the flows can be integrated.
Conclusion
This technical stack, built on real production needs rather than on theoretical choices, lets us today launch POCs quickly while using a foundation ready for production.
For the non-technical people who made it this far, bravo!