AI Prototype
to Production
Your AI prototype works. Leadership likes it. Now, when you try to broaden its usage internally, or worse, show it to your customers, questions start to emerge.
Start with the demo
Add the controls
Test live use
You get software the business can depend on.
Where the questions come from
Maybe it’s from your technology stakeholders - Can we really show this to customers? Has it been thoroughly tested? What happens when it doesn’t work?
Maybe it’s from security - What happens when someone gives the model inappropriate or unexpected input? Have you tried prompt-injection attacks, jailbreaking or tricking it into calling tools it shouldn’t have permissions for, leaking PII or other confidential information?
Maybe it’s from finance - Whoa. Why has this LLM bill increased so much this month?! Why are we paying for those tokens when the system was supposed to avoid doing X?
Maybe it’s from the operational owner of the system - I can’t tell if we’ve improved performance since making that change. How come we couldn’t replicate that example from last week that broke it? How are we supposed to improve this thing without breaking it again?
In all of these cases, your team did a great job building the prototype but the system is nowhere near production-grade, let alone ready for the challenges of enterprise-scale.
Prototype versus production
In a prototype, you’ve shown that it works for a handful of curated examples.
In production, you need an evaluation harness which runs against your (test) dataset in CI, which can tell you whether your latest change has broken anything and then break the build.
In a prototype, you’ve been tweaking prompts ad hoc and keeping them in your head.
In production, you need versioned prompts which are managed in a structured way alongside your application. You need evaluations against them, both deterministic checks and model-as-judge, so you know whether your latest prompt has improved things.
In a prototype, you’ve been using a single provider with no fallbacks.
In production, you need routing and failover, caching, retries, timeouts, rate limits, budget management, cost controls and alerting, so you know why your monthly bill looks nothing like what you expected.
In a prototype, you’ve been copy-pasting examples in and out.
In production, you need observability so you can search through recent examples and reproduce any given example. You also need traces on every LLM or tool call so you understand exactly what happened.
In a prototype, you’ve been trusting that it does the right thing with user-provided inputs.
In production, you need to consider malicious inputs and defend against prompt-injection attacks, jailbreaking, preventing it calling tools it shouldn’t have permissions for, leaking PII or other confidential information.
In a prototype, you’ve been winging it on governance, assuming it’s not necessary.
In production, you need to understand the level of risk involved in the use-case, aligning to frameworks like the NIST AI RMF or regulatory requirements such as the EU AI Act and the guidance from regulators such as the ICO.
How do we help?
We specialise in taking prototypes like yours and making them safe and secure to use in anger. We know what it takes to get to enterprise-scale and can help you understand where you are today and what’s needed to improve things.
We can help with building evaluations which combine deterministic checks with rubric-based or model-as-judge evaluations, calibrating the model-as-judge against human-labelled examples, embedding them in CI, alerting when performance dips. We can help you understand what’s happening via tracing, provide guidance on permissions, help you understand the risks with your use-case, build out a register of AI use and perform risk-assessments to satisfy governance needs. We’ll tell you where we think it’s worthwhile rebuilding something, what’s involved in a rebuild and give you options for proceeding.
How does engagement with us work?
We start with a production review to understand where things stand today, digging into your repo, prompts, usage, testing, any examples it’s gotten wrong recently. This is a fixed-fee exercise with a defined scope. From here, we can offer an engineering retainer to work with your team to improve things, breaking down the work into discrete areas, such as improving performance, cost-optimisation, observability, evals, security, governance, etc. We’ll then scope each area, agreeing on success criteria and how we’ll demonstrate it, pricing each phase. We’ll dig deeper in areas where it makes sense, prototyping or developing ideas before integrating them back in. Throughout, we’ll be using your prototype as the specification, assuming that it already provides the right workflows, user interface and achieves the right outcomes when working.
Get in touch
Where possible, we’ll be building around and incorporating the prompts and approach that you’ve already taken.