AI Ops
An AI feature needs ongoing oversight after launch
An AI feature left without oversight gradually answers worse and costs more. We set up quality measurement, a change process and documentation, so the team spots a problem before users do.
Written by Radosław Kołacki Updated
Trusted by
When is AI Ops worth it?
The company has an AI feature or product in production, built by Squares or by its own team, and someone on the team is responsible for it still working well six months after launch.
Models, prompts and content change, so answer quality and costs need to be measured continuously.
A Product Owner who accounts to the management board for the cost of an AI feature should have those numbers to hand.
Who is AI Ops for?
- AI features are already in production: assistants, chatbots, content generation or RAG, meaning answers grounded in the company's own knowledge base
- AI was shipped quickly and there is no process yet for overseeing quality and costs
- A Product Owner is responsible for an AI feature built by an external supplier
- A medical or health product, where a wrong AI answer has real consequences for the user
How it works
How does AI Ops work step by step?
01
Quality monitoring
We measure answer quality, the rate of errors and hallucinations (answers the model makes up), costs and response speed.
02
Evaluation process
We test model or prompt changes before they reach production and compare the results with the previous version.
03
Change rules
We agree who may change prompts and the model, and who approves a change before release.
04
Oversight and documentation
We record how the feature works, who approves changes and how the team responds to errors.
05
Regular reviews
At set intervals we assess with the team whether the feature still does what it was designed to do.
Oversight dashboard
Every conversation with an AI feature turns into data
Instead of guessing whether the assistant answers well, we look at the numbers: how many issues it resolved, where users struggled and which conversations need a human to step in.
-
Resolution
How many conversations end with the answer the user was looking for, and how many stay open.
-
User effort
How hard it was to reach an answer, on a scale of 1 to 5, plus the conversations where the AI goes round in circles.
-
Sentiment and risk
How the user felt at the end of the conversation and whether there was a sign of frustration or intent to leave.
-
Topics and intents
What people ask about most often and which topics the feature handles worst.
The team can be confident that the AI feature in production still works well, its costs are under control, and changes are tested and documented.
How much does AI Ops cost?
from PLN 1,500 net/month
Oversight costs from PLN 1,500 net per month. Oversight is billed as a fixed monthly fee, sized to the scale of the AI feature and the level of risk. We start with a review of the current state of the feature in production.
We give the monthly figure after a call about the AI feature and a review of the current state of the deployment. We write back within a few hours.
„A great collaboration for over 2 years. The team is professional, punctual and genuinely committed to the development of the project. The app is polished and the communication is excellent. Highly recommended.”
Daily Effect
Google review, translated from Polish · Full review
Work
How it works in practice
Daily Effect: an AI assistant with a conversation dashboard
The dashboard shows answer effectiveness, topics and sentiment of conversations. Over more than 6 months the assistant resolved 72.7% of issues without a human agent.
Case study
Tropokolagen.pl: an AI assistant overseen by the team
In the dashboard the client's team sees sessions, answer effectiveness, sentiment and the most frequent topics, and manages the starter questions itself.
Case study
Prose Captain: an AI assistant platform
We built our assistant platform on RAG and a vector database, and two deployments taught us what has to be measured after launch. Oversight as a separate service is new.
Case study
Why Squares
We design and build AI features, so we know where they break after launch: in the model, in the interface and in user trust.
Questions
Frequently asked questions
What is AI Ops?
AI Ops is ongoing oversight of an AI feature after launch: measuring answer quality, cost and speed, and running the process for changes. A feature left without oversight answers worse and costs more over time, because models, prompts and content change. The aim is for the team to notice a problem before the user does.
How to measure the answer quality of a chatbot or AI assistant?
The quality of an AI assistant is measured on conversations: how many cases it resolved, how much effort it took the user to reach an answer on a scale of 1 to 5, what the sentiment was at the end, and which topics the feature handles worst. Added to that are the rate of errors and hallucinations, meaning answers made up by the model, plus cost and speed.
When does an AI feature need ongoing oversight?
Ongoing oversight is needed when an AI feature runs in production and nobody measures the quality of its answers or its cost. This applies to assistants, chatbots, content generation and RAG, meaning answers based on the company's own knowledge base. It is most urgent where AI was shipped fast, or where a wrong answer has real consequences for the user, as in medical products.
How much does oversight of an AI feature cost?
Oversight is billed as a fixed monthly fee. The amount depends on the scale of the AI feature and the level of risk, so we give a figure after reviewing the current state of the deployment. The scope is matched to scale, from a single assistant to a larger system with several AI use cases.
What does oversight of a live AI feature include?
Oversight has five parts: monitoring answer quality, cost and speed; testing model or prompt changes before release; rules on who makes changes and who approves them; documentation of how the feature works; and regular reviews with the team. The effect: costs are under control and every change is tested and documented.
Is AI Ops the same as MLOps or model training?
No. MLOps is about infrastructure and model training; AI Ops is about the product and quality layer. We measure how the feature works for users, what it costs and how model and prompt changes are introduced. We do not run model infrastructure or train models. We work on the live feature, alongside the client's engineering team.
How is AI Ops different from AI product design?
AI product design happens before launch: it defines what the feature should do and how it shows a result, uncertainty and an error. AI Ops starts after launch and checks whether the feature still does what it was designed to do. The first service ends with an interface design; the second is steady monthly oversight.
How does the work with the client's engineering team look?
We work alongside the client's engineering team: we are responsible for quality measurement and the change process, and the split of implementation work is agreed at the start. Oversight covers features built by Squares, the company's own team or an external supplier. The first step is a review of the current state of the live feature.
Are there case studies for this service yet?
AI oversight as a separate service is new and does not have its own case study yet. It builds on AI assistants deployed for two clients, together with a dashboard for measuring conversations. At Tropokolagen.pl the client's team sees sessions, answer effectiveness, sentiment and the most common topics in the dashboard, and manages the starter questions itself.