Case study
A major Italian bank uses an LLM agent, built on Azure OpenAI, RAG and LangChain, to route more than 1 million internal help desk tickets a year into 400+ categories. It picks the right category 81% of the time, against 75% for human routing, and average resolution time went from 5–6 days to 2–3.
Employees send the bank's internal help desk over a million requests a year, written as free conversation. Each one has to go to one of 400+ categories in a hierarchy. Sorting them by hand was slow and inconsistent. Tickets sent to the wrong team bounced between teams, and resolution took 5–6 days on average.
An LLM agent reads the conversation, uses RAG to pull the definitions of the relevant categories, and assigns the ticket to the right node of the hierarchy. I was tech lead of a team of 4: I coordinated development and was the bank's technical contact.
If your team routes requests (tickets, emails, claims) across hundreds of categories, an LLM agent that looks up your taxonomy can route them more accurately than people do by hand. A wrong assignment costs days, so better accuracy shows up straight away in resolution time. To start, you need a labelled history to measure against and a cloud setup your compliance team already approves.
Here it did. On a hierarchy of 400+ categories the agent reached 81% accuracy, while human routing reached 75%. The gain comes from retrieval: for each ticket the agent pulls in the definitions of the candidate categories, so it doesn't rely on what the model remembers about an internal taxonomy.
This system handles more than 400 categories in a hierarchy, at over 1 million tickets a year. Putting every category into every prompt isn't practical, so RAG retrieves the relevant branches of the taxonomy and the model picks among them.
In this project the models ran on Azure OpenAI inside the bank's EU cloud environment, and sensitive data never went into the logs. With sensitive data, that's where you start: the model runs where your data already lives, under contracts your compliance team has already approved.
After the agent went live, average resolution time fell from 5–6 days to 2–3. In a help desk, most of the delay comes from tickets bouncing between the wrong teams. When the first assignment is right, that delay goes away.
Azure OpenAI, retrieval-augmented generation, and LangChain and LangGraph for orchestration, packaged as a Docker container on Azure Functions. A team of 4 built it, and I was tech lead.
Related
Next step
Is your team still sorting requests by hand? We can work out which ones an agent can route.
Tell me the workflow. We scope it on a free call, then a working version is live within 10 days.
Book a scoping call