In 2024 I worked at V75 Inc. as an NLP and generative AI engineer. We built agents on Jaseci that used named-entity recognition and retrieval, measured them with precision, recall and F1, and put them into the team's daily work. The systems and prompts belong to the company, so they stay out of this post. What follows is how I would build a support bot, at a level that gives nothing of theirs away.
Start narrow
A support bot should do a few jobs well and hand everything else to a person. Password resets and opening hours are good jobs for a bot. A billing dispute is a job for a person.
People here write English with Creole in it. They change topic halfway through a message, and they expect the other side to notice when they are upset. A bot that knows five intents will fail them all day, politely.
The order I would build it in
- Real transcripts. Label real conversations, not marketing copy, and remove personal details first. Even 500 labelled examples from your own customers beat guessing with none.
- A plain baseline. TF-IDF with logistic regression, or a small fine-tuned DistilBERT. Anything bigger has to beat it.
- Scores per intent. Report precision, recall and F1 for each intent. A few intents usually take most of the traffic, so accuracy alone hides the rare ones.
- Retrieval for policy. When the answer lives in a policy document, retrieve the passage and answer from it. A model left to answer on its own can make up a refund policy.
- Rules for handing over. Low confidence, an angry message, legal words, or the same question twice should open a ticket for a person.
- Logs. Put the bot behind an API (FastAPI is enough for a pilot) and keep every turn: what the customer wrote, the intent picked, its confidence, the reply, and how the chat ended.
- A weekly read. New product names, promotions and seasons change the language. Read a sample of the logs every week.
What to count
Count problems solved. A bot can close a chat without solving anything, and a deflection rate will still call that a win. Count finished tasks, how often the bot hands over, how long people wait compared with a human-only line, and what a person finds when they read a sample of the logs. Word-overlap scores such as ROUGE say little about whether a customer got help.
Who it affects
In a small market, a support bot touches real jobs. Keep a clear path to a person, hire locally for the hard cases, and know where your customers' data is stored. Build a bot for a problem you can define and measure, and skip it when the only reason is that bots are in fashion.