Building a Customer Support Chatbot with LLMs: Beginner Guide

Most support chatbots fail for a boring reason: they try to answer everything. A useful one does three things well. It works out what the customer actually wants, it answers only from material you trust, and it hands the conversation to a person at the right moment. Everything else — model choice, interface, analytics — is secondary. Here is how to build that assistant, in the order you should build it.

Start With a Scope You Can Write Down

Before you open any tool, spend an hour in your support inbox. Pull the last 200 tickets and sort them into groups. You will usually find that a handful of topics covers most of the volume: order status, returns, password resets, delivery delays, invoice copies.

Write that list down and pick the two or three groups you will handle first. Then write the opposite list — the things the assistant must never touch. Payment disputes, account deletions, anything involving safety, and anything that would need legal or financial advice belong with a human from the start. Having that boundary written down stops you from arguing about it later, when someone asks why the bot "refuses" to help.

Route the Conversation Before You Answer It

An intent is a short label for what the customer wants: track_order, start_return, reset_password, cancel_subscription. Keep the list between six and twelve. More than that and the labels start overlapping, which makes routing noisy and hard to debug.

A common pattern is two steps. First a small routing call that returns one intent plus a confidence score. Then a handler for that intent, which is where the real answer gets assembled.

For example, "I was charged twice for the same order" should route to billing_dispute, not order_status, even though both mention an order. That single mapping decides whether the customer gets a refund link or a human being.

Always keep a fallback

Plenty of messages will not fit a clean bucket. When the router is unsure, the assistant should ask one clarifying question with two or three concrete options rather than guess. "Is this about a delivery, a return, or a payment?" beats a confident wrong answer every time.

Give It Sources It Can Actually Use

Free-form answers are where support bots get into trouble. Instead, give the assistant a small library of approved pages — help centre articles, policy documents, internal runbooks — and make it answer only from passages retrieved from that library.

• Split documents by heading, not by character count. A section about "return window for furniture" should stay whole, so the model sees the qualifier alongside the rule.
• Keep the heading and page title attached to every chunk. Retrieval often depends on that context more than on the body text.
• Instruct it to say it does not know when the retrieved passages do not cover the question, and to offer a handoff instead of improvising.
• Put a review date on every document and retire anything older than your policy cycle.

When you test later, most wrong answers will trace back to a chunking problem or a stale page — not to the model.

Make Escalation a Feature, Not a Failure

Handing off is not an error state. It is the part of the design that keeps trust intact. Decide in advance which events trigger it:

• The customer asks for a person directly.
• Tone turns frustrated, sarcastic, or abusive.
• The same question has been answered twice without resolution.
• The topic touches money, account security, or anything on your off-limits list.
• Routing confidence stays low after one clarifying question.

When you hand over, pass context: a short transcript summary, the detected intent, the account or order reference, and what the assistant already tried. A customer who has to repeat everything will be angrier than when they started. Tell them what happens next — "I'm passing this to a colleague; they'll reply within one working day" — and give them a way to follow up if the queue is long.

A good handoff feels like a colleague picking up the conversation, not a system restarting it.

Never end a conversation on a dead end. If nothing can be resolved automatically, the last message should still offer a next step.

Test With Real Questions

A handful of friends typing questions is not a test. Build a set of 50 to 100 real messages from your inbox, label the correct intent and correct outcome for each, and run the whole set after every change.

Score three things:

1. Routing accuracy — did each message reach the right handler?
2. Retrieval hit rate — for answerable questions, was the right passage found?
3. Escalation precision — when it handed off, was that correct? When it answered, should it have handed off instead?

Then add awkward cases on purpose: typos, two questions in one message, a question in another language, off-topic requests, and messages that try to talk the assistant out of its rules. These break more assistants than ordinary traffic does.

Review a random sample of live transcripts every week. Every mistake becomes a documentation fix, a new intent, or a new test case. That loop is the whole job.

A Sensible First Two Weeks

Days one and two: pick the scope, write the off-limits list, gather 200 tickets. Days three to five: build the intent list and the routing step, then test against your labelled set. Week two: load the knowledge base, wire up escalation, and run the assistant internally with staff asking real questions before anyone outside sees it.

Release to a small share of traffic first, and keep a human reading transcripts daily. Fix documentation before you tune prompts — most complaints are content problems wearing a technical costume. If a conversation involves security, payments, or anything with legal weight, route it to a person and let them own it.

Photo: Yan Krukau / Pexels