Build Your Own AI Support System: What to Customize and What to Leave Alone
A custom assistant needs more than a model that sounds like your brand. Follow a return request through knowledge, rules, permissions and confirmed actions to decide where customization earns its place.
The Short Answer
Define the customer task, then separate language behavior, approved knowledge, business rules and authorized actions. Use current services for changing facts and software for permission checks. Customize model behavior only when a specific measured failure remains after the information and workflow are correct.
Key takeaways
- Custom service does not require training a new base model.
- Facts, behavior and authority need different update paths.
- An explanation, an eligibility decision and a submitted request have different finish conditions.
- Evaluate capabilities and critical failures separately from average completion scores.
- A tool should expose a bounded job, not unrestricted customer data.
- Include maintenance ownership and failure recovery in the architecture decision.

When a founder says, “We need our own AI customer service model,” I first want to know what the current assistant cannot do. Does it misunderstand the customer? Does it lack the right policy? Is it looking at yesterday’s stock? Can it see an order that the visitor has not proved they own? Those failures can produce equally unhelpful replies, but they belong to different parts of the system.
A custom support service rarely becomes dependable by putting every company fact into a model. It becomes dependable when the business decides where each kind of information belongs, who can change it, which actions are permitted, and how the assistant behaves when a required component is unavailable. The language model is one component within that arrangement.
This guide develops an architecture for a fictional bag company, Fern & Field. The requirements, test results and workload estimates are constructed examples, not DTCUP customer results or product promises. The purpose is to make a practical design decision: which parts should the company customize, and which responsibilities should remain outside the model entirely?
1. Define what “custom” needs to mean
There are several different forms of customization. A branded interface changes how customers encounter the service. Instructions change how a model approaches an answer. Retrieval supplies relevant material during the conversation. Business tools let software read or change a permitted record. Fine-tuning changes model behavior using additional training examples. Training a base model is a much larger undertaking that most support requirements do not demand.
These choices are not interchangeable levels on a single ladder. A company can have an excellent custom order-status workflow while using a general-purpose model. Another company can fine-tune a model and still have poor order-status answers because its shipment data is stale. The degree of model customization is therefore a weak measure of how well the service fits the business.
My preferred definition is operational: a custom assistant reliably completes the specific tasks the company has agreed to support, within its information and permission boundaries. Its advantage should be visible in those tasks. The company owns the task definitions, approved knowledge and acceptance criteria even if it buys much of the underlying software.
Fern & Field initially asks for an assistant that “knows the brand and handles returns.” That sentence conceals at least four jobs: explain the public return policy, identify a customer’s order, determine whether an item meets the recorded conditions, and submit a return request. The first job may need only approved knowledge. The others require identity, current records, a policy decision and potentially a state-changing operation.
Separate those jobs before estimating development. Otherwise a demonstration that answers a policy question may be mistaken for a finished returns service. The missing work appears later as manual exceptions, privacy problems and promises that the warehouse cannot honor. A clear scope makes the estimate more honest and the first release easier to judge.
If the main decision is whether to buy, configure or build the overall service, use the existing AI support software decision guide. Here the question is narrower: once a service is being designed, where should its intelligence, facts and authority live?
2. Write a contract for one customer request
Start with a real task stated in the customer’s language. Fern & Field chooses: “I bought the commuter bag last month. Can I return it, and what should I do next?” A successful service must not merely mention the return page. It should establish which answer is possible from the information available and take the customer to the appropriate next step without inventing eligibility.
The request contract records the task, required inputs, allowed sources, permissible output, possible actions and stopping conditions. For the public explanation route, the assistant needs the market and applicable policy version. For a personalized route, it also needs a verified session and an authorized order reference. A claimed purchase date in chat is context to investigate, not a replacement for an order record.
Write the observable finish line. An explanation can finish with the relevant conditions and a working next-step link. An eligibility check finishes with a decision from the approved rules service, including any unresolved condition. A submitted request finishes only after the destination confirms its receipt and supplies a reference. These are different outcomes and should not share an ambiguous “done” flag.
Now write the forbidden result. The assistant must not confirm a refund, guarantee acceptance of an inspected item, or reveal a record outside the verified customer’s access. This is not a list of frightening edge cases. It is the other half of the product definition: what the service does not have authority to decide.
The contract should also name the owner of each judgment. Support approves how the conditions are explained. Operations owns the return workflow. Engineering owns record access and execution. The business owner approves policy changes. The model provider supplies language capability; it does not decide what Fern & Field owes a customer.
This exercise often exposes a process gap before it exposes a model gap. If two staff members cannot agree which purchases qualify, training a model on their replies will not resolve the disagreement. The company first needs a rule it is prepared to apply. Automation can then make the rule easier to use consistently.

3. Give six layers different responsibilities
I would draw six layers for this service: the conversation interface, the language model, approved knowledge, business rules, authorized tools, and the operational record. The point is not to create six products or six engineering teams. A small implementation can combine several layers in one application while preserving their responsibilities.
The interface collects the message and presents the result. It should show whether the customer is signed in and whether a request is still processing. The model interprets the question, identifies missing information and drafts understandable language. It should not become the only place where eligibility logic or permissions exist.
Approved knowledge contains explanations that people can review: care instructions, return conditions, product distinctions and service processes. Business rules turn verified inputs into permitted decisions. Authorized tools retrieve the minimum current data or execute a bounded action. The operational record connects the request, evidence, decision and outcome so the team can investigate what actually happened.
Consider a customer asking whether a bag fits a particular laptop. A maintained compatibility table may provide the answer. The model can explain the relevant dimensions and ask for the laptop model if necessary. It should not estimate an exact fit from a photograph when the approved source does not establish it. That is a knowledge and evidence boundary, not a reason to invent a compatibility tool.
For “Where is my parcel?” the center of gravity changes. A document describing typical shipping time cannot establish the state of this shipment. An authorized tool must retrieve current information, including the timestamp and any limitations. The model translates that record into a clear explanation while preserving the distinction between a recorded event and a promised arrival.
These responsibilities also explain why a more autonomous agent is not automatically a better architecture. Anthropic distinguishes predefined workflows from systems in which the model directs its own tool sequence. For a narrow support task, a known sequence with explicit gates can be easier to operate. Add flexible planning when the task genuinely requires it and when its added behavior can be evaluated.
4. Put facts where their rate of change can be managed
A useful architecture question is: how quickly must this information change, and how would we prove that the old version stopped being used? A brand’s preferred writing style may change rarely. A return condition may change occasionally and depend on purchase date. An order’s fulfillment state may change during the conversation. They should not all follow the same update process.
Fern & Field puts stable explanation material in a reviewed knowledge collection. It records the product and market scope, effective dates and approval owner. A changed paragraph is published through that collection rather than buried in a developer’s instruction string. The assistant receives the applicable material when answering, with enough source information for the answer to be checked.
Current order and stock facts stay behind business services. The assistant requests them when necessary instead of treating an exported spreadsheet as live. The service response distinguishes unavailable data from zero stock or no shipment. That distinction matters: a connection failure must not be translated into a confident commercial statement.
Customer context also needs its own lifecycle. A delivery address attached to one order should not become a permanent preference. A customer’s statement that the bag is a gift may matter in this conversation without redefining their future purchase intent. Store only context that has an explicit purpose and a correction route; do not make memory the default destination for everything a customer says.
Behavioral guidance belongs in versioned instructions or, where justified, a trained behavior layer. Examples include answer structure, how to ask a concise clarification, and how to distinguish an explanation from a completed action. These instructions can tell the assistant how to use an approved policy, but the policy itself should remain independently reviewable.
The practical advantage is a smaller change radius. Updating a return period should not require retraining the entire assistant. Changing the model should not silently rewrite the return period. Replacing a shipment provider should not require revising every explanation about product care. Separate lifecycles reduce the number of unrelated things that must be retested together.

5. Decide what retrieval and fine-tuning are each meant to improve
Retrieval supplies selected information at the time of a request. Fine-tuning adjusts a model using training examples. For a support team, the useful distinction is between information the assistant needs to consult and behavior it needs to perform reliably. This is a design heuristic, not a claim that the two techniques can never overlap or be combined.
Suppose the assistant repeatedly gives last year’s return condition. Before considering training, inspect whether the current policy was approved, indexed, selected and supplied. If it never reached the answer, the problem is upstream. A trained model cannot be expected to infer that a missing business document changed yesterday. Repair the information path and test the same request again.
Now suppose the correct policy is always present, but the assistant regularly writes long, confusing replies that mix conditions with actions. Clearer instructions and a few carefully chosen examples may improve the behavior. If a stable, important behavior still fails across a representative test set, additional model adaptation becomes a reasonable experiment rather than a reflex.
A fine-tuning proposal should specify the behavior to improve, the available reviewed examples, the comparison model and the acceptance criteria. “Learn our business” is too broad. “Produce this structured classification for these defined support intents, including an unknown category” is testable. Even then, the classifier’s output needs validation before it controls a consequential action.
The training examples must reflect desired behavior rather than merely historical behavior. A reply that pleased one customer may have granted an exceptional remedy. A confident response may contain an outdated fact. A large archive is not automatically a high-quality training set, and a smaller carefully governed dataset may better describe the intended task.
OpenAI’s model optimization guidance emphasizes evaluating performance, supplying instructions and context, and measuring any tuning against representative tests. Treat this as a measurement principle; supported models and platform workflows change. The architecture decision should survive a supplier changing its interface or deprecating a particular optimization feature.
I would not fund model adaptation merely to avoid maintaining knowledge. It creates a different maintenance obligation and does not remove the need for current sources, record permissions or action confirmation. The relevant question is whether it improves a persistent behavioral failure enough to justify that obligation after simpler repairs have been tested.
6. Use a small comparison to locate the expensive problem
Fern & Field creates a constructed evaluation with 80 cases: 40 public product or policy questions, 25 authorized order-status tasks, and 15 return-request tasks. Expected outcomes are written before the candidate systems are compared. A case counts as complete only when every required condition for that task is met, not when the final sentence sounds helpful.
Three hypothetical candidates illustrate how to read the evidence. Candidate A uses instructions and selected knowledge but has no live order tools. It completes 31 of 40 public questions, none of the 25 live order tasks, and none of the 15 submission tasks. Its 31 total successes do not mean the language model is bad; it lacks the machinery required for most of the workload.
Candidate B adds authorized order retrieval and a bounded return workflow, without changing the base model. It completes 34 public questions, 22 order tasks and 12 returns: 68 of 80. Candidate C adds a behavior adaptation to B and completes 36 public questions, 22 order tasks and 12 returns: 70 of 80. These are invented teaching results, not an expected performance uplift.
The comparison suggests two different development questions. B’s large improvement comes with capabilities A did not possess. C’s two additional completions may justify investigation, but the team should inspect which cases changed, whether new failures appeared, and whether the improvement persists on held-out cases and repeated runs. Two additional successes are not enough by themselves to settle the investment.
Do not compare only the totals. If C fixes three confusing explanations but creates one unauthorized disclosure, the net gain of two is commercially misleading. Critical failures need separate release conditions. The relevant unit is the customer task and its consequences, not a single accuracy percentage averaged across incompatible outcomes.
The comparison also needs a capability label. A could be perfectly suitable for a public FAQ assistant even though it fails personalized tasks outside its scope. B may be the right choice for the selected service. C should earn its place through a specific behavioral improvement. Scope is part of the result, not a footnote added after scoring.
The AI support test-set guide explains how to protect held-out cases and compare releases. Here, the important architecture lesson is to change one meaningful layer at a time where feasible. Otherwise the team cannot tell whether it needs better information, a working integration or a different model behavior.

7. Keep business authority in a service people can inspect
A policy explanation and an eligibility decision should agree, but they do not have to be produced by the same mechanism. Fern & Field can let the model explain the return conditions while a maintained rules service decides whether a recorded order meets the conditions it can verify. Anything requiring physical inspection remains unresolved until that process occurs.
This separation gives policy owners something concrete to review. For a fictional rule, they can inspect the permitted market, purchase-date boundary, product exclusions and exception route. The decision response can state eligible, ineligible or more information required, with a reason code. The model then expresses that result without quietly changing the business rule.
A rule written only in a prompt is difficult to treat as the sole enforcement mechanism. The model may misread the input, confuse two dates or follow an irrelevant instruction in retrieved text. Deterministic checks are not infallible either, but their inputs, branches and tests can be made explicit. The goal is to put enforceable decisions in a place designed to enforce them.
Allow the business to express uncertainty. A rule service should not force every case into yes or no when required evidence is absent. An unknown product condition, incomplete order record or conflicting policy version may require review. A three-state response is often more truthful than a confident Boolean that hides missing information.
Keep version references in the decision record. If a customer disputes an answer next week, the team should be able to identify the policy version and inputs used at the time. Replaying today’s rule against yesterday’s transaction may produce a different result. That does not establish what the assistant originally did.
Exceptions need an authorized route, too. A support manager may approve a one-off remedy without changing the public policy. Record that decision as a case-specific authorization. Do not let the assistant generalize it into a new promise for everyone, and do not train future behavior from the exception without preserving its limits.
8. Design tools around bounded jobs, not unlimited database access
A useful tool is not simply a connection to a large system. It is a narrow operation with clear inputs, permission checks and an understandable result. “Retrieve shipment events for an order this session can access” is a much better starting point than “search all customer data.” The narrower description makes both testing and incident investigation more concrete.
The server should determine the workspace and authenticated identity from the trusted session. The model can propose an order reference, but it should not choose another customer’s identity or a different workspace as an argument. The service checks that the requested record belongs within the permitted scope before returning any personal information.
Return only the fields needed for the task. An order-status reply may need the shipment state and tracking link, not the entire payment history or internal risk notes. Reducing the returned data also makes the answer easier to inspect: the model has fewer irrelevant facts to confuse with the customer’s question.
For state-changing work, separate preparation from execution where the task warrants it. The assistant can gather a return reason and present the proposed request. The application can obtain the required confirmation, validate the current state, then submit a bounded operation. The model’s narrative about what should happen is not the execution itself.
Design the result for ambiguous outcomes. If a request times out after being sent, the service may not know whether it was accepted. Repeating it blindly could create a duplicate. The workflow needs a request identifier and a way to check the destination’s state. Until that check finishes, the customer should see that confirmation is pending rather than a fabricated success or failure.
These controls belong below the conversational surface. A customer asking politely, angrily or with a pasted instruction should not change record access or permitted operations. The assistant can adapt its tone while the service applies the same authorization and execution rules. That is the difference between a flexible conversation and an uncontrolled business process.
9. Walk through the return request from message to receipt
Return to the original customer message. The application first establishes whether the customer is signed in through the supported account process. If not, the assistant can explain the public policy and direct the person to verification. It does not ask the model to decide whether the message sounds like a genuine owner.
Once the customer is authorized, the order tool retrieves the relevant order and minimum product details. Suppose the account has two commuter-bag purchases. The assistant asks which purchase the customer means using an approved, privacy-conscious presentation. It should not silently select the newest order because that is easier to process.
The workflow then requests the applicable rule decision. Assume the fictional result says the purchase is within the request period, but item condition still requires confirmation. The assistant asks the required condition question. It describes this as a request eligibility check, not a guarantee that the final return will be accepted after inspection.
When the customer confirms the details, the application shows a concise summary and obtains the required action confirmation. The execution service checks that the order has not already entered the same return process. It submits the request with a unique operation reference and waits for a verified response.
If the destination returns request R-204, the assistant can state that the request was received and explain the recorded next step. If the destination cannot confirm, the assistant says that submission is not yet confirmed and gives the supported follow-up route. Neither response should promise money or shipping events that the destination has not established.
The operational record contains the request identity, verified order reference, policy version, tool outcomes and final state. It does not need a hidden model reasoning transcript to be useful. What matters for review is the observable evidence and sequence of actions. The team should also minimize and govern any personal information retained in that record.
At this point, the custom value is visible. Fern & Field has connected its approved policy, account access and return process into one understandable service. The model helped interpret and explain. It did not become the order database, identity provider, policy authority or proof of execution.

10. Assign failures to the layer that can actually repair them
A common maintenance trap is to send every bad answer to the person who writes prompts. That person may be able to make the reply more cautious, but they cannot repair an inaccessible order service or decide which of two conflicting policies the business intends to honor. The result is a growing instruction document that compensates for unresolved operating problems.
Create a small responsibility map. A wrong approved fact goes to the knowledge owner. A correct fact that was not retrieved goes to the retrieval implementation owner. A supplied fact that was misinterpreted goes to the instruction or model evaluation owner. A wrong eligibility decision goes to the rules owner. A failed action goes to the integration owner.
Some failures cross layers. A tool might return an unclear status that the model interprets incorrectly. Repairing only the wording may leave the same ambiguity for another model. The better change may be a clearer response contract with explicit confirmed, pending and failed states, followed by instructions and tests that preserve those distinctions.
Require a reproducible record before choosing the repair. The team needs the task, relevant source version, permitted inputs, returned evidence and observed result. A screenshot of the final sentence is often insufficient. It shows the symptom but not whether the underlying evidence was missing, wrong, stale or ignored.
The wrong-answer diagnosis guide provides a detailed method for that investigation. In the architecture review, ask a simpler question: can the person responsible for each layer see enough evidence to repair it without guessing what the other layers did?
This also creates a useful commercial test for vendors and developers. Ask them to demonstrate one failed case, explain which component owns the failure and show the correction path. An impressive successful conversation is welcome. An understandable failure and recovery process is stronger evidence that the service can be maintained after launch.
11. Price the maintenance work the design creates
Every customization creates a future change surface. A custom instruction set must be reviewed when task scope changes. A retrieval pipeline must cope with source updates and document failures. A tool integration must survive changed fields, expired access and destination outages. A tuned model adds training-data governance, evaluation and replacement decisions.
For a constructed planning exercise, Fern & Field lists twelve approved knowledge changes, two business-rule changes and one integration change per month. These are workload assumptions, not benchmarks. The question is how each candidate architecture handles those changes and how much coordinated work it requires.
Suppose a separated design takes 30 minutes to review and verify each knowledge change, two hours per rule change and four hours for the integration change. That is six plus four plus four, or fourteen hours. It is not the entire operating cost; it is the explicitly listed monthly change workload.
Now consider a tightly coupled design in which all fifteen changes require a two-hour combined release review because knowledge, rules and tool behavior are entangled. Its listed workload is thirty hours. The sixteen-hour difference is only a scenario for discussion. Actual timings, shared release efficiencies and failure rates must be measured before anyone claims a saving.
The point is not that separation always reduces work. A poorly designed collection of services can create more coordination than it removes. The useful question is whether an ordinary business change can be made by its owner without reopening unrelated parts of the system. If not, the company is buying an ongoing dependency, not merely a feature.
Also examine scarce skills. Four hours of a specialist developer’s time may be harder to obtain than six hours spread across trained support editors. A design that looks inexpensive at average hourly rates may still fail operationally if only one person can repair it. Include review access, documentation and backup ownership in the architecture decision.
Do not count every reduction in handling time as immediately recoverable cash. Staff may use the time for better service, and fixed commitments may remain. The architecture case should distinguish capacity released, actual spending avoided and additional maintenance required. That makes later evaluation less vulnerable to an attractive but imaginary savings number.
Include information handling in that maintenance estimate. Draw where a customer message, retrieved order field and final reply travel: the website, support application, model service, tool provider and monitoring system. For each destination, record the purpose, the access owner and the actual retention configuration. Hosting the chat interface yourself does not mean every part of the conversation stays on your server.
For example, a developer investigating an order-status failure may need the tool’s error category and request reference, not the customer’s full address. A knowledge editor checking a product answer may need the cited document version, not the entire conversation history. Designing different review views can reduce unnecessary exposure while preserving the evidence each owner needs. It also avoids making a large unrestricted transcript archive the only debugging method.
Ask how corrections and deletion requests propagate through the systems you actually use. A removed knowledge item may remain in an index or a cached answer until those copies are refreshed. A customer record may have separate handling in the commerce system and the support platform. These are implementation questions to verify with the responsible owners and applicable requirements, not assumptions the language model can resolve. Record any unresolved dependency before enabling the affected task.
12. Make the model replaceable without pretending it is interchangeable
A replaceable model is a useful goal. It means the company can evaluate a different model while keeping its task definitions, approved knowledge and business permissions intact. It does not mean every model will produce identical answers or understand the same tool descriptions equally well.
Keep the application contract stable where practical. Define the expected intent fields, source references, tool arguments and final outcome states. Validate those fields in software. When the model changes, run the same task evaluation and inspect behavior at the boundaries. A syntactically valid response can still select the wrong order or misstate a policy.
Preserve reviewed examples separately from provider-specific configuration. The company should be able to explain what a correct return conversation looks like without opening one vendor’s console. Similarly, approved knowledge should remain exportable in a form people can inspect, including ownership and applicability rather than only a collection of embeddings.
There may still be genuine switching costs. Different providers support different tool interfaces, context limits, adaptation methods and operating terms. Document the dependency instead of hiding it behind an abstraction that claims universal compatibility. A modest adapter with clear limitations is more useful than a grand promise that no migration work will be needed.
Practice a restricted fallback. If the order service is unavailable, can the assistant continue answering public questions while clearly declining personalized status? If the model is unavailable, can the customer still reach the supported account page or human queue? A fallback should preserve the remaining useful service without pretending that missing capabilities are functioning.
The exit plan should include open conversations and outstanding actions, not only exported documents. An unconfirmed return request cannot be abandoned because the chat supplier changed. Record where those requests live, who resolves them and how the next interface discovers their state. Business continuity is part of architecture, even when the interface itself is easy to replace.
13. Write an architecture decision a future operator can understand
A useful decision record is short enough to read and specific enough to challenge. For Fern & Field, it begins with the supported jobs: public product and policy explanations, authorized shipment status and bounded return-request submission. Refund approval and exceptional remedies remain outside the initial automated scope.
The chosen design uses a general-purpose model with versioned instructions, a reviewed knowledge collection, a deterministic eligibility service and narrow order tools. The reason is task ownership and change frequency. The company has no demonstrated behavioral failure that yet justifies a tuned model, although the design leaves room to evaluate one later.
The record names evidence that would change the decision. Persistent formatting or classification failures after instruction improvements could justify adaptation. Repeated retrieval misses could justify a different search approach. A task requiring unpredictable investigation might justify more flexible planning. Each trigger points to a specific deficiency rather than a desire to follow a technology trend.
It also records unresolved limitations: incomplete product compatibility knowledge, unsupported markets, the handling of destination timeouts and the staffed exception route. These are release boundaries. They should not disappear from the customer service merely because the architecture diagram looks complete.
Finally, assign owners and a review event. A new market, new permitted action, material policy change or model replacement should trigger the relevant checks. A routine knowledge correction should have a smaller, appropriate verification path. The record makes clear which changes require a broad review and which can remain local.

14. Questions worth settling before development
Do we need to train our own model to have a distinctive service?
No. Distinctive service can come from accurate knowledge, thoughtful clarification, reliable account access and a well-designed remedy process. Training becomes relevant when a specific stable behavior needs improvement and an evaluation can demonstrate the benefit. It is not a prerequisite for owning the customer experience or operating a custom workflow.
Can we put all the rules and facts in one large prompt?
You can supply substantial context, but that does not settle ownership, freshness or enforcement. A prototype may begin with a small approved context. As the service grows, identify which facts change independently and which permissions must be enforced by software. The question is whether the arrangement remains correct and maintainable, not whether the text technically fits.
Should the assistant automatically learn from every conversation?
Not as an unrestricted rule. Customer messages can be incorrect, incomplete or personal, and staff replies can contain exceptions. Use conversations to identify candidate improvements, then review what should become knowledge, an example or a process change. The ticket-to-knowledge method explains how to preserve that distinction.
How do we know the architecture is ready?
You should be able to follow a supported task from request to verified outcome, identify the authority behind every consequential decision, and explain what happens when each required component fails. Demonstrate those paths through tests and a limited operating release. A diagram, integration list or successful demonstration alone cannot establish readiness across the intended workload.
My recommendation is to customize the parts that carry your actual service promises: task boundaries, approved knowledge, business decisions and the path to a completed outcome. Keep the model’s job clear, measure its behavior and change it when evidence justifies the change. That produces a system the business can understand and improve, rather than a model that everyone hopes has somehow learned the company.
Sources
Still have a question?Ask the DTCUP Community ↗

