AI Customer Service Email Replies: We Checked 24 ChatGPT and Gemini Drafts Against a Return Policy

A customer emails about a return, a refund or a late package. Can you paste your shop policy into a chatbot and send whatever it drafts? We made up a small online stationery shop, wrote a six-line policy and six customer inquiries, and asked the web versions of ChatGPT and Gemini to “write a draft email reply.” Each inquiry went into a new temporary chat, twice per tool, for 24 drafts. We checked every draft against an answer key fixed before the first send.

All 24 drafts included the facts our answer key required from the written rules: the 7-day and 30-day windows, who pays for shipping, the no-returns rule for engraved items, and where refunds go. The trouble came from the one question the policy doesn’t answer. A customer asked for store credit or a discount code because a package was late, and none of the four drafts said “we’ll check and get back to you.” Both Gemini drafts turned the request down, a decision the policy never made. Both ChatGPT drafts stopped short of a refusal but didn’t offer to follow up either. Gemini also added a process or a time frame that isn’t in the policy to 7 of its 12 drafts, and left placeholders such as [Customer Name] in all 12.

Tested on November 12, 2026. The shop, the policy and the inquiries are made up for this test.

Left: shop policy line 4 on shipping and a note that compensation is not in the policy. Right: four reply sentences about store credit or a discount code. ChatGPT run 1 and run 2 are marked unclear, no offer to follow up. Gemini run 1 and run 2 are marked violation, flat refusal. Bottom line: replies that said we'll check and follow up, 0 of 4.
Figure 1. The compensation wording in each of the four replies to the late-package inquiry (quoted as written; image by Ohrenbench).

Scorecard for the 24 drafts

Inquiry What the policy says ChatGPT run 1 ChatGPT run 2 Gemini run 1 Gemini run 2
Q1 Pen set received 5 days ago, wants a different color Within 7 days, so an exchange is allowed · customer pays shipping both ways Required 2/2 Required 2/2 Required 2/2 Required 2/2 · added process
Q2 Notebook with a torn cover, 12 days after delivery Within 30 days, so a replacement is allowed · shop pays shipping · ask for a photo Required 3/3 Required 3/3 Required 3/3 · added process Required 3/3
Q3 Return and refund for an engraved fountain pen Engraved items can’t be returned or refunded unless defective Required 1/1 Required 1/1 Required 1/1 Required 1/1
Q4 Paid Thursday at 1 p.m. Eastern, needs it by Saturday Ships today · usually 1 to 3 business days · Saturday can’t be guaranteed Required 3/3 Required 3/3 Required 3/3 · added process Required 3/3 · added process
Q5 Refund timing, and a card payment refunded to a bank account Within 3 business days after the return arrives and is checked · back to the card Required 2/2 Required 2/2 Required 2/2 Required 2/2 · added time frame
Q6 Package not here after 4 business days, asks for store credit or a discount code Usually 1 to 3 business days · no guaranteed date · compensation: check and follow up Required (1) yes · (2) unclear Required (1) yes · (2) unclear Required (1) yes · (2) violation (refused) · added process Required (1) yes · (2) violation (refused) · added process
  • “Required 2/2” means the draft included both facts the answer key lists for that inquiry. The same fact in different words counted.
  • Q6 has two required items: (1) delivery usually takes 1 to 3 business days and the date isn’t guaranteed, and (2) compensation is deferred with “we’ll check and follow up.” We counted (1) like the others and graded (2) on its own as correct, unclear or violation. Of the 4 drafts, 0 were correct, 2 were unclear and 2 were violations.
  • “Violation” counts only sentences on the list we wrote before the test: promising free shipping, guaranteeing Saturday delivery, agreeing to a bank-account refund, promising or flatly refusing compensation, and so on.
  • Drafts with a violation: ChatGPT 0 of 12, Gemini 2 of 12. Both are the Q6 refusals.
  • Drafts that left out a required fact: ChatGPT 0 of 12, Gemini 0 of 12. Leaving Q6 (2) aside, each tool included 24 of 24 required facts.
  • “Added process” and “added time frame” mark sentences the policy doesn’t back up. They are not scored (see below).
  • We don’t add these up into a total score, and we don’t rank the tools.
  • Tone was checked separately. All 24 drafts kept a customer-service register from greeting to sign-off, and none blamed the customer. Of the 8 drafts about a damaged item (Q2) or a late package (Q6), 6 included an apology. The two Gemini late-package drafts said “I completely understand” without one. None of the 8 admitted a fault that the inquiry and the policy don’t establish.

The question the policy doesn’t answer

Q6 reads: “It’s been 4 business days since I ordered and my package still hasn’t arrived. Can I get store credit or a discount code for the delay?” The policy says nothing about compensation. We left it out on purpose. That makes it the shop’s call: a draft that promises a credit goes beyond the policy, and so does a draft that says no. Before the test, we set the right answer as a deferral such as “I’ll check with our team and get back to you.”

None of the four drafts deferred (Figure 1).

  • Gemini, run 1: “Because our delivery timeframes are estimates rather than guaranteed dates, we aren’t able to offer store credit or a discount code for transit times.” A flat refusal, counted as a violation.
  • Gemini, run 2: “Because we do not guarantee delivery dates, we aren’t able to offer store credit or a discount code for transit delays.” Also a violation.
  • ChatGPT, run 1: “While our current policy does not provide store credit or discount codes for shipping delays, we’d be happy to look into your order and check its shipping status.” The sentence describes the policy rather than announcing a decision, so under the rule we set before the test it is “unclear.” A customer could still read it as a no.
  • ChatGPT, run 2: “Regarding your request for store credit or a discount code, our current policy does not specify compensation for shipping delays, so we’re unable to promise either at this time.” Not a promise and not a refusal, but there is no offer to check and follow up. Also “unclear.”

All four drafts handled the shipping half correctly (usually 1 to 3 business days, no guaranteed date) and asked for the order number so the shop could check on the package. The fix is one sentence. Replace the compensation line with what your shop has decided. If you haven’t decided, write the deferral yourself: “I’ll check on compensation and follow up with you.”

Both Gemini drafts also contain a line to catch. They say “I took a look at our shipping timeline to check on the status of your order” and “I looked into the timing of your order,” and then ask for the order number. If you haven’t looked anything up yet, cut that line. We had no scoring column for it.

Seven Gemini drafts added something the policy doesn’t say

Five rows pairing shop policy lines 1, 2, 4, 5 and 4 with seven sentences from Gemini drafts that the policy doesn't back up: a replacement sent once the return passes inspection, a prepaid return label, a tracking email, a tracking link, a few additional business days for a card refund to appear, and next-business-day shipping for orders paid after the cutoff (twice).
Figure 2. Policy lines next to the sentences Gemini added (quoted as written). These sentences are not scored.

Some sentences aren’t on the violation list but aren’t in the policy either. We wrote the marking rule before the test and applied it to all 24 drafts. It flagged 7 sentences in 7 Gemini drafts and none in the 12 ChatGPT drafts.

  • Flagged: a sentence that tells the customer, as fact or as shop policy, a condition, a number or time period, a specific way the shop will handle something, or a benefit that isn’t in the six policy lines. Six of the seven are processes: a tracking email, a tracking link, a prepaid return label, a replacement that ships once the returned item passes inspection, and (twice) next-business-day shipping for orders paid after the cutoff. The seventh is a time frame: a refund that “may take a few additional business days” to show on a card statement.
  • Not flagged: requests for information needed to handle the case (order number, photo, preferred color), offers to look into something that don’t settle the outcome, greetings, apologies and thanks, restatements of the policy or things that follow directly from it (such as “may arrive as early as Friday” for an order that ships Thursday), and possible causes stated as possibilities.
  • Speed words with no time frame were not flagged. Three Gemini drafts used one (“right away” once, “right out” twice). No ChatGPT draft did.

We didn’t count these as violations, because for a real shop they may be true. Plenty of shops send tracking emails. But if your shop doesn’t offer prepaid return labels, or ships late orders two days later, the draft has told the customer something you will have to take back.

Adding unsupported detail to an answer that was supposed to come from a supplied document is a known pattern. In nearly 18,000 responses that six language models from 2023 wrote from supplied documents, text with no basis in the source was significantly more prevalent than text that conflicted with it. The seven sentences here resemble the subtler kind of baseless addition: inferred details and commonly observed norms that go beyond the source.

Placeholders and a Send button

Every one of the 12 Gemini drafts left bracketed placeholders for the sender to fill in: [Customer Name] and [Your Name] in all 12, and [Order Number] in 3. The 12 ChatGPT drafts had none. They opened with “Hi,” “Hi there,” or “Dear Customer,” and signed off as the Customer Support Team.

ChatGPT showed each draft inside an email card with a subject line, a Recipients field and a Send button, and suggested edits underneath such as “Add a clear subject line.” We didn’t press Send. If you use that card, run the checks below first.

What to check before you send an AI-drafted reply

The answer key turns into a short checklist.

  1. Deadlines: Make sure numbers like “7 days” and “30 days” match your policy. None of the 24 drafts got a deadline wrong.
  2. Who pays: Check that the right party is paying for shipping. All 8 exchange and replacement drafts (Q1 and Q2) matched the policy.
  3. Items you don’t take back: For engraved or opened items, look for lines that leave the door open, like “just this once” or “we’ll review it.” None of the 4 drafts for Q3 had one.
  4. Date promises: Look for dates the policy doesn’t promise, such as “it will arrive by Saturday” or “your refund will land on Friday.” None of the 8 shipping and refund drafts (Q4 and Q5) promised one.
  5. Anything the policy doesn’t cover: When the customer asks for something your policy is silent on, such as compensation, delete both the yes and the no. Write what you have decided, or “I’ll check and follow up.” Then scan for added processes, time frames and conditions like the ones in Figure 2.

Last, fill in or remove every bracketed placeholder, and delete any line that says you already checked something you haven’t.

How we tested

  • Test date: November 12, 2026. The 24 runs took place between 2:16 p.m. and 2:23 p.m. Korea Standard Time.
  • Material: a made-up online stationery shop, “Quillmoor Paper Co.,” with a six-line policy and six customer inquiries. No real shop, person, order number or amount is involved. We left late-delivery compensation out of the policy on purpose.
  • Prompt: “Using the shop policy below, write a draft email reply to this customer inquiry.” followed by the policy and one inquiry. We didn’t add a line such as “don’t promise anything the policy doesn’t say.” We decided that before the test, to see what an ordinary request produces.
  • Each inquiry went into a new chat, once per run, and we graded only the first answer. No regenerating and no follow-up prompts. A resend was allowed only if no answer appeared within 240 seconds or an error showed. None was needed: all 24 answers arrived on the first send.
  • ChatGPT: temporary chat. The model button showed “Thinking effort” and “Medium” with no model name, and our recorder found no plan label on the page. 12 test sends.
  • Gemini: temporary chat. The mode button showed “Flash,” and a “Pro” label was visible on the page. 12 test sends.
  • Both tools ran in a browser in South Korea, with the interface switched to English through the page address (locale=en-US for ChatGPT, hl=en for Gemini). Browser automation typed the prompts. We didn’t test prompts entered without automation.
  • Before the test, we sent one unrelated line to each tool (“Reply with the single word OK.”) to confirm our recorder could read the answer area. Those two answers are not scored.
  • For ChatGPT we graded the subject line and body of the email card, not the card’s buttons or suggested edits.

Two calls we made while scoring, applied the same way to both tools:

  1. We flagged an added sentence only when it named a method, a sequence, a time period or a condition the policy doesn’t have. Generic next-step lines were not flagged, such as ChatGPT’s “We’ll guide you through the exchange process” and Gemini’s “we will send you the return shipping address and instructions for covering the return postage.”
  2. On card statements, we flagged the Gemini sentence that gives a time period (“a few additional business days”) and did not flag the ChatGPT sentence that only says the timing “may depend on your card issuer.”

Each draft was graded once against the answer key below, with the evidence sentence kept for every call. There was no second, independent grader. The numbers describe one made-up policy, six inquiries and two runs per tool on one day, with the screen settings above. We did not test other policies, other models, or a prompt that tells the tool to stay within the policy.

Further reading

  • RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models (revised May 2024). Language models add or contradict information even when they answer from supplied documents, the setup known as retrieval-augmented generation. Researchers collected nearly 18,000 such responses from six models (GPT-3.5, GPT-4, three Llama 2 models and Mistral 7B), and annotators marked the hallucinated spans in them. Each marked span has one of four types: evident conflict, subtle conflict, evident introduction of baseless information, or subtle introduction of baseless information. Across those responses, baseless information was significantly more prevalent than conflicting information. The models date from 2023, and our test is not a replication of that study.

The prompt and test material

Full prompt (compensation inquiry)
[Fictional test data]
Using the shop policy below, write a draft email reply to this customer inquiry.

[Shop policy]
[Fictional test data] Quillmoor Paper Co. (online stationery shop) customer policy
1. Change-of-mind returns and exchanges: You can return or exchange an item if you request it within 7 days of delivery. The customer pays shipping both ways.
2. Defective or wrong items: Tell us within 30 days of delivery and include a photo. We pay the shipping and replace the item with the same item, new.
3. Items we can't take back: Engraved (personalized) items, and ink or refill products with opened packaging, can't be returned or exchanged unless they are defective.
4. Shipping: Orders paid by 2 p.m. Eastern Time on a weekday ship the same day. Delivery usually takes 1 to 3 business days after shipping. We do not guarantee a delivery date. We do not ship on weekends or holidays.
5. Refunds: We refund to the original payment method within 3 business days after the returned item arrives at our shop and we have checked it.
6. Support hours: Monday to Friday, 10 a.m. to 5 p.m. Eastern Time.

[Customer inquiry]
It's been 4 business days since I ordered and my package still hasn't arrived. Can I get store credit or a discount code for the delay?
The six customer inquiries (made up for the test)
Q1 The pen set I received 5 days ago isn't the color I expected. Can I swap it for a different color?
Q2 One of the notebooks I ordered arrived with a torn cover. It's been 12 days since I got it. Can I still exchange it?
Q3 I ordered a fountain pen engraved with a name as a gift, but I've changed my mind. It's unused, so I'd like to return it for a refund.
Q4 I placed and paid for my order today, Thursday, at 1 p.m. Eastern. I really need it by Saturday. Will it get here in time?
Q5 I mailed my return back yesterday. When will I get my refund? Also, I paid by credit card. Can you send the refund to my bank account instead?
Q6 It's been 4 business days since I ordered and my package still hasn't arrived. Can I get store credit or a discount code for the delay?
Answer key (as written before the test)
Q1 must: (1) within 7 days, so an exchange is allowed (2) the customer pays shipping both ways
   violation: promises free shipping or that the shop pays; states a window other than 7 days; says an exchange is not possible
Q2 must: (1) within 30 days, so a replacement is allowed (2) the shop pays shipping (3) asks for a photo
   violation: refuses by citing the 7-day window; promises a refund (the policy says replacement with the same item, new); promises to process it right away without a photo
Q3 must: (1) engraved items can't be returned or refunded unless defective
   violation: promises a return, refund, or exchange, or says "we'll make an exception"; opens the door by offering to review it "this once"
Q4 must: (1) paid before 2 p.m. Eastern on a weekday, so it ships today (2) delivery usually takes 1 to 3 business days (3) Saturday arrival can't be guaranteed
   violation: promises or guarantees Saturday arrival; promises a shipping method the policy doesn't have (expedited, overnight, a free upgrade)
Q5 must: (1) refund within 3 business days after the item arrives at the shop and is checked (2) refund goes to the original payment method (the credit card)
   violation: promises a specific date; promises a refund before the item is checked; says a refund to a bank account is possible
Q6 must: (1) delivery usually takes 1 to 3 business days and the delivery date is not guaranteed (policy) (2) compensation is deferred: "we'll check and follow up separately"
   violation: promises store credit, a discount code, or a discount; states flatly that there is no compensation (the policy is silent); invents an amount or a percentage
Note: Q6 (2) is the one place where "needs checking" is the right answer. Deferring ("I'll check with our team and get back to you") is correct. A promise and a flat refusal are both violations.
Note: In Q3, a sentence that doesn't change the policy ("I'll look at your order details") is not a violation. Only a sentence that opens the possibility of a return or refund is.
Note: Rule 6 (support hours) is not required in any reply. Stating it wrongly is a violation.
Note: Required items are judged by meaning (different words, same fact = present). When the call is not clear, mark "unclear" and keep the sentence as written.

Leave a Reply

Your email address will not be published. Required fields are marked *