How to Test Intercom Fin Before You Set It Live

The temptation with Fin is to add ten articles, hit the button, and see what happens. It usually works well enough that nothing looks broken, which is the problem. The failures are the twenty percent of questions your content never covered, and you will only find those by looking for them deliberately.

Intercom has built a decent amount of testing tooling here, more than most agents in this category ship. This guide covers what each tool does, what it does not do, and the order that catches the most problems for the least effort.

Before any of this, remember the gate. Intercom's chat deployment documentation states you need at least ten public articles live in your Help Center before you can set Fin live over chat, so testing on an empty knowledge base tells you nothing useful.

What are the four ways to test Fin?

Intercom publishes a page comparing three of them directly, and the fourth is a deployment setting rather than a testing tool. Here is what each is for.

The most useful thing on that table is what previews show you. Intercom's documentation describes previews as available throughout Fin's training areas, including Content, Guidance, Attributes, Escalation, Procedures and Deploy, so you can check a change without leaving the screen you made it on.

How do you run a batch test?

Batch testing lives under Fin AI Agent, then Test. It is the tool that answers the question you actually care about before launch, which is whether your content covers what people really ask.

Running a batch test

1. Open Fin AI Agent, then Test — This is the batch testing screen. It is separate from the preview panels you see while editing content.
2. Get your questions in — Intercom documents four routes: generate from past conversations across all conversations or a specific AI topic, generate by topic using AI Topics, add questions manually, or upload a CSV with a single column of questions.
3. Mind the limits — Intercom's page states up to 50 questions per test group, and notes that generating from your inbox needs at least one conversation in the last 90 days. Large question sets get split across groups.
4. Run it and read every answer — This is the unglamorous part and it is where the value is. Skimming defeats the exercise.
5. Rate each response — Intercom's documented ratings are Acceptable, where you can add an internal note, and Poor, where you pick a root cause from categories covering content, clarification, misuse, tone, length, language and other.
6. Use the evaluate panel to diagnose — The Evaluate answer panel shows what shaped a given response, including personality settings, guidance, the content sources used and any automations that triggered. That is how you tell a content gap from a guidance conflict.

Where do batch test questions come from?

Generate from past conversations first, because those are the questions customers genuinely ask, in the words they genuinely use. Questions written by your team are always cleaner and better spelled than real ones, which makes Fin look better than it is.

Then add manual questions for the cases you cannot afford to get wrong. Intercom's documentation notes manual entry is useful for compliance scenarios where you have no historical data. Refund policy, cancellation terms, anything with a legal edge, write those yourself and test them explicitly.

There is a decent middle path too. Intercom's live testing documentation suggests building a set of control questions and asking colleagues to mark whether Fin gave the best response, which doubles as a way to find gaps in your support content.

How do you test procedures?

Procedures need simulations rather than batch tests, because a procedure has branches and a batch test only checks one answer. Intercom describes a simulation as an automated end to end test where an AI acts as the customer and judges whether the outcome was correct.

The habit worth copying from their documentation is running the whole suite after every edit. Their guidance is to use Run all before going live, on the basis that a simulation which newly fails is a regression your last change introduced.

Their advice on suite coverage is also sound: build scenarios for the normal path, for graceful failure, and for customers behaving unexpectedly. Those three categories catch most of what breaks in production.

How do you launch to your own team first?

Use an audience rule on the deployment itself. Intercom's live testing documentation describes going to Fin AI Agent, then Deploy, then Chat, and using the Who will see Fin section to restrict who Fin appears for. The email equivalent is under Deploy, then Email.

The trick their documentation gives for internal testing is to select Users and filter on the email address containing your company domain. Fin is genuinely live, on real infrastructure, but only your colleagues can reach it.

For a limited customer rollout, the same audience mechanics accept richer conditions. Intercom's page describes targeting on custom data attributes such as plan or account tier, on user type using Users or Leads only, on product usage attributes, on the current page URL so Fin appears only on specific pages, and on combinations like last seen and region.

A note on how much of this is testing versus rehearsal

Everything above is real engineering practice and Intercom deserves credit for shipping it. It is also, honestly, a lot of process to run before a support agent can answer its first question.

That preparation load is what we approached differently at Sistava. Instead of building a configuration and then proving it works, you hire an AI Employee and tell it in plain English what you want handled. It has its tools connected already and it asks you when it is unsure, so the checking happens in the conversation rather than in a test suite you have to maintain alongside it.

Both approaches are defensible. A test suite gives you evidence you can show a stakeholder, and at high volume that evidence is worth the effort. A worker that asks gives you speed, and for a small team that speed is often the difference between shipping and not.

What does a sensible testing order look like?

If you only have an afternoon, run it in this sequence. Each step catches a different class of problem, and doing them out of order means re-testing work you already changed.

• Preview as you build. Every time you add content or write guidance, type one question at it. Cheap, immediate, unlimited.
• Batch test from past conversations. Fifty real questions will find your content gaps faster than anything else on this list.
• Fix the content, then batch test again. The second run is the one that tells you whether your fixes landed.
• Simulate every procedure. Run all, and treat a newly failing simulation as a regression rather than a flaky test.
• Deploy to your own domain. Live infrastructure, colleagues only, using the audience filter on the deploy screen.
• Deploy to a forgiving customer segment. Then widen it once the reporting agrees with what your tests predicted.

Sources

• Intercom Help, Batch test Fin AI Agent
• Intercom Help, Simulations vs. Batch tests vs. Previews
• Intercom Help, Live testing Fin with your team or customer segments
• Intercom Help, Deploy Fin AI Agent over chat
• Intercom Help, Add your content for Fin AI Agent