How are conversations evaluated by AI?

What is it for?

Every completed conversation is evaluated by AI against the criteria you define: each criterion is answered yes/no or scored, and the conversation gets an overall score from 0 to 100. Each result comes with its reasoning, what it's based on and how reliable that data is. For example, use it to pre-screen conversations or to decide which visitors to get back to first.

“Sentiment analysis” on the same tab reads the messages the visitor wrote to the agent and shows whether each message is positive, neutral or negative and whether the visitor is frustrated (see below).

Results appear on the agent's “Analytics” tab and in the conversation details; visitors never see them.

What does evaluation read?

For each completed conversation, evaluation reads:

• The visitor's answers to the steps.
• What the visitor wrote to the agent in AI Assistant steps.
• If you choose, the content of the documents and images the visitor uploaded in file upload steps. To include them, turn on the “Also evaluate the content of uploaded files” switch (see below). Files older than 90 days aren't read.

The agent's own messages are read only for context: something the agent said doesn't count as the visitor's information. Something the visitor only said in the conversation counts as their own statement; it isn't treated as being as reliable as an answer to a step or a detail in a document.

Turning on evaluation

1. Open the “Evaluation” tab in the flow editor.
2. Turn on the switch on the right of the “AI evaluation” card. While it's off, the criteria are kept but hidden, and completed conversations aren't scored.
3. To have files read as well, turn on the “Also evaluate the content of uploaded files” switch that appears on the card. It's off by default. Evaluations that read files use more credits.
4. Click “Add criterion”. In the window that opens, write the criterion's name (“Criterion”, for example Financial health), choose the “Evaluation type” (see below) and fill in “Evaluation instructions”: describe what the AI should pay attention to when evaluating this criterion, and which answer, which part of the conversation or which detail in a file to look at. Click “Save” in the window to add the criterion to the list.
5. Add more criteria if needed, then save the agent with “Save” in the editor.
6. On a published agent, click “Publish”. Evaluation is part of the agent's version: the new setting applies to conversations that start after you publish.

Each criterion appears in the list with its number, its name, the first line of its instructions and its type (“Yes / No” or something like “0–10 points”), plus how many references it has. To edit one, click its row or its pencil button: the same window opens; keep your changes with “Save” or drop them with “Cancel”.

If evaluation is on but there are no criteria, conversations aren't scored. While evaluation is on, the agent can't be saved if a criterion has no name or instructions; that criterion's row shows an “Incomplete” badge.

Only owners and admins can change the evaluation and sentiment analysis settings; other roles can only view the tab. Everyone can see the results.

Evaluation type

The “Evaluation type” selector in the criterion's window sets how the criterion is answered. New criteria start as “Yes / No”.

• “Yes / No”: the result is “Success”, “Failure” or “Unknown”. “Unknown” means the data wasn't enough to decide, and it isn't counted in the overall score. Good for questions like “Was the visitor's question answered by the end of the conversation?”
• “Score”: the criterion is scored from 0 up to the number you enter in “Maximum score”. The minimum is always 0; the maximum can be a whole number from 1 to 100, and new score criteria start at 10. In “Scoring rubric (optional)” you can describe how points are given, for example “Give 1 point for a greeting, 1 for understanding the issue, and up to 3 for how well it was solved.”

Criteria added before you could choose a type work as “Score” criteria with a maximum of 100; their results don't change.

Reference documents and links

In a criterion's window you can add optional “References”, such as your organization's scoring guide:

• “Upload document (PDF, DOCX, TXT, MD)”: up to 10 MB.
• “Reference link”: enter the address and click “Fetch”.

Each criterion can have one document and one link; the first 3,000 characters of each are used.

The reference's text goes to the model with the instructions; the file itself isn't stored. To delete a criterion, click the trash button on its row; if it has instructions or a reference, you're asked to confirm first. The published agent doesn't change until you save and publish.

Reading the results

Evaluation runs in the background once the conversation is completed. Results appear in two places:

• The “Evaluations” view on the “Analytics” tab: the average score and the score distribution. The list is sorted from the highest score; each row shows the visitor, the overall score, a 1–2 sentence summary written by the AI and the date. Click a row to open the result per criterion with its reasoning and confidence (“Confirmed by data”, “Partial / self-reported”, “No data”). The reasoning and the summary are written in the agent's main language. The overview also shows the “Avg. fit score”.
• The conversation details: the same results and reasoning under “AI evaluation” on the “Analysis” tab; the “Overall score” also appears at the top of the “Summary” tab.

Yes/no criteria show a badge: “Success” in green, “Failure” in red, “Unknown” in gray. Score criteria show the points as “4 / 5” (just the number when the maximum is 100); in the window opened from the “Evaluations” list, a bar appears under them too.

The conversation's overall score is the average on a 0–100 scale: a score criterion counts as its points divided by its maximum (4 out of 5 counts as 80), “Success” counts as 100 and “Failure” as 0. “Unknown” results and criteria the evaluation couldn't return are left out.

In the window that opens when you click a row in the “Evaluations” list, the comparison chart shows only score criteria; yes/no criteria are listed below with their badge.

Each criterion's result shows “Based on”: the result relies on an answer (“Answer: …”), a visitor message (“Message 3”) or a file (“File: …”). In the conversation details, click a message chip to scroll the conversation to that message.

If the visitor comes back to a finished conversation and writes again, or a new answer arrives, the conversation is evaluated again when it finishes again and the new score replaces the old one. A conversation is evaluated at most 3 times in total.

Criteria that couldn't be scored are listed separately. If no conversation has been scored, Analytics shows “Evaluation is off” or “No evaluations yet”; owners and admins also see a “Set up evaluation” link.

Credits

Every evaluation uses credits (reading files included). A conversation is evaluated at most 3 times, and the cost appears as “Evaluation” in the conversation's credit breakdown. If you run out of credits, evaluation and sentiment analysis don't run.

Sentiment analysis

AI reads each message the visitor wrote to the agent and rates it in two ways: sentiment (“Very negative”, “Negative”, “Neutral”, “Positive”, “Very positive”) and frustration (“None”, “Low”, “Moderate”, “High”). This shows you in which conversations the visitor wrote negatively or got frustrated.

The switch on the “Sentiment analysis” card is on by default for every agent. To turn it off, switch it off on the “Evaluation” tab, then click “Save” and, on a published agent, “Publish”. Like evaluation, it's part of the agent's version: the new setting applies to conversations that start after you publish.

The analysis runs in the background when the conversation ends or the visitor stops writing for 30 minutes. It only works in the parts where the visitor talks to the agent (AI Assistant step); answers to other steps aren't read.

The result appears on two tabs of the conversation details:

• In the list at the top of the “Summary” tab, the “Sentiment” row (based on the visitor's last message) and the “Frustration” row (based on the visitor's worst moment).
• On the “Analysis” tab, a message-by-message chart: hover over a point to see the message and its ratings; click it to scroll the conversation to that message.
• On the same tab, “Worst moment: message 3”: click it to scroll the conversation to that message.

In the transcript, messages that are negative or show frustration get a small badge under them.

The analysis uses credits (the note under the card reminds you); the cost appears as “Sentiment analysis” in the conversation's credit breakdown. Visitors never see any of it.

Frustrated visitors

While sentiment analysis is on, the agent also notices during the conversation when the visitor is frustrated: when they get annoyed, complain that they have to ask the same thing again, or want to talk to a person. There's no separate switch for this; on that message the agent's reply may use slightly more credits.

At that moment:

• The agent briefly apologizes and answers short and clear. If it can take requests (the request tool is on, or the step routes to a request step), it offers once to pass the matter to your team.
• Your team gets the “Frustrated visitor” notification, with a link to the conversation. It's sent once per conversation; choose where it goes on the Alerts and notifications page.
• The conversation's frustration becomes “High”: a red “Frustration: High” badge appears in the Conversations list. Once the analysis runs, the level is settled from the messages' ratings.

It works in conversations on your website and through the share link.