Eval Loop

Eval Loop

Version: 0.1.1
Type: tool

Pulls a Dify chat app's real conversation logs and evaluates the Q&A for groundedness, relevance, correctness, and reusability - using the actual knowledge-base passages Dify already retrieved for each answer, plus any user feedback already recorded. Returns a structured report; it does not create annotations or modify anything. It can evaluate specific end users' conversations, or - with a console access token - every conversation in the app.

As a Tool-type plugin, it drops directly into a Dify Workflow's canvas - including right after a native Schedule Trigger node, so you can run it on a cron schedule with no external infrastructure and no HTTP Request node.

Contents

• Why this exists
• Settings
• Install
• Setup
• Triggering a run
• Report format
• Viewing results in Dify
• Support

Why this exists

Dify's Service API only lets you pull conversations/messages scoped to one end user at a time, and there's no built-in way to systematically review whether an agent's real answers are actually correct. This plugin closes that gap: it walks the conversation history for the end users you specify, and for each message/answer pair asks an LLM to judge:

• Grounded - is the answer actually supported by the knowledge-base passages that were retrieved for it (Dify attaches these to every message when the app has retrieval enabled), or does it go beyond/contradict them? A deliberate decline/redirect on an out-of-scope message counts as grounded too, since it isn't making a factual claim that needs support.
• Relevant - is the answer the appropriate response to what the user was asking or trying to accomplish - which can be a redirect or brief acknowledgment when that's the right call, not only a literal on-topic answer.
• Correct - overall accuracy and appropriateness, combining groundedness and relevance.
• Reusable - is this a generalizable piece of knowledge worth turning into permanent support material, or a one-off case tied to specific names/order IDs/dates (or just a greeting/redirect, which is never reusable)?

The base criteria are domain-agnostic. If your app has its own scope-limiting policy (e.g. "always redirect off-topic questions"), describe it in the Custom Instruction setting so the eval judges against your app's actual intended behavior instead of guessing.

Real user feedback (like/dislike), where already recorded on a message, is passed to the eval model as an additional signal and included in the report.

Settings

Set once, when you add Eval Loop's credentials to your workspace (Tools → Eval Loop → Authorize):

Set per run, on the Run Eval tool node itself (so the same authorized tool can be reused across workflows with different targets):

Install

To install this plugin, specify the following GitHub repository when selecting "Install Plugin":

https://github.com/fr3on/eval-loop

Setup

1. In the Dify app you want to evaluate, go to API Access and generate (or copy) a Service API key.
2. Note the app's identifiers - whatever string is passed as when the app is invoked (via the chat API, a workflow, or any client integration). If nothing sets this explicitly, every conversation falls back to one shared default user, and you can pass that instead.
3. Install this plugin. In a Workflow's Tools panel (or Tools → Eval Loop), authorize it: select the target app, and fill in the base URL, API key, and Eval Model.
4. Drop a Run Eval node onto your workflow canvas and fill in the per-run parameters (target end users, lookback window, etc.).

Triggering a run

Eval Loop is a Tool-type plugin, so it appears directly in a Workflow's Tools panel - drag it onto the canvas like any other tool node, no webhook URL or HTTP Request node needed. Trigger it however fits your setup, e.g.:

• On a schedule - add a native Schedule Trigger node as your workflow's start, then chain a Run Eval tool node after it. This is the primary intended setup: fully automatic, entirely inside Dify.
• Manually - run the Workflow app by hand from Dify Studio whenever you want a report.
• As an agent tool - add it to an Agent app's tool list so an agent can trigger an eval run on request.

Report format

A successful run returns:

is deliberately not called - it's whatever the user typed (Dify's own field is called ), which is often a greeting, an acknowledgment, or small talk rather than a literal question.

is or . holds preference pairs built from flagged answers that have a corrected answer, ready for preference fine-tuning (e.g. TRL's DPOTrainer). If a batch of pairs gets a malformed response from the eval model, each pair is retried individually before being marked as an eval error. Retrieved passages are capped at 3000 characters each.

lists any per-user or per-conversation failures (e.g. an invalid end-user identifier) that didn't stop the rest of the run.

is a ready-to-display Markdown report - counts plus a table of flagged (incorrect) messages with their issues - meant to be shown directly rather than parsed. See below for where to actually see it.

Viewing results in Dify

This plugin doesn't keep its own run history by default - each run's report only exists in that run's node output, unless something stores or displays it. Two ways to see results inside Dify:

Option A - via a Workflow's own Logs (no extra setup)

1. Build a Dify Workflow app with a native Schedule Trigger node, followed by a Run Eval tool node (see Triggering a run).
2. Add an End node whose output is the Run Eval node's output variable.

Every scheduled run then shows up in that Workflow app's own Logs tab in Dify Studio, with the readable summary as the run's output. The node also exposes , , , , and as separate output variables, and the full raw report as its JSON output, if you need to feed a later automation step (e.g. an IF/ELSE branch on count, or a Knowledge Base write).

Option B - persistent history via a Knowledge Base

Turn on Save Report to Knowledge Base (with a Knowledge Base ID and its own API Key) and every run writes its as a new Document into that dataset automatically - no Workflow needed. This gives you an actual searchable history across runs (visible in that Knowledge Base's Documents list in Dify Studio), and since it's a real Dify dataset, another app can even use it as RAG context - e.g. an admin-facing "ask about this app's quality trends" chatbot.

Both options can be used together.

Sending answers to Slack for human review

The Run Eval node outputs (and ): every answer that was judged incorrect, couldn't be judged, or got a thumbs-down from the end user - each with the user's message, the answer, the issue, the suggested corrected answer, and the conversation/message IDs.

wires this up: Schedule Trigger -> Run Eval -> Build Slack message (Code) -> Anything to review? (IF/ELSE) -> Post to Slack (HTTP Request) -> End. Nothing is posted on runs with nothing to review.

To use it:

1. In Slack, create an Incoming Webhook for the review channel (Slack Apps -> Incoming Webhooks) and copy its URL.
2. Import the example, then paste the URL into the environment variable (stored as a secret).
3. Adjust the schedule and Run Eval settings.

The message lists up to 10 items (each field trimmed to 500 characters) and points to the workflow Logs / Knowledge Base report for the rest. If the Slack post fails, the run fails visibly in the workflow's Logs after 3 retries. Slack messages here are one-way notifications; approve/reject buttons would need a full Slack app rather than a webhook.

Limitations

• Each message/answer pair is judged on its own, without the earlier turns of the conversation.
• Load-all mode depends on the console API, which is not a stable public API and may differ between Dify versions; the token expires. For long unattended schedules, listing known Target End Users is more reliable.
• Listings are capped at 50 pages (Service API) / 200 pages (console) per scan, and runs at .

Support

• Source repository: https://github.com/fr3on/eval-loop
• Issues / questions: https://github.com/fr3on/eval-loop/issues
• Contact: @fr3on on GitHub