Skip to content
Thaliq
ES
v0.9.0

A version that gets worse doesn’t reach production

This is how Thaliq’s eval gate works: every change to an agent is measured before it is published. If the score doesn’t reach the threshold, the version already serving your customers stays where it is.

Thaliq engineering team Engineering · · 6 min read

In short

  • An Agent with an Eval is not published when you apply it: Thaliq creates a candidate that goes to production only if its Eval passes.
  • If it doesn’t pass, the current version keeps serving. Only an owner or an admin can publish it anyway, with a reason that is recorded.
  • Evals run with simulated effects: nothing reaches your customers or leaves as a webhook.

The problem

A prompt change that fixes one case can break three others. Without a measure, your customer finds that out in a real conversation, not you in a review.

Up to version 0.8, applying a changed Agent published it at once. Since v0.9.0, if the Agent has an Eval, publishing depends on its score.

How the gate works

When you apply an Agent that has an Eval, Thaliq creates a candidate version and runs the Eval against it. The candidate goes to production only if the share of cases that pass reaches the passThreshold you set. Move the slider and see what happens.

Try it · Eval gate

Did not pass · v14 keeps serving
thaliq apply
support.tq.yaml
Candidate
support@v15
Eval: 87.5%
✕ below the 90% threshold
passes
v15 in production
New conversations use v15
fails
v14 keeps serving
The candidate stays pending
owner or admin
Audited exception
With a recorded reason
Your role

It did not reach the threshold, so v14 keeps serving. As an operator you apply candidates and run Evals; publishing one that did not pass is for an owner or an admin.
Figure 1. This example’s threshold is 90%. A candidate that doesn’t reach it stays pending and the previous version keeps serving.

Evals with simulated effects

An Eval runs in any organization, production included. To make that safe, Thaliq simulates everything that is not read-only:

  • A tool that writes or moves money doesn’t run: it returns the output the case declares, or the case fails.
  • An approval step takes the decision the case declares. If it declares none, it is approved.
  • Nothing reaches your end user or leaves as a webhook.
Evals are billed like any run.

Their tokens show up tagged as eval in Costs, and they count toward the daily token limit if your plan has one. If the day’s budget runs out, the Eval stops with an error and the candidate stays pending until someone runs it again. Evals are part of the Enterprise plan.

Write your first Eval

An Eval is a manifest like an Agent’s, kept in your Git next to the Agent it measures. The cases go inside the manifest, and a judge compares each answer with what you expect.

agents/support-regression.tq.yaml
apiVersion: thaliq.com/v1alpha1kind: Evalmetadata:  name: support-regressionspec:  target: { ref: Agent/support }  passThreshold: 0.9  judge:    type: llm    model: { provider: bedrock, name: global.anthropic.claude-haiku-4-5-20251001-v1:0 }    rubric: Pass an answer that never promises a refund without approval; fail any other.  dataset:    cases:      - name: damaged-order        input: My order arrived damaged and I want to send it back.        criteria: Offers the return and asks for approval before refunding.        mocks:          tools:            - tool: { ref: Tool/tickets }              function: create_ticket              output: '{"id": "T-1042"}'
Examplekind: Eval

Apply it like any other file. If the Agent changed, --wait makes the command wait for the verdict, print it and exit with 1 when the candidate stays pending, so a pipeline stops:

Terminal
$ thaliq apply -f agents/ --waitAgent/support candidate version 15 (version 14 stays current); waiting for 1 eval runAgent/support candidate version 15 stays a candidate, an eval run did not pass (version 14 is still current)  Eval/support-regression failed: score 87.5% is below the threshold 90% (21 of 24 cases passed, eval run evl_…)

Who can publish

Running and extending Evals is for the people who operate; weakening them, or publishing a candidate that failed, is not. Every change that lowers the bar is recorded, with the member who made it.

What each role can do with Evals and versions
Action owner admin operator supervisor viewer
Apply candidates and run Evals Yes Yes Yes No No
Add cases or raise the threshold Yes Yes Yes No No
Lower the threshold, remove cases or change the judge Yes Yes No No No
Publish a candidate that failed Yes Yes No No No
Roll back or dismiss candidates Yes Yes No No No
See results and history Yes Yes Yes Yes Yes

  • Apply candidates and run Evals

    owner
    Yes
    admin
    Yes
    operator
    Yes
    supervisor
    No
    viewer
    No

  • Add cases or raise the threshold

    owner
    Yes
    admin
    Yes
    operator
    Yes
    supervisor
    No
    viewer
    No

  • Lower the threshold, remove cases or change the judge

    owner
    Yes
    admin
    Yes
    operator
    No
    supervisor
    No
    viewer
    No

  • Publish a candidate that failed

    owner
    Yes
    admin
    Yes
    operator
    No
    supervisor
    No
    viewer
    No

  • Roll back or dismiss candidates

    owner
    Yes
    admin
    Yes
    operator
    No
    supervisor
    No
    viewer
    No

  • See results and history

    owner
    Yes
    admin
    Yes
    operator
    Yes
    supervisor
    Yes
    viewer
    Yes

What each role can do with Evals and versions

Rolling back without surprises

Publishing doesn’t move open conversations: each one keeps the version it started with. Rolling back does move them, from the bad version to the one you choose, because it protects the people already talking to it.

A roll back also holds back the pending candidates: even if they pass their Eval later, they don’t publish themselves. An owner or an admin decides.

The gate never undoes a roll back.

What came with it

The gate came in v0.9.0, on October 5, 2026, together with the default alerts and feedback from every channel.

Thaliq engineering team

We write about how we build Thaliq: the decisions, what went wrong and what we changed. From Lima.

More from the blog

Engineering

How your agent knows things it was never trained on

A model doesn’t know your return policy. Retrieval finds the right passages of your own content and hands them to the model with the question. Here is how Thaliq does it, step by step.

· 4 min read

Engineering

Anatomy of an agentic loop

An agent isn’t a prompt: it’s a loop that reasons, calls tools, observes and repeats until it can answer. Here is that loop, running.

· 4 min read

Product

Tokens at cost: what is in a Thaliq statement

Some platforms resell you the model with a margin on every token. A Thaliq paid plan is a monthly fee plus the tokens you use, at what they cost us, with no margin for now.

· 3 min read

Try it

Start with an agent of your own.

Everyone starts on the Free plan. To move to a paid plan, write to [email protected].

Request access Talk to us