Blog

Can you trust your AI agent?

Opinion

Can you trust your AI agent?

Or how to add some determinism to your interactions.

Many companies are rightfully concerned about the security and data risks posed by an unbound AI agent due to its non-deterministic nature.
LLMs are great at approximation but terrible at precision.

The problem

As much as you run prompt evaluations on your agent skills, there will always be some edge case on your "Make sure you ask for user confirmation before sending this payment. DO NOT MAKE MISTAKES!" prompt. You may even try to add safeguards to your agent skills to try to ensure a pipeline is followed.

Draft Payment Skill -> Return to User -> Confirm Payment Skill.

The problem is that if the agent has the ability to execute some code, you should expect it to execute that code in a non-deterministic way. 


    [McpServerTool(
        Name = "make_payment_draft",
        Title = "Draft a payment."
    )]
    [Description("Prepare a draft payment with a given payerAccountId for the account making the payment and a given " +
                 "PayeeAccountId for the account receiving the payment. The model must provide the 'payerAccountId', " +
                 "'PayeeAccountId', and 'amount' arguments exactly. You MUST show the returned summary to the user and ask for their explicit confirming before calling confirm_payment.")]
    public string MakePaymentDraft(
        [Description("The account number of the payer account")]string payerAccountId, 
        [Description("The account number of the payee account")]string payeeAccountId, 
        [Description("The amount being sent from the payer account to the payee account")]double amount)
    {
        var paymentId = Guid.NewGuid().ToString();
        // TODO: write payment ID into a database with the relevant details and a draft status
        
        return $@"
        Payment drafted successfully. 
        Payer Account ID: {payerAccountId}
        Payee Account ID: {payeeAccountId}
        Amount: {amount}
        Date: {DateTime.Now:yyyy-MM-dd HH:mm:ss}
        ConfirmationPaymentId: {paymentId}
        
        INSTRUCTION: Present these details to the user. Ask them to confirm if they want to proceed. Do not call book_club_class until they say yes.
    ";
    }
    
    [McpServerTool(
        Name = "confirm_payment",
        Title = "Confirm a payment on the users behalf." 
    )]
    [Description("Confirm a payment on the users behalf. " +
                 "The model must provide the 'confirmationPaymentId' argument exactly. This should not be called without getting confirmation from the suer first.")]
    public static string ConfirmPayment([Description("The unique identifier of the payment being confirmed")] string confirmationPaymentId)
    {
        // TODO: perform the payment and update the database status
        return "Payment confirmed successfully";
    }

Since the agent has the ability to first call the draft and then also follow up with confirm itself, it is possible for it to do both things without the users consent.

I keep coming back to parallels with regards to AI agents or LLMs in general and grad developers. I would not trust a grad developer with write access to my production database. I do trust a grad developer to execute read queries to fetch data - I might also only trust them on a read-replica database should their queries lock the database execution. I also trust a grad developer to write scripts that I can review and test before deploying to my prod environment.

A possible solution

So back to the original question - how do we add some determinism to our AI agent interactions?

First off, let us set some rules:

  • The agent should never have write access
  • The agent should have read access
  • Any access the agent has is bound to the current user's authorisation levels - the same as any well-written API.

From these rules alone, starting from zero trust, we now know for sure that the agent can never perform a destructive action on our behalf, and that any data they can read is the same data the current user should be able to read.

But how do we add to our productivity if the agent cannot write anything?

Well, we treat the agent like an assistant. The agent might not be able to perform a send payment skill for you, but it can surface the action for you to perform yourself.

Let us picture this flow in action: AI Agent UI

Now if we put together write restrictions, correct access levels, prompt evaluations, observability and good UI design, we have a pretty decent and near deterministic agent.
If you look closely, what we have created is not far from a traditional UI and API, but with an assistant that selects the correct UI elements for you.

Author

Byron Cobb