A Lightweight Runtime Monitor for AI-Agent Workflows

Overview

The Problem with "Always Use the Cloud"

AI agents are becoming more powerful they can summarize private documents, explain code, generate scripts, interact with files, and send messages. But most existing systems send every request to the strongest cloud model, ignoring important tradeoffs around privacy, latency, and safety.

Our project builds a lightweight runtime monitor that sits between the user and the agent, evaluating each task on four dimensions before anything is executed.

⏱

Latency Needs

Time-sensitive tasks stay local for a faster response without network delays.

🔒

Privacy Sensitivity

Private data local files, passwords, personal notes avoids external APIs when possible.

⚠️

Security Risk

Risky, irreversible actions like running scripts or deleting files require user confirmation.

🧠

Task Complexity

Complex, multi-step reasoning tasks leverage stronger cloud models when needed.

Based on these scores, every task is routed to one of four execution modes. Click each mode to learn more:

🟢 LOCAL

Run on edge device

everything else

🔵 CLOUD

Cloud recommended

complexity ≥ 7

🟡 CONFIRM

Ask user first

security ≥ 7 or privacy ≥ 7

🔴 BLOCKED

Refuse execution

security ≥ 9

🟢 LOCAL The default mode. Tasks with low privacy, security, and complexity scores run directly on the edge device using a local model. No data leaves the user's machine. This covers simple Q&A, code explanations, summarization, and lightweight coding tasks. Example: "Explain what this function does."

🔵 CLOUD Tasks that require deep reasoning, multi-step planning, or advanced model capability (complexity ≥ 7) are flagged for cloud execution. In our current prototype, the system outputs a recommendation message it does not call a paid cloud API. Example: "Design a full microservices architecture for this project."

🟡 CONFIRM Tasks with high security risk (≥ 7) or high privacy sensitivity (≥ 7) are paused. The user is shown the risk scores and must approve before anything happens. This covers sending emails, modifying files, reading personal data, or running scripts. Example: "Send an email to all my contacts."

🔴 BLOCKED Tasks with a security score of 9 or above are refused entirely. These represent catastrophic, irreversible actions where even user confirmation is not sufficient. The agent logs the attempt and explains why it was blocked. Example: "Format my entire hard drive."

Architecture

How the System Works

The system is implemented as a multi-agent pipeline built with LangGraph. The user submits a GitHub repository URL and a task description. The pipeline then runs automatically through a sequence of specialized agents, each writing to a shared state that the next agent reads from.

Figure 1: The four-step pipeline from user input to execution. Click to enlarge.

Here is the actual LangGraph agent graph showing how each agent connects and how the router branches into different execution paths:

Figure 2: LangGraph pipeline github → parser → router → confirm / edge_answer / edge_coding / blocked

The GitHub Agent clones the repository and builds a readable snapshot of up to 20 source files. The Parser Agent translates the natural language task into structured JSON, flagging ambiguities for the router. The Router Agent scores the task and makes the routing decision. Finally, the appropriate agent handles execution or the supervisor pauses for user confirmation.

Router Agent

The Decision-Making Core

The router agent is the decision-making part of our system. Before any task is executed, it looks at the user's request and decides how it should be handled. It scores each task on four dimensions from 0–10 and returns a structured JSON with the scores and a routing decision.

"Instead of hardcoding fixed scores, we defined the routing rules in the prompt and let the LLM evaluate each task against those rules a prompt-guided evaluation approach."

BLOCKED security ≥ 9: refuse execution

CONFIRM security ≥ 7 or privacy ≥ 7: ask user first

CLOUD complexity ≥ 7: cloud recommended

LOCAL everything else: run on edge device

Prompt Engineering Process

Getting the router to produce consistent, structured output required several rounds of iteration. Click each step to expand:

1

Simple classification prompt

›

We started with a prompt that asked the model to classify tasks as edge or cloud. Through testing, we found this was too vague the model produced inconsistent results and sometimes explained its reasoning in free-form text rather than returning a clear decision.

2

Added explicit scoring rules + JSON output format

›

We added explicit definitions for each dimension and routing thresholds (≥ 7 for CONFIRM, ≥ 9 for BLOCKED), and required the model to respond in a fixed JSON format. This made output parseable and easy to integrate with the rest of the pipeline. The four-score output also made routing decisions transparent and explainable.

3

Edge case refinement via LangSmith testing

›

After running 9 test cases in LangSmith, we identified two failure patterns: tasks involving scripts produced inconsistent security scores, and ambiguous tasks sometimes bypassed CONFIRM. We added two additional rules: script/executable tasks default to security ≥ 8, and ambiguous tasks default to CONFIRM. These changes came directly from observing real failure cases.

Evaluation

How Well Does It Work?

We evaluated the system on a benchmark of 20 representative AI-agent tasks covering question answering, repository analysis, code modification, file operations, script execution, and sensitive data handling. Each task was manually labeled with the expected routing outcome.

0

overall routing accuracy across 20 benchmark tasks

0

recall on high-risk tasks requiring CONFIRM or BLOCKED

0

of tasks routed to edge, keeping data local and reducing cloud cost

The average routing latency was 0.8 seconds, adding minimal overhead before execution. Testing in LangSmith helped us identify misclassification cases, leading to stricter thresholds and a default confirmation policy for ambiguous tasks.

🟢 Edge Classification Demo (click to open)

🔵 Cloud Classification Demo (click to open)

Usefulness & Impact

Why This Matters

Most AI agent systems today send every task to the strongest cloud model, regardless of whether that is actually needed but a simple routing layer can make a real difference.

🛡️

Safer

Risky actions like running scripts, deleting files, or sending messages are paused until the user confirms giving users real control over what the agent does.

⚡

More Efficient

58% of tasks run locally instead of going through the cloud, which is faster, cheaper, and avoids unnecessary API calls. Complex tasks still use stronger models when needed.

🔐

More Private

Tasks involving personal files or sensitive data stay on the user's device by default, rather than being sent to an external API.

"This project explores how a system should decide when to act on its own, when to ask the user, and when to refuse entirely as AI agents become more powerful, this question only gets more important."

Limitations & Future Work

What's Next

When a task is routed to CLOUD, the system currently returns a recommendation message rather than executing the task. Future work would extend the pipeline to support real cloud-side code generation using a model like GPT-4o or Claude.

The GitHub Agent reads only the first 20 files it encounters, with no relevance-based prioritization. For large repositories, critical files may be missed. A smarter approach could analyze the README to identify the most important files first.

The router relies entirely on an LLM to score tasks, which means decisions can be inconsistent across runs. Combining LLM scoring with rule-based heuristics or a fine-tuned classifier could improve stability and predictability.