CUE — Computer Use Enforcer

![License](LICENSE) ![Python 3.11+](https://www.python.org/) ![MCP](https://modelcontextprotocol.io/)

English | 한국어

The missing safety layer between AI agents and your desktop.

CUE is safety middleware for AI computer use agents. It monitors, enforces policies, and provides guardrails so that AI agents can interact with your desktop — safely and compliantly.

---

Problem

AI computer use is accelerating fast, but safety infrastructure hasn't kept up:

Rapid adoption — 40% of enterprises plan to deploy AI computer use agents by end of 2026
No guardrails — only 50% of organizations have any safety controls for autonomous agents
Regulation is coming — the EU AI Act (Aug 2026) mandates human oversight for high-risk AI systems

There is no open-source framework that sits between AI agents and the desktop to enforce safety policies. CUE fills that gap.

What CUE Does

┌─────────────────────────────────┐
│  Any AI Agent                   │  Claude, GPT, Agent-S, ...
│  (natural language in/out)      │
└──────────┬──────────────────────┘
           │ MCP protocol (stdio)
┌──────────▼──────────────────────┐
│  CUE — Computer Use Enforcer   │  ← this project
│  ┌────────────────────────────┐ │
│  │ Policy Engine              │ │  risk classification, action filtering
│  │ Guardrails                 │ │  app blocklist, key blocking, rate limits
│  │ Audit Logger               │ │  JSONL compliance trail
│  │ Monitor (coming soon)      │ │  real-time dashboard & event streaming
│  └────────────────────────────┘ │
└──────────┬──────────────────────┘
           │ pyautogui / pygetwindow / pywin32
┌──────────▼──────────────────────┐
│  Desktop OS                     │
└─────────────────────────────────┘

Core Value Proposition

| Capability | Status | Description | |------------|--------|-------------| | Action Guardrails | Available | App blocklist, key blocking, per-session action limits | | Audit Logging | Available | Every action logged to JSONL for compliance review | | FAILSAFE | Available | Mouse to (0,0) aborts immediately | | Policy Engine | Phase 1 | Risk classification, rule-based action filtering | | Real-time Monitor | Phase 2 | Live dashboard with event streaming | | Human-in-the-Loop | Phase 3 | Approval workflows for high-risk actions | | Agent Adapters | Phase 4 | Agent-agnostic backends (Claude, GPT, open-source) | | Compliance Reports | Phase 5 | Automated audit reports for EU AI Act, SOC 2 |

Quick Start

1. Install

git clone https://github.com/yonghwan1106/computer-use-engine.git
cd computer-use-engine
pip install -e .

2. Register with Claude

python scripts/register.py

This automatically adds CUE to both Claude Desktop and Claude Code configurations.

3. Restart Claude and go

Restart Claude Desktop or Claude Code. Then just ask:

"Take a screenshot of my screen"

"Open Notepad and type 'Hello, CUE!'"

"Show me all open windows"

Current Features

MCP Tools (12)

Screenshot & Screen

| Tool | Description | Parameters | |------|-------------|------------| | cue_screenshot | Capture full screen or a region as JPEG | region_x, region_y, region_width, region_height (all optional) | | cue_screen_size | Get screen resolution | — | | cue_cursor_position | Get current cursor coordinates | — |

Mouse

| Tool | Description | Parameters | |------|-------------|------------| | cue_click | Click at coordinates | x, y, button (left/right/middle), clicks (1-3) | | cue_scroll | Scroll at position | x, y, clicks (positive=up, negative=down) | | cue_move | Move cursor | x, y | | cue_drag | Drag from point A to B | start_x, start_y, end_x, end_y, button, duration |

Keyboard

| Tool | Description | Parameters | |------|-------------|------------| | cue_type | Type text (auto clipboard fallback for non-ASCII like Korean/CJK) | text | | cue_key | Press key or combo | key (e.g. "enter", "ctrl+c", "alt+tab") |

Window Management

| Tool | Description | Parameters | |------|-------------|------------| | cue_list_windows | List all visible windows with geometry | — | | cue_focus_window | Focus a window by partial title match | title | | cue_window_info | Get active window info | — |

Safety Features

| Feature | Description | Default | |---------|-------------|---------| | Action limit | Max actions per session before requiring reset | 100 | | App blocklist | Prevents interaction with sensitive apps | Registry Editor, Windows Security | | Key blocklist | Blocks dangerous key combos | win+r, ctrl+alt+del | | Audit log | Every action logged to JSONL file | cue_audit.jsonl | | FAILSAFE | Move mouse to (0, 0) to abort immediately | Enabled | | Action delay | Pause between actions for safety | 50ms |

Safety Policy

All safety settings are configurable in config/default.yaml:

safety:
  max_actions_per_session: 100
  action_delay: 0.05
  failsafe: true
  allowed_apps: []
  blocked_apps:
    - "Windows Security"
    - "Registry Editor"
    - "Task Manager"
  blocked_keys:
    - "win+r"
    - "alt+f4"
    - "ctrl+alt+del"

Manual Registration

If you prefer to configure manually instead of using register.py:

Claude Desktop — edit %APPDATA%/Claude/claude_desktop_config.json: ``json { "mcpServers": { "cue": { "command": "python", "args": ["-m", "cue"] } } } ``

Claude Code — edit ~/.claude/settings.json: ``json { "mcpServers": { "cue": { "command": "python", "args": ["-m", "cue"] } } } ``

Project Structure

computer-use-engine/
├── cue/
│   ├── __init__.py            # Package version
│   ├── __main__.py            # python -m cue entry point
│   ├── server.py              # FastMCP server initialization
│   ├── tools/
│   │   ├── screenshot.py      # Screen capture tools (3)
│   │   ├── mouse.py           # Mouse control tools (4)
│   │   ├── keyboard.py        # Keyboard input tools (2)
│   │   └── window.py          # Window management tools (3)
│   ├── safety/
│   │   ├── guardrails.py      # Action limits, app/key blocking
│   │   └── logger.py          # JSONL audit logger
│   ├── core/                  # Policy engine, risk scoring (Phase 1)
│   ├── monitor/               # Real-time dashboard (Phase 2)
│   ├── adapters/              # Agent-agnostic backends (Phase 4)
│   └── utils/
│       ├── screen.py          # DPI awareness, image processing
│       └── keymap.py          # xdotool → pyautogui key mapping
├── config/
│   └── default.yaml           # Safety configuration
├── scripts/
│   └── register.py            # Auto-registration for Claude
├── tests/                     # Unit tests
├── pyproject.toml
├── LICENSE                    # Apache 2.0
└── README.md

Development

# Install with dev dependencies
pip install -e ".[dev]"

# Run tests
pytest tests/ -v

# Run the server directly (stdio mode)
python -m cue

Roadmap

| Phase | Focus | Status | |-------|-------|--------| | MVP | MCP server, 12 tools, basic guardrails | Done | | Phase 1 | Policy engine, risk classification, session management | Next | | Phase 2 | Real-time monitoring dashboard, event streaming | Planned | | Phase 3 | Human-in-the-loop approval workflows | Planned | | Phase 4 | Agent-agnostic adapters (Claude, GPT, Agent-S) | Planned | | Phase 5 | Compliance reports (EU AI Act, SOC 2) | Planned |

Requirements

Python 3.11+
Windows 10/11
Claude Desktop or Claude Code with MCP support

Contributing

Contributions are welcome! Please see LICENSE for details.

License

Apache 2.0

computer-use-engine