macos-sys-assist

macos-sys-assist

A focused macOS automation MCP server for reliable input simulation, window management, and window-specific screenshots — the three things AppleScript and bash do poorly.

  

---

What Is This?

macos-sys-assist is a Python-based MCP server that fills the gaps where bash + Chrome DevTools fall short. It uses pyobjc (native macOS APIs) and Core Graphics for low-level input simulation — more reliable than AppleScript's keystroke.

What It Does That bash/CDP Can't

| Capability | Why Not bash/CDP | |---|---| | Core Graphics click/type/key | AppleScript keystroke misses keys or fails silently. This uses CGEventPost — the same API macOS uses internally. | | Window-specific screenshots | bash screencapture captures the full screen; cropping is tedious. This captures just the window you want. | | Precise window geometry | osascript returns position inconsistently. This uses the Accessibility API for accurate pixel-level data. | | Multi-app window layouts | Arrange 3+ apps at specific positions in one command. bash needs multiple chained osascript calls. |

What It Does NOT Do (Use bash Instead)

| Tool | Why Use bash | |---|---| | Finding files | find / mdfind are simpler | | Reading files | cat / python3 -c | | Opening files | open command | | App queries | osascript -e 'tell app "System Events"...' | | Clipboard | pbpaste / pbcopy | | Screen resolution | system_profiler SPDisplaysDataType |

---

Quick Start

Installation

git clone https://github.com/YOUR_USERNAME/macos-sys-assist.git
cd macos-sys-assist
./setup.sh

Grant Permissions

1. Accessibility — System Settings → Privacy & Security → Accessibility → Add Terminal/Python
2. Screen Recording (for screenshots) — System Settings → Privacy & Security → Screen Recording → Add Terminal/Python

Configure Apps

Edit allowed_apps.json to control which apps can be automated.

---

Usage

Standalone Mode

./run.sh

OpenCode Integration

Add to opencode.jsonc:

"mcp": {
  "macos-sys-assist": {
    "type": "local",
    "command": ["/path/to/macos-sys-assist/run.sh"],
    "enabled": true
  }
}

Direct Python Usage (via bash)

.venv/bin/python3 -c "
import sys
sys.path.insert(0, '.')
from macos.input import InputSimulator
InputSimulator().click_at(100, 200, 'left')
"

---

Tool Reference

Input Simulation (Core Graphics)

| Tool | Description | Security | |------|-------------|----------| | click_at(x, y, button, double) | Click at screen coordinates | ⚠️ Confirmation | | type_string(text) | Type text character by character | ⚠️ Confirmation, max 500 chars | | press_key(combination) | Press key combo (e.g., cmd+tab) | ⚠️ Blocked combos enforced |

More reliable than AppleScript — uses CGEventPost instead of keystroke.

Window Management

| Tool | Description | Security | |------|-------------|----------| | move_window(x, y) | Move active window to coords | ⚠️ Confirmation | | resize_window(width, height) | Resize active window | ⚠️ Confirmation | | get_window_geometry(pid) | Get window position/size (Accurate) | Read-only |

Uses Accessibility API for pixel-level accuracy. More reliable than osascript.

Screenshots (Requires Screen Recording Permission)

| Tool | Description | |------|-------------| | screenshot(filepath, display_id) | Capture full screen | | screenshot_window(pid, filepath) | Capture specific window only — no cropping needed | | screenshot_region(x, y, w, h, filepath) | Capture a screen region | | get_displays() | Get all connected displays and resolutions |

---

When to Use This vs bash

✅ Use macos-sys-assist when:

• AppleScript keystroke or click fails silently
• You need a screenshot of just one window without browser chrome
• You're arranging 3+ app windows at specific positions for a workspace
• The task requires pixel-level coordinate accuracy

❌ Use bash when:

• Finding files (find, mdfind, ls)
• Reading files (cat, python3 -c)
• Opening files (open)
• Basic clipboard (pbpaste, pbcopy)
• Checking what app is frontmost (osascript)
• Launching apps (open -a)

🔄 Use Chrome DevTools when:

• Interacting with web pages (clicking buttons, filling forms)
• Uploading files to websites (base64 injection into <input type="file">)
• Reading page content
• Navigating multi-page web flows

---

Configuration

allowed_apps.json

Controls which apps can be automated:

{
  "allowed_apps": [
    {
      "bundle_id": "com.brave.Browser",
      "name": "Brave Browser",
      "allow_actions": true
    }
  ],
  "global_settings": {
    "require_confirmation_for_click": true,
    "require_confirmation_for_type": true,
    "max_string_length": 500,
    "blocked_key_combinations": [
      "cmd+q",
      "cmd+delete",
      "ctrl+alt+delete"
    ]
  }
}

---

Project Structure

macos-sys-assist/
├── server.py                 # Main MCP server entry point
├── config.py                 # Configuration management
├── security.py               # Security validation layer
├── allowed_apps.json         # Application allow-list
├── requirements.txt          # Python dependencies
├── setup.sh                  # Installation script
├── run.sh                    # Wrapper script
├── macos/                    # Native macOS API wrappers
│   ├── accessibility.py     # App queries, PID lookup
│   ├── window.py            # Window move/resize/geometry
│   ├── input.py             # Core Graphics click/type/key
│   ├── screenshot.py        # Screen capture (full/window/region)
│   └── task_engine.py       # Multi-step task execution
└── tools/                    # MCP tool definitions
    ├── information.py       # get_window_geometry
    ├── actions.py           # click_at, type_string, press_key, move/resize
    └── screenshot.py        # screenshot, screenshot_window, screenshot_region, get_displays

---

Roadmap

Completed ✅

• [x] Core Graphics input simulation (click, type, key)
• [x] Window management (move, resize, geometry)
• [x] Window-specific screenshots (no cropping)
• [x] Security layer (allow-list, blocked keys, confirmations)

Planned 📋

• [ ] Folder Watcher — Detect new files in Downloads, auto-organize by project
• [ ] System State — Battery, WiFi, disk space checks before long automations
• [ ] Window Layout Presets — Save/restore multi-app workspaces
• [ ] Calendar Integration — Meeting-aware automation scheduling

---

Security Model

Design Principles

1. No Shell Access — All operations use native macOS APIs
2. Explicit Allow-List — Only pre-approved apps can be controlled
3. Human-in-the-Loop — Invasive actions require user confirmation
4. Input Validation — Text length limits, key combo blocking

What's Blocked

| Threat | Mitigation | |--------|------------| | Unauthorized app control | Application allow-list | | Destructive key combos | Blocked combinations list | | Excessive text input | Maximum string length (500) | | Unconfirmed actions | Confirmation prompts |

---

Troubleshooting

"Accessibility permission not granted"

1. System Settings → Privacy & Security → Accessibility
2. Add Terminal.app or .venv/bin/python3
3. Ensure toggle is ON
4. Restart the server

"Screen Recording permission required"

1. System Settings → Privacy & Security → Screen Recording
2. Add Terminal.app or .venv/bin/python3
3. Ensure toggle is ON
4. Restart the server

"App not in allow-list"

1. Find the app's bundle ID: osascript -e 'id of app "AppName"'
2. Add it to allowed_apps.json
3. Restart the server

---

License

MIT License — see LICENSE

---

Acknowledgments

Built for the OpenCode AI assistant framework. Uses the Model Context Protocol for tool integration. Powered by pyobjc for native macOS API access.