HiveMind

🐝 HiveMind - MCP Subagent Orchestrator

  

A high-concurrency, asynchronous MCP (Model Context Protocol) Orchestrator that coordinates multiple AI coding agents using the Copilot CLI as the underlying intelligence engine. Built on an Actor Model architecture for non-blocking operations, robust fault tolerance, and strict resource synchronization.

📋 Table of Contents

• Overview
• Key Features
• Architecture
• Components
• Installation
• Usage
• Configuration
• API Reference
• Development
• Project Structure
• Roadmap
• Contributing

🎯 Overview

HiveMind enables a Main Agent to spawn and coordinate multiple Subagents, each running isolated Copilot CLI instances. This allows for:

• Parallel task execution across multiple files/modules
• Hierarchical planning with dependency-aware scheduling
• Safe concurrent file access via lock management
• Unified observability with structured logging and tracing

Problem Solved

Traditional single-agent coding assistants struggle with:

• Large codebases requiring parallel analysis
• Multi-file refactoring with dependencies
• Long-running tasks that block the main thread
• Resource contention when multiple tools access files

HiveMind solves these by orchestrating a swarm of specialized agents that work concurrently while respecting file locks and execution order.

✨ Key Features

| Feature | Description | | ----------------------------- | ---------------------------------------------------------------------------------- | | 🚀 Autonomous Execution | Submit a task and let the orchestrator plan, schedule, and execute without polling | | 🔒 File Locking (Warden) | Pessimistic locking with deadlock detection and wait queues | | 📊 Hierarchical Planning | Break complex tasks into dependency graphs with parallel execution groups | | 🔌 MCP Integration | Exposes orchestration capabilities as MCP tools | | 📈 Observability | Structured logging, OpenTelemetry tracing, and real-time metrics | | 💾 Persistence | SQLite-based checkpointing with crash recovery | | 🎛️ Dashboard | REST API + WebSocket for real-time monitoring |

🏗️ Architecture

HiveMind uses a three-layer architecture:

┌─────────────────────────────────────────────────────────────────────────────┐
│                              CONTROL PLANE                                  │
│  ┌─────────────┐  ┌──────────────┐  ┌────────────┐  ┌─────────────────────┐ │
│  │ Orchestrator│  │Task Planner  │  │ Scheduler  │  │ Resource Warden     │ │
│  │    Core     │──│ (Decomposer) │──│            │──│ (Lock Manager)      │ │
│  └─────────────┘  └──────────────┘  └────────────┘  └─────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
                                     │
                                     ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                               DATA PLANE                                    │
│  ┌──────────────────────────┐            ┌──────────────────────────────┐   │
│  │      Event Bus (IPC)     │            │      Trace Aggregator        │   │
│  │  Pub/Sub Message Passing │            │  Logs, Metrics, Spans        │   │
│  └──────────────────────────┘            └──────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────────────┘
                                     │
                                     ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                             EXECUTION PLANE                                 │
│  ┌───────────┐  ┌───────────┐  ┌───────────┐       ┌───────────────────┐   │
│  │ Agent Pod │  │ Agent Pod │  │ Agent Pod │  ...  │  Copilot CLI      │   │
│  │     1     │  │     2     │  │     N     │       │  (Child Process)  │   │
│  └───────────┘  └───────────┘  └───────────┘       └───────────────────┘   │
└─────────────────────────────────────────────────────────────────────────────┘

Request Flow

1. Main Agent calls MCP tool (e.g., submit_task_auto)
2. Planner analyzes task, creates dependency graph
3. Scheduler creates execution groups respecting dependencies
4. Warden acquires file locks before each group executes
5. Agent Pods spawn Copilot CLI processes in parallel
6. Results aggregated and returned to Main Agent

🧩 Components

Core Components

| Component | Location | Description | | --------------------- | ------------------ | --------------------------------------------- | | Orchestrator Core | src/core/ | Central coordination, lifecycle management | | Agent Manager | src/agents/ | Agent pool, pod lifecycle, message routing | | Event Bus | src/ipc/ | Pub/sub messaging, async communication | | Lock Manager | src/locks/ | File locking, deadlock detection, wait queues | | Planner | src/planner/ | Task decomposition, dependency graphs | | CLI Adapter | src/cli/ | Copilot CLI spawning, I/O handling | | MCP Layer | src/mcp/ | Tool definitions, request handling | | Tracing | src/tracing/ | Structured logging, OpenTelemetry | | Persistence | src/persistence/ | SQLite, checkpointing, recovery | | Dashboard | src/dashboard/ | REST API, WebSocket server |

Type System

| Module | Description | | --------------------- | --------------------------------- | | src/types/agent.ts | Agent status, config, results | | src/types/job.ts | Job lifecycle, subtasks, progress | | src/types/lock.ts | Lock modes, handles, conflicts | | src/types/events.ts | Event bus messages, channels | | src/types/mcp.ts | MCP tool schemas, responses | | src/types/cli.ts | CLI command construction, options |

📦 Installation

Prerequisites

• Node.js 20+
• npm or yarn
• GitHub Copilot CLI installed and authenticated
• Git for version control

Install

# Clone the repository
git clone https://github.com/sureshsankaran/mcp-subagent-orchestrator.git
cd mcp-subagent-orchestrator

# Install dependencies
npm install

# Build the project
npm run build

Verify Installation

# Run tests
npm test

# Check build output
ls dist/

Quick Demo

Run the end-to-end demo to see all features in action:

# Run comprehensive demo showcasing all phases (1-7)
npm run demo:e2e

# Run basic component demo
npm run demo

The demo showcases:

• Event Bus - Pub/sub messaging with wildcard patterns
• State Machine - Agent lifecycle transitions
• Lock Manager - Resource locking with deadlock detection
• DAG - Task dependency graphs with parallel execution groups
• MCP Tools - 14+ orchestration tools exposed via MCP
• Structured Logging - Winston logger with JSON/pretty formats
• Distributed Tracing - OpenTelemetry integration
• Metrics Registry - Counters, histograms, and gauges
• Health Checks - Component health monitoring
• Full Orchestration - End-to-end task execution simulation

🚀 Usage

As an MCP Server

Register HiveMind as an MCP server in your AI tool configuration:

{
  "mcpServers": {
    "hivemind": {
      "command": "node",
      "args": ["path/to/mcp-subagent-orchestrator/dist/index.js"],
      "env": {
        "GH_TOKEN": "your-github-token"
      }
    }
  }
}

Programmatic Usage

import { OrchestratorCore, OrchestratorConfig } from 'mcp-subagent-orchestrator';

// Create orchestrator
const config: OrchestratorConfig = {
  maxConcurrentAgents: 5,
  workspacePath: '/path/to/workspace',
  persistence: {
    enabled: true,
    dbPath: '.hivemind/state.db'
  }
};

const orchestrator = new OrchestratorCore(config);
await orchestrator.initialize();

// Spawn a single agent
const agent = await orchestrator.spawnAgent('Fix the authentication bug in auth.ts');
const result = await orchestrator.waitForAgent(agent.id);

// Submit an autonomous job
const job = await orchestrator.submitJob('Refactor the entire API layer to use async/await');
const status = await orchestrator.waitForJob(job.id);

MCP Tool Examples

// Spawn a focused subagent
await mcp.callTool('spawn_agent', {
  task: 'Review auth.ts for security vulnerabilities',
  config: {
    system_instruction: 'You are a security expert. Be thorough.',
    allowed_paths: ['src/auth/'],
    read_only: true
  }
});

// Submit autonomous job (orchestrator handles everything)
await mcp.callTool('submit_task_auto', {
  instruction: 'Add comprehensive unit tests to all service files',
  strategy: 'parallel',
  max_agents: 4
});

// Monitor job progress
await mcp.callTool('get_job_status', { job_id: 'job-123' });

// Acquire file lock (for manual coordination)
await mcp.callTool('acquire_lock', {
  resource: 'src/api/users.ts',
  mode: 'exclusive',
  timeout_ms: 30000
});

⚙️ Configuration

Environment Variables

| Variable | Description | Default | | --------------------- | ------------------------------------- | -------------------- | | GH_TOKEN | GitHub token with Copilot access | Required | | HIVEMIND_LOG_LEVEL | Logging level (debug/info/warn/error) | info | | HIVEMIND_MAX_AGENTS | Maximum concurrent agents | 10 | | HIVEMIND_DB_PATH | SQLite database path | .hivemind/state.db | | HIVEMIND_PORT | Dashboard server port | 3000 | | HIVEMIND_PREFERRED_CLI_TYPE | Override detected CLI type (copilot_v2, copilot_v1, codex, gh_copilot, claude) | auto-detect | | HIVEMIND_PREFERRED_CLI_PATH | Absolute path to CLI binary (bypass detection) | auto-detect | | OTEL_ENABLED | Enable OpenTelemetry | false |

Orchestrator Config (file-first)

Configuration is loaded in this order: user overrides → env → hivemind.config.json → defaults.

Add a hivemind.config.json at the repo root to set project-wide defaults (used for both decomposition and spawned agents):

{
  "cli": {
    // Force a specific adapter type instead of auto-preference (copilot_v2 > copilot_v1 > codex > gh_copilot > claude)
    "preferred_cli_type": "codex",
    // Optional explicit binary path (skips detection)
    "preferred_cli_path": "/usr/local/bin/codex",
    // Existing options
    "cli_command": "copilot",
    "default_timeout_ms": 300000,
    "auto_approve_tools": true
  }
}

interface OrchestratorConfig {
  // Agent management
  maxConcurrentAgents: number;       // Max parallel agents
  agentTimeout: number;              // Agent execution timeout (ms)
  agentPoolSize: number;             // Pre-warmed agent pool size
  
  // Workspace
  workspacePath: string;             // Root workspace path
  trustedFolders: string[];          // Pre-trusted Copilot folders
  
  // Locking
  lockTimeout: number;               // Default lock acquisition timeout
  deadlockDetection: boolean;        // Enable deadlock detection
  
  // Persistence
  persistence: {
    enabled: boolean;
    dbPath: string;
    checkpointInterval: number;      // Checkpoint frequency (ms)
  };
  
  // Observability
  logging: {
    level: 'debug' | 'info' | 'warn' | 'error';
    format: 'json' | 'pretty';
  };
  
  // Security
  toolApproval: {
    allowedShellCommands: string[];
    deniedShellCommands: string[];
  };

  // CLI (decomposition + spawned agents)
  cli: {
    cli_command: string;
    preferred_cli_type?: 'copilot_v2' | 'copilot_v1' | 'codex' | 'gh_copilot' | 'claude' | 'unknown';
    preferred_cli_path?: string;
    default_timeout_ms: number;
    auto_approve_tools: boolean;
  };
}

📖 API Reference

MCP Tools

| Tool | Description | | --------------------- | ------------------------------------- | | spawn_agent | Spawn a single subagent with a task | | submit_task_auto | Submit job for autonomous execution | | get_job_status | Get current job status and progress | | get_job_tree | Get hierarchical view of job subtasks | | cancel_job | Cancel a running job | | acquire_lock | Manually acquire a file lock | | release_lock | Release a held lock | | list_locks | List all active locks | | stream_logs | Stream logs for job/agent | | orchestrator_health | Get orchestrator health status |

REST API Endpoints

| Endpoint | Method | Description | | ------------------------- | ------ | ------------------ | | /api/v1/agents | GET | List all agents | | /api/v1/agents | POST | Spawn new agent | | /api/v1/agents/:id | GET | Get agent details | | /api/v1/agents/:id | DELETE | Kill agent | | /api/v1/jobs | GET | List all jobs | | /api/v1/jobs | POST | Submit new job | | /api/v1/jobs/:id | GET | Get job details | | /api/v1/jobs/:id/cancel | POST | Cancel job | | /api/v1/locks | GET | List active locks | | /api/v1/system/health | GET | Health check | | /api/v1/system/metrics | GET | Prometheus metrics |

🛠️ Development

Scripts

# Build TypeScript
npm run build

# Watch mode
npm run build:watch

# Run tests
npm test

# Run tests with coverage
npm run test:coverage

# Lint code
npm run lint
npm run lint:fix

# Format code
npm run format

# Clean build artifacts
npm run clean

Testing

# Run all tests
npm test

# Run specific test file
npm test -- tests/unit/locks/LockManager.test.ts

# Run with coverage
npm run test:coverage

Code Quality

• ESLint for linting
• Prettier for formatting
• Jest for testing
• TypeScript strict mode enabled

📁 Project Structure

mcp-subagent-orchestrator/
├── src/
│   ├── agents/          # Agent pod management
│   ├── cli/             # Copilot CLI adapter
│   ├── core/            # Orchestrator core
│   ├── dashboard/       # REST API & WebSocket
│   ├── errors/          # Custom error classes
│   ├── ipc/             # Event bus & messaging
│   ├── locks/           # Lock manager (Warden)
│   ├── mcp/             # MCP tool definitions
│   ├── persistence/     # SQLite & checkpointing
│   ├── planner/         # Task decomposition
│   ├── tracing/         # Logging & telemetry
│   ├── types/           # TypeScript interfaces
│   ├── utils/           # Shared utilities
│   └── index.ts         # Main entry point
├── tests/
│   ├── unit/            # Unit tests
│   └── integration/     # Integration tests
├── tasks/               # Phase task breakdowns
├── dist/                # Compiled output
├── spec.md              # Full specification
├── TASKS.md             # Development roadmap
└── package.json

🗺️ Roadmap

The project is developed in 9 phases:

| Phase | Name | Status | Tasks | | ----- | ----------------------------- | --------------- | ----- | | 1 | Project Skeleton & Interfaces | ✅ Complete | 68 | | 2 | Nervous System (EventBus/IPC) | 🔄 In Progress | 52 | | 3 | CLI Abstraction Layer | ⏳ Planned | 68 | | 4 | File Warden (Lock Manager) | ⏳ Planned | 58 | | 5 | Hierarchical Planner | ⏳ Planned | 72 | | 6 | MCP Exposure | ⏳ Planned | 58 | | 7 | Observability | ⏳ Planned | 52 | | 8 | Persistence & Recovery | ⏳ Planned | 62 | | 9 | Dashboard & REST API | ⏳ Planned | 55 |

Total: ~542 detailed subtasks

See TASKS.md for detailed task breakdowns.

🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

Development Guidelines

• Follow TypeScript strict mode
• Write tests for new features
• Update documentation as needed
• Use conventional commit messages

📄 License

This project is licensed under the ISC License - see the LICENSE file for details.

---

<p align="center"> Built with 🐝 by the HiveMind team </p>