Astro-LLM

大模型黄历 · Astro-LLM

The LLM almanac. One dashboard that tells you, every day, (1) which model is sharp today and whether you should spend or save your tokens, and (2) which model actually shares your values — all wrapped in an astrology / weather-report skin that's fun on the surface and a real personal eval harness underneath.

It looks like a fortune-telling almanac. It's actually a benchmark.

▶ Live demo: https://2-actual-hack.vercel.app · Repo: https://github.com/EthanPany/astro-llm

!Astro-LLM showcase

Five completely different themed layouts (almanac · oracle · court · ink · aurora), a step-by-step value-match quiz, and a live model-fortune board. Full walkthrough: docs/showcase.mp4.

---

Why

Coding plans (Claude Code, Codex, …) feel slightly different day to day — sometimes the same model thinks longer, sometimes it's sharper, sometimes worse. And which model "gets" your taste is deeply personal. Astro-LLM turns both into a daily reading:

| Organ | Astrology name | What it really measures | |------|----------------|--------------------------| | Capability probe | 运势 fortune | Is the model sharp today? — 18 deterministic, auto-graded tasks → an IQ, latency, cost, refusal rate, trend. | | Value matcher | 缘分 affinity | Does it judge like you? — a bank of subjective dilemmas; we measure agreement (footrule + verdict match). | | MCP server | — | How an agent runs the test on itself. | | Harmonize loop | 调和 harmony | Can we nudge a model toward you? — inject your taste, re-test on held-out dilemmas. |

The "luck" is the skin; the numbers underneath are real. See the honesty note at the bottom of the dashboard.

---

Quickstart (local)

Requirements: Python 3.11+, uv, Node 20+.

cp .env.example .env        # paste your API keys (all optional — see below)
make install                # backend venv (uv) + frontend deps
make seed                   # probe live models, write committed seed JSON  (optional)
make dev-backend            # API on http://localhost:8000   (terminal 1)
make dev-frontend           # UI  on http://localhost:5173   (terminal 2)

Open http://localhost:5173.

No keys? It still works. The repo ships committed seed data, so the dashboard, verdict grid and value-match all render offline. Keys unlock live re-probes (↻ 实时巡检), the live custom-dilemma panel, and the harmonize loop.

Providers

ANTHROPIC_API_KEY, OPENAI_API_KEY, DEEPSEEK_API_KEY are wired and verified. GEMINI_API_KEY / QWEN_API_KEY are implemented; without them those models show clearly-labelled simulated data. Adding a key flips them to live on the next make seed.

---

Deploy (one command)

make up           # docker compose: builds the frontend, serves SPA + API on :8000
# or:
docker build -t astro-llm . && docker run -p 8000:8000 --env-file .env astro-llm

In production FastAPI serves the built SPA and the API from the same origin, so there's nothing else to configure. The image bakes in the seed data, so it renders even with zero keys.

---

Architecture

backend/   FastAPI + httpx + pydantic + SQLite     (uv-managed)
  app/
    providers/        one adapter per provider (OpenAI/Anthropic/DeepSeek/Gemini/Qwen)
    benchmarks/       dilemma bank, capability tasks, judge + capability runners,
                      matching (footrule + agreement), scoring (IQ + suggestion)
    astrology/        deterministic zodiac / numerology / fortune / compatibility
    loop/             harmonize (train/test guardrail)
    storage/          SQLite DAL + seed generator
    service.py        assembly: runs + astrology -> AlmanacCards + headline
    api/routes.py     REST API   |   mcp_server.py   MCP tools   |   scheduler.py
frontend/  React + Vite + TypeScript, handwritten CSS, GSAP, hand-drawn SVG charts

Full data contracts: ARCHITECTURE.md.

Earlier prompt harness

experiments/astro-twin-v1 preserves the earlier AstroTwin dual-pipeline prompt-testing harness. It remains runnable as a standalone Python project and records its exact source revision in SOURCE_REPOSITORIES.md.

API

GET /api/dashboard · GET /api/models · GET /api/dilemmas · POST /api/match · POST /api/harmonize · POST /api/judge · GET /api/health

MCP

make mcp        # python -m app.mcp_server  (stdio)

Tools: list_models, get_almanac, judge_dilemma, match_me, score_model_against_user. Point any MCP client (e.g. Claude Desktop) at it and an agent can score itself against the benchmark.

Hourly almanac

Set ENABLE_SCHEDULER=true to probe available models on an interval (PROBE_INTERVAL_MINUTES), persisting real runs to SQLite so the trend becomes measured data over time.

---

Tests

make test       # 34 tests: astrology, matching, scoring, graders, judge parsing,
                # harmonize, API (offline via TestClient + monkeypatched seed)

Design

Handwritten CSS only — no Tailwind, no component library, no chart library. Charts are hand-drawn SVG. Type: Fraunces (display) + Space Grotesk (UI) + Space Mono (numbers). GSAP drives entrance reveals, number count-ups, and chart draw-on.

---

Honesty note

The capability score is a real stopwatch + autograder. The value-match measures how a model was tuned to present itself, not a hidden soul. The harmonize loop is scored on held-out dilemmas the model never saw during tuning — the guardrail that keeps "alignment" from collapsing into a flattery machine. The astrology is for fun; we say so.

⚠ Rotate any API keys that were shared in plaintext before making this repo public.