⚔ Valhalla — Autoresearch-Valhalla
> Stand on the shoulders of failed research. > A shared, version-controlled knowledge base of AI-research experiences — > what was tried, whether it failed or worked, the root cause, and the lesson — > that any Claude Code instance can consult before committing to a research > direction, so agents don't walk a failed path twice.
In the age of large-scale, agent-driven research, the bottleneck is no longer running experiments — it's not re-running the ones that already failed. Every unrecorded dead end is a lesson the field re-learns at the cost of GPU-hours and weeks. Valhalla turns one researcher's failure into everyone's shortcut.
you are here ───► ✗ ✗ ✗ ✗ ✗ ───► the frontier
│ │ │ │ │
others already
fell here — read
the lesson, step over---
One-command setup
Option A — Claude Code plugin (recommended)
Inside Claude Code:
/plugin marketplace add lezhang7/Autoresearch-Valhalla
/plugin install valhalla@valhallaThat's it. The valhalla skill now loads automatically during research tasks, and you get the /valhalla, /valhalla-search, and /valhalla-add commands. The experience base travels with the plugin; refresh it any time with
/plugin marketplace update valhalla.
Option B — shell installer (CLI + skill, anywhere)
curl -fsSL https://raw.githubusercontent.com/lezhang7/Autoresearch-Valhalla/main/install.sh | bashClones Valhalla to ~/.valhalla, puts the valhalla CLI on your PATH, and installs the Claude Code skill into ~/.claude/skills/valhalla. Re-run any time to update.
---
How Claude Code uses it
Once installed, the skill prompts Claude to consult Valhalla proactively — before designing an experiment, picking a learning rate, debugging a diverging run, or choosing an autoresearch agent's next step:
You: Let's pretrain a 1.3B model, I'm thinking lr=3e-3 with no warmup.
Claude: Before we commit — checking Valhalla…
⚔ found: "Aggressive LR with no warmup → irrecoverable loss spike"
[✗ failure] domain: llm-pretraining
lesson → above ~1e-3 at this scale, skipping warmup spikes the loss
within ~200 steps and it never recovers. Use ≥2000 warmup steps and
cap lr at 6e-4, or gradient-clip aggressively.
→ experiences/karpathy-lab/nano-scale/2025-... .md
I'd start at lr=6e-4 with 2000 warmup steps instead. Want me to?You can also drive it manually:
/valhalla-search rlhf reward hacking
/valhalla-add # record a new experience after a run resolves…or from any shell:
valhalla search "eval looks too good contamination" --outcome failure
valhalla list --domain rag
valhalla show experiences/<user>/<project>/<file>.md
valhalla stats---
What's an "experience"?
A single Markdown file with structured frontmatter and six short sections, organized as experiences/<contributor>/<project>/<slug>.md:
---
title: Aggressive LR with no warmup → irrecoverable loss spike
author: karpathy-lab
project: nano-scale
domain: llm-pretraining
outcome: failure
tags: [learning-rate, warmup, loss-spike, stability]
date: 2025-11-02
---
## Context What you were trying to do.
## Approach What you tried (config, scale, hyperparameters).
## Outcome What actually happened (metrics, failure mode).
## Root cause Why it happened — the mechanism.
## Lesson The one actionable takeaway. What to do instead.
## References Links, papers, commits, runs.Organized by contributor, by project — so Valhalla is the union of many people's hard-won lessons, browsable by who learned them and on what. See templates/experience.template.md.
---
Contributing
Valhalla is only as valuable as the lessons in it. After a run resolves with a non-obvious lesson, record it:
valhalla add --title "..." --author <your-handle> --project <name> \
--domain <domain> --outcome failure --tags "tag1, tag2"
# fill in the sections, then open a pull requestSee CONTRIBUTING.md. Every PR is auto-validated (valhalla validate) so the base stays parseable.
---
CLI reference
| Command | What it does | |---|---| | valhalla search <q> [--domain] [--outcome] [--tag] [--user] [--project] [--limit N] [--full] | Ranked relevance search | | valhalla show <path> | Print one experience in full | | valhalla list [filters] | Browse experiences | | valhalla stats | Counts by outcome / domain / contributor | | valhalla users / valhalla projects | Who and what is in here | | valhalla add [...] | Scaffold a new experience | | valhalla sync [--push] | git pull latest (and optionally push yours) | | valhalla validate | Check every record parses (used in CI) | | valhalla home | Print the resolved Valhalla directory |
Pure Python standard library — no dependencies, Python 3.8+.
---
Philosophy
> Skál. Drink to the runs that died so yours might live.
The hall of Valhalla is where fallen warriors are honored — not mourned. Here, fallen experiments are honored the same way: their failure made useful. The mission is to advance AI research faster by refusing to re-walk known dead ends. Consult Valhalla before you act; contribute to it after you learn.
License
MIT.
