A new MCP server and CLI called jevcrawl gives AI agents a way to crawl websites that finds the answer page more often, costs a fraction of current approaches, and returns only the passages that matter. The system uses a small decision model to score every link and passage, so the agent's LLM reads only what answers its question — at a few cents per crawl.
How jevcrawl picks links and passages
Agents that browse a site have two bad options. They can read everything a crawler fetches, which buries the answer in navigation and boilerplate and burns tokens. Or they can let the LLM pick each link, which is slow and costs dollars per question. jevcrawl does neither. A compact decision model scores every link and passage for relevance to the goal. The model only scores; it never follows links or generates text. This means page text is treated as data, never as instructions — making the system safe against prompt injection by design. An injected "ignore previous instructions" can at worst bias a score.
Every returned passage is copied verbatim from the source page with its URL, heading path, and character offsets. The character offsets index into the page text whose SHA-256 hash is stored in the result, so a quote can be verified later. The decision model returns a probability per yes/no question, so it cannot produce text. Everything the agent receives comes directly from the page.
Benchmark results show major gains
The benchmark uses WebWalkerQA from the WebWalker paper by Wu et al. at Alibaba Tongyi Lab (ACL 2025). The dataset contains questions whose answers sit several clicks deep in real websites — conferences, universities, organizations, games — in English and Chinese. Each task was re-checked live before the run; the site and answer page still load and still contain the reference answer. Tasks examined during development are excluded.
The accuracy run covered 50 tasks on 2026-10-12. The answering model was claude-opus-5-5 and the judge was claude-sonnet-5-5. jevcrawl found the answer page in 54 percent of tasks. When Claude picks every link, the success rate drops to 33 percent. Crawl4AI's goal-directed crawl, using its BestFirstCrawlingStrategy with a keyword scorer, was tested on 20 different tasks on 2026-10-13 with version 0.9.4. jevcrawl finds the answer page twice as often as Crawl4AI. Baselines included breadth-first search to depth two. All systems shared jevcrawl's fetcher and removed navigation and footers.
The authors note that 50 and 20 tasks are small samples: the cost gap is large and stable, while the accuracy gap is less so. Crawl4AI drives a real browser; jevcrawl does not run JavaScript.
Cost and speed comparison
Jev charges for input tokens only at $0.042 per million for model jev-1.13.0. In the benchmark, a crawl cost $0.011 on average. The crawl decisions cost about a cent. This makes jevcrawl about eight times cheaper per task and ten times cheaper per correct answer than letting Claude crawl. The system processes four pages at a time with one decision request per page, making it roughly twice as fast as an LLM-driven crawl.
A real result illustrates the efficiency. Crawling docs.python.org for "What does the walrus operator := do and which Python version added it?" fetched six pages. The raw page text totaled 80,034 tokens. jevcrawl returned 1,873 tokens of answer passages for $0.0069 in 8.3 seconds. The top block had relevance 0.97 and pointed to the section "What's New In Python 3.8 > New Features > Assignment expressions" at character offsets 307 to 580.
Results include blocks (relevant passages, best first) and uncertain (borderline passages worth reading before concluding something is missing). The stats object reports pages fetched, raw tokens, returned tokens, Jev calls, cache hits, Jev input tokens, Jev cost in USD, model version, degraded status, URLs needing JavaScript, stop reason, and elapsed seconds. Stop reasons include answered, frontier empty, max pages, max seconds, budget, or auth error. Expected failures return an error code and message; the tools never raise into the agent's session.
Tools and integration
Three main tools are available. The crawl tool takes a goal, start URL, and optional limits (max pages 30, max seconds 120, return tokens 4000, max decision USD 0.05) and returns relevant passages. The scrape tool handles one known page — with a goal it returns only relevant passages; without a goal it returns all content with no Jev call. The map tool returns ranked URLs to decide what to scrape, with a limit of 50 and max pages of 10.
jevcrawl works everywhere MCP does: Claude Code, Claude Desktop, Cursor, or the CLI. Setup requires a Jev API key from console.typesafe.ai and uv. In Claude Code: claude mcp add jevcrawl -e TYPESAFE_API_KEY=. For other MCP clients, configure mcpServers with command uvx, args ["jevcrawl"], and env TYPESAFE_API_KEY. From the command line: uvx jevcrawl crawl https://docs.python.org/3/ --goal "What does the walrus operator do?" (add --json for machine output). The doctor command checks the setup.
Environment variables control behavior. TYPESAFE_API_KEY is required. JEVCRAWL_BASE_URL defaults to https://api.typesafe.ai. JEVCRAWL_MODEL defaults to jev-1.13.0 (pinned because thresholds are calibrated per version). JEVCRAWL_CACHE_DIR defaults to ~/.jevcrawl/cache. All other thresholds, links per page, and concurrency settings have measured defaults in jevcrawl/config.py; Python callers can pass their own Config.
Reliability and limits
Links below the priority floor are kept in reserve, not dropped, so a crawl never stops early while it still has page budget. After three failed Jev requests, the crawl continues breadth-first, returns every block unscored in uncertain, and sets stats.degraded. Repeats are free — Jev's answers are cached locally, so the same goal on the same pages costs nothing. The API key is read from the environment and never written to disk or logs.
jevcrawl does not read robots.txt and does not rate-limit itself beyond fetching four pages at a time. Users are responsible for the sites they crawl, their terms of service, and applicable law. The tool never logs in, does not try to bypass paywalls or CAPTCHAs, and does not run JavaScript (pages that need it are listed in stats.needs_js_urls). Every call is bounded by max_pages, max_seconds, max_decision_usd, and return_tokens. The max_decision_usd is a hard limit: a request that could pass it is never sent, and the crawl stops with budget.
Tests run offline with all HTTP mocked. Live tests make one tiny real request, a fraction of a cent, marked with the live flag. The benchmark code and reproduction instructions live in bench/. The README charts are drawn by assets/build.py from measured numbers.