A new API service aims to solve a persistent problem for developers building RAG pipelines and LLM-powered applications: getting clean, readable markdown from messy web pages without burning through token budgets. The service, hosted at markdown.usemy.cloud, combines a two-tier extraction engine with a headless browser pool that can bypass Cloudflare Turnstile and JavaScript challenges that typically block standard scraping libraries.
Why most web scrapers fail at feeding LLMs
Raw HTML is a terrible input for language models. Navigation bars, cookie consent banners, sidebar widgets, and ad blocks all consume context window tokens without contributing useful content. A standard scraper might pull 10,000 tokens from a page, but only 3,000 of those tokens contain the actual article text. When you are paying per token for GPT-4 or Claude, that overhead adds up fast. The markdown.usemy.cloud service claims a 75% reduction in token costs by stripping boilerplate before returning markdown, which means developers can fit roughly four times more content into the same context window budget.
The API also pre-computes tiktoken counts for every response, so callers know exactly how many tokens they are about to send to their language model before making the request. This eliminates the guesswork when planning context windows across multiple source documents, which is a real pain point when building pipelines that pull from dozens of pages at once.
Two tiers of extraction for different problem classes
The extraction engine operates in two tiers. Tier 1 handles straightforward pages where the target content is accessible without executing JavaScript. This tier is fast and lightweight, suitable for blogs, documentation sites, and static pages. Tier 2 activates a headless Chromium pool for pages that require JavaScript execution or have anti-bot protections. Cloudflare Turnstile, JavaScript challenges, and other anti-bot mechanisms are handled automatically without manual intervention.
The headless browser pool is the key differentiator here. Most developer-facing scraping APIs either require users to configure their own browser instances or simply fail against Cloudflare-protected sites. A Turnstile challenge, for example, requires rendering JavaScript, generating browser fingerprints, and solving a challenge token. The Chromium pool manages this transparently, returning markdown as if the protection layer did not exist. This is a significant time saver for teams that would otherwise maintain their own browser automation infrastructure.
Caching reduces repeated fetches to 1ms
When the same URL is scraped more than once within an hour, the second request returns directly from RAM cache with a 1ms latency. There is no network egress and no margin cost on repeated queries. For teams that repeatedly reference the same documentation pages or news articles in their RAG pipelines, this means the marginal cost of re-fetching drops to effectively zero after the first request.
The caching layer is particularly relevant for development workflows where the same set of reference URLs gets queried across multiple test runs, pipeline iterations, and debugging sessions. Instead of hammering the same endpoints repeatedly, the cached responses serve instantly from memory. This also reduces the risk of getting rate-limited by target sites during development.
Integration with RAG and agent frameworks
The markdown output is designed to feed directly into popular LLM toolchains. LangChain agents, RAG vector stores, and custom retrieval pipelines can consume the markdown without further cleaning. The output format is clean markdown with no residual HTML tags, no script blocks, and no tracking pixels. This makes it straightforward to chunk the content, generate embeddings, and index it without an intermediate sanitization step.
For developers building document search or question-answering systems, the pipeline becomes simpler: send a URL to the API, receive clean markdown, chunk it, embed it, and store it. The tiktoken count on each response helps with chunking decisions, since you can plan token budgets per chunk rather than discovering overflow issues at query time.
Getting started through RapidAPI
The service is available through RapidAPI with a free tier for testing. The live showcase URLs on the product page demonstrate extraction latency and Cloudflare bypass in real time. Custom URL scraping requires a subscription through the RapidAPI gateway, which handles billing, rate limiting, and API key management. The setup is minimal: subscribe, grab your API key, and call the endpoint with a URL.
For teams that need higher throughput or custom configurations, the RapidAPI gateway scales through its standard tiered plans. The service itself handles the infrastructure complexity of maintaining a Chromium pool, managing Cloudflare bypass tokens, and keeping the extraction rules current as target sites change their HTML structures. That operational burden is the real value proposition, more so than the markdown conversion itself.