Objectives
What newgle is for, and how each goal is measured. From the project charter and the crawler's design.
A library of the useful web, built for agents
A locally searchable corpus of the useful public web, made for AI-agent retrieval rather than human browsing. Each page keeps its URL and provenance; duplicates and SEO spam go; storage stays compact.
Reading for a fraction of the tokens
Agents read text, not markup. The raw HTML of helmwise.com cost 1,283,103 tokens and its text 71,473: 18 times more, and 94.4% of it markup.
Search agents can rely on
Benchmark hit@10 at least .650 with the docs and official categories up, the known-item set above 54%, and median latency within 10% of 385 ms.
Fetch it ourselves, politely
For each site on the crawl list: at least 90% of its robots-allowed English sitemap pages, with no site above its rate limit, every Crawl-delay honoured, and nothing fetched that robots.txt disallows.
The BHAG
Proposal · not yet written down
A clean-text mirror in front of every useful site on the web, so that no AI agent has to read raw HTML again.
The bet behind newgle, the Dead Reader theory: the readers left, not the writers. "No one is reading your website anymore — their agents are." Those agents spend most of their reading budget on markup.
- Own the library, not the crawl: the archive answers first, and a live fetch covers what's new.
- 18 times cheaper reading for every agent, for good.
- The path: 345 priority sites now, then the top 50,000 sites, then every host.
The crawl: what's left to fetch
The crawl list names the sites newgle fetches itself, politely and within robots.txt. It started its full run on 29 October 2026.
All known URLs
The biggest queues
| Site | Queued | Fetched | Done | Pages / day | To finish |
|---|
To finish = what's queued divided by the site's pace since midnight UTC. www.canada.ca was held back by a robots.txt bug, fixed on 1 Nov at 11:53 UTC, and has fetched steadily since. Stopped = the circuit breaker stopped the site after repeated errors; resuming it is a manual call.
Milestones
What's done, what's under way and what's next. All dates are 2026.
Legal corpus
No legal documents are in newgle yet, on purpose. The crawl leaves legal sites and legal sections out. The legal corpus is its own project after the crawl, built mostly from official bulk data: statutes and regulations as XML, court decisions from bulk dumps, and PDFs, with the first contextIR L2 pilot. Below are the sources that exist, and what it takes to get each.
Terms as published on each source's site, read on 1 Nov 2026. Rows marked check have terms that conflict or aren't verified yet; read them closely before use. This is a planning list, not legal advice.