Objectives
What newgle is for, and how each goal is measured. From the project charter and the crawler's design.
A library of the useful web, built for agents
A locally searchable corpus of the useful public web, made for AI-agent retrieval rather than human browsing. Each page keeps its URL and provenance; duplicates and SEO spam go; storage stays compact.
Reading for a fraction of the tokens
Agents read text, not markup. The raw HTML of helmwise.com cost 1,283,103 tokens and its text 71,473: 18 times more, and 94.4% of it markup.
Search agents can rely on
Benchmark hit@10 at least .650 with the docs and official categories up, the known-item set above 54%, and median latency within 10% of 385 ms.
Fetch it ourselves, politely
For each site on the crawl list: at least 90% of its robots-allowed English sitemap pages, with no site above its rate limit, every Crawl-delay honoured, and nothing fetched that robots.txt disallows.
The BHAG
Proposal · not yet written down
A clean-text mirror in front of every useful site on the web, so that no AI agent has to read raw HTML again.
The bet behind newgle, the Dead Reader theory: the readers left, not the writers. "No one is reading your website anymore — their agents are." Those agents spend most of their reading budget on markup.
- Own the library, not the crawl: the archive answers first, and a live fetch covers what's new.
- 18 times cheaper reading for every agent, for good.
- The path: 345 priority sites now, then the top 50,000 sites, then every host.
The crawl: what's left to fetch
The crawl list names the sites newgle fetches itself, politely and within robots.txt. It started its full run on 29 October 2026.
All known URLs
The biggest queues
| Site | Queued | Fetched | Done | Pages / day | To finish |
|---|
To finish = what's queued divided by the site's pace since midnight UTC. www.canada.ca was held back by a robots.txt bug, fixed on 1 Nov at 11:53 UTC, and has fetched steadily since. Stopped = the circuit breaker stopped the site after repeated errors; resuming it is a manual call.
Milestones
What's done, what's under way and what's next. All dates are 2026.
Legal corpus
No legal documents are in newgle yet, on purpose. The crawl leaves legal sites and legal sections out. The legal corpus is its own project after the crawl, built mostly from official bulk data: statutes and regulations as XML, court decisions from bulk dumps, and PDFs, with the first contextIR L2 pilot. Below are the sources that exist, and what it takes to get each.
Terms as published on each source's site, read on 1 Nov 2026. Rows marked check have terms that conflict or aren't verified yet; read them closely before use. This is a planning list, not legal advice.
Keeping the whole web fresh
What it would take for newgle to fetch every page of the public web itself, once a month. The yardstick is the largest public web archive's monthly crawl. The costs per page are newgle's own, measured on its crawler.
What a monthly refresh needs
| Resource | Need | Based on |
|---|---|---|
| CPU | About 35 cores busy around the clock | Measured: 18 ms per request to fetch, 13 ms per page to turn it into text |
| Bandwidth in | About 110 TB a month: 350 Mbps on average, or 750 Mbps to finish in two weeks | Measured 27 KB per request; the archive takes in 100–120 TB per crawl |
| Bandwidth out | About 4% of what comes in | The requests themselves |
| Raw pages (WARC files) | 50–90 TB per full pass | Ours average 22 KB per page, the archive's 44 KB |
| Extracted text | About 6 TB per pass | The archive's text files for the same crawl |
| Search corpus and index | 3.5 TB today, for 745M documents; about 8 TB with room to build the next one beside it | The live index on minis1 |
| URL database | 3–13 TB: 3.2B URLs fetched each month, 13.3B known | About 1 KB per URL in today's design |
On one server
Proposal The whole crawl on one machine: a 64-core AMD EPYC (7002 series), 512 GB of memory, 6 TB enterprise NVMe drives and a 10 Gbps link. One server has done it before: in 2007, the research crawler IRLbot fetched 6.3 billion pages in 41 days on one machine.
| Resource | Monthly refresh | The server | Used |
|---|---|---|---|
| CPU | About 35 threads | 64 cores, 128 threads | ~27% 58% for a two-week pass |
| Memory | About 110 GB for a fingerprint of every known URL, plus working memory | 512 GB | ~⅓ |
| Link | 350 Mbps on average | 10 Gbps | ~4% 8% for a two-week pass |
| Fast storage | 9–12 TB: the URL database, WARC files waiting to upload, text waiting to publish | 6 drives: 30 TB usable (layout below) | 30–40% |
| Raw pages | 50–90 TB per pass | Not on the server: the MinIO cluster beside it, about 290 TB free | 3–5 passes |
| Search | 8 TB | Stays on minis1 | – |
Drive layout
| Array | Usable | Holds | Full |
|---|---|---|---|
| Mirror, 2 drives | 6 TB | The URL database and crawl state: the one part that can't be fetched again | ~55% less with a compact design |
| Stripe, 4 drives | 24 TB | WARC files until MinIO has them (1–2 days, 3–6 TB), and text until the weekly publish (about 3 TB) | 25–40% |
Writes come to under 0.3 drive writes a day on each striped drive, well within enterprise drives' rating. Moving search onto this server too would need two more drives in the stripe, to keep it near half full.
What limits it is software
The hardware fits. The crawler was built for 345 sites; at 40 million, these are the hard parts.
The URL database
Today one SQLite file holds 15M URLs in 15.7 GB, already more than the crawl machine's memory. Billions of URLs need a store split by site, one part for each fetch process.
Many fetch processes
Fetch is one process today. 1,330 requests a second takes about 32 of them, each owning a share of the sites, with a local DNS cache for 40M hosts.
Politeness without review by hand
Each site on today's list was reviewed by hand. At 40M sites, robots.txt, Crawl-delay, stops after errors and takedowns all have to run on their own.
Big sites can't be refreshed monthly
At newgle's limit of 4 requests a second, one site yields at most about 10M pages a month. A site that asks for 10 seconds between requests yields about 260,000.
Change rates save less than they seem
A third of each monthly crawl is URLs never fetched before: 0.7B of 2.14B. "Not modified" answers are 0.45% of requests, and sitemap dates are unreliable. Fetching by change rate buys freshness where it matters, not a smaller bill.
The plan
Agreed 2 Nov 2026
The archive for the long tail, newgle's own crawl for what has to be fresh.
- The archive's monthly crawl keeps the long tail: about 6 TB of text a month, around 20 Mbps averaged.
- newgle's crawler fetches what has to be fresher than monthly, or what the archive misses: the 345 listed sites now, then the top 50,000.
- Proposal: run it on the server above, and take on more of the web as the crawler is rebuilt for scale.