Objectives
What newgle is for, and how each goal is measured. From the project charter and the crawler's design.
A library of the useful web, built for agents
A locally searchable corpus of the useful public web, made for AI-agent retrieval rather than human browsing. Each page keeps its URL and provenance; duplicates and SEO spam go; storage stays compact.
Reading for a fraction of the tokens
Agents read text, not markup. The raw HTML of helmwise.com cost 1,283,103 tokens and its text 71,473: 18 times more, and 94.4% of it markup.
Search agents can rely on
Benchmark hit@10 at least .650 with the docs and official categories up, the known-item set above 54%, and median latency within 10% of 385 ms.
Fetch it ourselves, politely
For each site on the crawl list: at least 90% of its robots-allowed English sitemap pages, with no site above its rate limit, every Crawl-delay honoured, and nothing fetched that robots.txt disallows.
The BHAG
Proposal · not yet written down
A clean-text mirror in front of every useful site on the web, so that no AI agent has to read raw HTML again.
The bet behind newgle, the Dead Reader theory: the readers left, not the writers. "No one is reading your website anymore — their agents are." Those agents spend most of their reading budget on markup.
- Own the library, not the crawl: the archive answers first, and a live fetch covers what's new.
- 18 times cheaper reading for every agent, for good.
- The path: 345 priority sites now, then the top 50,000 sites, then every host.
The crawl: what's left to fetch
The crawl list names the sites newgle fetches itself, politely and within robots.txt. It started its full run on 29 October 2026.
All known URLs
The biggest queues
| Site | Queued | Fetched | Done | Pages / day | To finish |
|---|
To finish = what's queued divided by the site's pace since midnight UTC. www.canada.ca was held back by a robots.txt bug, fixed on 1 Nov at 11:53 UTC, and has fetched steadily since. Stopped = the circuit breaker stopped the site after repeated errors; resuming it is a manual call.
Milestones
What's done, what's under way and what's next. All dates are 2026.
Legal corpus
No legal documents are in newgle yet, on purpose. The crawl leaves legal sites and legal sections out. The legal corpus is its own project after the crawl, built mostly from official bulk data: statutes and regulations as XML, court decisions from bulk dumps, and PDFs, with the first contextIR L2 pilot. Below are the sources that exist, and what it takes to get each.
Terms as published on each source's site, read on 1 Nov 2026. Rows marked check have terms that conflict or aren't verified yet; read them closely before use. This is a planning list, not legal advice.
Keeping the whole web fresh
What it would take for newgle to fetch every page of the public web itself, once a month. The yardstick is the largest public web archive's monthly crawl. The costs per page are newgle's own, measured on its crawler.
What a monthly refresh needs
| Resource | Need | Based on |
|---|---|---|
| CPU | About 35 cores busy around the clock | Measured: 18 ms per request to fetch, 13 ms per page to turn it into text |
| Bandwidth in | About 110 TB a month: 350 Mbps on average, or 750 Mbps to finish in two weeks | Measured 27 KB per request; the archive takes in 100–120 TB per crawl |
| Bandwidth out | About 4% of what comes in | The requests themselves |
| Raw pages (WARC files) | 50–90 TB per full pass | Ours average 22 KB per page, the archive's 44 KB |
| Extracted text | About 6 TB per pass | The archive's text files for the same crawl |
| Search corpus and index | 3.5 TB today, for 745M documents; about 8 TB with room to build the next one beside it | The live index on minis1 |
| URL database | 3–13 TB: 3.2B URLs fetched each month, 13.3B known | About 1 KB per URL in today's design |
On one server
Proposal The whole crawl on one machine in a data centre: a 64-core AMD EPYC (7002 series), 512 GB of memory, eight 6 TB enterprise NVMe drives, two 26 TB hard drives, and a 10 Gbps internet link with no transfer cap. The parts are in hand; it isn't built yet. One server has done it before: in 2007, the research crawler IRLbot fetched 6.3 billion pages in 41 days on one machine.
The server turns pages into text itself, and keeps the raw WARC files on its own MinIO node. Only the text crosses to the other site, where search and the MinIO cluster are: about 6 TB a pass, 20 Mbps averaged.
| Resource | Monthly refresh | The server | Used |
|---|---|---|---|
| CPU | About 35 threads | 64 cores, 128 threads | ~27% 58% for a two-week pass |
| Memory | About 110 GB for a fingerprint of every known URL, plus working memory | 512 GB | ~⅓ |
| Link | 350 Mbps on average, 110 TB a month | 10 Gbps internet, no transfer cap | ~4% 8% for a two-week pass |
| Fast storage | 7–9 TB: the URL database, WARC files until the MinIO node has them, text waiting to publish | 8 NVMe drives: 42 TB usable (layout below) | ~20% |
| Raw pages | 50–90 TB per pass | A MinIO node on the two 26 TB drives, mirrored: 26 TB | the latest 9–15 days today's whole crawl is about 0.1 TB |
| Search | 8 TB | Stays on minis1, at the other site | – |
Drive layout
| Array | Usable | Holds | Full |
|---|---|---|---|
| NVMe mirror, 2 drives | 6 TB | The system, the URL database and crawl state: the one part that can't be fetched again | ~55% less with a compact design |
| NVMe stripe, 6 drives | 36 TB | WARC files until the MinIO node has them (hours, 1–3 TB), and text until the weekly publish (about 3 TB) | 10–20% about 40% if search moves here too |
| Hard-drive mirror, 2 × 26 TB | 26 TB | The MinIO node: WARC files, the raw pages kept for re-conversion. It speaks the same S3 as the cluster, so the crawler only gets a new address. | under 1% today full in 9–15 days at full scale |
Writes come to under 0.3 drive writes a day on each striped drive, well within enterprise drives' rating. While the crawl is small, the MinIO node copies everything to the cluster as an off-site backup. Past the top 50,000 sites that stops fitting: the cluster has about 290 TB free, 3–5 passes. Keeping one whole pass of raw pages near half full would then take about eight 26 TB drives with two for parity.
What limits it is software
The hardware fits. The crawler was built for 345 sites; at 40 million, these are the hard parts.
The URL database
Today one SQLite file holds 15M URLs in 15.7 GB, already more than the crawl machine's memory. Billions of URLs need a store split by site, one part for each fetch process.
Many fetch processes
Fetch is one process today. 1,330 requests a second takes about 32 of them, each owning a share of the sites, with a local DNS cache for 40M hosts.
Politeness without review by hand
Each site on today's list was reviewed by hand. At 40M sites, robots.txt, Crawl-delay, stops after errors and takedowns all have to run on their own.
Big sites can't be refreshed monthly
At newgle's limit of 4 requests a second, one site yields at most about 10M pages a month. A site that asks for 10 seconds between requests yields about 260,000.
Change rates save less than they seem
A third of each monthly crawl is URLs never fetched before: 0.7B of 2.14B. "Not modified" answers are 0.45% of requests, and sitemap dates are unreliable. Fetching by change rate buys freshness where it matters, not a smaller bill.
The plan
Agreed 2 Nov 2026
The archive for the long tail, newgle's own crawl for what has to be fresh.
- The archive's monthly crawl keeps the long tail: about 6 TB of text a month, around 20 Mbps averaged.
- newgle's crawler fetches what has to be fresher than monthly, or what the archive misses: the 345 listed sites now, then the top 50,000.
- Proposal: build the server above, move the 345-site crawl onto it, and take on more of the web as the crawler is rebuilt for scale.