How The Torch Search Engine Works: Crawling & Index

Quick answer: how onion search works

Onion search engines crawl .onion pages through Tor, store text in an index, and rank matches for a query. Coverage is partial because crawlers cannot reach every service.

From onion service to search result

An onion search engine performs three broad jobs. It discovers onion services, it fetches and stores information about their pages, and it answers queries against that stored information. Torch, like other engines in this category, is best understood as a pipeline rather than a single program.

Discovery and crawling

Crawlers start from seeds: known onion addresses, links found on pages already crawled, and addresses submitted by users. From each page the crawler extracts links and adds new addresses to a queue. Because .onion addresses are not resolved through public DNS, every fetch must go through the Tor network. That makes crawling slower and more fragile than crawling the clear web.

Many services are never found. A service that is not linked from anywhere the crawler visits, or that is only shared privately, will not appear in the queue. A service that is offline when the crawler calls may be skipped or recorded as unreachable. A service that blocks automated clients may be invisible even if it is popular.

Indexing

Indexing turns fetched pages into searchable data. The engine extracts text, titles, headings, and links, and stores them in a data structure that supports keyword lookup. It may also store a snippet, a content hash, a timestamp, and the response status. What is not stored cannot be searched later, so the design of the index determines what kinds of questions the engine can answer.

Ranking and query handling

When you submit a query, the engine matches your terms against the index and orders the results. Ranking may consider term frequency, title matches, link structure, recency, or service-level signals. Most onion engines publish little about their ranking, and rankings can change without notice. A high position means the engine thinks a page matches, not that the page is important, trustworthy, or online.

Freshness and drift

Onion services appear and disappear frequently. An index is always a snapshot with delays. A result can point to a service that has gone offline, changed ownership, moved to a new address, or been replaced by a different site. Older engines such as Torch can be especially useful for historical leads, but the age of an entry is a warning as well as an asset.

Coverage bias

Coverage is uneven in ways that are hard to measure. Well-linked services are crawled more often. Text-heavy services are indexed more fully. Services that require login, use heavy JavaScript, or return errors may be poorly represented. Any conclusion of the form this service does not exist, drawn only from a search engine, is unsafe.

Implications for a research database

Use several discovery methods, including curated directories, link following, and search engines. Keep provenance for each entry. Distinguish between what the engine returned and what you verified yourself. The next page, Torch features and indexing, looks at what this means in practice.

Last updated: October 2026. Site map.

Torch FAQ

How does an onion search engine work?

It discovers onion services, fetches pages through Tor, stores text in an index, and answers keyword queries against that index.

Why is onion search coverage partial?

Many services are unlinked, private, offline, or hostile to crawlers, so no index can see the whole network.

Does Torch see every .onion page?

No. Crawlers only index what they can discover and fetch. Absence from the index is not proof of absence.

How fresh is the Torch index?

Freshness varies and is rarely published. Repeat a query after a few weeks to estimate how quickly results change.