How long does it take to index a million documents on an ordinary computer? How large does the index grow, and how fast is search afterwards? Instead of a synthetic test, we took a real corpus of nearly one million files, indexed it with GroupDocs.Search for .NET 26.11 on a desktop PC and measured every step.

The short answer: 919,896 documents were indexed in 3 hours 48 minutes, and 17 of 20 kinds of search answered in 35 to 539 ms, although most queries were found in tens or hundreds of thousands of documents. The benchmark is open source, so you can measure your own documents the same way.

The results in one table

What Result
Documents indexed 919,896
Indexing 3 h 48 min, 67 documents per second
Index size 60.2 GB, 49.5 GB after optimization
Opening the optimized index 9.8 s
17 of 20 kinds of search 35 to 539 ms (median 158 ms) in repeated runs
The heaviest queries 8.5 to 87 s for results of millions to hundreds of millions of occurrences

The setup

  • Computer: Intel Core i5-11500 (6 cores, 12 threads), 32 GB of RAM, a SATA SSD, Windows 10 – an ordinary desktop PC, not a server.
  • Software: GroupDocs.Search for .NET 26.11 on .NET 10, with the default index settings.
  • Documents: the govdocs1 corpus of Digital Corpora – nearly one million files from US government web servers: PDF, HTML, Word, Excel, PowerPoint, text, XML, CSV, images and more. It is a tough corpus: many of its files are spreadsheets and data files, and every distinct number or code becomes a term, so the index ended up with 285.6 million unique terms.
  • Method: the text of the documents was extracted once and saved to files, so the indexing could be measured on its own. The data was then added to a new index in batches of up to 512 MB with 4 indexing threads, the index was optimized, and 20 kinds of search ran 3 times each.

Out of 983,840 extracted files, 63,943 yielded no text (PostScript, gzip, Flash and dBase files and files that could not be read, among others), and one document was too large for the index, so 919,896 documents were indexed.

Indexing: 920 thousand documents in under four hours

The indexing took 3 hours 48 minutes, or 67 documents per second on average, including 28 minutes spent reading the extracted data from disk. The index took 60.2 GB in 727 segments.

The most useful finding for planning is how the time of one call of Index.Add changed as the index grew:

Documents already in the index Average time of a call
0 to 27,000 12.7 s
208,000 to 338,000 52.7 s
497,000 to 639,000 98.9 s
639,000 to 920,000 109.4 s

The last call added only 1,240 documents and still took 113 s, as long as the calls with more than 9,000 documents before it. In a large index, the number of calls, not the number of documents, decides most of the indexing time. So add documents to a large index in batches as large as your memory allows.

Search: tenths of a second for most queries

The optimized index of 28 segments opened in 9.8 s. Each query then ran three times: the first run is what the first user of a freshly started application waits for, and the repeated runs are the usual case of a running service.

Kind of search Documents found First run Repeated runs
Simple word 202,079 539 ms 210 ms
Boolean 44,626 251 ms 149 ms
Phrase 241,277 591 ms 539 ms
Phrase with wildcards 32,270 183 ms 129 ms
Wildcard 267,231 347 ms 287 ms
Fuzzy 202,183 207 ms 157 ms
Case-sensitive 47,010 52 ms 35 ms
Faceted (in a field) 151,413 148 ms 124 ms
Word forms 150,522 163 ms 140 ms
Synonyms 202,299 211 ms 168 ms
Homophones 206,431 208 ms 160 ms
Spelling correction 213,795 168 ms 174 ms
Keyboard layout correction 213,720 182 ms 168 ms
Aliases 78,118 203 ms 143 ms
Document filter (PDF only) 88,732 147 ms 131 ms
Search by chunks 202,079 155 ms 137 ms
Object query 37,935 456 ms 236 ms
Numeric range 703,582 25.2 s 8.5 s
Regular expression 609,547 27.9 s 26.5 s
Date range of a year 138,876 92.1 s 87.2 s

Fuzzy search, spelling correction and keyboard layout correction are about as fast as a simple word search here, although the index holds 285.6 million terms.

The last three rows are deliberately extreme: the regular expression matched 310 million occurrences in two thirds of all documents, the numeric range 231 million occurrences in three quarters of them, and the date range covered a whole year. Since 26.11, repeated runs reuse the words matched by the first run, so the repeated times show where the time goes: into collecting the occurrences. A user rarely needs a result of that size, and a shorter range, a more specific pattern or a document filter keeps the result, and the time, small.

The full table with the queries and the numbers of occurrences is in the documentation.

Optimization: a job for the night

The optimization merged the 727 segments into 28 and shrank the index from 60.2 GB to 49.5 GB in 12 hours 24 minutes. The size of a merged segment is limited, so a large index is optimized into several segments rather than one. Searches are not blocked while an index is optimized, so the optimization of a large index can run in the background, for example at night.

Five takeaways for your project

  1. Extract once, index many times. Separate data extraction saves the extracted text to files, so you can rebuild the index or try other settings without extracting the documents again.
  2. Index in large batches. Each call of Index.Add costs more as the index grows, so fewer, larger calls index a large collection much faster.
  3. Open the index once. Opening a large index takes seconds; keep the Index object and run all searches on it – searches are thread-safe.
  4. Optimize in the background. Searches keep working while the index is optimized, so schedule the optimization for quiet hours.
  5. Keep the heaviest queries narrow. Regular expressions and ranges that match hundreds of millions of occurrences take seconds; filters and narrower patterns keep their results, and their time, small.

Run the benchmark on your documents

Your documents, queries and hardware decide your numbers, so measure them. The GroupDocs.Search for .NET benchmarks application measures the extraction, the indexing, the optimization and 20 kinds of search, reports the time and the peak memory of every operation and writes everything to a CSV file:

git clone https://github.com/groupdocs-search/full-text-search-benchmark-net.git
cd full-text-search-benchmark-net
dotnet run -c Release --project src/GroupDocs.Search.Benchmark

Before the run, set the license, the folders and the queries for the language of your documents at the top of Program.cs. A free temporary license is enough: without a license, the evaluation mode limits the number of indexed and found documents.

Resources