Skip to main content
Adding new documents to an index is a multi-threaded and memory-intensive operation. Meilisearch’s indexes are at the core of what makes our search engine fast, relevant, and reliable. This article explains some of the details regarding RAM consumption and multi-threading.

​
RAM

By default, our indexer uses the sysinfo Rust library to calculate a machine’s total memory size. Meilisearch then adapts its behavior so indexing uses a maximum two thirds of available resources. Alternatively, you can use the --max-indexing-memory instance option to manually control the maximum amount of RAM Meilisearch can consume. It is important to prevent Meilisearch from using all available memory during indexing. If that happens, there are two negative consequences:
  1. Meilisearch may be killed by the OS for over-consuming RAM
  2. Search performance may decrease while the indexer is processing an update
Memory overconsumption can still happen in two cases:
  1. When letting Meilisearch automatically set the maximum amount of memory used during indexing, sysinfo may not be able to calculate the amount of available RAM for certain OSes. Meilisearch still makes an educated estimate and adapts its behavior based on that, but crashes may still happen in this case. Follow this link for an exhaustive list of OSes supported by sysinfo
  2. Lower-end machines might struggle when processing huge datasets. Splitting your data payload into smaller batches can help in this case. For more information, consult the section below

​
Multi-threading

In machines with multi-core processors, the indexer avoids using more than half of the available processing units. For example, if your machine has twelve cores, the indexer will try to use six of them at most. This ensures Meilisearch is always ready to perform searches, even while you are updating an index. You can override Meilisearch’s default threading limit by using the --max-indexing-threads instance option. Allowing Meilisearch to use all processor cores for indexing might negatively impact your users’ search experience. Multi-threading is unfortunately not possible in machines with only one processor core.

​
Memory crashes

In some cases, the OS will interrupt Meilisearch and stop all its processes. Most of these crashes happen during indexing and are a result of a machine running out of RAM. This means your computer does not have enough memory to process your dataset.

​
Diagnosing memory issues

Before making changes, identify the root cause:
  • Check your --max-indexing-memory setting: If you have manually configured --max-indexing-memory to a value close to or exceeding your machine’s total available RAM, Meilisearch may consume too much memory during indexing. Try lowering this value to leave room for the OS and other processes.
  • Monitor RSS usage: Use tools such as top, htop, or ps to monitor the Resident Set Size (RSS) of the Meilisearch process during indexing. If RSS approaches the machine’s total available memory, the OS may kill the process via the OOM (Out Of Memory) killer.
  • Evaluate dataset size relative to available RAM: As a general guideline, your machine should have enough RAM to hold the full dataset in memory during indexing. If your dataset is significantly larger than available RAM, memory crashes become more likely.
  • Check system logs: On Linux, inspect dmesg or /var/log/syslog for OOM killer messages. These logs confirm whether the OS terminated Meilisearch due to memory pressure.

​
Mitigating memory crashes

If you are struggling with memory-related crashes, consider:
  • Adding new documents in smaller batches to reduce peak memory consumption during indexing
  • Lowering the --max-indexing-memory value so Meilisearch reserves less memory for indexing
  • Increasing your machine’s RAM
  • Reducing the number of searchable, filterable, and sortable attributes in your index settings, as each adds to indexing memory requirements
  • Following indexing best practices