Elasticsearch Frozen Indices
In Elasticsearch 6.6, two powerful features have been introduced, i.e., ILM and Frozen Indices. ILM refers to as Index Lifecycle Management, which we have already discussed in the previous chapter in detail. Click here to know Index Lifecycle Management. So, we will discuss Frozen indices in this chapter.
In this chapter, we will discuss the following topics -
What are Frozen indices?
Frozen indices are basically those indices which are rarely accessed. So, the users freeze these indices. Basically, we freeze those indices which are not in use to free up the memory. Such indices are called frozen indices. Elasticsearch provides _freeze API that helps the users to freeze the indexes.
Following are some important points of Frozen indices -
Note that the frozen indices are slower to search but take no heap memory.
Why freezes an index?
Elasticsearch allows the users to freeze the indices. So, it is important to know why freeze an index. Let's understand with the help of an example -
Some indices are frequently searched by the user, which takes time to re-build each time when they searched. So, these frequently searched indices are kept in memory to save time to re-build and search them efficiently. Thus, whenever a user searches for these indices, it can be searched efficiently in less time.
On the other hand, there may also be some indices that are rarely accessed are not kept in memory. Because it is not good to hold memory for indices, which are not often used. Therefore, users prefer to release the memory for the corresponding indices and free up space by freezing them. Such indices are called frozen indices.
One more possible solution is available that can close the index, which does not require memory. An index releases the heap memory after being closed, but we have to re-open that closed index to make a search. However, re-opening an index reduces operational cost, but it requires that heap memory which it was using before being closed.
Note: Whenever the frozen indices are required, they can be re-built, and resources are allocated for them.
How freezing works?
Elasticsearch provides a dedicated searched_throttled threadpool, which is used to search the Frozen indices. By default, it uses a single thread to ensure that the frozen indices are loaded into the memory one at a time.
Elasticsearch indices have some data structures in memory. These data structures make it easy for the users to search the indices and index into them efficiently. If in case there are too many indices, we can add up the sufficient memory for these data structures. Therefore, it is better to keep these data structures in memory for the frequently searched indices because they take time to re-build.
Elasticsearch builds transient data structures of each shard of a frozen index each time that shard is searched, and discards these data structure as soon as the search is complete. These transient data structures are not maintained by Elasticsearch in memory. When the indices become frozen, they consume less heap memory than the normal indices. Now, we will move to next how to freeze an index.
How to freeze an index?
We can freeze an index using _freeze API. These indices are searchable, but to write again, we have to unfreeze them. We can unfreeze the indices (frozen indices) using _unfreeze API. These API keywords are used in query while creating a query request.
Following is the syntax to freeze an index -
Here _freeze is Elasticsearch API and POST is the request method. In index_name, you have to pass the name of the index to perform the freeze operation on it.
In this below example, we will freeze the student index.
If your output same as the below response, the student index has frozen successfully.
How to unfreeze an index?
Frozen indices are searchable but we cannot write in them as they are read-only. If you try to write something in frozen index, it will give an error. To write something in frozen indices, we have to unfreeze them. Elasticsearch provides _unfreeze API to unfreeze the indices so that we can write in them.
Following is the syntax to unfreeze an index -
Here _unfreeze is an API and POST is request method. In index_name, you have to pass the name of the index on which you want to perform the operation.
In this below example, we will freeze the student index.
If your output same as the below response, the student index has unfrozen successfully.
Searching a Frozen Index
We can search the frozen indexes anytime whenever they needed. On frozen indices, regular search would not work. Therefore, to include the frozen indices in our search, we must have to use a parameter ignore_throttled = false while creating a query.
These indices re-build again and resources are also allocated to them. They can be loaded concurrently per node, limited by the number of threads in thread pool. The default value of it is 1.
Note: To control the number of concurrent searches, we can use search_throttled threadpool.
Create a search request to search the frozen index. This search query includes a request method, search API, name of the index. Along with that, it must contain a query parameter ignore_throttled = false that is used to include the frozen index in search.
Following syntax is used for searching a frozen index -
Here GET is request method and _search is API. In index_name, pass the name of index you want to search.
With the help of below examples, you will see how a normal search works differently on frozen indexes. See the example below -
Query without ignore_throttled
In this below example, we will find the document containing name: Alen Paul in student index, which is a frozen index.
You have not included ignore_throttled=false in your query. So, it will not include the student frozen index in search and not return any document.
Query with ignore_throttled parameter
Now, we will include ignore_throttled=false in our query and see the difference for the same index.
Here, you can see the difference that by including ignore_throttled=false, one document has returned that contains name: Alen Paul. Because it included the frozen index Student in its search.
Monitor Frozen Indices
Frozen indices are ordinary indices. These indices use a memory-efficient shard implementation and search throttling. Elasticsearch allows checking the status of the index using a command.
Execute the following code to check the status of your index-
Here, _cat indices command is used to provide a cross-section of each index, where v and h are parameters and GET is the request method. In index_name, pass the name of the index you want to search.
The response will be the same as the below output:
health status Index pri rep docs.count store.size yellow open Frozenexample 4 1 2 16.3
Error on adding document in frozen index
In case you try to add some data in the frozen index, it will generate an error. In the below screenshot, you can see that we have added a document in the student index, but it returned an error. See the below example -
By executing the above query on frozen index, it will generate unavailable shards exception error. Look at the screenshot below: