Elasticsearch Frozen Indices

In Elasticsearch 6.6, two powerful features have been introduced, i.e., ILM and Frozen Indices. ILM refers to as Index Lifecycle Management, which we have already discussed in the previous chapter in detail. Click here to know Index Lifecycle Management. So, we will discuss Frozen indices in this chapter.

In this chapter, we will discuss the following topics -

What are Frozen indices
Why freeze an index
How freezing works
How to freeze an index
How to unfreeze an index
Search an index
Monitor an index
Error on adding document in Frozen indices

What are Frozen indices?

Frozen indices are basically those indices which are rarely accessed. So, the users freeze these indices. Basically, we freeze those indices which are not in use to free up the memory. Such indices are called frozen indices. Elasticsearch provides _freeze API that helps the users to freeze the indexes.

Following are some important points of Frozen indices -

Frozen indices are read-only and its resources are no longer kept active. This means the resources of frozen indices are destroyed and re-allocated to them on demand.
Elasticsearch 6.6 came with a new API called freeze API that allows us to freeze the indices, which are not in use.
Frozen indices are always open for search means that we can search the data even in frozen indices anytime.

Note that the frozen indices are slower to search but take no heap memory.

Elasticsearch offers a dedicated ignore_throttled parameter, which is used to include the frozen indices in your search. Thus, we don't need to re-open them to make available for search.
Frozen indices are searchable, but to write again, we have to unfreeze them.

Why freezes an index?

Elasticsearch allows the users to freeze the indices. So, it is important to know why freeze an index. Let's understand with the help of an example -

Some indices are frequently searched by the user, which takes time to re-build each time when they searched. So, these frequently searched indices are kept in memory to save time to re-build and search them efficiently. Thus, whenever a user searches for these indices, it can be searched efficiently in less time.

On the other hand, there may also be some indices that are rarely accessed are not kept in memory. Because it is not good to hold memory for indices, which are not often used. Therefore, users prefer to release the memory for the corresponding indices and free up space by freezing them. Such indices are called frozen indices.

One more possible solution is available that can close the index, which does not require memory. An index releases the heap memory after being closed, but we have to re-open that closed index to make a search. However, re-opening an index reduces operational cost, but it requires that heap memory which it was using before being closed.

Note: Whenever the frozen indices are required, they can be re-built, and resources are allocated for them.

How freezing works?

Elasticsearch provides a dedicated searched_throttled threadpool, which is used to search the Frozen indices. By default, it uses a single thread to ensure that the frozen indices are loaded into the memory one at a time.

As the frozen indices are not kept in cache memory, it takes time to search. It may take seconds or minutes to complete.
It might be possible that when the indices were not frozen, they could take a bit of millisecond to search.
Generally, Frozen indices are not intended for high search load.

Elasticsearch indices have some data structures in memory. These data structures make it easy for the users to search the indices and index into them efficiently. If in case there are too many indices, we can add up the sufficient memory for these data structures. Therefore, it is better to keep these data structures in memory for the frequently searched indices because they take time to re-build.

Elasticsearch builds transient data structures of each shard of a frozen index each time that shard is searched, and discards these data structure as soon as the search is complete. These transient data structures are not maintained by Elasticsearch in memory. When the indices become frozen, they consume less heap memory than the normal indices. Now, we will move to next how to freeze an index.

How to freeze an index?

We can freeze an index using _freeze API. These indices are searchable, but to write again, we have to unfreeze them. We can unfreeze the indices (frozen indices) using _unfreeze API. These API keywords are used in query while creating a query request.

Syntax

Following is the syntax to freeze an index -

POST /index_name/
_freeze

Here _freeze is Elasticsearch API and POST is the request method. In index_name, you have to pass the name of the index to perform the freeze operation on it.

Example

In this below example, we will freeze the student index.

POST /student/
_freeze
{ }

Response

If your output same as the below response, the student index has frozen successfully.

{
	"acknowledged": true,
	"shards_acknowledged": true
}

Screenshot

How to unfreeze an index?

Frozen indices are searchable but we cannot write in them as they are read-only. If you try to write something in frozen index, it will give an error. To write something in frozen indices, we have to unfreeze them. Elasticsearch provides _unfreeze API to unfreeze the indices so that we can write in them.

Syntax

Following is the syntax to unfreeze an index -

POST /index_name/
_unfreeze

Here _unfreeze is an API and POST is request method. In index_name, you have to pass the name of the index on which you want to perform the operation.

Example

In this below example, we will freeze the student index.

POST /student/
_unfreeze
{ }

Response

If your output same as the below response, the student index has unfrozen successfully.

{
	"acknowledged": true,
	"shards_acknowledged": true
}

Screenshot

Searching a Frozen Index

We can search the frozen indexes anytime whenever they needed. On frozen indices, regular search would not work. Therefore, to include the frozen indices in our search, we must have to use a parameter ignore_throttled = false while creating a query.

These indices re-build again and resources are also allocated to them. They can be loaded concurrently per node, limited by the number of threads in thread pool. The default value of it is 1.

Note: To control the number of concurrent searches, we can use search_throttled threadpool.

Create a search request to search the frozen index. This search query includes a request method, search API, name of the index. Along with that, it must contain a query parameter ignore_throttled = false that is used to include the frozen index in search.

Syntax

Following syntax is used for searching a frozen index -

GET /index_name/
_search?q=parameter:parameter_value&ignore_throttled=false

Here GET is request method and _search is API. In index_name, pass the name of index you want to search.

Examples

With the help of below examples, you will see how a normal search works differently on frozen indexes. See the example below -

Query without ignore_throttled

In this below example, we will find the document containing name: Alen Paul in student index, which is a frozen index.

GET /student/
_search?q=name:Alen Paul
{ }

Response

You have not included ignore_throttled=false in your query. So, it will not include the student frozen index in search and not return any document.

{
	"took": 0,
"timed_out": true,
	"_shards": {
	     "total": 0,
	     "successful": 0,
	     "skipped": 0,
	     "failed": 0
	},
	"hits": {
	     "total": {
	         "value": 0,
	         "relation": "eq"
	       }
	     "max_score": 0,
	     "hits": [ ]
	}
}

Screenshot

Query with ignore_throttled parameter

Now, we will include ignore_throttled=false in our query and see the difference for the same index.

GET /student/
_search?q=name:Alen Paul&ignore_throttled=false
{ }

Response

Here, you can see the difference that by including ignore_throttled=false, one document has returned that contains name: Alen Paul. Because it included the frozen index Student in its search.

{
	"took": 17,
"timed_out": true,
	"_shards": {
	     "total": 1,
	     "successful": 1,
	     "skipped": 0,
	     "failed": 0
	},
	"hits": {
	     "total": {
	         "value": 1,
	         "relation": "eq"
	       }
	     "max_score": 0.5753642,
	     "hits": [ ]
	{
"_index": "student"
"_type": "_doc",
"_id": "01",
"_score": 0.5753642
    "_source": {
         "name": "Alen Paul",
         "gender": "Male",
         "phone": "9876543210",
         "street": "Malen Park",
         "city": "California",
         "country": "United States",
         "zip": "94025",
}

Screenshot

Monitor Frozen Indices

Frozen indices are ordinary indices. These indices use a memory-efficient shard implementation and search throttling. Elasticsearch allows checking the status of the index using a command.

Syntax

Execute the following code to check the status of your index-

GET _cat/indices/index_name?v&h=health,status,index,pri,rep,docs.count,store.size

Here, _cat indices command is used to provide a cross-section of each index, where v and h are parameters and GET is the request method. In index_name, pass the name of the index you want to search.

Response

The response will be the same as the below output:

health	status	Index	pri	rep	docs.count	store.size
yellow	open	Frozenexample	4	1	2	16.3

Error on adding document in frozen index

In case you try to add some data in the frozen index, it will generate an error. In the below screenshot, you can see that we have added a document in the student index, but it returned an error. See the below example -

Example

POST http://localhost:9200/student/_doc/02/
{
  "name": "Kelvin K",
  "dob": "17 Dec 1989",
  "course": "CFP",
  "duration": "6 months to 2 years",
  "phone": "+91141893548",
  "address": "Malen Beach 9A, NewCity, USA",
  "pin code": "56018"
}

Screenshot

By executing the above query on frozen index, it will generate unavailable shards exception error. Look at the screenshot below: