Elasticsearch Managing Index Lifecycle
In this chapter, we will talk about managing the index lifecycle. As we all know Elasticsearch performs various tasks as well as it also manages the index life cycle. It involves performing management actions based on certain factors such as shard size and performance requirements.
In Elasticsearch 6.7.0, a new feature Index Lifecycle Management (ILM) is introduced. The ILM (Index Lifecycle management) APIs enable us to automate the indices automation, which means how we want to manage our indices over time. Following are the crucial points of ILM (Index Lifecycle Management) -
- Elasticsearch REST API helps to set the ILM policies. However, we can directly set it in Kibana user-interface.
- ILM (Index Lifecycle Management) can be used to establish a hot-warm-cold architecture. In this phase and actions are optional and they can be configured if needed.
- An index has a lifecycle. An index goes through different stages, first from the hot stage, then warm, cold, and lastly from delete stage.
Hot Indices
Hot indices actively receive data in index and serve queries. In this stage, indices are sensitive and are used for data ingestion. It includes the following actions for this phase, which are -
- For the recovery, it set up high priority.
- If the current index is too large, old containing too many documents, then specify the rollover policy to create a new index.
Warm Indices
Warm indices do not contain data indexed in them. However, these indices can still perform queries. In this warm stage, indices are usually not used for ingestion. It includes the following actions for this phase, which are -
- It sets a medium priority for recovery.
- After that, it optimizes the indices by shrinking, setting, or force-merging them to read only.
- At last, it allocates the indices to hardware that is less performant.
Cold Indices
Cold indices queried very occasionally. It rarely queried at all, but it includes following actions for this phase are -
- It sets the low priority for recovery.
- After that, it freezes the indices.
- Similar to the warm indices, it also allocates the indices to the less performant hardware.
Delete Indices
This stage is all about to be deleted. According to this, delete all those indices which are older than the arbitrary retention period.
There are several ILM APIs provided by Elasticsearch. We will discuss those different API with their purpose (usage) and example. Below is a table containing API and their purpose to use with an example -
Policy Management APIs
There are three policy management APIs, which are given below -
| API Name |
Purpose |
Example |
| Create lifecycle policy |
This API policy is used to create a lifecycle policy. If the policy that you are going to create already exists, then in that case the existing policy is replaced and the policy version is extended. |
PUT_ilm/policy/policy_id |
| Get lifecycle policy |
This API is used to fetch the specified policy definition. The information returned by the policy includes the policy version and the last modified date. In case of any of the policy is not specified, it returns all the defined policies. |
GET_ilm/policy/policy_id |
| Delete lifecycle API |
The main objective of this API is to delete the specified lifecycle policy definition. But remember one thing that the policy, which is currently in use not is allowed to be deleted. |
DELETE_ilm/policy/policy_id |
Index Management APIs
There are three index management APIs, which are given below -
| API Name |
Purpose |
Example |
| Move to lifecycle step API |
As the name specifies, this API helps to move an index into a specified step. After moving the index, it executes that step. |
POST_ilm/move/index |
| Retry policy |
This policy is responsible for setting the policy back to the step where an error occurred. From there it executes the steps again. |
POST index/_ilm/retry |
| Remove policy from index API edit |
Basically, this policy removes the lifecycle assigned to an index. Apart from this, it also stops managing the specified index. |
POST index/_ilm/remove |
Operation Management APIs
There are four types of operation management APIs, which are given below -
| API Name |
Purpose |
Example |
| Get index lifecycle management status API |
This operation API is used to fetch the status of the ILM plugin. It shows one of the three states either STARTED, STOPPING, or STOPPED in operation_mode field returned by this API in response. |
POST /_ilm/status |
| Start index lifecycle management API |
This operation type of API is responsible for starting the ILM plugin, which is currently stopped. Whenever a cluster is formed, the ILM plugin starts automatically. |
POST /_ilm/start |
| Stop index lifecycle management (ILM) API |
As the name describes itself, this API halts all the lifecycle management operations as well as blocks the ILM plugin.
This API is very useful while performing maintenance on a cluster and requires to prevent ILM from performing any action on your indices. |
POST /_ilm/stop |
| Explain lifecycle API |
This API helps to retrieve the information about the current lifecycle state of index. It can be currently executing phase, step, and action. This display whenever the index entered each one, information about failure, and the definition of running phase. |
GET index/_ilm/explain |
|
For Videos Join Our Youtube Channel: Join Now
