Splunk Alerts

In this Splunk tutorial we are going to learn about the Alerts in the Splunk. How to create an Alert, Types of Alert, the workflow of Alert, Comparison between different types of Alert, Real time Alert, Scheduled Alert, Rolling time Window trigging.

Alerts

Alerts occur when particular criteria are met for the search results. When alerts activate, we can use the warning actions to respond. It is used to monitor specific events and respond to them. It includes facts, instructions, and warning action scenarios for use.

The alerting workflow

Alerts combine a saved search, type and trigger configurations, and action alerts. Here are some details of how the various portions of an alert work together.

Search: What would we like to have a track of?

Start by searching for the events that we wish to track. Save as Alert the search.

Alert type: How often do we want to check for events?

The alert uses the search we save to check for events. Set the type of alert to configure how often the search will run. Use a scheduled alert to check for the events regularly. We can also use an actual-time alert for continuous monitoring of events.

Alert trigger conditions and throttling in the Splunk platform: How often do we want to trigger an alert?

An alert doesn't have to trigger each time the search results are generated-set conditions to manage trigger when the alert triggers. We can also throttle an alert to control how quickly following an initial alert can trigger the next alert.

Alert Action: What happens in the Splunk environment when the alert triggers?

When an alert triggers, one or more alert actions may be initialized. An alert action can notify us of a triggered alert and can help us get started to respond. We may customize the frequency and type of warning action.

Alert types

Two types of alerts exist, scheduled and in real-time. Definitions of type of alert are based on the timing of the search alert. We can customize timing, activation, and other actions for either type of alarm according to the scenario.

Alert type comparison

Here is a comparison of the alerts in schedule and in real-time.

Alert type and triggering scenarios

Once we select a scheduled or real-time alert, we can configure how the alert will trigger outcomes. Depending on the events we are monitoring, we may need a real-time alert that triggers with each result or a scheduled alert that triggers only if results meet certain conditions. The following scenarios show various use cases for alert and trigger types.

Scheduled alert

Use a scheduled alert to search for events regularly, and monitor if they meet specific conditions. If immediate or real-time monitoring is not a priority, a scheduled alert is useful.

Scenarios

• An online retailer is targeting 500 sales daily. A retailer admin creates a scheduled alert for monitoring the performance of the sales. The administrator schedules the warning to try sales events at 23:00 each day. She configures the warning to activate if the result numbers are below 500.
• An administrator wants to monitor how often the users follow a bad link to the 404 error page. The admin generates a scheduled warning which searches every hour for 404 errors and triggers if there are over 100 results.
• An admin generates a scheduled warning to check if no data has been submitted to the Splunk platform by a given host in the last few hours. He schedules the alert to look every three hours for events from the host. The admin configures the warning to activate if search results do not exist.

Real-time alert

Real-time alerts constantly scan for incidents. In circumstances where immediate monitoring and responses are relevant, they can be useful. We can use real-time warnings that occur once per outcome or only if those conditions are met within a limited time span of rolling.

Per-result triggering

A real-time alert with a triggering condition is sometimes referred to as an "alert per outcome" Use this type of alert and trigger to search for events continuously and get notifications when events occur.

Caution: Use with caution per-outcome triggering in a high-availability deployment. If a peer is not available, a search in real-time does not warn that the search may be incomplete. For this deployment, it is recommended that a scheduled alert be used.

Scenarios

Here are a few examples of using an actual-time alarm with triggering per-result.

• A website administrator on social networking needs to learn if authentication errors occur. She sets up a real-time alert to look for failed attempts to log in. She chooses a trigger condition per-result so she can track any attempt at failed login.
• An admin requires real-time control of a series of hosts for errors. Some errors need a more immediate response than others. A real-time warning is set up by the admin with a trigger condition per-outcome. He is the one who controls the flow of the alert using a field representing the less urgent error code and a suppression period of one hour. The alarm causes any urgent error but for less critical errors at most once in an hour.

Rolling time window triggering

A real-time alert that triggers a rolling time window is sometimes called a "rolling window alert." This form of alarm and activation is useful when a specific time period is an important part of the sequence of events we are tracking in real-time.

Scenarios

Here are some examples of using an actual-time warning to activate the rolling time window.

• An admin wants a notification every time a user has three failed logins within 10 minutes. The administrator sets a real-time alert to search for failed logins and configures a ten-minute time window to roll. The admin throttles the alarm, so it causes failed logins from the same user only once in an hour.
• An administrator wants to know whenever a web application has more than five connection errors in a minute. The administrator configures a real-time alert to search for an error event and specifies a one-minute rolling window. If the search returns one result and four more results five minutes later, the alert does not trigger.

Create real-time alerts in Splunk Web

We use a real-time alert to monitor events or patterns of events as they occur. We can create real-time alerts with triggering of per-result or rolling time window. Real-time alerts can be expensive in terms of computing resources, so consider using a scheduled alert when possible.

Create a real-time alert with per-result triggering

Real-time alerts with triggering per-output are sometimes referred to as per-outcome alerts. This type of alert and trigger uses a continuous real-time search to search for events. Any outcome of the search activates the warning.

Note: If we have a high-availability deployment at Splunk Enterprise, use cautiously per-result triggering. If a peer is not available, a search in real-time does not warn that the search may be incomplete. Use a scheduled alert to avoid the issue.

Follow these steps to generate an actual time warning with triggering per-result.

1. Go to the Search page in the Search & Reporting app in the Splunk Web Platform.
2. Create a search.
3. Goto Select Save As than on Alert.
4. Enter a title and optional description for your Alert.
5. Also, Give the necessary permissions to the Alerts.
6. Select the Real-time alert type in the window.
7. (Optional) Change the Expires setting if you want your Alert to get expire after a period of time.
8. Select the Per-Resulttrigger option.
9. Select at least one alert action that occurs when the alert triggers.
10. Click Save.