Splunk Knowledge Management

What is Splunk knowledge?

Splunk software offers a powerful search and analysis engine to help we see the specifics as well as the broader trends in our IT knowledge. If we're using Splunk tools, we're doing more than looking at individual entries in our log files; we 're using the collective knowledge they carry to find out more about our IT setting.

Splunk software extracts from our IT data different kinds of knowledge (events, fields, timestamps, etc.) to help we harness that information in a better, smarter, more focused way. Some of this information is extracted at index time, because our IT data is indexed by Splunk software. But, at search time, the bulk of this information is created by both Splunk software and its users.

Unlike databases or schema-based analytical tools that decide what information to pull out or analyze in advance, Splunk software allows us to extract knowledge dynamically from raw data as needed.

As our organization uses Splunk software, additional Splunk software knowledge object categories are created including types of events, tags, searches, field extractions, workflow actions, and saved searches.

We can think of Splunk knowledge of software as a multitool that we use to discover and analyze various aspects of our IT results. Event types allow us to quickly and easily classify and similar group events together. We can then use them to perform analytical searches on precisely defined event subgroups.

The Knowledge Manager manually shows how to keep sets of knowledge objects for the organization through Splunk Web and configuration files. It also shows the way to use Splunk knowledge to solve the real-world problems for the organization.

Knowledge of Splunk software is grouped into five categories:

Data interpretation: Fields and field extractions

Fields and field extractions constitute the first order of knowledge of the Splunk software. The fields that Splunk software extracts from our IT data automatically help bring meaning to our raw data, clarifying what may seem incomprehensible at first glance. The fields, we manually remove extend and build upon this sense layer.

Data classification: Event types and transactions

We use event types and transactions to pool interesting sets of similar events together. Event types bring together sets of events found through searches, while transactions are collections of time-spanning, conceptually linked events.

Data enrichment: Lookups and workflow actions

Lookups and workflow actions are categories of objects of knowledge which extend our data 's usefulness in various ways. Field lookups allow us to add fields from external data sources, such as static tables or commands based on Python, to our data. Workflow actions allow interactions between data fields and other applications or web resources, such as a WHOIS search on a field containing an IP address.

Data normalization: Tags and aliases

Tags and aliases are used for administering and normalizing field information sets. Tags and aliases can be used to group sets of related field values together and to give tags of extracted fields that reflect different aspects of their identity. For instance, we can group events from a collection of hosts together in a similar location (such as a building or city)-just give each host the same tag. Or perhaps we have two different sources that use different field names to refer to the same data - we can normalize our data using aliases (for example, by aliasing clientip to ipaddress).

Data models

Data models are representations of one or more datasets and drive the Pivot tool, allowing Pivot users to quickly generate useful tables, complex visualizations and robust reports without interacting with the Splunk search language of the software. Data models are developed by the information managers who completely understand their indexed data format and semantics. A typical data model makes use of certain types of information objects discussed in this manual, including lookups, transactions, extractions of search-time fields and measured fields.

The Manual of Knowledge Managers contains information on the following topic:

Summary-based report and data model acceleration

Use the Splunk software to speed things up when searches and pivots are slow to complete. This chapter addresses acceleration of reports (for searches), acceleration of data models (for pivots) and indexing of summaries (for special case searches).

Knowledge managers should have a fundamental understanding of the concepts of data input setup, event processing and indexing.

Why manage Splunk knowledge?

When we have to maintain a relatively large number of objects of knowledge during our Splunk deployment, we know it is necessary to manage the information. It is especially true in companies with a large number in Splunk apps, and more so if we have several user teams operating with Splunk software. This is simply because a greater proliferation of users results in a greater proliferation of additional knowledge of Splunk.

If we leave a situation like this unchecked, our users may find themselves searching through vast sets of objects with confusing or contradictory names, trying to locate and use objects that have applied unevenly assignments and permissions, and wasting valuable time generating objects like reports and field extractions that already exist elsewhere in the system.

Splunk information managers have centralized control of information about Splunk applications. The benefits which managers of knowledge can offer include:

• Tracking the development and use of information objects through teams , departments and deployments; If we have a broad Splunk deployment spread through several user teams, we will inevitably find teams reinventing the wheel by designing artifacts which other teams have already created. Knowledge managers can mitigate these situations by monitoring the creation of objects and making sure that useful general purpose objects are shared across deployments globally.
• Data standardisation of events. To put it straight: objects of knowledge proliferate. While Splunk software is focused on data indexes and not databases, the same normalization principles still apply. It is easy for any robust, well-used Splunk implementation to end up with a dozen tags that have all been to the same field, but as these redundant objects of knowledge stack up, the end result is confusion and inefficiency on the part of its users. We will give we some tips on normalizing our libraries of information objects by applying standardized naming standards and using the Splunk Common Information Model.
• Manage the objects of information through configuration files. Some aspects of setting up object awareness are best performed via configuration files. This manual will teach information managers at Splunk Enterprise how to deal with objects of information in this way.
• Data model creation for Pivot users. Splunk software offers the Pivot tool for users who want to build tables , charts and dashboards easily without having to write search strings that can be long and complicated at times. The Pivot tool is driven by data models - Pivot has nothing to report on without a data model. Splunk information managers build data models: individuals who understand the structure and semantics of their indexed data, and who are familiar with the Splunk search language.
• Manage the configuration and use of summary search and pivot acceleration tools; Large volumes of data can result in slow performance for Splunk software, whether we start a search, run a report or try using Pivot. Report acceleration, data model acceleration, and summary indexing can be used to speed things up the knowledge manager to help ensure that the teams in our deployment get results quickly and efficiently. This manual teaches we how to centralize these acceleration techniques to ensure they are used safely and effectively.

Prerequisites for knowledge management

Most tasks in knowledge management are cantered around manipulation of search time events. In other words, a typical knowledge manager does not usually focus on work before events are indexed, such as setting up data inputs, adjusting event processing activities, correcting default field extraction problems, creating and maintaining indexes, setting up forwarding and receiving and so on. We do recommend, though, that all knowledge managers understand these concepts well. Strong grounding in these topics allows knowledge managers to better plan their approach to knowledge object management for their deployment ... and it helps them solve problems that will inevitably arise over time.

Here are some topics that should be familiar to knowledge managers, with links to get we started:

• Inherit Splunk Enterprise Deployment: If we inherited a Splunk Enterprise Deployment, we can find more information on the network characteristics of our deployment, data sources, user population and knowledge objects in the Inherited Deployment Manual.
• Working with Splunk Apps: If our deployment uses more than one Splunk App, we should have some background on how they are organized and how multi-app deployments work with app object management. See What's an application? App architecture and ownership of objects, and Manage app objects in the Admin manual.
• Managing the configuration file: Where are the configuration files? How does it organize them? How do the configuration files dominate over one another? See Admin manual for priority over configuration files and configuration file.
• Incoming Data Indexing: What is an index, and how does it work? What is the difference between time of index and time of search and why is that distinction significant? In the Managing Indexers and Clusters manual, start with About Indexes and Indexers and read the rest of the chapter. Special attention should be paid to Index time vs Search time.
• Getting event data into our Splunk deployment: At least a basic understanding of the Splunk data inputs is important. Find out What Splunk will index and, if necessary, read the other topics in the Getting Data Manual.
• Know our forwarding and receiving setup: If our Splunk implementation uses forwarders and receivers, it's a good idea to get a grip on how they've been implemented, because this can impact our information management strategy. Get an overview of the topic in The forwarding and receiving manual.
• Understand event processing: It's a good idea to get a good grounding in the steps Splunk software passes to parse data before indexing it. This knowledge can help we with our event data troubleshooting issues and recognize index time issues in event processing. Start with the Getting Data In Manual overview of event processing and read the entire chapter.
• Default field extraction: Most field extraction takes place at the time of search, except for some default fields that are extracted at index time. As a knowledge manager, most of the time we 're going to be concerned with search-time field extraction, but it's a good idea to know how to manage default field extraction when this is absolutely necessary. This will help we overcome problems with the fields of host, source, and sourcetype which Splunk software applies to each case. Start with Getting Data In manual About default fields.
• User and role management: Knowledge managers do not typically set up users and roles directly. However, understanding how they are set up within our deployment is a good idea, as this directly affects our efforts to share and promote objects of knowledge between groups of users.