Reporting Web Services Data Using KV Store Operations

Published: June 1, 2023

Scenario

In one of our customer implementations, one requirement was to pull data from a web service as side data for the entire application. The data from the web service is similar to the structure of a relational database and follows the same rules for inserting/updating records. Our solution had to deal with a primary key identifier as a reference for inserting and updating data.

Solution

We implemented a key-value method using Splunk’s KV Stores feature. This scalable way to store key-value pairs is functionally more efficient than a CSV look-up. Previous Splunk releases relied on CSV file lookups, which had limited capabilities for enterprise-scale operations. Since Splunk introduced KV Stores in version 6, this feature is a mature method to store side data, revolutionizing the ability to enrich raw data.

In the section below, you will see how we can create a KV Store and use some commands to create and manipulate them.

Design Considerations

• Scalability – KV Stores are ideal for increasing volume.
• Performance – They are superior to any side look-up option (e.g., CSV lookup).
• Duplicate Record – A composite primary key maintains the unique key constraints for the table data.
• Reliability – KV Stores can be backed up periodically, eliminating data loss and making it easier to rebuild them.
• Authorization – User access can be restricted, which ensures only authorized users connect with the data.

Creating a New KV Store

Step 1) Edit the collections.conf file
We need to edit the collections.conf file and add the name of the lookup as a stanza. It is recommended to create the lookup inside an app directory rather than use the default search app.
Path to collection.conf – $splunk_home/etc/apps/[appname]/collections.conf

Step 2) Create a lookup definition
To create a lookup definition, go to Settings 🡪 Lookups 🡪 Lookup Definitions 🡪 Add New.

Step 3) Modify the definition to add field names
In the collection name, supply the same name you created in collections.conf (in this case, solsys_kv_store). Provide the field names you want to have in the KV Store.

Step 4) Add some data into the lookup and verify the data
In this case, I added some data from one of the indexes. There are various options to get data into a KV Store:

• Use a scheduled script to onboard data into Splunk from other systems and applications employed for this scenario.
• Add data from the SPL command using the “outputlookup” command.
• Obtain it from other data sources like input files and CSVs.

Step 5) Include a unique key identifier
KV Store records are identified by a unique key. This key helps us to perform unique operations on the KV Store. We have to enable the key in the KV Store definition to fetch it.

Click on the Save button. Now we have the ability to see the Key value.

Corollary

Viewing the unique primary key

The primary key is an alphanumeric key auto-generated by Splunk that identifies a record and can serve as the unique key constraints (similar to traditional relational databases).

Since fields with underscores aren’t displayed using the inputlookup command, the approach is to eval the key to another field name that doesn’t include a leading underscore.

Example:

Using look-up to fetch values

KV Stores can be used to enrich and enhance raw data by adding side data to the original searches. They are also useful for fetching matching values. The “lookup” command can be used for this purpose.
The possible commands are outlined below.

• If the field name is the same as in the KV Store, you can use the query below.

• If the field name is different as in the KV Store, you can use this query instead.

Updating/appending the lookup using SPL

It is possible to write data to a KV Store using SPL in case the user must store processed data to retrieve later. This flexibility of inserting data makes KV Stores extremely useful. SPL has built-in commands to achieve this.

Use the “outputlookup” command to insert values into a lookup. If you use append=true, it will update the values. Otherwise, just overwrite the entire lookup table.

Add custom primary key

Splunk will generate an alphanumeric primary key (_key) when a record is inserted into the lookup. We can specify a custom key to override the _key. This can be helpful in identifying redundant records. Use the eval command to assign a field to the _key. The new value will be assigned as the primary key.

For general KV Store reference documentation from Splunk, see here.