Leveraging GEN AI for Splunk Development
AI has been in the forefront of technology news headlines for the past several months. While some businesses have embraced tools like ChatGPT and Copilot, others have continued doing things the traditional way. It has become increasingly apparent to me that ignoring those tools will leave you at a disadvantage when their use inevitably becomes ubiquitous in all industries. In this post, I will be looking specifically at how to leverage GPT to help you with a variety of tasks specific to Splunk.
ChatGPT Basics
Before diving into the practical application of GPT, you need to understand how to interact with GPT effectively. GPT is not a search engine. You can type in a few keywords and it might figure out what you are looking for, but to make full use of its potential, you should instead treat your interaction with the tool as a conversation. Begin by giving it a very specific description of what you’re expecting from it.
Bad example:
Blueberry pie recipe
Good example:
I would like to make a blueberry pie. I have 2 pints of blueberries available and a 9-inch pie pan. I need to have it ready in under 4 hours. It should feed 6 people. Avoid recipes that contain almonds.
Both of these examples will yield results, but the second will more quickly get you exactly the result you need rather than having to further refine your search. Bear in mind that your next interaction will be part of the same conversation. If you want to refine your pie recipe further, you will not need to re-enter everything again but rather tell GPT to make some adjustments to what it’s already presented. In this case, you may ask it to limit the number of calories or make the recipe vegan-friendly. The conversation will continue in the same thread until you implicitly or explicitly change the topic. If you start asking questions about Splunk, it should create a new conversation on the new topic.
Also important to note is that depending on the version that you are accessing, the information available to the AI is limited to a particular time-frame. Version 3.5, for example, which is the current publicly accessible version, only has information as recent as September 2021. Therefore, searching for current events or information about current technologies may be slightly out-of-date.
Splunk Installation
Now that we have the fundamentals of GPT down, let’s turn our attention to Splunk. Just as there are many aspects to working with Splunk, there are even more ways for GPT to help you with them. Many of us have installed Splunk dozens of times and some of us may even have pre-made scripts to help us with the process, but for those of us who are new to the procedure, here’s how you can get GPT to help you with the task.
I would like to install a Splunk Heavy Forwarder on a linux machine. I have downloaded the install file and it is located at /home/splunk/splunkforwarder.tgz. Splunk should be installed as an existing user named “splunk”. It should also be configured to automatically start when the machine reboots. Show me the commands to perform this task.
Within a few seconds, GPT will not only show you all the commands you will need to execute, but also explain what each one does. If you need clarification, you could simply ask it to
Tell me more about step 3.
If something goes wrong along the way, simply paste the error message right into the chat bar and GPT will diagnose the problem and provide a solution. More often than not, it will accomplish the task quickly and correctly. This is not to say that whoever you assign to the task of performing the installation shouldn’t have some understanding of what they’re doing. Problems inevitably arise and knowing how to deal with them is vital, however, they are no longer required to remember the exact syntax for every command in the process.
Splunk Configuration
While the previous example is impressive by itself, there is so much more we can do with GPT. It’s here that we need to begin thinking about what data we’re sharing with the AI. It’s best to avoid entering sensitive data such as real names or IP addresses into the chat bar. So, when it comes time to configure Splunk, obfuscating that data is a good idea. For example:
Next, I’d like to set up this forwarder to send its logs to 99.99.99.99
GPT will provide you with an exact outline of the steps you’ll need to take to set up the forwarder as well as describing exactly what information you’ll need in order to do it. Obviously, you’ll need to change the IP address to the appropriate one. If you’ve never used GPT, you should be slightly impressed at this point. The next logical step is to start adding some data.
I have some log files located in /path/to/logs.log I would like to have splunk monitor those logs and add them to index "blueberry" with a sourcetype of "pie_logs".
GPT will probably give you instructions for adding the new logs through the UI, but what if we want to use configuration files instead?
Could we do that using a new TA named "splunk_pies"? What files would I need to create and what would their contents be?
GPT will walk you through the process of creating your new TA and give you example inputs.conf and local.meta files. But what about the sourcetype? We’ll need sample data for that.
my log data looks like this:
Jun 14 2023 15:16:02.330 [ field1="abc" field2="def" ] could you generate a props.conf to ingest that?
GPT will not only define the sourcetype, it will remember what I called the sourcetype earlier and use that name appropriately. Again, the new configuration needs to be reviewed by someone who knows what they’re doing, but it’s orders of magnitude faster than having to do it from scratch.
I Have Some Data…Now What?
Now that GPT has helped you to configure Splunk, what’s left for it to do? How about suggesting some ideas for dashboards? How about creating those dashboards for you?
I have a Splunk index named sql_logs that contains sourcetypes named "oracle_errors" and "mysql_errors". I would like to make better use of the information in that index. What are some ideas for dashboard panels that could use the information in those logs?
Without even providing specific details about the contents of the logs, GPT will infer what data might exist in a sourcetype named “oracle_errors” and provide some ideas for making use of that data.
Show me what the XML for a dashboard containing these panels would look like.
GPT will then proceed to take seconds to perform what it would likely take several hours for a Splunk developer to do. The result may not be exactly perfect, but it will provide you with an excellent starting point.
Conclusion
I just spent a handful of minutes setting up Splunk, configuring it, coming up with dashboard ideas, and putting those ideas into practice. Only a few months ago, this would have taken several days of work by individuals with a variety of Splunk specialties. While interacting with the AI takes some getting used to, in the long-run it will save your organization valuable time and resources.
