Splunk as a Product Enabler for Observability

Published: June 2, 2023

In part one of this post, we talked about Product Thinking, what it is, and why it’s valuable to organizations. We also outlined a series of questions we normally consider when taking a product thinking approach. In this post, we’ll talk about Splunk as a product enabler, and what kinds of products it might be used to offer in the operations and observability space.

What kinds of Products might Splunk Enable?

Taking a product thinking approach means we don’t consider the solution or implementation of a product before addressing the product thinking questions from the first post. The reality is, however, that large organizations tend to face many similar problems that Splunk could help resolve, and may already have Splunk installations that could enable product and service offerings. Making some assumptions here, we can talk about what kinds of things Splunk can enable for an organization in terms of product and service offerings.

If you’re reading this, you probably already know that Splunk is a ‘big data’ time-series analytics and monitoring platform. It’s ideal for grabbing lots of high-volume machine outputs (log files, or other machine-generated events) and enabling searching and reporting over them, quickly and thoroughly. As such, it has a wide range of applications.

At Solsys we have been involved in and seen initiatives delivered that offer the following kinds of logging and monitoring services as a product, or as part of a product implementation:

Observability and Operational Support

The classic business case for Splunk is to enable grabbing lots of logs from various different components, as well as using off-the-shelf connectivity to pull in metrics about infrastructure and services (whether on-premise or cloud), and then providing operational teams (ideally DevOps teams!) with the ability to use their log data to diagnose operational issues, manage and determine capacity requirements. This good view into a product system enables the development and operations team to reduce the Mean Time To Reduction for technical issues or outages, do resource capacity planning as we grow our customer base, and even to get ahead of outages issues without customers having to tell us something is wrong. By running Splunk as a service, we can offer these capabilities to the many other product teams within our organization, and extend dash-boarding and even alerting services to other product teams. This can represent a strong internal platform product capability to extend world-class observability and operation tools across a company.

Agile DevOps Teams

DevOps teams are constantly running build and deployment pipelines that can involve complex steps and the execution of multiple unit and acceptance tests – even load automation tests! In these environments, when many DevOps teams are leveraging pipelines among complex products, it can be extremely beneficial to provide observability and insight into the DevOps deployment systems for the team. Hosting a centralized Splunk product, it is possible to give development teams visibility to issues with build and deployment components, enabling quicker diagnosis of problems or identification of subtle failures over time (‘when did this start happening?!’). It is also possible to provide reporting and statistics around testing successes and failures or metrics about load testing responsiveness and changes in performance. It can be extremely valuable to development teams to understand how frequent and complex their changes are per build or deployment, to know how long pipelines and tests are taking to execute over time, to understand their QA posture in terms of test results and volume over time, and even do things like automating release notes via live dashboards. This service could be called ‘DevOps observability’. When combined with the Observability and Operational support offering, above, this gives a full view into software from creation to use, through time.

Product Business Reporting & Analytics

With many applications, the log files contain user interactions and business operations such as sales transactions, payment activity (hopefully not payment information of course!), account openings and closings, activation of services and products, and so on. These logs can provide a quick and easy mechanism for a product team that knows how to read their log files (and can perhaps also customize them) to measure and report business stats to their product stakeholders. Log files contain timestamps too, so with some accuracy, you can often report on the performance aspects of a product or service as it relates to its users. This means you can use this information to measure against an SLA, for example. It allows product teams to see how they’re servicing their customers, and perhaps what events – such as slow page loads – cause customers to disengage. With Splunk, we’ve often observed that log file ingestion enables operational tasks, and over time this grows to more meaningful ‘performance’ or business reporting. Again, another service extension to log file ingestion and another capability that can be offered when running a Splunk logging and monitoring platform as a service within your organization. Real-time business reporting and dashboards are of huge value to product owners and other business stakeholders such as sales and marketing.

In the next part of this post, we’ll talk about what products and services might be enabled by Splunk in the security space.

About the Authors

John Tobin has been working in software development for around 30 years, and has been working with Agile methods in the role of scrum master and product owner for product development teams for about half of that time. John has worked extensively with teams to deliver products and services in challenging and complex environments, delivering hundreds of thousands of dollars of business value over this time.

Doug Brown joined Solsys Corporation in 2013 as a Managing Consultant. Doug has tenures with large software organizations such as Oracle and Sun Microsystems leading professional services teams across North America. Doug has experience working with customers canvassing requirements, identifying critical solution architecture components. Success looks like great product teams delivering great value for their customers.