Solsys Labs – Practical Learning for Applicable Success

Published: July 5, 2023

“Let me show you what that looks like,” is a powerful phrase when it comes to showcasing technology.  Long-winded explanations and fancy diagrams can only go so far but nothing can compare to a live demonstration with a realistic environment.  This is true not only for potential customers, but also for employees who are trying to learn new skills or hone their skills in existing areas.  This is the thinking that inspired us to create Solsys Labs.

Welcome to Solsys, Have a Splunk Server

Our lab began as a testing platform for new hires.  We wanted to be able to watch candidates try to solve realistic problems in an environment that closely resembled one in which they’d be expected to perform.  To that end, we did some research and found that setting up droplets in Digital Ocean was a simple and cost-effective way to set up a few Splunk servers and Heavy Forwarders that could easily be spun up and destroyed when it came time to perform the exercise.  We developed an effective test to provide a reasonable challenge and watched as candidates worked through it.  Often, we discovered that those candidates who didn’t necessarily complete the tasks correctly or at all still demonstrated the sort of intuitive critical-thinking skills that we look for at Solsys.  That sort of knowledge couldn’t have been gleaned from simply administering a written test and evaluating the results.

Building upon the success of the lab, we wanted to be able to provide targeted training for our new hires to ensure that when their contracts began, they would hit the ground running with hands-on experience performing relevant tasks.  Once again, we set up an environment in Digital Ocean that allowed us to replicate the infrastructure our new hires could expect to encounter on the job.  Further to this, we built a guided curriculum that the trainees could follow on their own and then review with one of our more experienced staff members.  We found that the trainers sometimes learned as much from the exercises as the trainees.  Our customers were benefiting from having competent contractors just as Solsys benefited from having more skillful employees.  Of course, we supplemented the internal training with official Splunk training courses for all our employees.

Practical Demos Aren’t Just for New Customers

When trying to demonstrate a new idea or feature of Splunk that a customer may not have access to on their own instance, we discovered that having a sandbox Splunk instance available allowed us to customize demos in a much more detailed way than we would have had we tried to use Splunk Show.  This could include deployment concepts, custom visualizations, or new apps.  While Splunk Show can accommodate highlighting features, there are times when setting up a demo could take several days of configuration.  Having access to that pre-configured environment is worth the small cost of keeping an additional droplet on hand.

Another feature that we were able to highlight was the ingestion of Azure event logs to simulate the collection of Splunk data from OT devices such as monitors or gauges.  We replicated the data creation using a simple Raspberry PI device whose logs were stored in an Event Log then designed an Azure Logic App to perform batch ingestion of that data into Splunk using the HEC.  We even made the Logic App freely available on Splunkbase and have provided assistance to several individuals to help them configure it correctly.

Keeping Ahead of the Curve

Solsys has a strong history pertaining to API microservice security and architecture.  In order to maintain our knowledge on the subject, we decided to implement a microservice environment in our lab, focusing on Kong as a gateway.  In addition to learning Kong, it gave our lab team the opportunity to become proficient in Terraform for deploying to both a GCP environment and Kubernetes.  Naturally, this also involved gathering splunk data from the microservices, Kong, Google Cloud, and Kubernetes.  Having the ability to see these real-world logs and how different attack vectors were presented in both Splunk Enterprise and Splunk Observability gave us insights that would be difficult to replicate simply by taking training courses.

Once our infrastructure was in place, we automated attacks against various aspects and used the data we gathered to create realistic Eventgen scripts to automate the synthesis of events for basic demos.  We also had the option to execute attacks in real-time to highlight the security features of Splunk.

Splunk Observability was particularly interesting when it came to instrumenting our infrastructure.  Its out-of-the-box dashboards for Kong, Google, Kubernetes, and many others simply worked once data began flowing through OpenTelemetry collectors.  This allowed us to focus more on exploring some of the practical benefits of the platform rather than working to configure data sources.

Next-Level Security

The most recent addition to our lab features a set of functional authentication services.  These include basic authentication, oAuth2 authentication through Google’s SSO, and a synthesized oAuth2 model that allows for more targeted testing.  We are able, with a click of a mouse, to replicate dozens of API attack vectors and demonstrate how they can be mitigated using Splunk.  This allows our lab team to design powerful use cases and dashboards specific to our current and potential customers’ needs.

Conclusion

Splunk offers several options when it comes to training, demos, and hands-on opportunities…but having an environment that can be stressed, broken, rebuilt, and shown off provides a unique opportunity for the growth of your team.  Let us show you what we’ve built or advise you on the creation of your own sandbox environment to bring your Splunk game to the next level.