15 Practices for Shielding Your APIs from Attack: #2 – Teams & Tests
This post shares experiences from our multi-year journey working with a Canadian client to help them improve their API security and security management. This post in the series discusses the approaches and activities that were important to that journey. This is a long post, so we start with an overview of the practices to let you jump around if you don’t wish to read the whole post.
Overview
These approaches and activities are extremely valuable to improving security during software development and largely revolve around DevSecOps Teams, Automated Tests and other testing validation:
- Sense of full ownership of security and operations across all environments as related to the product.
- Security-related tooling.
- Ability to own and prioritize security work.
- Ability to own and prioritize automated testing, particularly around security implementation and policies.
- Ability to own and prioritize and implement product-specific CI/CD pipelines to run tooling, testing, and deployment activities.
DevSecOps Product Teams
Dedicated DevSecOps development teams are a key approach to ensure a reduction in security exposures. Dedicated product teams that own a product’s code base, can iterate independently and are aware of security as an ongoing requirement can be very effective at reducing security risks. Enabling these teams with tooling and training is important, but also allowing their Product Owner to work with the team to prioritize security requirements and repairs, is also critical.
Enabled DevSecOps teams tend to work as a group to look at the big picture with a product. Owning deployment and configuration (CI/CD) pipelines ensures they have an eye for pushing products and configurations consistently and reliably.
Stable DevSecOps teams require business sponsorship to ensure consistent dedicated team
members over time. From a business perspective, stable and consistent teams owning a product pay off. Not only can these teams build a sense of pride and ownership around their work, but the ability to maintain and adapt these products to new business needs becomes rapid and costs less overall than bringing in people every time changes are required. Our client measured in one example year that they saved $100K in terms of administrative overhead alone, by simply being able to ask for particular changes to the product from an existing team and not having to go through the expenses of engaging new people every time. This saving does not even measure the business case offered by having this product team able to rapidly respond to these new business needs and implement a solution sustainably, ensuring the solution is secure, tested, and fits well within the product codebase.
Automated Security Testing
Teams concerned with product security and quality as a whole tend to prioritize automated testing. Our two teams working on this particular system have over ten thousand (and growing) fully automated programmatic functional tests for the platform. Programmatic just means that the tests (even for user interfaces) are orchestrated by functions in the code. This means that if part of the interface changes, the functions dealing with that part of the interface are quickly changed, and the orchestrated test continues to pass. It is worth knowing that the DevSecOps team as a group owns the development and maintenance of all these tests so that tests are applied at the time of the development or change in a feature. In fact, our team with this client spent several years doing this testing without any specialized or dedicated QA Developer on the team at all.
Several thousand of these automated are exclusively concerned with validating security policy and ‘edge case’ conditions that could result in a security failure or exploit. This is over and above countless unit tests in the code. These tests are programmatic, meaning the development team maintains the functions and steps that enable them every sprint as they are building functions and features. Tests have to pass completely to have a candidate build to deploy to any other system, and the tests are baked into the CI/CD pipeline.
Agile teams might need to change any part of their code base regularly, and Test Driven Development (TDD), and good functional test coverage help teams design code for change. Code that can be modified safely and verified reliably tends to have fewer defects over time. In addition, we use tools to perform regular (daily) scans of code to highlight possible security concerns with dependent libraries or implementation choices. I’ll talk more about these in the tooling post.
External Pen-Testing & Security Review
Product teams benefit from feedback. As such, we also ensure that external code evaluation and penetration testing by a third-party group is engaged from time to time. This type of testing gives the team insight into potential improvements or risk areas and may find existing gaps in the system. These activities can be performed without much engagement from the product team until it is time to review the results and decide how to prioritize identified improvements. This is a great ‘service’ for established partners or security departments to offer to product teams in their organization.
External tooling can help the team a lot. In a future post, we’ll discuss how some of these tools work within automation pipelines to measure and check for security issues. However, running external scanning tools periodically as a ‘state-of-the-nation’ assessment update is also valuable. An example of external tooling our DevSecOps team has used is an SSL checking tool, like the SSL Labs from Qualys.
Empowering our DevSecOps team to own the product and prioritize security and quality work has enabled us to act on the input from security reviews quickly, depending on risk and priority. Agility and automation are what enable our teams to roll out fixes to any identified gaps at low cost and close any high-risk findings in production rapidly and with confidence.
Security Consulting as a Service
It is unlikely that every team has the expertise for all types of security questions. While a cross-functional DevOps team should include the required day-to-day security skills whenever possible, there will always be situations where additional expertise and advice is needed, and sometimes that means more than finding online articles.
Our team benefits greatly from approachable security professionals in other parts of our customer’s organization. Sometimes this was because we needed to know what systems or software may already exist to meet a need, or if we were building some secure component, that the approach we were taking made sense and was addressing the correct things. Sometimes we need to know if specific security policies exist around a particular type of software implementation, and being able to reach out to other experts can help.
A prior example in our case was when we needed to implement SSL certificate signing and verification services. Some of this function was supported in the products we were already using, but parts of it were not. When considering something like a certificate signing service, we wanted additional support to ensure we were complying with our customer’s organizational standards and concerns, and that we weren’t building something that we could leverage from elsewhere in the organization.
In our case, using security expertise from a trusted professional within the customer’s organization enabled us to meet these goals. We were lucky to find that our specific customer experts are approachable and supportive in providing direction and advice. So often in large organizations employees tend to avoid security groups for fear of them shutting down work until they can ‘fully review’ the design, audit the implemented code, or other ‘work stopping’ blockers. In these cases, security is sometimes seen as an obstacle to avoid, rather than a consultation that will help.
It takes awareness and trust to work with Agile teams who are dealing with constant change and support them as they try to improve. A security department with a service mindset will be more readily engaged by those teams, will have much greater influence over those teams, and therefore greater success at improving an organization’s security.
What’s Next?
The next post in this series goes into the ideas, architecture, and design elements that help us to reduce the risk of APIs becoming an attack vector for our customers.
Solsys would be pleased to share more details of this experience with you. We have various workshops, such as our API attack vector workshop, and other services available to support you in your API & Software security journey. Please reach out to us by using the contact form on this site.
