Bringing software engineering principles, systems, and processes to cybersecurity
Offering specific examples of how the concepts, systems, and processes from software engineering can change the way we view, approach, and do cybersecurity
Welcome to Venture in Security! Before we begin, do me a favor and make sure you hit the “Subscribe” button. Subscriptions let me know that you care and keep me motivated to write more. Thanks folks!
I have previously discussed why security of the future is going to resemble software engineering and why the rise of security engineering is reshaping cybersecurity as we know it. Instead of repeating myself, I highly recommend reading the previous article before continuing with this one: The rise of security engineering and how it is changing the cybersecurity of tomorrow.
In this deep dive, I am offering specific examples of how the concepts, systems, and processes from software engineering can change the way we view, approach, and do cybersecurity.
This post is brought to you by… Audience 1st.
Join The CISO Track at Black Hat - The Peer-to-Peer Information Exchange Unconference
Audience 1st and The CISO Track have joined forces to create a unique, exclusive space for CSOs and CISOs to exchange insights and best practices with peers. Forget about sales pitches and long lectures; this is a forum for authentic, curated discussions. Each session starts with a brief 10-15 minute presentation by vetted industry practitioners, followed by in-depth discussions. Rest assured, our practitioners and moderators focus solely on stirring deep dialogue about the issue at hand, keeping product pitches at bay. You, the real experts, dissect the problem to gain real-world solution insights.
Start your journey towards enriched knowledge at The CISO Track, where we're all about learning, sharing, and growing together. The CISO Track runs August 8th. More information and tickets at audience1st.fm.
Reinventing the wheel in cybersecurity
Anyone fortunate to get exposure and experience working across several different markets knows that no industry exists in isolation. The way people bank changes the way they would like to access insurance; how retail customers get their orders delivered changes expectations around lead times for businesses, and the way e-commerce platforms are built changes the expectations around peer-to-peer commerce. True innovation is more often found at the intersection of disciplines rather than in silos of a particular industry.
Cybersecurity, as much as it may be tempting to think otherwise, isn’t a market capable of shielding itself from the rest of the technology industry. The way security is done, and the way security capabilities are delivered, should be closely tied to the way software is built.
The truth is that a lot of the challenges cybersecurity is dealing with have previously been solved in the field of software engineering; if not directly, then close enough to be referenced as examples. For instance,
Software teams have limited resources and a large number of competing priorities. It is well worth examining how the software engineering function solves this problem, how they plan work, how they iterate on requirements, and so on.
The number of bugs in the software is well above what the engineering team can fix. It would be valuable to see how this challenge is being addressed, how the right work is being prioritized, who does it, etc.
To build a product, technical teams have to ensure that it solves the right problem for the customer, while also aligning with the expectations from a wide variety of stakeholders such as legal, compliance, marketing, operations, security, and the like.
Every time a small change is deployed to production, there is the understanding that it can break or greatly impact the existing functionality. To avoid unexpected surprises, software engineering teams need to find a way to continuously test changes when they happen, and do it without unnecessarily slowing down the ability to ship code.
When I talk about adopting an engineering approach and making security look more like software engineering, I often discuss the necessary mindset and industry-wide shifts without which the evolution simply won’t happen. But, in addition to these fundamental shifts, there are tactical changes security teams can start implementing to improve their ability to respond to the challenges of the present.
While there are no easy answers, and nothing will magically solve all security problems, software engineering does offer a body of knowledge the cybersecurity profession can draw from, without having to reinvent the wheel once again.
Bringing software engineering principles, systems, and processes to cybersecurity
Product managers and their role in software engineering
As a product manager, the topic of the product’s role in the software development life cycle is near and dear to my heart. Over the past decade leading teams that built products across different industries, company types, and team sizes, I have seen a lot. I have witnessed that every company defines the role of the product manager slightly differently. My goal here isn’t to debate the merits of different approaches to the PM function but to zoom in on what I think matters, namely the critical problems the role is designed to solve.
Customer discovery
The number one way in which product managers add value to software delivery is by learning and becoming experts on customer needs. There are different types of personas that come in contact with the product (users, buyers, supervisors of the users, etc.), each with its unique needs. More often than not, to uncover the needs of the customers and to understand the problems they are trying to solve, it is not enough to look at the list of feature requests. One needs to foster closer relationships with customers, see them at work, and understand the context in which they are using the product, to be able to make good product decisions.
Prioritization
Customer discovery typically results in a long list of to-dos for the company. Not surprisingly, there is always more work than the team can reasonably do. The product manager crafts detailed product requirements and ensures that at any time, the team works on the highest-value items.
Team focus
When it comes to software development, distractions are the norm. Every day users report bugs, stakeholders come up with ideas, and tens of “urgent” requests come up. Additionally, developers get excited about the latest cool technologies, and if given full freedom, many would prefer to work on the latest and greatest tech instead of doing what the company needs today.
One of the responsibilities of the product manager is to protect the team’s focus so that engineers can ship useful software. While sometimes there is a legitimate need to switch focus on tackling a highly impactful issue, in most cases, PMs are funneling potential distractions into the backlog and following a rigorous prioritization process.
Cross-functional communication
Good product teams don’t just ship code; they build products that take into account the needs and requirements of the multitude of stakeholders - security, compliance, legal, marketing, data analytics, customer support, and more. One of the responsibilities of the product manager is to overcommunicate with everyone involved, to ensure that their input is taken into account and that the expectations around product, releases, and other parts are clear.
Most software engineers choose their profession so that they can focus on writing code and not spend days in meetings, and rightfully so: that is where their talent is better used.
How a PM-like role would benefit cybersecurity
At the fundamental level, cybersecurity teams are dealing with similar challenges:
there is a need to fully understand the business constraints and requirements before looking for solutions
there is always more work than can reasonably be completed within a reasonable time frame
distractions are never-ending, and someone must shield the team from constantly changing requirements
while most people don't go into cyber to spend time on calls with different stakeholders, someone must do this work if we want security to be a part of the organization
The concept and the role of a product manager can be highly relevant to security teams looking for ways to better organize their workflows, systems, and processes. Whoever takes this role on a security team, will need to understand the technical side of the craft, but also the business needs, how the company operates, and who is involved in different decisions. And, it goes without saying - they will need to invest in building relationships across the organization.
Structuring work: the organized sprint process
There was a time when software was shipped using the so-called “waterfall” model: a project manager would spend a month planning a perfect design, and then for six months or longer the team would build a solution. This approach didn’t work too well and resulted in continuous waste of resources, complex releases, frequent rework, and so on. This isn’t surprising as writing software cannot be planned and predicted to the same degree as, say, building a house.
Fast forward to today, and almost everyone in the field has embraced agile with its frequent iterations, regularly scheduled releases, continuous delivery, and minimum viable product (MVP) approaches. Even large banks and insurance companies that couldn’t have imagined shipping software more frequently than once a year, are now changing their mobile applications to what feels like weekly.
One of the most critical components that underpin agile software development is the iterative planning process. Instead of building new functionality for many months or years, and then shipping everything together, one of the methodologies of agile, Scrum, suggests working in sprints. Sprints are short blocks of development (2-4 weeks) to get valuable functionality to the customers as soon as possible. Work that is too big for one iteration, gets broken down into chunks that can be worked on, and ideally shipped within a sprint. Every sprint begins with a sprint planning meeting. In preparation for the meeting, the product manager ensures that the team has a healthy product backlog prioritized from top to bottom based on business and customer needs. The team will set a sprint goal, and take a certain number of work items, typically referred to as tickets or stories for that sprint. At the end of the sprint, the team holds a demo of what they’ve built and a retrospective where everyone looks back at their work and identifies what went well and therefore they should keep doing, what can be done better, and who should be recognized for their contributions over that period. Retrospectives enable software teams to continuously evolve their processes, and keep improving together as a team.
Since security work is also highly iterative, there is value for security teams to explore how frameworks like Scrum can be adapted to their needs. The part about backlog prioritization is especially worth emphasizing because both threats and the organization’s environment are continuously evolving. It’s critical that detection engineers, for instance, have a well-organized and prioritized backlog to decide what to work on next, instead of tackling items as they come up. The same applies to changes, additions, and others: similar to how every bug needs to have a priority and impact scores assigned for prioritization, every security gap needs to be evaluated and compared against all others waiting in the backlog. The process of tickets/user story management is foundational to software development and, if adopted well, can transform the way security teams operate.
Other concepts, systems, and processes
Working at scale
Software engineers are used to building everything for scale, thinking about latency, reliability, downtime, and other concepts. There is no capacity for manual intervention, so everything that can be built API-first should be built that way. Cybersecurity, on the other hand, has historically been reliant on collections of tools that can only be deployed, configured, fine-tuned, and maintained manually.
In the past decade, security vendors started to focus on making it easier for customers to adopt their products, which resulted in them prioritizing deployments at scale. Unfortunately, configuration and maintenance are still predominantly manual. Security teams should be looking for ways to do what they need to do at scale, and moving to security-as-code, enabled by the rise of detections-as-code, infrastructure-as-code, and other approaches well understood in software engineering.
Continuous testing and automation
There was a time when software companies needed to hire teams of quality assurance (QA) professionals to make sure that new code doesn’t break the existing functionality. At first, these were manual testers who would go into the application and try to find what’s broken by navigating to pages, clicking buttons, inputting information, and emulating other tasks usual for software users. Armed with findings, they would file bugs and send these to software engineers only to hear that “it works on their machine” and “sorry but I can’t reproduce it”.
As time went by and the amount of code released to production increased, QA started being asked to automate their testing. In addition, hundreds of outsourcing companies sprung up, promising to “take care of QA, for cheap”. While many startups had no choice but to work with outsourcing, with time it became apparent that it may not actually be the best way to solve the problem of quality.
More and more companies today see QA as one of the responsibilities of those building the code - software engineers. Large enterprises like Google and Amazon are clear that developers have to test their work. While experienced QA analysts still have jobs, the requirements for the role are rapidly changing. First and foremost, it became nearly impossible for those without skills in automation to get hired as no company can solely rely on manual testing. Second, over the past five years, I have been seeing the rise of a new role - software development engineer in test (software engineers with experience in testing). Engineers in test and QA today are acting akin to consultants, focusing on setting up infrastructure for testing, and supporting software engineers with best practices to test their work.
When I look at the evolution of QA in software development, I can’t help but think that penetration testing in security is going through a similar evolution. What started as manual work once every year, with time got outsourced to shops in countries where the cost of labor is lower. Today, bug bounty programs still work like QA did a long time ago: the bounty hunter reports an issue, there is a back and forth about the scope and severity of the issue, and then they receive payment. I think that similar to how QA became embedded in software development, blue teams will eventually be asked to build tests for their work. The future of security is inevitably in the hands of the so-called purple teams: one cannot defend if he or she doesn't know how to attack, and vice versa.
The fact of the matter is, however, that pen testing isn’t the only way to test the organization's security measures. Similar to how testing is now a part of the CI/CD pipeline in software development, it is inevitable that the same approach must be adopted in security. Every time a new detection or a new policy is released, every time a new asset appears on the network, every second the organization’s security posture should be tested and verified.
Thinking in systems and the role of systems design
Any complex solution requires strong technical design, an understanding of how and where it fits into the broader picture, what data it will use, what data it will send, how it will happen, and so on. The role of systems design in software engineering is hard to overestimate.
Many security teams do not have a deep understanding of their own environment, and most would likely panic if asked to present a map that explains their environment, data flow, and the relationship between all the security measures they have in place. Thinking in systems is critical for cybersecurity as a discipline, as without it, we are simply going to be adding more and more tools, not knowing where they overlap, where they conflict, and potentially even where they neutralize the benefits of one another.
Embracing the Agile Manifesto for cybersecurity
In 2001, seventeen software practitioners made a statement: the way software is delivered is not working and must be changed. What we take for granted today - frequent iterations, continuous development, and regular releases have all been around for just over 20 years. Before that, new software launches happened in huge, multi-year cycles. A group of programmers would spend 6-12 months planning, then another 3-4 years building software, and after that, the code was sent to be written on CDs, which would then need to be distributed.
Tired of explaining that developing software is different from manufacturing cars or building houses, the team of seventeen developed the Agile Manifesto:
“We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more.”
Based on that, 12 principles of Agile were crafted:
Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.
Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
Business people and developers must work together daily throughout the project.
Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
Working software is the primary measure of progress.
Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
Continuous attention to technical excellence and good design enhances agility.
Simplicity–the art of maximizing the amount of work not done–is essential.
The best architectures, requirements, and designs emerge from self-organizing teams.
At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
The impact of the Agile Manifesto and Agile principles on the software development practice and the technology industry as we know it is hard to overestimate. At the same time, the degree to which cybersecurity is going against the Agile Manifesto is uncanny:
Processes and tools are often valued over individuals and interactions. Many security teams still communicate with the rest of their organizations by issuing policies and directives.
Comprehensive compliance documentation is often valued more than working security measures (I have previously discussed this issue at length).
Formal requirements and contracts that check compliance boxes are often valued more than collaboration.
Following plans slows the ability to be nimble and adjust to ever-changing requirements.
I am not saying that every organization conflicts with these principles even if as an industry, we most certainly do. I am also not suggesting that these principles should be adopted as is for cybersecurity. But, security has a lot to learn from software engineering. The 12 Agile principles can be quite useful:
“Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.” Similar to software development, we should be able to maintain a constant pace indefinitely. It is apparent that we can’t - security professionals are understaffed and overworked. We need to look for ways to solve this problem.
“Continuous attention to technical excellence and good design enhances agility.” As previously discussed, we should think about our needs holistically and look for ways to better understand the environments we are in charge of defending, so that we can craft appropriate protective measures.
“Simplicity–the art of maximizing the amount of work not done–is essential.” Every day we hear that security teams “need to do more with less”; only a few times have I heard strong arguments to do less but do it better. Focused work in most fields is a better use of limited resources; cybersecurity might be one of those fields.
Closing thoughts
Blindly copying approaches and practices from any field is always a bad idea, and I am not suggesting that software engineering has all the answers for cybersecurity. However, since software development is about a decade or two ahead of security, it might be a good idea for us to see what worked there and what can be adopted, adjusted, and tweaked to fit the needs of our industry.
I have a strong conviction that the engineering approach to security is the answer to many of our needs. This approach isn’t new: hacking communities that started what is known today as security shared the same engineering mindset - they were tinkering, constantly developing new skills, and looking for ways to subvert the code into doing what it wasn’t designed to do. It’s just about time that we get back to the origins.
I’m not trying to be flippant, but is this not already happening? Who hasn’t received this message?
Security practitioners need to meet customers and partners where they are at and many (if not most) have already had to adapt to these ways of working.
This is a great take -- I largely agree but I suspect at the same time this is partially due to where the money is. In my experience companies with great software engineering disciplines tend to have higher expectations of their security team, and the security team expect more in kind. Just like in software engineering the most advanced teams/processes/etc tend to be found in the highest-performing organizations, the people tend to stay there too... and salary/benefits tend to be quite different as well. If the process and discipline knowledge is going to bleed out into more security teams in more industries I think there needs to be some external driver to promote this sharing.
Open-sourcing documentation for example is a good thing to do but it doesn't seem to be enough.