Challenges in Security Engineering Programs
Security Engineering is not on the horizon. It has already become mainstream … in certain circles. A guest post by Rami McCarthy.
Welcome to Venture in Security! Before we begin, do me a favor and make sure you hit the “Subscribe” button. Subscriptions let me know that you care and keep me motivated to write more. Thanks folks!
Please also check out my best selling book “Cyber for Builders: The Essential Guide to Building a Cybersecurity Startup” (over 2,200 copies sold).
This week, Venture in Security is excited to feature a guest article from Rami McCarthy. Rami is a security engineer, author of an active blog for technical security practitioners, and a frequent contributor to tl;dr sec, one of my favorite newsletters.
The topic of security engineering has gained a lot of attention in recent years, with a good number of mature security teams embracing an engineering mindset and changing the way they operate. While security engineering is frequently pictured as a solution to many of the industry’s problems, it comes with trade-offs and a variety of challenges. In this week’s post, Rami discusses the challenges of security engineering, and whether there is a path for security to use software engineering as a model.
The late Ross Anderson first published Security Engineering (the book) in 2001. The definition and philosophy he discussed have seeped across the industry.
“Security engineering is about building systems to remain dependable in the face of malice, error, or mischance. As a discipline, it focuses on the tools, processes, and methods needed to design, implement, and test complete systems, and to adapt existing systems as their environment evolves”. - What Is Security Engineering?
More recently, there has been a trend of evangelism for Security Engineering. Take:
Security needs more engineers by Frank Wang at Frankly Speaking (August, 2022)
What is Security Engineering? by Nielet D'Mello at The Pragmatic Engineer (April, 2024)
And of course, Venture in Security’s own: The rise of security engineering and how it is changing the cybersecurity of tomorrow.
With all due respect to these authors, Security Engineering is not on the horizon. It has already become mainstream … in certain circles.
“The future is already here – it’s just not very evenly distributed.“ - William Gibson
Who is doing Security Engineering?
There is no census for security engineering, but it’s evidently in practice across swaths of the industry. For example, we can take participation in Building Security In Maturity Model (BSIMM) as a proxy for the use of Security Engineering. You can find security companies, F500s, software startups, financial institutions, insurers, and healthcare firms.
Image Source: BSIMM13 report
Anecdotally, Security Engineering is strongly represented at companies where software engineering is a revenue source. This is likely because these companies understand the leverage engineering can provide against business problems, generally have leadership with literacy in engineering, and have an engineering leader of influence who can drive security engineering approaches.
Venture-funded companies share similar adoption rates. Partially, this may be due to network effects and the role of venture capital in motivating long-term efficiency in addition to early investment in preparation for scale. Of course, there is also correlation with younger companies. This is simply explained by the fact that transformation of an existing security program and company to a security engineering approach is hard and poorly specified. Well-established, and often compliance- or regulatory-mandated, processes can co-exist with security engineering, but it’s not the default.
Finally, we see Security Engineering adoption spreading at one degree of remove from these spheres. Companies selling into these pools especially can see motivation to adopt similar security programs. Additionally, the adjacency provides necessary cross-pollination. These characteristics that most commonly pair with Security Engineering are not a boundary on its utility, however.
Challenges of Security Engineering
The existing momentum and adoption of Security Engineering means we aren’t limited to abstractly discussing a future direction. We can also talk about the real-world challenges that crop up in building and operating this model of security program.
This discussion about the challenges of Security Engineering is long overdue. While there is no lack of thought leadership explaining how companies would benefit from embracing an engineering approach to security, what is currently missing is a discussion of the challenges surrounding this approach. Shared knowledge of these challenges better prepares companies to start Security Engineering practices, or to transform their legacy approaches.
Challenge 1: Hiring
This is one of the most discussed challenges in security, let alone security engineering. It’s been previously featured in Venture in Security:
Given that, the struggle in hiring for security engineering teams is easily explained. Candidates who are strong in two adjacent domains (Security, and Engineering) are categorically a smaller pool than traditional security or engineering hires. This is especially stark at junior levels, where having even a single area of competency is generally competitive. The result is that hiring these roles can be expensive, so figuring out a financial plan early helps build a Security Engineering program.
The further away the security domain from code, the more you currently see compromises on traditional security or engineering expertise. Finding “Application Security/Security Engineers” is generally slightly easier than “Cloud Security/Security Engineers” with “Security Operations or Detection Engineering/Security Engineers,” lagging and “Corporate Security/Security Engineers,” as the caboose.
Challenge 2: Relearning software engineering lessons
Much of Security Engineering maturity can be cribbed from Software Engineering teams.
Ross has already shared about Bringing software engineering principles, systems, and processes to cybersecurity, covering the role of product managers, the sprint process, Agile, and systems design.
There are more lessons and shared challenges to be spotlighted:
Promotion Based Development
Shiny Object Syndrome
Wherein security engineers focus on tackling the interesting problems at the bleeding edge, neglecting the boring basis. Almost universally true in terms of the relative focus of Security Engineering teams on Product Security and Enterprise Security.
“In the end, product security is a red herring; it’s enterprise security that urgently needs a paradigm shift. I know that we’ll end up with more regulation for software development: the narratives of “market failures” are unfalsifiable and it’s the nature of all bureaucracies to amass influence and expand. But I think we’re barking up the wrong tree.” - Source: Product security: barking up the wrong tree - Michał Zalewski
Metrics
Security programs, when they have metrics at all, are flooded with output metrics instead of outcome metrics. Measurement should be in the language of risk or business impact.
Product Management
It’s rare that security organizations have product managers, although I’ve seen some teams start to get great utility out of the TPM role. FAANG commonly uses TPMs -
Adobe has written repeatedly about setting up a Security Program Management Office. Lea Snyder and Devina Dhawan have both shared their career narratives featuring stints in (T)PM roles.
In addition to Ross’ Why security teams should start recruiting product managers, Matthew Sullivan also has written up The missing piece: the need for product management in security teams.
Driving Revenue
Perception as a cost center can severely limit the impact of a security engineering team. Balancing investment to take advantage of opportunities to drive revenue is important. This can involve improvements to velocity, but that’s often hard to quantify and recognize. Security Engineering building security features in the product is a more direct way to show business impact. Longer term there are big levers in sales enablement activities and compliance wedges for market making (like going through FedRAMP to break open a government market).
Challenge 3: Hammer & Nail
When you have a Security Engineering program, you can fall into the trap of trying to solve every problem with engineering. This can be challenging when integrating existing systems or processes without “APIs.” For example, sales enablement can stump security engineering teams, who have an allergic reaction to friction and spreadsheets but can’t figure out how to automate the process. Similar issues appear in work that still requires a human in the loop, GenAI or otherwise.
Security is already notorious for not using the tools companies are already paying for. Because of the engineering focus, many security engineering teams are especially weak at deploying, operating, and getting value from vendors. This can artificially inflate the total cost of ownership, and further exacerbate bias-to-build. The build-versus-buy decision is vulnerable to the not invented here pathology. The bigger the engineering team, the bigger the undifferentiated problem engineers will want to re-solve.
Security Engineering tends to pride itself on low toleration for toil. This can lead to low ROI engineering investment. “Ten hours of engineering could save us thirty minutes of toil a week” has a twenty week payback period, and then you need to consider what might change in that period as well as the opportunity cost.
Challenge 4: Engineering Disconnect
Security has historically experienced (and caused) a lot of friction with engineering teams. Even in companies where engineers value security, they have different incentive models and may not enjoy working on those challenges. Implementing Security Engineering brings a specific flavor of those challenges. Security Engineering should resemble engineering more closely than other security programs, and often needs to much more tightly integrate into the development lifecycle. Despite this, Security Engineering can be organizationally silo’d, especially with years of efforts to move CISOs under the CEO.
Handling Security Engineering Challenges
Despite these challenges, I still believe the path forward for security is to use software engineering as a model. However, we can do better to be proactive in how we manage these programs, and more aggressive in learning from software engineering.
To address Hiring challenges: figure out a financial plan early, be aware of tradeoffs in requiring deep expertise across two adjacent domains, and find ways to cross-train domain experts.
To address software engineering lessons: always consider mapping to software engineering practices, and justify any deviations that are differentiated or security specific. Tie yourself to revenue and value generation, and incrementally build up measurement and concrete outcomes for your program.
To avoid Hammer & Nail: be aware of the pitfalls of a builder bias, and make sure to standardize on a framework for making major build-versus-buy decisions. Make sure projects project ROI, and focus your engineering efforts on differentiated problems.
To address engineering disconnect: be careful in your organizational design, consider partnership-oriented engagement models, and make sure engineering is at least a co-owner of security risk.
This is a guest article from Rami McCarthy. Rami is a security engineer, author of an active blog for technical security practitioners, and a frequent contributor to tl;dr sec, one of my favorite newsletters.
 | A guest post by
|