Challenges of building products in cybersecurity, useful metrics, and ways to focus on what matters
Practical advice for cybersecurity founders and product leaders about building winning products in the industry
Welcome to Venture in Security! Before we begin, do me a favor and make sure you hit the “Subscribe” button. Subscriptions let me know that you care and keep me motivated to write more. Thanks folks!
Learn how your company can sponsor Venture in Security: Sponsorship.
Introduction
Building products and companies in cybersecurity is not an easy task because in many ways, the industry behaves differently from others. To start, it is very crowded, with hundreds and thousands of undifferentiated solutions promising to solve all security problems and more often than not, falling short of their promises. Moreover, it is an incredibly dynamic space with the landscape sometimes shifting and evolving overnight.
As a product leader, I meet many entrepreneurs and startup founders and see over and over how the vast majority get slowed down by the same types of problems. In this post, I am looking at six challenges of building products in cybersecurity and ways to overcome them. I will then cover metrics relevant to cybersecurity entrepreneurs and product leaders.
6 challenges of building products in cybersecurity and ways to overcome them
The challenge of customer discovery
In most industries, buyers and end users are open to talking to vendors because they recognize that software providers are there to identify their problems, pain points, and inefficiencies, and build solutions that remove them. The same, unfortunately, cannot be said about cybersecurity where few leaders (Chief Information Security Officers or CISOs) or practitioners are open to having transparent conversations with strangers. There are several reasons why this is the case:
Security teams are chronically overextended and understaffed, and therefore cannot prioritize talking to vendors over improving the security posture of their company
CISOs and practitioners alike are inundated by vendors who reach out from all fronts - calls, emails, social media messages, and conferences, to name a few
Product managers and founders tend to ask the same types of questions that an adversary would (what products the company is using, where their gaps are, etc.) - questions that can only be answered if there is a level of trust between parties
All this makes the lives of cybersecurity product leaders incredibly hard as it essentially makes them unable to do customer discovery, learn about the pain points, and brainstorm potential solutions. Here are some of the ways to address this:
Build relationships with CISOs and security practitioners by attending events, workshops, and webinars
Ask existing customers, VCs, and design partners for introductions to people in their network
When PMs get an opportunity to talk to security people - use that time to ask questions and be curious, instead of pitching their products and solutions
Using traditional product management frameworks
A lot of what is known as “product management best practices” comes from the hyper-growth experiences of consumer Silicon Valley companies such as Meta, Google, Instagram, and the like. The idea is that founders and product leaders can subsidize user growth, leverage referrals, growth loops, and virality to scale the number of daily active users (DAU) quickly, and then figure out the monetization. Although in theory, this approach sounds quite compelling, it breaks down when one tries to apply it in cybersecurity. This is because the vast majority of what we think of as cybersecurity is based on enterprise sales with long product evaluation, slow adoption cycles, and a heavy reliance on trust for purchasing.
Although there are no easy ways to deal with this problem, I have found the following to be a useful mental model for thinking about solutions:
Start with your ideal customer profile and build a deep understanding of their needs, pain points, use cases they want to see addressed, ability to influence the buying process, and so on
Assume that many of the so-called “best practices”, especially those that come from B2C space, are either not going to work or will have a different shape when implemented in security (this applies to things like growth models such as growth loops, product-led growth, and so on)
Adopt an evidence-based approach that enables you to test and validate your hypothesis and find what works for your organization
Over-optimizing technology and under-optimizing the business model
Because cybersecurity requires deep domain expertise, the vast majority of the founders and product leaders tend to have a technical background. This, combined with the objective need to ensure that technology is well designed, optimized, and works as intended, tends to make them put a lot of time and effort into building a good product. Since the time PMs have to tackle problems is finite, and conflicting priorities make it hard to do everything at once, the business side of the product tends to suffer.
In practical terms, this means that in some cases, the business model the company is trying to pursue is not viable from day one, while in others, the unit economics doesn’t leave any room for profitable margins, or the support costs are going to scale with the company reducing the customer lifetime value (CLV). Understanding the underlying business model and the economics of building a product or providing a service can help product leaders avoid costly mistakes.
Here are some of the ways to address this:
Before building the product, do the modeling in a simple excel spreadsheet to understand what needs to be true for metrics such as customer acquisition costs (CAC), customer lifetime value (CLV), average order size, cost of goods sold (COGS), and gross margin (GM), to name a few, to make financial sense
Identify scenarios in which the actual performance under these metrics puts the business model of the company under question
State assumptions as to what will each of these numbers look like when the product is launched and actively used by the customers
Compare the actual metrics against the projections to validate (or invalidate) assumptions, identify red flags early, and further optimize the business model
Oversimplifying the buying journey
It is common, especially for inexperienced product managers and first-time founders, to oversimplify the buying journey for their product. Some companies think “we just need to convince a CISO that we are better than alternative options” while others go another route and talk about “just doing product-led growth and going bottom-up”. Unfortunately for both, neither is how security products are actually purchased. The mechanics will heavily depend on the specific product and a specific market segment, but it is common for hands-on practitioners, CISOs, compliance, finance, and purchasing departments to evaluate new tools from their unique lenses. For some products, the list may include chief technology officers (CTOs), software developers and engineering managers, and others. Different parties will have different roles and different levels of influence on the final decision depending on the organization’s size, structure, power dynamics, and many other factors.
Although there are no easy ways to deal with this problem, I have found the following to be a useful mental model for thinking about solutions:
Map the customer’s buying journey and identify who is involved in purchasing process, what they care about, what their influence is, what angles they are looking at the problem from, and others
Get all of this information in one place, and understand what your product and the company need to offer to each party to satisfy their requirements
Custom ways in which every organization does security
Every organization’s security environment is different, and therefore every company does security in its own way. This means that workflows, playbooks, risks, attack vectors, and what constitutes “suitable security coverage” will vary widely from one organization to another. This complexity and variability make the lives of product leaders quite hard: every customer they talk to will inevitably bring their own “must have '' feature requests. This can be overwhelming, especially because from the outside, many of these organizations with unique needs will fall under the same customer profile.
Building everything prospects ask is not possible: not only would it result in Frankenstein products, but also no company has enough resources to do that. On the other hand, not responding to the needs customers are being vocal about can result in issues with adoption, or high churn rates.
To address this dilemma,
Companies should look for ways to design their products as building blocks and make them extendable by using open APIs and the latest technical design principles
Understand the willingness and the ability of customers in this particular segment to tailor out-of-the-box products to their needs
Do thorough customer research to understand what is the 80% of product functionality needed by all customers, and what is the other 20% that users can do on their own (this assumes that there are ways for them to do it)
The inability to easily move up- and down-market and the resulting need to pick core focus early
Naturally, some types of problems are only seen in consumer or enterprise space, while others span all types of customers at once. For those that are common across consumer, SMB, and enterprise space, it is common that a company would penetrate one of the segments first, and then seek adoption across others. Think about productivity tools as an example: people, small businesses, and large corporations all need to manage tasks and projects. While the scale and some nuances do vary, we often see that what distinguishes a consumer-focused task manager from an enterprise-focused one are multi-tenancy and multi-user support, rules-based access control (RBAC), API access, single sign-on (SSO) and access to audit logs.
Although everyone - people, Fortune 100 companies, and everything in between needs cybersecurity, what they see as “security” varies. People are notorious for not wanting to adopt (not to mention having to pay for) anything that introduces friction. Small and medium-sized businesses only now are starting to realize that they do need security. For SMBs, security means having an “everything-in-one solution”. For enterprises, on the other hand, security is about advanced capabilities that are API-first, and much more specialized than SMB-focused tools. A product that started in the consumer cannot typically move up the market, and vice versa. An enterprise solution for network detection and response (NDR), for example, cannot be simplified to the consumer level because consumers don’t need or care about an NDR.
To address this reality,
Founders and product leaders need to understand their target market, and the kind of solutions they are looking for
Startups need to commit to solving a problem for a segment of the market while realizing that with few exceptions, they cannot hope for cross-customer segment mobility
Metrics for cybersecurity product managers
The conventional product management wisdom suggests that one of the responsibilities of a product leader is to track and optimize metrics - quantitative measurements that reflect how people benefit from a specific solution. Anyone who has read product management books, attended workshops, or even simply gone through an interview, knows that “what is not measured, cannot be managed”.
The practice of product management is, however, much more nuanced. Context matters a lot, and the realities of different organizations, geographies, cultures, and market segments heavily influence what can be measured, and what actions can be taken based on these observations. In this article, I am looking at cybersecurity product management and how metrics product leaders are tempted to track and report on may not be what they seem.
3 core metrics for cybersecurity product managers
Conversion rate
Conversion rate is one of the most important metrics companies, and subsequently - product teams, obsess about. This metric tracks the percentage of all users or visitors who take a desired action.
Who owns conversions in the organization will depend upon who can influence the outcome. For example,
If the product is fully sales-led, and whether or not the deal gets closed is in the hands of sales, then conversion is owned by sales
If the product is fully product-led, and whether or not a free user becomes a paying customer is in the hands of product, then conversion is owned by marketing and product teams (marketing owns the signup on the website, product owns in-app conversion)
Conversion rate is an incredibly important metric, as it means that the company can efficiently capture customer interest and translate it into revenue. Although cybersecurity product managers should be monitoring conversion closely, they must understand some nuances that surround it.
In cybersecurity, the buying process is incredibly complex: not only it heavily relies on trust but also it takes a long time, and involves a large number of people - security practitioners, CISOs, legal, compliance, purchasing, and so on (most of the cybersecurity is enterprise sales). Even the so-called product-led companies are not an exception: the in-product experience is just one of many factors that influence the buying process. Moreover, there is typically no one-to-one relationship between users and customers: ten anonymous users who signed up at different times with gmail.com domains could all be a part of the same security team who evaluates the tool from different perspectives. Tracking conversion is therefore best done on the customer level rather than the user level.
Usage and Engagement
Traditionally, product teams are tasked with growing engagement and ensuring that users not only come back but that they spend a lot of time using the product. This is driven by the understanding that the more time users spend in the product, the more likely they are to experience real value, develop a “habit” and therefore continue using it. Low usage is often linked to a higher probability of churn while high user engagement helps companies validate that they are solving an important problem, raise capital (investors are looking for revenue and engagement), and identify new expansion opportunities. However, this approach doesn’t work the same way in cybersecurity.
Let’s start by looking at a real example. Say, a company bought a security automation and orchestration platform (often abbreviated as SOAR) to eliminate unnecessary manual tasks and simplify the communication between many of its security products. At first, the security team will be very active in configuring workflows, setting up automation, and creating playbooks to make the security operations center (SOC) run smoothly. After the initial setup is done, the team will stop engaging with the product and may only see one or two users log in twice a month to tweak some configuration here and there. While conventional wisdom may ring the alarm that the customer is disengaged, and with high potential for churn, the reality might be quite the opposite: the product is saving the security team a lot of time, and it is happy that everything works smoothly - so smoothly in fact that it doesn’t even need to check on the product often.
This example highlights a simple yet often forgotten truth: most cybersecurity products that add a lot of value to the security stack are invisible. Having to navigate 20-70 tools, security teams will only actively engage with a limited few, while leaving the rest to do their job in the background. Hence why product teams must adopt a different angle when seeking to understand if the product is being used: instead of seeking engagement, they can track the flow of data, alerts, and other core tasks that show that the product is alive.
After learning that an average security team will use tens of product dashboards to do their jobs, many vendors became excited about the idea of a “single pane of glass” - one screen to rule them all. Every vendor wants their product to be the one dashboard that is left, and with that, there is a constant fight for user’s attention. Product leaders need to understand this reality and embrace the fact that being a well-loved tool that works great in the background may be a better idea than being a shiny dashboard everyone complains about.
Detection accuracy
Although not all cybersecurity products are designed to generate some kind of detections, many do. Detection accuracy is a metric that applies to the security tooling that does trigger alerts notifying users that a specific behavior has been detected.
Two types of metrics are useful to track in the context of detection accuracy:
False positives (i.e., a false alarm, when the tool triggers a detection on normal behavior)
False negatives (i.e., a missed attack, when the tool misidentifies an attack as normal behavior and does not trigger a detection)
Security vendors are faced with a serious, and I dare to say - an impossible-to-win challenge: how to reduce the number of false positives and false negatives and bring them as close to zero as possible. The reason it is impossible to accomplish this is that every customer’s environment is unique, and applying generic detection logic across all organizations will inevitably lead to gaps in security coverage.
Product leaders need to keep in mind that false positives make it more likely that a real, critical detection will be missed, while false negatives mean that the product is not doing the job the tool was bought to do.
Focusing on what matters for cybersecurity product managers
Building high-performing products that solve hard security problems is hard, and taking them to the market is even harder. It is no wonder product leaders are tempted to look for shortcuts and easy ways to measure how they are tracking toward their goals. Unfortunately, although quantitative metrics have value, they can easily become a distraction.
The vast majority of cybersecurity companies are early-stage startups, primarily operating in the B2B space. Security teams themselves are quite small, too. In practical terms, all this means that product managers see a low volume of users and in-product activity, and metrics such as daily active users (DAU) and monthly active users (MAU) don’t provide a lot of useful information. In this environment, trying tools like A/B testing or making decisions based on quantitative data alone is not feasible.
To identify user needs and focus on what matters, product managers need to look for qualitative insights. I have found the following to be quite useful:
Talking to sales & sales engineering teams to get visibility into the sales process - what questions people ask, what problems they are looking to solve, what gaps have they experienced in other solutions, etc. Sometimes, being a “fly on the wall” in the sales call can provide more insights than several customer surveys combined.
Working with sales on conducting win-loss analysis. In particular, it can be very useful to connect with prospects who decided to go with competitive offerings and learn about their evaluation process, opportunities for improvement, and gaps they’ve identified.
Analyzing questions coming to support can resurface what areas of the product are hard to use, what critical features are missing, and which are hard to discover.
Most importantly, practicing continuous discovery habits and talking to users, prospects, and non-users will help understand the needs, pain points, and challenges cybersecurity teams are facing, and design products that solve them. When in doubt, it’s always better to ask a question and understand the actual motivators of users instead of going with what the data, devoid of a broader context, seems to be suggesting.
Note: the original version of this article was published on TechCrunch: