In this piece, I am discussing why it is so hard to find reliable sources of information about security, why we should not be creating data out of thin air, and where we should go from here
Some of the various small "institutes" where statistics are sourced from are likely making up survey results, in my opinion. There is a well-known one that produces many surveys that get play all over the media landscape, often claiming to have thousands of survey respondents...but no one actually works for them except a husband-wife team. They let vendors write obscenely long, "let's just add a few more questions" surveys that are often 50+ questions, some with as many as 20 multi-select answers. This is the kind of survey that, in a real survey-taking organization, would have a non-complete rate of 90%, and I say that as someone who did market research full time in my early career. However, the Institute claims their non-completion rates are very low and are happy to use pretty much any questions you like. They'll just charge you more for more questions.
They have no survey takers employed by them, no actual way to have the surveys completed (you're not going to talk a thousand CISOs into answering a detailed 50-question survey that takes an hour using cheap outsourced labor). There is literally no way they can be doing what they claim they are doing...and I've never yet met an executive who says they've been party to one of these surveys from this institute, and no one has ever complained online about the giant 30-50 question surveys they accept.
When I worked with this institute, they couldn't show us raw data or crosstabs for their survey results, and when asked for crosstabs took about as long as you'd think it'd take to...well...make some up.
When I think of how this Institute's statistics are shared as if they are some kind of home truths, it fills me with disgust. It's actively fraudulent -- I have seen literally nothing to indicate they can achieve any of what they claim to be achieving in the survey-taking space -- yet nothing is done, and they continue to collect funds on the basis that vendors are desperate for these statistics and the Institute will reliably deliver them to you.
Cybersecurity Ventures also employs almost no one. The people they do employ are basically "creatives" -- writers, editors, videographers. Real market research firms have only a small fraction of workers employed in these roles, and the roles doing the actual research and analysis dominate. But Cybersecurity Ventures and the Institute I'm speaking of? No, they don't have anyone like that employed, in spite of coming out with many pieces of "original research" each year based on large-scale surveys of high-ranking employees.
Thank you Jeanette, this is such a great perspective! We should also be mindful when looking at what "institutes" produce unless we know what they actually are and how they work.
Wow. Great post as always. Your ability to put into words what the collective security community knows and has harped on for years is unmatched. Keep up the great work, Ross!
Another great piece Ross. A recent straw poll from fellow front line security types (a closed door session with one or two incident response providers) suggests 70% to 90% don’t announce when affected by a “material” incident. This has certainly been my observation and whatever the percentage, a significant proportion is unreported. Also, the business impact of incidents has strayed into the domain of “death by a thousand cuts”. Since there’s not even a clear definition of the boundary of “Cyber” (other than the great CYBoK material from University of Bristol) the boundary of what gets counted, even if visible, is so distributed and often individually “small” that it would anyway fall out of view. What’s clear is this needs to become more disciplined, because the temperature has risen significantly over just the last 3 years (not a statistically relevant point just personal experience!).
Thanks John, 70% to 90% is a crazy number but I am not at all surprised. At the end of the day, business goals and publicity are more important to companies that contributing to collective knowledge.
That is quite a can of whoop-ass you’ve opened. The claims about the number of open cyber jobs are thinly sourced — I agree with Ben Rothke’s take.
Thanks Andrew, indeed!
Some of the various small "institutes" where statistics are sourced from are likely making up survey results, in my opinion. There is a well-known one that produces many surveys that get play all over the media landscape, often claiming to have thousands of survey respondents...but no one actually works for them except a husband-wife team. They let vendors write obscenely long, "let's just add a few more questions" surveys that are often 50+ questions, some with as many as 20 multi-select answers. This is the kind of survey that, in a real survey-taking organization, would have a non-complete rate of 90%, and I say that as someone who did market research full time in my early career. However, the Institute claims their non-completion rates are very low and are happy to use pretty much any questions you like. They'll just charge you more for more questions.
They have no survey takers employed by them, no actual way to have the surveys completed (you're not going to talk a thousand CISOs into answering a detailed 50-question survey that takes an hour using cheap outsourced labor). There is literally no way they can be doing what they claim they are doing...and I've never yet met an executive who says they've been party to one of these surveys from this institute, and no one has ever complained online about the giant 30-50 question surveys they accept.
When I worked with this institute, they couldn't show us raw data or crosstabs for their survey results, and when asked for crosstabs took about as long as you'd think it'd take to...well...make some up.
When I think of how this Institute's statistics are shared as if they are some kind of home truths, it fills me with disgust. It's actively fraudulent -- I have seen literally nothing to indicate they can achieve any of what they claim to be achieving in the survey-taking space -- yet nothing is done, and they continue to collect funds on the basis that vendors are desperate for these statistics and the Institute will reliably deliver them to you.
Cybersecurity Ventures also employs almost no one. The people they do employ are basically "creatives" -- writers, editors, videographers. Real market research firms have only a small fraction of workers employed in these roles, and the roles doing the actual research and analysis dominate. But Cybersecurity Ventures and the Institute I'm speaking of? No, they don't have anyone like that employed, in spite of coming out with many pieces of "original research" each year based on large-scale surveys of high-ranking employees.
Thank you Jeanette, this is such a great perspective! We should also be mindful when looking at what "institutes" produce unless we know what they actually are and how they work.
Wow. Great post as always. Your ability to put into words what the collective security community knows and has harped on for years is unmatched. Keep up the great work, Ross!
Thank you my friend!
Another great piece Ross. A recent straw poll from fellow front line security types (a closed door session with one or two incident response providers) suggests 70% to 90% don’t announce when affected by a “material” incident. This has certainly been my observation and whatever the percentage, a significant proportion is unreported. Also, the business impact of incidents has strayed into the domain of “death by a thousand cuts”. Since there’s not even a clear definition of the boundary of “Cyber” (other than the great CYBoK material from University of Bristol) the boundary of what gets counted, even if visible, is so distributed and often individually “small” that it would anyway fall out of view. What’s clear is this needs to become more disciplined, because the temperature has risen significantly over just the last 3 years (not a statistically relevant point just personal experience!).
Thanks John, 70% to 90% is a crazy number but I am not at all surprised. At the end of the day, business goals and publicity are more important to companies that contributing to collective knowledge.