Skip Links

Confounded nonsense

Potential pitfalls in security studies

Security Strategies Alert By M. E. Kabay, Network World
February 05, 2009 12:06 AM ET
Sign up for this newsletter now!

The long view of security strategies for your network.

A study sponsored by Cisco and carried out by InsightExpress using responses from more than 2,000 respondents in 10 countries indicated that accidental and deliberate violations of security by insiders are a serious threat to data confidentiality.

Jim Duffy of Network World wrote

"Thirty-nine percent of IT officials surveyed perceive negligence among employees as the main reason for the data security risk, while one in five pointed to disgruntled workers as the source. One in three IT respondents said portable hard drive devices are their top concern for how data is leaked -- more than e-mail (25%), lost or stolen devices (19%) and verbal communication with non-employees (8%)."

He added, "One in 10 employees surveyed admitted stealing data or corporate devices, selling them for a profit, or knowing fellow employees who did. This finding was most prevalent in France, where 21% of employees admitted knowledge of this behavior."

I’m sorry that the report (not Jim Duffy!) included arrant nonsense. Let’s look at some simple elements of statistical analysis (no, really simple: not even one formula today).

The problem with the statement about admitting stealing or knowing employees who stole is that it combines different causative factors that can result in the response. For example, suppose we are studying the effect of a new series of security-awareness cartoons on employees. One could form two groups, the cartoonified group (C+) and the uncartoonified group (C-) and then study their susceptibility to, say, phishing attacks sent to them via e-mail. Sounds great! We do the test and end up with:

Tricked: 72
Not Tricked: 128

Tricked: 52
Not Tricked: 148

For statistics aficionados, we compute a chi-square statistical test of independence with a value of 4.219 (with 1 degree of freedom) for a probability of 0.04 that there is no relationship between cartoon exposure and resistance to phishing. So, obviously exposure to the cartoons increased resistance to phishing messages, at least at the 0.05 level of significance, right?

Ah, but suppose that, without reporting the fact, we actually have an additional orthogonal (independent) factor defining two groups of employees: those who have previously been given a full-day security-awareness workshop (W+) and those who have not (W-). Well, that means that there are actually four test groups: W-C-, W-C+, W+C- and W+C+. And then we find out belatedly that the results, when classified with the additional information about security training, are as follows:

Tricked: 48
Not Tricked: 52

Tricked: 44
Not Tricked: 56

Tricked: 24
Not Tricked: 76

Tricked: 8
Not Tricked: 92

So, the results with both variables displayed indicate quite a different story: the cartoons have very little effect on people who had no security-awareness training but there was a noteworthy improvement after exposure to the cartoons among those who had been trained. In statistical terms, we call this phenomenon an interaction between the independent variables (workshops and cartoons); there are tests for decomposing the effects precisely (the log-likelihood ratio, G, is my favorite). For readers who have studied analysis of variance (ANOVA), the G-test is the non-parametric equivalent of a multivariate ANOVA. But enough of this airy persiflage.

M. E. Kabay, PhD, CISSP-ISSMP, specializes in security and operations management consulting services and teaching. He is Chief Technical Officer of Adaptive Cyber Security Instruments, Inc. and Associate Professor of Information Assurance in the School of Business and Management at Norwich University. Visit his Web site for white papers and course materials.

Our Commenting Policies
Latest News
rssRss Feed
View more Latest News