Background
This article will barely discuss technical issues if at all. After some experience with people and organization from Israel and outside of Israel i have noticed an issue with the approach of experienced, smart and capable staff to the methodology and logic of conducting a research / test. In the next page or so i will try to cover the top points for me while reading/conducting a research in the security field and in general. Please notice that by the word ‘research’ i am not referring to reverse engineering or malware analysis but rather to actual studies.
Part I: Common Warning Signs in Experimental Design
Appropriate Control Group
As in every experiment, in order to have something to match there must be a control group. Control group is a randomly selected group which is not modified in the way that the other subjects are. For example, we will use a traffic check exam. We want to understand if in a certain malware creates more HTTP traffic or not. We will initially take a couple of infected machines and check their traffic. The important thing to notice is that if we do not have an appropriate control group which we can compare the traffic to we cannot indicate whether 40% HTTP traffic is normal or if something out of the ordinary is occurring. Remember to aspire that the control group will be 10% of the size of your actual study at least. Less than that and you are prone to statistical errors.
Double-Blind Studies
We are human. Like it or not we are all prone to suggestions, paradigms we already have, things we believe in, and many more things that will cloud our judgment no matter how smart or capable we are. Our brain will make up information and will discard other pieces of information to get to the conclusions it wants without or consent or knowledge. In order to overcome these basic limitations one of the best things science have developed for us is the Double-Blind mechanism. As in the previous example, the best practice about double-blind will be to get a CAP files, analyze them without the knowledge of which groups they belong to (control, test-subjects, subgroups) and then, when we finished analyzing them, something else will divide them into groups without us knowing what are the titles of the groups and after we finish analyzing them, then we can know. In this way we avoid any biases which we already had since we do not know and therefore cannot influence the outcomes with our own internal thoughts.
Appropriate Study Size
Statistics is a problematic subject. It’s not that they lie or that they are incorrect, but again, we are the problem. We are not built to handle statistics and most of us do not know how to handle it. The first thing you should do to get results which are closer to the real world is have an appropriate subject base. If we wish to analyze information about all of the machines on the planet – for example, getting statistics about percentages of traffic world wide, we cannot have 30 subject. 30 machines is just not enough to expand your findings to the entire planet. And yes ,this is an extreme example but i have seen titles and studies saying 10% of world traffic is web with 100 machines as subject. Make sure that your subjects are appropriate to the amount you will like to expand your conclusions to.
The Wrong Subjects
This is a very common mistake and it is best explained using a demonstration. You will like to test for which is the most common browser in the world. You post on your Facebook wall asking your friend to vote so you have as much information as you can. After a week you go to view the survey. You find out that 60% are using Opera, 30% are using elinks and the other 10% are divide between Internet Explorer, Chrome and Firefox. Well that is a strong indication that Google and Mozilla are controlling the world media when they publish statistics which does not correlate with the real world.
You probably think ‘not enough people commented’. Well, let’s assume that 1000 people have commented. That is still not enough but not beause of the size of your subjects. The results are not compatible with the real world because most of your friends are BSD / *nix system administrators and they work from the console most of the time. The remained parts are friends of your spouse which is in the graphic design market and that’s their favorite browser. See how you have made a test with a good amount of subjects but they were not chosen according to the requests you have made. They could have all been from the same community, same city and you have a completely misguided (not wrong) indication. Remember, the results can still be real or even match reality, but the way they were harvested makes them invalid. Build your own standards for how a subject should look.
Summery
This is not a complete article, and there will be following articles but these are main points you should watch for when trying to conduct a research / experiment.
About The Author |
||
 |
Yuval Nativ |
|
| Yuval (tisf) Nativ is the manager of the Cyber Assualt Division of See-Security Technologies. As part of his job he is the manager of the Hacking Defined Experts program, a penetration tester, security researcher and developer. If you like to find him in our offices, just ask where Moriarty is… | ||