Lessons from my stint as a Statistical Consultant

School teaches you to be methodical. There is indeed merit to method. But good work requires more than mere method. For instance, take empirical research. No amount of method will make you curious enough to formulate a worthwhile research hypothesis. Curiosity is a blessing, fuelled by will. When one is curious, nothing else matters. One only itches to get to the bottom of the matter. And herein is the foundation for great research. Methods are mere tools, which you need to possess and these are necessary but hardly sufficient. And formal qualifications are not even necessary.

Research begins with good design. It is important to ensure rigor. One needs to give a lot of thought to the design. It is very similar to conceiving a baby. There has to be considerable passion to the plan in order to ensure that the conception is good. You need to ensure that all ends are fastened and for this it is important that you get your design reviewed by the grand masters. Even at this stage it helps to think beyond the obvious. Because its possible that the tool that can answer your queries does not exist. And then you’d need to create it – you should be so in love with your research.

You need to love your variables and constructs. These are the building blocks of your research – the cells. They need to be properly explored and understood. This does not mean a mechanical exploration of their distributions, which while necessary, comes later. You need to think of what is it that will cause your variables to act in a certain manner, what could cause them to break your heart! You need to wonder whether your equations are good enough for your research.

You need to be very careful when you are collecting data. Its like nurturing the baby for the data that you collect is what nourishes it. Statistics warns against the many biases that creep in at this stage. Remember, biases distort reality and are really bad for your analysis and research – bad influences on the baby. The next part is exploring the data on your variables. It is here that distributions become important. If you have thought adequately over what your variables prefer, its very likely that you’d start getting a fair idea of patterns hidden within your dataset. If there are missing values (babies can get sick), how do you treat the variable in a manner that it does not distort your research is critical. What could be the possible undesirable effects of the treatment (side effects) – is one treatment suitable over the other, so evaluate all possibilities. Then, there are outlying observations, what could explain them, think about it.

The main analysis comes next. This is the moment you have waited for. The results generally give you a high that is hardly matched. But there are heart breaks and you need to be aware of the latter possibility. And if you are truly in love with your research you’d still feel the high or else you could end shattered. Research is not for the faint hearted!