Making sense of a data set is fundamental to our role, not only as scientists, but also as clinicians. A simple method exists that allows for a systematic manner to answer the key question in research: Can we reject the null hypothesis?
As a short review, the null hypothesis is the general statement that states that there is no significant difference between populations being studied. The difference that may be present is simply due to error or chance. This means that in a given set of data, the majority will fall under the middle area of a normal curve, where the differences observed are simply due to chance, or naturally occurring randomness.
The p-value is short for probability value; a point at which a researcher can say the differences observed in a set of data did not occur due to chance, or random error. Using the p-value, we take the position that the sample follows a normal curve. Researchers, depending on the strictness of a study can decide where to set the alpha. By convention, most studies set this at <0.05. This means the difference observed in a data set is beyond 95% of a normal distribution, hence, less likely to be due to chance. The alpha is the point beyond which we can say the values are beyond the normal distribution, while the p-value represents how far away we are.
For example, a study determining whether a drug (Drug X) could lower blood pressure effectively compared to placebo, sets the alpha at <0.05. Their null hypothesis states: Drug X has no statistically significant difference in lowering blood pressure when compared to placebo. They do their statistical analysis, and get a p-value of 0.02; therefore, the null hypothesis can be rejected. This means the observed differences in blood pressure was not due to chance, or error.
Using the p-value, one cannot outright say the drug has an affect. Rather, we can only be certain that the difference we observed was probably an effect of the intervention, and not due to error. This always has to be taken in mind when interpreting the p-value.
Reference
Greenland S, Senn SJ, Rothman KJ, et al. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Eur J Epidemiol. 2016;31(4):337–350.