Using Cohen’s Effect Size to Identify Distinguishing Statements in Q-Methodology

Q-methodology was introduced more than 80 years ago to study subjective topics such as attitudes, perceptions, preferences, and feelings and there has not been much change in its statistical components since then. In Q-methodology, subjective topics are studied using a combination of qualitative and quantitative techniques. It involves development of a sample of statements and rank-ordering these statements by study participants using a grid known as Q-sort table. After completion of Q-sort tables by the participants, a by-person factor analysis (i.e., the factor analysis is performed on persons, not variables or traits) is used to analyze the data. Therefore, each factor represents a group of individuals with similar views, feelings, or preferences about the topic of the study. Then, each group (factor) is usually described by a set of statements, called distinguishing statements, or statements with high or low factor scores. In this article, we review one important statistical issue, i.e. the criteria for identifying distinguishing statements and provide a review of its mathematical calculation and statistical background. We show that the current approach for identifying distinguishing statements has no sound basis, which may result in erroneous findings and seems to be appropriate only when there are repeated evaluations of Q-sample from the same subjects. However, most Q-studies include independent subjects with no repeated evaluation. Finally, a new approach is suggested for identifying distinguishing statements based on Cohen’s effect size. We demonstrate the application of this new formula by applying the current and the suggested methods on a Q-dataset and explain the differences.