Reporting Standards for Research in Psychology

Problem under investigation Participants or subjects; specifying pertinent characteristics; in animal research, include genus and species Study method, including: Sample size Any apparatus used Outcome measures Data-gathering procedures Research design (e.g., experiment, observational study) Findings, including effect sizes and confidence intervals and/or statistical significance levels Conclusions and the implications or applications Introduction The importance of the problem: Theoretical or practical implications Review of relevant scholarship: Relation to previous work If other aspects of this study have been reported on previously, how the current report differs from these earlier reports Specific hypotheses and objectives: Theories or other means used to derive hypotheses Primary and secondary hypotheses, other planned analyses How hypotheses and research design relate to one another Method Participant characteristics Eligibility and exclusion criteria, including any restrictions based on demographic characteristics Major demographic characteristics as well as important topic-specific characteristics (e.g., achievement level in studies of educational interventions), or in the case of animal research, genus and species Sampling procedures Procedures for selecting participants, including: The sampling method if a systematic sampling plan was implemented Percentage of sample approached that participated Self-selection (either by individuals or units, such as schools or clinics) Settings and locations where data were collected Agreements and payments made to participants Institutional review board agreements, ethical standards met, safety monitoring Sample size, power, and precision Intended sample size Actual sample size, if different from intended sample size How sample size was determined: Power analysis, or methods used to determine precision of parameter estimates Explanation of any interim analyses and stopping rules Measures and covariates Definitions of all primary and secondary measures and covariates: Include measures collected but not included in this report Methods used to collect data Methods used to enhance the quality of measurements: Training and reliability of data collectors Use of multiple observations Information on validated or ad hoc instruments created for individual studies, for example, psychometric and biometric properties Research design Whether conditions were manipulated or naturally observed Type of research design; provided in Table 3 are modules for: Randomized experiments (Module A1) Quasi-experiments (Module A2) Other designs would have different reporting needs associated with them 842 December 2008 ● American Psychologist standards for reporting either (a) a study involving random assignment of participants to experimental or intervention conditions (Module A1) or (b) quasi-experiments, in which different groups of participants receive different experimental manipulations or interventions but the groups are formed (and perhaps equated) using a procedure other than random assignment (Module A2). Using this modular approach, the JARS Group was able to incorporate the general recommendations from the current APA Publication Manual and both the CONSORT and TREND standards into a single set of standards. This approach also makes it possible for other research designs (e.g., observational studies, longitudinal designs) to be added to the standards by adding new modules. The standards are categorized into the sections of a research report used by APA journals. To illustrate how the tables would be used, note that the Method section in Table 1 is divided into subsections regarding participant characteristics, sampling procedures, sample size, measures and covariates, and an overall categorization of the research design. Then, if the design being described involved an experimental manipulation or intervention, Table 1 (continued) Paper section and topic Description Results Participant flow Total number of participants Flow of participants through each stage of the study Recruitment Dates defining the periods of recruitment and repeated measurements or follow-up Statistics and data analysis Information concerning problems with statistical assumptions and/or data distributions that could affect the validity of findings Missing data: Frequency or percentages of missing data Empirical evidence and/or theoretical arguments for the causes of data that are missing, for example, missing completely at random (MCAR), missing at random (MAR), or missing not at random (MNAR) Methods for addressing missing data, if used For each primary and secondary outcome and for each subgroup, a summary of: Cases deleted from each analysis Subgroup or cell sample sizes, cell means, standard deviations, or other estimates of precision, and other descriptive statistics Effect sizes and confidence intervals For inferential statistics (null hypothesis significance testing), information about: The a priori Type I error rate adopted Direction, magnitude, degrees of freedom, and exact p level, even if no significant effect is reported For multivariable analytic systems (e.g., multivariate analyses of variance, regression analyses, structural equation modeling analyses, and hierarchical linear modeling) also include the associated variance–covariance (or correlation) matrix or matrices Estimation problems (e.g., failure to converge, bad solution spaces), anomalous data points Statistical software program, if specialized procedures were used Report any other analyses performed, including adjusted analyses, indicating those that were prespecified and those that were exploratory (though not necessarily in level of detail of primary analyses) Ancillary analyses Discussion of implications of ancillary analyses for statistical error rates Discussion Statement of support or nonsupport for all original hypotheses: Distinguished by primary and secondary hypotheses Post hoc explanations Similarities and differences between results and work of others Interpretation of the results, taking into account: Sources of potential bias and other threats to internal validity Imprecision of measures The overall number of tests or overlap among tests, and Other limitations or weaknesses of the study Generalizability (external validity) of the findings, taking into account: The target population Other contextual issues Discussion of implications for future research, program, or policy 843 December 2008 ● American Psychologist Table 2 Module A: Reporting Standards for Studies With an Experimental Manipulation or Intervention (in Addition to Material Presented in Table 1) Paper section and topic Description Method Experimental manipulations or interventions Details of the interventions or experimental manipulations intended for each study condition, including control groups, and how and when manipulations or interventions were actually administered, specifically including: Content of the interventions or specific experimental manipulations Summary or paraphrasing of instructions, unless they are unusual or compose the experimental manipulation, in which case they may be presented verbatim Method of intervention or manipulation delivery Description of apparatus and materials used and their function in the experiment Specialized equipment by model and supplier Deliverer: who delivered the manipulations or interventions Level of professional training Level of training in specific interventions or manipulations Number of deliverers and, in the case of interventions, the M, SD, and range of number of individuals/units treated by each Setting: where the manipulations or interventions occurred Exposure quantity and duration: how many sessions, episodes, or events were intended to be delivered, how long they were intended to last Time span: how long it took to deliver the intervention or manipulation to each unit Activities to increase compliance or adherence (e.g., incentives) Use of language other than English and the translation method Units of delivery and analysis Unit of delivery: How participants were grouped during delivery Description of the smallest unit that was analyzed (and in the case of experiments, that was randomly assigned to conditions) to assess manipulation or intervention effects (e.g., individuals, work groups, classes) If the unit of analysis differed from the unit of delivery, description of the analytical method used to account for this (e.g., adjusting the standard error estimates by the design effect or using multilevel analysis) Results Participant flow Total number of groups (if intervention was administered at the group level) and the number of participants assigned to each group: Number of participants who did not complete the experiment or crossed over to other conditions, explain why Number of participants used in primary analyses Flow of participants through each stage of the study (see Figure 1) Treatment fidelity Evidence on whether the treatment was delivered as intended Baseline data Baseline demographic and clinical characteristics of each group Statistics and data analysis Whether the analysis was by intent-to-treat, complier average causal effect, other or multiple ways Adverse events and side effects All important adverse events or side effects in each intervention group Discussion Discussion of results taking into account the mechanism by which the manipulation or intervention was intended to work (causal pathways) or alternative mechanisms If an intervention is involved, discussion of the success of and barriers to implementing the intervention, fidelity of implementation Generalizability (external validity) of the findings, taking into account: The characteristics of the intervention How, what outcomes were measured Length of follow-up Incentives Compliance rates The “clinical or practical significance” of outcomes and the basis for these interpretations 844 December 2008 ● American Psychologist Table 2 presents additional information about the research design that should be reported, including a description of the manipulation or intervention itself and the units of delivery and analysis. Next, Table 3 presents two separate sets of reporting standards to be used depending on whether the participants in the study were assigned to conditions using a random or nonrandom procedure. Figure 1, an adaptation of the chart recommended in the CONSORT guidelines, presents a chart that should be used to present the flow of participants through the stages of either an experiment or a quasiexperiment. It details the amount and cause of participant attrition at each stage of the research. In the future, new modules and flowcharts regarding other research designs could be added to the standards to be used in conjunction with Table 1. For example, tables could be constructed to replace Table 2 for the reporting of observational studies (e.g., studies with no manipulations as part of the data collection), longitudinal studies, structural equation models, regression discontinuity designs, single-case designs, or real-time data capture designs (Stone & Shiffman, 2002), to name just a few. Additional standards could be adopted for any of the parts of a report. For example, the Evidence-Based Behavioral Medicine Committee (Davidson et al., 2003) examined each of the 22 items on the CONSORT checklist and described for each special considerations for reporting of research on behavioral medicine interventions. Also, this group proposed an additional 5 items, not included in the CONSORT list, that they felt should be included in reports Table 3 Reporting Standards for Studies Using Random and Nonrandom Assignment of Participants to Experimental Groups Paper section and topic Description Module A1: Studies using random assignment Method Random assignment method Procedure used to generate the random assignment sequence, including details of any restriction (e.g., blocking, stratification) Random assignment concealment Whether sequence was concealed until interventions were assigned Random assignment implementation Who generated the assignment sequence Who enrolled participants Who assigned participants to groups Masking Whether participants, those administering the interventions, and those assessing the outcomes were unaware of condition assignments If masking took place, statement regarding how it was accomplished and how the success of masking was evaluated Statistical methods Statistical methods used to compare groups on primary outcome(s) Statistical methods used for additional analyses, such as subgroup analyses and adjusted analysis Statistical methods used for mediation analyses Module A2: Studies using nonrandom assignment Method Assignment method Unit of assignment (the unit being assigned to study conditions, e.g., individual, group, community) Method used to assign units to study conditions, including details of any restriction (e.g., blocking, stratification, minimization) Procedures employed to help minimize potential bias due to nonrandomization (e.g., matching, propensity score matching) Masking Whether participants, those administering the interventions, and those assessing the outcomes were unaware of condition assignments If masking took place, statement regarding how it was accomplished and how the success of masking was evaluated Statistical methods Statistical methods used to compare study groups on primary outcome(s), including complex methods for correlated data Statistical methods used for additional analyses, such as subgroup analyses and adjusted analysis (e.g., methods for modeling pretest differences and adjusting for them) Statistical methods used for mediation analyses 845 December 2008 ● American Psychologist on behavioral medicine interventions: (a) training of treatment providers, (b) supervision of treatment providers, (c) patient and provider treatment allegiance, (d) manner of testing and success of treatment delivery by the provider, and (e) treatment adherence. The JARS Group encourages other authoritative groups of interested researchers, practitioners, and journal editorial teams to use Table 1 as a similar starting point in their efforts, adding and deleting items and modules to fit the information needs dictated by research designs that are prominent in specific subdisciplines and topic areas. These revisions could then be incorporated into future iterations of the JARS. Information for Inclusion in Manuscripts That Report Meta-Analyses The same pressures that have led to proposals for reporting standards for manuscripts that report new data collections have led to similar efforts to establish standards for the reporting of other types of research. Particular attention has been focused on the reporting of meta-analyses. With regard to reporting standards for meta-analysis, the JARS Group began by contacting the members of the Society for Research Synthesis Methodology and asking Figure 1 Flow of Participants Through Each Stage of an Experiment or Quasi-Experiment Assessed for eligibility (n = ) Excluded (total n = ) because Did not meet inclusion criteria