Reporting on Experimental Design and Statistical Analysis.

Introduction All of us in experimental science are concerned with scientific rigor and most journals have a policy on statistical analysis and data representation to ensure that issues of experimental design can be reviewed thoroughly. The policy at the Journal of Neuroscience is to require full reporting of statistical analyses and we ask our reviewers to report on the adequacy of those analyses. However, because of the breadth of research published in the Journal, we have not mandated specific formats for statistical reporting. In reviewing our existing policy, we decided it was important to update the guidelines to ensure that reviewers and editors have all of the information needed to evaluate the rigor of submitted studies. We therefore convened a group of editors to discuss what types of information would be essential for papers submitted to the Journal that include statistical analyses. We wanted to come up with a reporting policy that was flexible enough to encompass the breadth of neuroscience studies, but clear enough that a reviewer, editor, or reader could find the details of experimental design and statistical analyses easily. We also wanted to make sure that all of this information was within the manuscript so that a reader would not have to go to a separate document to find these details. Once we agreed on a set of criteria that were fundamental for evaluating experimental design, the draft policy was circulated to the Editorial Board for comment and editing. Through this process, we have come up with a consensus policy that we believe will not be onerous to authors, yet will provide a set of standards to ensure inclusion of important information in every manuscript, simplifying evaluation of this information by reviewers and editors. The new policy can be found here. Every manuscript must now include a section entitled “Experimental Design and Statistical Analysis” in the Materials and Methods that describes the design (control groups, sample sizes, sex of subjects, etc.) and all of the statistical tests used in the study. The policy is now more explicit in asking for full details of critical variables required for independent replication. In addition to a description of the statistical tests used, we ask that planned comparisons and details of control studies be reported in this section (for studies that have highly complex and heterogeneous analyses, authors may simply indicate in this section where in the manuscript details can be found; e.g., “Statistical analyses for Experiment 1 can be found in the Results describing Figure 2”). We also ask that authors state whether they have preregistered their study in this section. The policy also requires authors to report complete results of the statistical analyses, including degrees of freedom, effects sizes and exact p-values rather than a range and encourages authors to report the full variability and complexity of the data, for example, by using histograms or scatter plots in addition to bar and line graphs. Finally, we encourage authors to share raw data and to report where these data can be found. This policy will affect reviewers and editors as well because the review form includes a question asking whether experimental design and statistical analyses meet Journal standards. We realize that not all reviewers will have the expertise to evaluate every aspect of a study, so in addition to “yes” and “no” checkboxes, we have added a “don’t know” option. This will ensure that studies are evaluated fully by the reviewers or editors with the appropriate expertise. Overall, the rigor of science comes from independent replication and the Journal of Neuroscience is committed to publishing studies that are rigorous and have all of the details necessary to be replicated by others. This update to our policy on reporting experimental design and analyses is an additional step to ensure the ability to evaluate the science in manuscripts published in the Journal.