The TOMS Initiative and Policies for

Replicated Computational Results (RCR)

TOMS accepts manuscripts for an additional, and presently optional, review of computational results. This Replicated Computational Results (RCR) review is focused solely on replicating any computational results that are included in a manuscript. If the results are successfully replicated, the manuscript receives a special RCR designation when published. This page outlines the TOMS policies for determining the RCR designation.

• RCR Review Initiation
• RCR Review Steps
• RCR Review Methods

RCR Initiation

If you are interested in submitting your manuscript for RCR designation, use the standard TOMS submission process. Then send an email message to the TOMS Editor-in-Chief expressing your interest in the RCR review process.

RCR Review Steps

The RCR process includes the following steps:

1. RCR review request: When authors submit a manuscript for review, they can optionally request a replicated computational results review, which will be conducted independently from the standard review process.
2. Standard reviewer assignment: Once the manuscript has been assigned to an associate editor (AE), the AE will assign referees for the standard review process.
3. RCR suitability review: Concurrent with assigning standard reviewers, the Editor-in-Chief (EiC) and AE will briefly review the manuscript to determine if it is suitable for an RCR review. The decision about RCR suitability may be delayed until the first round of standard reviews is complete.
4. RCR reviewer assignment: After determination of RCR suitability and concurrent with the standard peer review process, the AE will assign an RCR reviewer whose sole responsibility is to replicate manuscript computational results. Unlike the other reviewers, the RCR reviewer will be known to the authors and work together with the authors during the RCR process.
5. RCR review process: Replicating computational results will require a multi-faceted approach. TOMS editors will advise the RCR reviewer on acceptable approaches, but ultimately the RCR reviewer has the responsibility to declare whether or not computational results in the manuscript are replicated. RCR reviewers will document the details of how results were replicated.
6. RCR Determination: We anticipate that manuscripts submitted for RCR designation will almost surely succeed in eventually achieving this designation, at least in the introductory phase of the RCR initiative. This is important for assuring that RCR reviewers will obtain a published article as an outcome from their efforts, and to reduce authors' risk in voluntarily submitting to this review process.
7. RCR Review Failure: There is some risk now and in the future that RCR efforts will fail. In this case, we must acknowledge that the manuscript is not ready for publication with the presented results. During the introductory phase, the EiC will personally manage this situation if it occurs and will work with the authors to avoid rejecting the manuscript outright. As the RCR initiative matures, we anticipate that failed RCR reviews would constitute grounds for returning the manuscript back to the authors for revision, or for rejection if concerns were serious.
8. Publication: A manuscript whose computational results are successfully replicated will be published with a special RCR designation as a text-only note on the bottom of the first page. A special graphic designation may become possible in the future. The RCR referee will be acknowledged in the published paper as author of the RCR review report that will appear with the published manuscript. The RCR referee's report will be published as a TOMS article, immediately following the RCR-reviewed article. This report will also go through a light review process to assure that it is well written and contains required report elements.

Methods for Replicating Results

For all RCR reviews, we rely on the expertise of the RCR reviewer to make the final determination of the RCR designation. Presently we have two basic approaches for assessing replicability. The first is more desirable, but not always possible.

1. Independent replication: The authors provide the RCR reviewer access to, or sufficient description of, the computational platform used to produce the manuscript results. Access could be:
   1. A direct transfer of software to the reviewer or a pointer to an archive of the software, and a description of a commonly available computer system the reviewer can access.
   2. A guest account and access to the software on the system used to produce the results.
   3. Detailed observation of the authors replicating the results.
2. Review of computational results artifacts: In some situations, authors may not be able to readily replicate computational results. Results may be from a system that is no longer available, or may be on a leadership class computing system to which access is very limited. In these situations, careful documentation of the process used to produce results could be sufficient for an RCR designation. In this case, the software should have its own substantial verification process to give the reviewer confidence that computations were performed correctly. If timing results are reported, the authors' artifacts should include validation testing of the timers used to report results.