IPDPS 2007 Reproducibility Initiative

To produce a standardized, long-lasting impact, IPDPS is requiring a computational result reproducibility appendix for the accepted papers. This appendix aims to describe the processes used to obtain the computational results and will be appended to the accepted papers. 

The motivation for adding this requirement is that data sets and software artifacts are important elements of research reproducibility. However, these elements are often absent when research results are recorded in a journal article or conference proceedings, or their description often lacks standardization that would allow for long-term usability [1]. To cope with this, several computing conferences such as SC, ICPP, Europar, CCGrid, Cluster, Eurosys, and OSDI/ATC already enabled and/or required authors to include an appendix describing how computational results were obtained. These reproducibility appendices have been shown to have a positive impact on the wider applicability and adoption of research produced [2].

Reproducibility in practice

After acceptance notification, accepted papers that include computational results will have to include in their final version a mandatory Artifact Description appendix (AD) and a strongly encouraged Artifact Evaluation appendix (AE). IPDPS will use a similar template to those used in other conferences in the field, which has been refined over multiple years of experimentation. It has been well documented; several examples can be found.

A sample IPDPS template can be found here: https://github.com/IPDPS/ipdps26-repro/

The AD is mandatory for accepted papers when computational results are present. The authors can clearly mark which results are likely to be reproducible and which may not be. 

In certain cases, authors might be unable to share the full computational artifact accompanying a paper. Under these circumstances, they have two alternatives.

1. Authors may choose not to submit an AD Appendix, such as in cases of proprietary code restrictions. If authors “opt-out”, they must provide a detailed explanation on why an AD Appendix could not be provided. This explanation will be entered directly in the submission system (i.e., the explanation is not part of the AD Appendix).
2. (Preferred) Even in scenarios where the disclosure of source code is not feasible, authors are still encouraged to compile an Artifact Description (AD) Appendix, encapsulating all other important information of the computational artifact. Authors should follow the provided AD/AE template and fill in as much information as can be shared. For instance, if a Digital Object Identifier (DOI) for a computational artifact cannot be provided, authors may simply enter "N/A" or "N/A (proprietary)" in the corresponding section of the AD Appendix.

References
[1] Report on Challenges of Practical Reproducibility for Systems and HPC Computer Science, Kehaey, Kate, et al., 2025  https://www.arxiv.org/pdf/2505.01671

[2] Reproducibility Practice in High Performance Computing: Community Survey Results, Plale, Malik, Pouchard. Computing in Science & Engineering, 2021 https://www.osti.gov/servlets/purl/1827148

[3] See https://github.com/hunsa/sc24-repro/tree/main/for-paper-authors for an example.