|
In an effort to produce a standardized, long lasting impact, IPDPS is introducing a computational result reproducibility appendix. This appendix aims at describing the processes used to obtain the computational results and will be appended to the accepted papers.
The motivation for adding this requirement is that data sets and software artifacts are important elements of research reproducibility. However, these elements are often absent when research results are recorded in a journal article or conference proceedings, or their description often lacks standardization that would allow for long term usability [1]. To cope with this, several computing conferences such as SC, ICPP, Europar, CCGrid, Cluster, Eurosys and OSDI/ATC already enabled and/or required authors to include an appendix describing how computational results were obtained. These reproducibility appendices have been shown to have a positive impact on the wider applicability and adoption of research produced [2].
[1] Report on Challenges of Practical Reproducibility for Systems
and HPC Computer Science, Kehaey, Kate, et al., 2025 https://www.arxiv.org/pdf/2505.01671
[2] Reproducibility Practice in High Performance Computing: Community Survey Results, Plale, Malik, Pouchard. Computing in Science & Engineering, 2021 https://www.osti.gov/servlets/purl/1827148
Reproducibility In Practice
After acceptance notification, accepted papers that include computational results will have to include in their final version a mandatory Artifact Description appendix (AD) and a strongly encouraged Artifact Evaluation appendix (AE). IPDPS will use a similar template to the one used at SC’XY, which is a result of multiple years of experimenting with the form [3]. It has been well documented, several examples can be found.
The AD is mandatory when computational results are present but in certain cases, authors might be unable to share the full computational artifact accompanying a paper. Under these circumstances, they have two alternatives.
- Authors may choose not to submit an AD Appendix, such as in cases of proprietary code restrictions. If authors opt-out, they must provide a detailed explanation on why an AD Appendix could not be provided. This explanation will be entered directly in the submission system (i.e., the explanation is not part of the AD Appendix).
- (preferred) Even in scenarios where the disclosure of source code is not feasible, authors are still encouraged to compile an Artifact Description (AD) Appendix, encapsulating all other important information of the computational artifact. Authors should follow the provided AD/AE template and fill in as much information as can be shared. For instance, if a Digital Object Identifier (DOI) for a computational artifact cannot be provided, authors may simply enter "N/A" or "N/A (proprietary)" in the corresponding section of the AD Appendix.
[3] See https://github.com/hunsa/sc24-repro/tree/main/for-paper-authors for an example.
|
