Experiences With Replicable Experiments and Replication Kits for Software Engineering Research

Abstract

Replications and replicable research are currently gaining traction in the software engineering research community. Our research group made an effort in the recent years to make our own research accessible for other researchers, through the provision of replication kits that allow rerunning our experiments. Within this chapter, we present our experiences with replication kits. We first had to learn which contents are required, how to structure them, how to document them, and also how to best share them with other researchers. While this sounds very straightforward, there are many small potential mistakes, which may have a strong negative impact on the usefulness and long-term availability of replication kits. We derive best practices for the content and the sharing of replication kits based on our experiences. Moreover, we outline how platforms for replicable research may further help our community, especially with problems related to the external validity of results. Finally, we discuss the lack of integration of replication kits into most current review processes at conferences and journals. We also give one example for a review process into which replication kits were well-integrated. Altogether, this chapter demonstrates that making research replicable is a challenging task and there is a long road ahead until our community has a generally accepted and enforced standard of replicability.