Are unit and integration test definitions still valid for modern Java projects? An empirical study on open-source projects

Abstract

Context: Unit and integration testing are popular testing techniques. However, while the software development context evolved over time, the definitions remained unchanged. There is no empirical evidence, if these commonly used definitions still fit to modern software development. Objective: We analyze, if the existing standard definitions of unit and integration tests are still valid in modern software development contexts. Hence, we analyze if unit and integration tests detect different types of defects, as expected from the standard literature. Method: We classify 38782 test cases into unit and integration tests according to the definition of the IEEE and use mutation testing to assess their defect detection capabilities. All integrated mutations are classified into five different defect types. Afterwards, we evaluate if there are any statistically significant differences in the results between unit and integration tests. Results: We could not find any evidence that one test type is more capable of detecting certain defect types than the other one. Our results suggest that the currently used definitions do not fit modern software development contexts. Conclusions: This finding implies that we need to reconsider the definitions of unit and integration tests and suggest that the current property-based definitions may be exchanged with usage-based definitions.