A framework for similarity-based and resource-aware orchestration of end-to-end test cases

Continuous integration environments are characterized by the execution of the test suite at each repository change, which can dramatically increase total project costs, making it not feasible to test the system adequately. Several testing optimization techniques exist, which generally focus on optimizing test execution either by parallelizing and sharing test resources or by improving how the test suite is executed through regression batching, prioritization, or test suite reduction. This work introduces SIMphony, a framework for end-to-end testing that combines both perspectives to optimize end-to-end test execution in terms of time, test resource redeployments, and test-case execution efficiency. SIMphony combines a similarity-based test strategy with resource-aware orchestration: it first analyzes test case similarities, then selects and applies the most suitable test regression strategy, and finally schedules test execution based on resource compatibility, maximizing parallelism and minimizing the number of test resource redeployments. The framework is evaluated on a real-world application from a European project. The evaluation results show that SIMphony helps reduce the number of test resource redeployments and execution time, while it either preserves code-coverage metrics or manifests only small reduced impacts. Concisely, prioritization reduces the time to reach maximum coverage, whereas reduction achieves a trade-off between test coverage and execution time. These results show that combining regression strategies with resource-based orchestration can address time-budget constraints in continuous integration environments for end-to-end testing.