MOTEF: a testing framework for runtime monitoring infrastructures

Intelligent monitoring systems can effectively predict or detect anomalies and issues in smart working systems and ecosystems and implement the proper countermeasures. However, for their effective and efficient use, attributes like responsiveness, performance, and quality should be properly tested and assessed before integrating the monitoring system into an ecosystem. The work aims to present a framework, called MOnitoring TEsting Framework (MOTEF), focused on the testing and evaluation of a generic monitoring system performance. In particular, the framework allows testing the monitoring system in isolation or when used in a smart environment to provide functional or non-functional property predictions. By simulating the runtime execution of a smart environment, MOTEF lets testing and assessment of a generic monitoring system establish its working boundaries. The results collected can be used to design a smart environment architecture to fulfil its global performance constraints better. This work presents the architecture of MOTEF and its preliminary implementation. It also validated and showcased the use of MOTEF in evaluating the performance of an existing monitoring system in isolation and when it is used in a smart environment. The results have been assessed by considering two research questions about the monitoring system’s responsiveness and effectiveness in proving required functional or non-functional property predictions.