Test flakiness is a topmost concern in software test automation. While conducting pre-deployment testing, those tests that are flagged as ?flaky are put aside for being either repaired or discarded. We hypothesize that some flaky tests could provide useful insights if run in the field, and could help identify hard-to-detect failures that escape testing and present themselves in operation. We present the first study to investigate the behaviour of flaky tests when moved to the field. Our experimentation over 52 test methods labelled as flaky provides a first confirmation that moving from the laboratory to the field, the behaviour of tests changes and, in particular, the failure frequency of intermittently failing tests can increase. More importantly, we could identify few cases of field failures that would have been hard to detect while testing in house.

Insights from running flaky tests into the field: extended version

Barboni M;Bertolino A;De Angelis G
2022

Abstract

Test flakiness is a topmost concern in software test automation. While conducting pre-deployment testing, those tests that are flagged as ?flaky are put aside for being either repaired or discarded. We hypothesize that some flaky tests could provide useful insights if run in the field, and could help identify hard-to-detect failures that escape testing and present themselves in operation. We present the first study to investigate the behaviour of flaky tests when moved to the field. Our experimentation over 52 test methods labelled as flaky provides a first confirmation that moving from the laboratory to the field, the behaviour of tests changes and, in particular, the failure frequency of intermittently failing tests can increase. More importantly, we could identify few cases of field failures that would have been hard to detect while testing in house.
2022
Istituto di Analisi dei Sistemi ed Informatica ''Antonio Ruberti'' - IASI
Istituto di Scienza e Tecnologie dell'Informazione "Alessandro Faedo" - ISTI
Software testing
Flaky test
In Vivo testing
Intermittently failing test
File in questo prodotto:
File Dimensione Formato  
prod_466130-doc_183214.pdf

solo utenti autorizzati

Descrizione: Insights from running flaky tests into the field: extended version
Dimensione 507.63 kB
Formato Adobe PDF
507.63 kB Adobe PDF   Visualizza/Apri   Richiedi una copia

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/20.500.14243/444377
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus ND
  • ???jsp.display-item.citation.isi??? ND
social impact