Write a Blog >>
ICST 2020
Sat 24 - Wed 28 October 2020 Porto, Portugal

Modern software engineers automate as many tests as possible. Test automation allows tests to be run hundreds or thousands of times: hourly, daily, and sometimes continuously. This saves time and money, ensures reproducibility, and ultimately leads to software that is better and cheaper. Automated tests must include code to check that the output of the program on the test matches expected behavior. This code is called the test oracle and is typically implemented in assertions that flag the test as passing if the assertion evaluates to true and failing if not. Since automated tests require programming, many problems can occur. Some lead to false positives, where incorrect behavior is marked as correct, and others to false negatives, where correct behavior is marked as incorrect. This paper identifies and studies a common problem where test assertions are written incorrectly, leading to incorrect behavior that is not recognized. We call these tests blind because the test does not see the incorrect behavior. Blind tests cause false positives, essentially wasting the tests. This paper presents results from several human-based studies to assess the frequency of blind tests with different software and different populations of users. In our studies, the percent of blind tests ranged from a low of 39% to a high of 95%.

Mon 26 Oct
Times are displayed in time zone: Greenwich Mean Time : Lisbon change

15:30 - 17:00: RT9 - Empirical Analysis and Program RepairResearch Papers at Farfetch (D. Maria) +11h
Chair(s): Corina S. PasareanuCarnegie Mellon University Silicon Valley, NASA Ames Research Center
15:30 - 16:00
Talk
An Empirical Analysis of Blind Tests
Research Papers
Kesina BaralGeorge Mason University, Jeff OffuttGeorge Mason University
Link to publication DOI
16:00 - 16:30
Talk
A Family of Experiments to Assess the Impact of Page Object Pattern in Web Test Suite Development
Research Papers
Maurizio LeottaUniversità di Genova, Matteo BiagiolaFondazione Bruno Kessler (FBK), Filippo RiccaUniversità di Genova, Mariano CeccatoUniversity of Verona, Paolo TonellaUniversità della Svizzera Italiana (USI)
Link to publication DOI
16:30 - 17:00
Talk
Human-In-The-Loop Automatic Program Repair
Research Papers
Marcel BöhmeMonash University, Australia, Charaka GeethalMonash University, Van-Thuan PhamMonash University
Link to publication DOI

Tue 27 Oct
Times are displayed in time zone: Greenwich Mean Time : Lisbon change

02:30 - 04:00: RT9 - Empirical Analysis and Program RepairResearch Papers at Farfetch (D. Maria)
Chair(s): Corina S. PasareanuCarnegie Mellon University Silicon Valley, NASA Ames Research Center
02:30 - 03:00
Talk
An Empirical Analysis of Blind Tests
Research Papers
Kesina BaralGeorge Mason University, Jeff OffuttGeorge Mason University
Link to publication DOI
03:00 - 03:30
Talk
A Family of Experiments to Assess the Impact of Page Object Pattern in Web Test Suite Development
Research Papers
Maurizio LeottaUniversità di Genova, Matteo BiagiolaFondazione Bruno Kessler (FBK), Filippo RiccaUniversità di Genova, Mariano CeccatoUniversity of Verona, Paolo TonellaUniversità della Svizzera Italiana (USI)
Link to publication DOI
03:30 - 04:00
Talk
Human-In-The-Loop Automatic Program Repair
Research Papers
Marcel BöhmeMonash University, Australia, Charaka GeethalMonash University, Van-Thuan PhamMonash University
Link to publication DOI