Browse Definitions :
Definition

flaky test

Contributor(s): Eric Avidon

A flaky test is an analysis of web application code that fails to produce the same result each time the same analysis is run. Whenever new code is written to develop or update computer software, a web page or an app, it needs to be tested throughout the development process to make sure the application does what it’s supposed to do when it’s released for use. Logically, when put through the same test over and over, the code will produce the same result -- the application will either work properly every time, thus passing the test, or fail to work properly every time, thus failing the test.

However, seemingly at random, occasionally the same test of the same code will produce different results. Sometimes it will show that the code passed the test and the application worked as planned, and sometimes it will show that the code failed the test and didn’t work as planned. When the test fails to produce a consistent result, the test is deemed flaky.

Flaky tests can be caused by various factors:

  • an issue with the newly-written code
  • an issue with the test itself
  • some external factor compromising the test results

Once a test is deemed flaky, there are different approaches to dealing with the muddled results. Some developers will ignore the flakiness entirely, assuming that the issue is with the test and not with the newly-written code. Others will rerun their test multiple times and only go back to investigate further if the test fails a certain number of times in a row, indicating to them a true failure.

However, the safest approach -- the only way to truly find out whether there is a bug in the code -- is to halt the development of the application, fully investigate the cause of the flaky test and resolve it. If left unresolved and there truly is an issue with the code, one problem has the potential to wind up leading to another and another as more is built onto the faulty code.

When investigating the cause of a flaky test, the developer will need to gather data to try to discover differences within the seemingly random results in order to isolate the cause of the failed tests. The code should be re-examined, as should the test itself, and if no issues are found then external factors will need to be looked at to see if they might be at the core of the problem. The developer might look at whether the tests that passed were run at a certain time of day whereas the ones that failed were run at a different time of day, whether certain programs were running on the developer’s computer at the same time of failed tests that weren’t running when the tests passed or whether the tests that failed did so at the same point in the test or at different times during the test.

Sometimes, the cause of the flaky test is simple to diagnose and can be quickly fixed. That’s the best-case scenario. Other times, there is no easy fix, and though potentially costly and time-consuming, the developer may need to delete the test and rewrite it from scratch in order to ensure the accuracy of the test results.

Unfortunately, flaky tests are not uncommon -- Google, for example, reports that 16 percent of its tests show some level of flakiness. They can bring production to a temporary standstill, but they can be dealt with, and they can be resolved.

This was last updated in May 2019

Continue Reading About flaky test

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.

-ADS BY GOOGLE

File Extensions and File Formats

Powered by:

SearchCompliance

SearchSecurity

  • cybersecurity

    Cybersecurity is the protection of internet-connected systems, including hardware, software and data, from cyberattacks.

  • asymmetric cryptography (public key cryptography)

    Asymmetric cryptography, also called public key cryptography, uses a pair of numerical keys that are mathematically related to ...

  • digital signature

    A digital signature is a mathematical technique used to validate the authenticity and integrity of a message, software or digital...

SearchHealthIT

SearchDisasterRecovery

  • business continuity plan (BCP)

    A business continuity plan (BCP) is a document that consists of the critical information an organization needs to continue ...

  • disaster recovery team

    A disaster recovery team is a group of individuals focused on planning, implementing, maintaining, auditing and testing an ...

  • cloud insurance

    Cloud insurance is any type of financial or data protection obtained by a cloud service provider. 

SearchStorage

  • hard disk drive (HDD)

    A computer hard disk drive (HDD) is a non-volatile memory hardware device that controls the positioning, reading and writing of ...

  • byte

    In most computer systems, a byte is a unit of data that is eight binary digits long. Bytes are often used to represent a ...

  • network-attached storage (NAS)

    Network-attached storage (NAS) is dedicated file storage that enables multiple users and heterogeneous client devices to retrieve...

Close