Introducing flaky - a nose test plugin for automatically rerunning flaky tests

Have you ever been tempted to do this?

Done work item: Remove failing test

We've all had tests that fail; worse, we've had tests that fail only occasionally. Often, the solution is to fix the test, but sometimes the sporadic failures originate from a component that can't be fixed.  Now you've got a flaky test - one that fails, usually infrequently, for reasons other than an error in the code under test. This situation is truly something to avoid, because being able to trust test results is paramount to maintaining high code quality.

Introducing @flaky

When testing Sync 4, Box's desktop sync application, we also ran into this issue, but we also didn't want to simply remove our flaky tests. When we noticed that most flaky tests would pass when rerun, we realized we could make doing so automatic. Flaky is a nose plugin that can rerun flaky tests without interrupting your test run. Using it is as easy as decorating your test methods with @flaky:

[python]

@flaky
def test_something_that_usually_passes(self):
    value_to_double = 21
    result = get_result_from_flaky_doubler(value_to_double)
    self.assertEqual(result, value_to_double * 2, 'Result doubled incorrectly.')

[/python]

By default, flaky will automatically rerun any test marked flaky an additional time if (and only if) it fails. The maximum number of times to rerun, as well as the minimum number of times the test must pass, is configurable.

[python]

@flaky(max_runs=3, min_passes=2)
def test_something_that_usually_passes(self):
    """This test must pass twice, and it can be run up to three times."""
    value_to_double = 21
    result = get_result_from_flaky_doubler(value_to_double)
    self.assertEqual(result, value_to_double * 2, 'Result doubled incorrectly.')

[/python]

Installation, Use, and Code (oh my!)

Installation

flaky is available on PyPI, so installation is as simple as

[shell]

pip install flaky

[/shell]

Importing @flaky()

Once flaky is installed, simply import the flaky decorator in your test classes:

[python]

from box.test.flaky import flaky

[/python]

Running the tests

Finally, once your flaky tests have been appropriately decorated with @flaky(), simply pass a new argument to the nose test runner.

[shell]

nosetests --with-flaky

[/shell]

Check out the code

flaky is available on GitHub:  https://github.com/box/flaky. Check it out, and enjoy!