50: Flaky Tests and How to Deal with Them

Anthony Shaw joins Brian to discuss flaky tests and flaky test suites.

• What are flaky tests?
• Is it the same as fragile tests?
• Why are they bad?
• How do we deal with them?
• What causes flakiness?
• How can we fix them?
• How can we avoid them?
• Proactively rooting out flakiness
• Test design
• GUI tests
• Sharing solutions

Special Guest: Anthony Shaw.

Links:

• Dropbox article on flaky tests
• Microsoft article on flaky tests
• pytest-rerunfailures: a py.test plugin that re-runs failed tests up to -n times to eliminate flakey failures
• pytest-randomly: Pytest plugin to randomly order tests and control random.seed
• pytest-random-order: pytest plugin to randomise the order of tests with some control over the randomness
• math.isclose()
• numpy.isclose()
• pytest.approx() — approx
• Anthony's testing article on RealPython
• Ghost Inspector
• wily: A Python application for tracking, reporting on timing and complexity in tests