How to handle flaky tests?
Leankoala provides three solutions for flakey tests:
- Alert n in a row
The alerting mechanism of Leankoala can be configured to triggering an alert solely if the error occurs a specific number of times (=n) in a row. The alert will not be sent upon previous detections of errors, but you will be able to see the failures on your status dashboard.
- Non-Strict mode
The default behaviour of Leankoala is the display of every failure on the dashboard. However, when the non-strict mode is activated a failure must occur at least twice in a row to be a real failure. Please also read "What’s the non-strict mode?" for more information on this configuration.
- Tool Configuration
If the non-strict mode is not enough or too much effort to configure it is possible to set that option directly within the tool. This option can be found under „Configuration > Tools“. Just activate the overwrite switch and enter the number of "errors in a row". All checks from this tool in that project will now follow that same rule.