On ever big codebase flaky tests are part of the deal, and usually don't have enough attention or even workflow/tools to deal with them.
I wonder why I haven't found any automated test services (CircleCI/Travis/BuildKite/SemaphoreCI/etc...) with some kind of feature/help to deal with those 🤔
The trickiest part seems to be how to identify/reference them as lines/files/names change across time & branches.
What I would like from a tool like this would be:
- To know if a build has failed because tests that have been marked as flakies before. Being new to a codebase makes you wonder on every failing spec if its a false positive or not.
- To know which flakies have the most occurrences (and prioritize tackling those over the rest if I have time or just too many)
And I think the best way to interact with the developer would be through an integration with Github/Gitlab/etc... on every PR.
@bertocq My policy if a test is flaky:
1. delete it
2. ask yourself what was it testing?
3. rewrite it
A Mastodon instance for Rubyists & friends