On ever big codebase flaky tests are part of the deal, and usually don't have enough attention or even workflow/tools to deal with them.

I wonder why I haven't found any automated test services (CircleCI/Travis/BuildKite/SemaphoreCI/etc...) with some kind of feature/help to deal with those πŸ€”

The trickiest part seems to be how to identify/reference them as lines/files/names change across time & branches.

What I would like from a tool like this would be:
- To know if a build has failed because tests that have been marked as flakies before. Being new to a codebase makes you wonder on every failing spec if its a false positive or not.
- To know which flakies have the most occurrences (and prioritize tackling those over the rest if I have time or just too many)

And I think the best way to interact with the developer would be through an integration with Github/Gitlab/etc... on every PR.

@bertocq My policy if a test is flaky:

1. delete it
2. ask yourself what was it testing?
3. rewrite it

Sign in to participate in the conversation

A Mastodon instance for Rubyists & friends