What Is a Flaky Test? A Developer's Guide

flaky-testsci-cdtestinggithub-actions

A flaky test is a test that sometimes passes and sometimes fails without any changes to the code being tested. They're one of the most frustrating problems in software development.

Why Flaky Tests Matter

Flaky tests erode trust in your test suite. When developers see tests fail randomly, they start ignoring failures — including real ones.

The Cost of Flaky Tests

  • Wasted CI minutes — reruns burn compute time
  • Slower deployments — teams wait for green builds
  • Lost confidence — developers stop trusting the suite

Common Causes

1. Timing and Race Conditions

Tests that depend on timing are inherently fragile:

// Flaky — depends on setTimeout precision
await new Promise((resolve) => setTimeout(resolve, 100));
expect(element).toBeVisible();

2. Shared State

Tests that share database state or global variables can interfere with each other:

// Flaky — order-dependent
beforeAll(() => {
  db.seed(testData);
});

3. External Dependencies

Network calls, file system operations, and third-party APIs introduce non-determinism.

How to Detect Flaky Tests

The best approach is automated detection. Tools like TestGlance analyze your CI test results over time and flag tests that flip between pass and fail states.

Detection Algorithm

A simple flaky detection algorithm tracks test status across runs:

  1. Collect test results from multiple CI runs
  2. Compare status of each test across runs
  3. Flag tests that changed status without code changes

Next Steps

Start monitoring your test suite health today. The sooner you identify flaky tests, the sooner your team can trust their CI pipeline again.