Can AI detect flaky tests before they cause problems? Can it optimize test execution automatically? 🤔
In modern test automation, one of the biggest challenges QA engineers face is dealing with flaky tests—tests that randomly pass or fail due to unstable environments, timing issues, or dependencies.
💡 What if AI could analyze your test logs and predict which tests are flaky?
In this article, I’ll show you how we can use AI and machine learning (ML) to analyze test logs, predict flaky tests, and optimize test execution—all using Java! 🚀
🔍 What Are Flaky Tests & Why Are They a Problem?
A flaky test is one that fails intermittently without any code changes.
Common Causes of Flaky Tests:
❌ Timing issues – Network latency, slow responses
❌ Environment dependencies – Test relies on external systems
❌ Concurrency problems – Tests run in parallel but affect each other
❌ Test data inconsistencies – Data changes across runs
Flaky tests waste CI/CD pipeline resources and make debugging painful. But AI can help detect and fix them! 🤖
⚡ AI-Powered Test Log Analysis (Java POC)
We’ll use Java + OpenAI’s GPT API to analyze test execution logs and detect flaky tests.
🛠️ Tools & Technologies Used:
✔ Java 11+
✔ JUnit / TestNG Logs
✔ OpenAI GPT API (for AI-based log analysis)
✔ Jackson (JSON Parsing Library)
📝 Step 1: Collect Test Execution Logs
[Test] Running LoginTest
[INFO] Test Started: Login with valid credentials
[INFO] Assertion Passed: User redirected to Dashboard
[INFO] Test Ended: Passed ✅
[Test] Running CheckoutTest
[INFO] Test Started: Checkout with expired credit card
[ERROR] Assertion Failed: Payment declined
[INFO] Test Retried: Attempt 1
[INFO] Test Retried: Attempt 2
[ERROR] Test Failed After 2 Retries ❌
🧐 AI can analyze these logs, detect patterns, and highlight flaky tests!
📝 Step 2: Send Logs to AI for Analysis (Java Code)
import java.io.IOException;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.Scanner;
import org.json.JSONObject;
public class AIFlakyTestAnalyzer {
private static final String API_KEY = "your_openai_api_key";
private static final String OPENAI_URL = "https://api.openai.com/v1/chat/completions";
public static void main(String[] args) throws IOException {
String testLogs = """
[Test] Running LoginTest
[INFO] Test Started: Login with valid credentials
[INFO] Assertion Passed: User redirected to Dashboard
[INFO] Test Ended: Passed ✅
[Test] Running CheckoutTest
[INFO] Test Started: Checkout with expired credit card
[ERROR] Assertion Failed: Payment declined
[INFO] Test Retried: Attempt 1
[INFO] Test Retried: Attempt 2
[ERROR] Test Failed After 2 Retries ❌
""";
String analysis = analyzeLogsWithAI(testLogs);
System.out.println("🔍 AI Analysis: \n" + analysis);
}
public static String analyzeLogsWithAI(String logs) throws IOException {
JSONObject requestBody = new JSONObject();
requestBody.put("model", "gpt-4");
requestBody.put("messages", new org.json.JSONArray()
.put(new JSONObject().put("role", "system")
.put("content", "You are an AI test automation expert. Analyze test logs and detect flaky tests."))
.put(new JSONObject().put("role", "user")
.put("content", logs)));
URL url = new URL(OPENAI_URL);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("POST");
conn.setRequestProperty("Authorization", "Bearer " + API_KEY);
conn.setRequestProperty("Content-Type", "application/json");
conn.setDoOutput(true);
conn.getOutputStream().write(requestBody.toString().getBytes());
Scanner scanner = new Scanner(conn.getInputStream());
String response = scanner.useDelimiter("\\A").next();
scanner.close();
JSONObject jsonResponse = new JSONObject(response);
return jsonResponse.getJSONArray("choices").getJSONObject(0).getJSONObject("message").getString("content");
}
}
🎯 Step 3: AI’s Response – Detecting Flaky Tests
When we run this Java program, AI will analyze the logs and detect flaky tests!
Example AI Output:
🔍 AI Analysis:
- Test `CheckoutTest` is identified as a **flaky test** due to multiple retries before failure.
- Possible cause: **Unstable external API or timing issue.**
- Suggested Fix:
- Add `Thread.sleep(2000)` between retries to avoid API rate limiting.
- Validate if the payment API is responsive before retrying.
- Flakiness Score: **High (80%)**
🚀 What’s Next? Optimize Test Execution!
Now that AI detects flaky tests, we can use AI to optimize test execution by:
✅ Skipping unstable tests in CI/CD runs to save build time
✅ Prioritizing stable test cases to execute first
✅ Auto-tuning retry strategies based on AI recommendations
Imagine CI/CD pipelines automatically adjusting execution based on flaky test predictions. 🤯