Regression Testing for Software Systems

OpenOnco quality control: testing a closely integrated diagnostics database and codebase: LLM Data Review + UI Regression Testing. OpenOnco grew from prototype to production in about a month: 80+ diagnostic tests, complex filtering, PDF export, comparison tools. 12K lines of code. Manual QA stopped working, fortunately some smart software folks advised us. Here's our system: (1) Multi-LLM Data Verification Before each deploy, I run the full database through Claude, Grok, GPT-5, and Gemini 3. Each model reviews test data for: → Inconsistencies between related tests → Outdated info vs. current clinical guidelines → Missing fields that should be populated → Logical errors (FDA-approved test with no approval date) Different models catch different things. Claude finds logical inconsistencies. GPT-5 catches formatting. Grok flags outdated clinical data. Gemini spots missing cross-references. (2) Automated UI Regression Testing Regression testing: "Did my changes break something that was working?" For us this means testing actual user workflows — clicking buttons, filling forms, navigating between pages — and verifying the interface behaves correctly every time. We test the actual UI, not just components in isolation: → Filter interactions: Click "IVD Kit" filter → verify correct tests appear → click "MRD" category → verify intersection is correct → clear filters → verify all tests return → Test card workflows: Click test card → modal opens with correct data → click "Compare" → test added to comparison → open comparison modal → verify all fields populate → Search behavior: Type "EGFR" → verify matching tests surface → clear search → verify full list returns → Direct URL testing: Navigate to /mrd?test=mrd-1 → verify modal auto-opens with correct test → Navigate to /tds?compare=tds-1,tds-2,tds-3 → verify comparison modal loads with all three → PDF export: Generate comparison PDF → verify page count matches content → verify no repeated pages (caught a real bug where Page 1 rendered on every page) → Mobile responsiveness: Run full suite at 375px, 768px, 1024px, 1440px breakpoints We run these tests using Playwright — an open-source browser automation framework. It launches real browsers (Chrome, Firefox, Safari), executes user actions, and asserts outcomes. Tests run on every push via GitHub Actions; deploy is blocked if anything fails. Full suite takes ~4 minutes 🤯🤯🤯 The combination of LLM data review + real UI regression testing catches what unit tests miss: so far, hundreds of issues 👍🏼👍🏼👍🏼

I’ve never seen a team work this hard… They ran 2,500 tests every month… and found nothing. Four engineers. One massive app. And barely any bugs to show for it. In six months, just three issues. Not one made it to production. On paper, it looked like a win. But something felt off. Releases were slipping. Budgets were tightening. And the team? Stretched thin. That’s when the Application Director called me. We sat down with the business: Claims, Billing, Underwriting. Mapped their workflows. Tracked the risks. And asked a tough question. Why are we testing all of this? We rebuilt the regression suite from scratch: Grouped tests into Platinum, Gold, and Silver Defined when each tier should run (e.g., only Platinum for hotfixes) Automated the Platinum tier, fast, reliable, focused Plugged real coverage gaps Three months later, the numbers told the story: 1. Regression load cut in half 2. QA effort down 60% 3. Defects found: 5x more 4. Still zero production issues They weren’t just testing smarter. They were finally testing what mattered. Sometimes, less testing means better quality. You just have to be brave enough to change the rules. What part of your process feels "safe" but isn't working?

Don’t Focus Too Much On Writing More Tests Too Soon 📌 Prioritize Quality over Quantity - Make sure the tests you have (and this can even be just a single test) are useful, well-written and trustworthy. Make them part of your build pipeline. Make sure you know who needs to act when the test(s) should fail. Make sure you know who should write the next test. 📌 Test Coverage Analysis: Regularly assess the coverage of your tests to ensure they adequately exercise all parts of the codebase. Tools like code coverage analysis can help identify areas where additional testing is needed. 📌 Code Reviews for Tests: Just like code changes, tests should undergo thorough code reviews to ensure their quality and effectiveness. This helps catch any issues or oversights in the testing logic before they are integrated into the codebase. 📌 Parameterized and Data-Driven Tests: Incorporate parameterized and data-driven testing techniques to increase the versatility and comprehensiveness of your tests. This allows you to test a wider range of scenarios with minimal additional effort. 📌 Test Stability Monitoring: Monitor the stability of your tests over time to detect any flakiness or reliability issues. Continuous monitoring can help identify and address any recurring problems, ensuring the ongoing trustworthiness of your test suite. 📌 Test Environment Isolation: Ensure that tests are run in isolated environments to minimize interference from external factors. This helps maintain consistency and reliability in test results, regardless of changes in the development or deployment environment. 📌 Test Result Reporting: Implement robust reporting mechanisms for test results, including detailed logs and notifications. This enables quick identification and resolution of any failures, improving the responsiveness and reliability of the testing process. 📌 Regression Testing: Integrate regression testing into your workflow to detect unintended side effects of code changes. Automated regression tests help ensure that existing functionality remains intact as the codebase evolves, enhancing overall trust in the system. 📌 Periodic Review and Refinement: Regularly review and refine your testing strategy based on feedback and lessons learned from previous testing cycles. This iterative approach helps continually improve the effectiveness and trustworthiness of your testing process.

"We have to run a full regression test suite on every build!" First: you don’t *have* to do anything. There is no law of nature, nor any human regulation, that says you must repeat any particular test. You *choose* to do things. In the Rapid Software Testing namespace, we say that regression testing is *any testing motivated by change to a previously tested product*. When a product changes, risk is not evenly distributed. A brief pause for some analysis of where the changes might affect things can help focus your testing on plausible risk. Test — that is, challenge — the idea that a change had only the desired effects in the area it was made, and didn’t introduce undesirable effects. Test things that might be connected to or influenced by that change. It might make sense to do *some* testing in places where you believe risk is low — to reveal hidden risks. If you want to find problems that matter, though, diversifying your techniques, tools, and tactics is essential. Rote repetition can limit you badly. Obsession with looking for problems in exactly the same way as you've already looked displaces two things: 1) Your ability to find problems that were there all along, but that your testing has missed all along; and 2) Your ability to find new problems introduced by a change that your existing set of tests won’t cover. Don’t fixate on tests you’ve done before. Consider the gap between what you thought you knew about the system before the change, and what you need to know about the product as it is now. It’s the latter that’s the most important bit, and your old tests might not be up to the task. Need help convincing management of this? Let me know.

Week 1: Playwright tests running in CI. Green. Team posts the screenshot in Slack. Month 2: Suite takes 18 minutes. Devs open a PR, context-switch, forget to check results. Some merge before tests finish. Month 3: Designer moves the "Submit" button to a sticky header. Three tests break. An engineer adds // TODO: fix after redesign. They never come back. Month 5: CI is green. But 40% of E2E tests are quietly disabled. The signup flow hasn't been tested in six weeks. A regression ships. A customer emails support. I've watched this exact timeline across dozens of SaaS teams that set up GitHub Actions for automated testing. The setup takes an afternoon. That's the easy part. The hard part is what runs inside the pipeline, and whether it's still running three months later. Most teams I talk to have "automated testing" that only covers unit tests. Their CI passes. Their checkout flow throws a 500 error. That green checkmark means the pipeline executed. It doesn't mean the product works. We wrote a guide on building a GitHub Actions pipeline that doesn't fall apart after month 3. It also covers the point where maintaining Playwright scripts yourself stops being worth the time. Link in comments. #QAtesting #e2etesting #regressiontesting #apptesting #aitesting #ai

Releases break more than just new features. They often break things that used to work. That is why regression testing matters. I put together a 25-Point Regression Testing Strategy Checklist that shows how to: • Prioritize by risk so you are not testing everything blindly • Keep a smoke subset for quick validation before full runs • Refresh and seed test data so results are reliable • Track and fix flaky tests before they poison your regression suite • Integrate regression into CI/CD so issues are caught early Strong regression is not about running more tests. It is about running the right ones consistently. Grab the PDF below and use it to strengthen your regression strategy.

After completing seven projects where developers extensively used AI to write code and fix defects, I came to the conclusion that traditional risk-based testing is no longer enough in the AI era. For years, we have relied on risk-based testing and impact analysis to determine what to test and where to focus our efforts. These practices are still valuable, but they are no longer sufficient on their own. Let me explain you the reason why. AI has fundamentally changed the speed and nature of software development. AI can generate thousands of lines of code in minutes, refactor existing components, modify shared libraries and introduce changes across multiple parts of an application faster than ever before. As a result, the true impact of a small change is becoming increasingly difficult to predict. This has also changed how we should think about regression testing. In the past, we could often rely on focused or targeted regression testing based on the results of impact analysis. Today, that approach is becoming increasingly risky. AI-generated changes can affect components, dependencies and business flows that were NOT directly modified and may not be identified by traditional impact analysis. As a result, comprehensive and intelligent regression testing has become more important than ever. You can no longer assume you have identified every area that could be affected by a change. Historical risk assessments and traditional impact analysis which we used for ages were designed for human-driven development. They were never intended to evaluate software evolving at AI speed. That does NOT mean we should abandon these practices. It means we need to evolve them. In my opinion the future of Quality Engineering requires: a. AI-aware impact analysis that understands AI-generated code and hidden dependencies b. Intelligent regression testing that dynamically adapts based on actual code, data and business process changes c. Continuous production monitoring to validate that software behaves as expected after deployment d. Human expertise/intelligence to challenge assumptions, validate business outcomes and investigate risks AI may overlook In the AI era, quality can no longer rely solely on what changed in the code. We must also understand how those changes affect the business, customers, integrations and production behavior. Traditional risk-based testing has reached its limit. The next generation of Quality Engineering must combine AI, intelligent analytics, continuous validation and human judgment to manage software risk effectively. What's your experience? Has AI made your traditional risk-based testing and impact analysis less reliable, or have you found ways to evolve your approach and keep pace with AI-driven development?