Skip to content

Test Case 3: Persistent Test Failure - #880

Open
oxve wants to merge 17 commits into
mainfrom
test-case-3-deflake-fail
Open

oxve wants to merge 17 commits into
mainfrom
test-case-3-deflake-fail

Conversation

@oxve

@oxve oxve commented Aug 28, 2026 •

Copy link
Copy Markdown
Member

Verifying persistent test failure stops after max retries.

Bug: 551996294

oxve added 7 commits August 28, 2026 16:45
- Update junit_mini_parser.py to output failing tests as structured JSON.
- Update junit_mini_parser_test.py with full test coverage and typing annotations.
- Update main.yaml and process_test_results to parse JSON output using jq.

Tag: agy
Conv: fb32e968-5c28-4711-9b18-47b6c44fd1d0
Update process_test_results action to use set -euo pipefail and improve quoting and failure output parsing.

Tag: agy
Conv: 497490ac-8ae1-409a-88cb-30bc08e7ffe6
Bug: 551996294
Remove error handling and empty checks from junit_mini_parser.py.
Simplify test_failures.json parsing in process_test_results and main.yaml workflows to use a unified jq query without empty checks.

Tag: agy
Conv: ef023ce0-0476-4a39-82ad-153794531729
Bug: 546722232
Restore the command-line arguments check and logging configuration in the entry point of junit_mini_parser.py as requested.

Tag: agy
Conv: ef023ce0-0476-4a39-82ad-153794531729
Bug: 546722232
Update process_test_results and main.yaml to check if the first XML file exists before calling junit_mini_parser.py.
This prevents crashes when no XML files match the glob and nullglob is not enabled.

Tag: agy
Conv: ef023ce0-0476-4a39-82ad-153794531729
Bug: 546722232
…evice tests

Implement test retry logic in GitHub Actions for on-host and on-device
platforms. This introduces a mechanism to track and retry specific
failures in PRs while allowing full retries on push events.

Update test filtering and reporting to support automated generation of retry
filters and consolidated test reporting. This reduces manual developer
intervention for flaky tests and optimizes data uploads to external
monitoring tools by merging result sets.

Tag: agy
Conv: b0702142-38ef-440d-acc8-a87a139ea206
Bug: 546722232
Move test_filter.py and test_filter_test.py to cobalt/devinfra/github.
Update references in main.yaml, on_host_tests, and on_device_tests workflows.
Simplify generate_retry_filter action by removing unnecessary empty check.

Tag: agy
Conv: ef023ce0-0476-4a39-82ad-153794531729
Bug: 546722232
@github-actions

github-actions Bot commented Aug 28, 2026 •

Copy link
Copy Markdown
Contributor
Context State Description
CI / android ✅ PASS Passed
CI / linux ❌ FAIL Failed
@github-actions

Copy link
Copy Markdown
Contributor

🤖 Gemini Suggested Commit Message


Error: Could not parse a valid response from the Gemini API. Please check the API response logs in the workflow run.

💡 Pro Tips for a Better Commit Message:

  1. Influence the Result: Want to change the output? You can write custom prompts or instructions directly in the Pull Request description. The model uses that text to generate the message.
  2. Re-run the Generator: Post a comment with: /generate-commit-message

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request replaces several real GitHub Actions steps—including GN generation, Ninja builds, browser tests, on-host tests, and artifact archiving—with mocked steps using a newly introduced helper script mock_step.py and a configuration file mock_config.json. The feedback focuses on improving the robustness of the Python script, specifically handling empty directory paths in os.makedirs, catching potential regular expression errors when matching job patterns, using os.path.splitext to safely extract filenames with multiple dots, and validating that parsed test targets are indeed lists to prevent runtime type errors.

failure.text = 'Mocked failure details'

tree = ET.ElementTree(root)
os.makedirs(os.path.dirname(filepath), exist_ok=True)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If filepath has no directory component (e.g., it is just a filename), os.path.dirname(filepath) returns an empty string. Calling os.makedirs("", exist_ok=True) raises a FileNotFoundError. To prevent this, only call os.makedirs when the directory path is non-empty.

Suggested change
os.makedirs(os.path.dirname(filepath), exist_ok=True)
dir_name = os.path.dirname(filepath)
if dir_name:
os.makedirs(dir_name, exist_ok=True)
Comment on lines +40 to +46
def find_rule(config, job_name):
rules = config.get('rules', [])
for rule in rules:
pattern = rule.get('job_pattern')
if pattern and re.match(pattern, job_name):
return rule
return None

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If job_pattern in the configuration file contains an invalid regular expression, re.match raises a re.error exception, which crashes the script. Wrapping the match in a try-except block prevents crashes from malformed configuration patterns.

Suggested change
def find_rule(config, job_name):
rules = config.get('rules', [])
for rule in rules:
pattern = rule.get('job_pattern')
if pattern and re.match(pattern, job_name):
return rule
return None
def find_rule(config, job_name):
rules = config.get('rules', [])
for rule in rules:
pattern = rule.get('job_pattern')
if pattern:
try:
if re.match(pattern, job_name):
return rule
except re.error as e:
print(f'Invalid regex pattern "{pattern}": {e}')
return None
for target_path in targets:
filename = os.path.basename(target_path)
# Remove extension if any (like .exe or run_ prefix)
test_name = filename.split('.')[0]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using filename.split('.')[0] to remove the file extension fails if the filename contains multiple dots (e.g., test.suite.name). It is safer to use os.path.splitext to strip only the final extension.

Suggested change
test_name = filename.split('.')[0]
test_name = os.path.splitext(filename)[0]
Comment on lines +102 to +107
targets = []
if args.test_targets:
try:
targets = json.loads(args.test_targets)
except Exception as e: # pylint: disable=broad-except
print(f'Error parsing test targets: {e}')

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If args.test_targets is parsed as a non-list JSON type (such as a dictionary, integer, or boolean), iterating over targets later in the script raises a TypeError or yields unexpected keys. Validating that the parsed object is a list of strings prevents runtime errors.

Suggested change
targets = []
if args.test_targets:
try:
targets = json.loads(args.test_targets)
except Exception as e: # pylint: disable=broad-except
print(f'Error parsing test targets: {e}')
targets = []
if args.test_targets:
try:
parsed = json.loads(args.test_targets)
if isinstance(parsed, list):
targets = [str(t) for t in parsed]
else:
print(f'Warning: test targets is not a list: {parsed}')
except Exception as e: # pylint: disable=broad-except
print(f'Error parsing test targets: {e}')
oxve added 5 commits August 31, 2026 18:41
- Add cobalt/devinfra to CI_ESSENTIALS in main.yaml.
- Rename validation rollup jobs to ${{ matrix.name }}_validation and tvos_validation to avoid erroneous retries from shepherd/deflake workflows.
- Fix bash parameter expansion for test_failures in main.yaml validation step.
- Sanitize colon prefixes in test target names across test_filter.py and actions.
- Support dictionary-formatted and string-formatted test targets in on_device_tests and on_host_tests actions and on_device_tests_gateway_client.py.
- Update test_filter_test.py with coverage for colon-prefixed target names.

Tag: agy
Conv: a62819b8-a42e-437a-b0f5-972911759cd8
Bug: 546722232
Tag: agy
Conv: 6ba17f35-7b52-48e7-a275-931ef58d3366
Bug: 546722232
Fix formatting violation in cobalt/devinfra/github/test_filter_test.py
to satisfy pre-commit yapf hook check.

Tag: agy
Conv: f5adcc09-6b58-4e6a-9594-32e4a55670ba
Bug: 546722232
…ion reporting

Tag: agy
Conv: a62819b8-a42e-437a-b0f5-972911759cd8
Bug: 551996294
Add mock_step.py and actions/workflow mocking configuration to execute simulated fast builds and tests on hosted runners for CI Shepherd verification.

Tag: agy
Conv: a62819b8-a42e-437a-b0f5-972911759cd8
Bug: 551996294
@oxve
oxve force-pushed the test-case-3-deflake-fail branch from 0988a97 to 1b68631 Compare August 31, 2026 19:29
Mock .github/actions/checkout and .github/actions/depot_tools to skip multi-gigabyte repo cloning and tool bootstrapping in sandbox verification testing.

Tag: agy
Conv: a62819b8-a42e-437a-b0f5-972911759cd8
Bug: 551996294
@oxve
oxve force-pushed the test-case-3-deflake-fail branch from 1b68631 to 2607f72 Compare August 31, 2026 19:49
oxve added 2 commits August 31, 2026 19:49
Configure deflake.json with deflake_runs=2 and fail test on shard 1 through attempt 2 to verify custom max attempts limit.

Tag: agy
Conv: a62819b8-a42e-437a-b0f5-972911759cd8
Bug: 551996294
Ensure checkout action copies CI_ESSENTIALS into cobalt/src and build action populates test targets and dummy APK artifacts without network calls.

Tag: agy
Conv: a62819b8-a42e-437a-b0f5-972911759cd8
Bug: 551996294
oxve added 2 commits September 1, 2026 19:47
Tag: agy
Conv: a62819b8-a42e-437a-b0f5-972911759cd8
Bug: 551996294
Fail shard 1 persistently on all attempts to verify max retries exhaustion.

Tag: agy
Conv: a62819b8-a42e-437a-b0f5-972911759cd8
Bug: 551996294
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

1 participant