Create a consistency metric for EvalBench. I verified that for this c…

[gigistark-google]

…hange, our test output for datasets/bat/example_run_config.yaml is as expected: DBs Setup: |██████████████████████████████████████████████████| 100.0% Complete

Prompts: |██████████████████████████████████████████████████| 100.0% Complete
SQLGen: |██████████████████████████████████████████████████| 100.0% Complete
SQLExec: |██████████████████████████████████████████████████| 100.0% Complete
Scoring: |██████████████████████████████████████████████████| 100.0% Complete
I0502 17:41:42.837790 140424820953600 report.py:25] Total Prompts: 32.
I0502 17:41:42.841057 140424820953600 report.py:43] Prompt Errors: 0.
I0502 17:41:42.842391 140424820953600 report.py:45] SQLGen Errors: 0.
I0502 17:41:42.843745 140424820953600 report.py:46] SQLExec Gen Errors: 0.
I0502 17:41:42.844884 140424820953600 report.py:48] Golden Errors: 0.
I0502 17:41:42.856288 140424820953600 analyzer.py:104] exact_match: 2/32 = 6.25%
I0502 17:41:42.859278 140424820953600 analyzer.py:104] llmrater: 14/32 = 43.75%
I0502 17:41:42.862122 140424820953600 analyzer.py:104] returned_sql: 32/32 = 100.0%
I0502 17:41:42.865184 140424820953600 analyzer.py:104] set_match: 9/32 = 28.12%
I0502 17:41:42.868373 140424820953600 analyzer.py:104] executable_sql: 32/32 = 100.0%
I0502 17:41:42.878332 140424820953600 analyzer.py:104] exact_match_consistency: 8/16 = 50.0%
I0502 17:41:42.881638 140424820953600 analyzer.py:104] llm_consistency: 13/16 = 81.25%
I0502 17:41:42.890740 140424820953600 analyzer.py:104] executable: 32/32 = 100.0%

TAG=agy
CONV=9bb5d876-3619-4349-ab50-9187bda4aed2

[IsmailMehdi]

great work, thank you.
Could you remove the format only chaned files:
evalbench/databases/mongodb.py
evalbench/databases/mysql.py
evalbench/databases/postgres.py
evalbench/databases/spanner.py
evalbench/databases/sqlite.py
evalbench/dataset/dataset.py
evalbench/eval_service.py
evalbench/evaluator/streamingorchestrator.py
evalbench/generators/models/claude_code.py
evalbench/generators/models/gemini.py
evalbench/generators/models/gemini_cli.py
evalbench/generators/models/query_data_api.py
evalbench/scorers/examples/sample_python_validator.py
evalbench/scorers/llmrater.py
evalbench/scorers/score.py
evalbench/scorers/skillsbestpractices.py
evalbench/test/mongodb_test.py
evalbench/test/query_data_api_test.py
evalbench/test/test_db_cleanups.py
evalbench/test/test_spanner_cleanup.py
evalbench/test/trajectory_matcher_test.py
evalbench/util/fake_mcp_server.py
evalbench/util/scriptrunner.py
evalbench/util/sessionmgr.py

[IsmailMehdi]

see the pycodestyle we use: https://github.com/GoogleCloudPlatform/evalbench/blob/main/.pycodestyle

[IsmailMehdi]

the please resolve conflicts

[gigistark-google]

Thanks! I removed the format only chained files and verified that all 18 remaining files are now perfectly formatted according to the repository's .pycodestyle rules.

[IsmailMehdi]

thanks, could you look into these 2 issues:

1. evalbench/eval_service.py

• Line 182: Needs to unpack 5 return values instead of 4.
• Line 240+: The _process_results helper needs to load/append multi_trial_scores_tf and pass num_prompts/num_trials down to the
  analyzer just like evalbench.py now does.

2. evalbench/evaluator/streamingorchestrator.py

This orchestrator was completely missed and is currently broken:

• Line 122: evaluator.evaluate(...) now returns 3 variables, but this line only captures 2, which will raise a ValueError immediately upon
  execution.
• Line 184: Its process() method needs to be updated to return the 5th tuple member (multi_trial_scores_tf) so that it conforms to the
  base interface used by callers.

[gigistark-google]

Restricted num trials to dql and addressed the eval_service.py and streamingorchestrator.py issues. I am relatively rusty on GitHub. Curious what tests you ran to detect the eval_service.py and streamingorchestrator.py issues and how in the future, I can run these before sending for review.

[IsmailMehdi]

Restricted num trials to dql and addressed the eval_service.py and streamingorchestrator.py issues. I am relatively rusty on GitHub. Curious what tests you ran to detect the eval_service.py and streamingorchestrator.py issues and how in the future, I can run these before sending for review.

Restricted num trials to dql and addressed the eval_service.py and streamingorchestrator.py issues. I am relatively rusty on GitHub. Curious what tests you ran to detect the eval_service.py and streamingorchestrator.py issues and how in the future, I can run these before sending for review.

Running the GRPC mode using a local docker and testing with g3 datasets will show the issue. But i am familiar enough with the framework to figure out issues like that.

[IsmailMehdi]

/gcbrun

[IsmailMehdi]

Great work