ci: improve robustness of functional tests #184

eeaton · 2025-01-09T13:11:11Z

I identified an issue in the regexp for functional test checking for the result of airflow dag statuses. I had a syntax issue in regexp that meant it was greedily matching too many characters, and would pass a test even when the status was "fail". This investigation also led to a few other improvements:

address permission issues that are different between the testing environment (Classifier in different project) vs a customer environment
Improve reusability of code for adding more tests in the future
Improved debugging and error handling of tests
GCP project ID is now tied to the build ID, not commit ID, meaning that if CI fails for flaky reasons, the build can be retriggered from Github checks interface, instead of requiring an empty commit to force a trigger with a new ID.

…n Cloud Build trigger substitution for CI automation. I clarified the documentation and made manual changes to the trigger outside of this PR.

… ci projects

… central cicd project. Reduce the steps so I can run just the functional test with subsitution from Cloud Build

…GoogleCloudPlatform/enterprise-knowledge-solution into ci-functional-test-improvements

…e out and cause very slow tests. Subsequent retries are no more likely to succeed than the first try.

… examples that work seem to be 9m, so it the current 10m timeout is probably too aggressive

…onal test

…which is painful to modify and troubleshoot. 1: move sleep and retry logic into functional_test.go. 2: separate steps to prepare_functional and do_functional 3: separate helper script for adding cross-project IAM role to classifier service account, referencing terraform outputs

…iggered after a flaky failure

…lang code to exit instead of catching the error

…ry a possible error in the command

…me cmd object will fail on subsequent attempts.

and TestDAGIsSuccess

…rt.EventuallyWith is a built-in wrapper that helps handle this use case; the previous method of manually repeating assert in loops doesn't work well for polling because a single failed assert method will fail the whole test.

… even if the previous stage TestDagIsAvailable has passed. Might be another propagation issue

…INTJ.pdf HLXB.pdf that may trigger to timeout

… 35+ of more which was at the high end of my timers

eyalbenivri

LGTM

correct issue with regexp in test condition that meant everything passed

465a44e

eeaton marked this pull request as draft January 9, 2025 13:33

eeaton changed the title ~~CI: correct issue with regexp in functional tests~~ ci: correct issue with regexp in functional tests Jan 9, 2025

Input variables for custom_classifier_id were not the correct value i…

4a1741b

…n Cloud Build trigger substitution for CI automation. I clarified the documentation and made manual changes to the trigger outside of this PR.

eeaton marked this pull request as ready for review January 10, 2025 10:36

eeaton added 25 commits January 10, 2025 15:32

empty commit to retrigger build after flaky Composer error

3d96d1a

retrigger tests for debugging

2543ff7

Merge branch 'main' into ci-functional-test-improvements

3acbcdf

empty commit to retrigger tests after flaky Composer error

01b9b07

empty commit

577dc08

empty commit to retrigger tests after tweaking manual IAM settings in…

d1210e9

… ci projects

retrigger after fixing DocAI settings

8523ca0

add IAM role Cloud Run Job in test project to use Doc AI Processor in…

2532ad4

… central cicd project. Reduce the steps so I can run just the functional test with subsitution from Cloud Build

Merge branch 'ci-functional-test-improvements' of https://github.com/…

7187303

…GoogleCloudPlatform/enterprise-knowledge-solution into ci-functional-test-improvements

format for cloud build variable substituion

4e9f5d4

reduce retries on specialized_parser_processor_job, this tends to tim…

38191d1

…e out and cause very slow tests. Subsequent retries are no more likely to succeed than the first try.

syntax for cloud build variable substitution

759e3af

syntax for cloud build variable subsitution

1fd1669

syntax for cloud build variable subsitution

d830957

try a much longer timeout for self.processor_config.timeout. The rare…

778b825

… examples that work seem to be 9m, so it the current 10m timeout is probably too aggressive

split up the steps between preparing functional test and doing functi…

c42148d

…onal test

add uuid to gcp project names so that they build can be manually retr…

05cb569

…iggered after a flaky failure

fix recursive copy command for ingest_test_docs.sh

0c0fb78

fix reference to changed script filename

00191d1

fix typo

a49266b

suppress interactive prompt about IAM conditions

9697b05

try loosening -o errexit setting in script, this might cause the go…

0f2208c

…lang code to exit instead of catching the error

add retry to TestDAGIsTriggered

434ac78

remove use of log.fatal in golang, because I want to tolerate and ret…

df28065

…ry a possible error in the command

eeaton added 2 commits January 16, 2025 13:41

Create a new instance of cmd for each retry, otherwise reusing the sa…

1b9acdf

…me cmd object will fail on subsequent attempts.

separate tests for TestDAGIsComplete

7ad0155

and TestDAGIsSuccess

eeaton changed the title ~~ci: correct issue with regexp in functional tests~~ ci: improve robustness of functional tests Jan 16, 2025

eeaton added 14 commits January 17, 2025 10:43

improve message on retry

99ad509

manually reconfigure test docs in cicd_project source

6309b4e

modify the retry logic to handle failures better

6a53217

modify the retry logic to handle failures better

e9b9a09

troubleshooting trigger_workflow.sh. Sometimes this fails to trigger,…

41584be

… even if the previous stage TestDagIsAvailable has passed. Might be another propagation issue

improve evaluation logic to fail sooner if DAG fails

c195d70

retrigger after reducing a few particularly large invoices RYDE.pdf, …

59793c5

…INTJ.pdf HLXB.pdf that may trigger to timeout

fix typo

985a04e

extend the retryInterval and retries for functional_test, it can take…

99f51d3

… 35+ of more which was at the high end of my timers

rename some stages to be more intuitive/clear

9709b3c

cleanup golang conventions

3e08803

revert changes made to specialized parser as part of troubleshooting

5582ba6

typo

51abc6a

eyalbenivri approved these changes Jan 22, 2025

View reviewed changes

eeaton merged commit 67a8d3b into main Jan 22, 2025
32 checks passed

eeaton deleted the ci-functional-test-improvements branch January 22, 2025 11:50

eeaton mentioned this pull request Jan 22, 2025

feat: composer 3 serverless upgrade #182

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: improve robustness of functional tests #184

ci: improve robustness of functional tests #184

eeaton commented Jan 9, 2025 •

edited

Loading

eyalbenivri left a comment

ci: improve robustness of functional tests #184

ci: improve robustness of functional tests #184

Conversation

eeaton commented Jan 9, 2025 • edited Loading

eyalbenivri left a comment

Choose a reason for hiding this comment

eeaton commented Jan 9, 2025 •

edited

Loading