Skip to content

Check all unified tests in UTF makefile #1794

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

alcaeus
Copy link
Member

@alcaeus alcaeus commented Apr 30, 2025

This PR does not check any specifications or test files, but it completely revamps the makefile used to test specifications. The change was prompted by me realising that not all folders were properly checked. As a result, I changed the following:

  • Introduced separate targets for each folder containing test files. This creates multiple targets in specifications like the unified test format, e.g. unified-test-format/invalid and unified-test-format/valid-pass. In those cases, I also added a compound target, e.g. unified-test-format, that runs all individual targets
  • Sorted target names alphabetically

The following specifications didn't have their test files checked previously:

  • auth
  • connection-monitoring-and-pooling
  • index-management
  • run-command
  • server-discovery-and-monitoring
  • server-selection

Copy link
Member

@GromNaN GromNaN left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not having a glob pattern for all locations?

Like this:

@ajv test -s $(SCHEMA) -d "../../**/*.yml" --valid

@alcaeus
Copy link
Member Author

alcaeus commented Apr 30, 2025

Why not having a glob pattern for all locations?

There are a number of tests that aren't written in the unified test format. Some are legacy tests waiting to be replaced, others (e.g. some SDAM tests) aren't yet possible in the unified test format and we have no plans to convert them as they test internal functionality that can't easily be tested in the test runners.

@@ -1,48 +1,54 @@
SCHEMA=../schema-1.23.json

.PHONY: all invalid valid-fail valid-pass atlas-data-lake versioned-api load-balancers gridfs transactions transactions-convenient-api crud collection-management read-write-concern retryable-reads retryable-writes sessions command-logging-and-monitoring client-side-operations-timeout HAS_AJV
.PHONY: all atlas-data-lake auth change-streams client-side-encryption client-side-operations-timeout collection-management command-logging-and-monitoring command-logging-and-monitoring/logging command-logging-and-monitoring/monitoring connection-monitoring-and-pooling connection-monitoring-and-pooling/logging crud gridfs index-management load-balancers read-write-concern retryable-reads retryable-writes run-command server-discovery-and-monitoring server-selection server-selection/logging sessions transactions-convenient-api transactions unified-test-format unified-test-format/invalid unified-test-format/valid-fail unified-test-format/valid-pass versioned-api HAS_AJV
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[optiona] split target list for readability?

Suggested change
.PHONY: all atlas-data-lake auth change-streams client-side-encryption client-side-operations-timeout collection-management command-logging-and-monitoring command-logging-and-monitoring/logging command-logging-and-monitoring/monitoring connection-monitoring-and-pooling connection-monitoring-and-pooling/logging crud gridfs index-management load-balancers read-write-concern retryable-reads retryable-writes run-command server-discovery-and-monitoring server-selection server-selection/logging sessions transactions-convenient-api transactions unified-test-format unified-test-format/invalid unified-test-format/valid-fail unified-test-format/valid-pass versioned-api HAS_AJV
.PHONY: all \
atlas-data-lake \
auth \
change-streams \
client-side-encryption \
client-side-operations-timeout \
collection-management \
command-logging-and-monitoring \
command-logging-and-monitoring/logging \
command-logging-and-monitoring/monitoring \
connection-monitoring-and-pooling \
connection-monitoring-and-pooling/logging \
crud \
gridfs \
index-management \
load-balancers \
read-write-concern \
retryable-reads \
retryable-writes \
run-command \
server-discovery-and-monitoring \
server-selection \
server-selection/logging \
sessions \
transactions-convenient-api \
transactions \
unified-test-format \
unified-test-format/invalid \
unified-test-format/valid-fail \
unified-test-format/valid-pass \
versioned-api \
HAS_AJV


all: invalid valid-fail valid-pass atlas-data-lake versioned-api load-balancers gridfs transactions transactions-convenient-api change-streams crud collection-management read-write-concern retryable-reads retryable-writes sessions command-logging-and-monitoring client-side-operations-timeout client-side-encryption
all: atlas-data-lake auth change-streams client-side-encryption client-side-operations-timeout collection-management command-logging-and-monitoring connection-monitoring-and-pooling crud gridfs index-management load-balancers read-write-concern retryable-reads retryable-writes run-command server-discovery-and-monitoring server-selection sessions transactions-convenient-api transactions unified-test-format versioned-api
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[optional] split the target list for readability

Suggested change
all: atlas-data-lake auth change-streams client-side-encryption client-side-operations-timeout collection-management command-logging-and-monitoring connection-monitoring-and-pooling crud gridfs index-management load-balancers read-write-concern retryable-reads retryable-writes run-command server-discovery-and-monitoring server-selection sessions transactions-convenient-api transactions unified-test-format versioned-api
all: atlas-data-lake \
auth \
change-streams \
client-side-encryption \
client-side-operations-timeout \
collection-management \
command-logging-and-monitoring \
connection-monitoring-and-pooling \
crud \
gridfs \
index-management \
load-balancers \
read-write-concern \
retryable-reads \
retryable-writes \
run-command \
server-discovery-and-monitoring \
server-selection \
sessions \
transactions-convenient-api \
transactions \
unified-test-format \
versioned-api

@ajv test -s $(SCHEMA) -d "invalid/*.yml" --invalid > /dev/null && echo "invalid/*.yml passed test"
# Keep specifications sorted alphabetically
# When adding a new specification, remember to add it to the all and .PHONY targets above
# For specifications that contain
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For specifications that contain

Was information lost?

transactions: HAS_AJV
@ajv test -s $(SCHEMA) -d "../../transactions/tests/unified/*.yml" --valid

unified-test-format: unified-test-format/invalid unified-test-format/valid-fail unified-test-format/valid-pass
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The .PHONY all target already calls unified-test-format/invalid unified-test-format/valid-fail unified-test-format/valid-pass individually. Should we remove this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants