Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: schemachange/mixed-versions failed #133339

Closed
cockroach-teamcity opened this issue Oct 24, 2024 · 14 comments · Fixed by #134407
Closed

roachtest: schemachange/mixed-versions failed #133339

cockroach-teamcity opened this issue Oct 24, 2024 · 14 comments · Fixed by #134407
Assignees
@spilchen
Labels
branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 branch-release-24.2 Used to mark GA and release blockers, technical advisories, and bugs for 24.2 branch-release-24.3 Used to mark GA and release blockers, technical advisories, and bugs for 24.3 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Oct 24, 2024 •
edited by cockroach-jira-scripts
Loading

roachtest.schemachange/mixed-versions failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

(mixedversion.go:732).Run: mixed-version test failure while running step 42 (run "run schemachange workload and validation in mixed version"): full command output in run_101745.452827211_n4_COCKROACHRANDOMSEED1.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-43548
@cockroach-teamcity cockroach-teamcity added branch-release-24.3 Used to mark GA and release blockers, technical advisories, and bugs for 24.3 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) labels Oct 24, 2024
@exalate-issue-sync exalate-issue-sync bot removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Oct 24, 2024
This was referenced Oct 25, 2024
Open
Closed
@cockroach-teamcity
Copy link
Member Author

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.schemachange/mixed-versions failed with artifacts on release-24.3 @ 3839fb143f98e5009a994ad58fb6b82dc7badc55:

(mixedversion.go:732).Run: mixed-version test failure while running step 19 (run "run schemachange workload and validation in mixed version"): full command output in run_105050.275040878_n4_COCKROACHRANDOMSEED3.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=true
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Oct 27, 2024
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-24.3 @ 3839fb143f98e5009a994ad58fb6b82dc7badc55:

(mixedversion.go:732).Run: mixed-version test failure while running step 25 (run "run schemachange workload and validation in mixed version"): full command output in run_105057.101037257_n4_COCKROACHRANDOMSEED3.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-24.3 @ 416acdb7560e4548f3b82c5a522ef48592471c7d:

(mixedversion.go:732).Run: mixed-version test failure while running step 23 (run "run schemachange workload and validation in mixed version"): full command output in run_103015.681724651_n4_COCKROACHRANDOMSEED2.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@fqazi
Copy link
Collaborator

fqazi commented Oct 29, 2024

This looks similar to: #133016

@exalate-issue-sync exalate-issue-sync bot added the P-2 Issues/test failures with a fix SLA of 3 months label Oct 29, 2024
@cockroach-teamcity cockroach-teamcity mentioned this issue Oct 30, 2024
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-24.3 @ c357ebbfdd72bb6aecef57c84850845b5955382f:

(mixedversion.go:732).Run: mixed-version test failure while running step 14 (run "run schemachange workload and validation in mixed version"): full command output in run_101504.891420142_n4_COCKROACHRANDOMSEED7.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/cpu_arch=arm64/run_1

Parameters:

  • ROACHTEST_arch=arm64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Oct 31, 2024
Open
@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-24.3 @ 29d723d5f5e6ed2075feb728e9632fb3516e4fc2:

(mixedversion.go:732).Run: mixed-version test failure while running step 13 (run "run schemachange workload and validation in mixed version"): full command output in run_174906.104752585_n4_COCKROACHRANDOMSEED9.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-24.3 @ bbce415047c9896ee3b33b1eb4c06e3d2cab5bd6:

(mixedversion.go:732).Run: mixed-version test failure while running step 20 (run "run schemachange workload and validation in mixed version"): full command output in run_123135.054043885_n4_COCKROACHRANDOMSEED5.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@spilchen
Copy link
Contributor

spilchen commented Nov 4, 2024 •
edited
Loading

The last two failures have been because of attempts to create a table with a column of type BIT(0).

 {
   "sql": "CREATE TABLE schema_w2_113.table_w2_114 (col114_w2_115 TIMETZ NOT NULL, col114_w2_116 BOOL NOT NULL, col114_w2_117 BIT(0) NOT NULL, col114_w2_118 BIT(22) NOT NULL, col114_w2_119 TIME NOT NULL, col114_w2_120 STRING NOT NULL AS (lower(CAST(col114_w2_119 AS STRING))) VIRTUAL, col114_w2_121 STRING NOT NULL AS (lower(CAST(col114_w2_119 AS STRING))) VIRTUAL, PRIMARY KEY (col114_w2_116 DESC), INDEX (col114_w2_116 DESC) VISIBILITY 0.84, FAMILY (col114_w2_117, col114_w2_119), FAMILY (col114_w2_118), FAMILY (col114_w2_115), FAMILY (col114_w2_116))",
   "expectedExecErr": "3F000"
  }
 ],
 "expectedExecErrors": "3F000",
 "expectedCommitErrors": "",
 "message": "***UNEXPECTED ERROR; Received an unexpected execution error.: ERROR: at or near \")\": syntax error: length for type bit must be at least 1 (SQLSTATE 22023)",

This was recently allowed and backported in #132944. It's probably because it's running an old version that doesn't include the backport.

@rafiss
Copy link
Collaborator

rafiss commented Nov 4, 2024

It might be simplest to avoid using bit(0) at all in any mixed version state.

@spilchen
Copy link
Contributor

spilchen commented Nov 5, 2024

How do we verify if we're in a mixed-version state? I know we can use version gate checks like isClusterVersionLessThan, but that typically checks against a specific version. How can we determine if the workload is running with different, unspecified versions?

This issue should resolve itself once the new patch releases come out on November 18th. If we’re okay with this failing for another two weeks, it may just fix itself.

@rafiss
Copy link
Collaborator

rafiss commented Nov 6, 2024

How do we verify if we're in a mixed-version state? I know we can use version gate checks like isClusterVersionLessThan, but that typically checks against a specific version. How can we determine if the workload is running with different, unspecified versions?

That's a good point -- it would mean we need slightly different isClusterVersionLessThan calls for each branch. That is, each branch would need to pass in a different argument to that function.

This issue should resolve itself once the new patch releases come out on November 18th. If we’re okay with this failing for another two weeks, it may just fix itself.

It does make the test a bit noisy, so maybe we should go ahead and add the mixed version checks. I also have #133790 and #131162 assigned to me, so I may get to it first.

@spilchen
Copy link
Contributor

spilchen commented Nov 6, 2024

Got it, thanks for clarifying! I see now that in a mixed-version test, the test on a given branch always runs against a specific previous version. If I'm following correctly, the table below should outline the version checks:

Branch Down Level Version Function call
master 24.2.x isClusterVersionLessThan(ctx, tx, clusterversion.V24_3.Version())
release-24.3 24.2.x isClusterVersionLessThan(ctx, tx, clusterversion.V24_3.Version())
release-24.2 24.1.x isClusterVersionLessThan(ctx, tx, clusterversion.V24_2.Version())
release-24.1 23.2.x isClusterVersionLessThan(ctx, tx, clusterversion.V24_1.Version())
release-23.2 23.1.x isClusterVersionLessThan(ctx, tx, clusterversion.V23_2.Version())

spilchen added a commit to spilchen/cockroach that referenced this issue Nov 6, 2024
@spilchen
schemachange/mixed-versions: handle possible syntax error for BIT(0) …
ffdfb40 
…column usage

A recent fix enabled support for BIT(0) columns (see issue cockroachdb#132944),
which has since been backported. However, since this backport hasn’t
been released yet, running against an older down-level client may still
trigger a syntax error. This change adds a temporary check to handle
such errors until the backports for cockroachdb#132944 are publicly released.

This adjustment will be backported to older branches, with slight
variations per release:
- release-24.3: Reuse the same fix as in master.
- release-24.2: Use isClusterVersionLessThan with clusterversion.V24.2.
- release-24.1: Use isClusterVersionLessThan with clusterversion.V24.1.
- release-23.2: Use isClusterVersionLessThan with clusterversion.V23.2.

Epic: None
Release note: None
Closes: cockroachdb#133339, cockroachdb#133790, cockroachdb#131162
@spilchen spilchen mentioned this issue Nov 6, 2024
Merged
spilchen added a commit to spilchen/cockroach that referenced this issue Nov 6, 2024
@spilchen
schemachange/mixed-versions: handle possible syntax error for BIT(0) …
e66b80d 
…column usage

A recent fix enabled support for BIT(0) columns (see issue cockroachdb#132944),
which has since been backported. However, since this backport hasn’t
been released yet, running against an older down-level client may still
trigger a syntax error. This change adds a temporary check to handle
such errors until the backports for cockroachdb#132944 are publicly released.

This adjustment will be backported to older branches, with slight
variations per release:
- release-24.3: Reuse the same fix as in master.
- release-24.2: Use isClusterVersionLessThan with clusterversion.V24.2.
- release-24.1: Use isClusterVersionLessThan with clusterversion.V24.1.
- release-23.2: Use isClusterVersionLessThan with clusterversion.V23.2.

Epic: None
Release note: None
Closes: cockroachdb#133339, cockroachdb#133790, cockroachdb#131162
spilchen added a commit to spilchen/cockroach that referenced this issue Nov 6, 2024
@spilchen
schemachange/mixed-versions: handle possible syntax error for BIT(0) …
5750fb9 
…column usage

A recent fix enabled support for BIT(0) columns (see issue cockroachdb#132944),
which has since been backported. However, since this backport hasn’t
been released yet, running against an older down-level client may still
trigger a syntax error. This change adds a temporary check to handle
such errors until the backports for cockroachdb#132944 are publicly released.

This adjustment will be backported to older branches, with slight
variations per release:
- release-24.3: Reuse the same fix as in master.
- release-24.2: Use isClusterVersionLessThan with clusterversion.V24.2.
- release-24.1: Use isClusterVersionLessThan with clusterversion.V24.1.
- release-23.2: Use isClusterVersionLessThan with clusterversion.V23.2.

Epic: None
Release note: None
Closes: cockroachdb#133339, cockroachdb#133790, cockroachdb#131162
@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-24.3 @ 7252771de42bf0752052fae4f945c0e7a88d0ecd:

(mixedversion.go:732).Run: mixed-version test failure while running step 21 (run "run schemachange workload and validation in mixed version"): full command output in run_102556.272523242_n4_COCKROACHRANDOMSEED4.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

craig bot pushed a commit that referenced this issue Nov 7, 2024
@spilchen
Merge #134407
16a78de 
134407: schemachange/mixed-versions: handle possible syntax error for BIT(0) column usage r=spilchen a=spilchen

A recent fix enabled support for BIT(0) columns (see issue #132944), which has since been backported. However, since this backport hasn’t been released yet, running against an older down-level client may still trigger a syntax error. This change adds a temporary check to handle such errors until the backports for #132944 are publicly released.

This adjustment will be backported to older branches, with slight variations per release:
- release-24.3: Reuse the same fix as in master.
- release-24.2: Use isClusterVersionLessThan with clusterversion.V24.2.
- release-24.1: Use isClusterVersionLessThan with clusterversion.V24.1.
- release-23.2: Use isClusterVersionLessThan with clusterversion.V23.2.

Epic: None
Release note: None
Closes #133339
Closes #133790
Closes #131162

Co-authored-by: Matt Spilchen <[email protected]>
@craig craig bot closed this as completed in b66f3b7 Nov 7, 2024
@blathers-crl blathers[crl]
Copy link

blathers-crl bot commented Nov 7, 2024

Based on the specified backports for linked PR #134407, I applied the following new label(s) to this issue: branch-release-23.2, branch-release-24.1, branch-release-24.2. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@blathers-crl blathers-crl bot added branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 branch-release-24.2 Used to mark GA and release blockers, technical advisories, and bugs for 24.2 labels Nov 7, 2024
blathers-crl bot pushed a commit that referenced this issue Nov 7, 2024
@spilchen
schemachange/mixed-versions: handle possible syntax error for BIT(0) …
9b62053 
…column usage

A recent fix enabled support for BIT(0) columns (see issue #132944),
which has since been backported. However, since this backport hasn’t
been released yet, running against an older down-level client may still
trigger a syntax error. This change adds a temporary check to handle
such errors until the backports for #132944 are publicly released.

This adjustment will be backported to older branches, with slight
variations per release:
- release-24.3: Reuse the same fix as in master.
- release-24.2: Use isClusterVersionLessThan with clusterversion.V24.2.
- release-24.1: Use isClusterVersionLessThan with clusterversion.V24.1.
- release-23.2: Use isClusterVersionLessThan with clusterversion.V23.2.

Epic: None
Release note: None
Closes: #133339, #133790, #131162
@blathers-crl blathers-crl bot mentioned this issue Nov 7, 2024
Merged
blathers-crl bot pushed a commit that referenced this issue Nov 7, 2024
@spilchen
schemachange/mixed-versions: handle possible syntax error for BIT(0) …
c58c5a2 
…column usage

A recent fix enabled support for BIT(0) columns (see issue #132944),
which has since been backported. However, since this backport hasn’t
been released yet, running against an older down-level client may still
trigger a syntax error. This change adds a temporary check to handle
such errors until the backports for #132944 are publicly released.

This adjustment will be backported to older branches, with slight
variations per release:
- release-24.3: Reuse the same fix as in master.
- release-24.2: Use isClusterVersionLessThan with clusterversion.V24.2.
- release-24.1: Use isClusterVersionLessThan with clusterversion.V24.1.
- release-23.2: Use isClusterVersionLessThan with clusterversion.V23.2.

Epic: None
Release note: None
Closes: #133339, #133790, #131162
@blathers-crl blathers-crl bot mentioned this issue Nov 7, 2024
Merged
spilchen added a commit that referenced this issue Nov 7, 2024
@spilchen
schemachange/mixed-versions: handle possible syntax error for BIT(0) …
69e93ac 
…column usage

A recent fix enabled support for BIT(0) columns (see issue #132944),
which has since been backported. However, since this backport hasn’t
been released yet, running against an older down-level client may still
trigger a syntax error. This change adds a temporary check to handle
such errors until the backports for #132944 are publicly released.

This adjustment will be backported to older branches, with slight
variations per release:
- release-24.3: Reuse the same fix as in master.
- release-24.2: Use isClusterVersionLessThan with clusterversion.V24.2.
- release-24.1: Use isClusterVersionLessThan with clusterversion.V24.1.
- release-23.2: Use isClusterVersionLessThan with clusterversion.V23.2.

Epic: None
Release note: None
Closes: #133339, #133790, #131162
spilchen added a commit to spilchen/cockroach that referenced this issue Nov 7, 2024
@spilchen
schemachange/mixed-versions: handle possible syntax error for BIT(0) …
e73b424 
…column usage

A recent fix enabled support for BIT(0) columns (see issue cockroachdb#132944),
which has since been backported. However, since this backport hasn’t
been released yet, running against an older down-level client may still
trigger a syntax error. This change adds a temporary check to handle
such errors until the backports for cockroachdb#132944 are publicly released.

This adjustment will be backported to older branches, with slight
variations per release:
- release-24.3: Reuse the same fix as in master.
- release-24.2: Use isClusterVersionLessThan with clusterversion.V24.2.
- release-24.1: Use isClusterVersionLessThan with clusterversion.V24.1.
- release-23.2: Use isClusterVersionLessThan with clusterversion.V23.2.

Epic: None
Release note: None
Closes: cockroachdb#133339, cockroachdb#133790, cockroachdb#131162
@spilchen spilchen mentioned this issue Nov 7, 2024
Merged
spilchen added a commit to spilchen/cockroach that referenced this issue Nov 7, 2024
@spilchen
schemachange/mixed-versions: handle possible syntax error for BIT(0) …
142130d 
…column usage

A recent fix enabled support for BIT(0) columns (see issue cockroachdb#132944),
which has since been backported. However, since this backport hasn’t
been released yet, running against an older down-level client may still
trigger a syntax error. This change adds a temporary check to handle
such errors until the backports for cockroachdb#132944 are publicly released.

This adjustment will be backported to older branches, with slight
variations per release:
- release-24.3: Reuse the same fix as in master.
- release-24.2: Use isClusterVersionLessThan with clusterversion.V24.2.
- release-24.1: Use isClusterVersionLessThan with clusterversion.V24.1.
- release-23.2: Use isClusterVersionLessThan with clusterversion.V23.2.

Epic: None
Release note: None
Closes: cockroachdb#133339, cockroachdb#133790, cockroachdb#131162
@spilchen spilchen mentioned this issue Nov 7, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@spilchen spilchen

Labels
branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 branch-release-24.2 Used to mark GA and release blockers, technical advisories, and bugs for 24.2 branch-release-24.3 Used to mark GA and release blockers, technical advisories, and bugs for 24.3 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

schemachange/mixed-versions: handle possible syntax error for BIT(0) column usage spilchen/cockroach
4 participants
@spilchen @rafiss @fqazi @cockroach-teamcity