Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: schemachange/mixed-versions failed #131162

Closed
cockroach-teamcity opened this issue Sep 22, 2024 · 59 comments · Fixed by #134407
Closed

roachtest: schemachange/mixed-versions failed #131162

cockroach-teamcity opened this issue Sep 22, 2024 · 59 comments · Fixed by #134407
Assignees
@rafiss
Labels
B-runtime-assertions-enabled branch-master Failures and bugs on the master branch. branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 branch-release-24.2 Used to mark GA and release blockers, technical advisories, and bugs for 24.2 branch-release-24.3 Used to mark GA and release blockers, technical advisories, and bugs for 24.3 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. P-3 Issues/test failures with no fix SLA T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Sep 22, 2024 •
edited by cockroach-jira-scripts
Loading

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar issue without the B-runtime-assertions-enabled label. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.schemachange/mixed-versions failed with artifacts on master @ 83589fb87caa92fb42e83994f1691978f37e4cbb:

(mixedversion.go:720).Run: mixed-version test failure while running step 18 (run "run schemachange workload and validation in mixed version"): full command output in run_064232.570264793_n4_COCKROACHRANDOMSEED9.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=true
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-42418
@cockroach-teamcity cockroach-teamcity added B-runtime-assertions-enabled branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) labels Sep 22, 2024
@fqazi
Copy link
Collaborator

fqazi commented Sep 24, 2024

So, this failure is linked to renames occurring back to back:

 "message": "***UNEXPECTED ERROR; Received an unexpected execution error.: ERROR: function udf_w1_647(composite_w3_74[],composite_w4_114[],composite_w4_58[],composite_w4_721[],composite_w3_74,composite_w4_114,composite_w4_58,composite_w4_721,composite_w3_280[],composite_w4_263[],enum_w0_351[],composite_w3_280,composite_w4_263,enum_w0_351,composite_w4_354[],enum_w0_41[],enum_w4_101[],composite_w4_354,enum_w0_41,enum_w4_101,enum_w0_243[],enum_w1_126[],enum_w1_368[],enum_w0_243,enum_w1_126,enum_w1_368,composite_w0_288[],enum_w1_46[],enum_w2_220[],composite_w0_288,enum_w1_46,\"char\",bit,bool,box2d,char,bytes,string,date,decimal,float,geography,geometry,inet,int,int2,int2vector,int,interval,jsonb,name,oidvector,pg_lsn,refcursor,string,time,timestamp,timestamptz,timetz,tsquery,tsvector) does not exist (SQLSTATE 42883)",

the object we recently created via a rename:

ALTER FUNCTION schema_w3_478.udf_w1_583(public._composite_w3_74, public._composite_w4_114, public._composite_w4_58, public._composite_w4_721, public.composite_w3_74, public.composite_w4_114, public.composite_w4_58, public.composite_w4_721, schema_w1_15._composite_w3_280, schema_w1_15._composite_w4_263, schema_w1_15._enum_w0_351, schema_w1_15.composite_w3_280, schema_w1_15.composite_w4_263, schema_w1_15.enum_w0_351, schema_w3_61._composite_w4_354, schema_w3_61._enum_w0_41, schema_w3_61._enum_w4_101, schema_w3_61.composite_w4_354, schema_w3_61.enum_w0_41, schema_w3_61.enum_w4_101, schema_w2_103._enum_w0_243, schema_w2_103._enum_w1_126, schema_w2_103._enum_w1_368, schema_w2_103.enum_w0_243, schema_w2_103.enum_w1_126, schema_w2_103.enum_w1_368, schema_w0_77._composite_w0_288, schema_w0_77._enum_w1_46, schema_w0_77._enum_w2_220, schema_w0_77.composite_w0_288, schema_w0_77.enum_w1_46, \"char\", BIT, BOOL, BOX2D, BPCHAR, BYTES, STRING, DATE, DECIMAL, FLOAT8, GEOGRAPHY, GEOMETRY, INET, INT8, INT2, INT2VECTOR, INT8, INTERVAL, JSONB, NAME, OIDVECTOR, PG_LSN, REFCURSOR, STRING, TIME, TIMESTAMP, TIMESTAMPTZ, TIMETZ, TSQUERY, TSVECTOR) RENAME TO udf_w1_647

A simple repro here doesn't work

@fqazi
Copy link
Collaborator

fqazi commented Sep 24, 2024

Maybe the mixed version state was at play here:

  |                                                                  n1          n2          n3          n4
  | released versions                                                v24.2.1     v24.2.1     v24.2.1     master
  | logical binary versions                                          24.2        24.2        24.2        24.2-upgrading-to-24.3-step-016
  | cluster versions before failure (system)                         24.2        24.2        24.2        24.2
  | cluster versions after failure (system)                          24.2        24.2        24.2        24.2
  | cluster versions before failure (mixed-version-tenant-czrnv)     24.2        24.2        24.2        24.2
  | cluster versions after failure (mixed-version-tenant-czrnv)      24.2        24.2        24.2        24.2
Wraps: (2) attached stack trace

@fqazi
Copy link
Collaborator

fqazi commented Sep 24, 2024

I'm gonna stick this in the triage pile, lets see if we observe it again.

This was referenced Sep 25, 2024
Closed
Open
@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on master @ 41084720464c4144f64d9ddcb46508b4d762c4e8:

(mixedversion.go:720).Run: mixed-version test failure while running step 16 (run "run schemachange workload and validation after upgrade has finalized"): full command output in run_083059.671973709_n4_COCKROACHRANDOMSEED3.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/cpu_arch=arm64/run_1

Parameters:

  • ROACHTEST_arch=arm64
  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.schemachange/mixed-versions failed with artifacts on master @ 67dc7a1c9bf117046b10513c3277bf7ccf0db975:

(mixedversion.go:737).Run: mixed-version test failure while running step 19 (run "run schemachange workload and validation in mixed version"): full command output in run_064632.864983100_n4_COCKROACHRANDOMSEED8.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=true
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.schemachange/mixed-versions failed with artifacts on master @ 67dc7a1c9bf117046b10513c3277bf7ccf0db975:

(mixedversion.go:737).Run: mixed-version test failure while running step 11 (run "run schemachange workload and validation in mixed version"): full command output in run_123841.551664357_n4_COCKROACHRANDOMSEED8.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=true
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on master @ 5400cb9a70e63bfe1aa2849a566c195ad63130d1:

(mixedversion.go:737).Run: mixed-version test failure while running step 27 (run "run schemachange workload and validation in mixed version"): full command output in run_073614.179943298_n4_COCKROACHRANDOMSEED2.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on master @ b6c13686495bbe9ad476b28033461ef7628e18a8:

(mixedversion.go:737).Run: mixed-version test failure while running step 12 (run "run schemachange workload and validation in mixed version"): full command output in run_061511.239883760_n4_COCKROACHRANDOMSEED4.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on master @ b6c13686495bbe9ad476b28033461ef7628e18a8:

(mixedversion.go:737).Run: mixed-version test failure while running step 8 (run "run schemachange workload and validation in mixed version"): full command output in run_070234.145815425_n4_COCKROACHRANDOMSEED3.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Oct 1, 2024
Open
@cockroach-teamcity
Copy link
Member Author

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.schemachange/mixed-versions failed with artifacts on master @ 74333311616b937fea6a995462215a1cb5962686:

(mixedversion.go:737).Run: mixed-version test failure while running step 11 (run "run schemachange workload and validation in mixed version"): full command output in run_071626.787954313_n4_COCKROACHRANDOMSEED6.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=true
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on master @ ec2573dc6aaeefc226440bb2c5a7c94a63989868:

(mixedversion.go:737).Run: mixed-version test failure while running step 11 (run "run schemachange workload and validation in mixed version"): full command output in run_071314.492703135_n4_COCKROACHRANDOMSEED7.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.schemachange/mixed-versions failed with artifacts on master @ 0c0af9540ed3f9d63eba523bc870eeb6c7eebe90:

(mixedversion.go:732).Run: mixed-version test failure while running step 13 (run "run schemachange workload and validation in mixed version"): full command output in run_064606.857018228_n4_COCKROACHRANDOMSEED5.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=true
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on master @ f842c3b4b5adc040d411bd17d7d10005273fc1b6:

(mixedversion.go:732).Run: mixed-version test failure while running step 29 (run "run schemachange workload and validation in mixed version"): full command output in run_081656.342236161_n4_COCKROACHRANDOMSEED5.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on master @ fd4b1464dbd6e385c6e51af26fe294fd2023a259:

(mixedversion.go:732).Run: mixed-version test failure while running step 19 (run "run schemachange workload and validation in mixed version"): full command output in run_093143.180400942_n4_COCKROACHRANDOMSEED1.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on master @ 645eb8c99796b3b88f5631aa0fc92a011010ce64:

(mixedversion.go:732).Run: mixed-version test failure while running step 17 (run "run schemachange workload and validation in mixed version"): full command output in run_061138.887300800_n4_COCKROACHRANDOMSEED8.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on master @ a0f39e7ac9574756063bc90bba6bc532b45c33d4:

(mixedversion.go:732).Run: mixed-version test failure while running step 23 (run "run schemachange workload and validation in mixed version"): full command output in run_084240.422619629_n4_COCKROACHRANDOMSEED3.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on master @ 49ca24cedb042579e9645c206640d59975805d12:

(mixedversion.go:732).Run: mixed-version test failure while running step 9 (run "run schemachange workload and validation in mixed version"): full command output in run_094022.054694636_n4_COCKROACHRANDOMSEED6.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on master @ 5be5b0b52ff79b98689b2282a8b25cf9eb50ec40:

(mixedversion.go:732).Run: mixed-version test failure while running step 12 (run "run schemachange workload and validation in mixed version"): full command output in run_125417.691846160_n4_COCKROACHRANDOMSEED6.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Oct 17, 2024
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on master @ 42f40f59cae3c0fd8842e194d6991c951ab4382f:

(mixedversion.go:732).Run: mixed-version test failure while running step 16 (run "run schemachange workload and validation in mixed version"): full command output in run_065824.620219033_n4_COCKROACHRANDOMSEED7.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on master @ 015b2f48cf80a6d8b60d7038c8c3457d934c716a:

(mixedversion.go:732).Run: mixed-version test failure while running step 12 (run "run schemachange workload and validation in mixed version"): full command output in run_092926.884501308_n4_COCKROACHRANDOMSEED6.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@rimadeodhar
Copy link
Collaborator

@rafiss - this test seems to consistently fail. Is there anything we can do to stabilize it in addition to #133877?
I have assigned a P3 to this for now but I wonder whether we should skip this test until we have made it more reliable?

@rimadeodhar rimadeodhar added the P-3 Issues/test failures with no fix SLA label Nov 5, 2024
@rafiss
Copy link
Collaborator

rafiss commented Nov 6, 2024

@rimadeodhar yeah, we have been discussing this test in our recent team tech discussions. we recently (about a month ago) decided to start using potentialExecErrors instead of expectedExecErrors to make the error assertions in this test less flaky. we want to try this strategy a bit and see if it helps, and if not, we will work on #133877.

I don't think we need to skip it, but i will take a look at the latest failures here. not sure how this remained unassigned.

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on master @ d0e07efe30dfe64d36412363000a1b977b4d5d2e:

(mixedversion.go:732).Run: mixed-version test failure while running step 9 (run "run schemachange workload and validation in mixed version"): full command output in run_084754.000852829_n4_COCKROACHRANDOMSEED9.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/cpu_arch=arm64/run_1

Parameters:

  • ROACHTEST_arch=arm64
  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

Same failure on other branches

This test on roachdash | Improve this report!

spilchen added a commit to spilchen/cockroach that referenced this issue Nov 6, 2024
@spilchen
schemachange/mixed-versions: handle possible syntax error for BIT(0) …
ffdfb40 
…column usage

A recent fix enabled support for BIT(0) columns (see issue cockroachdb#132944),
which has since been backported. However, since this backport hasn’t
been released yet, running against an older down-level client may still
trigger a syntax error. This change adds a temporary check to handle
such errors until the backports for cockroachdb#132944 are publicly released.

This adjustment will be backported to older branches, with slight
variations per release:
- release-24.3: Reuse the same fix as in master.
- release-24.2: Use isClusterVersionLessThan with clusterversion.V24.2.
- release-24.1: Use isClusterVersionLessThan with clusterversion.V24.1.
- release-23.2: Use isClusterVersionLessThan with clusterversion.V23.2.

Epic: None
Release note: None
Closes: cockroachdb#133339, cockroachdb#133790, cockroachdb#131162
@spilchen spilchen mentioned this issue Nov 6, 2024
Merged
@rimadeodhar
Copy link
Collaborator

Aah my bad. I thought this #133877 was the tracking issue for what we discussed in the tech discussion which is why I referenced it above.
Thanks for taking a look.

spilchen added a commit to spilchen/cockroach that referenced this issue Nov 6, 2024
@spilchen
schemachange/mixed-versions: handle possible syntax error for BIT(0) …
e66b80d 
…column usage

A recent fix enabled support for BIT(0) columns (see issue cockroachdb#132944),
which has since been backported. However, since this backport hasn’t
been released yet, running against an older down-level client may still
trigger a syntax error. This change adds a temporary check to handle
such errors until the backports for cockroachdb#132944 are publicly released.

This adjustment will be backported to older branches, with slight
variations per release:
- release-24.3: Reuse the same fix as in master.
- release-24.2: Use isClusterVersionLessThan with clusterversion.V24.2.
- release-24.1: Use isClusterVersionLessThan with clusterversion.V24.1.
- release-23.2: Use isClusterVersionLessThan with clusterversion.V23.2.

Epic: None
Release note: None
Closes: cockroachdb#133339, cockroachdb#133790, cockroachdb#131162
spilchen added a commit to spilchen/cockroach that referenced this issue Nov 6, 2024
@spilchen
schemachange/mixed-versions: handle possible syntax error for BIT(0) …
5750fb9 
…column usage

A recent fix enabled support for BIT(0) columns (see issue cockroachdb#132944),
which has since been backported. However, since this backport hasn’t
been released yet, running against an older down-level client may still
trigger a syntax error. This change adds a temporary check to handle
such errors until the backports for cockroachdb#132944 are publicly released.

This adjustment will be backported to older branches, with slight
variations per release:
- release-24.3: Reuse the same fix as in master.
- release-24.2: Use isClusterVersionLessThan with clusterversion.V24.2.
- release-24.1: Use isClusterVersionLessThan with clusterversion.V24.1.
- release-23.2: Use isClusterVersionLessThan with clusterversion.V23.2.

Epic: None
Release note: None
Closes: cockroachdb#133339, cockroachdb#133790, cockroachdb#131162
spilchen added a commit to spilchen/cockroach that referenced this issue Nov 6, 2024
@spilchen
schemachange/mixed-versions: handle possible syntax error for BIT(0) …
b66f3b7 
…column usage

A recent fix enabled support for BIT(0) columns (see issue cockroachdb#132944),
which has since been backported. However, since this backport hasn’t
been released yet, running against an older down-level client may still
trigger a syntax error. This change adds a temporary check to handle
such errors until the backports for cockroachdb#132944 are publicly released.

This adjustment will be backported to older branches, with slight
variations per release:
- release-24.3: Reuse the same fix as in master.
- release-24.2: Use isClusterVersionLessThan with clusterversion.V24.2.
- release-24.1: Use isClusterVersionLessThan with clusterversion.V24.1.
- release-23.2: Use isClusterVersionLessThan with clusterversion.V23.2.

Epic: None
Release note: None
Closes: cockroachdb#133339, cockroachdb#133790, cockroachdb#131162
@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on master @ a60d739746648922134ec3c0a22bb069bf1d283c:

(mixedversion.go:755).Run: mixed-version test failure while running step 16 (run "run schemachange workload and validation in mixed version"): full command output in run_075327.695059786_n4_COCKROACHRANDOMSEED5.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on master @ a60d739746648922134ec3c0a22bb069bf1d283c:

(mixedversion.go:755).Run: mixed-version test failure while running step 18 (run "run schemachange workload and validation in mixed version"): full command output in run_080022.954517571_n4_COCKROACHRANDOMSEED7.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_encrypted=false
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@craig craig bot closed this as completed in 16a78de Nov 7, 2024
@blathers-crl blathers[crl]
Copy link

blathers-crl bot commented Nov 7, 2024

Based on the specified backports for linked PR #134407, I applied the following new label(s) to this issue: branch-release-23.2, branch-release-24.1, branch-release-24.2, branch-release-24.3. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@blathers-crl blathers-crl bot added branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 branch-release-24.2 Used to mark GA and release blockers, technical advisories, and bugs for 24.2 branch-release-24.3 Used to mark GA and release blockers, technical advisories, and bugs for 24.3 labels Nov 7, 2024
blathers-crl bot pushed a commit that referenced this issue Nov 7, 2024
@spilchen
schemachange/mixed-versions: handle possible syntax error for BIT(0) …
9b62053 
…column usage

A recent fix enabled support for BIT(0) columns (see issue #132944),
which has since been backported. However, since this backport hasn’t
been released yet, running against an older down-level client may still
trigger a syntax error. This change adds a temporary check to handle
such errors until the backports for #132944 are publicly released.

This adjustment will be backported to older branches, with slight
variations per release:
- release-24.3: Reuse the same fix as in master.
- release-24.2: Use isClusterVersionLessThan with clusterversion.V24.2.
- release-24.1: Use isClusterVersionLessThan with clusterversion.V24.1.
- release-23.2: Use isClusterVersionLessThan with clusterversion.V23.2.

Epic: None
Release note: None
Closes: #133339, #133790, #131162
@blathers-crl blathers-crl bot mentioned this issue Nov 7, 2024
Merged
blathers-crl bot pushed a commit that referenced this issue Nov 7, 2024
@spilchen
schemachange/mixed-versions: handle possible syntax error for BIT(0) …
c58c5a2 
…column usage

A recent fix enabled support for BIT(0) columns (see issue #132944),
which has since been backported. However, since this backport hasn’t
been released yet, running against an older down-level client may still
trigger a syntax error. This change adds a temporary check to handle
such errors until the backports for #132944 are publicly released.

This adjustment will be backported to older branches, with slight
variations per release:
- release-24.3: Reuse the same fix as in master.
- release-24.2: Use isClusterVersionLessThan with clusterversion.V24.2.
- release-24.1: Use isClusterVersionLessThan with clusterversion.V24.1.
- release-23.2: Use isClusterVersionLessThan with clusterversion.V23.2.

Epic: None
Release note: None
Closes: #133339, #133790, #131162
@blathers-crl blathers-crl bot mentioned this issue Nov 7, 2024
Merged
spilchen added a commit that referenced this issue Nov 7, 2024
@spilchen
schemachange/mixed-versions: handle possible syntax error for BIT(0) …
69e93ac 
…column usage

A recent fix enabled support for BIT(0) columns (see issue #132944),
which has since been backported. However, since this backport hasn’t
been released yet, running against an older down-level client may still
trigger a syntax error. This change adds a temporary check to handle
such errors until the backports for #132944 are publicly released.

This adjustment will be backported to older branches, with slight
variations per release:
- release-24.3: Reuse the same fix as in master.
- release-24.2: Use isClusterVersionLessThan with clusterversion.V24.2.
- release-24.1: Use isClusterVersionLessThan with clusterversion.V24.1.
- release-23.2: Use isClusterVersionLessThan with clusterversion.V23.2.

Epic: None
Release note: None
Closes: #133339, #133790, #131162
spilchen added a commit to spilchen/cockroach that referenced this issue Nov 7, 2024
@spilchen
schemachange/mixed-versions: handle possible syntax error for BIT(0) …
e73b424 
…column usage

A recent fix enabled support for BIT(0) columns (see issue cockroachdb#132944),
which has since been backported. However, since this backport hasn’t
been released yet, running against an older down-level client may still
trigger a syntax error. This change adds a temporary check to handle
such errors until the backports for cockroachdb#132944 are publicly released.

This adjustment will be backported to older branches, with slight
variations per release:
- release-24.3: Reuse the same fix as in master.
- release-24.2: Use isClusterVersionLessThan with clusterversion.V24.2.
- release-24.1: Use isClusterVersionLessThan with clusterversion.V24.1.
- release-23.2: Use isClusterVersionLessThan with clusterversion.V23.2.

Epic: None
Release note: None
Closes: cockroachdb#133339, cockroachdb#133790, cockroachdb#131162
@spilchen spilchen mentioned this issue Nov 7, 2024
Merged
spilchen added a commit to spilchen/cockroach that referenced this issue Nov 7, 2024
@spilchen
schemachange/mixed-versions: handle possible syntax error for BIT(0) …
142130d 
…column usage

A recent fix enabled support for BIT(0) columns (see issue cockroachdb#132944),
which has since been backported. However, since this backport hasn’t
been released yet, running against an older down-level client may still
trigger a syntax error. This change adds a temporary check to handle
such errors until the backports for cockroachdb#132944 are publicly released.

This adjustment will be backported to older branches, with slight
variations per release:
- release-24.3: Reuse the same fix as in master.
- release-24.2: Use isClusterVersionLessThan with clusterversion.V24.2.
- release-24.1: Use isClusterVersionLessThan with clusterversion.V24.1.
- release-23.2: Use isClusterVersionLessThan with clusterversion.V23.2.

Epic: None
Release note: None
Closes: cockroachdb#133339, cockroachdb#133790, cockroachdb#131162
@spilchen spilchen mentioned this issue Nov 7, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rafiss rafiss

Labels
B-runtime-assertions-enabled branch-master Failures and bugs on the master branch. branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 branch-release-24.2 Used to mark GA and release blockers, technical advisories, and bugs for 24.2 branch-release-24.3 Used to mark GA and release blockers, technical advisories, and bugs for 24.3 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. P-3 Issues/test failures with no fix SLA T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

schemachange/mixed-versions: handle possible syntax error for BIT(0) column usage spilchen/cockroach
4 participants
@rafiss @fqazi @cockroach-teamcity @rimadeodhar