Core, Spark3.5: Fix tests failure due to timeout

[manuzhang]

This PR attempts to fix following tests failure due to timeout.

#11047
#11066
#11651

[manuzhang]

@nastra Please take another look, thanks!

[nastra]

@RussellSpitzer or @amogh-jahagirdar can any of you also take a look at this please?

[nastra]

I think you've got too many commas here. I believe this should be ('%s' '%s', '%s' '%s', '%s' '%s') and same for the other Spark test

[amogh-jahagirdar]

So if I understand right, to address flakiness of these tests that are timing out, we're reducing a lot of commit properties (like the max retry and the min retry wait), that way we just fail fast? Typically these should not exceed these values and any tests which do, would just end up failing fast.

Do we have any indication on what could be the root cause for the commits in these tests taking so long?

To be clear, I'm open to giving the fix in this PR a try but I think we should be a bit cautious that we're not introducing flakiness the other way where another set of tests end up failing more frequently due to not waiting enough for example.

[nastra]

@amogh-jahagirdar from what I saw in some local debugging is that these specific tests typically took quite a long time due to exponential retries and eventually timed out, hence this is lowering the min/max wait times for these retries in these 3 particular tests. Other tests shouldn't be affected by this

[amogh-jahagirdar]

Sounds good,

these specific tests typically took quite a long time due to exponential retries and eventually timed out

Yeah this was the part I was trying to unpack was why are these particular tests retrying a lot? Like I said though, that may be difficult to reason about and we may just want to tune these properties in the interim, so all good!