Various errors when running a spark cluster driver with mirrord

[t4lz]

User runs a spark cluster driver with mirrord.
The program downloads heavy jar files from s3 and then uploads them to the workers. This process seems to be slow with mirrord.
At some point the ping-pong between the intproxy and the agent fails.

Additionally, there is this error log from mirrord:

2024-11-21T12:22:31.234085Z ERROR ThreadId(02) mirrord_layer::socket::ops: connect -> Failed call to libc::connect with ConnectResult {
    result: -1,
    error: Some(
        Errno {
            code: 22,
            description: Some(
                "Invalid argument",
            ),
        },
    ),
}
2024-11-21T12:22:31.234203Z ERROR ThreadId(02) mirrord_layer::error: Error occured in Layer >> IO(Os { code: 22, kind: InvalidInput, message: "Invalid argument" })

Sometimes the run seems to be successful (enough?) despite the errors.

[linear]

MBE-530 Various errors when running a spark cluster driver with mirrord