[BUG] Container overrides broken in flytekit 1.11.0

[ai-rnatour]

Describe the bug

See this related issue.

This was working for us in version 1.10.2 of flytekit, however when we upgraded to 1.11.0 we confirmed that this no longer works. In our case, the default base image is used instead of the runtime override applied through with_overrides.

Expected behavior

When using with_overrides, I would expect the container_image I provide to override our base image.

Additional context to reproduce

No response

Screenshots

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[runllm]

To address the issue of container image overrides not working in flytekit version 1.11.0, it's important to note that this problem has been identified in previous versions and discussed in various contexts. The core issue lies in how flytekit handles task-level metadata, specifically the container image, during the registration process. As per discussions on Flyte forums, overriding the container image using with_overrides may not work as expected when tasks are registered because the last override tends to win, and the registration process does not handle these overrides at the task node level effectively.

A workaround involves using dynamic tasks to register new task entities with overridden container images, as suggested in the discussions. This method ensures that each task can have its unique container image. For more details on implementing this workaround or to understand the underlying issue better, you might want to refer to the discussions and examples provided in the Flyte community resources.

Would you like a detailed explanation or a code snippet demonstrating how to implement the workaround using dynamic tasks?

Sources:

• Overriding the container image isn't working for me
• Override container image by using with_overrides

---

This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.

Mention @RunLLM to ask follow-up questions. See runllm.com for more!

[fg91]

Discussion ongoing in #4543.

[eapolinario]

As discussed in #4543 (comment), this ended up being a breaking change that should have been feature more prominently in the 1.11.0 release notes.

I'm going to close this issue since we cleared the confusion. Let me know if you need to discuss this again.