providing validation for specialized FlyteFormat types

[zeryx]

Motivation: Why do you think this is important?

This issue revolves around the fact that today, specialized filetypes within Flyte are not validated to actually conform to the requested type - just the file type metadata is provided to s3. This means I could store a png image as a JPEGImageFile object, which could end up adding a lot of confusion to end users who may expect JPEGImageFile types to be validated.

Goal: What should the final outcome look like, ideally?

Ideally, there should be a mechanism at runtime that validates that a particular file actually corresponds to the format requested. If I had a PDFFile type, I would expect to be able to ensure that the actual serialized file is a PDF type.

Describe alternatives you've considered

https://en.wikipedia.org/wiki/File_(command)

^ Previously used this in other projects to verify image format and filetype.

Checking file extensions is also a potential mechanism for validation (albeit a much worse one)

Propose: Link/Inline OR Additional context

example of a wf that succeeds (when it shouldn't)
https://gist.github.com/zeryx/31266bbe21d4dcfeca9f1b0e7dc3a883

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[eapolinario]

The functionality provided by the file utility is present in a library called libmagic which has python bindings.

We could hook python-libmagic up with the aliases exported in the FlyteFile type transformers, more specifically in the to_literal.

[hhcs9527]

Hi @eapolinario,
I am interested in this topic but still new to flyte.
Can I try this issue?

[eapolinario]

@hhcs9527 , for sure. I'm going to assign it to you. Feel free to ping me if you need to discuss it further or simply to review PRs, etc. Thanks!

[jasonlai1218]

@hhcs9527 @pingsutw @eapolinario
I would like to work on this, can you please assign me and give me a chance to participate in open source together?