Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Video display #118

Closed
light-matters opened this issue Sep 17, 2024 · 19 comments
Closed

Support Video display #118

light-matters opened this issue Sep 17, 2024 · 19 comments
Assignees
Labels
enhancement New feature or request

Comments

@light-matters
Copy link
Contributor

Manipulating/creating a Wolfram Video returns a strange format that no visual tool knows how to interpret.

(w/VideoTrim [12 24])

Returns

(Video /home/ctw/Documents/Wolfram/Video/VideoTrim-5ef8ad95-051d-4e6f-a1b2-1f1e24b62217.mkv (-> Appearance Automatic) (-> AudioOutputDevice Automatic) (-> SoundVolume Automatic))

We should some implement some sort of 'kindly' support (e.g. metadata) that allows visual tools such as Clay and Clerk to display these outputs correctly.

@light-matters light-matters added the enhancement New feature or request label Sep 17, 2024
@holyjak
Copy link
Member

holyjak commented Sep 17, 2024

@daslu Does Clay/Kindly support metadata on return values to know how to display them? So that when we return (Video "/file/path/..") we could add some metadata to tell Clay to get the video path (i.e. second on the data structure) and display as a video? 🙏

I obviously cannot simply (with-meta '(Video "path..." #_...) {:kind/video true}) because that does not communicate that the value of interest is a local file path, and how to get it out of the data structure (here, a List). I don't know if :kindly/options could be leveraged? Currently, it seems from the Kindly Book that only embedded video from youtube are supported... Perhaps I should rather use hiccup of html and the video tag - but still need to instruct the viewer tool how to get from the actual value to that... Perhaps something like :kind/custom (fn [wolfram-video] ^:kind/hiccup [:video [:source {:src (second wolfram-video)}]]) ?

@daslu
Copy link
Member

daslu commented Sep 17, 2024

Hi.

I'm copying a few details from a private chat with @light-matters:

  • kind/video supports only youtube, and following your proposal, we will add support for local files.
  • kind/hiccup is supposed to support anything that HTML can do.
  • If you render through Quarto (incliding Reveal slideshows), you include in your markdown comments anything that Quarto supports: Videos.

If you are pointing to files, you need to put them under a directory which is copied by Clay under the target directory, as explained in Referring to files. Then, you can just refer to them with the path relative to the target directory.

There is also some support for custom functions, as you suggested: Functions

Does it seem to make sense?

@holyjak
Copy link
Member

holyjak commented Sep 26, 2024

@daslu thank you! How would we make Clay output Quarto's {{< video local-video.mp4 >}} ?

@light-matters perhaps we can just do

^:kind/hiccup
(let [url (->> (wl/eval (w/VideoTrim ...)) second)]
  [:video [:source {:src url :type "video/mp4" }]])

WDYT?

Or, using Clay fns:

(defn wolf-video->hiccup [wl-result]
  {:pre [(list? wl-result) (= 'Video (first wl-result))]}
  ^:kind/hiccup
  [:video [:source {:src (second wl-result) :type "video/mp4"}]])

(k/fn [wolf-video->hiccup (wl/eval ....)])

The fn could also make the video to another place, if desired...

@daslu
Copy link
Member

daslu commented Sep 26, 2024

Hi!

Quarto Markdown can simply be included in Clojure comments or kind/md.

I hope to add some better support to the other ways during the coming weekend.

@light-matters
Copy link
Contributor Author

My feeling is that we should avoid explicit quarto notation where possible. Whether the output uses quarto or not to garner the particular effect seems like an implementation detail and not something for Wolframite to worry about. An argument can be made for the case of the website etc. where the exact output goal is known in advance.

@light-matters
Copy link
Contributor Author

@daslu thank you! How would we make Clay output Quarto's {{< video local-video.mp4 >}} ?

@light-matters perhaps we can just do

^:kind/hiccup
(let [url (->> (wl/eval (w/VideoTrim ...)) second)]
  [:video [:source {:src url :type "video/mp4" }]])

WDYT?

Or, using Clay fns:

(defn wolf-video->hiccup [wl-result]
  {:pre [(list? wl-result) (= 'Video (first wl-result))]}
  ^:kind/hiccup
  [:video [:source {:src (second wl-result) :type "video/mp4"}]])

(k/fn [wolf-video->hiccup (wl/eval ....)])

The fn could also make the video to another place, if desired...

Can we highjack the return from Wolfram and add the kind information automatically (after detecting the return)?

@holyjak
Copy link
Member

holyjak commented Sep 26, 2024

Can we highjack the return from Wolfram and add the kind information automatically (after detecting the return)?

That would be best, but not possible with the way kind/fn works now, as far as I understand it. It either wants a vector with the fn and value (where value in our case is the returned (Video ...) or a map with.. @daslu we would want to be able to do

^{:kind/f (fn [the-value] ^:kind/hiccup [:video [:source {:src (second the-value)}]])}
the-value

because then we could leave the value as-is and just attach metadata to it. However that is not how kind/fn works, as far as I see. It requires either a vector or a map. WDYT?

@light-matters
Copy link
Contributor Author

Maybe we should avoid the kind function? Let's discuss this later.

@daslu
Copy link
Member

daslu commented Sep 29, 2024

@holyjak having :kind/f as metadata is an interesting idea worth thinking about.

Anyway, kind/video should support a file path, and that will happen in the coming Clay release tomorrow.

@daslu
Copy link
Member

daslu commented Sep 29, 2024

Oh, sorry, I see what you mean now.

You need the custom function to avoid changing the value (Video "path/to/file.mkv" ...) and just define the way it is rendered.

I'll sleep on the :kind/f metadata idea.

@daslu
Copy link
Member

daslu commented Sep 29, 2024

Clay version 2-beta17 adds kind/video support for urls, and also additional support to passing the function as metadata as @holyjak proposed.

See: https://scicloj.github.io/clay/#functions

So, we can do something like:

(k/fn (w/eval ...)
  {:kindly/f wolf-video->hiccup})

Thanks for these helpful ideas.

holyjak added a commit that referenced this issue Oct 1, 2024
Issue: Wolfram may return `(Video <path to file> ...)` and we want to be able to display it nicely in a Clay or compatible notebook.

Fix: Add ns with a fn to try add display metadata to all results, call it from eval.
holyjak added a commit that referenced this issue Oct 1, 2024
Issue: Wolfram may return `(Video <path to file> ...)` and we want to be able to display it nicely in a Clay or compatible notebook.

Fix: Add ns with a fn to try add display metadata to all results, call it from eval.
@holyjak
Copy link
Member

holyjak commented Oct 1, 2024

@light-matters could you be so kind and verify that #131 works for you?

@daslu This is amazing, thanks a lot! Does the meta we attach here https://github.com/scicloj/wolframite/pull/131/files#diff-62bf9ddf1ee0d76b66dec5a61c7d8766df142e9471c228ce2e7925b67a5403d3R18-R20 look good to you?

@daslu
Copy link
Member

daslu commented Oct 1, 2024

Looks good!

@holyjak

This comment was marked as outdated.

@light-matters
Copy link
Contributor Author

From the other thread (#131),

The format

(k/video {:src "./resources/video/Baby.mp4"}) 

works for me rather than file://...
The DIYness can be extracted from whatever k/video produces.

@light-matters
Copy link
Contributor Author

We also need to make those folders visible to clay.

@holyjak
Copy link
Member

holyjak commented Oct 4, 2024

Thank you, the code works now.

However, as you point out, the video must be in a folder which Clay can read. Not sure how to ensure that, when we work with a video from Wolfram. When using a notebook, we could start with having the video in an appropriate folder - just need to document this well. But what about the output of w/Video and friends? Do they return relative, or absolute paths? If we start with a relative path and they keep it relative, and in the same folder, then we are good. If they absolutize it, or add files to a different location, then we have a problem.

We could/should perhaps check that the url is relative, and return something else such as hiccup with a warning message, when it is not...?

@light-matters
Copy link
Contributor Author

For reference, absolute paths can be given to the clay options, see this thread

This is particularly necessary for operations like VideoTrim which store the results centrally. Probably we need to pass some of these default paths to whatever viewer the user uses at startup.

@light-matters
Copy link
Contributor Author

PRing this for now because it works. The extra things mentioned above now have their own issues: #143 #142

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants