Skip to content
This repository has been archived by the owner on Apr 24, 2024. It is now read-only.

Add support for videos #22

Open
iMuzz opened this issue Apr 6, 2021 · 9 comments · May be fixed by #24
Open

Add support for videos #22

iMuzz opened this issue Apr 6, 2021 · 9 comments · May be fixed by #24
Labels
enhancement New feature or request

Comments

@iMuzz
Copy link

iMuzz commented Apr 6, 2021

React-static-tweets doesn't current support videos. Here's an example tweet:

On Twitter: https://twitter.com/sassal0x/status/1364207094614749184
Rendered Version: https://react-static-tweets.vercel.app/1364207094614749184

Would love any pointers on how/where one might start to implement this! 🙂

@transitive-bullshit transitive-bullshit added the enhancement New feature or request label Apr 7, 2021
@DaniGuardiola
Copy link

DaniGuardiola commented Apr 7, 2021

Working on this already (as agreed with @iMuzz).

Here's an update on my progress:

  • The static-tweets ast parser is not finding the video media.
  • That's because it's not there in the HTML in the first place (likely Twitter API change).
  • The media URI is available through this URL:
syndication.twimg.com/tweet?id=1378452375296442371

Or, alternatively:

cdn.syndication.twimg.com/tweet?id=1378452375296442371
  • Here's the relevant part of the JSON returned:
{
  "video": {
    "aspectRatio": [
      249,
      220
    ],
    "contentType": "media_entity",
    "durationMs": 9934,
    "mediaAvailability": {
      "status": "available"
    },
    "poster": "https://pbs.twimg.com/ext_tw_video_thumb/1378452274570264576/pu/img/ytz1zFSu4mFOGGI_.jpg",
    "variants": [
      {
        "type": "video/mp4",
        "src": "https://video.twimg.com/ext_tw_video/1378452274570264576/pu/vid/498x440/s2-MGvS6ilwspHei.mp4?tag=12"
      },
      {
        "type": "application/x-mpegURL",
        "src": "https://video.twimg.com/ext_tw_video/1378452274570264576/pu/pl/bScvTVcujd7bUNjO.m3u8?tag=12"
      },
      {
        "type": "video/mp4",
        "src": "https://video.twimg.com/ext_tw_video/1378452274570264576/pu/vid/304x270/0xP7m72L6iuRzaht.mp4?tag=12"
      }
    ],
    "videoId": {
      "type": "tweet",
      "id": "1378452375296442371"
    },
    "viewCount": 473
  },
}
  • Currently debugging the static-tweets fetcher and parser to figure out how to fix it. Looks promising though. I'm only worried about CORS-type issues, but maybe there's still a workaround by using iframes or similar. Will continue this afternoon.

Will keep updating on this issue, and create a draft PR once I start having actual changes to push.

@transitive-bullshit
Copy link
Owner

That's because the video's not there in the HTML in the first place (likely Twitter API change).

One option would be to switch from using cheerio to puppeteer.. but I would really like to avoid this if at all possible since puppeteer is more heavyweight and introduces difficulties in serverless environments.

but maybe there's still a workaround by using iframes or similar

Let's avoid this if at all possible. What CORS issues are you referring to? I can't imagine there would be any issues playing the video from the client-side if we already have the video URL.

We circumvent all the CORS stuff in general by fetching the tweet info + metadata server-side where CORS doesn't exist 😄

@DaniGuardiola
Copy link

I am gonna get the data from the new source. I've found the endpoint that provides that data now (I got it by reverse-engineering the official tweet embeds), so I'll just add it.

We circumvent all the CORS stuff in general by fetching the tweet info + metadata server-side where CORS doesn't exist

The video is precisely the one thing that isn't requested by the server, but it'll probably be fine.

One option would be to switch from using cheerio to puppeteer.. but I would really like to avoid this

Puppeteer is definitely too heavy, I agree. Cheerio is okay. Btw, I've been thinking about revamping the scraper with surgeon, I used to work on scraping and this tool is pretty amazing. But that's for a different time :P

Fortunately, this has a pretty straightforward solution that doesn't involve changing the HTML parsing :D

@DaniGuardiola
Copy link

I got it, but doing this I realized that the code is indeed due for a bit of refactoring, especially since the changes in the Twitter API. We could release an experimental version or publish it under something like unstable_fetchTweetAst named export if we want to ship it right away.

I'll push the working code later and create a PR. @iMuzz you should be able to use it directly from GitHub temporarily if it is urgent.

@transitive-bullshit
Copy link
Owner

@DaniGuardiola try to keep the refactoring and feature additions as separate as possible (ideally in separate PRs).

@DaniGuardiola DaniGuardiola linked a pull request Apr 7, 2021 that will close this issue
@kevcodez
Copy link

kevcodez commented Apr 26, 2021

Any updates on this @DaniGuardiola ?

@DaniGuardiola
Copy link

@kevcodez I got it working but as mentioned in my last comment, it'd be better imo to do a more global refactor of the AST fetching code, because my current solution duplicates the calls to Twitter's API and I think there might be a way to avoid it.

Maybe we could ship this under an option so that it's opt-in for users who don't mind the cost (extra requests could lead to problems with Twitter's API rate-limits). Sort of like an experimental feature flag. What do you think @transitive-bullshit?

@kevcodez if you need to use this right now, feel free to build the package from my branch yourself, but be aware of the trade-offs. Also, although it works just fine, it hasn't been tested thoroughly so it could break. Keep that in mind.

@remorses
Copy link

Gifs aren't rendered too, here is an example https://react-static-tweets.vercel.app/1461738430698860544

@kasem-sm
Copy link

any updates?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants