Fix multiple video audio overlay bug and remove GSPlayer dependency

[bryanmontz]

Although you can't hear audio in this, you can see that the videos pause and play automatically as you scroll them into and out of the viewport.

Closes #1386

[bryanmontz]

@jb55 @kroucis Please review. I didn't have a note that had multiple videos in a carousel. I'm guessing that won't work without a little more effort. But this should be close.

[jb55]

Reviewed-by: William Casarin ***@***.***>

…

On Wed, Sep 06, 2023 at 11:16:14AM -0500, Bryan Montz wrote: Closes: #1539 --- damus.xcodeproj/project.pbxproj | 4 ++++ damus/Views/Video/AVPlayerView.swift | 31 ++++++++++++++++++++++++++++ 2 files changed, 35 insertions(+) create mode 100644 damus/Views/Video/AVPlayerView.swift diff --git a/damus.xcodeproj/project.pbxproj b/damus.xcodeproj/project.pbxproj index 82e92fc81..25bec02cf 100644 --- a/damus.xcodeproj/project.pbxproj +++ b/damus.xcodeproj/project.pbxproj @@ -388,6 +388,7 @@ 504323A72A34915F006AE6DC /* RelayModel.swift in Sources */ = {isa = PBXBuildFile; fileRef = 504323A62A34915F006AE6DC /* RelayModel.swift */; }; 504323A92A3495B6006AE6DC /* RelayModelCache.swift in Sources */ = {isa = PBXBuildFile; fileRef = 504323A82A3495B6006AE6DC /* RelayModelCache.swift */; }; 5053ACA72A56DF3B00851AE3 /* DeveloperSettingsView.swift in Sources */ = {isa = PBXBuildFile; fileRef = 5053ACA62A56DF3B00851AE3 /* DeveloperSettingsView.swift */; }; + 50A16FFB2AA6C06600DFEC1F /* AVPlayerView.swift in Sources */ = {isa = PBXBuildFile; fileRef = 50A16FFA2AA6C06600DFEC1F /* AVPlayerView.swift */; }; 50A50A8D29A09E1C00C01BE7 /* RequestTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 50A50A8C29A09E1C00C01BE7 /* RequestTests.swift */; }; 50A60D142A28BEEE00186190 /* RelayLog.swift in Sources */ = {isa = PBXBuildFile; fileRef = 50A60D132A28BEEE00186190 /* RelayLog.swift */; }; 50B5685329F97CB400A23243 /* CredentialHandler.swift in Sources */ = {isa = PBXBuildFile; fileRef = 50B5685229F97CB400A23243 /* CredentialHandler.swift */; }; @@ -1063,6 +1064,7 @@ 504323A62A34915F006AE6DC /* RelayModel.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = RelayModel.swift; sourceTree = "<group>"; }; 504323A82A3495B6006AE6DC /* RelayModelCache.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = RelayModelCache.swift; sourceTree = "<group>"; }; 5053ACA62A56DF3B00851AE3 /* DeveloperSettingsView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = DeveloperSettingsView.swift; sourceTree = "<group>"; }; + 50A16FFA2AA6C06600DFEC1F /* AVPlayerView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = AVPlayerView.swift; sourceTree = "<group>"; }; 50A50A8C29A09E1C00C01BE7 /* RequestTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = RequestTests.swift; sourceTree = "<group>"; }; 50A60D132A28BEEE00186190 /* RelayLog.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = RelayLog.swift; sourceTree = "<group>"; }; 50B5685229F97CB400A23243 /* CredentialHandler.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = CredentialHandler.swift; sourceTree = "<group>"; }; @@ -1359,6 +1361,7 @@ children = ( 4C1A9A2929DDF54400516EAC /* DamusVideoPlayer.swift */, 4CCF9AAE2A1FDBDB00E03CFB /* VideoPlayer.swift */, + 50A16FFA2AA6C06600DFEC1F /* AVPlayerView.swift */, ); path = Video; sourceTree = "<group>"; @@ -2795,6 +2798,7 @@ 4C3EA63D28FF52D600C48A62 /* bolt11.c in Sources */, 4C9BB83129C0ED4F00FC4E37 /* DisplayName.swift in Sources */, 7CFF6317299FEFE5005D382A /* SelectableText.swift in Sources */, + 50A16FFB2AA6C06600DFEC1F /* AVPlayerView.swift in Sources */, 4CA352A82A76B37E003BB08B /* NewMutesNotify.swift in Sources */, 4CFF8F6929CC9ED1008DB934 /* ImageContainerView.swift in Sources */, 7527271E2A93FF0100214108 /* Block.swift in Sources */, diff --git a/damus/Views/Video/AVPlayerView.swift b/damus/Views/Video/AVPlayerView.swift new file mode 100644 index 000000000..7a69aef78 --- /dev/null +++ b/damus/Views/Video/AVPlayerView.swift @@ -0,0 +1,31 @@ +// +// AVPlayerView.swift +// damus +// +// Created by Bryan Montz on 9/4/23. +// + +import Foundation +import AVKit +import SwiftUI + +struct AVPlayerView: UIViewControllerRepresentable { + + let player: AVPlayer + + func makeUIViewController(context: Context) -> AVPlayerViewController { + AVPlayerViewController() + } + + func updateUIViewController(_ uiViewController: AVPlayerViewController, context: Context) { + if uiViewController.player == nil { + uiViewController.player = player + player.play() + } + } + + static func dismantleUIViewController(_ uiViewController: AVPlayerViewController, coordinator: ()) { + uiViewController.player?.pause() + uiViewController.player = nil + } +}

[jb55]

Reviewed-by: William Casarin ***@***.***>

…

On Wed, Sep 06, 2023 at 11:25:07AM -0500, Bryan Montz wrote: Closes: #1539 --- damus.xcodeproj/project.pbxproj | 4 + .../Video/DamusVideoPlayerViewModel.swift | 105 ++++++++++++++++++ 2 files changed, 109 insertions(+) create mode 100644 damus/Views/Video/DamusVideoPlayerViewModel.swift diff --git a/damus.xcodeproj/project.pbxproj b/damus.xcodeproj/project.pbxproj index d78d57f64..a4e198b08 100644 --- a/damus.xcodeproj/project.pbxproj +++ b/damus.xcodeproj/project.pbxproj @@ -389,6 +389,7 @@ 504323A92A3495B6006AE6DC /* RelayModelCache.swift in Sources */ = {isa = PBXBuildFile; fileRef = 504323A82A3495B6006AE6DC /* RelayModelCache.swift */; }; 5053ACA72A56DF3B00851AE3 /* DeveloperSettingsView.swift in Sources */ = {isa = PBXBuildFile; fileRef = 5053ACA62A56DF3B00851AE3 /* DeveloperSettingsView.swift */; }; 50A16FFB2AA6C06600DFEC1F /* AVPlayerView.swift in Sources */ = {isa = PBXBuildFile; fileRef = 50A16FFA2AA6C06600DFEC1F /* AVPlayerView.swift */; }; + 50A16FFD2AA7525700DFEC1F /* DamusVideoPlayerViewModel.swift in Sources */ = {isa = PBXBuildFile; fileRef = 50A16FFC2AA7525700DFEC1F /* DamusVideoPlayerViewModel.swift */; }; 50A16FFF2AA76A0900DFEC1F /* VideoController.swift in Sources */ = {isa = PBXBuildFile; fileRef = 50A16FFE2AA76A0900DFEC1F /* VideoController.swift */; }; 50A50A8D29A09E1C00C01BE7 /* RequestTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 50A50A8C29A09E1C00C01BE7 /* RequestTests.swift */; }; 50A60D142A28BEEE00186190 /* RelayLog.swift in Sources */ = {isa = PBXBuildFile; fileRef = 50A60D132A28BEEE00186190 /* RelayLog.swift */; }; @@ -1066,6 +1067,7 @@ 504323A82A3495B6006AE6DC /* RelayModelCache.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = RelayModelCache.swift; sourceTree = "<group>"; }; 5053ACA62A56DF3B00851AE3 /* DeveloperSettingsView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = DeveloperSettingsView.swift; sourceTree = "<group>"; }; 50A16FFA2AA6C06600DFEC1F /* AVPlayerView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = AVPlayerView.swift; sourceTree = "<group>"; }; + 50A16FFC2AA7525700DFEC1F /* DamusVideoPlayerViewModel.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = DamusVideoPlayerViewModel.swift; sourceTree = "<group>"; }; 50A16FFE2AA76A0900DFEC1F /* VideoController.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = VideoController.swift; sourceTree = "<group>"; }; 50A50A8C29A09E1C00C01BE7 /* RequestTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = RequestTests.swift; sourceTree = "<group>"; }; 50A60D132A28BEEE00186190 /* RelayLog.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = RelayLog.swift; sourceTree = "<group>"; }; @@ -1362,6 +1364,7 @@ isa = PBXGroup; children = ( 4C1A9A2929DDF54400516EAC /* DamusVideoPlayer.swift */, + 50A16FFC2AA7525700DFEC1F /* DamusVideoPlayerViewModel.swift */, 50A16FFE2AA76A0900DFEC1F /* VideoController.swift */, 4CCF9AAE2A1FDBDB00E03CFB /* VideoPlayer.swift */, 50A16FFA2AA6C06600DFEC1F /* AVPlayerView.swift */, @@ -2733,6 +2736,7 @@ 4CB88393296F798300DC99E7 /* ReactionsModel.swift in Sources */, 5C42E78C29DB76D90086AAC1 /* EmptyUserSearchView.swift in Sources */, 4CB88396296F7F8B00DC99E7 /* ReactionView.swift in Sources */, + 50A16FFD2AA7525700DFEC1F /* DamusVideoPlayerViewModel.swift in Sources */, 4CFF8F6B29CD0079008DB934 /* RepostedEvent.swift in Sources */, 4C8682872814DE470026224F /* ProfileView.swift in Sources */, 5C0707D12A1ECB38004E7B51 /* DamusLogoGradient.swift in Sources */, diff --git a/damus/Views/Video/DamusVideoPlayerViewModel.swift b/damus/Views/Video/DamusVideoPlayerViewModel.swift new file mode 100644 index 000000000..09551a272 --- /dev/null +++ b/damus/Views/Video/DamusVideoPlayerViewModel.swift @@ -0,0 +1,105 @@ +// +// DamusVideoPlayerViewModel.swift +// damus +// +// Created by Bryan Montz on 9/5/23. +// + +import AVFoundation +import Combine +import Foundation +import SwiftUI + ***@***.*** +final class DamusVideoPlayerViewModel: ObservableObject { + + private let url: URL + private let player_item: AVPlayerItem + let player: AVPlayer + private let controller: VideoController + let id = UUID() + + @published var has_audio = false + @binding var video_size: CGSize? + @published var is_muted = true + @published var is_loading = true + + private var cancellables = Set<AnyCancellable>() + + private var is_scrolled_into_view = false { + didSet { + if is_scrolled_into_view && !oldValue { + // we have just scrolled from out of view into view + controller.focused_model_id = id + } else if !is_scrolled_into_view && oldValue { + // we have just scrolled from in view to out of view + if controller.focused_model_id == id { + controller.focused_model_id = nil + } + } + } + } + + init(url: URL, video_size: Binding<CGSize?>, controller: VideoController) { + self.url = url + player_item = AVPlayerItem(url: url) + player = AVPlayer(playerItem: player_item) + self.controller = controller + _video_size = video_size + + Task { + await load() + } + + is_muted = controller.should_mute_video(url: url) + player.isMuted = is_muted + + NotificationCenter.default.addObserver( + self, + selector: #selector(did_play_to_end), + name: Notification.Name.AVPlayerItemDidPlayToEndTime, + object: player_item + ) + + controller.$focused_model_id + .sink { [weak self] model_id in + model_id == self?.id ? self?.player.play() : self?.player.pause() + } + .store(in: &cancellables) + } + + private func load() async { + if let meta = controller.metadata(for: url) { + has_audio = meta.has_audio + video_size = meta.size + } else { + has_audio = await video_has_audio(player: player) + if let video_size = await get_video_size(player: player) { + self.video_size = video_size + let meta = VideoMetadata(has_audio: has_audio, size: video_size) + controller.set_metadata(meta, url: url) + } + } + + is_loading = false + } + + func did_tap_mute_button() { + is_muted.toggle() + player.isMuted = is_muted + controller.toggle_should_mute_video(url: url) + } + + func set_view_is_visible(_ is_visible: Bool) { + is_scrolled_into_view = is_visible + } + + func view_did_disappear() { + set_view_is_visible(false) + } + + @objc private func did_play_to_end() { + player.seek(to: CMTime.zero) + player.play() + } +}

[jb55]

Reviewed-by: William Casarin ***@***.***>

…

On Wed, Sep 06, 2023 at 11:29:45AM -0500, Bryan Montz wrote: Closes: #1539 --- damus/Components/ImageCarousel.swift | 8 +- damus/Views/Images/ImageContainerView.swift | 6 +- damus/Views/Images/ImageView.swift | 6 +- damus/Views/Video/DamusVideoPlayer.swift | 116 ++++++++++---------- 4 files changed, 69 insertions(+), 67 deletions(-) diff --git a/damus/Components/ImageCarousel.swift b/damus/Components/ImageCarousel.swift index f99b67295..c7553518b 100644 --- a/damus/Components/ImageCarousel.swift +++ b/damus/Components/ImageCarousel.swift @@ -112,10 +112,6 @@ struct ImageCarousel: View { } } - func video_model(_ url: URL) -> VideoPlayerModel { - return state.events.get_video_player_model(url: url) - } - func Media(geo: GeometryProxy, url: MediaUrl, index: Int) -> some View { Group { switch url { @@ -125,7 +121,7 @@ struct ImageCarousel: View { open_sheet = true } case .video(let url): - DamusVideoPlayer(url: url, model: video_model(url), video_size: $video_size) + DamusVideoPlayer(url: url, video_size: $video_size, controller: state.video) .onChange(of: video_size) { size in guard let size else { return } @@ -194,7 +190,7 @@ struct ImageCarousel: View { } .tabViewStyle(PageTabViewStyle(indexDisplayMode: .never)) .fullScreenCover(isPresented: $open_sheet) { - ImageView(cache: state.events, urls: urls, disable_animation: state.settings.disable_animation) + ImageView(video_controller: state.video, urls: urls, disable_animation: state.settings.disable_animation) } .frame(height: height) .onChange(of: selectedIndex) { value in diff --git a/damus/Views/Images/ImageContainerView.swift b/damus/Views/Images/ImageContainerView.swift index cd4499a79..64396ad30 100644 --- a/damus/Views/Images/ImageContainerView.swift +++ b/damus/Views/Images/ImageContainerView.swift @@ -10,7 +10,7 @@ import Kingfisher struct ImageContainerView: View { - let cache: EventCache + let video_controller: VideoController let url: MediaUrl @State private var image: UIImage? @@ -47,7 +47,7 @@ struct ImageContainerView: View { case .image(let url): Img(url: url) case .video(let url): - DamusVideoPlayer(url: url, model: cache.get_video_player_model(url: url), video_size: .constant(nil)) + DamusVideoPlayer(url: url, video_size: .constant(nil), controller: video_controller) } } } @@ -57,6 +57,6 @@ let test_image_url = URL(string: "https://jb55.com/red-me.jpg")! struct ImageContainerView_Previews: PreviewProvider { static var previews: some View { - ImageContainerView(cache: test_damus_state().events, url: .image(test_image_url), disable_animation: false) + ImageContainerView(video_controller: test_damus_state().video, url: .image(test_image_url), disable_animation: false) } } diff --git a/damus/Views/Images/ImageView.swift b/damus/Views/Images/ImageView.swift index a51133c79..850dc583d 100644 --- a/damus/Views/Images/ImageView.swift +++ b/damus/Views/Images/ImageView.swift @@ -8,7 +8,7 @@ import SwiftUI struct ImageView: View { - let cache: EventCache + let video_controller: VideoController let urls: [MediaUrl] @Environment(\.presentationMode) var presentationMode @@ -39,7 +39,7 @@ struct ImageView: View { TabView(selection: $selectedIndex) { ForEach(urls.indices, id: \.self) { index in ZoomableScrollView { - ImageContainerView(cache: cache, url: urls[index], disable_animation: disable_animation) + ImageContainerView(video_controller: video_controller, url: urls[index], disable_animation: disable_animation) .aspectRatio(contentMode: .fit) .padding(.top, Theme.safeAreaInsets?.top) .padding(.bottom, Theme.safeAreaInsets?.bottom) @@ -80,6 +80,6 @@ struct ImageView: View { struct ImageView_Previews: PreviewProvider { static var previews: some View { let url: MediaUrl = .image(URL(string: "https://jb55.com/red-me.jpg")!) - ImageView(cache: test_damus_state().events, urls: [url], disable_animation: false) + ImageView(video_controller: test_damus_state().video, urls: [url], disable_animation: false) } } diff --git a/damus/Views/Video/DamusVideoPlayer.swift b/damus/Views/Video/DamusVideoPlayer.swift index ca64e42ed..bbca2a375 100644 --- a/damus/Views/Video/DamusVideoPlayer.swift +++ b/damus/Views/Video/DamusVideoPlayer.swift @@ -17,79 +17,85 @@ func globalCoordinate(localX x: CGFloat, localY y: CGFloat, } struct DamusVideoPlayer: View { - var url: URL - @ObservedObject var model: VideoPlayerModel - @binding var video_size: CGSize? + let url: URL + @StateObject var model: DamusVideoPlayerViewModel @EnvironmentObject private var orientationTracker: OrientationTracker - var mute_icon: String { - if model.has_audio == false || model.muted { - return "speaker.slash" - } else { - return "speaker" - } - } - - var mute_icon_color: Color { - switch self.model.has_audio { - case .none: - return .white - case .some(let has_audio): - return has_audio ? .white : .red - } - } - - var MuteIcon: some View { - ZStack { - Circle() - .opacity(0.2) - .frame(width: 32, height: 32) - .foregroundColor(.black) - - Image(systemName: mute_icon) - .padding() - .foregroundColor(mute_icon_color) - } + init(url: URL, video_size: Binding<CGSize?>, controller: VideoController) { + self.url = url + _model = StateObject(wrappedValue: DamusVideoPlayerViewModel(url: url, video_size: video_size, controller: controller)) } var body: some View { GeometryReader { geo in let localFrame = geo.frame(in: .local) let centerY = globalCoordinate(localX: 0, localY: localFrame.midY, localGeometry: geo).y - let delta = localFrame.height / 2 - ZStack(alignment: .bottomTrailing) { - VideoPlayer(url: url, model: model) - if model.has_audio == true { - MuteIcon - .zIndex(11.0) - .onTapGesture { - self.model.muted = !self.model.muted - } + ZStack { + AVPlayerView(player: model.player) + + if model.is_loading { + ProgressView() + .progressViewStyle(.circular) + .tint(.white) + .scaleEffect(CGSize(width: 1.5, height: 1.5)) } - } - .onChange(of: model.size) { size in - guard let size else { - return + + if model.has_audio { + mute_button } - video_size = size } .onChange(of: centerY) { _ in - /// pause video when it is scrolled beyond visible range - let isBelowTop = centerY + delta > 100, /// 100 =~ approx. bottom (y) of ContentView's TabView - isAboveBottom = centerY - delta < orientationTracker.deviceMajorAxis - if isBelowTop && isAboveBottom { - model.start() - } else { - model.stop() + update_is_visible(centerY: centerY) + } + .onAppear { + update_is_visible(centerY: centerY) + } + } + .onDisappear { + model.view_did_disappear() + } + } + + private func update_is_visible(centerY: CGFloat) { + let isBelowTop = centerY > 100, /// 100 =~ approx. bottom (y) of ContentView's TabView + isAboveBottom = centerY < orientationTracker.deviceMajorAxis + model.set_view_is_visible(isBelowTop && isAboveBottom) + } + + private var mute_icon: String { + !model.has_audio || model.is_muted ? "speaker.slash" : "speaker" + } + + private var mute_icon_color: Color { + model.has_audio ? .white : .red + } + + private var mute_button: some View { + HStack { + Spacer() + VStack { + Spacer() + + Button { + model.did_tap_mute_button() + } label: { + ZStack { + Circle() + .opacity(0.2) + .frame(width: 32, height: 32) + .foregroundColor(.black) + + Image(systemName: mute_icon) + .padding() + .foregroundColor(mute_icon_color) + } } } } } } struct DamusVideoPlayer_Previews: PreviewProvider { - @StateObject static var model: VideoPlayerModel = VideoPlayerModel() - static var previews: some View { - DamusVideoPlayer(url: URL(string: "http://cdn.jb55.com/s/zaps-build.mp4")!, model: model, video_size: .constant(nil)) + DamusVideoPlayer(url: URL(string: "http://cdn.jb55.com/s/zaps-build.mp4")!, video_size: .constant(nil), controller: VideoController()) } }

[jb55]

On Wed, Sep 06, 2023 at 11:23:33AM -0500, Bryan Montz wrote: diff --git a/damus/ContentView.swift b/damus/ContentView.swift index bc9628a50..8782d5835 100644 --- a/damus/ContentView.swift +++ b/damus/ContentView.swift @@ -654,7 +654,8 @@ struct ContentView: View { wallet: WalletModel(settings: settings), nav: self.navigationCoordinator, user_search_cache: user_search_cache, - music: MusicController(onChange: music_changed) + music: MusicController(onChange: music_changed), + video: VideoController()

DamusState is getting so big! Starting to wonder if we'll need a CacheManager or something... Reviewed-by: William Casarin ***@***.***>

[jb55]

Reviewed-by: William Casarin ***@***.***>

…

On Wed, Sep 06, 2023 at 11:49:06AM -0500, Bryan Montz wrote: Closes: #1539 --- damus.xcodeproj/project.pbxproj | 8 +- damus/Components/ImageCarousel.swift | 2 +- damus/Util/AVPlayer+Additions.swift | 35 ++ damus/Util/EventCache.swift | 26 -- .../Video/DamusVideoPlayerViewModel.swift | 12 + damus/Views/Video/VideoController.swift | 4 + damus/Views/Video/VideoPlayer.swift | 350 ------------------ 7 files changed, 56 insertions(+), 381 deletions(-) create mode 100644 damus/Util/AVPlayer+Additions.swift delete mode 100644 damus/Views/Video/VideoPlayer.swift diff --git a/damus.xcodeproj/project.pbxproj b/damus.xcodeproj/project.pbxproj index a4e198b08..acf8a6806 100644 --- a/damus.xcodeproj/project.pbxproj +++ b/damus.xcodeproj/project.pbxproj @@ -317,7 +317,6 @@ 4CC7AAFA297F64AC00430951 /* EventMenu.swift in Sources */ = {isa = PBXBuildFile; fileRef = 4CC7AAF9297F64AC00430951 /* EventMenu.swift */; }; 4CCEB7AE29B53D260078AA28 /* SearchingEventView.swift in Sources */ = {isa = PBXBuildFile; fileRef = 4CCEB7AD29B53D260078AA28 /* SearchingEventView.swift */; }; 4CCEB7B029B5415A0078AA28 /* SearchingProfileView.swift in Sources */ = {isa = PBXBuildFile; fileRef = 4CCEB7AF29B5415A0078AA28 /* SearchingProfileView.swift */; }; - 4CCF9AAF2A1FDBDB00E03CFB /* VideoPlayer.swift in Sources */ = {isa = PBXBuildFile; fileRef = 4CCF9AAE2A1FDBDB00E03CFB /* VideoPlayer.swift */; }; 4CCF9AB22A1FE80C00E03CFB /* GSPlayer in Frameworks */ = {isa = PBXBuildFile; productRef = 4CCF9AB12A1FE80C00E03CFB /* GSPlayer */; }; 4CD348EF29C3659D00497EB2 /* ImageUploadModel.swift in Sources */ = {isa = PBXBuildFile; fileRef = 4CD348EE29C3659D00497EB2 /* ImageUploadModel.swift */; }; 4CD7641B28A1641400B6928F /* EndBlock.swift in Sources */ = {isa = PBXBuildFile; fileRef = 4CD7641A28A1641400B6928F /* EndBlock.swift */; }; @@ -394,6 +393,7 @@ 50A50A8D29A09E1C00C01BE7 /* RequestTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 50A50A8C29A09E1C00C01BE7 /* RequestTests.swift */; }; 50A60D142A28BEEE00186190 /* RelayLog.swift in Sources */ = {isa = PBXBuildFile; fileRef = 50A60D132A28BEEE00186190 /* RelayLog.swift */; }; 50B5685329F97CB400A23243 /* CredentialHandler.swift in Sources */ = {isa = PBXBuildFile; fileRef = 50B5685229F97CB400A23243 /* CredentialHandler.swift */; }; + 50C3E08A2AA8E3F7006A4BC0 /* AVPlayer+Additions.swift in Sources */ = {isa = PBXBuildFile; fileRef = 50C3E0892AA8E3F7006A4BC0 /* AVPlayer+Additions.swift */; }; 50DA11262A16A23F00236234 /* Launch.storyboard in Resources */ = {isa = PBXBuildFile; fileRef = 50DA11252A16A23F00236234 /* Launch.storyboard */; }; 5C0707D12A1ECB38004E7B51 /* DamusLogoGradient.swift in Sources */ = {isa = PBXBuildFile; fileRef = 5C0707D02A1ECB38004E7B51 /* DamusLogoGradient.swift */; }; 5C42E78C29DB76D90086AAC1 /* EmptyUserSearchView.swift in Sources */ = {isa = PBXBuildFile; fileRef = 5C42E78B29DB76D90086AAC1 /* EmptyUserSearchView.swift */; }; @@ -990,7 +990,6 @@ 4CC7AAF9297F64AC00430951 /* EventMenu.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = EventMenu.swift; sourceTree = "<group>"; }; 4CCEB7AD29B53D260078AA28 /* SearchingEventView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = SearchingEventView.swift; sourceTree = "<group>"; }; 4CCEB7AF29B5415A0078AA28 /* SearchingProfileView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = SearchingProfileView.swift; sourceTree = "<group>"; }; - 4CCF9AAE2A1FDBDB00E03CFB /* VideoPlayer.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = VideoPlayer.swift; sourceTree = "<group>"; }; 4CD348EE29C3659D00497EB2 /* ImageUploadModel.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ImageUploadModel.swift; sourceTree = "<group>"; }; 4CD7641A28A1641400B6928F /* EndBlock.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = EndBlock.swift; sourceTree = "<group>"; }; 4CDA128929E9D10C0006FA5A /* SignalView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = SignalView.swift; sourceTree = "<group>"; }; @@ -1072,6 +1071,7 @@ 50A50A8C29A09E1C00C01BE7 /* RequestTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = RequestTests.swift; sourceTree = "<group>"; }; 50A60D132A28BEEE00186190 /* RelayLog.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = RelayLog.swift; sourceTree = "<group>"; }; 50B5685229F97CB400A23243 /* CredentialHandler.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = CredentialHandler.swift; sourceTree = "<group>"; }; + 50C3E0892AA8E3F7006A4BC0 /* AVPlayer+Additions.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = "AVPlayer+Additions.swift"; sourceTree = "<group>"; }; 50DA11252A16A23F00236234 /* Launch.storyboard */ = {isa = PBXFileReference; lastKnownFileType = file.storyboard; path = Launch.storyboard; sourceTree = "<group>"; }; 5C0707D02A1ECB38004E7B51 /* DamusLogoGradient.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = DamusLogoGradient.swift; sourceTree = "<group>"; }; 5C42E78B29DB76D90086AAC1 /* EmptyUserSearchView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = EmptyUserSearchView.swift; sourceTree = "<group>"; }; @@ -1366,7 +1366,6 @@ 4C1A9A2929DDF54400516EAC /* DamusVideoPlayer.swift */, 50A16FFC2AA7525700DFEC1F /* DamusVideoPlayerViewModel.swift */, 50A16FFE2AA76A0900DFEC1F /* VideoController.swift */, - 4CCF9AAE2A1FDBDB00E03CFB /* VideoPlayer.swift */, 50A16FFA2AA6C06600DFEC1F /* AVPlayerView.swift */, ); path = Video; @@ -1791,6 +1790,7 @@ 3A8CC6CB2A2CFEF900940F5F /* StringUtil.swift */, D2277EE92A089BD5006C3807 /* Router.swift */, 4C2B10272A7B0F5C008AA43E /* Log.swift */, + 50C3E0892AA8E3F7006A4BC0 /* AVPlayer+Additions.swift */, ); path = Util; sourceTree = "<group>"; @@ -2507,7 +2507,6 @@ 4C216F34286F5ACD00040376 /* DMView.swift in Sources */, 4C32B9572A9AD44700DC3548 /* Root.swift in Sources */, 4C3EA64428FF558100C48A62 /* sha256.c in Sources */, - 4CCF9AAF2A1FDBDB00E03CFB /* VideoPlayer.swift in Sources */, 504323A72A34915F006AE6DC /* RelayModel.swift in Sources */, 4CA9276A2A290FC00098A105 /* ContextButton.swift in Sources */, 4CF0ABF62985CD5500D66079 /* UserSearch.swift in Sources */, @@ -2554,6 +2553,7 @@ 4C7D09742A0AEF9000943473 /* AlbyGradient.swift in Sources */, 4C687C272A6039500092C550 /* TestData.swift in Sources */, 3AA247FD297E3CFF0090C62D /* RepostsModel.swift in Sources */, + 50C3E08A2AA8E3F7006A4BC0 /* AVPlayer+Additions.swift in Sources */, 4C198DF229F88C6B004C165C /* BlurHashDecode.swift in Sources */, F75BA12F29A18EF500E10810 /* BookmarksView.swift in Sources */, 4CB883B6297730E400DC99E7 /* LNUrls.swift in Sources */, diff --git a/damus/Components/ImageCarousel.swift b/damus/Components/ImageCarousel.swift index c7553518b..811b9ff2b 100644 --- a/damus/Components/ImageCarousel.swift +++ b/damus/Components/ImageCarousel.swift @@ -105,7 +105,7 @@ struct ImageCarousel: View { } } .onAppear { - if self.image_fill == nil, let size = state.events.lookup_media_size(url: url) { + if self.image_fill == nil, let size = state.video.size_for_url(url) { let fill = ImageFill.calculate_image_fill(geo_size: geo_size, img_size: size, maxHeight: maxHeight, fillHeight: fillHeight) self.image_fill = fill } diff --git a/damus/Util/AVPlayer+Additions.swift b/damus/Util/AVPlayer+Additions.swift new file mode 100644 index 000000000..f870d9214 --- /dev/null +++ b/damus/Util/AVPlayer+Additions.swift @@ -0,0 +1,35 @@ +// +// AVPlayer+Additions.swift +// damus +// +// Created by Bryan Montz on 9/6/23. +// + +import AVFoundation +import Foundation +import UIKit + +extension AVPlayer { +#if !os(macOS) + var currentImage: UIImage? { + guard + let playerItem = currentItem, + let cgImage = try? AVAssetImageGenerator(asset: playerItem.asset).copyCGImage(at: currentTime(), actualTime: nil) + else { return nil } + + return UIImage(cgImage: cgImage) + } +#else + var currentImage: NSImage? { + guard + let playerItem = currentItem, + let cgImage = try? AVAssetImageGenerator(asset: playerItem.asset).copyCGImage(at: currentTime(), actualTime: nil) + else { + return nil + } + let width: CGFloat = CGFloat(cgImage.width) + let height: CGFloat = CGFloat(cgImage.height) + return NSImage(cgImage: cgImage, size: NSMakeSize(width, height)) + } +#endif +} diff --git a/damus/Util/EventCache.swift b/damus/Util/EventCache.swift index d48fea6c7..29d3229c4 100644 --- a/damus/Util/EventCache.swift +++ b/damus/Util/EventCache.swift @@ -141,7 +141,6 @@ class EventCache { private var replies = ReplyMap() private var cancellable: AnyCancellable? private var image_metadata: [String: ImageMetadataState] = [:] // lowercased URL key - private var video_meta: [URL: VideoPlayerModel] = [:] private var event_data: [NoteId: EventData] = [:] //private var thread_latest: [String: Int64] @@ -204,30 +203,6 @@ class EventCache { return image_metadata[url.absoluteString.lowercased()] } - @mainactor - func lookup_media_size(url: URL) -> CGSize? { - if let img_meta = lookup_img_metadata(url: url) { - return img_meta.meta.dim?.size - } - - return get_video_player_model(url: url).size - } - - func store_video_player_model(url: URL, meta: VideoPlayerModel) { - video_meta[url] = meta - } - - @mainactor - func get_video_player_model(url: URL) -> VideoPlayerModel { - if let model = video_meta[url] { - return model - } - - let model = VideoPlayerModel() - video_meta[url] = model - return model - } - func parent_events(event: NostrEvent, keypair: Keypair) -> [NostrEvent] { var parents: [NostrEvent] = [] @@ -289,7 +264,6 @@ class EventCache { private func prune() { events = [:] - video_meta = [:] event_data = [:] replies.replies = [:] } diff --git a/damus/Views/Video/DamusVideoPlayerViewModel.swift b/damus/Views/Video/DamusVideoPlayerViewModel.swift index 09551a272..761b8a4d5 100644 --- a/damus/Views/Video/DamusVideoPlayerViewModel.swift +++ b/damus/Views/Video/DamusVideoPlayerViewModel.swift @@ -10,6 +10,18 @@ import Combine import Foundation import SwiftUI +func get_video_size(player: AVPlayer) async -> CGSize? { + let res = Task.detached(priority: .background) { + return player.currentImage?.size + } + return await res.value +} + +func video_has_audio(player: AVPlayer) async -> Bool { + let tracks = try? await player.currentItem?.asset.load(.tracks) + return tracks?.filter({ t in t.mediaType == .audio }).first != nil +} + @mainactor final class DamusVideoPlayerViewModel: ObservableObject { diff --git a/damus/Views/Video/VideoController.swift b/damus/Views/Video/VideoController.swift index 8f524976c..9377d5cb4 100644 --- a/damus/Views/Video/VideoController.swift +++ b/damus/Views/Video/VideoController.swift @@ -37,4 +37,8 @@ final class VideoController: ObservableObject { func metadata(for url: URL) -> VideoMetadata? { metadatas[url] } + + func size_for_url(_ url: URL) -> CGSize? { + metadatas[url]?.size + } } diff --git a/damus/Views/Video/VideoPlayer.swift b/damus/Views/Video/VideoPlayer.swift deleted file mode 100644 index 2a0a04d49..000000000 --- a/damus/Views/Video/VideoPlayer.swift +++ /dev/null @@ -1,350 +0,0 @@ -// -// VideoPlayer.swift -// damus -// -// Created by William Casarin on 2023-05-25. -// - -import Foundation -// -// VideoPlayer.swift -// VideoPlayer -// -// Created by Gesen on 2019/7/7. -// Copyright © 2019 Gesen. All rights reserved. -// - -import AVFoundation -import GSPlayer -import SwiftUI - -public enum VideoState { - /// From the first load to get the first frame of the video - case loading - - /// Playing now - case playing(totalDuration: Double) - - /// Pause, will be called repeatedly when the buffer progress changes - case paused(playProgress: Double, bufferProgress: Double) - - /// An error occurred and cannot continue playing - case error(NSError) -} - -enum VideoHandler { - case onBufferChanged((Double) -> Void) - case onPlayToEndTime(() -> Void) - case onReplay(() -> Void) - case onStateChanged((VideoState) -> Void) -} - ***@***.*** -public class VideoPlayerModel: ObservableObject { - @published var autoReplay: Bool = true - @published var muted: Bool = true - @published var play: Bool = true - @published var size: CGSize? = nil - @published var has_audio: Bool? = nil - @published var contentMode: UIView.ContentMode = .scaleAspectFill - - fileprivate var time: CMTime? - - var handlers: [VideoHandler] = [] - - init() { - } - - func stop() { - self.play = false - } - - func start() { - self.play = true - } - - func mute() { - self.muted = true - } - - func unmute() { - self.muted = false - } - - /// Whether the video will be automatically replayed until the end of the video playback. - func autoReplay(_ value: Bool) -> Self { - autoReplay = value - return self - } - - /// Whether the video is muted, only for this instance. - func mute(_ value: Bool) -> Self { - muted = value - return self - } - - /// A string defining how the video is displayed within an AVPlayerLayer bounds rect. - /// scaleAspectFill -> resizeAspectFill, scaleAspectFit -> resizeAspect, other -> resize - func contentMode(_ value: UIView.ContentMode) -> Self { - contentMode = value - return self - } - - /// Trigger a callback when the buffer progress changes, - /// the value is between 0 and 1. - func onBufferChanged(_ handler: @escaping (Double) -> Void) -> Self { - self.handlers.append(.onBufferChanged(handler)) - return self - } - - /// Playing to the end. - func onPlayToEndTime(_ handler: @escaping () -> Void) -> Self { - self.handlers.append(.onPlayToEndTime(handler)) - return self - } - - /// Replay after playing to the end. - func onReplay(_ handler: @escaping () -> Void) -> Self { - self.handlers.append(.onReplay(handler)) - return self - } - - /// Playback status changes, such as from play to pause. - func onStateChanged(_ handler: @escaping (VideoState) -> Void) -> Self { - self.handlers.append(.onStateChanged(handler)) - return self - } -} - ***@***.***(iOS 13, *) -public struct VideoPlayer { - private(set) var url: URL - - @ObservedObject var model: VideoPlayerModel - - /// Init video player instance. - /// - Parameters: - /// - url: http/https URL - /// - play: play/pause - /// - time: current time - public init(url: URL, model: VideoPlayerModel) { - self.url = url - self._model = ObservedObject(wrappedValue: model) - } -} - ***@***.***(iOS 13, *) -public extension VideoPlayer { - - /// Set the preload size, the default value is 1024 * 1024, unit is byte. - static var preloadByteCount: Int { - get { VideoPreloadManager.shared.preloadByteCount } - set { VideoPreloadManager.shared.preloadByteCount = newValue } - } - - /// Set the video urls to be preload queue. - /// Preloading will automatically cache a short segment of the beginning of the video - /// and decide whether to start or pause the preload based on the buffering of the currently playing video. - /// - Parameter urls: URL array - static func preload(urls: [URL]) { - VideoPreloadManager.shared.set(waiting: urls) - } - - /// Set custom http header, such as token. - static func customHTTPHeaderFields(transform: @escaping (URL) -> [String: String]?) { - VideoLoadManager.shared.customHTTPHeaderFields = transform - } - - /// Get the total size of the video cache. - static func calculateCachedSize() -> UInt { - return VideoCacheManager.calculateCachedSize() - } - - /// Clean up all caches. - static func cleanAllCache() { - try? VideoCacheManager.cleanAllCache() - } -} - -func get_video_size(player: AVPlayer) async -> CGSize? { - let res = Task.detached(priority: .background) { - return player.currentImage?.size - } - return await res.value -} - -func video_has_audio(player: AVPlayer) async -> Bool { - let tracks = try? await player.currentItem?.asset.load(.tracks) - return tracks?.filter({ t in t.mediaType == .audio }).first != nil -} - ***@***.***(iOS 13, *) -extension VideoPlayer: UIViewRepresentable { - - public func makeUIView(context: Context) -> VideoPlayerView { - let uiView = VideoPlayerView() - - uiView.playToEndTime = { - if self.model.autoReplay == false { - self.model.play = false - } - DispatchQueue.main.async { - for handler in model.handlers { - if case .onPlayToEndTime(let cb) = handler { - cb() - } - } - } - } - - uiView.contentMode = self.model.contentMode - - uiView.replay = { - DispatchQueue.main.async { - for handler in model.handlers { - if case .onReplay(let cb) = handler { - cb() - } - } - } - } - - uiView.stateDidChanged = { [unowned uiView] _ in - let state: VideoState = uiView.convertState() - - if case .playing = state { - context.coordinator.startObserver(uiView: uiView) - - if let player = uiView.player { - Task { - let has_audio = await video_has_audio(player: player) - let size = await get_video_size(player: player) - Task { @mainactor in - if let size { - self.model.size = size - } - self.model.has_audio = has_audio - } - } - } - - } else { - context.coordinator.stopObserver(uiView: uiView) - } - - DispatchQueue.main.async { - for handler in model.handlers { - if case .onStateChanged(let cb) = handler { - cb(state) - } - } - } - } - - return uiView - } - - public func makeCoordinator() -> Coordinator { - Coordinator(self) - } - - public func updateUIView(_ uiView: VideoPlayerView, context: Context) { - if context.coordinator.observingURL != url { - context.coordinator.clean() - context.coordinator.observingURL = url - } - - if model.play { - uiView.play(for: url) - } else { - uiView.pause(reason: .userInteraction) - } - - uiView.isMuted = model.muted - uiView.isAutoReplay = model.autoReplay - - if let observerTime = context.coordinator.observerTime, let modelTime = model.time, - modelTime != observerTime && modelTime.isValid && modelTime.isNumeric { - uiView.seek(to: modelTime, completion: { _ in }) - } - } - - public static func dismantleUIView(_ uiView: VideoPlayerView, coordinator: VideoPlayer.Coordinator) { - uiView.pause(reason: .hidden) - } - - public class Coordinator: NSObject { - var videoPlayer: VideoPlayer - var observingURL: URL? - var observer: Any? - var observerTime: CMTime? - var observerBuffer: Double? - - init(_ videoPlayer: VideoPlayer) { - self.videoPlayer = videoPlayer - } - - @mainactor - func startObserver(uiView: VideoPlayerView) { - guard observer == nil else { return } - - observer = uiView.addPeriodicTimeObserver(forInterval: .init(seconds: 0.25, preferredTimescale: 60)) { [weak self, unowned uiView] time in - guard let `self` = self else { return } - - Task { @mainactor in - self.videoPlayer.model.time = time - } - self.observerTime = time - - self.updateBuffer(uiView: uiView) - } - } - - func stopObserver(uiView: VideoPlayerView) { - guard let observer = observer else { return } - - uiView.removeTimeObserver(observer) - - self.observer = nil - } - - func clean() { - self.observingURL = nil - self.observer = nil - self.observerTime = nil - self.observerBuffer = nil - } - - @mainactor - func updateBuffer(uiView: VideoPlayerView) { - let bufferProgress = uiView.bufferProgress - guard bufferProgress != observerBuffer else { return } - - for handler in videoPlayer.model.handlers { - if case .onBufferChanged(let cb) = handler { - DispatchQueue.main.async { - cb(bufferProgress) - } - } - } - - observerBuffer = bufferProgress - } - } -} - -private extension VideoPlayerView { - - func convertState() -> VideoState { - switch state { - case .none, .loading: - return .loading - case .playing: - return .playing(totalDuration: totalDuration) - case .paused(let p, let b): - return .paused(playProgress: p, bufferProgress: b) - case .error(let error): - return .error(error) - } - } -}

[jb55]

On Wed, Sep 06, 2023 at 10:09:38AM -0700, Bryan Montz wrote: @jb55 @kroucis Please review. I didn't have a note that had multiple videos in a carousel. I'm guessing that won't work without a little more effort. But this should be close.

I'm running it now. It seems to be working much better. Will keep testing to see if I run into anything weird.

[shawnyeager]

I see issue is closed, but @jb55 please confirm author for bounty payout. 🙏🏼

[jb55]

On Thu, Sep 07, 2023 at 11:32:07AM -0700, Shawn Yeager wrote: I see issue is closed, but @jb55 please confirm author for bounty payout. 🙏🏼

Montz npub1qlk0nqupxmlyxravg0aqscxmcc4q4tq898z6x003rykwwh3npj0syvyayc LNURL1DP68GURN8GHJ7AMPD3KX2AR0VEEKZAR0WD5XJTNRDAKJ7TNHV4KXCTTTDEHHWM30D3H82UNVWQHKWCTNWP5KUEMNDAUNSDQ7Y4KMJ

[shawnyeager]

Paid.

https://damus.io/note143x77x7xfvvyxqquwag464mpk954uw2djhj2camprmhxnkc47z0quzuw6u