Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Megaphone API publishing integration #1112

Draft
wants to merge 82 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
82 commits
Select commit Hold shift + click to select a range
b78a7a0
Add link header parser
kookster Oct 1, 2024
0fd257f
Add in active record encryption config
kookster Oct 1, 2024
03e5b7e
Add in megaphone feed and config
kookster Oct 1, 2024
8cb9e11
starter tests for megaphone config
kookster Oct 1, 2024
90083a2
Initial megaphone api support
kookster Oct 1, 2024
972f67c
Podcast megaphone api implementation
kookster Oct 1, 2024
4c76d44
Fix test
kookster Oct 1, 2024
9ca679c
Linting fixes
kookster Oct 1, 2024
844fb20
fix that test with an actual feed
kookster Oct 1, 2024
a8f1c18
Merge
kookster Oct 1, 2024
6341784
Start of episode, basic tests
kookster Oct 3, 2024
82f2bee
set external id to dt episode guid
kookster Oct 3, 2024
e2fa41d
Fix some bugs on link header handling
kookster Oct 8, 2024
03ef603
Initial episode attribute mapping
kookster Oct 8, 2024
b405ab2
Set some defaults for encryption in test
kookster Oct 8, 2024
edcb50f
Merge branch 'main' into feat/megaphone
kookster Oct 9, 2024
cc836de
Merge branch 'main' into feat/megaphone
kookster Oct 10, 2024
0eff44c
Prototype integrations interface
svevang Oct 21, 2024
c5873e0
Move prx access to an api module
kookster Oct 22, 2024
1c8a5e4
Move prx access, remove announce
kookster Oct 22, 2024
29f5463
Use private_feed arg
kookster Oct 22, 2024
44d5f29
Include all the attributes
kookster Oct 22, 2024
cb95dbb
Update gems
kookster Oct 22, 2024
7d6e782
Merge branch 'main' into feat/megaphone
kookster Oct 22, 2024
0be1b5d
merge main
kookster Oct 25, 2024
b24c264
In progress work on saving episodes to megaphone
kookster Oct 29, 2024
7670bae
Get the basic augury test working
kookster Oct 29, 2024
ced2219
prettier
kookster Oct 29, 2024
92f8b64
Merge branch 'main' into feat/megaphone
kookster Oct 29, 2024
a2e0ff8
Use env vars for urls
kookster Oct 29, 2024
59d192a
Clarify naming and extract episode status instance methods
svevang Nov 4, 2024
9da9dd2
Fix quirk on soft deleted dependent destroy
svevang Oct 29, 2024
2aa0e4d
Move in related methods
svevang Oct 29, 2024
fd6c5cf
Break up method refactor
svevang Oct 29, 2024
7ea7896
Add the uploaded state marker
svevang Oct 29, 2024
f2fc129
Break out the upload flow
svevang Oct 30, 2024
7d7ead7
Split out coverage for update reference
svevang Oct 31, 2024
0a94b59
fixup! Add the uploaded state marker
svevang Oct 31, 2024
f9793b2
Break out methods for testing
svevang Oct 31, 2024
a6dfab1
Fix return value
svevang Nov 5, 2024
7ba1c29
Remove dupe methods
svevang Nov 5, 2024
53a95a0
Coverage for apple_prepare_for_delivery
svevang Nov 5, 2024
3cc10c8
Bifucated uploaded/delivered states match
svevang Nov 5, 2024
8ff26c7
Merge branch 'main' into feat/megaphone
kookster Nov 23, 2024
77a9750
Merge main
kookster Nov 23, 2024
9b5aff3
Merge branch 'retry-asset-wait-file-processing-timeouts' into feat/me…
kookster Nov 23, 2024
3454fe6
Merge branch 'integrations-interface' into feat/megaphone
kookster Nov 25, 2024
dd8a552
Refactor pipeline and episode status for use in multiple integrations
kookster Nov 26, 2024
1c01abb
Initial megaphone feed UI
kookster Nov 26, 2024
e2114ca
Fix tests
kookster Nov 26, 2024
9f5bfc1
Linting issue
kookster Nov 26, 2024
484c193
Fix saving and deleting megaphone feed config
kookster Nov 27, 2024
732dee8
Refactor sync log to include integration type
kookster Nov 27, 2024
5fb3048
MP publisher start, publishes show
kookster Nov 30, 2024
f562866
Start episode publishing and tests
kookster Dec 3, 2024
9031291
New unfinished scope for episode integrations
kookster Dec 7, 2024
f23eed2
Refactor Feed#apple? to Feed#integration_type
kookster Dec 7, 2024
9f4bed8
Fix megaphone api setting post/put body
kookster Dec 7, 2024
f13bffe
More sensible, and largely unused, feed defaults
kookster Dec 7, 2024
40f57dd
Basic megaphone episode crud methods
kookster Dec 7, 2024
d579e5c
Instantiate megaphone episodes with a megaphone podcast
kookster Dec 7, 2024
6a41795
Get publisher to create megaphone episode drafts
kookster Dec 7, 2024
f070ccc
Reuse methods in the super class
kookster Dec 8, 2024
03c0757
Update episode delivery status access to optionally create a default …
kookster Dec 8, 2024
2c709b6
Megaphone episode audio publishing
kookster Dec 8, 2024
f4ea918
Move some sync log updates to integration models
kookster Dec 8, 2024
ad179a7
Move podcast sync log to model as well
kookster Dec 8, 2024
74c371a
Make migration reversible
kookster Dec 8, 2024
37a0574
Merge branch 'main' into feat/megaphone
kookster Dec 11, 2024
84e6b56
Initial cuepoint impl
kookster Dec 12, 2024
0449918
Retry checking audio status, raise error when failing
kookster Dec 12, 2024
3676a78
Fix episodes typo
kookster Dec 12, 2024
8812b73
cuipoint test
kookster Dec 12, 2024
008d087
Add megaphone form labels
kookster Dec 12, 2024
f1177ff
Episode form status for megaphone
kookster Dec 12, 2024
a6d133d
Podcast form megaphone status
kookster Dec 12, 2024
d49da30
New integration status helper methods
kookster Dec 12, 2024
889d52f
Add new podcast integrations concern
kookster Dec 12, 2024
d871257
Have episode know if it is integrated
kookster Dec 12, 2024
f7ebfc5
No permission to optin to span
kookster Dec 12, 2024
94ae8ac
Include house zones better
kookster Dec 12, 2024
e81280a
Fix update episode
kookster Dec 13, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Gemfile
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ gem "aws-sdk-s3"
gem "csv"
gem "excon"
gem "faraday", "~> 0.17.4"
gem "link-header-parser", "~> 6.0", ">= 6.0.1"
gem "fiddle"
gem "hyperresource"
gem "mutex_m"
Expand Down
2 changes: 2 additions & 0 deletions Gemfile.lock
Original file line number Diff line number Diff line change
Expand Up @@ -267,6 +267,7 @@ GEM
kaminari-core (= 1.2.2)
kaminari-core (1.2.2)
language_server-protocol (3.17.0.3)
link-header-parser (6.0.1)
local_time (2.1.0)
logger (1.6.1)
lograge (0.14.0)
Expand Down Expand Up @@ -783,6 +784,7 @@ DEPENDENCIES
importmap-rails
jbuilder
kaminari
link-header-parser (~> 6.0, >= 6.0.1)
local_time
logger
lograge
Expand Down
14 changes: 12 additions & 2 deletions app/controllers/feeds_controller.rb
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ def new
end

def get_apple_show_options(feed)
if feed.apple? && feed.apple_config&.key
if feed.integration_type == :apple && feed.apple_config&.key
feed.apple_show_options
end
end
Expand All @@ -39,6 +39,15 @@ def new_apple
render "new"
end

def new_megaphone
@feed = Feeds::MegaphoneFeed.new(podcast: @podcast, private: true)
@feed.build_megaphone_config
authorize @feed

@feed.assign_attributes(feed_params)
render "new"
end

# POST /feeds
def create
@feed = @podcast.feeds.new(feed_params)
Expand Down Expand Up @@ -154,7 +163,8 @@ def nilified_feed_params
feed_tokens_attributes: %i[id label token _destroy],
feed_images_attributes: %i[id original_url size alt_text caption credit _destroy _retry],
itunes_images_attributes: %i[id original_url size alt_text caption credit _destroy _retry],
apple_config_attributes: [:id, :publish_enabled, :sync_blocks_rss, {key_attributes: %i[id provider_id key_id key_pem_b64]}]
apple_config_attributes: [:id, :publish_enabled, :sync_blocks_rss, {key_attributes: %i[id provider_id key_id key_pem_b64]}],
megaphone_config_attributes: [:id, :publish_enabled, :sync_blocks_rss, :network_id, :network_name, :token]
)
end
end
26 changes: 4 additions & 22 deletions app/controllers/placements_preview_controller.rb
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
class PlacementsPreviewController < ApplicationController
include PrxAccess
include Prx::Api

before_action :set_podcast

# GET /podcasts/1/placements_preview/2
def show
@fetch_error = cached_placements.nil?
@zones = get_zones(params[:id].to_i)
@fetch_error = @zones.nil?
end

private
Expand All @@ -16,27 +16,9 @@ def set_podcast
authorize @podcast, :show?
end

def placements_href
"/api/v1/podcasts/#{@podcast.id}/placements"
end

def fetch_placements
if ENV["AUGURY_HOST"].present?
api(root: augury_root, account: "*").tap { |a| a.href = placements_href }.get
end
rescue HyperResource::ClientError, HyperResource::ServerError, NotImplementedError => e
Rails.logger.error("Error fetching placements", error: e.message)
nil
end

def cached_placements
Rails.cache.fetch(placements_href, expires_in: 1.minute) do
fetch_placements
end
end

def get_placement(original_count)
cached_placements&.find { |i| i.original_count == original_count }
placements = Prx::Augury.new.placements(@podcast.id)
placements&.find { |i| i.original_count == original_count }
end

def get_zones(original_count)
Expand Down
2 changes: 1 addition & 1 deletion app/controllers/podcasts_controller.rb
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
class PodcastsController < ApplicationController
include PrxAccess
include Prx::Api
include SlackHelper

before_action :set_podcast, only: %i[show edit update destroy]
Expand Down
2 changes: 1 addition & 1 deletion app/helpers/embed_player_helper.rb
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
module EmbedPlayerHelper
include PrxAccess
include Prx::Api

EMBED_PLAYER_LANDING_PATH = "/listen"
EMBED_PLAYER_PATH = "/e"
Expand Down
23 changes: 23 additions & 0 deletions app/helpers/episodes_helper.rb
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,29 @@ def episode_explicit_options
end
end

def episode_integration_status(integration, episode)
status = episode.episode_delivery_status(integration, true)
if !status
"not_found"
elsif status.new_record?
"new"
elsif !status.uploaded?
"incomplete"
elsif !status.delivered?
"processing"
elsif status.delivered?
"complete"
else
"not_found"
end
end

def episode_integration_updated_at(integration, episode)
episode.sync_log(integration)&.updated_at ||
episode.episode_delivery_status(integration)&.created_at ||
episode.updated_at
end

def episode_apple_status(episode)
apple_episode = episode.apple_episode
if !apple_episode
Expand Down
4 changes: 4 additions & 0 deletions app/helpers/feeds_helper.rb
Original file line number Diff line number Diff line change
Expand Up @@ -72,4 +72,8 @@ def feed_retry_image_path(feed, form)
def apple_feed?(feed)
feed.type == "Feeds::AppleSubscription"
end

def megaphone_feed?(feed)
feed.type == "Feeds::MegaphoneFeed"
end
end
17 changes: 17 additions & 0 deletions app/helpers/podcasts_helper.rb
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,23 @@ module PodcastsHelper

RSS_LANGUAGE_CODES = %w[af sq eu be bg ca zh-cn zh-tw hr cs da nl nl-be nl-nl en en-au en-bz en-ca en-ie en-jm en-nz en-ph en-za en-tt en-gb en-us en-zw et fo fi fr fr-be fr-ca fr-fr fr-lu fr-mc fr-ch gl gd de de-at de-de de-li de-lu de-ch el haw hu is in ga it it-it it-ch ja ko mk no pl pt pt-br pt-pt ro ro-mo ro-ro ru ru-mo ru-ru sr sk sl es es-ar es-bo es-cl es-co es-cr es-do es-ec es-sv es-gt es-hn es-mx es-ni es-pa es-py es-pe es-pr es-es es-uy es-ve sv sv-fi sv-se tr uk]

def podcast_integration_status(integration, podcast)
sync = podcast.public_feed.sync_log(integration)
if !sync
"not_found"
elsif !sync.external_id
"new"
elsif sync.updated_at <= podcast.updated_at
"incomplete"
else
"complete"
end
end

def podcast_integration_updated_at(integration, podcast)
podcast.public_feed.sync_log(integration)&.updated_at || podcast.updated_at
end

def feed_description(feed, podcast)
[feed.description, podcast.description].detect { |d| d.present? } || ""
end
Expand Down
20 changes: 0 additions & 20 deletions app/jobs/publish_apple_job.rb

This file was deleted.

29 changes: 17 additions & 12 deletions app/jobs/publish_feed_job.rb
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,17 @@ def perform(podcast, pub_item)
# grab the current publishing pipeline.
return :null if null_publishing_item?(podcast, pub_item)
return :mismatched if mismatched_publishing_item?(podcast, pub_item)

Rails.logger.info("Starting publishing pipeline via PublishFeedJob", {podcast_id: podcast.id, publishing_queue_item_id: pub_item.id})

PublishingPipelineState.start!(podcast)
podcast.feeds.each { |feed| publish_apple(podcast, feed) }

# Publish each integration for each feed (e.g. apple, megaphone)
podcast.feeds.each { |feed| publish_integration(podcast, feed) }

# After integrations, publish RSS, if appropriate
podcast.feeds.each { |feed| publish_rss(podcast, feed) }

PublishingPipelineState.complete!(podcast)
rescue Apple::AssetStateTimeoutError => e
fail_state(podcast, "apple_timeout", e)
Expand All @@ -26,18 +32,17 @@ def perform(podcast, pub_item)
PublishingPipelineState.settle_remaining!(podcast)
end

def publish_apple(podcast, feed)
return unless feed.publish_to_apple?

res = PublishAppleJob.do_perform(podcast.apple_config)
PublishingPipelineState.publish_apple!(podcast)
def publish_integration(podcast, feed)
return unless feed.publish_integration?
res = feed.publish_integration!
PublishingPipelineState.publish_integration!(podcast)
res
rescue => e
if podcast.apple_config.sync_blocks_rss
fail_state(podcast, "apple", e)
if feed.config.sync_blocks_rss
fail_state(podcast, feed.integration_type, e)
else
Rails.logger.error("Error publishing to Apple, but continuing to publish RSS", {podcast_id: podcast.id, error: e.message})
PublishingPipelineState.error_apple!(podcast)
Rails.logger.error("Error publishing to #{feed.integration_type}, but continuing to publish RSS", {podcast_id: podcast.id, error: e.message})
PublishingPipelineState.error_integration!(podcast)
end
end

Expand All @@ -51,9 +56,9 @@ def publish_rss(podcast, feed)

def fail_state(podcast, type, error)
(pipeline_method, log_level) = case type
when "apple" then [:error_apple!, :warn]
when "rss" then [:error_rss!, :warn]
when "apple_timeout", "error" then [:error!, :error]
when "rss" then [:error_rss!, :warn]
else [:error_integration!, :warn]
end

PublishingPipelineState.public_send(pipeline_method, podcast)
Expand Down
4 changes: 2 additions & 2 deletions app/models/apple/config.rb
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ def self.build_apple_config(podcast, key)
def self.mark_as_delivered!(apple_publisher)
apple_publisher.episodes_to_sync.each do |episode|
if episode.podcast_container&.needs_delivery? == false
episode.feeder_episode.apple_has_delivery!
episode.feeder_episode.apple_mark_as_delivered!
end
end
end
Expand Down Expand Up @@ -96,7 +96,7 @@ def apple_key

def apple_data
episode_data = [
SyncLog.where(feeder_type: "episodes", feeder_id: podcast.episodes.pluck(:id)),
SyncLog.apple.where(feeder_type: "episodes", feeder_id: podcast.episodes.pluck(:id)),
Apple::PodcastContainer.where(episode: podcast.episodes)
].flatten.compact

Expand Down
54 changes: 33 additions & 21 deletions app/models/apple/episode.rb
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# frozen_string_literal: true

module Apple
class Episode
class Episode < Integrations::Base::Episode
include Apple::ApiWaiting
include Apple::ApiResponse
attr_accessor :show,
Expand All @@ -15,10 +15,8 @@ class Episode
EPISODE_ASSET_WAIT_TIMEOUT = 15.minutes.freeze
EPISODE_ASSET_WAIT_INTERVAL = 10.seconds.freeze

# Cleans up old delivery/delivery files iff the episode is to be delivered
# Cleans up old delivery/delivery files iff the episode is to be uploaded
def self.prepare_for_delivery(episodes)
episodes = episodes.select { |ep| ep.needs_delivery? }

episodes.map do |ep|
Rails.logger.info("Preparing episode #{ep.feeder_id} for delivery", {episode_id: ep.feeder_id})
ep.feeder_episode.apple_prepare_for_delivery!
Expand Down Expand Up @@ -205,7 +203,13 @@ def self.upsert_sync_log(ep, res)
apple_id = res.dig("api_response", "val", "data", "id")
raise "Missing remote apple id" unless apple_id.present?

sl = SyncLog.log!(feeder_id: ep.feeder_episode.id, feeder_type: :episodes, external_id: apple_id, api_response: res)
sl = SyncLog.log!(
integration: :apple,
feeder_id: ep.feeder_episode.id,
feeder_type: :episodes,
external_id: apple_id,
api_response: res
)
# reload local state
if ep.feeder_episode.apple_sync_log.nil?
ep.feeder_episode.reload
Expand All @@ -221,6 +225,14 @@ def initialize(show:, feeder_episode:, api:)
@api = api || Apple::Api.from_env
end

def synced_with_integration?
synced_with_apple?
end

def integration_new?
apple_new?
end

def api_response
feeder_episode.apple_sync_log&.api_response
end
Expand All @@ -247,7 +259,7 @@ def podcast

def enclosure_url
url = EnclosureUrlBuilder.new.base_enclosure_url(podcast, feeder_episode, private_feed)
EnclosureUrlBuilder.mark_authorized(url, show.private_feed)
EnclosureUrlBuilder.mark_authorized(url, private_feed)
end

def enclosure_filename
Expand All @@ -256,7 +268,7 @@ def enclosure_filename
end

def sync_log
SyncLog.episodes.find_by(feeder_id: feeder_episode.id, feeder_type: :episodes)
SyncLog.apple.episodes.find_by(feeder_id: feeder_episode.id, feeder_type: :episodes)
end

def self.get_episode_bridge_params(api, feeder_id, apple_id)
Expand Down Expand Up @@ -420,10 +432,6 @@ def publishing_state_parameters(state)
self.class.publishing_state_params(apple_id, state)
end

def apple?
feeder_episode.apple?
end

def apple_json
return nil unless api_response.present?

Expand Down Expand Up @@ -498,16 +506,6 @@ def audio_asset_state_success?
audio_asset_state == AUDIO_ASSET_SUCCESS
end

def has_media_version?
return false unless delivery_status.present? && delivery_status.source_media_version_id.present?

delivery_status.source_media_version_id == feeder_episode.media_version_id
end

def needs_media_version?
!has_media_version?
end

def needs_delivery?
return true if missing_container?

Expand Down Expand Up @@ -575,5 +573,19 @@ def apple_episode_delivery_statuses
alias_method :delivery_files, :podcast_delivery_files
alias_method :delivery_status, :apple_episode_delivery_status
alias_method :delivery_statuses, :apple_episode_delivery_statuses
alias_method :apple_status, :apple_episode_delivery_status

# Delegate methods to feeder_episode
def method_missing(method_name, *arguments, &block)
if feeder_episode.respond_to?(method_name)
feeder_episode.send(method_name, *arguments, &block)
else
super
end
end

def respond_to_missing?(method_name, include_private = false)
feeder_episode.respond_to?(method_name) || super
end
end
end
Loading
Loading