Image Harvests

Deprecated / 2022-11-17 EWL

Background

The B1G Geoportal harvests images for thumbnail and reference images for contributed documents.

Process

1. Check prior states

RAILS_ENV=production bundle exec rake geoportal:sidecar_states

2. Process ALL images

RAILS_ENV=production bundle exec rake geoblacklight_sidecar_images:images:precache_all

3. Re-run incomplete states

RAILS_ENV=production bundle exec rake geoportal:queue_incomplete_states

Check that all background jobs have completed. When enqueued is 0 everything has processed.

4. Check Sidecar States

RAILS_ENV=production bundle exec rake geoportal:sidecar_states

5. Check Sidekiq Stats RAILS_ENV=production bundle exec rake geoportal:sidekiq_stats

6. Run Sidecar Report RAILS_ENV=production bundle exec rake geoportal:sidecar_report

Outputs a datetime stamped CSV file of sidecar states and last transitions. Example: https://geo.btaa.org/2018-11-07_10-23-01.sidecar_report.csv

Manually

sc = SolrDocumentSidecar.find_by(:document_id => 'ANT-REF-MS2509-001')
cat = Blacklight::SearchService.new(config: CatalogController.blacklight_config)
resp, doc = cat.fetch(sc.document_id)
is = GeoblacklightSidecarImages::ImageService.new(doc)
image_url = is.send(:image_url) # Shows what URL it want to store
is.store

--

Notes

# 2018-04-24 / GeoProd / Full harvest
#<Sidekiq::Stats:0x0055d2c341e2e0 @stats={:processed=>12480, :failed=>71, :scheduled_size=>0, :retry_size=>0, :dead_size=>0, :processes_size=>1, :default_queue_latency=>0, :workers_size=>0, :enqueued=>0}>

initialized - 0
queued - 0
processing - 0
succeeded - 6824
failed - 1681
placeheld - 3904

# 2018-05-01 / GeoProd / Incomplete state harvest
#<Sidekiq::Stats:0x0056305346c6a0 @stats={:processed=>23677, :failed=>76, :scheduled_size=>0, :retry_size=>0, :dead_size=>0, :processes_size=>1, :default_queue_latency=>0, :workers_size=>0, :enqueued=>0}>

initialized - 0
queued - 0
processing - 0
succeeded - 7815
failed - 1322
placeheld - 3374

Added: 953

# 2018-11-07 / GeoProd / Incomplete state harvest

initialized - 1
queued - 0
processing - 0
succeeded - 6896
failed - 3127
placeheld - 7095

Lost: 919

# 2018-11-08 / GeoProd / Incomplete state harvest

initialized - 31
queued - 0
processing - 0
succeeded - 7293
failed - 2736
placeheld - 7108