Code

Major rewrite to v3#4

Code

Draft

MarkusShepherd wants to merge 104 commits intomasterrecommend-games/board-game-scraper:masterfrom v3recommend-games/board-game-scraper:v3Copy head branch name to clipboard

+5,458-144

Commits

Commits on Nov 11, 2024

• Cruft

  MarkusShepherd

  committed
  de8fc46

  Copy full SHA for de8fc46

• Adding scrapy to dependencies

  MarkusShepherd

  committed
  47415be

  Copy full SHA for 47415be

• Move most old code to v2 dir

  MarkusShepherd

  committed
  84698f0

  Copy full SHA for 84698f0

• Exclude v2 from pre-commits

  MarkusShepherd

  committed
  dae4907

  Copy full SHA for dae4907

• docker-compose.yaml -> compose.yaml

  MarkusShepherd

  committed
  1c4ea90

  Copy full SHA for 1c4ea90

• Update shuffle.sh

  MarkusShepherd

  committed
  998d97c

  Copy full SHA for 998d97c

• Scrapy scaffolding

  MarkusShepherd

  committed
  7ad9f15

  Copy full SHA for 7ad9f15

• Comment out middleware

  MarkusShepherd

  committed
  0a9e205

  Copy full SHA for 0a9e205

• Clean up shit rules

  MarkusShepherd

  committed
  13d4d95

  Copy full SHA for 13d4d95

• First draft of BggSpider

  MarkusShepherd

  committed
  c687da5

  Copy full SHA for c687da5

Commits on Nov 15, 2024

• Added itemadapter, jmespath, pyyaml requests and w3lib as dependencies

  MarkusShepherd

  committed
  0c40916

  Copy full SHA for 0c40916

• {v2 => src}/board_game_scraper/download_bgg_dump.py

  MarkusShepherd

  committed
  2942d51

  Copy full SHA for 2942d51

• Require itemloaders>=1.1.0 which adds support for JMESPath

  MarkusShepherd

  committed
  02fe6fa

  Copy full SHA for 02fe6fa

Commits on Nov 16, 2024

• Added python-dotenv as dependency and corrected BASE_DIR in download_bgg_dump.py

  MarkusShepherd

  committed
  9798b41

  Copy full SHA for 9798b41

• scrapy.responsetypes.Response -> scrapy.http.Response

  MarkusShepherd

  committed
  dfb39fc

  Copy full SHA for dfb39fc

Commits on Nov 17, 2024

• Added new scrapy-extensions dependency

  MarkusShepherd

  committed
  2b6b225

  Copy full SHA for 2b6b225

• Settings

  MarkusShepherd

  committed
  ca2f7e3

  Copy full SHA for ca2f7e3

• scrapy-extensions v1.0.1

  MarkusShepherd

  committed
  cfbaa90

  Copy full SHA for cfbaa90

Commits on Nov 19, 2024

• Added attrs as dependency

  MarkusShepherd

  committed
  3ef4f54

  Copy full SHA for 3ef4f54

• Removed redundant TWISTED_REACTOR setting

  MarkusShepherd

  committed
  78253da

  Copy full SHA for 78253da

• Addressing some minor mypy complaints

  MarkusShepherd

  committed
  48658c7

  Copy full SHA for 48658c7

• GameItem

  MarkusShepherd

  committed
  3ae1c88

  Copy full SHA for 3ae1c88

• UserItem

  MarkusShepherd

  committed
  d4315d8

  Copy full SHA for d4315d8

• CollectionItem

  MarkusShepherd

  committed
  f85d689

  Copy full SHA for f85d689

• Added FEEDS setting

  MarkusShepherd

  committed
  5496de3

  Copy full SHA for 5496de3

• Actually start scraping something

  MarkusShepherd

  committed
  1c8f64a

  Copy full SHA for 1c8f64a

Commits on Nov 23, 2024

• Default for scraped_at in items.py

  MarkusShepherd

  committed
  e978e3f

  Copy full SHA for e978e3f

• Use ItemLoader in board_game_scraper/spiders/bgg.py

  MarkusShepherd

  committed
  c066072

  Copy full SHA for c066072

Commits on Nov 24, 2024

• Added board_game_scraper.utils

  MarkusShepherd

  committed
  fe9e61b

  Copy full SHA for fe9e61b

• Adding some converters to items

  MarkusShepherd

  committed
  a02012e

  Copy full SHA for a02012e

• Added python-dateutil to dependencies

  MarkusShepherd

  committed
  8a6e6b8

  Copy full SHA for 8a6e6b8

• Added parse_float and parse_date

  MarkusShepherd

  committed
  a3db058

  Copy full SHA for a3db058

• Added utils.now

  MarkusShepherd

  committed
  67758f4

  Copy full SHA for 67758f4

• Added to_str() and normalize_space() to utils

  MarkusShepherd

  committed
  e49b22b

  Copy full SHA for e49b22b

• Move conversion from items to loaders

  MarkusShepherd

  committed
  bab6f24

  Copy full SHA for bab6f24

• normalize_space_with_newline for descriptions and comments

  MarkusShepherd

  committed
  98cba4a

  Copy full SHA for 98cba4a

Commits on Nov 25, 2024

• Added cron/games.recommend.curl_hotness.plist

  MarkusShepherd

  committed
  9ac1cef

  Copy full SHA for 9ac1cef

• Added cron/games.recommend.download_bgg_dump.plist

  MarkusShepherd

  committed
  3d1a421

  Copy full SHA for 3d1a421

• Added cron/games.recommend.bgg_ranked_csv.plist

  MarkusShepherd

  committed
  2e998ef

  Copy full SHA for 2e998ef

Commits on Nov 26, 2024

• Small changes

  MarkusShepherd

  committed
  b375fd2

  Copy full SHA for b375fd2

• Whitespace

  MarkusShepherd

  committed
  7b13ed7

  Copy full SHA for 7b13ed7

• Introduced BggGameLoader; added more scraped fields

  MarkusShepherd

  committed
  698c63e

  Copy full SHA for 698c63e

• Added more (all?) input and output processors to GameLoader

  MarkusShepherd

  committed
  1d24b94

  Copy full SHA for 1d24b94

• Added UserLoader

  MarkusShepherd

  committed
  02fc4f9

  Copy full SHA for 02fc4f9

• Added some input processors to CollectionLoader

  MarkusShepherd

  committed
  7f9a6e5

  Copy full SHA for 7f9a6e5

• Added response_urljoin() to loaders

  MarkusShepherd

  committed
  2ff17c2

  Copy full SHA for 2ff17c2

• More fields scraped

  MarkusShepherd

  committed
  5a4dbc8

  Copy full SHA for 5a4dbc8

• More fields

  MarkusShepherd

  committed
  6c9cb7c

  Copy full SHA for 6c9cb7c

• Better CollectionItem scraping

  MarkusShepherd

  committed
  d11856c

  Copy full SHA for d11856c

• Some cleanup

  MarkusShepherd

  committed
  b6df965

  Copy full SHA for b6df965

Commits on Nov 27, 2024

• Added more-itertools

  MarkusShepherd

  committed
  af21eba

  Copy full SHA for af21eba

• Collect BGG IDs in chunks

  MarkusShepherd

  committed
  49ccb45

  Copy full SHA for 49ccb45

• Added BggSpider.has_seen_bgg_id()

  MarkusShepherd

  committed
  ae5c5f5

  Copy full SHA for ae5c5f5

• Warning if no state

  MarkusShepherd

  committed
  34969f4

  Copy full SHA for 34969f4

• Added RankingItem and RankingLoader

  MarkusShepherd

  committed
  e5ed156

  Copy full SHA for e5ed156

• Use RankingLoader in BggSpider

  MarkusShepherd

  committed
  333e96d

  Copy full SHA for 333e96d

Commits on Nov 28, 2024

• Added SparseJsonLinesItemExporter

  MarkusShepherd

  committed
  3c4c447

  Copy full SHA for 3c4c447

• Comments and TODOs

  MarkusShepherd

  committed
  55e4b83

  Copy full SHA for 55e4b83

• Set FEED_EXPORT_BATCH_ITEM_COUNT and add %(batch_id)05d to paths

  MarkusShepherd

  committed
  3b5b651

  Copy full SHA for 3b5b651

• Small changes

  MarkusShepherd

  committed
  89914e8

  Copy full SHA for 89914e8

• Moving around things

  MarkusShepherd

  committed
  f5e8200

  Copy full SHA for f5e8200

• Added BggSpider.game_requests()

  MarkusShepherd

  committed
  3916de5

  Copy full SHA for 3916de5

• Remove BggSpider.sitemap_filter() and batch requests in _parse_sitemap() instead

  MarkusShepherd

  committed
  d4b0bf7

  Copy full SHA for d4b0bf7

• Extracted BggSpider.scrape_game_item()

  MarkusShepherd

  committed
  7d03cd3

  Copy full SHA for 7d03cd3

• Factored out scrape_ranking_item()

  MarkusShepherd

  committed
  e4642f2

  Copy full SHA for e4642f2

• Factor out extract_collection_item()

  MarkusShepherd

  committed
  6762f60

  Copy full SHA for 6762f60

• Replace assert isinstance() with cast()

  MarkusShepherd

  committed
  a9ffc1c

  Copy full SHA for a9ffc1c

• Added contract to _parse_sitemap()

  MarkusShepherd

  committed
  9913137

  Copy full SHA for 9913137

• Added bgg_ids to request.meta

  MarkusShepherd

  committed
  9e35c76

  Copy full SHA for 9e35c76

• Querying the next pages of a game item; added extract_page_number

  MarkusShepherd

  committed
  2423307

  Copy full SHA for 2423307

• Remove debug message

  MarkusShepherd

  committed
  75798f6

  Copy full SHA for 75798f6

Commits on Nov 29, 2024

• Reduce priority of next pages

  MarkusShepherd

  committed
  27af0e9

  Copy full SHA for 27af0e9

• Lower priority for higher page numbers; corrected max_page_from_response calculation

  MarkusShepherd

  committed
  373d122

  Copy full SHA for 373d122

• Custom DOWNLOAD_DELAY

  MarkusShepherd

  committed
  1350d00

  Copy full SHA for 1350d00

• Added BggSpider.collection_request()

  MarkusShepherd

  committed
  dd16e6b

  Copy full SHA for dd16e6b

Commits on Nov 30, 2024

• Make BggSpider.collection_request()

  MarkusShepherd

  committed
  0cf6b8b

  Copy full SHA for 0cf6b8b

• Added BggSpider.scrape_ratings, scrape_collections and scrape_users

  MarkusShepherd

  committed
  d15f679

  Copy full SHA for d15f679

• Make scrape_ratings, scrape_collections and scrape_users explicit arguments of BggSpider()

  MarkusShepherd

  committed
  6f9a5c6

  Copy full SHA for 6f9a5c6

• Properly parse_collection()

  MarkusShepherd

  committed
  c36e4f7

  Copy full SHA for c36e4f7

• Move around things; always parse scrape_* correctly

  MarkusShepherd

  committed
  a425ea7

  Copy full SHA for a425ea7

• Added BggSpider.user_request(), parse_user() and extract_user_item()

  MarkusShepherd

  committed
  d4f131b

  Copy full SHA for d4f131b

• Tweak settings

  MarkusShepherd

  committed
  a178c58

  Copy full SHA for a178c58

• game_files and user_files as args

  MarkusShepherd

  committed
  15d335a

  Copy full SHA for 15d335a

• Stub game_requests_from_files() and user_and_collection_requests_from_files()

  MarkusShepherd

  committed
  446812a

  Copy full SHA for 446812a

• Added utils.files.extract_field_from_jsonlines_file()

  MarkusShepherd

  committed
  c9f0686

  Copy full SHA for c9f0686

• Added extract_field_from_csv_file()

  MarkusShepherd

  committed
  3f68199

  Copy full SHA for 3f68199

• Added load_premium_users()

  MarkusShepherd

  committed
  22df276

  Copy full SHA for 22df276

• Added extract_field_from_files()

  MarkusShepherd

  committed
  c967b06

  Copy full SHA for c967b06

• Moved parse_file_paths() to utils and extract_field_from_files() in *_requests_from_files()

  MarkusShepherd

  committed
  aec287e

  Copy full SHA for aec287e

• Scraping num_owned, num_trading, num_wanting, num_wishing, num_comments, num_weights

  MarkusShepherd

  committed
  45e7244

  Copy full SHA for 45e7244

• Better logging

  MarkusShepherd

  committed
  3dbc9aa

  Copy full SHA for 3dbc9aa

• Longer DOWNLOAD_DELAY

  MarkusShepherd

  committed
  6b12119

  Copy full SHA for 6b12119

Commits on Dec 1, 2024

• Load premium users in BggSpider

  MarkusShepherd

  committed
  9f0d683

  Copy full SHA for 9f0d683

• Contracts

  MarkusShepherd

  committed
  bc9a8d3

  Copy full SHA for bc9a8d3

• Added iterables.clear_iterable() and clear_list()

  MarkusShepherd

  committed
  d4bab99

  Copy full SHA for d4bab99

• Added utils.ids

  MarkusShepherd

  committed
  1a51f5d

  Copy full SHA for 1a51f5d

• Move extract_query_param() and parse_url() to utils.urls and other minor changes

  MarkusShepherd

  committed
  75fd39f

  Copy full SHA for 75fd39f

• Extract min_players_rec, max_players_rec, min_players_best, max_players_best

  MarkusShepherd

  committed
  84be337

  Copy full SHA for 84be337

• Extract min_age_rec

  MarkusShepherd

  committed
  5a60b2c

  Copy full SHA for 5a60b2c

• Added language_dependency

  MarkusShepherd

  committed
  21f165b

  Copy full SHA for 21f165b

• TODO

  MarkusShepherd

  committed
  3bc44b9

  Copy full SHA for 3bc44b9

• Leaner Generators

  MarkusShepherd

  committed
  dafb2d2

  Copy full SHA for dafb2d2

• Added board_game_scraper.pipelines.LimitImagesPipeline

  MarkusShepherd

  committed
  a25ef6d

  Copy full SHA for a25ef6d

• Small corrections

  MarkusShepherd

  committed
  16057c3

  Copy full SHA for 16057c3