Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Sitemap-based request list implementation #2498

Merged
merged 27 commits into from
Jul 4, 2024
Merged

feat: Sitemap-based request list implementation #2498

merged 27 commits into from
Jul 4, 2024

Commits on May 24, 2024

  1. refactor: Extract IRequestList interface from the RequestList class

    @janbuchar
    janbuchar committed May 24, 2024
    Configuration menu
    Copy the full SHA
    fbe7ee9 View commit details
    Browse the repository at this point in the history

  2. Implement a basic SitemapRequestList

    @janbuchar
    janbuchar committed May 24, 2024
    Configuration menu
    Copy the full SHA
    f20d14a View commit details
    Browse the repository at this point in the history

  3. Lint

    @janbuchar
    janbuchar committed May 24, 2024
    Configuration menu
    Copy the full SHA
    e16a76d View commit details
    Browse the repository at this point in the history

Commits on May 28, 2024

  1. Refactor sitemap utils to allow processing in the background

    @janbuchar
    janbuchar committed May 28, 2024
    Configuration menu
    Copy the full SHA
    27f6dd3 View commit details
    Browse the repository at this point in the history

  2. Handle all sitemap.xml fields

    @janbuchar
    janbuchar committed May 28, 2024
    Configuration menu
    Copy the full SHA
    610e6b6 View commit details
    Browse the repository at this point in the history

Commits on May 29, 2024

  1. Implement sitemap loading in the background

    @janbuchar
    janbuchar committed May 29, 2024
    Configuration menu
    Copy the full SHA
    d8951ef View commit details
    Browse the repository at this point in the history

  2. inheritdoc -> inheritDoc

    @janbuchar
    janbuchar committed May 29, 2024
    Configuration menu
    Copy the full SHA
    09b3bb1 View commit details
    Browse the repository at this point in the history

  3. Test sitemap metadata extraction

    @janbuchar
    janbuchar committed May 29, 2024
    Configuration menu
    Copy the full SHA
    f45e957 View commit details
    Browse the repository at this point in the history

  4. Basic SitemapRequestList functionality tests

    @janbuchar
    janbuchar committed May 29, 2024
    Configuration menu
    Copy the full SHA
    13d7b62 View commit details
    Browse the repository at this point in the history

  5. Fill in JSDoc

    @janbuchar
    janbuchar committed May 29, 2024
    Configuration menu
    Copy the full SHA
    a695f91 View commit details
    Browse the repository at this point in the history

Commits on May 31, 2024

  1. Implement persistence + better queueing

    @janbuchar
    janbuchar committed May 31, 2024
    Configuration menu
    Copy the full SHA
    a6fd628 View commit details
    Browse the repository at this point in the history

  2. Lint

    @janbuchar
    janbuchar committed May 31, 2024
    Configuration menu
    Copy the full SHA
    a4d77b7 View commit details
    Browse the repository at this point in the history

Commits on Jun 25, 2024

  1. chore: simplify RequestList.load() async syntax

    @barjin
    barjin committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    3baaa7b View commit details
    Browse the repository at this point in the history

  2. chore: more fitting variable naming

    @barjin
    barjin committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    ff97283 View commit details
    Browse the repository at this point in the history

  3. chore: get rid of requestData for the sake of simplicity

    @barjin
    barjin committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    ec437fd View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2024

  1. feat: more granular sitemap parsing persistence, simplify code

    @barjin
    barjin committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    1462b32 View commit details
    Browse the repository at this point in the history

  2. feat: add waitForNextRequest helper for IRequestList

    @barjin
    barjin committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    9e4a620 View commit details
    Browse the repository at this point in the history

  3. feat: add signal and timeoutMillis options for SitemapRequestList

    @barjin
    barjin committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    b1498a8 View commit details
    Browse the repository at this point in the history

Commits on Jun 27, 2024

  1. feat: remove the old queuedUrlsBySitemap property

    @barjin
    barjin committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    31efd21 View commit details
    Browse the repository at this point in the history

  2. chore: improve comment wording

    @barjin
    barjin committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    7e71afe View commit details
    Browse the repository at this point in the history

  3. chore: naming, documentation

    @barjin
    barjin committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    99a12a6 View commit details
    Browse the repository at this point in the history

  4. feat: minimizing the memory footprint with buffering streams

    @barjin
    barjin committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    53bee4b View commit details
    Browse the repository at this point in the history

Commits on Jul 3, 2024

  1. feat: customizable bufferSize, Request changes persistence

    @barjin
    barjin committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    7965373 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into sitemap-request-list

    @barjin
    barjin committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    c53dba2 View commit details
    Browse the repository at this point in the history

  3. fix: forever waiting in RequestList.requestIterator

    @barjin
    barjin committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    4b62f08 View commit details
    Browse the repository at this point in the history

  4. fix: align the tests with the current interface

    @barjin
    barjin committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    a796104 View commit details
    Browse the repository at this point in the history

Commits on Jul 4, 2024

  1. feat: swap requestIterator for [Symbol.asyncIterator]

    @barjin
    barjin committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    ead067b View commit details
    Browse the repository at this point in the history