Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

What happens if the same account is in two lists? #352

Open
slifty opened this issue Apr 6, 2020 · 2 comments
Open

What happens if the same account is in two lists? #352

slifty opened this issue Apr 6, 2020 · 2 comments
Labels
conversation Let's talk about this

Comments

@slifty
Copy link
Contributor

slifty commented Apr 6, 2020

When we scrape, we have some logic to prevent redundant scrapes of the same claim.

If an account is listed in two separate newsletters, does this mean a given claim will only appear in one of them, or is it smart enough to handle that.

@slifty slifty added the conversation Let's talk about this label Apr 6, 2020
@reefdog
Copy link
Collaborator

reefdog commented Apr 6, 2020

It's "smart" enough to handle that, in that claims are not fetched by list name. The newsletter fetches the accounts associated with the newsletter's list, and then fetches claims associated with those account names. (The claims themselves are associated with the account directly, not the account-list tuple.)

@reefdog
Copy link
Collaborator

reefdog commented Apr 6, 2020

We aren't doing any de-duping of the account names themselves when we scrape, but I believe that's handled by the scrape URL history that you allude to.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
conversation Let's talk about this
Projects
None yet
Development

No branches or pull requests

2 participants