You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
For example, when a free plan user performs a scrape for a club already scraped with a paid plan, metadata will be overwritten. To fix this we can check the number of available post_contents to the scraping user and compare with the existing metadata. If more are available to the user, overwrite and continue downloading. If more are availble in the existing metadata, skip the post entirely.
We can also check which plan status is currently joined, or even simplify this entirely by specifying the type of metadata e.g. metadata_plan0.json.
The text was updated successfully, but these errors were encountered:
For example, when a free plan user performs a scrape for a club already scraped with a paid plan, metadata will be overwritten. To fix this we can check the number of available
post_contents
to the scraping user and compare with the existing metadata. If more are available to the user, overwrite and continue downloading. If more are availble in the existing metadata, skip the post entirely.We can also check which plan status is currently joined, or even simplify this entirely by specifying the type of metadata e.g.
metadata_plan0.json
.The text was updated successfully, but these errors were encountered: