Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make API importer only update new records #126

Open
tamcap opened this issue May 16, 2014 · 0 comments
Open

Make API importer only update new records #126

tamcap opened this issue May 16, 2014 · 0 comments

Comments

@tamcap
Copy link
Contributor

tamcap commented May 16, 2014

I am currently working on some improvements to the current API importer.

However, as it is right now, we import the whole database from Durham County - despite their API providing us with a field Update_Date and syntax to filter it by Update_Date__gt=mm/dd/yyy that we can use to significantly limit our data / cpu use. We should take advantage of it.

Implementation: we will need to run a query on our db before import starts to establish the "latest" Update_Date value for each table (establishments, inspections, violations) and then incorporate it into the query string.

@copelco copelco added this to the Phase 3 milestone Jul 1, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants