Our first production Crawly spider #231
dogweather
started this conversation in
Show and tell
Replies: 1 comment
-
|
Thank you for the example, learned a few things! Unfortunately the service seems to be down right now... |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
https://github.com/public-law/oregon-revised-statutes-crawler
Code review / critique is appreciated.
Coding it was a very smooth experience. No unpleasant surprises; everything worked as expected. The Domo library and the new dbg debugger contributed the smooth experience. In some ways, it was easier than Scrapy/Python.
The spider creates a 60mb JSON lines file. It uses one API call and approx. 500 web pages as its source.
Beta Was this translation helpful? Give feedback.
All reactions