Skip to content

Latest commit

 

History

History
49 lines (39 loc) · 3.07 KB

README.md

File metadata and controls

49 lines (39 loc) · 3.07 KB

Quandl Scraper

Downloads Quandl Data (Bulks) without a premium account and import it to a PostgreSQL (+ 9.2) database.

Everything gets structured N I C E L Y

Dependencies

  • PostgreSQL (9.2 or later) must be installed

Setup

Thats it. The program will handle by itself the creation of database/schemas/tables etc..

Usage

  1. Program will download all databases available and insert them into quandl.databases (in a database named quandl in the postgres instance) and then pause
  • Here you have to select the databases in postgres by setting import to true. Example:
update quandl.databases set import = true where databasecode = 'WIKI';
  1. Program will download all datasets available from WIKI and insert them into quandl.datasets and then pause
  • Here you have to select the datasets in postgres by setting import to true. Example:
update quandl.datasets set import = true where databasecode = 'WIKI' and datasetcode = 'AAPL';
update quandl.datasets set import = true where databasecode = 'WIKI' and datasetcode = 'FB';
  1. Program will download all the data available from WIKI/AAPL and WIKI/FB, dynamically creates table quandl.wiki and inserts all the data

(Extra) Usage for datatables

  1. Program will insert all datatables available and insert them into quandl.datatables (in a database named quandl in the postgres instance) and then pause
  • Here you have to select the datatables in postgres by setting import to true. Example:
update quandl.datatables set import = true where name = 'ZACKS/CP';
  1. Program will download all the data available from ZACKS/CP, dynamically creates table ZACKS/CP and inserts all the data

Info

The program consists in 3 key tasks (4 if you count datatables fetch). Each one is independient from each other. So you can skip a task (commenting the code line) and the program will still run fine.

  1. Fetch Databases available
  2. Fetch Datasets available (from selected databases downloaded in step 1)
  3. Fetch Data (from selected databases downloaded in step 2)
  4. Fetch Datatables (Extra) (from selected datatables downloaded)

If something unexpected happens a file log.txt (inside same directory where the excutable is running) will be created with the information, however, the program will do his best to handle the error and keep working.