-
Notifications
You must be signed in to change notification settings - Fork 11
Load to SQL and save back to CSV
Peter edited this page Sep 14, 2020
·
9 revisions
There are many options to load CSV into PostgreSQL without knowledge of the table structure, or using script-generator.
Load the CSV
With previous CSVkit installation you can use
wget -c https://raw.githubusercontent.com/datasets-br/state-codes/master/data/br-state-codes.csv
csvsql --db postgresql:///database --insert br-state-codes.csv
See code generator as pack2sql, it generates the sh
and SQL scripts, something as
wget -O /tmp/br-state-codes.csv -c https://raw.githubusercontent.com/datasets-br/state-codes/master/data/br-state-codes.csv
CREATE EXTENSION file_fdw;
CREATE SERVER files FOREIGN DATA WRAPPER file_fdw;
DROP FOREIGN TABLE IF EXISTS tmpcsv_br_state_codes CASCADE;
CREATE FOREIGN TABLE tmpcsv_br_state_codes (
subdivision text,
region text,
name_prefix text,
name text,
id integer,
"idIBGE" text,
"wdId" text,
"lexLabel" text,
creation integer,
extinction integer,
category text,
"timeZone" text,
"utcOffset" integer,
"utcOffset_DST" integer,
"postalCode_ranges" text,
km2 int,
borders text,
centroid_geohash text,
utm_zones text,
bounds_geohash text,
bounds_lat text,
bounds_long text,
notes text
) SERVER files OPTIONS (
filename '/tmp/br-state-codes.csv', -- windows transforma '-' em '_'?
format 'csv',
header 'true'
);
REGIONS:
wget -O /tmp/br-region-codes.csv -c https://raw.githubusercontent.com/datasets-br/state-codes/master/data/br-region-codes.csv
DROP FOREIGN TABLE IF EXISTS tmpcsv_br_region_codes CASCADE;
CREATE FOREIGN TABLE tmpcsv_br_region_codes (
region text,
"wdId" text,
name text,
fullname text,
creation integer,
extinction integer,
"postalCode_ranges" text,
notes text
) SERVER files OPTIONS (
filename '/tmp/br-region-codes.csv',
format 'csv',
header 'true'
);
After create a table from the header and inferred datatypes as a table t
,
psql -h remotehost -d remote_mydb -U myuser -c \
"copy t from STDIN with delimiter as ','" < ./relative_path/file.csv
Exemplo, gravando via psql
para /tmp
:
COPY (select * from tmpcsv_br_state_codes) TO '/tmp/test_ibge.csv' CSV HEADER;