I've been throughly duckpilled. I've spent the better part of the last year wrangling a LOT of CSVs and let me tell you, duckdb has been right there with me, helping me along the way.
me, talking about duckdb to anyone who will listen
And after a year of doing this DAY AFTER DAY AFTER DAY
β― duckdb
v1.1.3 19864453f7
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
D create table t as select * from read_csv('ome_more_csv');
I decided that enough was enough and I made duckit. BEHOLD MY MONSTROUS CREATION!
Does it clean up after itself? Of course not!
Does it check inputs? Psh.
What happens if duckdb isn't installed? You'll probably die who can tell?!?
But it lets me do this, which I've been dying to do for months now:
β― source duckit.sh
β― duckit free_company_dataset.csv
100% ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
v1.1.3 19864453f7
Enter ".help" for usage hints.
D summarize t;
ββββββββββββββββ¬ββββββββββββββ¬ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ¬ββββββββββββββββ¬βββββββββββββββββββββ¬βββββββββββββββββββββ¬ββββββββββ¬ββββββββββ¬ββββββββββ¬βββββββββββ¬ββββββββββββββββββ
β column_name β column_type β min β max β approx_unique β avg β std β q25 β q50 β q75 β count β null_percentage β
β varchar β varchar β varchar β varchar β int64 β varchar β varchar β varchar β varchar β varchar β int64 β decimal(9,2) β
ββββββββββββββββΌββββββββββββββΌββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββΌββββββββββββββββΌβββββββββββββββββββββΌβββββββββββββββββββββΌββββββββββΌββββββββββΌββββββββββΌβββββββββββΌββββββββββββββββββ€
β country β VARCHAR β afghanistan β Γ₯land islands β 283 β β β β β β 22734322 β 9.98 β
β founded β BIGINT β 1001 β 2024 β 937 β 2006.3161400677536 β 24.825956869579304 β 2003 β 2013 β 2018 β 22734322 β 60.79 β
β id β VARCHAR β 000053ktcMRCSIbGGboE0QlBPQlh β zzzzgepytuhECxjtjLcHyAGn554K β 22169331 β β β β β β 22734322 β 0.00 β
β industry β VARCHAR β "glass β writing and editing β 164 β β β β β β 22734322 β 23.83 β
β linkedin_url β VARCHAR β ceramics & concrete" β linkedin.com/company/οΏ½οΏ½οΏ½ashiraj-education-overseas-consultants-private-limited β 26144844 β β β β β β 22734322 β 0.00 β
β locality β VARCHAR β "aeroporto \"b\"" β βaΓ―n el melh β 248569 β β β β β β 22734322 β 32.66 β
β name β VARCHAR β el" β π«§sl-wash𫧠spΓ©cialiste station de lavage auto β 24204406 β β β β β β 22734322 β 0.13 β
β region β VARCHAR β registered estate surveyors & valβ¦ β π§βοΈmaison sΓ©rΓ©nitΓ© opΓ΄no πͺ· β 139478 β β β β β β 22734322 β 22.99 β
β size β VARCHAR β 1-10 β ΕΎilinskΓ½ β 2558 β β β β β β 22734322 β 0.38 β
β website β VARCHAR β "google.com/maps/place/28Β°36'36.4\"n+77Β°01'43.6\"e/@28.610122.77.β¦ β πππ.fm β 20771265 β β β β β β 22734322 β 30.49 β
ββββββββββββββββ΄ββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ΄βββββββββββββββββββββ΄βββββββββββββββββββββ΄ββββββββββ΄ββββββββββ΄ββββββββββ΄βββββββββββ΄ββββββββββββββββββ€
β 10 rows 12 columns β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
D select count(0) from t;
ββββββββββββ
β count(0) β
β int64 β
ββββββββββββ€
β 22734322 β
ββββββββββββ
It can even do json!
β― duckit gists.json
v1.1.3 19864453f7
Enter ".help" for usage hints.
D select * from t;
ββββββββββββββββββββββββ¬βββββββββββββββββββββββ¬βββββββββββββββββββββββ¬βββββββββββββββββββββββ¬βββββββββββββββββββββββ¬βββββββββββββββββββββββ¬βββββββββββββββββββββββ¬ββββ¬ββββββββββββββββββββββ¬βββββββββββββββββββββββ¬βββββββββββ¬βββββββ¬βββββββββββββββββββββββ¬βββββββββββββββββββββββ¬ββββββββββββ
β url β forks_url β commits_url β id β node_id β git_pull_url β git_push_url β β¦ β updated_at β description β comments β user β comments_url β owner β truncated β
β varchar β varchar β varchar β varchar β varchar β varchar β varchar β β timestamp β varchar β int64 β json β varchar β struct(login varchβ¦ β boolean β
ββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββΌββββΌββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββΌβββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββΌββββββββββββ€
β https://api.githubβ¦ β https://api.githubβ¦ β https://api.githubβ¦ β ee86f0e9dd841661e9β¦ β G_kwDNAdPaACBlZTg2β¦ β https://gist.githuβ¦ β https://gist.githuβ¦ β β¦ β 2024-11-24 18:05:13 β Duckit β 0 β β https://api.githubβ¦ β {'login': mando, 'β¦ β false β
β https://api.githubβ¦ β https://api.githubβ¦ β https://api.githubβ¦ β 51b2f3fa672ad6e359β¦ β G_kwDNAdPaACA1MWIyβ¦ β https://gist.githuβ¦ β https://gist.githuβ¦ β β¦ β 2024-11-05 19:29:08 β mando.git.ht: logiβ¦ β 0 β β https://api.githubβ¦ β {'login': mando, 'β¦ β false β
β https://api.githubβ¦ β https://api.githubβ¦ β https://api.githubβ¦ β ae9712f7ff00430bacβ¦ β G_kwDNAdPaACBhZTk3β¦ β https://gist.githuβ¦ β https://gist.githuβ¦ β β¦ β 2024-11-05 19:28:54 β Testing testing teβ¦ β 0 β β https://api.githubβ¦ β {'login': mando, 'β¦ β false β
β https://api.githubβ¦ β https://api.githubβ¦ β https://api.githubβ¦ β f0bebf661fbb351165β¦ β G_kwDNAdPaACBmMGJlβ¦ β https://gist.githuβ¦ β https://gist.githuβ¦ β β¦ β 2024-03-22 15:01:42 β β 0 β β https://api.githubβ¦ β {'login': mando, 'β¦ β false β
β https://api.githubβ¦ β https://api.githubβ¦ β https://api.githubβ¦ β c6c55590e0072be1adβ¦ β G_kwDNAdPaACBjNmM1β¦ β https://gist.githuβ¦ β https://gist.githuβ¦ β β¦ β 2024-01-04 17:41:12 β mando.git.ht: logiβ¦ β 0 β β https://api.githubβ¦ β {'login': mando, 'β¦ β false β
β https://api.githubβ¦ β https://api.githubβ¦ β https://api.githubβ¦ β 50b947bba620eec256β¦ β G_kwDNAdPaACA1MGI5β¦ β https://gist.githuβ¦ β https://gist.githuβ¦ β β¦ β 2023-11-16 17:39:04 β Githoot - the blogβ¦ β 0 β β https://api.githubβ¦ β {'login': mando, 'β¦ β false β
β https://api.githubβ¦ β https://api.githubβ¦ β https://api.githubβ¦ β d997732792dc12c896β¦ β MDQ6R2lzdGQ5OTc3Mzβ¦ β https://gist.githuβ¦ β https://gist.githuβ¦ β β¦ β 2021-08-12 18:08:33 β β 0 β β https://api.githubβ¦ β {'login': mando, 'β¦ β false β
β https://api.githubβ¦ β https://api.githubβ¦ β https://api.githubβ¦ β 71f7421580c0371ad7β¦ β MDQ6R2lzdDcxZjc0Mjβ¦ β https://gist.githuβ¦ β https://gist.githuβ¦ β β¦ β 2021-08-12 17:50:53 β β 0 β β https://api.githubβ¦ β {'login': mando, 'β¦ β false β
β https://api.githubβ¦ β https://api.githubβ¦ β https://api.githubβ¦ β 61358dd5bc19f6dd84β¦ β MDQ6R2lzdDYxMzU4ZGβ¦ β https://gist.githuβ¦ β https://gist.githuβ¦ β β¦ β 2024-10-29 17:07:37 β 4 months into workβ¦ β 0 β β https://api.githubβ¦ β {'login': mando, 'β¦ β false β
β https://api.githubβ¦ β https://api.githubβ¦ β https://api.githubβ¦ β a5bb9dfc2bd8f2ebd1β¦ β MDQ6R2lzdGE1YmI5ZGβ¦ β https://gist.githuβ¦ β https://gist.githuβ¦ β β¦ β 2020-07-22 18:06:06 β how to cleanup youβ¦ β 0 β β https://api.githubβ¦ β {'login': mando, 'β¦ β false β
β https://api.githubβ¦ β https://api.githubβ¦ β https://api.githubβ¦ β f28b701828986c3872β¦ β MDQ6R2lzdGYyOGI3MDβ¦ β https://gist.githuβ¦ β https://gist.githuβ¦ β β¦ β 2020-04-17 21:30:31 β β 0 β β https://api.githubβ¦ β {'login': mando, 'β¦ β false β
β https://api.githubβ¦ β https://api.githubβ¦ β https://api.githubβ¦ β a7402b9ed557c2ded7β¦ β MDQ6R2lzdGE3NDAyYjβ¦ β https://gist.githuβ¦ β https://gist.githuβ¦ β β¦ β 2024-11-07 14:14:16 β Old Timer Drinkingβ¦ β 321 β β https://api.githubβ¦ β {'login': mando, 'β¦ β false β
β https://api.githubβ¦ β https://api.githubβ¦ β https://api.githubβ¦ β 002d6ea1a6da89a97dβ¦ β MDQ6R2lzdDAwMmQ2ZWβ¦ β https://gist.githuβ¦ β https://gist.githuβ¦ β β¦ β 2017-06-15 18:06:03 β β 0 β β https://api.githubβ¦ β {'login': mando, 'β¦ β false β
β https://api.githubβ¦ β https://api.githubβ¦ β https://api.githubβ¦ β 7bb2f1cd3d2306209fβ¦ β MDQ6R2lzdDdiYjJmMWβ¦ β https://gist.githuβ¦ β https://gist.githuβ¦ β β¦ β 2017-01-16 18:48:06 β Vig README β 0 β β https://api.githubβ¦ β {'login': mando, 'β¦ β false β
β https://api.githubβ¦ β https://api.githubβ¦ β https://api.githubβ¦ β 6738de7fa55b572373β¦ β MDQ6R2lzdDY3MzhkZTβ¦ β https://gist.githuβ¦ β https://gist.githuβ¦ β β¦ β 2016-10-25 15:57:18 β β 0 β β https://api.githubβ¦ β {'login': mando, 'β¦ β false β
β https://api.githubβ¦ β https://api.githubβ¦ β https://api.githubβ¦ β c33cb129af8f2f57cf25 β MDQ6R2lzdGMzM2NiMTβ¦ β https://gist.githuβ¦ β https://gist.githuβ¦ β β¦ β 2016-01-21 00:51:33 β Decimal degrees toβ¦ β 0 β β https://api.githubβ¦ β {'login': mando, 'β¦ β false β
β https://api.githubβ¦ β https://api.githubβ¦ β https://api.githubβ¦ β da18482e2b89f4b6ed5f β MDQ6R2lzdGRhMTg0ODβ¦ β https://gist.githuβ¦ β https://gist.githuβ¦ β β¦ β 2016-01-14 06:20:54 β β 0 β β https://api.githubβ¦ β {'login': mando, 'β¦ β false β
β https://api.githubβ¦ β https://api.githubβ¦ β https://api.githubβ¦ β f134d6a8c14ed2b0c92f β MDQ6R2lzdGYxMzRkNmβ¦ β https://gist.githuβ¦ β https://gist.githuβ¦ β β¦ β 2015-08-29 14:10:44 β β 0 β β https://api.githubβ¦ β {'login': mando, 'β¦ β false β
β https://api.githubβ¦ β https://api.githubβ¦ β https://api.githubβ¦ β d2941c0a6d11c34b828c β MDQ6R2lzdGQyOTQxYzβ¦ β https://gist.githuβ¦ β https://gist.githuβ¦ β β¦ β 2015-08-29 14:04:54 β Diffy diff OS X β 0 β β https://api.githubβ¦ β {'login': mando, 'β¦ β false β
β https://api.githubβ¦ β https://api.githubβ¦ β https://api.githubβ¦ β ca2fa6c9f58acd9bec92 β MDQ6R2lzdGNhMmZhNmβ¦ β https://gist.githuβ¦ β https://gist.githuβ¦ β β¦ β 2015-08-29 14:03:21 β β 0 β β https://api.githubβ¦ β {'login': mando, 'β¦ β false β
β https://api.githubβ¦ β https://api.githubβ¦ β https://api.githubβ¦ β e1b93463d7bc81eedb5e β MDQ6R2lzdGUxYjkzNDβ¦ β https://gist.githuβ¦ β https://gist.githuβ¦ β β¦ β 2015-08-29 14:02:24 β β 0 β β https://api.githubβ¦ β {'login': mando, 'β¦ β false β
β https://api.githubβ¦ β https://api.githubβ¦ β https://api.githubβ¦ β f3e5d30dbd3687eddeb0 β MDQ6R2lzdGYzZTVkMzβ¦ β https://gist.githuβ¦ β https://gist.githuβ¦ β β¦ β 2015-08-29 14:01:00 β IM A VAMPIRE! β 0 β β https://api.githubβ¦ β {'login': mando, 'β¦ β false β
β https://api.githubβ¦ β https://api.githubβ¦ β https://api.githubβ¦ β 9788368 β MDQ6R2lzdDk3ODgzNjg= β https://gist.githuβ¦ β https://gist.githuβ¦ β β¦ β 2015-08-29 13:57:46 β β 0 β β https://api.githubβ¦ β {'login': mando, 'β¦ β false β
β https://api.githubβ¦ β https://api.githubβ¦ β https://api.githubβ¦ β 828efa2fa0e40339024c β MDQ6R2lzdDgyOGVmYTβ¦ β https://gist.githuβ¦ β https://gist.githuβ¦ β β¦ β 2015-08-29 13:56:58 β β 0 β β https://api.githubβ¦ β {'login': mando, 'β¦ β false β
β https://api.githubβ¦ β https://api.githubβ¦ β https://api.githubβ¦ β bd6f9317afb952628516 β MDQ6R2lzdGJkNmY5Mzβ¦ β https://gist.githuβ¦ β https://gist.githuβ¦ β β¦ β 2015-08-29 13:56:53 β β 0 β β https://api.githubβ¦ β {'login': mando, 'β¦ β false β
β https://api.githubβ¦ β https://api.githubβ¦ β https://api.githubβ¦ β 9028074 β MDQ6R2lzdDkwMjgwNzQ= β https://gist.githuβ¦ β https://gist.githuβ¦ β β¦ β 2024-11-23 02:39:47 β β 2 β β https://api.githubβ¦ β {'login': mando, 'β¦ β false β
β https://api.githubβ¦ β https://api.githubβ¦ β https://api.githubβ¦ β 0618a7aa275ebd6bd6bd β MDQ6R2lzdDA2MThhN2β¦ β https://gist.githuβ¦ β https://gist.githuβ¦ β β¦ β 2016-01-04 20:59:39 β pojos por favor β 0 β β https://api.githubβ¦ β {'login': mando, 'β¦ β false β
β https://api.githubβ¦ β https://api.githubβ¦ β https://api.githubβ¦ β ccfda8a27f24afd2e30c β MDQ6R2lzdGNjZmRhOGβ¦ β https://gist.githuβ¦ β https://gist.githuβ¦ β β¦ β 2016-01-03 09:59:11 β β 0 β β https://api.githubβ¦ β {'login': mando, 'β¦ β false β
β https://api.githubβ¦ β https://api.githubβ¦ β https://api.githubβ¦ β 242aab0fc41a98fd853a β MDQ6R2lzdDI0MmFhYjβ¦ β https://gist.githuβ¦ β https://gist.githuβ¦ β β¦ β 2016-01-03 09:28:57 β β 0 β β https://api.githubβ¦ β {'login': mando, 'β¦ β false β
β https://api.githubβ¦ β https://api.githubβ¦ β https://api.githubβ¦ β 7805553 β MDQ6R2lzdDc4MDU1NTM= β https://gist.githuβ¦ β https://gist.githuβ¦ β β¦ β 2015-12-30 08:49:16 β β 0 β β https://api.githubβ¦ β {'login': mando, 'β¦ β false β
ββββββββββββββββββββββββ΄βββββββββββββββββββββββ΄βββββββββββββββββββββββ΄βββββββββββββββββββββββ΄βββββββββββββββββββββββ΄βββββββββββββββββββββββ΄βββββββββββββββββββββββ΄ββββ΄ββββββββββββββββββββββ΄βββββββββββββββββββββββ΄βββββββββββ΄βββββββ΄βββββββββββββββββββββββ΄βββββββββββββββββββββββ΄ββββββββββββ€
β 30 rows 18 columns (14 shown) β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
D
I dunno, I toss duckit into my .zshrc like a sicko, but you could put it somewhere better and source it like a good
person. But however you get it into your shell, just duckit filename and you're g2g.