visidata VS q

I'd like to call out one of my favorite pieces of software from the past 10 years: VisiData [1] has completely changed the way I do ad-hoc data processing, and is now my go-to for pretty much all use cases that I previously used spreadsheets for, and about half of those I previously used databases for.

It's a TUI application, not strictly CLI, but scriptable, and I figure anyone building pipelines using tools like jq, q, awk, grep, etc. to process tabular data will find it extremely useful.

----

[1]: https://visidata.org

[4] "Is it possible to "flatten" structured data (like JSON?)": https://github.com/saulpw/visidata/discussions/1605

Visidata - https://visidata.org

You can try Visidata

VisiData is an awesome terminal spreadsheet tool. edbrowse for internet browsing.

Have you ever tried out visidata? It's not vim, but it's a terminal app with vim-like keybindings for visualizing tabular data (and it can convert from other types like json). Not quite a neovim buffer, but you could always open visidata in a new terminal buffer.

If you are a command-line user, try visidata[0]

[0] https://github.com/saulpw/visidata

The datasets will be available as SQLite, JSONL, and CSV. This will be great for sharpening your SQL/Python/VisiData skills.

Help Sarah find the family holiday tapestry before her father notices it's missing! Sharpen your SQL/Python/VisiData skills with Hanukkah of Data.

CSV utilities (still haven't pick a favorite one...): https://github.com/harelba/q https://github.com/BurntSushi/xsv https://github.com/wireservice/csvkit https://github.com/johnkerl/miller

I think they're talking about https://github.com/harelba/q, which is not very fast.

Apparently, there were many who thought the same thing; Tools to execute SQL against CSV were trdsql, q, csvq, TextQL. They were highly functional, hoewver, had many options and no input completion. I found it just a little difficult to use.

Hi, author of q here.

Regarding the error you got, q currently does not autodetect headers, so you'd need to add -H as a flag in order to use the "country" column name. You're absolutely correct on failing-fast here - It's a bug which i'll fix.

In general regarding speed - q supports automatic caching of the CSV files (through the "-C readwrite" flag). Once it's activated, it will write the data into another file (with a .qsql extension), and will use it automatically in further queries in order to speed things considerably.

Effectively, the .qsql files are regular sqlite3 files (with some metadata), and q can be used to query them directly (or any regular sqlite3 file), including the ability to seamlessly join between multiple sqlite3 files.

http://harelba.github.io/q/#auto-caching-examples