SaaSHub helps you find the best software and product alternatives Learn more →
Top 23 Go JSON Projects
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
jsoniter
A high-performance 100% compatible drop-in replacement of "encoding/json" (by json-iterator)
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
-
-
miller
Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON
-
-
Ponzu
Headless CMS with automatic JSON API. Featuring auto-HTTPS from Let's Encrypt, HTTP/2 Server Push, and flexible server framework written in Go.
-
-
dasel
Select, put and delete data from JSON, TOML, YAML, XML and CSV files with a single tool. Supports conversion between formats and can be used as a Go package.
-
octosql
OctoSQL is a query tool that allows you to join, analyse and transform data from multiple databases and file formats using SQL.
-
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
Someone made a benchmark of serialization libraries in go [1], and I was surprised to see gobs is one of the slowest ones, specially for decoding. I suspect part of the reason is that the API doesn't not allow reusing decoders [2]. From my explorations it seems like both JSON [3], message-pack [4] and CBOR [5] are better alternatives.
By the way, in Go there are a like a million JSON encoders because a lot of things in the std library are not really coded for maximum performance but more for easy of usage, it seems. Perhaps this is the right balance for certain things (ex: the http library, see [6]).
There are also a bunch of libraries that allow you to modify a JSON file "in place", without having to fully deserialize into structs (ex: GJSON/SJSON [7] [8]). This sounds very convenient and more efficient that fully de/serializing if we just need to change the data a little.
--
1: https://github.com/alecthomas/go_serialization_benchmarks
2: https://github.com/golang/go/issues/29766#issuecomment-45492...
--
3: https://github.com/goccy/go-json
4: https://github.com/vmihailenco/msgpack
5: https://github.com/fxamacker/cbor
--
6: https://github.com/valyala/fasthttp#faq
--
7: https://github.com/tidwall/gjson
8: https://github.com/tidwall/sjson
Since most of the time would be spent decoding json, you could try to cut this time using https://github.com/bytedance/sonic or https://github.com/json-iterator/go, both are drop-in replacements for the stdlib, sonic is faster.
What I often use to just get the full key paths is yq (https://github.com/mikefarah/yq), piping into grep when necessary
yq -o=props
Be aware that there is a performance impact compared to using zerolog directly [0] (my uneducated guess is it is likely due to pointer indirection).
[0]: https://github.com/rs/zerolog/issues/571#issuecomment-166202...
I see, thanks for replying and no worries! yeap some of the "self-describing" formats like msgpack, cbor etc will because of how fq works have to be decoded into something more of a meta-msgpack etc.
About blobs, if you want to change how (possibly large) binaries are represented as JSON you can use the bits_format options, see https://github.com/wader/fq/blob/master/doc/usage.md#options, so fq -o bits_format=md5 torepr ...
I can highly recommend to learn jq, it's what makes fq really useful, and as a bonus you will learn jq in general! :)
Project mention: The API database architecture – Stop writing HTTP-GET endpoints | news.ycombinator.com | 2024-05-10Yeah, I fully agree. The tooling for putting that much logic into the database is just not great. I've been decently happy with Sqitch[0] for DB change management, but even with that you don't really get a good basis for testing some of the logic you could otherwise test in isolation in app code.
I've also tried to rely heavily on the database handling security and authorization, but as soon as you start to do somewhat non-trivial attribute-/relationship-based authorization (as you would find in many products nowadays), it really isn't fun anymore, and you spend a lot of the time you saved on manually building backend routes on trying to fit you authz model into those basic primitives (and avoiding performance bottlenecks). Especially compares to other modern authz solutions like OPA[1] or oso[2] it really doesn't stack up.
[0]: https://github.com/sqitchers/sqitch
[1]: https://www.openpolicyagent.org
[2]: https://www.osohq.com
Also, something that comes pretty close to what I want to do is stash. However, it doesn't handle ebooks, and my library is not nsfw and shouldn't rely on an external database (as no external database for my field of interest exists afaik). But stash ticks almost all the requirements.
Project mention: Introducing astjson: Transform and Merge JSON Objects with Unmatched Speed in Go | dev.to | 2023-11-29In this article, I will introduce you to a new package called astjson that I have been working on for the last couple of weeks. It is a Go package that allows you to transform and merge JSON objects with unmatched speed. It is based on the jsonparser package by buger aka Leonid Bugaev and extends it with the ability to transform and merge JSON objects at unparalleled performance.
Project mention: HCL: Toolkit for Structured Configuration Languages | news.ycombinator.com | 2024-04-23
Project mention: Wazero: Zero dependency WebAssembly runtime written in Go | news.ycombinator.com | 2023-07-01Never got it to anything close to a finished state, instead moving on to doing the same prototype in llvm and then cranelift.
That said, here's some of the wazero-based code on a branch - https://github.com/cube2222/octosql/tree/wasm-experiment/was...
It really is just a very very basic prototype.
You might want to look at tsv-utils, or a similar project: https://github.com/eBay/tsv-utils
For the SQL part, but maybe a lot heavier, you can use one of the projects listed on this page: https://github.com/multiprocessio/dsq (No longer maintained, but has links to lots of other projects)
GitHub repo: https://github.com/Jeffail/gabs
Yeap i've talked to itchyny quite a lot about various changes https://github.com/itchyny/gojq/issues/153 and also upstreamed quite a lot https://github.com/itchyny/gojq/issues?q=author%3Awader like custom iterators (to allow eval, own iterators and "empty" functions), query marshalling (query rewrite tricks) and a bunch of small things and bug fixes. But the largest change to add a JQValue interface is quite complex, other changes like extended literals is also a bit tricky.
Hmm weird list of changes for https://github.com/wader/gojq/compare/fq...itchyny:gojq:main but i guess it is because i haven't kept my main branch in sync. The fq branch should be based on latest gojq/main as of now. I usually try to rebase as quick as possible.
Let me know if you have any other questions or want to help out! maybe email etc as i usually don't check HN comments replies that often :)
For go -> you can actually get away with the standard json encoding package. Or if you want a slightly better one, I prefer goccy/go-json
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
Go JSON related posts
-
Using ARG in a Dockerfile – beware the gotcha
-
HCL: Toolkit for Structured Configuration Languages
-
How to Visualize and Analyze Data in Open Source Communities
-
Show HN: Flatito, grep for YAML and JSON files
-
OSS: Relicense to Apache 2 Globally
-
Bash/Zsh autocomplete for JSON fields
-
Fx 32.0, now with YAML support too
-
A note from our sponsor - SaaSHub
www.saashub.com | 20 May 2024
Index
What are some of the best open-source JSON projects in Go? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | urfave/cli | 21,702 |
| 2 | fx | 18,570 |
| 3 | GJSON | 13,675 |
| 4 | jsoniter | 13,100 |
| 5 | yq | 10,911 |
| 6 | zerolog | 9,900 |
| 7 | fq | 9,422 |
| 8 | OPA (Open Policy Agent) | 9,185 |
| 9 | miller | 8,598 |
| 10 | stash | 8,291 |
| 11 | jid | 6,809 |
| 12 | Ponzu | 5,645 |
| 13 | jsonparser | 5,357 |
| 14 | hcl | 5,117 |
| 15 | dasel | 4,889 |
| 16 | octosql | 4,707 |
| 17 | easyjson | 4,358 |
| 18 | dsq | 3,652 |
| 19 | minify | 3,604 |
| 20 | gorequest | 3,398 |
| 21 | gabs | 3,384 |
| 22 | gojq | 3,103 |
| 23 | go-json | 2,800 |
Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com