What is the best solution to unique data in golang

• jsoniter

  12 13,100 0.0 Go

  A high-performance 100% compatible drop-in replacement of "encoding/json" (by json-iterator)

  

I think you have to parse the json, if you dont know exactly what you are looking for and want some validation und prevent manual parsing errors. For parsing big json files it is recommend to read and decode it as stream. Here is an example. If you have serious performance criteria take a look at jsoniter. It can be used as 1 to 1 replacement for standard library.

• go-sqlite3

  40 7,495 6.2 C

  sqlite3 driver for go using database/sql

Once in sqlite you can use something like https://gorm.io/ (or use a golang sqlite package like: github.com/mattn/go-sqlite3) to work with the data.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• GORM

  74 35,589 8.5 Go

  The fantastic ORM library for Golang, aims to be developer friendly

  

Once in sqlite you can use something like https://gorm.io/ (or use a golang sqlite package like: github.com/mattn/go-sqlite3) to work with the data.

• dasel

  44 4,889 8.1 Go

  Select, put and delete data from JSON, TOML, YAML, XML and CSV files with a single tool. Supports conversion between formats and can be used as a Go package.

Or try dasel, which is like jq but written in Go, and if it works, look at the source to see how they do it :-)

• jstream

  1 553 0.0 Go

  Streaming JSON parser for Go

I suggest to use a streaming library to parse your file. Like jstream or simdjson-go

• simdjson-go

  6 1,764 4.0 Go

  Golang port of simdjson: parsing gigabytes of JSON per second

  

I suggest to use a streaming library to parse your file. Like jstream or simdjson-go

Suggest a related project