unify-jdocs VS hof

JSONPath is good when it comes to querying large JSON documents. But in my opinion, more than this is the need to simplify reading and writing from JSON documents. We use POJOs / model classes which can become a chore for large JSON documents. While it is possible to read paths, I had not seen any tool using which we could read and write JSON paths in a document without using POJOs. And so I wrote unify-jdocs - read and write any JSON path with a single line of code without ever using POJOs. And also use model documents to replace JSONSchema. You can find this library here -> https://github.com/americanexpress/unify-jdocs.

I went the other way - because I so much disliked working with DTOs to work with JSON, I wrote a library that allows you to get rid of DTOs in the first place. No more POJO / DTO in order to work with JSON data and much more. You could take a look at https://github.com/americanexpress/unify-jdocs. I think you will find it interesting!

"And that’s when a real problem emerged; nobody knew who I was" and "I felt invisible, like I was all alone in this world. I could have had the best thing in the world, and no one would care": Could not agree with you more on this - over the past many years it has been more and more important to build a digital brand for yourself and have many followers - to be networked with people and know people who can provide you with reach - something which is difficult for introverts and challenging to do now since I have crossed 50. I personally have felt this helplessness in trying to promote a couple of opensource Java libraries I released on behalf of my employer (shameless plug here -> https://github.com/americanexpress/unify-jdocs and https://github.com/americanexpress/unify-flowret). I thought I had done something of value which would be readily adopted by people after they saw it - guess what - first I have not been able to get through to a wide audience and second - I underestimated what it takes to get people out of their comfort zone and their traditional thinking. It is so difficult for people to accept that there may be better ways of doing things once they get used to a certain way. Anyway, I don't think you failed - you learnt and it is never too late to learn and do something new. Something will click eventually and I wish you all the very best.

Auther here - thanks for responding! I really do appreciate it.

> The trend seems to be in the opposite direction. People became frustrated with the lack of types in Python and JavaScript, hence we get Python with typing and TypeScript.

Regarding typing, what I have tried to achieve is the best of both worlds. In unify-jdocs, we have the concept of typed document. This is a document that defines the structure of the JSON document. It defines what the "type" of a leaf node is i.e. integer, string etc. The validation / determination against this type is however done at runtime. This, I feel is acceptable because whenever we add a JSON path to a document, the first thing we would do is to test the read / write of that path. And any type mismatch would get caught there immediately. And so, from the point of view of being able to read / write / validate in a single line of code (even though dynamically) provides much simplicity and ease of use as compared to using POJOs. Plus we always know the exact JSON path we are dealing with.

Usually, I would prefer static typing but, in a scenario, where we can have hundreds of JSON document types (we deal with more than 500 in the same application), complex JSON document (ours go down more than ten levels deep with hundreds of JSON paths) and where the document structures may undergo change over project lifetime, the use of read / writing using a single line of code has many benefits. I shudder to think of hundreds (if not thousands) of POJO classes, the writing of accessor methods (null / empty handling, namespace etc.) and what it would take to refactor in the face of change. Just my opinion based on my experience in the past.

> I think you will find less traction with this method of posting to HN. People want a clickable link to look at the project

The clickable link to the repo / documentation is actually in the text (https://github.com/americanexpress/unify-jdocs). In the case of this post, I felt it more important for people to read the text rather than be directly pointed to the contents of the link and hence this approach.

Nice work! I see that that this is for processing / parsing large data sets and where documents do not conform to a fixed structure and for Go language.

I made something similar in Java - unify-jdocs - https://github.com/americanexpress/unify-jdocs - though this is not for parsing - it is more for reading and writing when the structure of the document is known - read and write any JSONPath in one line of code and use model documents to define the structure of the data document (instead of using JSONSchema which I found very unwieldy to use) - no POJOs or model classes - along with many other features. Posting here as the topic is relevant and it may help people in the Java world. We have used it intensively within Amex for a very large complex project and it has worked great for us.

Nice! Talking of JSON schemas and validating JSON documents against schemas, for Java, I wrote unify-jdocs where I do not use JSON schemas but still do validations (I found them unwieldy to use and was looking for something simpler). You can find details here -> https://github.com/americanexpress/unify-jdocs. Also, no POJOs / model classes, just reading and writing JSON paths in a single line of code. It's helped us tremendously in managing complexity in a very large internal project. I am hoping it helps others.

This article got my attention. Related to what you are saying, in Java, the problem that I was really fed up of was creating domain specific JSON object models to map the JSON documents into to use in code. In other words, mapping JSON to rigidly typed language structure. Its boiler plate, is tedious to do (as the author points out in the article), difficult to change and usually a pain. I solved this problem by creating unify-jdocs which completely eliminates the need to create object models or POJO classes to represent your JSON object. You can read more about it here -> https://github.com/americanexpress/unify-jdocs

For a auth product, you probably should go with traditional code generation, not LLMs (re: all the unsolvable problems they have (hallucinations, incomplete code answers). For an auth product, you want something definite and reproducible, or you are asking for trouble.

If you are interested in a flexible code gen framework, I built one that had this very use case in mind. (https://github.com/hofstadter-io/hof)

I'd be happy to think / talk this through with you

I recently built a TUI for exploring and developing CUE, JSON, Yaml.

Large inspiration for the overall UX comes from the Bloomberg terminal, where every function is reachable with four letter shortcut from the command box. It was really the command box that I liked. I've also built a sort of "flex" panel component for the layout so you can create as many panels as you like.

https://docs.hofstadter.io/getting-started/hof-tui/

https://github.com/hofstadter-io/hof/tree/_dev/lib/tui

Many business analysts use SQL, have for a long time. They are probably not your target audience. With the problem being JAVA specific, you'd likely want to start there

This sounds similar to the goals of my hof tool (https://github.com/hofstadter-io/hof), lift type definitions out of code so they can be defined in one place, then generate the code for all the places. Is that sounding like what you are after?

Having built something similar, the biggest challenge for users is that they have to use a bespoke language, like WASP here. I suspect that it is also your biggest challenge as well.

Mine is built on CUE, which at least has the potential to become a more widely used language. CUE hasn't reached sufficient maturity for broader adoption yet, so I continue to face this same problem.

https://github.com/hofstadter-io/hof

Here's a big one I needed to get ChatGPT to do something more sophisticated with a JSON object response (predates functions and all that)

https://github.com/hofstadter-io/hof/blob/_dev/flow/chat/pro...

It no longer worked after a model update some time ago, haven't tried recently.

I found codellama to be much better for this and require fewer instructions, an anecdotal validation for smaller, focussed models

I was surprised out how you can define arbitrary grammars using arbitrary formulation and it would follow it. Of course you have to redo the prompt every time there is an update... such a pain

https://github.com/hofstadter-io/hof/blob/_dev/flow/chat/pro...

CUE is indeed a beautiful language, will get those mind juices flowing for sure!

There is more work to be done on the codec implementation, but if you just want to split yaml/json across files, CUE is a great option

You might also like my project, built on CUE: https://github.com/hofstadter-io/hof We have a TUI where you can explore and work with CUE, JSON, Yaml

You can pretty much make up any pseudo grammar like this one, which is a reduced JSON object that is close to CUE: https://github.com/hofstadter-io/hof/blob/_dev/flow/chat/pro...

No need to be formal or use a standard format

Yea, in particular for this project, they have created a bespoke templating system.

You can get the same thing with Go text/templates by adding chat function(s) as custom a helper: https://github.com/hofstadter-io/hof/blob/_dev/lib/templates...

100% one of the best things about building a TUI is not having the pain of modern web development. I do think there is a way to have a CLI & TUI come from the same code, so you can get the best of both, or pick the best for the task at hand.

experiments in progress here: https://github.com/hofstadter-io/hof/tree/_dev/lib/tui