Hacker News API VS devdocs

I built a plugin that allows Coze bots to make requests to the Hacker News API.

I recognise the huge amounts of effort involved in this and I applaud the moderators for keeping HN an interesting place to be.

That said, I think it's reasonable for us to have visibility on their manual interventions, and this could be easily surfaced via the Hacker News API (https://github.com/HackerNews/API), if the Story JSON included the values of "contro," "bury," and "gag" fields, which are currently opaque to users of the API

See https://medium.com/hacking-and-gonzo/how-hacker-news-ranking... for more discussion on terminology

Google probably stopped updating BigQuery when they started hosting live Hacker News data in Firebase: https://github.com/HackerNews/API

The live nature of the Firebase data is awesome, but the lack of ability to query is a loss.

Nice work! I did something similar with a personal project a few months ago using an open source llm. Also, not sure if you know, there is an api you can use. https://github.com/HackerNews/API

Hi HN, I made a live feed for viewing all of the new items on Hacker News in almost-realtime. It is a single, modern HTML file that doesn't use polling.

It works by establishing a websocket connection to the Hacker News Firebase database, to receive updates every time the HN server updates Firebase, which is about once every 30 seconds. This is very efficient, putting no load on HN's server and using minimal bandwidth.

To make the feed continuous despite the delay, it waits to display each item until exactly 30 seconds before displaying it. I think this is a fair tradeoff, it gives you a sense of how active HN is. For comparision, there are something like 6-7 thousand tweets every second.

Official HN Firebase API: https://github.com/HackerNews/API

Source code: https://github.com/jerbear2008/hn-live/blob/main/index.html

You could use the API: https://github.com/HackerNews/API

You can now get Hacker News data in real time from the Hacker News API powered by Firebase: https://github.com/HackerNews/API

This is great (real time!), but also kind of a pain (38+ millions individual http requests to get the whole thing).

Thankfully there's no authentication or apparent rate limiting. I fumbled my way through downloading the whole thing with curl. I screwed up a few times so made over 70 million requests in total.

Toy analysis of the data I downloaded here: https://public.tableau.com/app/profile/isna/viz/HackerNewsDa...

Get your submissions from this API

https://hacker-news.firebaseio.com/v0/user/ohjeez.json?print...

and then scan the "submitted" articles as described here

https://github.com/HackerNews/API

I have a crawler that sucks down all the posts from HN and then I read it into Pandas and write all sorts of queries. The boggle I have now is that I want to use the same system to (1) make sure YOShInOn never submits duplicate articles, and (2) have accurate vote and comment scores. (1) requires picking up articles as soon as possible, (2) requires waiting two weeks or so until the scores have settled down to what they are going to be. I guess I gotta come back and rescan things in 2 weeks so I have the right scores.

Hi HN,

I am Jiayuan, and I'm here to introduce a tool we've been building over the past few months: Devv (https://devv.ai). In simple terms, it is an AI-powered search engine specifically designed for developers.

Now, you might ask, with so many AI search engines already available—Perplexity, You.com, Phind, and several open-source projects—why do we need another one?

We all know that Generative Search Engines are built on RAG (Retrieval-Augmented Generation)[1] combined with Large Language Models (LLMs). Most of the products mentioned above use indexes from general search engines (like Google/Bing APIs), but we've taken a different approach.

We've created a vertical search index focused on the development domain, which includes:

- Documents: These are essentially the single source of truth for programming languages or libraries; I believe many of you are users of Dash (https://kapeli.com/dash) or devdocs (https://devdocs.io/).

- Code: While not natural language, code contains rich contextual information. If you have a question related to the Django framework, nothing is more convincing than code snippets from Django's repository.

- Web Search: We still use data from search engines because these results contain additional contextual information.

Our reasons for doing this include:

- The quality of the index is crucial to the RAG system; its effectiveness determines the output quality of the entire system.

- We focus more on the Index (RAG) rather than LLMs because LLMs evolve rapidly; even models performing well today may be superseded by better ones in a few months, and fine-tuning an LLM now has relatively low costs.

- All players are currently exploring what kind of LLM product works best; we hope to contribute some different insights ourselves (and plan to open source parts of our underlying infrastructure in return for contributions back into open source communities).

Some brief product features:

- Three modes: - Fast mode: Offers quick answers within seconds. - Agent mode: For complex queries where Devv Agent infers your question before selecting appropriate solutions. - GitHub mode(currently in beta): Links directly with your own GitHub repositories allowing inquiries about specific codebases.

- Clean & intuitive UI/UX design.

- Currently only available as web version but Chrome extension & VSCode plugin planned soon!

Technical details regarding how we build our Index:

- Documents section involves crawling most documentation sources using scripts inspired by devdocs project’s crawler logic then slicing them up according function/symbol dimensions before embedding into vector databases;

- Codes require special treatment beyond just embeddings alone hence why custom parsers were developed per language type extracting logical structures within repos such as architectural layouts calling relationships between functions definitions etc., semantically processed via LMM;

- Web searches combine both selfmade indices targeting developer niches alongside traditional API based methods. We crawled relevant sites including blogs forums tech news outlets etc..

For the Agent Mode, we have actually developed a multi-agent framework. It first categorizes the user's query and then selects different agents based on these categories to address the issues. These various agents employ different models and solution steps.

Future Plans:

- Build a more comprehensive index that includes internal context (The Devv for Teams version will support indexing team repositories, documents, issue trackers for Q&A)

- Fully localized: All of the above technologies can be executed locally, ensuring privacy and security through complete localization.

Devv is still in its very early stages and can be used without logging in. We welcome everyone to experience it and provide feedback on any issues; we will continue to iterate on it.

[1]: https://arxiv.org/abs/2005.11401

> I've started to preface all python searches with 'site:python.org'

You might find DevDocs to be useful: https://devdocs.io/

DevDocs - Aggregates documentation from various sources into a single, easy-to-navigate interface, covering frontend and backend technologies. DevDocs

DevDocs, an offline API documentation browser, supports multilingual, offering developers a quick and efficient way to access tech docs. From front-end to back-end and mobile development, it integrates official documentation, providing a sleek, user-friendly interface.

Environmental lighting conditions rule the day! I have astigmatism and I prefer bright backgrounds; #000 text on #fff backgrounds works great for me, but that's because I work in a room lit by a 250W 30,000 lumen corn-cob LED bulb[0] that makes my small office as bright on the inside as the shaded ground from a tree on an overcast day (which is quite bright compared to usual indoor lighting). In a room that bright, high contrast text works great and is highly readable, with "dark mode" often looking washed out and muddy. Even small reductions in contrast (such as what https://devdocs.io does with text of #333 in light mode) can make me notice and wish for greater contrast.

[0] - https://www.benkuhn.net/lux/

Here's how to add a new scraper: https://github.com/freeCodeCamp/devdocs/blob/main/.github/CO...

Or open an issue and wait for somebody else to implement the scraper.

Imagine a single, intuitive platform where you can access comprehensive documentation for a vast array of programming languages, frameworks, libraries, and tools. That's the magic of DevDocs. This exceptional resource eliminates the frustration of juggling multiple tabs and websites in your quest for information. DevDocs brings everything together into one easy-to-use interface.

Mosh for a stable connection, Offline documentation such as msdn, wikipedia (via kiwi etc), zeal for local access to https://devdocs.io/; Self host tabby for ai autocompletion. For many shell programs check what mulinux was using back then, and what are the modern replacements such as elinks instead of links. Mutt for mail, for irc doesn't matter much, use a desktop one but setup a bouncher on a vps, I used to have one on a raspberry pi 1, you can use rss reader for reddit (not sure if still works) and blogs