excon VS RESTClient

A internal network boundary is probably worth it for heavy jobs, since you usually don't want it to interfere with serving web requests (no matter the tech).

You probably already know what I would say to each of those examples.

> Rails timing out after 30s while allocating 500MB of memory (mostly) in ActiveRecord to compute 5MB of JSON to return to an API caller.

I can make a JS or Go program perform the same way. In fact the exact same thing happened in my shop with Go/Gorm. The key question is: how do you compute the 5mb of JSON? The devil is in those details. We changed the way we computed ours, and the issue was gone.

> 90% of request latency of ~10s spent waiting for downstream services to respond to requests. Most of these could be fired off concurrently (ie `Promise.all` in node). 9s/10s this Rails worker is sitting around doing nothing and eating up ~300MB of memory.

This sounds broken. Why is the worker doing nothing for 9 out of 10s? But like I said earlier, there are a bunch of ways to use HTTP1.1 pipelining to run them concurrently. (https://github.com/excon/excon and https://github.com/HoneyryderChuck/httpx support it, but you can also do that with Net::HTTP I believe) And you can still start threads, which are still concurrent while blocking on IO.

> trying to extract out Authorization to a centralized service (so that other extracted services don't have to call into the monolith in order to make authorization decisions) is a major pain as the monolith now has to make calls out to the centralized auth system to in order to make authz decisions.

This seems unrelated to Rails. Not sure why monolith can't continue handling authorization.

For practical reasons, I decided to re-implement Ruby's rest-client gem to dive into the Crystal.

The Met has a really robust API of their collection - over 470,000 distinct objects! Talk about a lot of data! I needed to figure out how to get at least a sampling of this data into my backend API. I found just what I was looking for in RestClient. In theory it is super easy to use and I will give a few quick steps below. I say in theory because I think it depends on how well set up the API is that you are pulling your data from. If you are wondering, the Met's was really easy sometimes and really hard others - some of the art objects were missing attributes that I wanted to pull so I had hand hold my seed file during some of the departments. For brevity sake, I will use one of the easier bit of data to pull - the Met's list of departments. (seeding the art objects themselves was another story all together!)

I am considering to use the rest-client gem, because many of my call need nested data. But I actually don't think they support the structure Microsoft are using for their "rest" api.

API *An application programming interface ( API) is a computing interface which defines interactions between multiple…*en.wikipedia.org rest-client/rest-client *A simple HTTP and REST client for Ruby, inspired by the Sinatra's microframework style of specifying actions: get, put…*github.com JSON *JavaScript Object Notation ( JSON, pronounced ; also ) is an open standard file format, and data interchange format…*en.wikipedia.org Web API | Spotify for Developers *Note: By using Spotify developer tools, you accept the Spotify Developer Terms of Service. Based on simple REST…*developer.spotify.com Ruby-Doc.org *Fast, searchable Ruby documentation for core and standard libraries. Plus, links to tutorials, guides, books, and…*ruby-doc.org