ferrum
Kiba
ferrum | Kiba | |
---|---|---|
9 | 7 | |
1,666 | 1,722 | |
1.9% | - | |
8.4 | 0.0 | |
22 days ago | over 1 year ago | |
Ruby | Ruby | |
MIT License | GNU Lesser General Public License v3.0 only |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
ferrum
- Generating PDFs in Rails using Grover
-
Learning Ruby Basics
What are you using for automation? There's a relatively new gem that I heard good things of, vessel: https://github.com/rubycdp/vessel . It uses ferrum under the hood, a set of Ruby bindings to Chrome/Chromium (https://github.com/rubycdp/ferrum).
-
Ruby web scraping gem that can handle JS?
I've used https://github.com/rubycdp/ferrum as driver for automated testing with capybara for which it works great. It recommends https://github.com/rubycdp/vessel as higher level abstraction for web scraping.
-
Automating Jekyll card generation with ruby’s Ferrum gem
require "Rubygems" require "Ferrum" def generate_card(browser, card, png, options={}) browser.go_to("http://localhost:4000/cards/#{card}") # see all the options here https://github.com/rubycdp/ferrum#screenshots browser.screenshot(path: "./images/cards/#{png}", full: true, # final image size is window_size x scale scale: 2) end browser = Ferrum::Browser.new(window_size: [800, 418]) # Check what cards we need to make Dir.glob("_posts/*").each do |post| post = File.basename(post, ".md") png = post + ".png" card = post + ".html" generate_card(browser, card, png) unless File.exists?("./images/cards/#{png}") end
- Best library for scraping dynamic page in Sidekiq background job (Selenium/Puppeteer/Cypress/Playwright)
- Ferrum – high-level API to control Chrome in Ruby
-
Migrating Selenium system tests to Cuprite
That is why we were happy to find out that a new ruby testing driver approach is being developed. It is called Cuprite, it runs the Ferrum library under the hood which, in turn, is an API that directly instruments the Chrome browser using the Chrome DevTools Protocol (CDP). About a week ago, we finally made a serious attempt to make our system test suite run on Cuprite, with especially two questions in our minds:
-
Web scraping with rails
I've used Ferrum for a couple small scripts in the past before.
-
My favorite Ruby gems
Ferrum
Kiba
-
Ask HN: What side projects landed you a job?
I started https://github.com/thbar/kiba#kiba-etl to scratch my own itch & be able to write properly structured ETL jobs in Ruby. It was a blank-slate rewrite of something larger (activewarehouse-etl) which I could not maintain anymore.
This landed me not strictly a job, but long term consulting gigs with a number of companies in EU, UK & US.
The job was directly related to the project: companies wanted the expertise of data engineering & ETL, often with Kiba directly, but also in general.
This "side project" was totally worth it :-)
-
Ruby's Hash Is a Swiss-Army Knife
Definitely! As a matter of fact, this is the default data structure I use when writing Ruby ETL code (e.g. https://github.com/thbar/kiba/wiki).
Methods like "except" (https://docs.ruby-lang.org/en/3.2/Hash.html#method-i-except) or "fetch" (raising an error on missing key) are very convenient to write defensive data processing code!
Similarly, in Elixir, I use Maps a lot for the same type of jobs (https://hexdocs.pm/elixir/1.15.4/Map.html), with similar properties.
-
Thinking in learn Ruby
Ruby has a very cool ETL library named Kiba that fits wonderfully with Ruby's strengths.
- What ETL tool do you use?
-
Massive SQL import from csv file, nulls, best practices.
Though it might be overkill for your problem, but have you had a look at [kiba-etl](https://github.com/thbar/kiba/blob/master/README.md)?
-
My favorite Ruby gems
Kiba
-
Ruby ETL Strategies: Organizing block-based Kiba Pipelines
If you don’t use Kiba, but work with data, check it out.
What are some alternatives?
Selenium WebDriver - A browser automation framework and ecosystem.
Nokogiri - Nokogiri (鋸) makes it easy and painless to work with XML and HTML from Ruby.
cuprite - Headless Chrome/Chromium driver for Capybara
data-science-with-ruby - Practical Data Science with Ruby based tools.
puppeteer - Headless Chrome Node.js API [Moved to: https://github.com/puppeteer/puppeteer]
Roo - Roo provides an interface to spreadsheets of several sorts.
Capybara - Acceptance test framework for web applications
chronicle-etl - 📜 A CLI toolkit for extracting and working with your digital history
puffing-billy - A rewriting web proxy for testing interactions between your browser and external sites. Works with ruby + rspec.
slay
puphpeteer - A Puppeteer bridge for PHP, supporting the entire API.
AW Datapipe - Unofficial ruby wrapper for the AWS SDK Data Pipeline API.