Top 23 Scala Spark Projects

• Apache Spark

  101 38,414 10.0 Scala

  Apache Spark - A unified analytics engine for large-scale data processing

  

Project mention: "xAI will open source Grok" | news.ycombinator.com | 2024-03-11

• delta

  69 6,919 9.8 Scala

  An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs (by delta-io)

  

Project mention: Delta Lake vs. Parquet: A Comparison | news.ycombinator.com | 2024-01-19

Delta is pretty great, let's you do upserts into tables in DataBricks much easier than without it.

I think the website is here: https://delta.io

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• SynapseML

  18 4,970 9.0 Scala

  Simple and Distributed Machine Learning

  

Project mention: FLaNK Stack Weekly for 12 September 2023 | dev.to | 2023-09-12

• spark-nlp

  87 3,695 9.3 Scala

  State of the Art Natural Language Processing

  

Project mention: Spark NLP 5.1.0: Introducing state-of-the-art OpenAI Whisper speech-to-text, OpenAI Embeddings and Completion transformers, MPNet text embeddings, ONNX support for E5 text embeddings, new multi-lingual BART Zero-Shot text classification, and much more! | /r/Python | 2023-09-06

• deequ

  17 3,134 7.4 Scala

  Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.

  

• Quill

  15 2,136 9.0 Scala

  Compile-time Language Integrated Queries for Scala

  

Project mention: Dear Sir, You Have Built a Compiler (2022) | news.ycombinator.com | 2023-08-17

https://github.com/zio/zio-quill

This library does exactly what you prescribe. Pretty sure under the hood it's using macros with string templates

• kyuubi

  1 1,941 9.8 Scala

  Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.

  

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

• spark-cassandra-connector

  1 1,930 5.1 Scala

  DataStax Connector for Apache Spark to Apache Cassandra (by datastax)

  

• Jupyter Scala

  6 1,564 9.0 Scala

  A Scala kernel for Jupyter

  

• mleap

  1 1,494 5.2 Scala

  MLeap: Deploy ML Pipelines to Production

  

• LearningSparkV2

  1 1,095 0.0 Scala

  This is the github repo for Learning Spark: Lightning-Fast Data Analytics [2nd Edition]

  

• adam

  3 967 6.1 Scala

  ADAM is a genomics analysis platform with specialized file formats built using Apache Avro, Apache Spark, and Apache Parquet. Apache 2 licensed.

  

• H2O

  0 952 7.7 Scala

  Sparkling Water provides H2O functionality inside Spark cluster

  

• tispark

  2 878 5.1 Scala

  TiSpark is built for running Apache Spark on top of TiDB/TiKV

  

• frameless

  9 870 8.1 Scala

  Expressive types for Spark.

  

• incubator-livy

  4 856 5.2 Scala

  Apache Livy is an open source REST interface for interacting with Apache Spark from anywhere.

  

• spark-daria

  4 742 0.0 Scala

  Essential Spark extensions and helper methods ✨😲

• spark-rapids

  3 722 9.8 Scala

  Spark RAPIDS plugin - accelerate Apache Spark with GPUs

  

• delta-sharing

  4 676 7.8 Scala

  An open protocol for secure data sharing

  

Project mention: Azure data lake - Data Share | /r/dataengineering | 2023-06-29

• sparkMeasure

  1 642 7.5 Scala

  This is the development repository for sparkMeasure, a tool and library designed for efficient analysis and troubleshooting of Apache Spark jobs. It focuses on easing the collection and examination of Spark metrics, making it a practical choice for both developers and data engineers.

• spline

  1 582 6.3 Scala

  Data Lineage Tracking And Visualization Solution (by AbsaOSS)

  

• metorikku

  0 576 2.4 Scala

  A simplified, lightweight ETL Framework based on Apache Spark

  

• spark-solr

  2 445 2.8 Scala

  Tools for reading data from Solr as a Spark RDD and indexing objects from Spark into Solr using SolrJ.

  

Project mention: How to store 175 million rows and query them | /r/datasets | 2023-05-10

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

Scala Spark related posts

• Spark NLP 5.1.0: Introducing state-of-the-art OpenAI Whisper speech-to-text, OpenAI Embeddings and Completion transformers, MPNet text embeddings, ONNX support for E5 text embeddings, new multi-lingual BART Zero-Shot text classification, and much more!

  1 project | /r/Python | 6 Sep 2023

• Azure data lake - Data Share

  1 project | /r/dataengineering | 29 Jun 2023

• Pandas was faster and less memory intensive then crealytics pyspark. How is it possible?

  2 projects | /r/dataengineering | 17 Jun 2023

• The "Big Three's" Data Storage Offerings

  2 projects | /r/dataengineering | 15 Jun 2023

• Medallion/lakehouse architecture data modelling

  1 project | /r/dataengineering | 3 Jun 2023

• How to build a data pipeline using Delta Lake

  2 projects | dev.to | 19 May 2023

• PySpark for NLP Workshop - Materials and Jupyter Notebooks

  2 projects | /r/dataengineering | 14 May 2023
• A note from our sponsor - SaaSHub
  www.saashub.com | 6 May 2024

  SaaSHub helps you find the best software and product alternatives Learn more →

Index

Sponsored

SaaSHub - Software Alternatives and Reviews

SaaSHub helps you find the best software and product alternatives

www.saashub.com

Do not miss the trending Scala projects with our weekly report!