Apache Spark, Hive, and Spring Boot — Testing Guide

• apache-spark-integration-testing-example

  1 3 0.0 Java

  An example of integration tests for Apache Spark Spring Boot application that transfers data from Apache Hive table to Aerospike Database.

In this article, I'm showing you how to create a Spring Boot app that loads data from Apache Hive via Apache Spark to the Aerospike Database. More than that, I'm giving you a recipe for writing integration tests for such scenarios that can be run either locally or during the CI pipeline execution. The code examples are taken from this repository.

• shadow

  15 3,572 0.0 Groovy

  Gradle plugin to create fat/uber JARs, apply file transforms, and relocate packages for applications and libraries. Gradle version of Maven's Shade plugin.

The result .jar is going to submitted to Apache Spark cluster (e.g. spark-submit command). So, it should contain all runtime artefacts. Unfortunately, the standard Spring Boot packaging does not put the dependencies in the way Apache Spark expects it. So, we'll use shadow-jar Gradle plugin. Take a look at the example below.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• initializr

  260 3,361 8.8 Java

  A quickstart generator for Spring projects

  

The project is bootstrapped with Spring Initializr. Nothing special here. But the dependencies list should be clarified.

• Apache Spark

  101 38,530 10.0 Scala

  Apache Spark - A unified analytics engine for large-scale data processing

  

In this article, I'm showing you how to create a Spring Boot app that loads data from Apache Hive via Apache Spark to the Aerospike Database. More than that, I'm giving you a recipe for writing integration tests for such scenarios that can be run either locally or during the CI pipeline execution. The code examples are taken from this repository.

• Apache Hive

  14 5,344 9.6 Java

  Apache Hive

  

In this article, I'm showing you how to create a Spring Boot app that loads data from Apache Hive via Apache Spark to the Aerospike Database. More than that, I'm giving you a recipe for writing integration tests for such scenarios that can be run either locally or during the CI pipeline execution. The code examples are taken from this repository.

• Aerospike

  16 975 8.7 C

  Aerospike Database Server – flash-optimized, in-memory, nosql database

  

In this article, I'm showing you how to create a Spring Boot app that loads data from Apache Hive via Apache Spark to the Aerospike Database. More than that, I'm giving you a recipe for writing integration tests for such scenarios that can be run either locally or during the CI pipeline execution. The code examples are taken from this repository.

Suggest a related project