Spring Cloud Data Flow

Spring Cloud Data Flow is a cloud-native orchestration service for composable data microservices on modern runtimes. With Spring Cloud Data Flow, developers can create and orchestrate data pipelines for common use cases such as data ingest, real-time analytics, and data import/export.

Quick Start
Fork me on GitHub

Spring Cloud Data Flow is the cloud native redesign of Spring XD – a project that aimed to simplify development of Big Data applications. The streaming and batch modules from Spring XD are refactored into Spring Boot based data microservice applications that are now autonomous deployment units and they can "natively" run in modern runtimes such as Cloud Foundry, Apache YARN, Apache Mesos, and Kubernetes.

Spring Cloud Data Flow offers a collection of patterns and best practices for microservices-based distributed streaming and batch data pipelines.

Features

  • Orchestrate applications across a variety of distributed modern runtimes including: Cloud Foundry, Apache YARN, Apache Mesos, and Kubernetes
  • Separate runtime dependencies backed by ‘spring profiles’
  • Consume streaming and batch data microservices as maven or docker artifacts
  • Create, unit-test, troubleshoot and manage data microservices in isolation
  • Develop using: DSL, Shell, REST-APIs, Admin-UI, and Flo
  • Take advantage of metrics, health checks, and remote management at each data microservice level
  • Scale stream and batch pipelines without interrupting data flows

Quick Start

Step 1 - Clone Spring Cloud Data Flow repository

Step 2 - Start Redis [for: analytics store] and Kafka [for: message bus]

Step 3 - Build Project mvn clean package

Step 4 - Launch ‘Local’ Server [spring-cloud-dataflow-server-local/target] java -jar spring-cloud-dataflow-server-local-1.0.0.BUILD-SNAPSHOT.jar

Step 5 - Launch Shell [spring-cloud-dataflow-shell/target] java -jar spring-cloud-dataflow-shell-1.0.0.BUILD-SNAPSHOT.jar

Step 6 - Create ‘ticktock’ Stream dataflow:>stream create ticktock --definition "time | log" --deploy

Step 7 - Launch Admin UI at: http://localhost:9393/admin-ui

Release Trains

Component spring-cloud-dataflow.1.0.0.M2 spring-cloud-dataflow.1.0.0.M3
spring-cloud-deployer NA 1.0.0.M1
spring-cloud-deployer-cloudfoundry NA 1.0.0.M1
spring-cloud-deployer-yarn NA 1.0.0.M1
spring-cloud-deployer-mesos NA 1.0.0.M1
spring-cloud-deployer-kubernetes NA 1.0.0.M1
spring-cloud-dataflow-server-cloudfoundry 1.0.0.M1 1.0.0.M2
spring-cloud-dataflow-server-yarn 1.0.0.M1 1.0.0.M2
spring-cloud-dataflow-server-mesos 1.0.0.M1 1.0.0.M2
spring-cloud-dataflow-server-kubernetes 1.0.0.M1 1.0.0.M2
spring-cloud-stream 1.0.0.M3 1.0.0.RC3
spring-cloud-stream-app-starters NA 1.0.0.M1
spring-cloud-task NA 1.0.0.M2**
spring-cloud-task-app-starters NA 1.0.0.M1**
spring-cloud-dataflow-ui 1.0.0.M2 1.0.0.M3
Flo for spring-cloud-dataflow-ui 1.0.0.M2 1.0.0.M3
spring-cloud-dataflow-rpm NA 1.0-1
spring-cloud-dataflow-ambari-rpm NA 1.0-1

** compatible only with Spring Cloud Data Flow's Local or YARN servers