4. Concourse Pipeline

The repository contains an opinionated pipeline that will build and deploy - Github Webhook application.

All in all there are the following projects taking part in the whole microservice setup for this demo.

4.1 Step by step

If you want to just run the demo as far as possible using PCF Dev and Docker Compose

Below you can find optional steps needed to be taken when you want to customize the pipeline

4.1.1 Fork repos

There are 4 apps that are composing the pipeline

You need to fork only these. That’s because only then will your user be able to tag and push the tag to repo.

4.1.2 Start Concourse and Artifactory

Concourse + Artifactory can be run locally. To do that just execute the start.sh script from this repo.

git clone https://github.com/spring-cloud/spring-cloud-pipelines
cd spring-cloud-pipelines/concourse
./setup_docker_compose.sh
./start.sh 192.168.99.100

The setup_docker_compose.sh script should be executed once only to allow generation of keys.

The 192.168.99.100 param is an example of an external URL of Concourse (equal to Docker-Machine ip in this example).

Then Concourse will be running on port 8080 and Artifactory 8081.

Deploy the infra JARs to Artifactory

When Artifactory is running, just execute the tools/deploy-infra.sh script from this repo.

git clone https://github.com/spring-cloud/spring-cloud-pipelines
cd spring-cloud-pipelines/
./tools/deploy-infra.sh

As a result both eureka and stub runner repos will be cloned, built and uploaded to Artifactory.

4.1.3 Start PCF Dev

TIP: You can skip this step if you have CF installed and don’t want to use PCF Dev The only thing you have to do is to set up spaces.

[Warning]Warning

It’s more than likely that you’ll run out of resources when you reach stage step. Don’t worry! Keep calm and clear some apps from PCF Dev and continue.

You have to download and start PCF Dev. A link how to do it is available here.

The default credentials when using PCF Dev are:

username: user
password: pass
email: user
org: pcfdev-org
space: pcfdev-space
api: api.local.pcfdev.io

You can start the PCF dev like this:

cf dev start

You’ll have to create 3 separate spaces (email admin, pass admin)

cf login -a https://api.local.pcfdev.io --skip-ssl-validation -u admin -p admin -o pcfdev-org

cf create-space pcfdev-test
cf set-space-role user pcfdev-org pcfdev-test SpaceDeveloper
cf create-space pcfdev-stage
cf set-space-role user pcfdev-org pcfdev-stage SpaceDeveloper
cf create-space pcfdev-prod
cf set-space-role user pcfdev-org pcfdev-prod SpaceDeveloper

You can also execute the ./tools/cf-helper.sh setup-spaces to do this.

4.1.4 Setup the fly CLI

If you go to Concourse website you should see sth like this:

   

running concourse

   

You can click one of the icons (depending on your OS) to download fly, which is the Concourse CLI. Once you’ve downloaded that (and maybe added to your PATH) you can run:

fly --version

If fly is properly installed then it should print out the version.

4.1.5 Setup your credentials.yml

The repo comes with credentials-sample.yml which is set up with sample data (most credentials) are set to be applicable for PCF Dev. Copy this file to a new file credentials.yml (the file is added to .gitignore so don’t worry that you’ll push it with your passwords) and edit it as you wish. For our demo jus setup:

  • app-url - url pointing to your forked github-webhook repo
  • github-private-key - your private key to clone / tag GitHub repos
  • repo-with-binaries - the IP is set to the defaults for Docker Machine. You should update it to point to your setup

If you don’t have a Docker Machine just execute ./whats_my_ip.sh script to get an external IP that you can pass to your repo-with-binaries instead of the default Docker Machine IP.

Below you can see what environment variables are required by the scripts. To the right hand side you can see the default values for PCF Dev that we set in the credentials-sample.yml.

Property NameProperty DescriptionDefault value

BUILD_OPTIONS

Additional options you would like to pass to the Maven / Gradle build

 

PAAS_TEST_API_URL

The URL to the CF Api for TEST env

api.local.pcfdev.io

PAAS_STAGE_API_URL

The URL to the CF Api for STAGE env

api.local.pcfdev.io

PAAS_PROD_API_URL

The URL to the CF Api for PROD env

api.local.pcfdev.io

PAAS_TEST_ORG

Name of the org for the test env

pcfdev-org

PAAS_TEST_SPACE

Name of the space for the test env

pcfdev-space

PAAS_STAGE_ORG

Name of the org for the stage env

pcfdev-org

PAAS_STAGE_SPACE

Name of the space for the stage env

pcfdev-space

PAAS_PROD_ORG

Name of the org for the prod env

pcfdev-org

PAAS_PROD_SPACE

Name of the space for the prod env

pcfdev-space

REPO_WITH_BINARIES

URL to repo with the deployed jars

http://192.168.99.100:8081/artifactory/libs-release-local

M2_SETTINGS_REPO_ID

The id of server from Maven settings.xml

artifactory-local

PAAS_HOSTNAME_UUID

Additional suffix for the route. In a shared environment the default routes can be already taken

 

APP_MEMORY_LIMIT

How much memory should be used by the infra apps (Eureka, Stub Runner etc.)

256m

JAVA_BUILDPACK_URL

The URL to the Java buildpack to be used by CF

https://github.com/cloudfoundry/java-buildpack.git#v3.8.1

4.1.6 Build the pipeline

Log in (e.g. for Concourse running at 192.168.99.100 - if you don’t provide any value then localhost is assumed). If you execute this script (it assumes that either fly is on your PATH or it’s in the same folder as the script is):

./login.sh 192.168.99.100

Next run the command to create the pipeline.

./set_pipeline.sh

Then you’ll create a github-webhook pipeline under the docker alias, using the provided credentials.yml file. You can override these values in exactly that order (e.g. ./set-pipeline.sh some-project another-target some-other-credentials.yml)

4.1.7 Run the github-webhook pipeline

   

Figure 4.1. Click Login

concourse login

   

Figure 4.2. Pick main team

concourse team main

   

Figure 4.3. Log in with concourse user and changeme password

concourse user pass

   

Figure 4.4. Your screen should look more or less like this

concourse pipeline

   

Figure 4.5. Unpause the pipeline by clicking in the top lefr corner and then clicking the play button

start pipeline

   

Figure 4.6. Click 'generate-version'

generate version

   

Figure 4.7. Click + sign to start a new build

run pipeline

   

Figure 4.8. The job is pending

concourse pending

   

Figure 4.9. Job is pending in the main screen

job running

   

Figure 4.10. Job is running in the main screen

running pipeline

4.2 FAQ

4.2.1 Can I use the pipeline for some other repos?

Sure! Just change the app-url in credentials.yml!

4.2.2 Will this work for ANY project out of the box?

Not really. This is an opinionated pipeline that’s why we took some opinionated decisions like:

  • usage of Spring Cloud, Spring Cloud Contract Stub Runner and Spring Cloud Eureka
  • application deployment to Cloud Foundry
  • For Maven:

    • usage of Maven Wrapper
    • artifacts deployment by ./mvnw clean deploy
    • stubrunner.ids property to retrieve list of collaborators for which stubs should be downloaded
    • running smoke tests on a deployed app via the smoke Maven profile
    • running end to end tests on a deployed app via the e2e Maven profile
  • For Gradle (in the github-analytics application check the gradle/pipeline.gradle file):

    • usage of Gradlew Wrapper
    • deploy task for artifacts deployment
    • running smoke tests on a deployed app via the smoke task
    • running end to end tests on a deployed app via the e2e task
    • groupId task to retrieve group id
    • artifactId task to retrieve artifact id
    • currentVersion task to retrieve the current version
    • stubIds task to retrieve list of collaborators for which stubs should be downloaded

This is the initial approach that can be easily changed in the future.

4.2.3 Can I modify this to reuse in my project?

Sure! It’s open-source! The important thing is that the core part of the logic is written in Bash scripts. That way, in the majority of cases, you could change only the bash scripts without changing the whole pipeline. You can check out the scripts here.

Furthermore, if you only want to customize a particular function under common/src/main/bash, you can provide your own function under common/src/main/bash/<some custom identifier> where <some custom identifier> is equal to the value of the CUSTOM_SCRIPT_IDENTIFIER environment variable. For more on this technique, see Override specific project type pipeline functions.

4.2.4 I ran out of resources!!

When deploying the app to stage or prod you can get an exception Insufficient resources. The way to solve it is to kill some apps from test / stage env. To achieve that just call

cf target -o pcfdev-org -s pcfdev-test
cf stop github-webhook
cf stop github-eureka
cf stop stubrunner

You can also execute ./tools/cf-helper.sh kill-all-apps that will remove all demo-related apps deployed to PCF dev.

4.2.5 The rollback step fails due to missing JAR ?!

You must have pushed some tags and have removed the Artifactory volume that contained them. To fix this, just remove the tags

git tag -l | xargs -n 1 git push --delete origin

4.2.6 Can I see the output of a job from the terminal?

Yes! Assuming that pieline name is github-webhook and job name is build-and-upload you can running

fly watch --job github-webhook/build-and-upload -t docker

4.2.7 I clicked the job and it’s constantly pending…​

Don’t worry…​ most likely you’ve just forgotten to click the play button to unpause the pipeline. Click to the top left, expand the list of pipelines and click the play button next to github-webhook.

Another problem that might occur is that you need to have the version branch. Concourse will wait for the version branch to appear in your repo. So in order for the pipeline to start ensure that when doing some git operations you haven’t forgotten to create / copy the version branch too.

4.2.8 The route is already in use

If you play around with Jenkins / Concourse you might end up with the routes occupied

Using route github-webhook-test.local.pcfdev.io
Binding github-webhook-test.local.pcfdev.io to github-webhook...
FAILED
The route github-webhook-test.local.pcfdev.io is already in use.

Just delete the routes

yes | cf delete-route local.pcfdev.io -n github-webhook-test
yes | cf delete-route local.pcfdev.io -n github-eureka-test
yes | cf delete-route local.pcfdev.io -n stubrunner-test
yes | cf delete-route local.pcfdev.io -n github-webhook-stage
yes | cf delete-route local.pcfdev.io -n github-eureka-stage
yes | cf delete-route local.pcfdev.io -n github-webhook-prod
yes | cf delete-route local.pcfdev.io -n github-eureka-prod

You can also execute the ./tools/cf-helper.sh delete-routes

4.2.9 I’m unauthorized to deploy infrastructure jars

Most likely you’ve forgotten to update your local settings.xml with the Artifactory’s setup. Check out this section of the docs and update your settings.xml.