10. Jenkins Pipeline (Cloud Foundry)

[Important]Important

In this chapter, we assume that you deploy your application to Cloud Foundry PaaS.

The Spring Cloud Pipelines repository contains job definitions and the opinionated setup pipeline, which uses the Jenkins Job DSL plugin. Those jobs form an empty pipeline and a opinionated sample pipeline that you can use in your company.

The following projects take part in the microservice setup for this demo.

10.1 Step-by-step

This is a guide for the Jenkins Job DSL based pipeline.

If you want only to run the demo as far as possible using PCF Dev and Docker Compose, do the following:

10.1.1 Fork Repositories

Four applications compose the pipeline:

You need to fork only the following, because only then can you tag and push the tag to your repository:

10.1.2 Start Jenkins and Artifactory

Jenkins + Artifactory can be ran locally. To do so, run the start.sh script from this repository. The following listing shows the script:

git clone https://github.com/spring-cloud/spring-cloud-pipelines
cd spring-cloud-pipelines/jenkins
./start.sh yourGitUsername yourGitPassword yourForkedGithubOrg

Then Jenkins runs on port 8080, and Artifactory runs on port 8081. The parameters are passed as environment variables to the Jenkins VM, and credentials are set. That way, you need not do any manual work on the Jenkins side. In the above parameters, the third parameter could be yourForkedGithubOrg or yourGithubUsername. Also the REPOS environment variable contains your GitHub org (in which you have the forked repos).

Instead of the Git username and password parameters, you could pass -key <path_to_private_key> (if you prefer to use key-based authentication with your Git repositories).

Deploy the Infra JARs to Artifactory

When Artifactory is running, run the tools/deploy-infra.sh script from this repo. The following listing shows the script:

git clone https://github.com/spring-cloud/spring-cloud-pipelines
cd spring-cloud-pipelines/
./tools/deploy-infra.sh

As a result, both the eureka and stub runner repositories are cloned, built, and uploaded to Artifactory.

10.1.3 Start PCF Dev

[Tip]Tip

You can skip this step if you have CF installed and do not want to use PCF Dev. In that case, the only thing you have to do is to set up spaces.

[Warning]Warning

Servers often run run out of resources at the stage step. If that happens clear some apps from PCF Dev and continue.

You have to download and start PCF Dev, as described here.

The default credentials when using PCF Dev are as follows:

username: user
password: pass
email: user
org: pcfdev-org
space: pcfdev-space
api: api.local.pcfdev.io

You can start PCF Dev as follows:

cf dev start

You must create three separate spaces, as follows:

cf login -a https://api.local.pcfdev.io --skip-ssl-validation -u admin -p admin -o pcfdev-org

cf create-space pcfdev-test
cf set-space-role user pcfdev-org pcfdev-test SpaceDeveloper
cf create-space pcfdev-stage
cf set-space-role user pcfdev-org pcfdev-stage SpaceDeveloper
cf create-space pcfdev-prod
cf set-space-role user pcfdev-org pcfdev-prod SpaceDeveloper

You can also run the ./tools/cf-helper.sh setup-spaces script to do this.

10.1.4 Run the Seed Job

We created the seed job for you, but you have to run it. When you do run it, you have to provide some properties. By default we create a seed that has all the properties options, but you can delete most of it. If you set the properties as global environment variables, you have to remove them from the seed.

To run the demo, provide a comma-separated list of the URLs of the two aforementioned forks (github-webhook and github-analytics') in the `REPOS variable.

The following images shows the steps involved:

   

Figure 10.1. Click the 'jenkins-pipeline-seed-cf' job for Cloud Foundry and jenkins-pipeline-seed-k8s for Kubernetes

seed click

   

Figure 10.2. Click the 'Build with parameters'

seed run

   

Figure 10.3. The REPOS parameter should already contain your forked repos (you’ll have more properties than the ones in the screenshot)

seed

   

Figure 10.4. This is how the results of seed should look like

seed built

10.1.5 Run the github-webhook Pipeline

We already created the seed job for you, but you have to run it. When you do run it, you have to provide some properties. By default, we create a seed that has all the properties options, but you can delete most of it. If you set the properties as global environment variables, you have to remove them from the seed.

To run the demo, provide a comma-separated list of URLs of the two aforementioned forks (github-webhook and github-analytics) in the REPOS variable.

The following images shows the steps involved:

   

Figure 10.5. Click the 'github-webhook' view

seed views

   

Figure 10.6. Run the pipeline

pipeline run

   

[Important]Important

If your build fails on deploy previous version to stage due to a missing jar, that means that you forgot to clear the tags in your repository. Typically, that happens because you removed the Artifactory volume with a deployed jar while a tag in the repository still points there. See here for how to remove the tag.

   

Figure 10.7. Click the manual step to go to stage (remember about killing the apps on test env). To do this click the ARROW next to the job name

pipeline manual

   

[Important]Important

Servers often run run out of resources at the stage step. For that reason, we suggest killing all applications on test. See the FAQ for more detail.

   

Figure 10.8. The full pipeline should look like this

pipeline finished

   

10.2 Declarative Pipeline & Blue Ocean

You can also use the declarative pipeline approach with the Blue Ocean UI.

The Blue Ocean UI is available under the blue/ URL (for example, for Docker Machine-based setup: http://192.168.99.100:8080/blue).

The following images show the various steps involved:

   

Figure 10.9. Open Blue Ocean UI and click on github-webhook-declarative-pipeline

blue 1

   

Figure 10.10. Your first run will look like this. Click Run button

blue 2

   

Figure 10.11. Enter parameters required for the build and click run

blue 3

   

Figure 10.12. A list of pipelines will be shown. Click your first run.

blue 4

   

Figure 10.13. State if you want to go to production or not and click Proceed

blue 5

   

Figure 10.14. The build is in progress…​

blue 6

   

Figure 10.15. The pipeline is done!

blue 7

   

[Important]Important

There is no possibility of restarting a pipeline from a specific stage after failure. See this issue for more information

[Warning]Warning

Currently, there is no way to introduce manual steps in a performant way. Jenkins blocks an executor when a manual step is required. That means that you run out of executors pretty quickly. See this issue and this StackOverflow question for more information.

10.3 Jenkins Cloud Foundry Customization

You can customize Jenkins for Cloud Foundry by setting a variety of environment variables.

[Note]Note

You need not see all the environment variables described in this section to run the demo. They are needed only when you want to make custom changes.

10.3.1 Environment Variable Summary

The environment variables that are used in all of the jobs are as follows:

Property NameProperty DescriptionDefault value

BINARY_EXTENSION

Extension of the binary uploaded to Artifactory / Nexus. Example: war for WAR artifacts

jar

PAAS_TEST_API_URL

The URL to the CF API for the TEST environment

api.local.pcfdev.io

PAAS_STAGE_API_URL

The URL to the CF API for the STAGE environment

api.local.pcfdev.io

PAAS_PROD_API_URL

The URL to the CF API for the PROD environment

api.local.pcfdev.io

PAAS_TEST_ORG

Name of the org for the test env

pcfdev-org

PAAS_TEST_SPACE_PREFIX

Prefix of the name of the CF space for the test environment to which the app name is appended

sc-pipelines-test

PAAS_STAGE_ORG

Name of the org for the stage environment

pcfdev-org

PAAS_STAGE_SPACE

Name of the space for the stage environment

sc-pipelines-stage

PAAS_PROD_ORG

Name of the org for the prod environment

pcfdev-org

PAAS_PROD_SPACE

Name of the space for the prod environment

sc-pipelines-prod

REPO_WITH_BINARIES_FOR_UPLOAD

URL of the repository with the deployed jars

http://artifactory:8081/artifactory/libs-release-local

M2_SETTINGS_REPO_ID

The ID of server from Maven settings.xml

artifactory-local

JDK_VERSION

The name of the JDK installation

jdk8

PIPELINE_VERSION

The version of the pipeline (ultimately, also the version of the jar)

1.0.0.M1-${GROOVY,script ="new Date().format('yyMMdd_HHmmss')"}-VERSION

GIT_EMAIL

The email used by Git to tag the repository

[email protected]

GIT_NAME

The name used by Git to tag the repository

Pivo Tal

PAAS_HOSTNAME_UUID

Additional suffix for the route. In a shared environment, the default routes can be already taken

 

AUTO_DEPLOY_TO_STAGE

Whether deployment to stage be automatic

false

AUTO_DEPLOY_TO_PROD

Whether deployment to prod be automatic

false

API_COMPATIBILITY_STEP_REQUIRED

Whether the API compatibility step is required

true

DB_ROLLBACK_STEP_REQUIRED

Whether the DB rollback step is present

true

DEPLOY_TO_STAGE_STEP_REQUIRED

Whether to the deploy-to-stage step be present

true

BUILD_OPTIONS

Additional options you would like to pass to the Maven / Gradle build

 

10.3.2 Jenkins Credentials

Our scripts reference the credentials by IDs. The following table describes the defaults for the credentials:

Property NameProperty DescriptionDefault value

PAAS_PROD_CREDENTIAL_ID

Credential ID for CF Prod environment access

cf-prod

GIT_CREDENTIAL_ID

Credential ID used to tag a Git repo

git

GIT_SSH_CREDENTIAL_ID

SSH credential ID used to tag a Git repo

gitSsh

GIT_USE_SSH_KEY

If true, pick the SSH credential id to use

false

REPO_WITH_BINARIES_CREDENTIAL_ID

Credential ID used for the repository with jars

repo-with-binaries

PAAS_TEST_CREDENTIAL_ID

Credential ID for CF Test environment access

cf-test

PAAS_STAGE_CREDENTIAL_ID

Credential ID for CF Stage environment access

cf-stage

If you already have in your system a credential to (for example) tag a repository, you can use it by passing the value of the GIT_CREDENTIAL_ID property.

[Tip]Tip

See the cf-helper script for all the configuration options.