2. How the Scripts Work

This section describes how the scripts and jobs correspond to each other. If you need to see detailed documentation of the bash scripts, go to the code repository and read common/src/main/bash/README.adoc.

2.1 Build and Deployment

The following text image (created via textart.io) shows a high-level overview:

+---------+                      +-----------+                      +-----------+ +-------+ +---------------+
| script  |                      | language  |                      | framework | | paas  | | customization |
+---------+                      +-----------+                      +-----------+ +-------+ +---------------+
     |                                 |                                  |           |             |
     | What is your language?          |                                  |           |             |
     |-------------------------------->|                                  |           |             |
     |                                 |                                  |           |             |
     |       I'm written in X language |                                  |           |             |
     |<--------------------------------|                                  |           |             |
     |                                 |                                  |           |             |
     |                                 | What framework do you use?       |           |             |
     |                                 |--------------------------------->|           |             |
     |                                 |                                  |           |             |
     |                                 |                I use Y framework |           |             |
     |<-------------------------------------------------------------------|           |             |
     |                                 |                                  |           |             |
     | I know that you use Z PAAS?     |                                  |           |             |
     |------------------------------------------------------------------------------->|             |
     |                                 |                                  |           |             |
     |                                 |  Here are all Z-related deployment functions |             |
     |<-------------------------------------------------------------------------------|             |
     |                                 |                                  |           |             |
     | Anything custom to override in bash?                               |           |             |
     |--------------------------------------------------------------------------------------------->|
     |                                 |                                  |           |             |
     |                                 |                                  |        Not this time... |
     |<---------------------------------------------------------------------------------------------|
     |                                 |                                  |           |             |
     | Ok, run the script              |                                  |           |             |
     |-------------------              |                                  |           |             |
     |                  |              |                                  |           |             |
     |<------------------              |                                  |           |             |
     |                                 |                                  |           |             |

Before we run the script, we need to answer a few questions related to your repository:

  • What is your language (for example, jvm,php, or something else)?
  • what framework do you use (for example, maven or gradle)?
  • what PAAS do you use (for example, cf or k8s)?

The following sequence diagram (created via textart.io) describes how the sourcing of bash scripts takes place:

+---------+                                         +-----------+                                            +-------------+                   +-----------+            +-----------+                                   +-------+                            +---------+
| script  |                                         | pipeline  |                                            | projectType |                   | language  |            | framework |                                   | paas  |                            | custom  |
+---------+                                         +-----------+                                            +-------------+                   +-----------+            +-----------+                                   +-------+                            +---------+
     |                                                    |                                                         |                                |                        |                                             |                                     |
     | [source pipeline.sh]                               |                                                         |                                |                        |                                             |                                     |
     |--------------------------------------------------->|                                                         |                                |                        |                                             |                                     |
     |                                                    | ------------------------------\                         |                                |                        |                                             |                                     |
     |                                                    |-| loading functions, env vars |                         |                                |                        |                                             |                                     |
     |                                                    | |-----------------------------|                         |                                |                        |                                             |                                     |
     |         -----------------------------------------\ |                                                         |                                |                        |                                             |                                     |
     |         | hopefully all functions get overridden |-|                                                         |                                |                        |                                             |                                     |
     |         | otherwise nothing will work            | |                                                         |                                |                        |                                             |                                     |
     |         |----------------------------------------| |                                                         |                                |                        |                                             |                                     |
     |                                                    | Source the [projectType/pipeline-projectType.sh]        |                                |                        |                                             |                                     |
     |                                                    |-------------------------------------------------------->|                                |                        |                                             |                                     |
     |                                                    |                        -------------------------------\ |                                |                        |                                             |                                     |
     |                                                    |                        | What do we have here...?     |-|                                |                        |                                             |                                     |
     |                                                    |                        | A [mvnw] file,               | |                                |                        |                                             |                                     |
     |                                                    |                        | it has to be a [jvm] project | |                                |                        |                                             |                                     |
     |                                                    |                        |------------------------------| | Source [pipeline-jvm.sh]       |                        |                                             |                                     |
     |                                                    |                                                         |------------------------------->|                        |                                             |                                     |
     |                                                    |                                                         |                                |                        |                                             |                                     |
     |                                                    |                                                         |                                | Maven or Gradle?       |                                             |                                     |
     |                                                    |                                                         |                                |----------------------->|                                             |                                     |
     |                                                    |                                                         |                                |                        | ----------------------------------------\   |                                     |
     |                                                    |                                                         |                                |                        |-| There's a [mvnw] file?                |   |                                     |
     |                                                    |                                                         |                                |                        | | So the [PROJECT_TYPE] must be [maven] |   |                                     |
     |                                                    |                                                         |                                |                        | |---------------------------------------|   |                                     |
     |                                                    |                                                         |                                |   It's a Maven project |                                             |                                     |
     |                                                    |<------------------------------------------------------------------------------------------------------------------|                                             |                                     |
     |                                                    |                                                         |                                |                        |                                             |                                     |
     |                                                    | The [PAAS_TYPE] is [cf] so I'll source [pipeline-cf.sh] |                                |                        |                                             |                                     |
     |                                                    |---------------------------------------------------------------------------------------------------------------------------------------------------------------->|                                     |
     |                                                    |                                                         |                                |                        |                                             | -------------------------------\    |
     |                                                    |                                                         |                                |                        |                                             |-| Loading all                  |    |
     |                                                    |                                                         |                                |                        |                                             | | deployment-related functions |    |
     |                   -------------------------------\ |                                                         |                                |                        |                                             | |------------------------------|    |
     |                   | Ok, we know that it's Maven  |-|                                                         |                                |                        |                                             |                                     |
     |                   | and should be deployed to CF | |                                                         |                                |                        |                                             |                                     |
     |                   |------------------------------| |                                                         |                                |                        |                                             |                                     |
     |                                                    | Try to source [custom/build_and_upload.sh]              |                                |                        |                                             |                                     |
     |                                                    |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------>|
     |                                                    |                                                         |                                |                        |                                             |                                     | ----------------------------\
     |                                                    |                                                         |                                |                        |                                             |                                     |-| No such file so           |
     |                                                    |                                                         |                                |                        |                                             |                                     | | nothing custom to be done |
     | ---------------------------------------------\     |                                                         |                                |                        |                                             |                                     | |---------------------------|
     |-| All build related functions                |     |                                                         |                                |                        |                                             |                                     |
     | | overridden by language / framework scripts |     |                                                         |                                |                        |                                             |                                     |
     | -------------------------------\-------------|     |                                                         |                                |                        |                                             |                                     |
     |-| All deploy related functions |                   |                                                         |                                |                        |                                             |                                     |
     | | overridden by paas scripts   |                   |                                                         |                                |                        |                                             |                                     |
     | |------------------------------|                   |                                                         |                                |                        |                                             |                                     |
     | run [build] function                               |                                                         |                                |                        |                                             |                                     |
     |---------------------                               |                                                         |                                |                        |                                             |                                     |
     |                    |                               |                                                         |                                |                        |                                             |                                     |
     |<--------------------                               |                                                         |                                |                        |                                             |                                     |
     |                                                    |                                                         |                                |                        |                                             |                                     |

The process works as follows:

  1. A script (for example, build_and_upload.sh) is called.
  2. It sources the pipeline.sh script that contains all the essential function “interfaces” and environment variables.
  3. pipeline.sh needs information about the project type. It sources projectType/pipeline-projectType.sh.
  4. projectType/pipeline-projectType.sh contains logic to determine the language.

    1. Verify whether a repository contains files that correspond to the given languages (for example, mvnw or composer.json).
    2. Verify whether a concrete framework that we support (for example, maven or gradle) is present.
  5. Once we know what the project type is, we can deal with PAAS. Depending on the value of the PAAS_TYPE environment variable, we can source proper PAAS functions (for example, pipeline-cf.sh for Cloud Foundry).
  6. Determine whether we can do some further customization.

    1. Search for a file called ${sc-pipelines-root}/common/src/main/bash/custom/build_and_upload.sh to override any functions you want.
  7. Run the build function from build_and_upload.sh

2.2 Project Crawler

In Jenkins, you can generate the deployment pipelines by passing an environment variable with a comma-separated list of repositories. This, however, does not scale. We would like to automatically fetch a list of all repositories from a given organization and team.

To do so, we use the Project Crawler library, which can:

  • Fetch all projects for a given organization.
  • Fetch contents of a file for a given repository.

The following diagram depicts this situation:

+---------+                                                  +-------+                                                                           +-------------+ +---------+
| Jenkins |                                                  | Seed  |                                                                           | SCPipelines | | Github  |
+---------+                                                  +-------+                                                                           +-------------+ +---------+
     |                                                           |                                                                                      |             |
     | Copy the seed job from the repo                           |                                                                                      |             |
     |------------------------------------------------------------------------------------------------------------------------------------------------->|             |
     |                                                           |                                                                                      |             |
     | Run seed job to generate Spinnaker pipelines and jobs     |                                                                                      |             |
     |---------------------------------------------------------->|                                                                                      |             |
     |                                                           |                                                                                      |             |
     |                                                           | Crawl org [foo] and fetch all repositories                                           |             |
     |                                                           |--------------------------------------------------------------------------------------------------->|
     |                                                           |                                                                                      |             |
     |                                                           |                                                                   In org [foo] there [a,b,c] repos |
     |                                                           |<---------------------------------------------------------------------------------------------------|
     |                                                           |                                                                                      |             |
     |                                                           | For each repo fetch pipeline descriptor                                              |             |
     |                                                           |--------------------------------------------------------------------------------------------------->|
     |                                                           |                                                                                      |             |
     |                                                           |                      There you go. [a] wants no [test] env, [b] no [stage] env, [c] wants all envs |
     |                                                           |<---------------------------------------------------------------------------------------------------|
     |                                                           |                                                                                      |             |
     |                                                           | Build pipelines. For [a] without [test], for [b] without [stage]. All for [c]        |             |
     |                                                           |------------------------------------------------------------------------------        |             |
     |                                                           |                                                                             |        |             |
     |                                                           |<-----------------------------------------------------------------------------        |             |
     |                             ----------------------------\ |                                                                                      |             |
     |                             | By having descriptors,    |-|                                                                                      |             |
     |                             | we can tune the pipelines | |                                                                                      |             |
     |                             | as the app wanted it to.  | |                                                                                      |             |
     |                             |---------------------------| | Build jobs / pipelines for [a,b,c] repos                                             |             |
     |                                                           |-----------------------------------------                                             |             |
     |                                                           |                                        |                                             |             |
     |                                                           |<----------------------------------------                                             |             |
     |                                                           |                                                                                      |             |

Thanks to the Project Crawler, you can run the seed job, and ,automatically, all the new repositories are picked and pipelines are created for them. Project Crawler supports repositories stored at Github, Gitlab, and Bitbucket. You can also register your own implementation. See the Project Crawler repository for more information.

2.3 How Scripts Work with Spinnaker

With Spinnaker, the deployment pipeline is inside of Spinnaker. No longer do we treat Jenkins or Concourse as a tool that does deployments. In Jenkins, we create only the CI jobs (that is, build and test) and prepare the JSON definitions of Spinnaker pipelines.

The following diagram shows how Jenkins, the seed job for Spinnaker, and Spinnaker cooperate:

+---------+                                                  +-------+                                                                           +-------------+                          +---------+ +-----------+
| Jenkins |                                                  | Seed  |                                                                           | SCPipelines |                          | Github  | | Spinnaker |
+---------+                                                  +-------+                                                                           +-------------+                          +---------+ +-----------+
     |                                                           |                                                                                      |                                      |            |
     | Copy the seed job from the repo                           |                                                                                      |                                      |            |
     |------------------------------------------------------------------------------------------------------------------------------------------------->|                                      |            |
     |                                                           |                                                                                      |                                      |            |
     | Run seed job to generate Spinnaker pipelines and jobs     |                                                                                      |                                      |            |
     |---------------------------------------------------------->|                                                                                      |                                      |            |
     |                                                           |                                                                                      |                                      |            |
     |                                                           | Crawl org [foo] and fetch all repositories                                           |                                      |            |
     |                                                           |---------------------------------------------------------------------------------------------------------------------------->|            |
     |                                                           |                                                                                      |                                      |            |
     |                                                           |                                                                                      |     In org [foo] there [a,b,c] repos |            |
     |                                                           |<----------------------------------------------------------------------------------------------------------------------------|            |
     |                                                           |                                                                                      |                                      |            |
     |                                                           | For each repo fetch pipeline descriptor                                              |                                      |            |
     |                                                           |---------------------------------------------------------------------------------------------------------------------------->|            |
     |                                                           |                                                                                      |                                      |            |
     |                                                           |                                                            There you go. [a] wants no [test], [b] no [stage], [c] wants all |            |
     |                                                           |<----------------------------------------------------------------------------------------------------------------------------|            |
     |                                                           |                                                                                      |                                      |            |
     |                                                           | Build pipelines. For [a] without [test], for [b] without [stage]. All for [c]        |                                      |            |
     |                                                           |------------------------------------------------------------------------------        |                                      |            |
     |                                                           |                                                                             |        |                                      |            |
     |                                                           |<-----------------------------------------------------------------------------        |                                      |            |
     |                             ----------------------------\ |                                                                                      |                                      |            |
     |                             | By having descriptors,    |-|                                                                                      |                                      |            |
     |                             | we can tune the pipelines | |                                                                                      |                                      |            |
     |                             | as the app wanted it to.  | |                                                                                      |                                      |            |
     |                             |---------------------------| | Build CI jobs for [a,b,c] repos                                                      |                                      |            |
     |                                                           |--------------------------------                                                      |                                      |            |
     |                                                           |                               |                                                      |                                      |            |
     |                                                           |<-------------------------------                                                      |                                      |            |
     |                                                           |                                                                                      |                                      |            |
     |                                                           | Build Spinnaker pipelines JSON definitions                                           |                                      |            |
     |                                                           |-------------------------------------------                                           |                                      |            |
     |                                                           |                                          |                                           |                                      |            |
     |                                                           |<------------------------------------------                                           |                                      |            |
     |                                                           |                                                                                      |                                      |            |
     |                                             Seed job done |                                                                                      |                                      |            |
     |<----------------------------------------------------------|                                                                                      |                                      |            |
     |                                                           |                                                                                      |                                      |            |
     | Upload JSON pipelines to Spinnaker                        |                                                                                      |                                      |            |
     |----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->|
     |                                                           |                                                                                      |                                      |            |
     |                                                           |                                                                                      |                                      |            | The pipelines for [a,b,c] successfully created
     |                                                           |                                                                                      |                                      |            |-----------------------------------------------
     |                                                           |                                                                                      |                                      |            |                                              |
     |                                                           |                                                                                      |                                      |            |<----------------------------------------------
     |                                                           |                                                                                      |                                      |            |
     |                                                           |                                                                                Waiting for [spinnaker-a-build] build to start & complete |
     |<-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
     |                                                           |                                                                                      |                                      |            |
     | New commit! Running a build [spinnaker-a-build]           |                                                                                      |                                      |            |
     |------------------------------------------------           |                                                                                      |                                      |            |
     |                                               |           |                                                                                      |                                      |            |
     |<-----------------------------------------------           |                                                                                      |                                      |            |
     |                                                           |                                                                                      |                                      |            |
     | Run the [build_and_upload.sh] script                      |                                                                                      |                                      |            |
     |------------------------------------------------------------------------------------------------------------------------------------------------->|                                      |            |
     |                                                           |                                                                                      | --------------------------------\    |            |
     |                                                           |                                                                                      |-| Proceed with all the sourcing |    |            |
     |                                                           |                                                                                      | | depending on language etc.    |    |            |
     |                                                           |                                                                                      | |-------------------------------|    |            |
     |                                                           |                                                                     Build completed! |                                      |            |
     |<-------------------------------------------------------------------------------------------------------------------------------------------------|                                      |            |
     |                                                           |                                                                                      |                                      |            |
     | [spinnaker-a-build] started and completed                 |                                                                                      |                                      |            |
     |----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->|
     |                                                           |                                                                                      |                                      |            | ------------------------------------\
     |                                                           |                                                                                      |                                      |            |-| Running the rest of the pipeline! |
     |                                                           |                                                                                      |                                      |            | |-----------------------------------|
     |                                                           |                                                                                      |                                      |            |
     |                                                           |                                                                                      |                                      |            | Pipeline for [a] in progress. Deploy [a] to test env
     |                                                           |                                                                                      |                                      |            |-----------------------------------------------------
     |                                                           |                                                                                      |                                      |            |                                                    |
     |                                                           |                                                                                      |                                      |            |<----------------------------------------------------
     |                                                           |                                                                                      |                                      |            |
     |                                                           |                                                                                   Calling [spinnaker-a-test-on-test] to run test on test |
     |<-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
     |                                                           |                                                                                      |                                      |            |
     | [spinnaker-a-test-on-test] started and completed          |                                                                                      |                                      |            |
     |----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->|
     |                                                           |                                                                                      |                                      |            |
     |                                                           |                                                                                      |                                      |            | ... we continue like this throughout the pipeline ...
     |                                                           |                                                                                      |                                      |            |------------------------------------------------------
     |                                                           |                                                                                      |                                      |            |                                                     |
     |                                                           |                                                                                      |                                      |            |<-----------------------------------------------------
     |                                                           |                                                                                      |                                      |            |
     |                                                           |                                                                                      |                                      |            | ... and the pipeline is done
     |                                                           |                                                                                      |                                      |            |-----------------------------
     |                                                           |                                                                                      |                                      |            |                            |
     |                                                           |                                                                                      |                                      |            |<----------------------------
     |                                                           |                                                                                      |                                      |            |

2.4 Deployment & languages compatibility matrix

In the following table we present which language is supported by which deployment mechanism.

Table 2.1. Deployment & languages compatibility matrix

LanguageCFK8SAnsible

JVM with Gradle

JVM with Maven

PHP with Composer

NodeJS with NPM

Dotnet core


[Tip]Tip

For K8S, a deployment unit is a docker image so any language and framework can be used.