CI/CD gitlab pipelines for ESB projects

Series: Data

Facts:

  • the Carbon console is ugly: the less you see it, the better
  • development and testing are iterative processes: you might need to deploy several versions before everything is OK and you do not want to perform these actions manually
  • you need a simple way to determine which version of the project is in production and be sure that your sources are up to date.

Here is the most straightforward way I found to achieve these goals with Gitlab

Requirements

Besides versioning your ESB projects (in this case, with Gitlab), you will need to take care of the following details

Project structure

The following example requires that the different modules of your projects

  • ESB: ESBConfigProject
  • CA: CompositApplicationProject
  • CE: ConnectorExporterProject
  • REG: RegistryRessourceProject

are explicitly linked in a mavenmultimodule pom.xml. Here is an example of such a pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" 
xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.someCompany</groupId>
  <artifactId>someProject</artifactId>
  <version>1.0.0</version>
  <packaging>pom</packaging>
  <name>someProject</name>
  <description>some description</description>
  <modules>
    <module>someProject_ESB</module>
    <module>someProject_REG</module>
    <module>someProject_CE</module>
    <module>someProject_CA</module>
  </modules>
  <build>
    <plugins>
      <plugin>
        <artifactId>maven-eclipse-plugin</artifactId>
        <version>2.9</version>
        <configuration>
          <buildcommands />
          <projectnatures>
            <projectnature>org.wso2.developerstudio.eclipse.mavenmultimodule.project.nature</projectnature>
          </projectnatures>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

Except for the CompositApplicationProject name of which should end with _CA, you are free to use different names from one project to another provided that all your project share the following folder structure:

[someProject]
    pom.xml (the one above)
    .gitlab-ci.yml (see bellow)
    [someProject_CA]
        pom.xml
        .....
    [someProject_ESB]
        pom.xml
        src
            main
            ....
        .....
    [someProject_CE]
        pom.xml
        .....
    [someProject_REG]
        pom.xml
        .....

Local runners for deployment

You’ll also need on premise gitlab runners (shell type) with write rights on you $WSO2_EI_HOME/repository/deployment/server/carbonapps. That’s all you need. installation details

A look at .gitlab-ci.yml

build_car: 
  stage: build
  image: maven:3.6.3-jdk-8
  only: 
    - master
    - dev
  script: 
    - export CI_PROJECT_NAME
    - echo "Starting build ${CI_PROJECT_NAME}"
    - mvn clean install -Dmaven.test.skip=true -Dversion=$CI_COMMIT_SHORT_SHA
  artifacts:
    paths: 
      - ./*_CA/target/*.car
  tags:
    - docker

dev_deploy:
  stage: deploy
  script: 
    - $carbon_app_name = $(Get-ChildItem ./*_CA/target/*.car).name.split('_')[0]
    - $old = Get-ChildItem  $carbon_app_name*.car -Path "$($env:WSO2_EI_HOME)\repository\deployment\server\carbonapps"
    - if ($old) {rm $old.fullname}
    - $new = Get-ChildItem ./*_CA/target/*
    - Copy-Item -Path $new -Destination "$($env:WSO2_EI_HOME)\repository\deployment\server\carbonapps"
  only: 
    - dev
  tags:
    - wso2
    - esb
    - dev

production_deploy:
  stage: deploy
  script: 
    - $carbon_app_name = $(Get-ChildItem ./*_CA/target/*.car).name.split('_')[0]
    - $old = Get-ChildItem  $carbon_app_name*.car -Path "$($env:WSO2_EI_HOME)\repository\deployment\server\carbonapps"
    - if ($old) {rm $old.fullname}
    - $new = Get-ChildItem ./*_CA/target/*
    - Copy-Item -Path $new -Destination "$($env:WSO2_EI_HOME)\repository\deployment\server\carbonapps"
  only: 
    - master
  tags:
    - wso2
    - esb
    - prod

The following pipeline has two steps.

Build

The build_car job will be realized by shared runners for every new commit on either the dev or master branches. Here, we are using maven:3.6.3-jdk-8 official docker image. Here you probably noticed that we are using -Dversion=$CI_COMMIT_SHORT_SHA during mvn clean install. That’s because we are using variables in pom.xml’s versions, like so:

<project 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" 
xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <modelVersion>4.0.0</modelVersion>
  <parent>
    <groupId>com.someCompany</groupId>
    <artifactId>someProject</artifactId>
    <version>1.0.0</version>
  </parent>
  <groupId>com.someCompany.someProject_CA</groupId>
  <artifactId>someProject_CA</artifactId>
  <version>${version}</version>
  <packaging>carbon/application</packaging>
  <name>someProject_CA</name>
  <description>someProject_CA</description>

(you’ll understand later why). Once build, the .car is stored in the project’s artifacts

Deploy

Depending on whether trigged by the dev or prod branch, the success of the build job will trigger the deploy job for the relevant gitlab-runner. In this powershell based example, the runner

  • identifies the project name based on the .car’s name (e.g. someProject_CA_c5ad90bf.car -> someProject)
  • looks for a previous version in $WSO2_EI_HOME/repository/deployment/server/carbonapps
  • removes it
  • and copy the new one

This is a post in the Data series.
Other posts in this series: