Introduction to Bitbucket pipelines

Those are pipes.

After Bitbucket announced their pipelines — I was little skeptical. You know — after circle ci — is there any other CI/CD environment that can compete? After some testing it appears — that it can. Basically I was able to set up fully working CI/CD flow for my python/django project.

In this post I will try to introduce how to setup basic flow for the Bitbucket pipelines. Because of the obvious reasons — I will write a setup for backend application written in django — it is my main field of expertise.

Maybe start from the beginning and definitions.

What is CI/CD?

On the wikipedia we can read (whole article):

continuous integration (CI) is the practice of merging all developer working copies to a shared mainline several times a day. Grady Booch first named and proposed CI in his 1991 method, although he did not advocate integrating several times a day. Extreme programming (XP) adopted the concept of CI and did advocate integrating more than once per day — perhaps as many as tens of times per day.

and few lines below:

In XP, CI was intended to be used in combination with automated unit tests written through the practices of test-driven development. Initially this was conceived of as running all unit tests in the developer’s local environment and verifying they all passed before committing to the mainline. This helps avoid one developer’s work-in-progress breaking another developer’s copy. If necessary, partially complete features can be disabled before commit, such as by using feature toggles.

What is CD?

Again wikipedia (whole article)

Continuous delivery (CD) is a software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time.[1] It aims at building, testing, and releasing software faster and more frequently. The approach helps reduce the cost, time, and risk of delivering changes by allowing for more incremental updates to applications in production. A straightforward and repeatable deployment process is important for continuous delivery.

It seems that integrating your project with some CI/CD tools can improve performance a lot :) So lets try.

Disclaimer: In this post I will show how to setup basic CI flow. Maybe in the next one I will show you how to set up the last step — releasing an application to production servers.

What is bitbucket pipeline?

Bitbucket pipelines allows you to run images in docker; Your main project will use one docker image — and you are allowed to set up three more. It is really nice for the backend tests — you can set up, eg.: postgresql image, or elasticsearch image, or some rabbitmq if needed; This allows you to test your application in the environment almost similar (or even exactly the same) condition than your production application.

From the official Bitbucket documentation:

Bitbucket Pipelines brings continuous integration and delivery to Bitbucket Cloud, empowering teams to build, test, and deploy their code within Bitbucket.

In different words: Bitbucket pipelines allows you to define a list of steps which will build your project. The steps can be very different:

• from linters
• through tests
• till the deploy

You can specify special build steps for eg. develop branch;

What is currently supported?

From the official Bitbucket pipelines documentation:

• java (and android applications)
• python
• node.js
• php
• ruby
• others (whatever this mean — taken from official docs)

Sadly iOS at the moment is not supported; You can try to use some magic, and open source swift images — but I do not foretell a success here.

Set-up required steps

1. Turn on the pipelines for the given repository;
   (Repository Settings -> PIPELINES -> Settings -> Enable Pipelines)
2. Add bitbucket-pipelines.yml file to your project in the root directory.

app_root_dir
    bitbucket-pipelines.yml
    ...

Push the bitbucket-pipelines.yml to the repository (start with master branch);

Sample bitbucket-pipelines.yml

So lets write very simple bitbucket-pipelines.yml:

image: python:3.6 
pipelines: default: 
  - step: 
      script: 
        - pip install -r requirements.txt 
        - flake8 .

In this example — the default build (default means on every commit on every branch — if not specified otherwise); will install the requirements stored in the requirements.txt - it is python example, but it is totally valid to use npm, gradle, yarn there; Sky is the limit :) The second step (defined under the step -> script section) just checks the pep8 - because it is a python code, as mentioned above.

Pipelines services

... 
- step: 
   ... 
   services: postgres 
definitions: 
  services: 
    postgres: 
      image: postgres

In each pipeline you can setup to 4 services; (actually 3 — because one of the container is the main one - the one which run you pipelines); To do that - you need a definitions block in your pipelines file at the highest level - then you specify services. In the example above I added the postgres service. To use in in your build step - just add services section under your step. You need to know what is the service host and port - and in database engine case - also database user and password. Usually you can find it in the Bitbucket pipelines documentation. It is worth to know this services are docker images and you can simply change the default credentials (using environments variables - you can find an example in section below where I present the fully working example) to ones that suits you more.

Pipelines branch steps

branches: 
  master: 
    - step: 
      ... 
  develop: 
    - step: 
      ...

Usually the base setup is to define default pipelines. The default pipeline will be run on every commit on every branch (if bitbucket-pipelines.yml file is present in application root directory). It is true unless some more specific pipelines is defined. Let say you have defined default and branches on master branch. When you push to the master the default will be skipped and pipelines will be built for master branch - where you have specific steps defined.

Usually on merge to master branch you should run integrations tests, do the deploy and run post deploy test. Bitbucket pipelines allows you to do that.

Fully working bitbucket-pipelines.yml

Here is an example, and I will describe each line below:

image: python:3.6 
pipelines: 
  default: 
    - step: 
      caches: 
        - pip 
      script: 
        - pip install -r requirements.txt 
        - pip install -r requirements-ci.txt 
        - flake8 . - isort -rc --check-only . 
        - python manage.py test --settings=settings.tests 
      services: 
        - elasticsearch 
        - rabbitmq 
        - postgres 
definitions: 
  services: 
    elasticsearch: 
      image: elasticsearch 
      environment: 
        ES_JAVA_OPTS: -Xms512m -Xmx512m 
    rabbitmq: 
      image: rabbitmq 
      environment: 
        RABBITMQ_ERLANG_COOKIE: "FDASFASAFDASFWEFGSDF"   
        RABBITMQ_DEFAULT_USER: "rabbitmq" 
        RABBITMQ_DEFAULT_PASS: "rabbitmq" 
        RABBITMQ_DEFAULT_VHOST: "/" 
    postgres: 
      image: postgres

I will describe above pipeline file line by line.

In the first line I define the docker image — image: python:3.6 this mean that I want to use image with python 3.6 support. In the next line - I define pipelines and default behavior. My pipeline has only one step (but you can add up to 10); This step cache the pip installations (python package manager) - keyword: caches. It is very nice feature which allows you to cache all the packages installed via pip in python. Later I define that I want to execute some commands in the terminal script - first I install the needed requirements; then I install the requirements needed for CI (usually in my case flake8 and isort). Later the flake8 . command is executed which check if my python code is compliant with the pep8. After that isort with --check-only attribute is run. Isort check if all imports are sorted properly (usually set up rules per project). And after all of the static code checks - the tests are run. It is a standard django command for running test - usually I override the settings module - to be able to connect with all of the services in the test environment.

After the script section there is a services section - there I am telling the pipelines to use elasticsearch, rabbitmq and postgres in my default pipeline. With those lines I will be able to connect to each one of those services from my main container.

Later on in the file there are a services definitions — which are pretty straightforward. The environment section is worth to note - as this allows you to change the default service set up. In the example above - I've changed the default user and password for the rabbitmq service.

Thanks

Thank you for reading!

Originally published at showpy.tech on October 4, 2017.