Python and CI/CD adventure

Overview

Talking points

Traditionally a Java-based shop
The goal is to create a CI/CD Pipeline which supports Python, Docker Images, and the OpenShift target platform.
Sub-goal: Adhere to enterprise security requirements for container and dependency scanning.
Consistency and similarity to existing pipelines is desirable
Greater Security comes at the cost of developer speed and the “Continuous” part of Continuous Delivery.

Infrastructure

Category	Tool	Purpose
Build	Jenkins	Build and Deploy scripts for applications
Security	VeraCode	Security Scanning
Repository	Nexus	Artifact Repository (wheels and docker images)
Security	Nexus IQ	Security Scanning of dependencies
Runtime Environment	OpenShift	Container Orchestration and Runtime Environment

Python

Pytest

PIPs

pytest
pytest-mock
pytest-pythonpath
pytest-env

PIPs go in the requirements.txt file

pytest.ini

Explicit markers were defined to allow selective running of different types of tests.

unit
integration -m unit

# content of pytest.ini
[pytest]
#...
markers = 
  unit: unit tests - explicit rather than implicitly defined
  integration: integration tests reach out to external dependencies
python_paths=src
env=
  LOGGING_LEVEL=DEBUG

Example unit test

@pytest.mark.unit
def test_dynamic_table_name(caplet):
  caplog.set_level(logging.debug)
  logger = logging.getLogger()
  issue_type = 'Story'
  table_name = issue_type_to_table_name_dynamic(issue_type)
  logger.debug(f'{issue_type=}, {table_name=}')
  assert table_name == 'stories'

The CI/CD Pipeline will execute Pytest unit tests with the following command

python -m pytest

Python packaging

Files

setup.cfg
setup.py

setup.py

Setup.py is required for two reasons. First, it triggers setup tools to use the setup.cfg configuration. Secondly, use of setup.py allows the requirements.txt file contents to used by the packager.

If the team prefers, they do still have the option of

explicitly maintaining packages in requirements.txt and also setup.cfg (install_requires). This violates DRY.
Specify packages explicitly in setup.cfg and set requirements.txt to “-e”. That would exclude making their own docker build for local runs. This is my preference, even though it is a little wonky.
Specify packages explicitly in setup.cfg and don’t define requirements.txt. That’s fine, but it assumes we are always making a package of the local service before running it. It makes it more difficult just to build / test / run the service locally, without Docker.

from setup tools import setup
import os 

with open('requirements.txt') as f:
  required = f.read().splitlines()
setup(
  installed_requires=required
)

setup.cfg

[metadata]
name = servicename
version = 1.0.0
description = Service Description
# ... more attributes

[options]
package_dir = 
  = src
packages = find:
python_requires = >= 3.8
include_package_data = True

scripts = 
  app.py
# app.py is used as the Standardized Python entry point packages/services built in this pipeline.

[options.packages.find]
where = src

app.py

import bogusco.servicename
if __name__ == '__main__':
  bogusco.packagename.main('/etc/servicename/secrets')

Artifacts

Python
- tar gz
- wheel
Docker
- Image

Shared Docker file

FROM rhel8/python-39
ENV PIP_CONFIG_FILE=/etc/pip.conf
# COPY certificates here
COPY example.pem /etc/pki/ca-trust/source/anchors/example.crt
# ...
USER root
RUN update-ca-trust extract
ENV REQUESTS_CA_BUNDLE=...

# COPY secrets into image
# COPY pip.conf into image

# Download PIP for our package/service from nexus
# PIP install it

CMD ["python", "/opt/app-root/bin/app.py"]