CI/CD Integration

Integrate GoMask CLI into your CI/CD pipelines

CI/CD Integration Guide

Automate synthetic data generation and data masking in your CI/CD pipelines.

Overview

The GoMask CLI is designed for automation:

Non-interactive commands
Environment variable configuration
Exit codes for error handling
JSON output for parsing
Timeout controls

GitHub Actions

Basic Workflow

name: Generate Test Data

on:
  schedule:
    - cron: '0 0 * * *'  # Daily at midnight
  workflow_dispatch:      # Manual trigger

jobs:
  generate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install GoMask CLI
        run: pip install gomask-cli

      - name: Initialize GoMask
        run: gomask init --secret ${{ secrets.GOMASK_SECRET }}

      - name: Validate Routine
        run: gomask validate routines/test-data.yaml

      - name: Generate Data
        run: gomask run routines/test-data.yaml --watch

With Environment Variables

name: Generate Environment-Specific Data

on:
  workflow_dispatch:
    inputs:
      environment:
        description: 'Target environment'
        required: true
        default: 'staging'
        type: choice
        options:
          - staging
          - development
      record_count:
        description: 'Number of records'
        required: true
        default: '1000'

jobs:
  generate:
    runs-on: ubuntu-latest
    environment: ${{ github.event.inputs.environment }}
    steps:
      - uses: actions/checkout@v4

      - name: Install GoMask CLI
        run: pip install gomask-cli

      - name: Initialize
        run: gomask init --secret ${{ secrets.GOMASK_SECRET }}

      - name: Generate Data
        run: |
          gomask run routines/data.yaml \
            --param record_count=${{ github.event.inputs.record_count }} \
            --env-file .env.${{ github.event.inputs.environment }} \
            --watch

Matrix Strategy for Multiple Routines

name: Generate All Test Data

on:
  schedule:
    - cron: '0 2 * * *'

jobs:
  generate:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        routine:
          - customers
          - orders
          - products
      fail-fast: false
    steps:
      - uses: actions/checkout@v4

      - name: Install GoMask CLI
        run: pip install gomask-cli

      - name: Initialize
        run: gomask init --secret ${{ secrets.GOMASK_SECRET }}

      - name: Generate ${{ matrix.routine }}
        run: gomask run routines/${{ matrix.routine }}.yaml --watch

GitLab CI

Basic Pipeline

stages:
  - validate
  - generate

variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"

cache:
  paths:
    - .cache/pip

validate:
  stage: validate
  image: python:3.11-slim
  script:
    - pip install gomask-cli
    - gomask validate routines/test-data.yaml --detailed

generate:
  stage: generate
  image: python:3.11-slim
  script:
    - pip install gomask-cli
    - gomask init --secret $GOMASK_SECRET
    - gomask run routines/test-data.yaml --watch
  only:
    - main
    - schedules
  variables:
    GOMASK_SECRET: $GOMASK_SECRET

Environment-Specific

.generate_template: &generate_template
  image: python:3.11-slim
  script:
    - pip install gomask-cli
    - gomask init --secret $GOMASK_SECRET
    - gomask run routines/data.yaml --env-file .env.$CI_ENVIRONMENT_NAME --watch

generate:staging:
  <<: *generate_template
  stage: generate
  environment:
    name: staging
  only:
    - develop

generate:production:
  <<: *generate_template
  stage: generate
  environment:
    name: production
  only:
    - main
  when: manual

Jenkins

Declarative Pipeline

pipeline {
    agent any

    environment {
        GOMASK_SECRET = credentials('gomask-api-secret')
    }

    stages {
        stage('Setup') {
            steps {
                sh 'pip install gomask-cli'
                sh 'gomask init --secret $GOMASK_SECRET'
            }
        }

        stage('Validate') {
            steps {
                sh 'gomask validate routines/test-data.yaml --detailed'
            }
        }

        stage('Generate') {
            steps {
                sh 'gomask run routines/test-data.yaml --watch --timeout 3600'
            }
        }
    }

    post {
        failure {
            echo 'Data generation failed!'
        }
        success {
            echo 'Data generation completed successfully!'
        }
    }
}

Parameterized Build

pipeline {
    agent any

    parameters {
        choice(
            name: 'ENVIRONMENT',
            choices: ['staging', 'development'],
            description: 'Target environment'
        )
        string(
            name: 'RECORD_COUNT',
            defaultValue: '1000',
            description: 'Number of records to generate'
        )
    }

    stages {
        stage('Generate') {
            steps {
                sh """
                    pip install gomask-cli
                    gomask init --secret \$GOMASK_SECRET
                    gomask run routines/data.yaml \
                        --param record_count=${params.RECORD_COUNT} \
                        --env-file .env.${params.ENVIRONMENT} \
                        --watch
                """
            }
        }
    }
}

Azure DevOps

trigger:
  - main

pool:
  vmImage: 'ubuntu-latest'

variables:
  - group: gomask-secrets

stages:
  - stage: Generate
    jobs:
      - job: GenerateTestData
        steps:
          - task: UsePythonVersion@0
            inputs:
              versionSpec: '3.11'

          - script: pip install gomask-cli
            displayName: 'Install GoMask CLI'

          - script: gomask init --secret $(GOMASK_SECRET)
            displayName: 'Initialize GoMask'

          - script: gomask validate routines/test-data.yaml
            displayName: 'Validate Routine'

          - script: gomask run routines/test-data.yaml --watch
            displayName: 'Generate Data'

CircleCI

version: 2.1

jobs:
  generate:
    docker:
      - image: python:3.11-slim
    steps:
      - checkout
      - run:
          name: Install GoMask CLI
          command: pip install gomask-cli
      - run:
          name: Initialize
          command: gomask init --secret $GOMASK_SECRET
      - run:
          name: Generate Data
          command: gomask run routines/test-data.yaml --watch
          no_output_timeout: 30m

workflows:
  nightly:
    triggers:
      - schedule:
          cron: "0 0 * * *"
          filters:
            branches:
              only: main
    jobs:
      - generate

Best Practices

1. Store Secrets Securely

Never commit secrets to version control:

# GitHub Actions
run: gomask init --secret ${{ secrets.GOMASK_SECRET }}

# GitLab CI
variables:
  GOMASK_SECRET: $GOMASK_SECRET  # From CI/CD settings

2. Validate Before Running

Always validate YAML before execution:

- name: Validate
  run: gomask validate routine.yaml --detailed

- name: Generate
  run: gomask run routine.yaml --watch

3. Use Timeouts

Set appropriate timeouts for large datasets:

gomask run routine.yaml --timeout 7200  # 2 hours

4. Handle Failures

Use exit codes for error handling:

Exit Code	Meaning	Action
0	Success	Continue
1	Error	Fail build
130	Interrupted	Retry

- name: Generate Data
  run: gomask run routine.yaml --watch
  continue-on-error: false

5. Use Dry Runs for Testing

Test pipeline changes without executing:

gomask run routine.yaml --dry-run

6. Version Your Routines

Store YAML files in git alongside code:

project/
├── src/
├── tests/
└── routines/
    ├── test-data.yaml
    ├── staging-data.yaml
    └── .env.staging

7. Environment-Specific Configuration

Use .env files for environment differences:

# Load environment-specific config
gomask run routine.yaml --env-file .env.$ENVIRONMENT

8. Parameterize Routines

Make routines flexible:

runtime_parameters:
  record_count:
    type: integer
    default: 1000

gomask run routine.yaml --param record_count=$RECORD_COUNT

Monitoring Executions

Check Status

# List recent executions
gomask executions list --limit 10

# Get specific execution
gomask executions show 123

Follow Logs

# Stream logs in real-time
gomask executions logs 123 --follow

Cancel if Needed

gomask executions cancel 123

Retry Logic

For network issues, implement retry logic:

#!/bin/bash
MAX_RETRIES=3
RETRY_DELAY=30

for i in $(seq 1 $MAX_RETRIES); do
  gomask run routine.yaml --watch && break
  echo "Attempt $i failed. Retrying in $RETRY_DELAY seconds..."
  sleep $RETRY_DELAY
done

Docker

Run in a container:

FROM python:3.11-slim

RUN pip install gomask-cli

COPY routines/ /app/routines/
WORKDIR /app

ENTRYPOINT ["gomask"]
CMD ["--help"]

docker build -t gomask-runner .
docker run -e GOMASK_SECRET=$SECRET gomask-runner run routines/data.yaml

Next Steps

Environment Variables - Variable substitution
Synthetic Data Guide - Configuration options
Command Reference - All CLI options