Synthetic Data Generation

Generate realistic test data with the GoMask CLI

Synthetic Data Generation Guide

Learn how to generate realistic synthetic test data using the GoMask CLI.

Overview

Synthetic data generation creates new, realistic test data that:

Maintains referential integrity across tables
Respects data types and constraints
Uses realistic values (names, emails, addresses, etc.)
Is safe for testing environments

Quick Start

Using the Setup Wizard (Recommended)

gomask setup

The wizard automatically:

Scans your database schema
Detects foreign key relationships
Assigns appropriate generation functions
Creates a ready-to-run YAML file

Manual Configuration

Create a YAML file:

routine:
  name: "My Synthetic Data Routine"
  type: synthetic
  connector_id: 1

tables:
  - table_name: users
    schema_name: public
    target_record_count: 1000
    columns:
      - column_name: email
        generation_function: generate_email
      - column_name: first_name
        generation_function: generate_first_name

Run it:

gomask run my-routine.yaml --watch

YAML Configuration

Basic Structure

routine:
  name: "Display Name"
  type: synthetic
  connector_id: 1
  description: "Optional description"

settings:
  generation_mode: hierarchical
  global_record_count: 1000
  batch_size: 1000

tables:
  - table_name: users
    schema_name: public
    hierarchy_level: 0
    target_record_count: 1000
    columns:
      - column_name: email
        generation_function: generate_email
        generation_parameters:
          - name: domain
            value: "example.com"

Table Configuration

tables:
  - table_name: users
    schema_name: public
    hierarchy_level: 0
    target_record_count: 1000
    order_index: 0
    columns:
      - column_name: id
        generation_function: sequential_integer
        is_primary_key: true
      - column_name: email
        generation_function: generate_email

Column Configuration

columns:
  - column_name: email
    generation_function: generate_email
    generation_parameters:
      - name: domain
        value: "example.com"
    is_nullable: false
    is_unique: true
    is_excluded: false

Foreign Key Relationships

Hierarchy Levels

Tables with foreign keys must be generated in the correct order:

tables:
  # Parent table - generate first (level 0)
  - table_name: customers
    schema_name: public
    hierarchy_level: 0
    target_record_count: 100
    columns:
      - column_name: id
        generation_function: sequential_integer
        is_primary_key: true

  # Child table - generate after parent (level 1)
  - table_name: orders
    schema_name: public
    hierarchy_level: 1
    parent_table_name: customers
    target_record_count: 500
    columns:
      - column_name: id
        generation_function: sequential_integer
        is_primary_key: true
      - column_name: customer_id
        is_foreign_key: true
        referenced_table: customers
        referenced_column: id

Record Distribution

Control how many child records are created per parent:

tables:
  - table_name: orders
    hierarchy_level: 1
    parent_table_name: customers
    target_record_count: 500
    record_distribution:
      distribution_type: uniform
      min_records_per_parent: 1
      max_records_per_parent: 10

Common Patterns

Basic Person Data

columns:
  - column_name: first_name
    generation_function: generate_first_name
  - column_name: last_name
    generation_function: generate_last_name
  - column_name: email
    generation_function: generate_email
  - column_name: phone
    generation_function: generate_phone_number
  - column_name: birthdate
    generation_function: generate_date_of_birth
    generation_parameters:
      - name: minimum_age
        value: 18
      - name: maximum_age
        value: 65

Personalized Emails

Generate emails based on name columns:

columns:
  - column_name: first_name
    generation_function: generate_first_name
  - column_name: last_name
    generation_function: generate_last_name
  - column_name: email
    generation_function: generate_email
    generation_parameters:
      - name: first_name_column
        valueType: reference
        columnReference: first_name
      - name: last_name_column
        valueType: reference
        columnReference: last_name
      - name: domain
        value: "company.com"

Financial Data

columns:
  - column_name: account_number
    generation_function: generate_iban
  - column_name: balance
    generation_function: generate_decimal
    generation_parameters:
      - name: min
        value: 0
      - name: max
        value: 100000
      - name: precision
        value: 2
  - column_name: currency
    generation_function: generate_currency_code

Date Ranges

columns:
  - column_name: created_at
    generation_function: generate_datetime
    generation_parameters:
      - name: start
        value: "2023-01-01"
      - name: end
        value: "2024-12-31"
  - column_name: expires_at
    generation_function: generate_future_datetime
    generation_parameters:
      - name: days
        value: 365

Static Values

columns:
  - column_name: status
    generation_function: constant
    generation_parameters:
      - name: value
        value: "active"
  - column_name: deleted_at
    generation_function: null_value

Random Selection

columns:
  - column_name: status
    generation_function: random_choice
    generation_parameters:
      - name: choices
        value: ["pending", "active", "completed", "cancelled"]
      - name: weights
        value: [0.1, 0.5, 0.3, 0.1]

Runtime Parameters

Make routines configurable at runtime:

routine:
  name: "Parameterized Routine"
  type: synthetic
  connector_id: 1

settings:
  runtime_parameter_definitions:
    - key: "param_record_count"
      name: "record_count"
      type: integer
      defaultValue: 1000
      description: "Number of records to generate"
    - key: "param_environment"
      name: "environment"
      type: string
      defaultValue: "dev"
      description: "Target environment"

tables:
  - table_name: users
    target_record_count: ${record_count}

Run with parameters:

gomask run routine.yaml --param record_count=5000 --param environment=staging

Validation

Always validate before running:

# Basic validation
gomask validate routine.yaml

# With detailed errors
gomask validate routine.yaml --detailed

# With environment variables
gomask validate routine.yaml --env-file .env

# Show parsed configuration
gomask validate routine.yaml --show-config

Execution

Basic Run

gomask run routine.yaml

With Progress Monitoring

gomask run routine.yaml --watch

Dry Run (Preview)

gomask run routine.yaml --dry-run

With Timeout

gomask run routine.yaml --timeout 7200  # 2 hours

Best Practices

Use the wizard - gomask setup detects relationships automatically
Validate first - Always gomask validate before running
Start small - Test with low record counts before scaling up
Version control - Store YAML files in git
Use parameters - Make routines reusable across environments

Troubleshooting

Foreign Key Violations

Problem: Child records reference non-existent parent IDs

Solution: Check hierarchy levels - parents must be lower than children

tables:
  - table_name: customers
    hierarchy_level: 0    # Generate first
  - table_name: orders
    hierarchy_level: 1    # Generate after customers
    parent_table_name: customers

Column Not Found

Problem: Function references a column that doesn't exist

Solution: Ensure referenced columns are defined earlier in the column list

Type Mismatch

Problem: Generated data doesn't match column type

Solution: Check the function's output type matches the column data type

Next Steps

Browse Functions - All available generators
Data Masking - Mask production data
CI/CD Integration - Automate generation