Operations: Multi-Environment Validation

Overview

Applications deployed across dev, staging, and production environments should behave consistently despite different configurations. Environment-specific details (hostnames, credentials, debug logging) make direct log comparison impossible. Normalizing logs reveals whether core behavior matches across environments.

Core Problem Statement

"Environment configurations differ, making it impossible to verify application behavior is consistent." Each environment has unique hostnames, database endpoints, session IDs, and logging levels, but the core business logic should be identical.

Example Scenario

Your e-commerce application runs in three environments:

• Dev: Localhost with debug logging enabled
• Staging: Internal staging infrastructure with standard logging
• Production: Production infrastructure with minimal logging

You need to verify that an order flow (login → create order → process payment → ship) works identically across all environments.

Input Data

Development Environment

2024-11-15T10:00:01.234Z [DEBUG] localhost:8080 - Connecting to database: postgres@localhost:5432/myapp_dev
2024-11-15T10:00:01.567Z [INFO] localhost:8080 - User login: alice@example.com (session: dev-sess-a1b2c3)
2024-11-15T10:00:02.890Z [DEBUG] localhost:8080 - Query executed: SELECT * FROM orders WHERE user_id=123 (15ms)
2024-11-15T10:00:03.123Z [INFO] localhost:8080 - Order created: #5001 for user 123
2024-11-15T10:00:03.456Z [DEBUG] localhost:8080 - Cache miss: order:5001
2024-11-15T10:00:03.789Z [INFO] localhost:8080 - Payment processed: order #5001, amount $99.99
2024-11-15T10:00:04.012Z [DEBUG] localhost:8080 - Email queued: order-confirmation to alice@example.com
2024-11-15T10:00:04.345Z [INFO] localhost:8080 - Order shipped: #5001

Development logs include DEBUG-level output, localhost hostnames, and local database connections.

Staging Environment

2024-11-15T10:15:23.456Z [INFO] api-staging-01.internal:8080 - Connecting to database: postgres@db-staging.internal:5432/myapp
2024-11-15T10:15:23.789Z [INFO] api-staging-01.internal:8080 - User login: alice@example.com (session: stg-sess-x7y8z9)
2024-11-15T10:15:25.012Z [INFO] api-staging-01.internal:8080 - Order created: #5001 for user 123
2024-11-15T10:15:25.345Z [INFO] api-staging-01.internal:8080 - Payment processed: order #5001, amount $99.99
2024-11-15T10:15:26.678Z [INFO] api-staging-01.internal:8080 - Order shipped: #5001

Staging logs use INFO-level only, staging hostnames, and staging infrastructure.

Production Environment

2024-11-15T14:32:45.123Z [INFO] api-prod-us-east-1a.internal:8080 - Connecting to database: postgres@db-prod-primary.internal:5432/myapp
2024-11-15T14:32:45.456Z [INFO] api-prod-us-east-1a.internal:8080 - User login: alice@example.com (session: prod-sess-p9q0r1)
2024-11-15T14:32:46.789Z [INFO] api-prod-us-east-1a.internal:8080 - Order created: #5001 for user 123
2024-11-15T14:32:47.012Z [INFO] api-prod-us-east-1a.internal:8080 - Payment processed: order #5001, amount $99.99
2024-11-15T14:32:48.345Z [INFO] api-prod-us-east-1a.internal:8080 - Order shipped: #5001

Production logs use INFO-level, production hostnames with availability zones, and production database endpoints.

Normalization Rules

Create rules that extract business logic while ignoring environment-specific details:

Multi-Environment Normalization Rules

rules:
  # Database connection - ignore hostname, port, database name
  - name: db_connection
    pattern:
      - field: timestamp
      - text: " ["
      - field: level
      - text: "] "
      - field: host
      - text: " - Connecting to database: "
      - field: connection_string
    output: "[db-connect]"

  # User login - keep email, ignore session ID and hostname
  - name: user_login
    pattern:
      - field: timestamp
      - text: " [INFO] "
      - field: host
      - text: " - User login: "
      - field: email
      - text: " (session: "
      - field: session_id
      - text: ")"
    output: "[user-login:{email}]"

  # Debug query - ignore in normalization (DEBUG logs vary by environment)
  - name: debug_query
    pattern:
      - field: timestamp
      - text: " [DEBUG] "
      - field: host
      - text: " - Query executed: "
      - field: query
      - text: " ("
      - field: duration
      - text: ")"
    output: "[query-executed]"

  # Debug cache - ignore in normalization
  - name: debug_cache
    pattern:
      - field: timestamp
      - text: " [DEBUG] "
      - field: host
      - text: " - Cache miss: "
      - field: key
    output: "[cache-miss]"

  # Debug email - ignore in normalization
  - name: debug_email
    pattern:
      - field: timestamp
      - text: " [DEBUG] "
      - field: host
      - text: " - Email queued: "
      - field: template
      - text: " to "
      - field: recipient
    output: "[email-queued]"

  # Order created - keep order ID and user ID
  - name: order_created
    pattern:
      - field: timestamp
      - text: " [INFO] "
      - field: host
      - text: " - Order created: #"
      - field: order_id
      - text: " for user "
      - field: user_id
    output: "[order-created:{order_id},user:{user_id}]"

  # Payment processed - keep order ID and amount
  - name: payment_processed
    pattern:
      - field: timestamp
      - text: " [INFO] "
      - field: host
      - text: " - Payment processed: order #"
      - field: order_id
      - text: ", amount $"
      - field: amount
    output: "[payment-processed:{order_id},amount:${amount}]"

  # Order shipped - keep order ID
  - name: order_shipped
    pattern:
      - field: timestamp
      - text: " [INFO] "
      - field: host
      - text: " - Order shipped: #"
      - field: order_id
    output: "[order-shipped:{order_id}]"

Rules preserve: business events (orders, payments, shipping), user identifiers. Rules ignore: hostnames, database endpoints, session IDs, timestamps, DEBUG-level logs.

Implementation

CLIPython

# Normalize logs from all three environments
patterndb-yaml --rules multi-env-rules.yaml dev.log \
    --quiet > normalized-dev.log

patterndb-yaml --rules multi-env-rules.yaml staging.log \
    --quiet > normalized-staging.log

patterndb-yaml --rules multi-env-rules.yaml prod.log \
    --quiet > normalized-prod.log

# Extract core business logic (INFO-level events only)
grep -v '^\[query-executed\]\|\[cache-miss\]\|\[email-queued\]' \
    normalized-dev.log > dev-core.log

# Compare core behavior
if diff -q dev-core.log normalized-staging.log && \
   diff -q normalized-staging.log normalized-prod.log; then
    echo "✓ All environments behave identically"
else
    echo "✗ Environment behavior differs"
    diff dev-core.log normalized-staging.log
    diff normalized-staging.log normalized-prod.log
fi

import sys
from patterndb_yaml import PatterndbYaml
from pathlib import Path
import subprocess

# Redirect stdout to file for testing
_original_stdout = sys.stdout
output_file = open("output.txt", "w")
sys.stdout = output_file

# Normalize all three environments
processor = PatterndbYaml(rules_path=Path("multi-env-rules.yaml"))

environments = ['dev', 'staging', 'prod']
for env in environments:
    with open(f"{env}.log") as f:
        with open(f"normalized-{env}.log", "w") as out:
            processor.process(f, out)

# Extract core business events (filter debug events)
debug_events = {'[query-executed]', '[cache-miss]', '[email-queued]'}

def extract_core_events(log_path):
    with open(log_path) as f:
        return [line.strip() for line in f
                if line.strip() not in debug_events]

dev_core = extract_core_events("normalized-dev.log")
staging_core = extract_core_events("normalized-staging.log")
prod_core = extract_core_events("normalized-prod.log")

# Compare
if dev_core == staging_core == prod_core:
    print("✓ All environments behave identically")
    print("\nCore business flow:")
    for event in dev_core:
        print(f"  {event}")
else:
    print("✗ Environment behavior differs")
    if dev_core != staging_core:
        print("\nDev vs Staging differences:")
        print(f"  Dev only: {set(dev_core) - set(staging_core)}")
        print(f"  Staging only: {set(staging_core) - set(dev_core)}")
    if staging_core != prod_core:
        print("\nStaging vs Prod differences:")
        print(f"  Staging only: {set(staging_core) - set(prod_core)}")
        print(f"  Prod only: {set(prod_core) - set(staging_core)}")

# Restore stdout and close output file
sys.stdout = _original_stdout
output_file.close()

Expected Output

Core Business Logic (All Environments)

[db-connect]
[user-login:alice@example.com]
[order-created:5001,user:123]
[payment-processed:5001,amount:$99.99]
[order-shipped:5001]

All environments produce identical normalized output for core business events.

Note: Development environment includes additional debug events ([query-executed], [cache-miss], [email-queued]) that are filtered out when comparing core business logic.

Practical Workflows

1. Deployment Validation

Verify new deployments match expected behavior:

#!/bin/bash
# Capture baseline from production
kubectl logs -l app=myapp -n production --tail=1000 > prod-baseline.log
patterndb-yaml --rules env-rules.yaml prod-baseline.log --quiet > prod-norm.log

# Deploy to staging and capture logs
kubectl logs -l app=myapp -n staging --tail=1000 > staging-test.log
patterndb-yaml --rules env-rules.yaml staging-test.log \
    --quiet > staging-norm.log

# Compare core events
if diff <(grep '^\[order\|payment\|ship' prod-norm.log | sort) \
        <(grep '^\[order\|payment\|ship' staging-norm.log | sort); then
    echo "✓ Staging matches production behavior"
    echo "Safe to promote to production"
else
    echo "✗ Staging behavior differs from production"
    echo "Review changes before promoting"
fi

2. Configuration Drift Detection

Detect when environments drift in behavior:

import sys
from patterndb_yaml import PatterndbYaml
from pathlib import Path
from collections import Counter

# Redirect stdout to file for testing
_original_stdout = sys.stdout
output_file = open("output.txt", "w")
sys.stdout = output_file

processor = PatterndbYaml(rules_path=Path("env-rules.yaml"))

# Normalize logs from each environment
def get_event_distribution(log_file):
    """Extract normalized events and count their frequency"""
    with open(log_file) as f:
        from io import StringIO
        output = StringIO()
        processor.process(f, output)
        output.seek(0)
        events = [line.strip() for line in output if line.strip()]
        return Counter(events)

prod_events = get_event_distribution("prod.log")
staging_events = get_event_distribution("staging.log")

# Compare event distributions
for event in set(prod_events.keys()) | set(staging_events.keys()):
    prod_count = prod_events.get(event, 0)
    staging_count = staging_events.get(event, 0)

    # Allow 10% variance
    variance = abs(prod_count - staging_count) / \
        max(prod_count, staging_count, 1)
    if variance > 0.10:
        print(f"⚠ Event frequency differs: {event}")
        print(f"  Production: {prod_count}, Staging: {staging_count}")

# Restore stdout and close output file
sys.stdout = _original_stdout
output_file.close()

3. Feature Flag Verification

Verify feature flags work correctly across environments:

# Normalize logs with feature flag enabled in dev
patterndb-yaml --rules env-rules.yaml dev-flag-on.log \
    --quiet > dev-flag-norm.log

# Normalize logs with feature flag enabled in staging
patterndb-yaml --rules env-rules.yaml staging-flag-on.log \
    --quiet > staging-flag-norm.log

# Compare to ensure feature behaves identically
diff dev-flag-norm.log staging-flag-norm.log || {
    echo "⚠ Feature flag behavior differs between environments"
    diff dev-flag-norm.log staging-flag-norm.log
}

4. Load Testing Validation

Verify load test in staging matches production patterns:

import sys
from patterndb_yaml import PatterndbYaml
from pathlib import Path

# Redirect stdout to file for testing
_original_stdout = sys.stdout
output_file = open("output.txt", "w")
sys.stdout = output_file

processor = PatterndbYaml(rules_path=Path("env-rules.yaml"))

# Normalize production baseline
with open("prod-normal-load.log") as f:
    with open("prod-norm.log", "w") as out:
        processor.process(f, out)

# Normalize staging load test
with open("staging-load-test.log") as f:
    with open("staging-norm.log", "w") as out:
        processor.process(f, out)

# Extract event sequences
def extract_sequences(log_file, user_field="user"):
    """Group events by user to extract workflows"""
    sequences = {}
    with open(log_file) as f:
        for line in f:
            if user_field in line:
                # Extract user ID and event
                # Format: [event:data,user:123]
                import re
                if match := re.search(r'\[([^:]+):.*?user:(\d+)', line):
                    event, user_id = match.groups()
                    sequences.setdefault(user_id, []).append(event)
    return sequences

prod_sequences = extract_sequences("prod-norm.log")
staging_sequences = extract_sequences("staging-norm.log")

# Compare workflow patterns
prod_workflows = set(tuple(seq) for seq in prod_sequences.values())
staging_workflows = set(tuple(seq) for seq in staging_sequences.values())

if staging_workflows.issubset(prod_workflows):
    print("✓ Staging load test exhibits production workflows")
else:
    print("⚠ Staging has unexpected workflows:")
    for workflow in staging_workflows - prod_workflows:
        print(f"  {' → '.join(workflow)}")

# Restore stdout and close output file
sys.stdout = _original_stdout
output_file.close()

5. Smoke Test Across Environments

Run same smoke test in all environments and verify identical behavior:

#!/bin/bash
# Run smoke test in each environment
for env in dev staging prod; do
    echo "Running smoke test in $env..."
    case $env in
        dev)     ENDPOINT="http://localhost:8080" ;;
        staging) ENDPOINT="https://staging.example.com" ;;
        prod)    ENDPOINT="https://api.example.com" ;;
    esac

    # Run test suite
    ./run-smoke-test.sh "$ENDPOINT" 2>&1 | tee "$env-smoke.log"

    # Normalize logs
    patterndb-yaml --rules env-rules.yaml "$env-smoke.log" \
        --quiet > "$env-smoke-norm.log"
done

# Compare all environments
if diff -q dev-smoke-norm.log staging-smoke-norm.log && \
   diff -q staging-smoke-norm.log prod-smoke-norm.log; then
    echo "✓ Smoke test passed identically in all environments"
else
    echo "✗ Smoke test results differ across environments"
    for env1 in dev staging; do
        for env2 in staging prod; do
            if [ "$env1" != "$env2" ]; then
                echo "\n$env1 vs $env2:"
                diff "$env1-smoke-norm.log" "$env2-smoke-norm.log" | head -20
            fi
        done
    done
fi

Key Benefits

• Verify environment parity: Ensure dev, staging, and prod behave identically
• Catch configuration drift: Detect when environments diverge
• Validate deployments: Confirm new releases work correctly before production
• Debug environment issues: Understand differences between environments
• Test with confidence: Verify load tests and feature flags across environments

Related Topics

• Rules - Pattern matching and normalization
• Statistics - Measure match coverage
• Explain Mode - Debug pattern matching