Maison/arti-api/auth-service/pipeline/EXTERNAL-BUILDAH-SYSTEM.md

External Buildah Build System - Updated Documentation

🎯 Overview

Updated external build system with dynamic pod discovery and concurrent build protection.

✨ New Features

🔍 Dynamic Pod Discovery

• Automatically finds running Buildah pods using labels
• No more hardcoded pod names
• Resilient to pod restarts and recreations

🔒 Concurrent Build Protection

• Lock file mechanism prevents simultaneous builds
• Automatic cleanup of stale locks (older than 10 minutes)
• Timeout protection (5-minute maximum wait)
• Guaranteed lock release even on build failure

🛠️ Enhanced Management

• Updated management script with dynamic pod discovery
• Lock management commands
• Better error handling and status reporting

📋 How It Works

Dynamic Pod Discovery

BUILDAH_POD=$(kubectl get pods -n apps--droneio--prd -l app=buildah-external --field-selector=status.phase=Running -o jsonpath='{.items[0].metadata.name}')

Locking Mechanism

1. Acquire Lock: Creates /workspace/locks/build-${DRONE_BUILD_NUMBER}.lock
2. Wait for Lock: Up to 5 minutes timeout with 5-second intervals
3. Auto-cleanup: Removes locks older than 10 minutes
4. Guaranteed Release: Cleanup step runs on success OR failure

Build Process

1. Find current Buildah pod dynamically
2. Acquire build lock with timeout
3. Transfer source code
4. Execute build in isolated workspace
5. Retrieve results
6. Clean up workspace and release lock

🚀 Usage

Deploy the System

./deploy-external-buildah.sh

Use Production Configuration

cp .drone.yml.external-buildah-production .drone.yml
# OR use the current updated version
git add .drone.yml
git commit -m "Implement dynamic external Buildah build"
git push

Management Commands

# Complete status overview
./manage-external-buildah.sh status

# Lock management
./manage-external-buildah.sh locks list    # List current locks
./manage-external-buildah.sh locks clean   # Remove old locks
./manage-external-buildah.sh locks clear   # Remove ALL locks

# Test functionality
./manage-external-buildah.sh test

# Clean old builds
./manage-external-buildah.sh clean

🔧 Configuration Files

Updated Files

• ✅ .drone.yml - Updated with dynamic discovery and locking
• ✅ manage-external-buildah.sh - Enhanced management script
• ✅ buildah-external-deployment.yaml - External Buildah service
• ✅ buildah-rbac.yaml - RBAC configuration

Key Configuration Elements

Pod Discovery

- BUILDAH_POD=$(kubectl get pods -n apps--droneio--prd -l app=buildah-external --field-selector=status.phase=Running -o jsonpath='{.items[0].metadata.name}')

Lock Management

- LOCK_FILE="/workspace/locks/build-${DRONE_BUILD_NUMBER}.lock"
- timeout=300  # 5 minutes maximum wait

Cleanup Step

- name: cleanup-build-lock
  when:
    status:
    - success
    - failure

📊 Benefits

Reliability

• ✅ No hardcoded pod names
• ✅ Automatic pod discovery
• ✅ Resilient to restarts

Concurrency

• ✅ Prevents build conflicts
• ✅ Automatic lock cleanup
• ✅ Timeout protection

Maintenance

• ✅ Self-managing system
• ✅ Comprehensive status reporting
• ✅ Easy troubleshooting

🎯 Next Steps

1. Test the Updated System:

   ./manage-external-buildah.sh status
   

2. Commit the Configuration:

   git add .drone.yml
   git commit -m "Add dynamic pod discovery and build locking"
   git push
   

3. Monitor First Build:

   • Watch Drone CI interface for build progress
   • Check locks: ./manage-external-buildah.sh locks list
   • Verify cleanup: ./manage-external-buildah.sh status

🔍 Troubleshooting

No Buildah Pod Found

kubectl get pods -n apps--droneio--prd -l app=buildah-external
kubectl apply -f buildah-external-deployment.yaml

Lock Issues

# Clean old locks
./manage-external-buildah.sh locks clean

# Clear all locks (emergency)
./manage-external-buildah.sh locks clear

Build Failures

# Check pod logs
./manage-external-buildah.sh logs

# Check pod details
./manage-external-buildah.sh details

The system is now production-ready with robust error handling, dynamic discovery, and concurrent build protection!