Maison/arti-api/auth-service/pipeline/EXTERNAL-BUILDAH-SYSTEM.md

# External Buildah Build System - Updated Documentation

## 🎯 Overview

Updated external build system with dynamic pod discovery and concurrent build protection.

## ✨ **New Features**

### 🔍 **Dynamic Pod Discovery**
- Automatically finds running Buildah pods using labels
- No more hardcoded pod names
- Resilient to pod restarts and recreations

### 🔒 **Concurrent Build Protection**
- Lock file mechanism prevents simultaneous builds
- Automatic cleanup of stale locks (older than 10 minutes)
- Timeout protection (5-minute maximum wait)
- Guaranteed lock release even on build failure

### 🛠️ **Enhanced Management**
- Updated management script with dynamic pod discovery
- Lock management commands
- Better error handling and status reporting

## 📋 **How It Works**

### **Dynamic Pod Discovery**
```bash
BUILDAH_POD=$(kubectl get pods -n apps--droneio--prd -l app=buildah-external --field-selector=status.phase=Running -o jsonpath='{.items[0].metadata.name}')
```

### **Locking Mechanism**
1. **Acquire Lock**: Creates `/workspace/locks/build-${DRONE_BUILD_NUMBER}.lock`
2. **Wait for Lock**: Up to 5 minutes timeout with 5-second intervals
3. **Auto-cleanup**: Removes locks older than 10 minutes
4. **Guaranteed Release**: Cleanup step runs on success OR failure

### **Build Process**
1. Find current Buildah pod dynamically
2. Acquire build lock with timeout
3. Transfer source code
4. Execute build in isolated workspace
5. Retrieve results
6. Clean up workspace and release lock

## 🚀 **Usage**

### **Deploy the System**
```bash
./deploy-external-buildah.sh
```

### **Use Production Configuration**
```bash
cp .drone.yml.external-buildah-production .drone.yml
# OR use the current updated version
git add .drone.yml
git commit -m "Implement dynamic external Buildah build"
git push
```

### **Management Commands**
```bash
# Complete status overview
./manage-external-buildah.sh status

# Lock management
./manage-external-buildah.sh locks list    # List current locks
./manage-external-buildah.sh locks clean   # Remove old locks
./manage-external-buildah.sh locks clear   # Remove ALL locks

# Test functionality
./manage-external-buildah.sh test

# Clean old builds
./manage-external-buildah.sh clean
```

## 🔧 **Configuration Files**

### **Updated Files**
- ✅ `.drone.yml` - Updated with dynamic discovery and locking
- ✅ `manage-external-buildah.sh` - Enhanced management script
- ✅ `buildah-external-deployment.yaml` - External Buildah service
- ✅ `buildah-rbac.yaml` - RBAC configuration

### **Key Configuration Elements**

#### **Pod Discovery**
```yaml
- BUILDAH_POD=$(kubectl get pods -n apps--droneio--prd -l app=buildah-external --field-selector=status.phase=Running -o jsonpath='{.items[0].metadata.name}')
```

#### **Lock Management**
```yaml
- LOCK_FILE="/workspace/locks/build-${DRONE_BUILD_NUMBER}.lock"
- timeout=300  # 5 minutes maximum wait
```

#### **Cleanup Step**
```yaml
- name: cleanup-build-lock
  when:
    status:
    - success
    - failure
```

## 📊 **Benefits**

### **Reliability**
- ✅ No hardcoded pod names
- ✅ Automatic pod discovery
- ✅ Resilient to restarts

### **Concurrency**
- ✅ Prevents build conflicts
- ✅ Automatic lock cleanup
- ✅ Timeout protection

### **Maintenance**
- ✅ Self-managing system
- ✅ Comprehensive status reporting
- ✅ Easy troubleshooting

## 🎯 **Next Steps**

1. **Test the Updated System**:
   ```bash
   ./manage-external-buildah.sh status
   ```

2. **Commit the Configuration**:
   ```bash
   git add .drone.yml
   git commit -m "Add dynamic pod discovery and build locking"
   git push
   ```

3. **Monitor First Build**:
   - Watch Drone CI interface for build progress
   - Check locks: `./manage-external-buildah.sh locks list`
   - Verify cleanup: `./manage-external-buildah.sh status`

## 🔍 **Troubleshooting**

### **No Buildah Pod Found**
```bash
kubectl get pods -n apps--droneio--prd -l app=buildah-external
kubectl apply -f buildah-external-deployment.yaml
```

### **Lock Issues**
```bash
# Clean old locks
./manage-external-buildah.sh locks clean

# Clear all locks (emergency)
./manage-external-buildah.sh locks clear
```

### **Build Failures**
```bash
# Check pod logs
./manage-external-buildah.sh logs

# Check pod details
./manage-external-buildah.sh details
```

The system is now production-ready with robust error handling, dynamic discovery, and concurrent build protection!