Rolling Updates

Rolling updates deploy new Docker images to a running cluster with zero downtime. Only container images are updated. Existing configuration, secrets, ingress, and persistent data are preserved.

Configuration

Add the following to your .env file:

# =============================================================================
# ROLLING UPDATE CONFIGURATION
# =============================================================================
UPDATE_IMAGES_ONLY=true            # Only update images - skip values regeneration and secret recreation
SKIP_MONGODB_UPDATE=true           # Exclude MongoDB from the image update (recommended for safety)
HELM_ATOMIC=true                   # Auto-rollback if upgrade fails (default: true)
HELM_WAIT=true                     # Wait for pods to be ready after upgrade (default: true)
HELM_TIMEOUT=5m                    # Timeout for Helm wait (default: 5m)

Flag	Description	Default
UPDATE_IMAGES_ONLY	Skip values.yaml regeneration and secret recreation; only update container images	false
SKIP_MONGODB_UPDATE	Pin MongoDB to its current image tag during the rolling update	false
HELM_ATOMIC	Automatically roll back if the Helm upgrade fails	true
HELM_WAIT	Wait for all pods to become ready before marking the upgrade successful	true
HELM_TIMEOUT	How long to wait for pods to become ready	5m

Usage

1. Update IMAGE_TAG in .env.

   # Edit .env and set the new image tag
   IMAGE_TAG=v1.0.2

2. Run the rolling update.

   # Update all images except MongoDB
   UPDATE_IMAGES_ONLY=true SKIP_MONGODB_UPDATE=true ./k8s-deploy.sh
   
   # Update all images including MongoDB
   UPDATE_IMAGES_ONLY=true ./k8s-deploy.sh
   
   # Skip image loading (images already in registry)
   UPDATE_IMAGES_ONLY=true SKIP_MONGODB_UPDATE=true SKIP_LOAD=true ./k8s-deploy.sh

   Or set the flags in your .env file and just run ./k8s-deploy.sh.

3. Monitor the rollout.

   # Watch pods rolling (new pods start before old ones terminate)
   kubectl get pods -n kmi -w
   
   # Verify ingress is intact
   kubectl get ingress -n kmi

4. Verify after deployment.

   # All pods should be 1/1 Running with 0 restarts
   kubectl get pods -n kmi
   
   # Confirm new image tags
   kubectl get pods -n kmi -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}{end}'
   
   # PVCs still bound (MongoDB data intact)
   kubectl get pvc -n kmi

What Happens During a Rolling Update

1. Images are pushed to the container registry with the new tag.
2. The MongoDB StatefulSet is pre-deleted (with --cascade=orphan) to avoid immutable-field errors. PVCs and data are preserved.
3. Helm applies the new image tags and Kubernetes performs a rolling update:
   • New pods start alongside old pods.
   • Once new pods pass readiness probes, old pods are terminated.
   • Frontend and backend use maxUnavailable: 0, maxSurge: 1, so at least one pod is always serving traffic.
4. The backend waits for MongoDB via an init container before starting, which prevents crash loops.
5. Post-deploy verification confirms all deployments rolled out successfully.

Troubleshooting Rolling Updates

Pods stuck in CrashLoopBackOff:

kubectl describe pod <pod-name> -n kmi
kubectl logs <pod-name> -n kmi --previous

Helm upgrade timed out:

# Check pod events
kubectl get events -n kmi --sort-by=.lastTimestamp | tail -20

# If HELM_ATOMIC=true, Helm auto-rolled back. Check release history:
helm history kafka-mobility-intelligence -n kmi

Ingress shows CLASS <none> after update:

• Ensure your .env has PLATFORM=eks (or the appropriate platform) set.

• The deploy script auto-detects from the registry URL, but an explicit PLATFORM is more reliable.

• Quick fix:

  kubectl patch ingress kafka-mobility-intelligence-ingress -n kmi -p '{"spec":{"ingressClassName":"alb"}}' --type=merge

Manual rollback:

helm rollback kafka-mobility-intelligence -n kmi