datablogs

MongoDB 8.0 Production Deployment on Azure AKS with Internal LoadBalancer + SRV Setup

Got free Azure credits to learn and explore Azure AKS with MongoDB. Tried my best to gain hands-on experience and learn as much as possible within the available limits. Sharing the detailed step-by-step implementation for anyone looking for reference or learning purposes.

Overview

This guide explains how to deploy a production-grade MongoDB 8.0 ReplicaSet on Azure Kubernetes Service (AKS) with:

  • 3-node MongoDB ReplicaSet

  • StatefulSet deployment

  • Persistent storage using Azure Premium SSD

  • Internal Azure LoadBalancer

  • MongoDB authentication

  • KeyFile internal authentication

  • DNS SRV connection setup

  • Expandable storage

  • Production-ready architecture


Architecture



Prerequisites

  • Azure AKS Cluster

  • kubectl configured

  • Azure Private DNS Zone

  • MongoDB compatible VM sizing

  • Premium SSD enabled AKS node pool


Step 1: Create Namespace

kubectl create namespace mongodb-prod

kubectl config set-context --current --namespace=mongodb-prod

Verify:

kubectl get ns

Step 2: Create StorageClass

This StorageClass uses Azure Premium SSD and allows future storage expansion.

cat <<EOF | kubectl apply -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: mongodb-storage-prod
provisioner: disk.csi.azure.com
parameters:
  skuName: Premium_LRS
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
EOF

Verify:

kubectl get storageclass

Step 3: Create MongoDB KeyFile

MongoDB ReplicaSet internal authentication requires a shared keyfile.

Generate keyfile:

openssl rand -base64 756 > mongo-keyfile

Create Kubernetes secret:

kubectl create secret generic mongodb-keyfile \
  --from-file=keyfile=mongo-keyfile \
  -n mongodb-prod

Verify:

kubectl get secrets -n mongodb-prod

Step 4: Create Headless Service

MongoDB StatefulSet requires a headless service for stable DNS resolution.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: mongodb-headless
  namespace: mongodb-prod
spec:
  clusterIP: None
  selector:
    app: mongodb
  ports:
    - name: mongodb
      port: 27717
      targetPort: 27717
EOF

Step 5: Deploy MongoDB StatefulSet (Without Auth Initially)

Deploy without auth first to create admin user cleanly.

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mongodb
  namespace: mongodb-prod
spec:
  serviceName: mongodb-headless
  replicas: 3
  selector:
    matchLabels:
      app: mongodb
  template:
    metadata:
      labels:
        app: mongodb
    spec:
      terminationGracePeriodSeconds: 60

      containers:
        - name: mongodb
          image: mongo:8.0

          command:
            - mongod
            - "--replSet"
            - rs0
            - "--bind_ip_all"
            - "--port"
            - "27717"

          ports:
            - containerPort: 27717

          volumeMounts:
            - name: mongodb-data
              mountPath: /data/db

          resources:
            requests:
              memory: "2Gi"
              cpu: "500m"
            limits:
              memory: "4Gi"
              cpu: "1000m"

  volumeClaimTemplates:
    - metadata:
        name: mongodb-data
      spec:
        accessModes:
          - ReadWriteOnce

        storageClassName: mongodb-storage-prod

        resources:
          requests:
            storage: 20Gi
EOF

Step 6: Verify Pods

kubectl get pods -n mongodb-prod -w

Expected:

mongodb-0   Running
mongodb-1   Running
mongodb-2   Running

Step 7: Initialize ReplicaSet

Connect to primary pod:

kubectl exec -it mongodb-0 -n mongodb-prod -c mongodb -- mongosh \
  --host localhost \
  --port 27717

Initialize ReplicaSet:

rs.initiate({
  _id: "rs0",
  members: [
    {
      _id: 0,
      host: "mongodb-0.mongodb-headless.mongodb-prod.svc.cluster.local:27717",
      priority: 2
    },
    {
      _id: 1,
      host: "mongodb-1.mongodb-headless.mongodb-prod.svc.cluster.local:27717",
      priority: 1
    },
    {
      _id: 2,
      host: "mongodb-2.mongodb-headless.mongodb-prod.svc.cluster.local:27717",
      priority: 1
    }
  ]
})

Verify:

rs.status()

Wait until mongodb-0 becomes PRIMARY.


Step 8: Create MongoDB Admin User

use admin

db.createUser({
  user: "admin",
  pwd: "StrongProductionPassword",
  roles: [
    {
      role: "root",
      db: "admin"
    }
  ]
})

Verify authentication:

db.auth("admin", "StrongProductionPassword")

Expected output:

1

Exit:

exit

Step 9: Enable MongoDB Authentication + KeyFile

Patch StatefulSet:

kubectl patch statefulset mongodb -n mongodb-prod --type='json' -p='[
  {
    "op":"add",
    "path":"/spec/template/spec/containers/0/volumeMounts/-",
    "value":{
      "name":"keyfile-volume",
      "mountPath":"/etc/mongodb"
    }
  },

  {
    "op":"add",
    "path":"/spec/template/spec/volumes",
    "value":[
      {
        "name":"keyfile-secret",
        "secret":{
          "secretName":"mongodb-keyfile"
        }
      },
      {
        "name":"keyfile-volume",
        "emptyDir":{}
      }
    ]
  },

  {
    "op":"add",
    "path":"/spec/template/spec/initContainers",
    "value":[
      {
        "name":"fix-keyfile-permissions",
        "image":"busybox",
        "command":[
          "sh",
          "-c",
          "cp /tmp/keyfile /etc/mongodb/keyfile && chmod 400 /etc/mongodb/keyfile && chown 999:999 /etc/mongodb/keyfile"
        ],
        "volumeMounts":[
          {
            "name":"keyfile-secret",
            "mountPath":"/tmp/keyfile",
            "subPath":"keyfile"
          },
          {
            "name":"keyfile-volume",
            "mountPath":"/etc/mongodb"
          }
        ]
      }
    ]
  },

  {
    "op":"add",
    "path":"/spec/template/spec/containers/0/command/-",
    "value":"--keyFile"
  },

  {
    "op":"add",
    "path":"/spec/template/spec/containers/0/command/-",
    "value":"/etc/mongodb/keyfile"
  },

  {
    "op":"add",
    "path":"/spec/template/spec/containers/0/command/-",
    "value":"--auth"
  }
]'

Wait for rollout:

kubectl rollout status statefulset/mongodb -n mongodb-prod

Step 10: Validate Authentication

kubectl exec -it mongodb-0 -n mongodb-prod -c mongodb -- mongosh \
  --host localhost \
  --port 27717 \
  -u admin \
  -p 'StrongProductionPassword' \
  --authenticationDatabase admin \
  --eval "rs.status()"

Step 11: Create Internal LoadBalancers

mongodb-0 ILB

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: mongodb-0-ilb
  namespace: mongodb-prod
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
spec:
  type: LoadBalancer

  selector:
    statefulset.kubernetes.io/pod-name: mongodb-0

  ports:
    - name: mongodb
      port: 27717
      targetPort: 27717
EOF

mongodb-1 ILB

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: mongodb-1-ilb
  namespace: mongodb-prod
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
spec:
  type: LoadBalancer

  selector:
    statefulset.kubernetes.io/pod-name: mongodb-1

  ports:
    - name: mongodb
      port: 27717
      targetPort: 27717
EOF

mongodb-2 ILB

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: mongodb-2-ilb
  namespace: mongodb-prod
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
spec:
  type: LoadBalancer

  selector:
    statefulset.kubernetes.io/pod-name: mongodb-2

  ports:
    - name: mongodb
      port: 27717
      targetPort: 27717
EOF

Step 12: Verify Internal IPs

kubectl get svc -n mongodb-prod

Expected:

mongodb-0-ilb   10.x.x.x
mongodb-1-ilb   10.x.x.x
mongodb-2-ilb   10.x.x.x

Step 13: Configure DNS

Create Private DNS records:

mongo-0.prodmongo.internal
mongo-1.prodmongo.internal
mongo-2.prodmongo.internal

Point each record to respective internal ILB IP.


Step 14: Configure SRV Record

SRV Record:

_mongodb._tcp.db.prodmongo.internal

Values:

0 0 27717 mongo-0.prodmongo.internal
0 0 27717 mongo-1.prodmongo.internal
0 0 27717 mongo-2.prodmongo.internal

Step 15: Configure TXT Record

TXT Record:

db.prodmongo.internal

Value:

replicaSet=rs0&authSource=admin

Step 16: Reconfigure ReplicaSet to External DNS

Connect:

kubectl exec -it mongodb-0 -n mongodb-prod -c mongodb -- mongosh \
  --host localhost \
  --port 27717 \
  -u admin \
  -p 'StrongProductionPassword' \
  --authenticationDatabase admin

Update ReplicaSet:

cfg = rs.conf()

cfg.members[0].host = "mongo-0.prodmongo.internal:27717"
cfg.members[1].host = "mongo-1.prodmongo.internal:27717"
cfg.members[2].host = "mongo-2.prodmongo.internal:27717"

rs.reconfig(cfg)

Step 17: Test SRV Connection

mongosh "mongodb+srv://admin:StrongProductionPassword@db.prodmongo.internal/admin"

Step 18: Final Validation

kubectl get all -n mongodb-prod

kubectl get pvc -n mongodb-prod

kubectl get svc -n mongodb-prod

Production Best Practices

Recommended

  • Use Azure Private DNS Zone

  • Use Premium SSD

  • Use Retain reclaimPolicy

  • Enable monitoring

  • Configure backups

  • Use TLS/SSL

  • Use Azure Monitor

  • Use Percona Backup for MongoDB

  • Enable alerts

Backup Recommendations

  • mongodump

  • Azure Disk Snapshots

  • Velero

  • Azure Backup Vault

Monitoring Recommendations

  • MongoDB Exporter

  • Prometheus

  • Grafana

  • Azure Monitor


Storage Expansion

Example:

kubectl patch pvc mongodb-data-mongodb-0 -n mongodb-prod \
  -p '{"spec":{"resources":{"requests":{"storage":"50Gi"}}}}'

Final SRV Connection String

mongodb+srv://admin:StrongProductionPassword@db.prodmongo.internal/admin


Still if you are facing any further issues , Please Contact me !!!

0 Comments