diff --git a/cluster/manifests/freeleaps-data-platform/kafka/README.md b/cluster/manifests/freeleaps-data-platform/kafka/README.md new file mode 100644 index 00000000..931c402b --- /dev/null +++ b/cluster/manifests/freeleaps-data-platform/kafka/README.md @@ -0,0 +1,75 @@ +# Kafka Configuration for Freeleaps Data Platform + +This directory contains the Kafka configuration for the Freeleaps Data Platform. + +## Files + +- `values.yaml` - Original Bitnami Kafka Helm values (legacy) +- `vpa.yaml` - Vertical Pod Autoscaler configuration +- `strimzi-kafka-cluster.yaml` - Strimzi Kafka Operator cluster configuration +- `README.md` - This file + +## Migration from Bitnami to Strimzi + +### Current State +- **Existing**: Bitnami Kafka deployment (single node) +- **Target**: Strimzi Kafka Operator with HA cluster (3 brokers + 3 ZK nodes) + +### Migration Steps + +1. **Install Strimzi Operator** (already done) + ```bash + helm install strimzi-kafka-operator strimzi/strimzi-kafka-operator \ + --namespace freeleaps-data-platform + ``` + +2. **Delete existing Kafka deployment** + ```bash + kubectl delete statefulset kafka-controller -n freeleaps-data-platform + kubectl delete service kafka kafka-controller-headless -n freeleaps-data-platform + kubectl delete configmap kafka-controller-configuration -n freeleaps-data-platform + kubectl delete pvc data-kafka-controller-0 -n freeleaps-data-platform + ``` + +3. **Deploy new Strimzi Kafka cluster** + ```bash + kubectl apply -f strimzi-kafka-cluster.yaml + ``` + +### New Cluster Features + +- **High Availability**: 3 Kafka brokers + 3 Zookeeper nodes +- **Authentication**: SCRAM-SHA-512 +- **Storage**: 20Gi per broker, 10Gi per ZK node +- **Replication**: 3x replication factor +- **Security**: TLS support for internal communication + +### Monitoring + +Check cluster status: +```bash +kubectl get kafka -n freeleaps-data-platform +kubectl get kafkatopic -n freeleaps-data-platform +kubectl get kafkauser -n freeleaps-data-platform +``` + +### Client Configuration + +For applications connecting to the new Kafka cluster, use: +- **Bootstrap servers**: `kafka-cluster-kafka-bootstrap.freeleaps-data-platform.svc.freeleaps.cluster:9092` +- **Authentication**: SCRAM-SHA-512 +- **Username**: `freeleaps-user` +- **Password**: Generated by Strimzi (check the secret) + +## Storage Requirements + +- **Kafka brokers**: 3 × 20Gi = 60Gi +- **Zookeeper**: 3 × 10Gi = 30Gi +- **Total**: 90Gi + +## Network Policy + +The cluster includes network policies that allow: +- Internal cluster communication +- External client access (with proper authentication) +- Metrics collection (if enabled) diff --git a/cluster/manifests/freeleaps-data-platform/kafka/storage-class.yaml b/cluster/manifests/freeleaps-data-platform/kafka/storage-class.yaml new file mode 100644 index 00000000..0d0170df --- /dev/null +++ b/cluster/manifests/freeleaps-data-platform/kafka/storage-class.yaml @@ -0,0 +1,14 @@ +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: kafka-storage-class + annotations: + storageclass.kubernetes.io/is-default-class: "false" +provisioner: kubernetes.io/azure-disk +parameters: + skuName: StandardSSD_LRS + cachingMode: ReadWrite + kind: Managed +reclaimPolicy: Retain +volumeBindingMode: WaitForFirstConsumer +allowVolumeExpansion: true diff --git a/cluster/manifests/freeleaps-data-platform/kafka/strimzi-kafka-cluster.yaml b/cluster/manifests/freeleaps-data-platform/kafka/strimzi-kafka-cluster.yaml new file mode 100644 index 00000000..e28118a6 --- /dev/null +++ b/cluster/manifests/freeleaps-data-platform/kafka/strimzi-kafka-cluster.yaml @@ -0,0 +1,90 @@ +apiVersion: kafka.strimzi.io/v1beta2 +kind: Kafka +metadata: + name: kafka-cluster + namespace: freeleaps-data-platform + labels: + app: kafka + component: messaging +spec: + kafka: + version: 3.8.0 + replicas: 3 + listeners: + - name: plain + port: 9092 + type: internal + tls: false + authentication: + type: scram-sha-512 + - name: tls + port: 9093 + type: internal + tls: true + authentication: + type: tls + config: + offsets.topic.replication.factor: 3 + transaction.state.log.replication.factor: 3 + transaction.state.log.min.isr: 2 + default.replication.factor: 3 + min.insync.replicas: 2 + inter.broker.protocol.version: "3.8" + log.retention.hours: 168 # 7 days + log.segment.bytes: 1073741824 # 1GB + num.partitions: 8 + log.retention.check.interval.ms: 300000 + storage: + type: persistent-claim + size: 20Gi + deleteClaim: false + zookeeper: + replicas: 3 + storage: + type: persistent-claim + size: 10Gi + deleteClaim: false + entityOperator: + topicOperator: {} + userOperator: {} +--- +apiVersion: kafka.strimzi.io/v1beta2 +kind: KafkaTopic +metadata: + name: system-events + namespace: freeleaps-data-platform + labels: + strimzi.io/cluster: kafka-cluster +spec: + partitions: 3 + replicas: 3 + config: + retention.ms: 604800000 # 7 days + segment.bytes: 1073741824 # 1GB + cleanup.policy: delete +--- +apiVersion: kafka.strimzi.io/v1beta2 +kind: KafkaUser +metadata: + name: freeleaps-user + namespace: freeleaps-data-platform + labels: + strimzi.io/cluster: kafka-cluster +spec: + authentication: + type: scram-sha-512 + authorization: + type: simple + acls: + - resource: + type: topic + name: "*" + patternType: literal + operation: All + host: "*" + - resource: + type: group + name: "*" + patternType: literal + operation: All + host: "*"