574 lines
28 KiB
Markdown
574 lines
28 KiB
Markdown
# Custom Resources & Operators Guide
|
|
|
|
## 🎯 **Overview**
|
|
|
|
This guide explains **Custom Resources (CRs)**, **Custom Resource Definitions (CRDs)**, **Kubernetes Operators**, and how your `freeleaps-devops-reconciler` works as an operator to manage your DevOps infrastructure.
|
|
|
|
---
|
|
|
|
## 📊 **What Are Custom Resources?**
|
|
|
|
### **🔄 CR vs CRD Relationship**
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────────────┐
|
|
│ CRD vs CR RELATIONSHIP │
|
|
│ │
|
|
│ ┌─────────────────────────────────────────────────────────────────────┐ │
|
|
│ │ CUSTOM RESOURCE DEFINITION (CRD) │ │ │
|
|
│ │ │ │ │
|
|
│ │ apiVersion: apiextensions.k8s.io/v1 │ │ │
|
|
│ │ kind: CustomResourceDefinition │ │ │
|
|
│ │ metadata: │ │ │
|
|
│ │ name: devopsprojects.freeleaps.com │ │ │
|
|
│ │ spec: │ │ │
|
|
│ │ group: freeleaps.com │ │ │
|
|
│ │ names: │ │ │
|
|
│ │ kind: DevOpsProject │ │ │
|
|
│ │ plural: devopsprojects │ │ │
|
|
│ │ scope: Namespaced │ │ │
|
|
│ │ versions: │ │ │
|
|
│ │ - name: v1alpha1 │ │ │
|
|
│ │ schema: │ │ │
|
|
│ │ # Schema definition... │ │ │
|
|
│ └─────────────────────────────────────────────────────────────────────┘ │ │
|
|
│ │ │ │
|
|
│ ▼ │ │
|
|
│ ┌─────────────────────────────────────────────────────────────────────┐ │ │
|
|
│ │ CUSTOM RESOURCE (CR) │ │ │
|
|
│ │ │ │ │
|
|
│ │ apiVersion: freeleaps.com/v1alpha1 │ │ │
|
|
│ │ kind: DevOpsProject │ │ │
|
|
│ │ metadata: │ │ │
|
|
│ │ name: my-project │ │ │
|
|
│ │ namespace: freeleaps-devops-system │ │ │
|
|
│ │ spec: │ │ │
|
|
│ │ projectName: "My Awesome Project" │ │ │
|
|
│ │ projectId: "my-awesome-project" │ │ │
|
|
│ │ git: │ │ │
|
|
│ │ url: "https://github.com/myorg/myproject" │ │ │
|
|
│ │ branch: "main" │ │ │
|
|
│ │ registry: │ │ │
|
|
│ │ url: "https://harbor.example.com" │ │ │
|
|
│ │ project: "myproject" │ │ │
|
|
│ └─────────────────────────────────────────────────────────────────────┘ │ │
|
|
└─────────────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### **🎯 Why Custom Resources?**
|
|
|
|
```yaml
|
|
# Instead of managing multiple resources manually:
|
|
# - Namespace
|
|
# - ServiceAccount
|
|
# - Role/RoleBinding
|
|
# - ConfigMap
|
|
# - Secret
|
|
# - Deployment
|
|
# - Service
|
|
# - Ingress
|
|
|
|
# You can create ONE custom resource:
|
|
apiVersion: freeleaps.com/v1alpha1
|
|
kind: DevOpsProject
|
|
metadata:
|
|
name: my-project
|
|
spec:
|
|
projectName: "My Project"
|
|
projectId: "my-project"
|
|
git:
|
|
url: "https://github.com/myorg/myproject"
|
|
branch: "main"
|
|
registry:
|
|
url: "https://harbor.example.com"
|
|
project: "myproject"
|
|
```
|
|
|
|
---
|
|
|
|
## 🏭 **Your DevOps Reconciler Architecture**
|
|
|
|
### **📊 Reconciler vs DevOps Repo Relationship**
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────────────┐
|
|
│ RECONCILER + DEVOPS ARCHITECTURE │
|
|
│ │
|
|
│ ┌─────────────────────────────────────────────────────────────────────┐ │
|
|
│ │ FRELEAPS.COM PLATFORM │ │ │
|
|
│ │ │ │ │
|
|
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
|
|
│ │ │ User │ │ Project │ │ Git │ │ Registry │ │ │
|
|
│ │ │ Creates │ │ Manager │ │ Webhook │ │ Manager │ │ │
|
|
│ │ │ Project │ │ Creates │ │ Triggers │ │ Creates │ │ │
|
|
│ │ │ │ │ DevOps │ │ Event │ │ Repo │ │ │
|
|
│ │ │ │ │ Project │ │ │ │ │ │ │
|
|
│ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │ │
|
|
│ └─────────────────────────────────────────────────────────────────────┘ │ │
|
|
│ │ │ │
|
|
│ ▼ │ │
|
|
│ ┌─────────────────────────────────────────────────────────────────────┐ │ │
|
|
│ │ RABBITMQ MESSAGE QUEUE │ │ │
|
|
│ │ │ │ │
|
|
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
|
|
│ │ │DevOpsInit │ │DevOpsReconcile│ │DevOpsDeploy │ │DevOpsDelete │ │ │
|
|
│ │ │Event │ │Event │ │Event │ │Event │ │ │
|
|
│ │ │ │ │ │ │ │ │ │ │ │
|
|
│ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │ │
|
|
│ └─────────────────────────────────────────────────────────────────────┘ │ │
|
|
│ │ │ │
|
|
│ ▼ │ │
|
|
│ ┌─────────────────────────────────────────────────────────────────────┐ │ │
|
|
│ │ FRELEAPS-DEVOPS-RECONCILER (OPERATOR) │ │ │
|
|
│ │ │ │ │
|
|
│ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │
|
|
│ │ │ CONTROLLERS │ │ │ │
|
|
│ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────┐ │ │ │ │
|
|
│ │ │ │DevOpsProject│ │ArgoSettings │ │Jenkins │ │... │ │ │ │ │
|
|
│ │ │ │Controller │ │Controller │ │Settings │ │ │ │ │ │ │
|
|
│ │ │ │ │ │ │ │Controller │ │ │ │ │ │ │
|
|
│ │ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────┘ │ │ │ │
|
|
│ │ └─────────────────────────────────────────────────────────────┘ │ │ │
|
|
│ │ │ │ │ │
|
|
│ │ ▼ │ │ │
|
|
│ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │
|
|
│ │ │ CUSTOM RESOURCES │ │ │ │
|
|
│ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────┐ │ │ │ │
|
|
│ │ │ │DevOpsProject│ │ArgoSettings │ │Jenkins │ │... │ │ │ │ │
|
|
│ │ │ │CR │ │CR │ │Settings │ │ │ │ │ │ │
|
|
│ │ │ │ │ │ │ │CR │ │ │ │ │ │ │
|
|
│ │ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────┘ │ │ │ │
|
|
│ │ └─────────────────────────────────────────────────────────────┘ │ │ │
|
|
│ └─────────────────────────────────────────────────────────────────────┘ │ │
|
|
│ │ │ │
|
|
│ ▼ │ │
|
|
│ ┌─────────────────────────────────────────────────────────────────────┐ │ │
|
|
│ │ KUBERNETES RESOURCES │ │ │
|
|
│ │ │ │ │
|
|
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
|
|
│ │ │ArgoCD │ │Jenkins │ │Harbor │ │Namespaces │ │ │
|
|
│ │ │Applications │ │Pipelines │ │Repositories │ │Services │ │ │
|
|
│ │ │Projects │ │Jobs │ │Credentials │ │Deployments │ │ │
|
|
│ │ │ │ │ │ │ │ │Ingresses │ │ │
|
|
│ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │ │
|
|
│ └─────────────────────────────────────────────────────────────────────┘ │
|
|
└─────────────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## 🔧 **Your Custom Resources**
|
|
|
|
### **1. DevOpsProject CRD**
|
|
|
|
```yaml
|
|
# 🏭 ACTUAL CRD FROM YOUR CODEBASE
|
|
# freeleaps-devops-reconciler/deploy/crds.yaml
|
|
apiVersion: apiextensions.k8s.io/v1
|
|
kind: CustomResourceDefinition
|
|
metadata:
|
|
name: devopsprojects.freeleaps.com
|
|
spec:
|
|
group: freeleaps.com
|
|
scope: Namespaced
|
|
names:
|
|
kind: DevOpsProject
|
|
plural: devopsprojects
|
|
shortNames: [dop, dops]
|
|
versions:
|
|
- name: v1alpha1
|
|
served: true
|
|
storage: true
|
|
schema:
|
|
openAPIV3Schema:
|
|
type: object
|
|
required: ['spec']
|
|
properties:
|
|
spec:
|
|
type: object
|
|
required:
|
|
- projectName
|
|
- projectId
|
|
- git
|
|
- registry
|
|
- environments
|
|
properties:
|
|
projectName:
|
|
type: string
|
|
description: "Human readable project name"
|
|
projectId:
|
|
type: string
|
|
description: "Unique project identifier"
|
|
pattern: "^[a-z0-9]([a-z0-9-]*[a-z0-9])?$"
|
|
git:
|
|
type: object
|
|
required: [url, branch]
|
|
properties:
|
|
url:
|
|
type: string
|
|
description: "Git repository URL"
|
|
branch:
|
|
type: string
|
|
description: "Default git branch"
|
|
default: "main"
|
|
registry:
|
|
type: object
|
|
required: [url, project]
|
|
properties:
|
|
url:
|
|
type: string
|
|
description: "Container registry URL"
|
|
project:
|
|
type: string
|
|
description: "Registry project name"
|
|
```
|
|
|
|
### **2. DevOpsProject CR Example**
|
|
|
|
```yaml
|
|
# 🏭 ACTUAL CR EXAMPLE
|
|
apiVersion: freeleaps.com/v1alpha1
|
|
kind: DevOpsProject
|
|
metadata:
|
|
name: magicleaps-frontend
|
|
namespace: freeleaps-devops-system
|
|
labels:
|
|
app.kubernetes.io/name: magicleaps-frontend
|
|
app.kubernetes.io/instance: magicleaps-frontend
|
|
spec:
|
|
projectName: "Magicleaps Frontend"
|
|
projectId: "magicleaps-frontend"
|
|
git:
|
|
url: "https://github.com/freeleaps/magicleaps-frontend"
|
|
branch: "main"
|
|
credentialsRef:
|
|
name: "github-credentials"
|
|
namespace: "freeleaps-devops-system"
|
|
registry:
|
|
url: "https://harbor.freeleaps.mathmast.com"
|
|
project: "magicleaps"
|
|
credentialsRef:
|
|
name: "harbor-credentials"
|
|
namespace: "freeleaps-devops-system"
|
|
environments:
|
|
- name: "production"
|
|
domain: "magicleaps.mathmast.com"
|
|
replicas: 3
|
|
- name: "alpha"
|
|
domain: "alpha.magicleaps.mathmast.com"
|
|
replicas: 1
|
|
```
|
|
|
|
### **3. Other Custom Resources**
|
|
|
|
```yaml
|
|
# 🏭 YOUR COMPLETE CRD SET
|
|
# From freeleaps-devops-reconciler/docs/design/one-click-deployment.md
|
|
|
|
# DevOpsProject - Main project configuration
|
|
# ArgoSettings - ArgoCD settings for the DevOpsProject
|
|
# JenkinsSettings - Jenkins settings and generated pipelines
|
|
# ContainerRegistry - Container registry information
|
|
# ContainerImage - Every image manufactured by Jenkins pipeline
|
|
# DeploymentRecord - Track deployment records
|
|
# GitCredential - Git repository credentials
|
|
# IngressResource - Ingress configuration
|
|
```
|
|
|
|
---
|
|
|
|
## 🤖 **How Your Operator Works**
|
|
|
|
### **🔄 Reconciliation Loop**
|
|
|
|
```python
|
|
# 🏭 ACTUAL CODE FROM YOUR RECONCILER
|
|
# freeleaps-devops-reconciler/reconciler/controllers/devops_projects/controller.py
|
|
|
|
@kopf.on.create(group=consts.GROUP, version=consts.VERSION, kind=consts.DEVOPS_PROJECT_KIND)
|
|
def on_devops_proj_created(name: str, namespace: Optional[str], body: Body, logger: Logger, **kwargs):
|
|
logger.info(f"Newly created DevOpsProject resource and named {name} in namespace {namespace}, start to reconciling...")
|
|
|
|
devops_proj = DevOpsProject(body)
|
|
|
|
try:
|
|
devops_proj.parse_spec()
|
|
devops_proj.get_spec().validate(logger)
|
|
except SpecError as e:
|
|
devops_proj.update_status({
|
|
'devopsProject': {
|
|
'status': DevOpsProjectDiagStatus.INVALID.value,
|
|
'synced': False,
|
|
'ready': False,
|
|
'lastProbeTime': isotime(),
|
|
}
|
|
})
|
|
devops_proj.error(action='CreateDevOpsProject',
|
|
reason='InvalidSpecArgument', msg=str(e))
|
|
raise kopf.TemporaryError(f"Error found in DevOpsProject spec: {e}")
|
|
|
|
# Create resource manager and handle the project
|
|
resource_manager = DevOpsProjectResourceManager(namespace, logger)
|
|
# ... implementation details
|
|
```
|
|
|
|
### **📊 Event Flow**
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────────────┐
|
|
│ EVENT FLOW │
|
|
│ │
|
|
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
|
|
│ │ User │ │ RabbitMQ │ │ Operator │ │ Kubernetes │ │
|
|
│ │ Action │ │ Message │ │ Controller│ │ Resources │ │
|
|
│ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │
|
|
│ │ │ │ │ │
|
|
│ │ 1. Create │ │ │ │
|
|
│ │ Project │ │ │ │
|
|
│ │───────────────▶│ │ │ │
|
|
│ │ │ 2. DevOpsInit │ │ │
|
|
│ │ │ Event │ │ │
|
|
│ │ │───────────────▶│ │ │
|
|
│ │ │ │ 3. Create CR │ │
|
|
│ │ │ │───────────────▶│ │
|
|
│ │ │ │ │ 4. CR Created │
|
|
│ │ │ │ │◀───────────────│
|
|
│ │ │ │ 5. Reconcile │ │
|
|
│ │ │ │◀───────────────│ │
|
|
│ │ │ │ 6. Create │ │
|
|
│ │ │ │ Resources │ │
|
|
│ │ │ │───────────────▶│ │
|
|
│ │ │ │ │ 7. Resources │
|
|
│ │ │ │ │ Ready │
|
|
│ │ │ │ │◀───────────────│
|
|
│ │ │ │ 8. Update │ │
|
|
│ │ │ │ Status │ │
|
|
│ │ │ │◀───────────────│ │
|
|
└─────────────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## 🎯 **Understanding the Relationship**
|
|
|
|
### **📊 Reconciler vs DevOps Repo**
|
|
|
|
| Component | Purpose | Location | Responsibility |
|
|
|-----------|---------|----------|----------------|
|
|
| **freeleaps-devops-reconciler** | **Kubernetes Operator** | `freeleaps-ops/freeleaps-devops-reconciler/` | Watches CRs, creates K8s resources |
|
|
| **freeleaps.com Platform** | **Business Logic** | `freeleaps-service-hub/` | User interface, project management |
|
|
| **RabbitMQ** | **Message Queue** | Infrastructure | Event communication |
|
|
| **ArgoCD** | **GitOps** | `freeleaps-ops/cluster/manifests/` | Application deployment |
|
|
| **Jenkins** | **CI/CD** | Infrastructure | Pipeline execution |
|
|
|
|
### **🔄 How They Work Together**
|
|
|
|
```yaml
|
|
# 1. User creates project on freeleaps.com
|
|
# 2. Platform sends DevOpsInit event to RabbitMQ
|
|
# 3. Reconciler receives event and creates DevOpsProject CR
|
|
# 4. Reconciler watches CR and creates:
|
|
# - ArgoCD Application
|
|
# - Jenkins Pipeline
|
|
# - Harbor Repository
|
|
# - Namespace and RBAC
|
|
# 5. ArgoCD deploys the application
|
|
# 6. Jenkins runs the pipeline
|
|
```
|
|
|
|
---
|
|
|
|
## 🔧 **Practical Examples**
|
|
|
|
### **1. Creating a DevOpsProject**
|
|
|
|
```bash
|
|
# Create a DevOpsProject CR
|
|
kubectl apply -f - <<EOF
|
|
apiVersion: freeleaps.com/v1alpha1
|
|
kind: DevOpsProject
|
|
metadata:
|
|
name: my-new-project
|
|
namespace: freeleaps-devops-system
|
|
spec:
|
|
projectName: "My New Project"
|
|
projectId: "my-new-project"
|
|
git:
|
|
url: "https://github.com/myorg/myproject"
|
|
branch: "main"
|
|
registry:
|
|
url: "https://harbor.example.com"
|
|
project: "myproject"
|
|
environments:
|
|
- name: "production"
|
|
domain: "myproject.example.com"
|
|
replicas: 2
|
|
EOF
|
|
|
|
# Check the CR status
|
|
kubectl get devopsprojects -n freeleaps-devops-system
|
|
kubectl describe devopsproject my-new-project -n freeleaps-devops-system
|
|
```
|
|
|
|
### **2. Monitoring the Operator**
|
|
|
|
```bash
|
|
# Check operator logs
|
|
kubectl logs -n freeleaps-devops-system deployment/freeleaps-devops-reconciler
|
|
|
|
# Check CR status
|
|
kubectl get devopsprojects --all-namespaces
|
|
kubectl get argosettings --all-namespaces
|
|
kubectl get jenkinssettings --all-namespaces
|
|
|
|
# Check created resources
|
|
kubectl get applications -n freeleaps-devops-system
|
|
kubectl get namespaces | grep my-new-project
|
|
```
|
|
|
|
### **3. Troubleshooting**
|
|
|
|
```bash
|
|
# Check CRD installation
|
|
kubectl get crd | grep freeleaps.com
|
|
|
|
# Check operator events
|
|
kubectl get events -n freeleaps-devops-system --sort-by='.lastTimestamp'
|
|
|
|
# Check resource creation
|
|
kubectl get all -n my-new-project
|
|
kubectl describe devopsproject my-new-project -n freeleaps-devops-system
|
|
```
|
|
|
|
---
|
|
|
|
## 🎯 **Best Practices**
|
|
|
|
### **1. CRD Design**
|
|
|
|
```yaml
|
|
# ✅ DO: Use clear, descriptive names
|
|
apiVersion: freeleaps.com/v1alpha1
|
|
kind: DevOpsProject # Clear, descriptive
|
|
|
|
# ❌ DON'T: Use generic names
|
|
kind: Project # Too generic
|
|
```
|
|
|
|
### **2. Validation**
|
|
|
|
```yaml
|
|
# ✅ DO: Include validation in CRD
|
|
spec:
|
|
openAPIV3Schema:
|
|
type: object
|
|
required: ['spec']
|
|
properties:
|
|
spec:
|
|
type: object
|
|
required: ['projectName', 'projectId']
|
|
properties:
|
|
projectId:
|
|
pattern: "^[a-z0-9]([a-z0-9-]*[a-z0-9])?$"
|
|
```
|
|
|
|
### **3. Status Management**
|
|
|
|
```yaml
|
|
# ✅ DO: Include status in CR
|
|
status:
|
|
conditions:
|
|
- type: Ready
|
|
status: "True"
|
|
reason: "ReconciliationSucceeded"
|
|
message: "All resources created successfully"
|
|
- type: Synced
|
|
status: "True"
|
|
reason: "ReconciliationSucceeded"
|
|
message: "Spec has been reconciled"
|
|
```
|
|
|
|
### **4. Error Handling**
|
|
|
|
```python
|
|
# ✅ DO: Proper error handling
|
|
try:
|
|
# Create resources
|
|
resource_manager.create_resources()
|
|
except Exception as e:
|
|
# Update status with error
|
|
devops_proj.update_status({
|
|
'status': 'Error',
|
|
'message': str(e)
|
|
})
|
|
raise kopf.TemporaryError(f"Failed to create resources: {e}")
|
|
```
|
|
|
|
---
|
|
|
|
## 🔍 **Debugging Your Operator**
|
|
|
|
### **1. Check Operator Status**
|
|
|
|
```bash
|
|
# Check if operator is running
|
|
kubectl get pods -n freeleaps-devops-system -l app=freeleaps-devops-reconciler
|
|
|
|
# Check operator logs
|
|
kubectl logs -n freeleaps-devops-system deployment/freeleaps-devops-reconciler -f
|
|
|
|
# Check CRD installation
|
|
kubectl get crd devopsprojects.freeleaps.com
|
|
```
|
|
|
|
### **2. Check CR Status**
|
|
|
|
```bash
|
|
# Check CR status
|
|
kubectl get devopsprojects --all-namespaces -o wide
|
|
|
|
# Check CR events
|
|
kubectl describe devopsproject <project-name> -n <namespace>
|
|
|
|
# Check CR YAML
|
|
kubectl get devopsproject <project-name> -n <namespace> -o yaml
|
|
```
|
|
|
|
### **3. Check Created Resources**
|
|
|
|
```bash
|
|
# Check what resources were created
|
|
kubectl get all -n <project-namespace>
|
|
|
|
# Check ArgoCD applications
|
|
kubectl get applications -n freeleaps-devops-system
|
|
|
|
# Check Jenkins pipelines
|
|
kubectl get jenkinssettings --all-namespaces
|
|
```
|
|
|
|
---
|
|
|
|
## 📚 **Next Steps**
|
|
|
|
### **1. Learn More About Operators**
|
|
- [Kubernetes Operators](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/)
|
|
- [Kopf Framework](https://kopf.readthedocs.io/) (what your reconciler uses)
|
|
- [Operator SDK](https://sdk.operatorframework.io/)
|
|
|
|
### **2. Understand Your Architecture**
|
|
- Study your `freeleaps-devops-reconciler` code
|
|
- Understand the event flow from RabbitMQ
|
|
- Learn how CRs trigger resource creation
|
|
|
|
### **3. Extend Your Operator**
|
|
- Add new custom resources
|
|
- Implement new controllers
|
|
- Add validation and error handling
|
|
|
|
---
|
|
|
|
**Last Updated**: September 3, 2025
|
|
**Version**: 1.0
|
|
**Maintainer**: Infrastructure Team
|