28 KiB
28 KiB
Custom Resources & Operators Guide
🎯 Overview
This guide explains Custom Resources (CRs), Custom Resource Definitions (CRDs), Kubernetes Operators, and how your freeleaps-devops-reconciler works as an operator to manage your DevOps infrastructure.
📊 What Are Custom Resources?
🔄 CR vs CRD Relationship
┌─────────────────────────────────────────────────────────────────────────────┐
│ CRD vs CR RELATIONSHIP │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ CUSTOM RESOURCE DEFINITION (CRD) │ │ │
│ │ │ │ │
│ │ apiVersion: apiextensions.k8s.io/v1 │ │ │
│ │ kind: CustomResourceDefinition │ │ │
│ │ metadata: │ │ │
│ │ name: devopsprojects.freeleaps.com │ │ │
│ │ spec: │ │ │
│ │ group: freeleaps.com │ │ │
│ │ names: │ │ │
│ │ kind: DevOpsProject │ │ │
│ │ plural: devopsprojects │ │ │
│ │ scope: Namespaced │ │ │
│ │ versions: │ │ │
│ │ - name: v1alpha1 │ │ │
│ │ schema: │ │ │
│ │ # Schema definition... │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │ │
│ │ │ │
│ ▼ │ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │ │
│ │ CUSTOM RESOURCE (CR) │ │ │
│ │ │ │ │
│ │ apiVersion: freeleaps.com/v1alpha1 │ │ │
│ │ kind: DevOpsProject │ │ │
│ │ metadata: │ │ │
│ │ name: my-project │ │ │
│ │ namespace: freeleaps-devops-system │ │ │
│ │ spec: │ │ │
│ │ projectName: "My Awesome Project" │ │ │
│ │ projectId: "my-awesome-project" │ │ │
│ │ git: │ │ │
│ │ url: "https://github.com/myorg/myproject" │ │ │
│ │ branch: "main" │ │ │
│ │ registry: │ │ │
│ │ url: "https://harbor.example.com" │ │ │
│ │ project: "myproject" │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │ │
└─────────────────────────────────────────────────────────────────────────────┘
🎯 Why Custom Resources?
# Instead of managing multiple resources manually:
# - Namespace
# - ServiceAccount
# - Role/RoleBinding
# - ConfigMap
# - Secret
# - Deployment
# - Service
# - Ingress
# You can create ONE custom resource:
apiVersion: freeleaps.com/v1alpha1
kind: DevOpsProject
metadata:
name: my-project
spec:
projectName: "My Project"
projectId: "my-project"
git:
url: "https://github.com/myorg/myproject"
branch: "main"
registry:
url: "https://harbor.example.com"
project: "myproject"
🏭 Your DevOps Reconciler Architecture
📊 Reconciler vs DevOps Repo Relationship
┌─────────────────────────────────────────────────────────────────────────────┐
│ RECONCILER + DEVOPS ARCHITECTURE │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ FRELEAPS.COM PLATFORM │ │ │
│ │ │ │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
│ │ │ User │ │ Project │ │ Git │ │ Registry │ │ │
│ │ │ Creates │ │ Manager │ │ Webhook │ │ Manager │ │ │
│ │ │ Project │ │ Creates │ │ Triggers │ │ Creates │ │ │
│ │ │ │ │ DevOps │ │ Event │ │ Repo │ │ │
│ │ │ │ │ Project │ │ │ │ │ │ │
│ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │ │
│ │ │ │
│ ▼ │ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │ │
│ │ RABBITMQ MESSAGE QUEUE │ │ │
│ │ │ │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
│ │ │DevOpsInit │ │DevOpsReconcile│ │DevOpsDeploy │ │DevOpsDelete │ │ │
│ │ │Event │ │Event │ │Event │ │Event │ │ │
│ │ │ │ │ │ │ │ │ │ │ │
│ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │ │
│ │ │ │
│ ▼ │ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │ │
│ │ FRELEAPS-DEVOPS-RECONCILER (OPERATOR) │ │ │
│ │ │ │ │
│ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │
│ │ │ CONTROLLERS │ │ │ │
│ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────┐ │ │ │ │
│ │ │ │DevOpsProject│ │ArgoSettings │ │Jenkins │ │... │ │ │ │ │
│ │ │ │Controller │ │Controller │ │Settings │ │ │ │ │ │ │
│ │ │ │ │ │ │ │Controller │ │ │ │ │ │ │
│ │ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────┘ │ │ │ │
│ │ └─────────────────────────────────────────────────────────────┘ │ │ │
│ │ │ │ │ │
│ │ ▼ │ │ │
│ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │
│ │ │ CUSTOM RESOURCES │ │ │ │
│ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────┐ │ │ │ │
│ │ │ │DevOpsProject│ │ArgoSettings │ │Jenkins │ │... │ │ │ │ │
│ │ │ │CR │ │CR │ │Settings │ │ │ │ │ │ │
│ │ │ │ │ │ │ │CR │ │ │ │ │ │ │
│ │ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────┘ │ │ │ │
│ │ └─────────────────────────────────────────────────────────────┘ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │ │
│ │ │ │
│ ▼ │ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │ │
│ │ KUBERNETES RESOURCES │ │ │
│ │ │ │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
│ │ │ArgoCD │ │Jenkins │ │Harbor │ │Namespaces │ │ │
│ │ │Applications │ │Pipelines │ │Repositories │ │Services │ │ │
│ │ │Projects │ │Jobs │ │Credentials │ │Deployments │ │ │
│ │ │ │ │ │ │ │ │Ingresses │ │ │
│ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
🔧 Your Custom Resources
1. DevOpsProject CRD
# 🏭 ACTUAL CRD FROM YOUR CODEBASE
# freeleaps-devops-reconciler/deploy/crds.yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: devopsprojects.freeleaps.com
spec:
group: freeleaps.com
scope: Namespaced
names:
kind: DevOpsProject
plural: devopsprojects
shortNames: [dop, dops]
versions:
- name: v1alpha1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
required: ['spec']
properties:
spec:
type: object
required:
- projectName
- projectId
- git
- registry
- environments
properties:
projectName:
type: string
description: "Human readable project name"
projectId:
type: string
description: "Unique project identifier"
pattern: "^[a-z0-9]([a-z0-9-]*[a-z0-9])?$"
git:
type: object
required: [url, branch]
properties:
url:
type: string
description: "Git repository URL"
branch:
type: string
description: "Default git branch"
default: "main"
registry:
type: object
required: [url, project]
properties:
url:
type: string
description: "Container registry URL"
project:
type: string
description: "Registry project name"
2. DevOpsProject CR Example
# 🏭 ACTUAL CR EXAMPLE
apiVersion: freeleaps.com/v1alpha1
kind: DevOpsProject
metadata:
name: magicleaps-frontend
namespace: freeleaps-devops-system
labels:
app.kubernetes.io/name: magicleaps-frontend
app.kubernetes.io/instance: magicleaps-frontend
spec:
projectName: "Magicleaps Frontend"
projectId: "magicleaps-frontend"
git:
url: "https://github.com/freeleaps/magicleaps-frontend"
branch: "main"
credentialsRef:
name: "github-credentials"
namespace: "freeleaps-devops-system"
registry:
url: "https://harbor.freeleaps.mathmast.com"
project: "magicleaps"
credentialsRef:
name: "harbor-credentials"
namespace: "freeleaps-devops-system"
environments:
- name: "production"
domain: "magicleaps.mathmast.com"
replicas: 3
- name: "alpha"
domain: "alpha.magicleaps.mathmast.com"
replicas: 1
3. Other Custom Resources
# 🏭 YOUR COMPLETE CRD SET
# From freeleaps-devops-reconciler/docs/design/one-click-deployment.md
# DevOpsProject - Main project configuration
# ArgoSettings - ArgoCD settings for the DevOpsProject
# JenkinsSettings - Jenkins settings and generated pipelines
# ContainerRegistry - Container registry information
# ContainerImage - Every image manufactured by Jenkins pipeline
# DeploymentRecord - Track deployment records
# GitCredential - Git repository credentials
# IngressResource - Ingress configuration
🤖 How Your Operator Works
🔄 Reconciliation Loop
# 🏭 ACTUAL CODE FROM YOUR RECONCILER
# freeleaps-devops-reconciler/reconciler/controllers/devops_projects/controller.py
@kopf.on.create(group=consts.GROUP, version=consts.VERSION, kind=consts.DEVOPS_PROJECT_KIND)
def on_devops_proj_created(name: str, namespace: Optional[str], body: Body, logger: Logger, **kwargs):
logger.info(f"Newly created DevOpsProject resource and named {name} in namespace {namespace}, start to reconciling...")
devops_proj = DevOpsProject(body)
try:
devops_proj.parse_spec()
devops_proj.get_spec().validate(logger)
except SpecError as e:
devops_proj.update_status({
'devopsProject': {
'status': DevOpsProjectDiagStatus.INVALID.value,
'synced': False,
'ready': False,
'lastProbeTime': isotime(),
}
})
devops_proj.error(action='CreateDevOpsProject',
reason='InvalidSpecArgument', msg=str(e))
raise kopf.TemporaryError(f"Error found in DevOpsProject spec: {e}")
# Create resource manager and handle the project
resource_manager = DevOpsProjectResourceManager(namespace, logger)
# ... implementation details
📊 Event Flow
┌─────────────────────────────────────────────────────────────────────────────┐
│ EVENT FLOW │
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ User │ │ RabbitMQ │ │ Operator │ │ Kubernetes │ │
│ │ Action │ │ Message │ │ Controller│ │ Resources │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │ │ │ │ │
│ │ 1. Create │ │ │ │
│ │ Project │ │ │ │
│ │───────────────▶│ │ │ │
│ │ │ 2. DevOpsInit │ │ │
│ │ │ Event │ │ │
│ │ │───────────────▶│ │ │
│ │ │ │ 3. Create CR │ │
│ │ │ │───────────────▶│ │
│ │ │ │ │ 4. CR Created │
│ │ │ │ │◀───────────────│
│ │ │ │ 5. Reconcile │ │
│ │ │ │◀───────────────│ │
│ │ │ │ 6. Create │ │
│ │ │ │ Resources │ │
│ │ │ │───────────────▶│ │
│ │ │ │ │ 7. Resources │
│ │ │ │ │ Ready │
│ │ │ │ │◀───────────────│
│ │ │ │ 8. Update │ │
│ │ │ │ Status │ │
│ │ │ │◀───────────────│ │
└─────────────────────────────────────────────────────────────────────────────┘
🎯 Understanding the Relationship
📊 Reconciler vs DevOps Repo
| Component | Purpose | Location | Responsibility |
|---|---|---|---|
| freeleaps-devops-reconciler | Kubernetes Operator | freeleaps-ops/freeleaps-devops-reconciler/ |
Watches CRs, creates K8s resources |
| freeleaps.com Platform | Business Logic | freeleaps-service-hub/ |
User interface, project management |
| RabbitMQ | Message Queue | Infrastructure | Event communication |
| ArgoCD | GitOps | freeleaps-ops/cluster/manifests/ |
Application deployment |
| Jenkins | CI/CD | Infrastructure | Pipeline execution |
🔄 How They Work Together
# 1. User creates project on freeleaps.com
# 2. Platform sends DevOpsInit event to RabbitMQ
# 3. Reconciler receives event and creates DevOpsProject CR
# 4. Reconciler watches CR and creates:
# - ArgoCD Application
# - Jenkins Pipeline
# - Harbor Repository
# - Namespace and RBAC
# 5. ArgoCD deploys the application
# 6. Jenkins runs the pipeline
🔧 Practical Examples
1. Creating a DevOpsProject
# Create a DevOpsProject CR
kubectl apply -f - <<EOF
apiVersion: freeleaps.com/v1alpha1
kind: DevOpsProject
metadata:
name: my-new-project
namespace: freeleaps-devops-system
spec:
projectName: "My New Project"
projectId: "my-new-project"
git:
url: "https://github.com/myorg/myproject"
branch: "main"
registry:
url: "https://harbor.example.com"
project: "myproject"
environments:
- name: "production"
domain: "myproject.example.com"
replicas: 2
EOF
# Check the CR status
kubectl get devopsprojects -n freeleaps-devops-system
kubectl describe devopsproject my-new-project -n freeleaps-devops-system
2. Monitoring the Operator
# Check operator logs
kubectl logs -n freeleaps-devops-system deployment/freeleaps-devops-reconciler
# Check CR status
kubectl get devopsprojects --all-namespaces
kubectl get argosettings --all-namespaces
kubectl get jenkinssettings --all-namespaces
# Check created resources
kubectl get applications -n freeleaps-devops-system
kubectl get namespaces | grep my-new-project
3. Troubleshooting
# Check CRD installation
kubectl get crd | grep freeleaps.com
# Check operator events
kubectl get events -n freeleaps-devops-system --sort-by='.lastTimestamp'
# Check resource creation
kubectl get all -n my-new-project
kubectl describe devopsproject my-new-project -n freeleaps-devops-system
🎯 Best Practices
1. CRD Design
# ✅ DO: Use clear, descriptive names
apiVersion: freeleaps.com/v1alpha1
kind: DevOpsProject # Clear, descriptive
# ❌ DON'T: Use generic names
kind: Project # Too generic
2. Validation
# ✅ DO: Include validation in CRD
spec:
openAPIV3Schema:
type: object
required: ['spec']
properties:
spec:
type: object
required: ['projectName', 'projectId']
properties:
projectId:
pattern: "^[a-z0-9]([a-z0-9-]*[a-z0-9])?$"
3. Status Management
# ✅ DO: Include status in CR
status:
conditions:
- type: Ready
status: "True"
reason: "ReconciliationSucceeded"
message: "All resources created successfully"
- type: Synced
status: "True"
reason: "ReconciliationSucceeded"
message: "Spec has been reconciled"
4. Error Handling
# ✅ DO: Proper error handling
try:
# Create resources
resource_manager.create_resources()
except Exception as e:
# Update status with error
devops_proj.update_status({
'status': 'Error',
'message': str(e)
})
raise kopf.TemporaryError(f"Failed to create resources: {e}")
🔍 Debugging Your Operator
1. Check Operator Status
# Check if operator is running
kubectl get pods -n freeleaps-devops-system -l app=freeleaps-devops-reconciler
# Check operator logs
kubectl logs -n freeleaps-devops-system deployment/freeleaps-devops-reconciler -f
# Check CRD installation
kubectl get crd devopsprojects.freeleaps.com
2. Check CR Status
# Check CR status
kubectl get devopsprojects --all-namespaces -o wide
# Check CR events
kubectl describe devopsproject <project-name> -n <namespace>
# Check CR YAML
kubectl get devopsproject <project-name> -n <namespace> -o yaml
3. Check Created Resources
# Check what resources were created
kubectl get all -n <project-namespace>
# Check ArgoCD applications
kubectl get applications -n freeleaps-devops-system
# Check Jenkins pipelines
kubectl get jenkinssettings --all-namespaces
📚 Next Steps
1. Learn More About Operators
- Kubernetes Operators
- Kopf Framework (what your reconciler uses)
- Operator SDK
2. Understand Your Architecture
- Study your
freeleaps-devops-reconcilercode - Understand the event flow from RabbitMQ
- Learn how CRs trigger resource creation
3. Extend Your Operator
- Add new custom resources
- Implement new controllers
- Add validation and error handling
Last Updated: September 3, 2025 Version: 1.0 Maintainer: Infrastructure Team