add source code and readme

manifests/infrastructure/elasticsearch/README.md

@@ -0,0 +1,261 @@
+# Elasticsearch Infrastructure
+
+This directory contains the Elasticsearch setup using ECK (Elastic Cloud on Kubernetes) operator for full-text search on the Kubernetes cluster.
+
+## Architecture
+
+- **ECK Operator**: Production-grade Elasticsearch deployment on Kubernetes
+- **Single-node cluster**: Optimized for your 2-node cluster (can be scaled later)
+- **Security enabled**: X-Pack security with custom role and user for Mastodon
+- **Longhorn storage**: Distributed storage with 2-replica redundancy
+- **Self-signed certificates**: Internal cluster communication with TLS
+
+## Components
+
+### **Core Components**
+- `namespace.yaml`: Elasticsearch system namespace
+- `repository.yaml`: Elastic Helm repository
+- `operator.yaml`: ECK operator deployment
+- Uses existing `longhorn-retain` storage class with backup labels on PVCs
+- `cluster.yaml`: Elasticsearch and Kibana cluster configuration
+
+### **Security Components**
+- `secret.yaml`: SOPS-encrypted credentials for Elasticsearch admin and Mastodon user
+- `security-setup.yaml`: Job to create Mastodon role and user after cluster deployment
+
+### **Monitoring Components**
+- `monitoring.yaml`: ServiceMonitor for OpenObserve integration + optional Kibana ingress
+- Built-in metrics: Elasticsearch Prometheus exporter
+
+## Services Created
+
+ECK automatically creates these services:
+
+- `elasticsearch-es-http`: HTTPS API access (port 9200)
+- `elasticsearch-es-transport`: Internal cluster transport (port 9300)
+- `kibana-kb-http`: Kibana web UI (port 5601) - optional management interface
+
+## Connection Information
+
+### For Applications (Mastodon)
+
+Applications should connect using these connection parameters:
+
+**Elasticsearch Connection:**
+```yaml
+host: elasticsearch-es-http.elasticsearch-system.svc.cluster.local
+port: 9200
+scheme: https  # ECK uses HTTPS with self-signed certificates
+user: mastodon
+password: <password from elasticsearch-credentials secret>
+```
+
+### Getting Credentials
+
+The Elasticsearch credentials are stored in SOPS-encrypted secrets:
+
+```bash
+# Get the admin password (auto-generated by ECK)
+kubectl get secret elasticsearch-es-elastic-user -n elasticsearch-system -o jsonpath="{.data.elastic}" | base64 -d
+
+# Get the Mastodon user password (set during security setup)
+kubectl get secret elasticsearch-credentials -n elasticsearch-system -o jsonpath="{.data.password}" | base64 -d
+```
+
+## Deployment Steps
+
+### 1. Encrypt Secrets
+Before deploying, encrypt the secrets with SOPS:
+
+```bash
+# Edit and encrypt the Elasticsearch credentials
+sops manifests/infrastructure/elasticsearch/secret.yaml
+
+# Edit and encrypt the Mastodon Elasticsearch credentials  
+sops manifests/applications/mastodon/elasticsearch-secret.yaml
+```
+
+### 2. Deploy Infrastructure
+The infrastructure will be deployed automatically by Flux when you commit:
+
+```bash
+git add manifests/infrastructure/elasticsearch/
+git add manifests/cluster/flux-system/elasticsearch.yaml
+git add manifests/cluster/flux-system/kustomization.yaml
+git commit -m "Add Elasticsearch infrastructure for Mastodon search"
+git push
+```
+
+### 3. Wait for Deployment
+```bash
+# Monitor ECK operator deployment
+kubectl get pods -n elasticsearch-system -w
+
+# Monitor Elasticsearch cluster startup
+kubectl get elasticsearch -n elasticsearch-system -w
+
+# Check cluster health
+kubectl get elasticsearch elasticsearch -n elasticsearch-system -o yaml
+```
+
+### 4. Verify Security Setup
+```bash
+# Check if security setup job completed successfully
+kubectl get jobs -n elasticsearch-system
+
+# Verify Mastodon user was created
+kubectl logs -n elasticsearch-system job/elasticsearch-security-setup
+```
+
+### 5. Update Mastodon
+After Elasticsearch is running, deploy the updated Mastodon configuration:
+
+```bash
+git add manifests/applications/mastodon/
+git commit -m "Enable Elasticsearch in Mastodon"
+git push
+```
+
+### 6. Populate Search Indices
+Once Mastodon is running with Elasticsearch enabled, populate the search indices:
+
+```bash
+# Get a Mastodon web pod
+MASTODON_POD=$(kubectl get pods -n mastodon-application -l app.kubernetes.io/component=web -o jsonpath='{.items[0].metadata.name}')
+
+# Run the search deployment command
+kubectl exec -n mastodon-application $MASTODON_POD -- bin/tootctl search deploy
+```
+
+## Configuration Details
+
+### Elasticsearch Configuration
+- **Version**: 7.17.27 (latest 7.x compatible with Mastodon)
+- **Preset**: `single_node_cluster` (optimized for single-node deployment)
+- **Memory**: 2GB heap size (50% of 4GB container limit)
+- **Storage**: 50GB persistent volume with existing `longhorn-retain` storage class
+- **Security**: X-Pack security enabled with custom roles
+
+### Security Configuration
+Following the [Mastodon Elasticsearch documentation](https://docs.joinmastodon.org/admin/elasticsearch/), the setup includes:
+
+- **Custom Role**: `mastodon_full_access` with minimal required permissions
+- **Dedicated User**: `mastodon` with the custom role
+- **TLS Encryption**: All connections use HTTPS with self-signed certificates
+
+### Performance Configuration
+- **JVM Settings**: Optimized for your cluster's resource constraints
+- **Discovery**: Single-node discovery (can be changed for multi-node scaling)
+- **Memory**: Conservative settings for 2-node cluster compatibility
+- **Storage**: Optimized for SSD performance with proper disk watermarks
+
+## Mastodon Integration
+
+### Search Features Enabled
+Once configured, Mastodon will provide full-text search for:
+
+- Public statuses from accounts that opted into search results
+- User's own statuses
+- User's mentions, favourites, and bookmarks
+- Account information (display names, usernames, bios)
+
+### Search Index Deployment
+The `tootctl search deploy` command will create these indices:
+
+- `accounts_index`: User accounts and profiles
+- `statuses_index`: User's own statuses, mentions, favourites, bookmarks
+- `public_statuses_index`: Public searchable content
+- `tags_index`: Hashtag search
+
+## Monitoring Integration
+
+### OpenObserve Metrics
+Elasticsearch metrics are automatically collected and sent to OpenObserve:
+
+- **Cluster Health**: Node status, cluster state, allocation
+- **Performance**: Query latency, indexing rate, search performance
+- **Storage**: Disk usage, index sizes, shard distribution
+- **JVM**: Memory usage, garbage collection, heap statistics
+
+### Kibana Management UI
+Optional Kibana web interface available at `https://kibana.keyboardvagabond.com` for:
+
+- Index management and monitoring
+- Query development and testing
+- Cluster configuration and troubleshooting
+- Visual dashboards for Elasticsearch data
+
+## Scaling Considerations
+
+### Current Setup
+- **Single-node cluster**: Optimized for current 2-node Kubernetes cluster
+- **50GB storage**: Sufficient for small-to-medium Mastodon instances
+- **2GB heap**: Conservative memory allocation
+
+### Future Scaling
+When adding more Kubernetes nodes:
+
+1. Update `discovery.type` from `single-node` to `zen` in cluster configuration
+2. Increase `nodeSets.count` to 2 or 3 for high availability
+3. Change `ES_PRESET` to `small_cluster` in Mastodon configuration
+4. Consider increasing storage and memory allocations
+
+## Troubleshooting
+
+### Common Issues
+
+**Elasticsearch pods pending:**
+- Check storage class and PVC creation
+- Verify Longhorn is healthy and has available space
+
+**Security setup job failing:**
+- Check Elasticsearch cluster health
+- Verify admin credentials are available
+- Review job logs for API errors
+
+**Mastodon search not working:**
+- Verify Elasticsearch credentials in Mastodon secret
+- Check network connectivity between namespaces
+- Ensure search indices are created with `tootctl search deploy`
+
+### Useful Commands
+
+```bash
+# Check Elasticsearch cluster status
+kubectl get elasticsearch -n elasticsearch-system
+
+# View Elasticsearch logs
+kubectl logs -n elasticsearch-system -l elasticsearch.k8s.elastic.co/cluster-name=elasticsearch
+
+# Check security setup
+kubectl describe job elasticsearch-security-setup -n elasticsearch-system
+
+# Test connectivity from Mastodon
+kubectl exec -n mastodon-application deployment/mastodon-web -- curl -k https://elasticsearch-es-http.elasticsearch-system.svc.cluster.local:9200/_cluster/health
+```
+
+## Backup Integration
+
+### S3 Backup Strategy
+- **Longhorn Integration**: Elasticsearch volumes are automatically backed up to Backblaze B2
+- **Volume Labels**: `backup.longhorn.io/enable: "true"` enables automatic S3 backup
+- **Backup Frequency**: Follows existing Longhorn backup schedule
+
+### Index Backup
+For additional protection, consider periodic index snapshots:
+
+```bash
+# Create snapshot repository (one-time setup)
+curl -k -u "mastodon:$ES_PASSWORD" -X PUT "https://elasticsearch-es-http.elasticsearch-system.svc.cluster.local:9200/_snapshot/s3_repository" -H 'Content-Type: application/json' -d'
+{
+  "type": "s3",
+  "settings": {
+    "bucket": "longhorn-backup-bucket",
+    "region": "eu-central-003",
+    "endpoint": "<REPLACE_WITH_S3_ENDPOINT>"
+  }
+}'
+
+# Create manual snapshot
+curl -k -u "mastodon:$ES_PASSWORD" -X PUT "https://elasticsearch-es-http.elasticsearch-system.svc.cluster.local:9200/_snapshot/s3_repository/snapshot_1"
+``` 

manifests/infrastructure/elasticsearch/cluster.yaml

@@ -0,0 +1,149 @@
+---
+apiVersion: elasticsearch.k8s.elastic.co/v1
+kind: Elasticsearch
+metadata:
+  name: elasticsearch
+  namespace: elasticsearch-system
+  labels:
+    app: elasticsearch
+    backup.longhorn.io/enable: "true"  # Enable Longhorn S3 backup
+spec:
+  version: 7.17.27  # Latest 7.x version compatible with Mastodon
+  
+  # Single-node cluster (can be scaled later)
+  nodeSets:
+  - name: default
+    count: 1
+    config:
+      # Node configuration
+      node.store.allow_mmap: false  # Required for containers
+      
+      # Performance optimizations for 2-node cluster (similar to PostgreSQL)
+      cluster.routing.allocation.disk.threshold_enabled: true
+      cluster.routing.allocation.disk.watermark.low: "85%"
+      cluster.routing.allocation.disk.watermark.high: "90%"
+      cluster.routing.allocation.disk.watermark.flood_stage: "95%"
+      
+      # Memory and performance settings
+      indices.memory.index_buffer_size: "20%"
+      indices.memory.min_index_buffer_size: "48mb"
+      indices.fielddata.cache.size: "30%"
+      indices.queries.cache.size: "20%"
+      
+      # ECK manages discovery configuration automatically for single-node clusters
+      
+      # Security settings - ECK manages TLS automatically
+      xpack.security.enabled: true
+    
+    # Pod template for Elasticsearch nodes
+    podTemplate:
+      metadata:
+        labels:
+          app: elasticsearch
+      spec:
+        # Node selection and affinity - Prefer n2 but allow n1 if needed
+        nodeSelector: {}
+        tolerations: []
+        affinity:
+          nodeAffinity:
+            # PREFERRED: Prefer n2 for optimal distribution, but allow n1 if needed
+            preferredDuringSchedulingIgnoredDuringExecution:
+            - weight: 100
+              preference:
+                matchExpressions:
+                - key: kubernetes.io/hostname
+                  operator: In
+                  values: ["n2"]
+        
+        # Resource configuration - Optimized for resource-constrained environment
+        containers:
+        - name: elasticsearch
+          resources:
+            requests:
+              cpu: 500m       # 0.5 CPU core 
+              memory: 2Gi     # 2GB RAM (increased from 1Gi)
+            limits:
+              cpu: 1000m      # Max 1 CPU core
+              memory: 4Gi     # Max 4GB RAM (increased from 2Gi)
+          env:
+          # JVM heap size - should be 50% of container memory limit  
+          - name: ES_JAVA_OPTS
+            value: "-Xms2g -Xmx2g"
+          
+          # Security context - ECK manages this automatically
+          securityContext: {}
+    
+    # Volume claim templates
+    volumeClaimTemplates:
+    - metadata:
+        name: elasticsearch-data
+        labels:
+          backup.longhorn.io/enable: "true"  # Enable S3 backup
+      spec:
+        accessModes:
+        - ReadWriteOnce
+        resources:
+          requests:
+            storage: 50Gi
+        storageClassName: longhorn-retain
+
+  # HTTP configuration
+  http:
+    service:
+      spec:
+        type: ClusterIP
+        selector:
+          elasticsearch.k8s.elastic.co/cluster-name: "elasticsearch"
+    tls:
+      selfSignedCertificate:
+        disabled: true  # Disable TLS for internal Kubernetes communication
+
+  # Transport configuration
+  transport:
+    service:
+      spec:
+        type: ClusterIP
+
+---
+# Kibana deployment for optional web UI management
+apiVersion: kibana.k8s.elastic.co/v1
+kind: Kibana
+metadata:
+  name: kibana
+  namespace: elasticsearch-system
+spec:
+  version: 7.17.27
+  count: 1
+  elasticsearchRef:
+    name: elasticsearch
+  
+  config:
+    server.publicBaseUrl: "https://kibana.keyboardvagabond.com"
+    
+  podTemplate:
+    metadata:
+      labels:
+        app: kibana
+    spec:
+      containers:
+      - name: kibana
+        resources:
+          requests:
+            cpu: 50m        # Reduced from 200m - actual usage ~26m
+            memory: 384Mi   # Reduced from 1Gi - actual usage ~274MB
+          limits:
+            cpu: 400m       # Reduced from 1000m but adequate for log analysis
+            memory: 768Mi   # Reduced from 2Gi but adequate for dashboards
+        securityContext: {}
+
+  http:
+    service:
+      metadata:
+        annotations:
+          tailscale.com/hostname: kibana
+      spec:
+        type: LoadBalancer
+        loadBalancerClass: tailscale
+    tls:
+      selfSignedCertificate:
+        disabled: false 

manifests/infrastructure/elasticsearch/kustomization.yaml

@@ -0,0 +1,21 @@
+---
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+namespace: elasticsearch-system
+
+resources:
+- namespace.yaml
+- repository.yaml
+- operator.yaml
+- cluster.yaml
+- secret.yaml
+- security-setup.yaml
+- monitoring.yaml
+
+# Apply resources in order
+# 1. Namespace and repository first
+# 2. Storage class and operator
+# 3. Cluster configuration
+# 4. Security setup (job runs after cluster is ready)
+# 5. Monitoring and ingress 

manifests/infrastructure/elasticsearch/monitoring.yaml

@@ -0,0 +1,67 @@
+---
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: elasticsearch-metrics
+  namespace: elasticsearch-system
+  labels:
+    app: elasticsearch
+spec:
+  selector:
+    matchLabels:
+      elasticsearch.k8s.elastic.co/cluster-name: elasticsearch
+  endpoints:
+  - port: https
+    path: /_prometheus/metrics
+    scheme: https
+    tlsConfig:
+      insecureSkipVerify: true  # Use self-signed certs
+    basicAuth:
+      username:
+        name: elasticsearch-es-elastic-user
+        key: elastic
+      password:
+        name: elasticsearch-es-elastic-user  
+        key: elastic
+    interval: 30s
+    scrapeTimeout: 10s
+  namespaceSelector:
+    matchNames:
+    - elasticsearch-system
+
+---
+# Optional: Kibana ServiceMonitor if you want to monitor Kibana as well
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: kibana-metrics
+  namespace: elasticsearch-system
+  labels:
+    app: kibana
+spec:
+  selector:
+    matchLabels:
+      kibana.k8s.elastic.co/name: kibana
+  endpoints:
+  - port: https
+    path: /api/status
+    scheme: https
+    tlsConfig:
+      insecureSkipVerify: true
+    basicAuth:
+      username:
+        name: elasticsearch-es-elastic-user
+        key: elastic
+      password:
+        name: elasticsearch-es-elastic-user
+        key: elastic
+    interval: 60s
+    scrapeTimeout: 30s
+  namespaceSelector:
+    matchNames:
+    - elasticsearch-system
+
+---
+# Note: Kibana is exposed via Tailscale LoadBalancer service (configured in cluster.yaml)
+# No Ingress needed - the service type LoadBalancer with loadBalancerClass: tailscale
+# automatically creates a Tailscale proxy pod and exposes the service via MagicDNS 

manifests/infrastructure/elasticsearch/namespace.yaml

@@ -0,0 +1,8 @@
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: elasticsearch-system
+  labels:
+    name: elasticsearch-system
+    backup.longhorn.io/enable: "true"  # Enable Longhorn S3 backup 

manifests/infrastructure/elasticsearch/operator.yaml

@@ -0,0 +1,55 @@
+---
+apiVersion: helm.toolkit.fluxcd.io/v2
+kind: HelmRelease
+metadata:
+  name: eck-operator
+  namespace: elasticsearch-system
+spec:
+  interval: 5m
+  timeout: 10m
+  chart:
+    spec:
+      chart: eck-operator
+      version: "2.16.1"  # Latest stable version
+      sourceRef:
+        kind: HelmRepository
+        name: elastic
+        namespace: elasticsearch-system
+      interval: 1m
+  values:
+    # ECK Operator Configuration
+    installCRDs: true
+    
+    # Resource limits for operator - optimized based on actual usage
+    resources:
+      requests:
+        cpu: 25m      # Reduced from 100m - actual usage ~4m
+        memory: 128Mi # Reduced from 150Mi - actual usage ~81MB
+      limits:
+        cpu: 200m     # Reduced from 1000m but still adequate for operator tasks
+        memory: 256Mi # Reduced from 512Mi but still adequate
+    
+    # Node selection for operator
+    nodeSelector: {}
+    tolerations: []
+    
+    # Security configuration
+    podSecurityContext:
+      runAsNonRoot: true
+    
+    # Webhook configuration
+    webhook:
+      enabled: true
+    
+    # Metrics
+    metrics:
+      port: 0  # Disable metrics endpoint for now
+    
+    # Logging
+    config:
+      logVerbosity: 0
+      metricsPort: 0
+    
+    # Additional volumes/mounts if needed
+    extraVolumes: []
+    extraVolumeMounts: [] 

manifests/infrastructure/elasticsearch/repository.yaml

@@ -0,0 +1,9 @@
+---
+apiVersion: source.toolkit.fluxcd.io/v1
+kind: HelmRepository
+metadata:
+  name: elastic
+  namespace: elasticsearch-system
+spec:
+  interval: 24h
+  url: https://helm.elastic.co 

manifests/infrastructure/elasticsearch/secret.yaml

@@ -0,0 +1,45 @@
+apiVersion: v1
+kind: Secret
+metadata:
+    name: elasticsearch-credentials
+    namespace: elasticsearch-system
+type: Opaque
+stringData:
+    #ENC[AES256_GCM,data:xbndkZj3CeTZN5MphjUAxKiQbYIAAV0GuPmueWw7JwPk5fk6KpG/8FGrG00=,iv:0FV6SB6Ng+kaE66uVdDlx8Tv/3LAHCjuoWObi2mpUbU=,tag:1vLYGHl2WHvRVGz1bAqYFw==,type:comment]
+    #ENC[AES256_GCM,data:Jg3rWRjashFNg+0fEc7nELrCrCVTUOuCly2bYpMjiELrqxz7Xr5NzR4xiIByw/Ra9k6KC3AIliqprRq6zg==,iv:Iin+CpprebHEWq6JwmGYKdwraxuMIgJBODyLcL0/SGo=,tag:xzJgp/dyR7lfTlOHLySWHg==,type:comment]
+    username: ENC[AES256_GCM,data:PKlxhJfU4CY=,iv:9Bsw4V+yjWquFB4O9o3WxPMkAgOacsHrNf5DVNaU5hM=,tag:a9fyeD52Q/9amVeZ4U1Rzg==,type:str]
+    password: ENC[AES256_GCM,data:AsYI0SYTPCzxCxBfrk/aNSqKiBg+pXXxG0Ao0kshsO//WjKkCohBbSM54/oesjEylZk=,iv:skXOKX9ZshzJF3e+zJKGL67XT5rgTIfetUbobY/SSH0=,tag:08SrG9iAtGLzc/Ie9LK+/Q==,type:str]
+    #ENC[AES256_GCM,data:2r1sPMzdY0Pm00UNo+PD56tSm3p0SFzOclIfisaubHzG4xfDzffyO6fBGbqXJHvARkRzp+8ZWuaSWnQQae9O2EjyTlO0xt9U,iv:KXzBL1VFnj7cYXuhcPXSxS5LUYOGkUT301VLkyCPxsI=,tag:wv5XuHZMSV3FQqzMrTEQlg==,type:comment]
+    #ENC[AES256_GCM,data:V/09hOJMrROOeg9Jicj+PA1JowWmwabb5BsRvUcrJabcyJQ8Alm+QIyjK86zLVnz,iv:9qO//4Nf0Bb5a4VmFUZBx6QEP1dhCipHpv3GmKm7YkA=,tag:HYwPfqQwJTF8gGVoTUNi5Q==,type:comment]
+    admin-username: ENC[AES256_GCM,data:tLJw1egNQQ==,iv:7VvP+EdNIMB3dfIOa9xR+RYtUg+MJhJHrhux0Vy3BME=,tag:Av5j8jBG7vo4Si1oqphLAg==,type:str]
+    admin-password: ENC[AES256_GCM,data:2wOb7lAY+T92s/zYFr0ladWDFePyMZ/r,iv:CRK5FIbmG+SFtbPqvaUKi/W3HTAR+zn/C2DtU55J/7E=,tag:1TULM84wl8mkUU9FPg0Zkw==,type:str]
+sops:
+    lastmodified: "2025-11-30T09:38:26Z"
+    mac: ENC[AES256_GCM,data:eY+5GdSvqXhbK+5HTmru9ItqZ3ivBls+6twWswhd3CnYtem3D++SyXxZlGuV9C8RPoiIUddl8XDNJBB6F+wC9MmbvokigYP3GsqCem2V1pvLpP5B0bMMO4y8JeyRVmXkTVIkA+syBDgPz3D05GSA0n9BNxh303Dmvv0EtCJ7pbI=,iv:H1pT3DnQmjqp7Pp6KHTHdj5etAx08IO1i+mjpvoQLcE=,tag:6thUf1j7bgQEfBzifni1nA==,type:str]
+    pgp:
+        - created_at: "2025-11-27T09:39:43Z"
+          enc: |-
+            -----BEGIN PGP MESSAGE-----
+
+            hF4DZT3mpHTS/JgSAQdAXiRkqvjErdtK7Mx1NbAHLYiybYUmto2yThAGLvCpzHcw
+            8b8b3RO6b9WQwYdtn6Ld3ghcXBhR/eUu8RX5TZwDL3uw4+sinRWzBYeMU2llFnwb
+            1GgBCQIQbKSPq4uVXVgUPEAmISfla/qePymV8eABHa3rRwYwnVsj5fez6bFoLfOz
+            wJfSDSrRDUmZT/rTLvHi3GXTfnaOYbg0aScf3SCbxaMf2K4zGTyPXwQUnRFUn9KI
+            yXvR8SRAC0SG3g==
+            =KCYR
+            -----END PGP MESSAGE-----
+          fp: B120595CA9A643B051731B32E67FF350227BA4E8
+        - created_at: "2025-11-27T09:39:43Z"
+          enc: |-
+            -----BEGIN PGP MESSAGE-----
+
+            hF4DSXzd60P2RKISAQdAZGa0E49mmUHnjAStIf6zY0n5lQJ7Zr+DRZkd7cIP5V0w
+            +fWI4RcQ3rfzZljfP9stegszFwL7MMuRes0PeDxT+zk3HAvOnJIocBoM96P48Ckm
+            1GgBCQIQA4kzGLnFD/pPsofvMjDXP2G+bGrvxBRgHG/vRpsTCI6tiOEd3VeSR9qe
+            DtaudhgKbbAfWSj9cKHULRkxrQoLHjoeIlN4V/4tRxYp3Mxj4t5myaZqxUY1+Kmc
+            IaU4qoz4LQAZ0Q==
+            =0MwX
+            -----END PGP MESSAGE-----
+          fp: 4A8AADB4EBAB9AF88EF7062373CECE06CC80D40C
+    encrypted_regex: ^(data|stringData)$
+    version: 3.10.2

manifests/infrastructure/elasticsearch/security-setup.yaml

@@ -0,0 +1,88 @@
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: elasticsearch-security-setup
+  namespace: elasticsearch-system
+  annotations:
+    # Run this job after Elasticsearch is ready
+    "helm.sh/hook": post-install,post-upgrade
+    "helm.sh/hook-weight": "10"
+    "helm.sh/hook-delete-policy": before-hook-creation
+spec:
+  template:
+    metadata:
+      labels:
+        app: elasticsearch-security-setup
+    spec:
+      restartPolicy: Never
+      initContainers:
+      # Wait for Elasticsearch to be ready
+      - name: wait-for-elasticsearch
+        image: curlimages/curl:8.10.1
+        command:
+        - /bin/sh
+        - -c
+        - |
+          echo "Waiting for Elasticsearch to be ready..."
+          until curl -u "elastic:${ELASTIC_PASSWORD}" "http://elasticsearch-es-http:9200/_cluster/health?wait_for_status=yellow&timeout=300s"; do
+            echo "Elasticsearch not ready yet, sleeping..."
+            sleep 10
+          done
+          echo "Elasticsearch is ready!"
+        env:
+        - name: ELASTIC_PASSWORD
+          valueFrom:
+            secretKeyRef:
+              name: elasticsearch-es-elastic-user
+              key: elastic
+      containers:
+      - name: setup-security
+        image: curlimages/curl:8.10.1
+        command:
+        - /bin/sh
+        - -c
+        - |
+          echo "Setting up Elasticsearch security for Mastodon..."
+          
+          # Create mastodon_full_access role
+          echo "Creating mastodon_full_access role..."
+          curl -X POST -u "elastic:${ELASTIC_PASSWORD}" \
+            "http://elasticsearch-es-http:9200/_security/role/mastodon_full_access" \
+            -H 'Content-Type: application/json' \
+            -d '{
+              "cluster": ["monitor"],
+              "indices": [{
+                "names": ["*"],
+                "privileges": ["read", "monitor", "write", "manage"]
+              }]
+            }'
+          
+          echo "Role creation response: $?"
+          
+          # Create mastodon user
+          echo "Creating mastodon user..."
+          curl -X POST -u "elastic:${ELASTIC_PASSWORD}" \
+            "http://elasticsearch-es-http:9200/_security/user/mastodon" \
+            -H 'Content-Type: application/json' \
+            -d '{
+              "password": "'"${MASTODON_PASSWORD}"'",
+              "roles": ["mastodon_full_access"]
+            }'
+          
+          echo "User creation response: $?"
+          echo "Security setup completed!"
+        env:
+        - name: ELASTIC_PASSWORD
+          valueFrom:
+            secretKeyRef:
+              name: elasticsearch-es-elastic-user
+              key: elastic
+        - name: MASTODON_PASSWORD
+          valueFrom:
+            secretKeyRef:
+              name: elasticsearch-credentials
+              key: password
+        securityContext: {}
+      nodeSelector: {}
+      tolerations: []