URL: https://mastodon.keyboardvagabond.com
Federation Domain: keyboardvagabond.com (CRITICAL: Never change this!)
Architecture: Multi-container design with Web, Sidekiq, and Streaming deployments
Authentication: Authentik OIDC integration + local accounts
Storage: Backblaze B2 S3-compatible storage with Cloudflare CDN
Database: Shared PostgreSQL cluster with CloudNativePG
Cache: Shared Redis cluster

Directory Structure

mastodon/
├── namespace.yaml           # mastodon-application namespace
├── repository.yaml          # Official Mastodon Helm chart repository
├── secret.yaml              # SOPS-encrypted secrets (credentials, tokens)
├── helm-release.yaml        # Main HelmRelease configuration
├── ingress.yaml            # NGINX ingress with SSL and external-dns
├── monitoring.yaml         # ServiceMonitor for OpenObserve integration
├── kustomization.yaml      # Resource list
└── README.md               # This documentation

🔑 Pre-Deployment Setup

1. Generate Mastodon Secrets

Important: Replace placeholder values in secret.yaml before deployment:

# Generate SECRET_KEY_BASE (using modern Rails command)
docker run --rm -it tootsuite/mastodon bundle exec rails secret

# Generate OTP_SECRET (using modern Rails command)
docker run --rm -it tootsuite/mastodon bundle exec rails secret

# Generate VAPID Keys (after setting SECRET_KEY_BASE and OTP_SECRET)
docker run --rm -it \
  -e SECRET_KEY_BASE="your_secret_key_base" \
  -e OTP_SECRET="your_otp_secret" \
  tootsuite/mastodon bundle exec rake mastodon:webpush:generate_vapid_key

2. Database Setup

Create Mastodon database and user in the existing PostgreSQL cluster:

kubectl exec -it postgresql-shared-1 -n postgresql-system -- psql -U postgres

-- Create database and user
CREATE DATABASE mastodon_production;
CREATE USER mastodon_user WITH PASSWORD 'SECURE_PASSWORD_HERE';
GRANT ALL PRIVILEGES ON DATABASE mastodon_production TO mastodon_user;
ALTER DATABASE mastodon_production OWNER TO mastodon_user;
\q

3. Update Secret Values

Edit secret.yaml and replace:

REPLACE_WITH_GENERATED_SECRET_KEY_BASE
REPLACE_WITH_GENERATED_OTP_SECRET
REPLACE_WITH_GENERATED_VAPID_PRIVATE_KEY
REPLACE_WITH_GENERATED_VAPID_PUBLIC_KEY
REPLACE_WITH_POSTGRESQL_PASSWORD
REPLACE_WITH_REDIS_PASSWORD

4. Encrypt Secrets

sops --encrypt --in-place manifests/applications/mastodon/secret.yaml

🚀 Deployment

Add to Applications Kustomization

Add mastodon to manifests/applications/kustomization.yaml:

resources:
# ... existing apps
- mastodon/

Commit and Deploy

git add manifests/applications/mastodon/
git commit -m "feat: Add Mastodon fediverse application"
git push origin k8s-fleet

Flux will automatically deploy within 5-10 minutes.

📋 Post-Deployment Configuration

1. Initial Admin Setup

Wait for pods to be ready, then create admin account:

# Check deployment status
kubectl get pods -n mastodon-application

# Create admin account (single-user mode enabled initially)
kubectl exec -n mastodon-application deployment/mastodon-web -- \
  tootctl accounts create admin \
  --email admin@keyboardvagabond.com \
  --confirmed \
  --role Admin

2. Disable Single-User Mode

After creating admin account, edit helm-release.yaml:

mastodon:
  single_user_mode: false  # Change from true to false

Commit and push to apply changes.

3. Federation Testing

Test federation with other Mastodon instances:

Search for accounts from other instances
Follow accounts from other instances
Verify media attachments display correctly via CDN

🔧 Configuration Details

Resource Allocation

Starting Resources (Phase 1):

Web: 2 replicas, 1-2 CPU, 2-4Gi memory
Sidekiq: 2 replicas, 0.5-1 CPU, 1-2Gi memory
Streaming: 2 replicas, 0.25-0.5 CPU, 0.5-1Gi memory
Total: ~5.5 CPU requests, ~9Gi memory requests

External Dependencies

✅ PostgreSQL: postgresql-shared-rw.postgresql-system.svc.cluster.local:5432
✅ Redis: redis-ha-haproxy.redis-system.svc.cluster.local:6379
✅ S3 Storage: Backblaze B2 mastodon-bucket
✅ CDN: Cloudflare mm.keyboardvagabond.com
✅ SMTP: <YOUR_SMTP_SERVER> <YOUR_EMAIL_ADDRESS>
✅ OIDC: Authentik auth.keyboardvagabond.com
❌ Elasticsearch: Not configured (Phase 2)

Security Features

HTTPS: Enforced with Let's Encrypt certificates
Headers: Security headers via NGINX ingress
OIDC: Single Sign-On with Authentik
S3: Media storage with CDN distribution
Secrets: SOPS-encrypted in Git

📊 Monitoring

OpenObserve Integration

Metrics automatically collected via ServiceMonitor:

URL: https://obs.keyboardvagabond.com
Metrics: Mastodon application metrics, HTTP requests, response times
Logs: Application logs via OpenTelemetry collector

Health Checks

# Check pod status
kubectl get pods -n mastodon-application

# Check ingress and certificates
kubectl get ingress,certificates -n mastodon-application

# Check logs
kubectl logs -n mastodon-application deployment/mastodon-web
kubectl logs -n mastodon-application deployment/mastodon-sidekiq

🔄 Phase 2: Elasticsearch Integration

When to Add Elasticsearch

Add Elasticsearch when you need:

Full-text search within Mastodon
Better search performance for content discovery
Enhanced user experience with search features

Implementation Steps

Add Elasticsearch infrastructure to manifests/infrastructure/elasticsearch/
Uncomment Elasticsearch configuration in helm-release.yaml
Update dependencies to include Elasticsearch
Enable search features in Mastodon admin panel

🆘 Troubleshooting

Common Issues

Database Connection Errors:

# Check PostgreSQL connectivity
kubectl exec -n mastodon-application deployment/mastodon-web -- \
  pg_isready -h postgresql-shared-rw.postgresql-system.svc.cluster.local -p 5432

Redis Connection Errors:

# Check Redis connectivity  
kubectl exec -n mastodon-application deployment/mastodon-web -- \
  redis-cli -h redis-ha-haproxy.redis-system.svc.cluster.local -p 6379 ping

S3 Upload Issues:

Verify Backblaze B2 credentials
Check bucket permissions and CORS configuration
Test CDN connectivity to mm.keyboardvagabond.com

OIDC Authentication Issues:

Verify Authentik provider configuration
Check client ID and secret
Confirm issuer URL accessibility

Support Commands

# Run Mastodon CLI commands
kubectl exec -n mastodon-application deployment/mastodon-web -- tootctl help

# Database migrations
kubectl exec -n mastodon-application deployment/mastodon-web -- \
  rails db:migrate

# Clear cache
kubectl exec -n mastodon-application deployment/mastodon-web -- \
  tootctl cache clear

📚 References

Official Documentation: https://docs.joinmastodon.org/
Helm Chart: https://github.com/mastodon/chart
Admin Guide: https://docs.joinmastodon.org/admin/
Federation Guide: https://docs.joinmastodon.org/spec/activitypub/