414 lines
11 KiB
Markdown
414 lines
11 KiB
Markdown
# Local Path Provisioner
|
|
|
|
## Overview
|
|
|
|
The Local Path Provisioner provides local storage using hostPath volumes. It's useful for:
|
|
- Testing and development
|
|
- Stateful workloads that don't require high availability
|
|
- Single-node scenarios
|
|
- Temporary storage needs
|
|
|
|
**Important**: This is NOT recommended for production workloads that require:
|
|
- High availability
|
|
- Data replication
|
|
- Cross-node pod mobility
|
|
- Disaster recovery
|
|
|
|
For production, use Ceph block storage (`ceph-block` StorageClass) instead.
|
|
|
|
## Current Configuration
|
|
|
|
- **Namespace**: `local-path-storage`
|
|
- **Storage Path**: `/var/mnt/local-path-provisioner` (Talos-compatible path)
|
|
- **Default StorageClass**: `false` (Ceph is the default)
|
|
- **Pod Security**: `privileged` (required for hostPath access)
|
|
- **Sync Wave**: `-2` (deploys before storage operators)
|
|
|
|
## When to Use Local Path vs Ceph
|
|
|
|
### Use Local Path For:
|
|
✅ **Development/Testing**
|
|
- Quick pod restarts on the same node
|
|
- No network overhead
|
|
- Fast I/O for local development
|
|
|
|
✅ **Node-Specific Data**
|
|
- Logs collection
|
|
- Monitoring agents data
|
|
- Cache that can be rebuilt
|
|
|
|
✅ **Temporary Storage**
|
|
- Build artifacts
|
|
- Scratch space
|
|
- Non-critical data
|
|
|
|
### Use Ceph Block Storage For:
|
|
✅ **Production Databases**
|
|
- PostgreSQL, MySQL, MongoDB
|
|
- Requires replication and HA
|
|
|
|
✅ **Stateful Applications**
|
|
- When pods need to move between nodes
|
|
- Data must survive node failures
|
|
|
|
✅ **Critical Data**
|
|
- Persistent volumes that need backups
|
|
- Data requiring disaster recovery
|
|
|
|
✅ **Multi-Replica Apps**
|
|
- ReadWriteOnce volumes that may reschedule
|
|
|
|
## Architecture & Limitations
|
|
|
|
```
|
|
┌─────────────────────────────────────────┐
|
|
│ Node 1 │
|
|
│ ┌─────────────────────────────────┐ │
|
|
│ │ /var/mnt/local-path-provisioner │ │
|
|
│ │ └── pvc-abc123/ │ │
|
|
│ │ └── data │ │
|
|
│ └─────────────────────────────────┘ │
|
|
│ ▲ │
|
|
│ │ hostPath mount │
|
|
│ ┌──────┴──────┐ │
|
|
│ │ Pod (fixed) │ │
|
|
│ └─────────────┘ │
|
|
└─────────────────────────────────────────┘
|
|
|
|
⚠️ If pod moves to Node 2, data is NOT accessible!
|
|
```
|
|
|
|
### Key Limitations:
|
|
|
|
1. **Node Affinity**: Pods are pinned to the node where PVC was created
|
|
2. **No Replication**: Data exists only on one node
|
|
3. **No HA**: If node fails, data is inaccessible until node recovers
|
|
4. **No Migration**: Cannot move volumes between nodes
|
|
5. **Disk Space**: Limited by node's local disk capacity
|
|
|
|
## Storage Classes Comparison
|
|
|
|
| Feature | local-path | ceph-block | ceph-filesystem |
|
|
|---------|-----------|------------|-----------------|
|
|
| **HA** | ❌ No | ✅ Yes | ✅ Yes |
|
|
| **Replication** | ❌ No | ✅ Yes (2x) | ✅ Yes (2x) |
|
|
| **Multi-node** | ❌ No | ✅ Yes (RWO) | ✅ Yes (RWX) |
|
|
| **Performance** | ⚡ Fast | 📊 Medium | 📊 Medium |
|
|
| **Snapshots** | ❌ No | ✅ Yes | ✅ Yes |
|
|
| **Resize** | ⚠️ Manual | ✅ Auto | ✅ Auto |
|
|
| **Backup** | ❌ Difficult | ✅ Velero | ✅ Velero |
|
|
| **Use Case** | Dev/Test | Production | Shared Data |
|
|
|
|
## Usage Examples
|
|
|
|
### Basic PVC
|
|
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: PersistentVolumeClaim
|
|
metadata:
|
|
name: local-test-pvc
|
|
namespace: default
|
|
spec:
|
|
storageClassName: local-path
|
|
accessModes:
|
|
- ReadWriteOnce
|
|
resources:
|
|
requests:
|
|
storage: 10Gi
|
|
```
|
|
|
|
### StatefulSet with Local Path
|
|
|
|
```yaml
|
|
apiVersion: apps/v1
|
|
kind: StatefulSet
|
|
metadata:
|
|
name: test-app
|
|
spec:
|
|
serviceName: test-app
|
|
replicas: 1 # ⚠️ Keep at 1 for local-path
|
|
selector:
|
|
matchLabels:
|
|
app: test-app
|
|
volumeClaimTemplates:
|
|
- metadata:
|
|
name: data
|
|
spec:
|
|
storageClassName: local-path
|
|
accessModes: ["ReadWriteOnce"]
|
|
resources:
|
|
requests:
|
|
storage: 5Gi
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app: test-app
|
|
spec:
|
|
containers:
|
|
- name: app
|
|
image: nginx
|
|
volumeMounts:
|
|
- name: data
|
|
mountPath: /data
|
|
```
|
|
|
|
### Pod with Explicit Node Affinity
|
|
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: Pod
|
|
metadata:
|
|
name: local-path-test
|
|
spec:
|
|
# Pin to specific node where PVC exists
|
|
nodeSelector:
|
|
kubernetes.io/hostname: worker1
|
|
|
|
volumes:
|
|
- name: data
|
|
persistentVolumeClaim:
|
|
claimName: local-test-pvc
|
|
|
|
containers:
|
|
- name: test
|
|
image: busybox
|
|
command: ["sh", "-c", "sleep 3600"]
|
|
volumeMounts:
|
|
- name: data
|
|
mountPath: /data
|
|
```
|
|
|
|
## Recommendations
|
|
|
|
### 1. Storage Path Configuration ✅
|
|
|
|
Current path `/var/mnt/local-path-provisioner` is correct for Talos Linux:
|
|
- Talos only allows persistent storage in `/var`
|
|
- This path persists across reboots
|
|
- Properly configured in your setup
|
|
|
|
### 2. Not Default StorageClass ✅
|
|
|
|
Good decision to keep `ceph-block` as default:
|
|
```yaml
|
|
storageclass.kubernetes.io/is-default-class: "false"
|
|
```
|
|
|
|
This ensures:
|
|
- PVCs without explicit `storageClassName` use Ceph
|
|
- Production workloads default to HA storage
|
|
- Local path is opt-in only
|
|
|
|
### 3. Use Cases for Your Cluster
|
|
|
|
**Good Uses:**
|
|
```yaml
|
|
# Development namespace with local-path default
|
|
apiVersion: v1
|
|
kind: Namespace
|
|
metadata:
|
|
name: dev-test
|
|
annotations:
|
|
# All PVCs in this namespace can use local-path
|
|
storageclass.kubernetes.io/is-default-class: "local-path"
|
|
|
|
---
|
|
# Prometheus node exporter temp storage
|
|
apiVersion: v1
|
|
kind: PersistentVolumeClaim
|
|
metadata:
|
|
name: node-exporter-data
|
|
spec:
|
|
storageClassName: local-path
|
|
accessModes: [ReadWriteOnce]
|
|
resources:
|
|
requests:
|
|
storage: 5Gi
|
|
```
|
|
|
|
**Bad Uses (Use Ceph Instead):**
|
|
```yaml
|
|
# ❌ Don't do this - use ceph-block
|
|
apiVersion: v1
|
|
kind: PersistentVolumeClaim
|
|
metadata:
|
|
name: postgres-data # Database needs HA!
|
|
spec:
|
|
storageClassName: local-path # ❌ Wrong choice
|
|
|
|
# ✅ Do this instead
|
|
spec:
|
|
storageClassName: ceph-block # ✅ Right choice
|
|
```
|
|
|
|
### 4. Monitoring Storage Usage
|
|
|
|
```bash
|
|
# Check PVs and their node
|
|
kubectl get pv -o custom-columns=NAME:.metadata.name,STORAGECLASS:.spec.storageClassName,NODE:.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0],SIZE:.spec.capacity.storage
|
|
|
|
# Check disk usage on nodes
|
|
talosctl -n <node-ip> df | grep local-path-provisioner
|
|
|
|
# List all local-path PVCs
|
|
kubectl get pvc -A --field-selector spec.storageClassName=local-path
|
|
```
|
|
|
|
### 5. Migration Strategy
|
|
|
|
If you need to migrate from local-path to Ceph:
|
|
|
|
```bash
|
|
# 1. Create new PVC with ceph-block
|
|
kubectl apply -f - <<EOF
|
|
apiVersion: v1
|
|
kind: PersistentVolumeClaim
|
|
metadata:
|
|
name: data-ceph
|
|
namespace: default
|
|
spec:
|
|
storageClassName: ceph-block
|
|
accessModes: [ReadWriteOnce]
|
|
resources:
|
|
requests:
|
|
storage: 10Gi
|
|
EOF
|
|
|
|
# 2. Use a temporary pod to copy data
|
|
kubectl apply -f - <<EOF
|
|
apiVersion: v1
|
|
kind: Pod
|
|
metadata:
|
|
name: data-migration
|
|
spec:
|
|
volumes:
|
|
- name: source
|
|
persistentVolumeClaim:
|
|
claimName: data-local-path
|
|
- name: dest
|
|
persistentVolumeClaim:
|
|
claimName: data-ceph
|
|
containers:
|
|
- name: migrate
|
|
image: ubuntu
|
|
command: ["sh", "-c", "cp -av /source/* /dest/ && echo 'Done'"]
|
|
volumeMounts:
|
|
- name: source
|
|
mountPath: /source
|
|
- name: dest
|
|
mountPath: /dest
|
|
EOF
|
|
|
|
# 3. Verify data, then update your app to use new PVC
|
|
```
|
|
|
|
### 6. Cleanup Old PVCs
|
|
|
|
Local path PVs can accumulate. Regular cleanup:
|
|
|
|
```bash
|
|
# List unused PVCs
|
|
kubectl get pvc -A --field-selector status.phase=Bound
|
|
|
|
# Delete unused PVCs (will also remove hostPath data)
|
|
kubectl delete pvc <pvc-name> -n <namespace>
|
|
|
|
# Cleanup orphaned directories on nodes (if needed)
|
|
talosctl -n <node-ip> ls /var/mnt/local-path-provisioner
|
|
```
|
|
|
|
### 7. Resource Limits
|
|
|
|
Consider node disk capacity:
|
|
|
|
```bash
|
|
# Check available space on each node
|
|
for node in $(kubectl get nodes -o name | cut -d/ -f2); do
|
|
echo "=== $node ==="
|
|
talosctl -n $node df | grep "/$"
|
|
done
|
|
|
|
# Set PVC size limits based on node capacity
|
|
# Example: If node has 100GB free, don't create PVCs > 50GB
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### PVC Stuck in Pending
|
|
|
|
```bash
|
|
# Check events
|
|
kubectl describe pvc <pvc-name> -n <namespace>
|
|
|
|
# Common causes:
|
|
# 1. No nodes have enough disk space
|
|
# 2. Provisioner pod not running
|
|
kubectl get pods -n local-path-storage
|
|
```
|
|
|
|
### Pod Can't Mount Volume
|
|
|
|
```bash
|
|
# Check if pod is on same node as PV
|
|
kubectl get pod <pod-name> -o jsonpath='{.spec.nodeName}'
|
|
kubectl get pv <pv-name> -o jsonpath='{.spec.nodeAffinity}'
|
|
|
|
# If different nodes, delete and recreate pod
|
|
# (pod will reschedule to correct node)
|
|
```
|
|
|
|
### Disk Space Issues
|
|
|
|
```bash
|
|
# Check node disk usage
|
|
kubectl get nodes -o custom-columns=NAME:.metadata.name,STORAGE:.status.allocatable.ephemeral-storage
|
|
|
|
# Free up space by deleting old PVCs
|
|
kubectl get pvc -A --sort-by=.metadata.creationTimestamp
|
|
```
|
|
|
|
## Best Practices
|
|
|
|
1. ✅ **Use for ephemeral workloads only** - Anything that can afford data loss
|
|
2. ✅ **Set PVC size limits** - Prevent filling up node disks
|
|
3. ✅ **Monitor disk usage** - Set up alerts for node disk space
|
|
4. ✅ **Document dependencies** - Note which apps use local-path and why
|
|
5. ✅ **Plan for migration** - Have a strategy to move to Ceph if needed
|
|
6. ❌ **Don't use for databases** - Use ceph-block instead
|
|
7. ❌ **Don't use for multi-replica apps** - Pods will be pinned to one node
|
|
8. ❌ **Don't assume HA** - Data is lost if node fails
|
|
|
|
## Decision Tree
|
|
|
|
```
|
|
Need persistent storage?
|
|
│
|
|
├─ Is data critical? ──→ YES ──→ Use ceph-block
|
|
│
|
|
├─ Need HA/replication? ──→ YES ──→ Use ceph-block
|
|
│
|
|
├─ Pod needs to move between nodes? ──→ YES ──→ Use ceph-block
|
|
│
|
|
├─ Multiple replicas need same data? ──→ YES ──→ Use ceph-filesystem
|
|
│
|
|
├─ Development/testing only? ──→ YES ──→ Consider local-path
|
|
│
|
|
└─ Temporary/cache data? ──→ YES ──→ Consider local-path or emptyDir
|
|
```
|
|
|
|
## Summary
|
|
|
|
Local path provisioner is included in your cluster but should be used sparingly:
|
|
|
|
✅ **When to use**: Dev/test, non-critical data, node-specific storage
|
|
❌ **When NOT to use**: Production databases, HA apps, critical data
|
|
|
|
Your configuration is correct:
|
|
- Talos-compatible path ✅
|
|
- Not default StorageClass ✅
|
|
- Privileged pod security ✅
|
|
- Automated prune enabled ✅
|
|
|
|
For most production workloads, stick with `ceph-block` (your default).
|