Guides
Troubleshooting
KubeCell spans four systems — the management plane, OCM, the host cluster, and the child K3s cluster. Almost every failure is diagnosed by walking that chain in order instead of guessing.
Walk the chain
diagnose.sh
# 1. Management plane: what does the controller believe?kubectl-n kubecell-system get vc <vc> -o yamlkubectl-n kubecell-system describe vc <vc># 2. OCM: did the work get applied?kubectl get manifestwork-n <managed-cluster-namespace>kubectl get manifestwork-n <managed-cluster-namespace> <work> -o yaml# 3. Host: is the work-agent healthy and did it land?KUBECONFIG=<host-kubeconfig> kubectl -n open-cluster-management-agent \ get podsKUBECONFIG=<host-kubeconfig> kubectl -n <vc>-ns get all,quota,limitrange,netpol# 4. Child: is the control plane up and reachable?KUBECONFIG=<host-kubeconfig> kubectl -n k3k-<vc> get pods,svc,pvcOne rule that saves time
Never edit host state by hand to "unstick" a child cluster. Fix the input (the manifest, the class, the inventory source) and let reconciliation re-render from the snapshot. Manual edits are overwritten on the next reconcile and hide the real cause.Symptom reference
| Symptom | Where to look | Action |
|---|---|---|
| Cell not Ready | Check in order: OCM Joined/Available, then managed-cluster-lease freshness, then Cell conditions. Available=True alone is not trusted. | If the lease is stale, treat the host as offline and stop creating child clusters on it. |
| Cell Degraded but OCM looks fine | Lease freshness or an inventory source failed. | Inspect conditions: ProviderReady and InventoryFresh. Fix the reported subsystem before anything else. |
| Work not Applied | ManifestWork status, then host work-agent, then the target namespace. | Check namespace quota and admission rejections — a rejected manifest will retry forever. |
| VirtualCluster stuck Provisioning | Foundation Work applied but Instance Work blocked, or K3k cannot schedule its server Pod. | Check K3k controller health, server Pod events, and TopoLVM PVC binding on the host. |
| Child API unreachable | status.endpoint versus the published Secret, then host NodePort Service. | Re-fetch the Secret after endpoint changes; verify the NodePort is allocated and fresh. |
| Child PVC Pending | Topology labels missing or TopoLVM has no free capacity. | Re-apply topology.topolvm.io/node and topology.kubernetes.io/zone, then check lvmd reporting. |
| Zero accelerators visible | Device Plugin registration mismatch. | Read plugin Pod logs on the host before changing any declaration; observed allocatable wins. |
| CapabilityMismatch condition | Cell declares more than the host allocates. | Align the machine profile with the observed resource key and capacity, then re-observe. |
| Tenant Pod rejected | Privileged access or hostPath in the manifest. | This is intended. Use the rejection hint and switch to a PVC on the storage class. |
| Ingress returns 503 | Mirrored Endpoints carry a port name that does not match. | The controller strips port names; for older clusters clear the endpoint port name to an empty string. |
| Image pull failures | Registry reachability or OCI-archive transfer problems. | Fix mirrors first; helpers live in hack/image/. Record digests and architectures. |
| Volumes remain after deletion | Orphaned logical volumes once the child API is gone. | Remove only cluster-created volumes with lvremove; leave the volume group intact. |
A Plan was rejected — now what?
- Read
status.checks. Each failed item names the resource that could not be satisfied. - Check
expiresAt. If the conclusion expired, re-apply the Plan for a fresh evaluation. - If the missing resource is storage or accelerators, verify that the Cell inventory is fresh before concluding the host is full.
- Adjust the request or switch Cells. Do not create the VirtualCluster "to see what happens" — the controller rechecks and blocks anyway.
Collecting evidence for an issue
- Cell and VirtualCluster YAML (including status) with timestamps.
- ManifestWork status for the affected child cluster.
- Host events for the child namespace, plus K3k server Pod logs.
- Observed versions from Cell status versus the pinned release.
- Reproduction steps and expected versus actual behavior.
File bugs at github.com/taosher/KubeCell/issues using the structure above. See also the host onboarding troubleshooting table for host-side symptoms.