Technical principles
Small control plane, honest data plane
Three layers
One management cluster, many Cells, many child clusters
Each layer has a single job, and boundaries are enforced by the API surface rather than by convention.
Management cluster
Single-node K3s. Desired state only; it runs no tenant workloads.
- OCM Hub: ManagedCluster, ManifestWork, ManagedServiceAccount, cluster proxy.
- KubeCell management controller plus the CRD admission webhook.
- Four CRDs: Cell, VirtualNodeClass, VirtualClusterPlan, VirtualCluster.
- Installer orchestrates release bundles; charts never mutate OCM objects.
Cell host cluster
One Cell equals one OCM ManagedCluster equals one physical host cluster.
- OCM klusterlet and work-agent apply the two works per child cluster.
- K3k v1.2.0 in shared mode creates and reflects child clusters.
- Host webhook admits platform Pods and rejects tenant host access.
- Pinned baseline: CNI/CoreDNS, TopoLVM, Traefik, vendor Device Plugins.
VirtualCluster (child)
A real K3s control plane with one logical node named kubelet.
- Independent API server and data volume, stored on TopoLVM.
- Child admin kubeconfig always published to a management-plane Secret.
- Child Pods and PVCs are reflected to the host namespace one-to-one.
- Scale vertically by switching quota tiers, never horizontally.
KubeCell CRDs
Cell · Class · Plan · VC
OCM Hub
desired state only
Host setup + cluster creation
applied on the host
Cell host
work-agent applies
VirtualCluster
child K3s API + kubelet
Control flow
What happens between kubectl apply and a kubeconfig
Every step is idempotent and safe to retry. Failed work is retried by re-rendering from the snapshot, not by patching host state in place.
- 1
Resolve and snapshot
The controller resolves the Cell and VirtualNodeClass, then writes an immutable resolution snapshot to status.resolved. All steady-state rendering reads only from that snapshot.
- 2
Re-run feasibility
Before creating anything, the controller re-runs the same feasibility evaluation the Plan used. A non-Accepted conclusion stops the flow and the report lands in status.feasibilityChecks.
- 3
Create the Foundation Work
Host namespace with policy labels, ResourceQuota, LimitRange, NetworkPolicy, and port-forward RBAC. Only when the work is Applied does the next step start.
- 4
Create the Instance Work
A K3k Cluster in Shared mode plus the K3k VirtualClusterPolicy. The work-agent on the host executes it; K3k then creates the server Pod, data volume, and NodePort Service.
- 5
Publish credentials
The controller reads the K3k kubeconfig Secret through the OCM proxy, rewrites the server address to the auto-discovered host address and NodePort, then republishes it in the management plane.
- 6
Converge and report
Steady-state reconciliation compares observed host state with the snapshot. Drift, stale inventory, and version mismatches surface as conditions — never as silent fixes.
# Foundation Work (excerpt) — host namespace and guardrailsapiVersion: work.open-cluster-management.io/v1kind: ManifestWorkmetadata: name: kubecell-<vc>-foundation namespace: <managed-cluster-namespace>spec: workload: manifests: - apiVersion: v1 kind: Namespace metadata: name: <vc>-ns labels: kubecell.io/virtualcluster: <vc> kubecell.io/managed: "true" - apiVersion: v1 kind: ResourceQuota # hard requests/limits from the class - apiVersion: v1 kind: LimitRange # default 100m / 256Mi - apiVersion: networking.k8s.io/v1 kind: NetworkPolicy # deny by default# Instance Work (excerpt) — K3k shared-mode cluster - apiVersion: k3k.io/v1beta1 kind: Cluster spec: mode: shared servers: 1 agents: 1 persistence: type: dynamic storageClassName: topolvm-provisionerTruth and freshness
A Cell is Ready only when its observations are alive
KubeCell separates “the object exists” from “the system is healthy”. Lease freshness and inventory freshness are first-class conditions, because stale data is the most common cause of wrong capacity decisions.
| Signal | Means | Failure mode |
|---|---|---|
| OCM Joined | The klusterlet on the host registered successfully. | Registration can be revoked or replaced. |
| OCM Available | The hub considers the cluster reachable. | Observed to go stale without updates; not trusted alone. |
| managed-cluster-lease fresh | The host Lease keeps renewing. | The decisive liveness signal for a Cell. |
| InventoryFresh | Inventory and storage observations are recent and from a live source. | When false, no feasibility conclusion is trustworthy. |
Endpoint discovery
Addresses are observed, never configured
The API address is not part of spec. The controller reads Ready host nodes through the proxy, takes the InternalIP of the first node by name, and records it in status.
$ kubectl-n kubecell-system get vc child-dev \ -o jsonpath='{.status.endpoint}'{"address":"10.0.12.7","port":31234}# When the address changes, kubeconfigs are republished# automatically. Clients pick up the new file; no CR edit.Exposure method, port range, and proxy identity are platform constants. That is why they are absent from the CRDs: there is exactly one safe value, so exposing them would only create drift.
Code layout
Where to look in the repository
The design is small enough to hold in your head: four CRDs in api/, one controller in internal/, two charts, and an installer that owns the release bundle.
Controllers never install host software; the installer does, at pinned versions. Charts never mutate OCM objects. Those two rules explain most of the directory structure.
Built on upstream
Upstream does the work; KubeCell owns the contract
K3k, K3s, TopoLVM, Traefik, and OCM are used through their public APIs and never patched. You get upstream fixes and behavior directly, and every release can pin and verify the exact combination it was tested with.
- No custom scheduler, CNI, CSI, device allocator, or host agent: standard Kubernetes behavior applies.
- No automatic cross-Cell placement, migration, or failover in the first version.
- No strong-adversary multi-tenancy: developer credentials are child-cluster admin kubeconfigs.
- port-forward and automatic physical data reclamation are outside the locked commitments.
Locked combination
Versions that move together
Changing any of these requires a release, not a CR edit.
| Component | Version | Notes |
|---|---|---|
| Management cluster | Single-node K3s | Holds your manifests and reports state; runs no tenant workloads. |
| Host Kubernetes | K3s v1.36.3+k3s1 | Validated on ARM64 bare metal. |
| K3k | v1.2.0 | Creates and manages the per-developer control planes on each host. |
| Cluster Kubernetes | v1.34.2+k3s1 | The version your developers see and target with kubectl. |
| OCM Hub | v1.3.1 | The audited channel between the management cluster and hosts. |
| Storage | TopoLVM on hosts | Persistent volumes for clusters, backed by local disks on the host. |
Ingress and north-south
One host ingress, predictable hostnames
Host Traefik owns ports 80 and 443 for the whole host. It watches only VirtualCluster namespaces that carry the KubeCell label, so one noisy tenant cannot claim the edge.
You choose the Ingress name
Create a standard Ingress with class kubecell in the child cluster. The hostname derives as <ingress>.<vc>.<apps-suffix>; the suffix is a platform constant.
Endpoints stay in sync
The controller mirrors child Endpoints into the host namespace and strips port names that would break routing.
TLS is an operator decision
Certificates are issued at the host ingress. The design keeps v1 to HTTP(S) rather than promising protocols it cannot keep.
Symptom: Cell reports Degraded even though OCM still says Available.
Action: Trust the lease, not the condition. Check klusterlet health and node time skew before creating new child clusters.
Symptom: storageInventory.freeCapacityKnown is false, or the observation is old.
Action: No feasibility conclusion is trustworthy. Fix TopoLVM/lvmd reporting and wait for a fresh Cell observation.
Symptom: A VirtualCluster stays in Provisioning and never becomes Ready.
Action: Walk the chain: the work status in the hub, the work-agent on the host, then the target namespace quota and any admission rejections.
Symptom: The kubeconfig exists but connections time out.
Action: Compare status.endpoint with the published Secret, then check K3k server Pods, the NodePort Service, and TopoLVM PVCs on the host.
Symptom: A Cell declares more accelerator capacity than the host actually allocates.
Action: Observed allocatable always wins. Align the machine profile declaration with the Device Plugin registration, then re-observe.
Symptom: Admission denies a child Pod for privileged access or hostPath volumes.
Action: This is expected. Use the rejection hint to pick a supported replacement, such as a PVC on the TopoLVM storage class.