Sidecar Container GPU Resource Accounting

This design was implemented in HAMi PR #2723, after v2.10.0. The problem descriptions below refer to the earlier init-container accounting; the proposal describes the sidecar-aware changes.

Problem Summary

Native sidecar containers are declared in spec.initContainers with restartPolicy: Always, but unlike regular init containers they run for the whole pod lifetime, next to the app containers. Before PR #2723, HAMi (Init Container GPU Resource Accounting) classified only by which list a container appears in. That accounting did not check RestartPolicy, so a sidecar received run-to-completion semantics it does not have.

This design adds sidecars as a third container class.

Problems Before Sidecar-Aware Accounting

1. Under-accounting: CollapseInitContainerUsage (pkg/device/initContainer.go). Classification is by index (cidx < numInit), so a sidecar lands in the init-peak (max) bucket. A 4000 MiB sidecar plus a 4000 MiB app container on one card is accounted as max(4000, 4000) = 4000; real demand is 8000, so the scheduler can oversubscribe the card.

2. A running sidecar blocks the shrink gate. The shrink waits for every init container to be Terminated. A running sidecar prevents that gate from opening, so the regular init containers' memory stays reserved. Case 4 of the init design stops working.

3. The shrink target omits sidecars. AppContainersOnlyDeviceUsage. The shrink target skips all init containers, sidecars included. An exit-0 gap between sidecar restarts can satisfy the old gate and drop the sidecar's usage from accounting. The gate and target have to change together.

The Core Idea

The upstream formula (what the apiserver itself charges):

effective = max( max over non-sidecar init_i ( init_i + sum(sidecars declared before init_i) ),
                 sum(apps) + sum(all sidecars) )

A flat sidecar_sum + max(init_peak, app_sum) (same spirit as the init design's assumption, using sum(all sidecars) instead of the ordering-aware term) was considered as a cheaper approximation. It turned out no harder to walk spec.initContainers in declaration order and fold each sidecar's usage into the running peak as it's seen, so PR #2723 implements the ordering-aware term above. This follows the shape of the apiserver calculation; HAMi applies it per device UUID. Per device UUID and resource (count, mem, cores):

effective[uuid] = max( max over non-sidecar init_i ( init_i[uuid] + sidecar_sum_so_far[uuid] ),
                        app_sum[uuid] + sidecar_sum[uuid] )

where sidecar_sum_so_far accumulates only the sidecars declared earlier in spec.initContainers, and sidecar_sum in the second term is the total across all sidecars. If there are no non-sidecar init containers, the first term is 0, and a missing per-UUID entry also counts as 0 before the max() and addition.

Classification:

isSidecar(c) := c ∈ spec.initContainers && c.RestartPolicy != nil &&
                *c.RestartPolicy == corev1.ContainerRestartPolicyAlways

The nil check makes this safe everywhere: absent field means no sidecars, behavior stays exactly as today. No version gating, no new config.

Design Implemented by PR #2723

• Admission quota check: walk spec.initContainers in order, accumulating a running sidecar sum and folding it into each non-sidecar init container's peak (pkg/scheduler/webhook.go's fitResourceQuota); the final effectiveReq is that peak compared against the total sidecar sum plus the app sum; memory factor applied once to the result.
• Scheduler fit & scoring: a steady-state pass fits sidecars plus app containers cumulatively against a shared node copy; each non-sidecar init container gets its own fresh copy, deep-copied from that shared one at the point it's reached, so it's pre-charged with only the sidecars declared before it; merge per UUID via max().
• Usage recording: CollapseInitContainerUsage walks containers in declaration order, adding only preceding sidecars to each regular init container's usage and all sidecars to the app total. The same split applies to the per-entry slot count (HAMi#2623): sidecar slots add like app containers, non-sidecar inits keep their peak of 1. getNodesUsage only consumes the stored output, so it needs no change of its own. Add/update/delete symmetry stays as it is.

Annotations don't change, but a sidecar keeps its position in the init range of hami.io/vgpu-devices-allocated, and the annotation itself carries no sidecar identity. Position i maps to pod.Spec.InitContainers[i] when i < len(InitContainers), otherwise to pod.Spec.Containers[i - len(InitContainers)]; accounting readers must use that container's restartPolicy to distinguish a sidecar from a regular init container.

Cases (one node, single 24Gi GPU)

• Oversubscription prevented: sidecar 10Gi + app 10Gi. Before: accounted max(10,10) = 10Gi, so a later 12Gi pod schedules → real demand 32Gi. After: accounted 20Gi → the 12Gi pod is rejected.
• Shrink restored: init 20Gi + sidecar 2Gi + app 10Gi. Before: while the sidecar is running, the gate stays closed and 20Gi remains reserved. After, admission depends on declaration order: sidecar declared before the init container charges max(2+20, 2+10) = 22Gi; init container declared first charges max(20, 2+10) = 20Gi. Either way, once the init container exits 0, usage shrinks to the steady state 2+10 = 12Gi.
• No sidecars, or terminal phase: identical to the init design.

Shrink Rules

Same three rules as the init design, with non-sidecar inserted: shrink to steady-state usage (apps + sidecars) once non-sidecar inits exit 0; hold on non-zero exit; zero at terminal phase. A sidecar can crash-loop through Terminated states; its usage stays counted through the gap and never briefly reads zero. Stored usage only changes at add, at the shrink (whose target includes sidecars), or at terminal phase, and a restart triggers none of these. But an exit-0 gap can momentarily satisfy the old gate and fire the shrink, permanently dropping the sidecar's usage; the non-sidecar gate closes that. A test should pin both. If all init containers are sidecars (init_peak = 0), the gate is satisfied immediately and the shrink recomputes the stored value (delta 0). initContainerResourceReleased keeps its semantics. PR #2723 replaces AppContainersOnlyDeviceUsage with SteadyStateDeviceUsage to include sidecars in the shrink target.

Interaction with Kubernetes ResourceQuota

The apiserver charges the ordering-aware upstream formula, walking the same spec.initContainers order. Because admission computes that same formula rather than the flat simplification, the ordering corner case this design originally warned about, where a pod near the limit passes the apiserver at 20Gi but is rejected by HAMi at 22Gi, does not arise: both sides agree on the shape of the calculation for a given pod. The absolute numbers can still differ, since HAMi applies its memory factor to its own value before checking its internal quota cache. As before, the shrink only frees capacity inside HAMi; the apiserver holds its charge until the pod ends, which is fine, because the sidecar's share has to be held that long anyway.