Security, Troubleshooting, and Performance
Free NVIDIA-Certified Professional: AI Networking practice — 6 questions on Security, Troubleshooting, and Performance, with explanations. No sign-up. Full 12-question mixed test →
A cloud provider runs two independent AI tenants on a shared 400G InfiniBand NDR fabric managed by UFM. Tenant A must never be able to send or receive RDMA traffic to/from Tenant B's HCAs, even if a misconfigured application attempts it. Which mechanism BEST enforces this isolation at the fabric level?
InfiniBand has no native VLAN concept; isolation is enforced through Partition Keys (P_Keys), which the Subnet Manager (via UFM's Partition Manager) programs into every port's partition table. Membership limited/full is enforced in hardware at the HCA and switch level, so traffic between partitions is dropped before it ever reaches the destination, giving true fabric-level isolation without extra physical infrastructure.
A RoCEv2-based AI cluster experiences periodic ~50ms freezes in GPU-to-GPU collective operations across many nodes simultaneously. Switch telemetry shows synchronized spikes in ingress buffer occupancy right before each freeze, and no CRC or symbol errors are reported on any link. Which condition is most likely occurring, and what should be checked to confirm it?
Synchronized multi-node freezes with rising buffer occupancy and no bit/CRC errors are the signature of a PFC (Priority Flow Control) storm on a lossless RoCE fabric: congestion on one queue triggers pause frames that propagate backward through the fabric, freezing unrelated flows sharing the same priority. Checking pause-frame counters on switch interfaces directly confirms whether PFC XOFF events are being triggered and cascading.
In a Kubernetes cluster deployed with the NVIDIA Network Operator to support GPUDirect RDMA workloads, which component is responsible for discovering RDMA/SR-IOV virtual functions on the NICs and advertising them as schedulable extended resources so pods can request dedicated RDMA interfaces?
The SR-IOV Network Device Plugin, deployed by the NVIDIA Network Operator, scans nodes for configured RDMA-capable SR-IOV virtual functions and registers them as Kubernetes extended resources (e.g., nvidia.com/roce_gdr). Pods request these resources in their spec, and the plugin allocates and binds the VF to the pod for GPUDirect RDMA use.
An enterprise wants to enforce zero-trust micro-segmentation policies between AI compute nodes on the east-west traffic path, inspecting and filtering traffic at line rate without consuming host CPU cycles. Which BlueField DPU capability is designed to deliver this?
BlueField DPUs run DOCA-based services, including flow-table-driven firewall/micro-segmentation applications built with the DOCA Flow library, that offload packet classification and policy enforcement onto the DPU's hardware accelerators. This enables per-flow zero-trust policies at line rate for east-west traffic without burdening the host x86/GPU CPU.
You need to configure lossless RoCEv2 transport on a Spectrum switch fabric for an AI training cluster while avoiding PFC deadlock and maintaining near line-rate throughput. Which combination of configuration steps correctly achieves this?
Robust lossless RoCEv2 requires PFC to prevent buffer overflow drops, ECN marking on switch queues to signal incipient congestion before PFC XOFF is triggered, and DCQCN running on the adapters to react to ECN marks by throttling sender rate. Together these three elements prevent PFC deadlock/storms while preserving throughput; using PFC alone risks deadlock and pause-frame cascades under sustained congestion.
On a DGX H100 cluster with NDR InfiniBand, an NCCL all-reduce benchmark achieves only 60% of expected peak GPU-to-GPU bandwidth. Raw HCA-to-HCA testing with ib_write_bw between the same nodes reaches full line rate with zero errors. What is the most likely cause, and how should it be confirmed?
Since raw RDMA bandwidth (ib_write_bw) already reaches line rate with no errors, the fabric and cabling are healthy — the bottleneck must be above the RDMA transport layer. A common cause is GPUDirect RDMA traffic traversing a PCIe path that crosses NUMA nodes or PCIe switches to reach a non-local HCA instead of the nearest one, adding latency and reducing effective bandwidth. 'nvidia-smi topo -m' reveals GPU-NIC affinity, and NCCL_IB_HCA/NCCL topology environment variables can be checked and corrected to bind each GPU to its nearest HCA.
Ready for the real thing?
The full course has two full-length practice tests, video lessons for every exam domain, hands-on labs and detailed answer explanations.
Start my full course on Udemy →