InfiniBand Fabric Configuration and Management
Free NVIDIA-Certified Professional: AI Networking practice — 6 questions on InfiniBand Fabric Configuration and Management, with explanations. No sign-up. Full 12-question mixed test →
A GPU training cluster uses NVIDIA SHARP for NCCL all-reduce acceleration over the InfiniBand fabric. Mid-training, the UFM-hosted SHARP Aggregation Manager (AM) process crashes and does not restart before the next collective operation is issued. No other fabric components fail. What is the MOST LIKELY outcome for the running job?
SHARP is an acceleration path, not the only path — MPI/NCCL collective libraries are designed to detect unavailable SHARP resources and fall back to standard host-based reduction algorithms (e.g., recursive doubling), so the job continues but loses the in-network compute speedup.
An AI training cluster's all-reduce operations show high tail latency driven by the time to complete data reduction across GPUs, not by path congestion or hot-spotting. Which InfiniBand fabric feature is SPECIFICALLY designed to reduce this by performing arithmetic reduction operations inside the switch ASICs rather than at the endpoints?
SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) performs the actual reduction math within the switch fabric, aggregating data from multiple sources before it reaches the destination, directly reducing collective completion time rather than just managing traffic distribution.
You are configuring two OpenSM instances, Node A and Node B, for subnet manager high availability on an InfiniBand fabric. OpenSM priority values range from 0 (lowest) to 15 (highest), and priority is the primary factor in master SM election, with port GUID used only to break ties when priorities are equal. You want Node A to always become the master SM whenever both instances are running, regardless of their GUID values. Which configuration guarantees this?
OpenSM master election compares SM priority first; the instance with the strictly higher priority value always wins, independent of GUID. Only when priorities tie does the algorithm fall back to comparing GUIDs. Setting Node A strictly higher guarantees deterministic mastership.
A fabric administrator defines partition PKey 0x0005 (high bit clear = limited membership) on two GPU compute nodes and PKey 0x8005 (high bit set = full membership) on a storage node in the same partition. Both compute nodes can successfully open RDMA queue pairs to the storage node, but they cannot open a QP directly to each other, even though ibportstate and opensm show the partition applied correctly on all three ports. What explains this behavior?
InfiniBand partition membership semantics restrict limited members (high bit 0) to communicating only with full members (high bit 1) in the same partition; two limited members are explicitly prevented from establishing a QP with each other, which is exactly the isolation behavior observed.
What is the primary function of the SL2VL (Service Level to Virtual Lane) mapping table configured on each InfiniBand port to support QoS for mixed AI training and storage traffic?
SL2VL mapping determines, on a per-port/per-hop basis, which Virtual Lane a packet's Service Level is assigned to. Since each VL has independent buffering and arbitration, this mapping is the mechanism that prevents head-of-line blocking and enforces relative QoS priority between traffic classes such as storage vs. training traffic.
An ibdiagnet scan of a Quantum-2 based fabric flags a rising SymbolErrorCounter and PortXmitDiscards on port 12 of switch SW03, while ibnetdiscover still shows the link as Active at its full negotiated speed. Which command provides detailed physical-layer diagnostics — including eye margin, FEC correction statistics, and module/cable health — needed to determine whether this is a marginal cable or a transceiver hardware issue?
mlxlink is the ConnectX/Quantum physical-layer diagnostic tool that exposes low-level signal integrity data — eye margins, FEC pre/post-correction error counts, and module (cable/transceiver) health — which is exactly what's needed to distinguish a marginal cable from a failing switch ASIC/port when logical link state still shows Active.
Ready for the real thing?
The full course has two full-length practice tests, video lessons for every exam domain, hands-on labs and detailed answer explanations.
Start my full course on Udemy →