ETL and Distributed Processing
Free NVIDIA-Certified Professional: Accelerated Data Science practice — 6 questions on ETL and Distributed Processing, with explanations. No sign-up. Full 12-question mixed test →
A team must join two datasets totaling 1.5TB on a single DGX node with 8x A100 80GB GPUs (640GB aggregate GPU memory). The join triggers a large shuffle that temporarily needs more memory than fits on any single GPU. Which configuration BEST handles this on the single node?
1.5TB comfortably fits within 640GB aggregate GPU memory when combined with spilling, and NVLink-connected GPUs on a single DGX node give the fastest intra-node shuffle bandwidth — no need to add nodes.
An out-of-core groupby-aggregation on a dataset larger than aggregate GPU memory keeps failing with GPU out-of-memory errors on a dask-cuda LocalCUDACluster. Which cluster configuration correctly enables automatic spilling from GPU memory to host RAM and then to disk as needed?
device_memory_limit sets the GPU memory threshold that triggers spilling, and jit_unspill enables the just-in-time unspilling mechanism that can move data to host memory and further to disk, allowing the workload to exceed aggregate GPU memory safely.
A 500GB dataset will be written to Parquet for later ingestion by dask_cudf on GPUs with 32GB of memory each. Which write/read strategy minimizes GPU memory overhead while maximizing read throughput?
Row groups in the 128-256MB range balance GPU memory usage against per-partition scheduling overhead, and split_row_groups lets dask_cudf control how row groups are combined into partitions sized appropriately for 32GB GPUs.
An organization needs to run daily ETL processing 50TB of Parquet data with heavy shuffle-based joins and sorts. They have 4 DGX A100 nodes (8 GPUs per node, NVLink intra-node, InfiniBand inter-node). Which setup best supports this throughput requirement?
50TB with heavy shuffles benefits from scaling across all available GPUs, and UCX-enabled dask-cuda automatically selects the fastest available transport (NVLink within a node, InfiniBand between nodes), maximizing shuffle throughput across the full cluster.
When repartitioning a dask_cudf DataFrame before a memory-intensive shuffle operation like a sort-based join, what partition sizing guideline best balances GPU utilization against out-of-memory risk during the shuffle?
Shuffle operations like sorts and joins require temporary buffers on top of the base partition data; sizing partitions to a fraction of GPU memory leaves headroom for these intermediates, preventing OOM while still keeping partition counts reasonable for scheduling efficiency.
A 2TB ORC dataset has 50 columns, but a downstream GPU pipeline only needs 3 columns and rows within a specific date range. Which approach minimizes I/O and GPU memory usage when loading this with dask_cudf?
Projecting columns at read time via the columns parameter avoids decoding and transferring the other 47 columns from disk into GPU memory, and applying date filtering on the reduced projection minimizes both I/O and GPU memory footprint before heavier processing.
Ready for the real thing?
The full course: two full-length practice tests, video lessons for every exam domain, hands-on labs and detailed explanations.
$129.99 $34.99 with code FREETEST33 — valid through September 9.
Get my $34.99 deal →