Transformer Architecture and Large Language Models

Free NVIDIA Generative AI & LLMs (NCA-GENL) practice — 6 questions on Transformer Architecture and Large Language Models, with explanations. No sign-up. Full 12-question mixed test →

Question 1 of 6 · Transformer Architecture and Large Language Models

A research team is fine-tuning a decoder-only transformer model for code generation. During training, they notice the model performs well on short code snippets but struggles with generating complete functions exceeding 2048 tokens. The team wants to maintain computational efficiency while enabling the model to handle longer contexts. Which architectural modification would BEST address this requirement?

ALiBi removes absolute positional embeddings and uses linear biases in attention scores, allowing the model to extrapolate to sequences longer than those seen during training without architectural changes or retraining. This maintains computational efficiency while enabling better length generalization.

Question 2 of 6 · Transformer Architecture and Large Language Models

An AI startup is deploying a mixture-of-experts (MoE) language model with 8 expert networks and a gating mechanism. During inference, they observe that only 2 of the 8 experts are consistently activated for 90% of inputs, while the other 6 experts remain underutilized. This is causing suboptimal model performance. What is the MOST likely cause of this expert collapse?

Expert collapse in MoE models typically occurs when the auxiliary load balancing loss is too weak. This loss term encourages equal distribution of inputs across experts. When set too low, the router learns to favor certain experts, creating a feedback loop where popular experts get better training signal and become even more preferred.

Question 3 of 6 · Transformer Architecture and Large Language Models

A financial services company needs to deploy a large language model for contract analysis. Their compliance requirements mandate that they must be able to completely remove all knowledge about specific individuals from the model after processing. They are evaluating different architectures. Which approach would BEST meet this requirement?

RAG systems store knowledge externally in retrievable databases rather than encoded in model weights. This allows complete and verifiable deletion of specific information by removing entries from the vector database, while the base model remains unchanged. This is the only architecture that provides true 'machine unlearning' capability.

Question 4 of 6 · Transformer Architecture and Large Language Models

During the development of a multilingual translation model, engineers implement grouped-query attention (GQA) to reduce memory consumption. The model has 48 attention heads and they configure 8 key-value head groups. After deployment, they observe that translation quality for low-resource languages has degraded compared to the baseline multi-head attention model, while high-resource languages maintain quality. What is the MOST effective solution to address this issue?

Language-specific adapter layers add trainable parameters for each language while keeping the core GQA mechanism intact. This allows low-resource languages to have dedicated representational capacity without increasing the memory costs of the attention mechanism itself. Adapters have proven effective for multilingual models where different languages have varying complexity.

Question 5 of 6 · Transformer Architecture and Large Language Models

A machine learning team is implementing flash attention for their transformer model training pipeline. They need to process sequences of 16,384 tokens with a batch size of 32 on GPUs with 40GB of memory. The standard attention implementation causes out-of-memory errors. After implementing flash attention, they still experience memory issues during backward pass. What is the MOST likely configuration error?

Flash attention optimizes memory during the forward pass by avoiding materialization of the full attention matrix. However, the backward pass still requires attention matrices. Without gradient checkpointing, all intermediate activations must be stored. Flash attention with gradient checkpointing recomputes attention during backward pass, trading computation for memory—this combination is essential for long sequences.

Question 6 of 6 · Transformer Architecture and Large Language Models

A healthcare AI company is evaluating whether to use sparse attention patterns for their clinical notes processing model. Their notes average 32,768 tokens with critical information distributed throughout the document. They are comparing different sparse attention patterns: local attention (fixed window), stride attention (dilated), and global attention (dedicated global tokens). Which pattern would provide the BEST balance between computational efficiency and the ability to capture long-range dependencies in clinical documentation?

Clinical notes have structured sections (diagnosis, history, medications, etc.) where section headers contain critical organizational information. Longformer's hybrid approach uses efficient local attention for detailed context while designated global tokens (section headers) can attend to and be attended by all tokens, enabling long-range dependencies. This matches the document structure and information flow of clinical notes.

Ready for the real thing?

The full course: two full-length practice tests, video lessons for every exam domain, hands-on labs and detailed explanations.

$54.99 $17.99 with code FREETEST33 — valid through August 23.

Get my $17.99 deal →