Free Optimize generative AI systems and model performance Practice Questions — Microsoft MLOps Engineer Associate (AI-300)

Question 1 of 6 · Optimize generative AI systems and model performance

A financial services company has deployed a RAG system using Azure AI Search with semantic ranking for regulatory document retrieval. Users report that queries about recent policy changes (2025-2026) return older documents from 2020-2022 with high confidence scores, while relevant recent documents rank lower. The embedding model is text-embedding-ada-002, chunk size is 512 tokens with 50-token overlap, and similarity threshold is 0.78. Which optimization will MOST effectively address this relevance issue?

Hybrid search with recency weighting directly addresses the temporal relevance problem. The alpha parameter in Azure AI Search hybrid search (typically 0.5-0.7 for balanced weighting) allows combining semantic similarity with keyword/metadata filters. Applying exponential decay or linear boosting based on document timestamp ensures recent documents receive higher scores even with slightly lower semantic similarity. This is the recommended Azure AI Search pattern for time-sensitive domains and doesn't require model retraining.

Question 2 of 6 · Optimize generative AI systems and model performance

An MLOps team is fine-tuning GPT-4o-mini on Azure OpenAI Service for a customer support chatbot using 800 real support ticket conversations. After deployment, the fine-tuned model shows 15% higher accuracy on internal test cases but generates verbose responses that exceed token limits in 30% of production requests, causing failures. The team used default training hyperparameters (learning_rate_multiplier: 0.1, batch_size: auto, n_epochs: 3). What is the BEST approach to resolve this issue while maintaining accuracy gains?

This approach uses synthetic data augmentation (a key AI-300 fine-tuning technique) to correct the undesired behavior while preserving learned knowledge. Creating 200 synthetic examples (20-25% of dataset) with explicit brevity patterns, reducing learning rate (0.05 vs 0.1) and epochs (2 vs 3) prevents the new behavior from overwriting previous learning too aggressively. This is the recommended pattern for behavioral correction in production fine-tuned models without starting over.

Question 3 of 6 · Optimize generative AI systems and model performance

A manufacturing company's RAG system retrieves technical documentation using Azure AI Search with a custom embedding model deployed on Azure Machine Learning. The system uses 256-token chunks with no overlap, cosine similarity, and returns the top 10 chunks. Evaluation shows precision@10 of 0.42 and recall@10 of 0.68 on a labeled test set of 500 queries. The team has a budget to optimize ONE component. Which single optimization provides the HIGHEST expected improvement in both precision and recall?

Fine-tuning the embedding model with domain-specific data addresses the root cause of poor precision and recall. Contrastive learning with hard negatives (documents that are lexically similar but semantically irrelevant) is the most impactful RAG optimization technique. With 10,000 labeled pairs, the embedding model learns domain-specific semantic relationships, improving both precision (fewer irrelevant results in top-10) and recall (relevant docs score higher). This is the recommended approach in Azure ML documentation for domain-specific RAG optimization.

Question 4 of 6 · Optimize generative AI systems and model performance

An AI team deployed a fine-tuned GPT-4o model for legal contract analysis on Azure OpenAI. After two months in production, they observe that model accuracy has decreased from 89% to 76% on new contract types introduced in Q1 2026, while maintaining 88% accuracy on contract types from the training data (2024-2025). The team has collected 1,200 new labeled examples from Q1 2026 failures. They want to update the model without losing performance on original contract types. What is the BEST strategy that follows Azure OpenAI fine-tuning best practices?

This approach prevents catastrophic forgetting by including representative samples from the original training distribution (1:1 ratio with new data). Retraining from the base model with the combined dataset ensures the model learns both old and new patterns with equal weight. This is the Azure OpenAI recommended approach for addressing distribution shift while maintaining original capabilities. Validation on both distributions confirms no regression. AI-300 emphasizes this pattern for production model updates.

Question 5 of 6 · Optimize generative AI systems and model performance

A healthcare organization's RAG system uses Azure AI Search with 1,024-token chunks and text-embedding-3-large embeddings (3072 dimensions). A production evaluation framework measures answer quality using LLM-as-judge with GPT-4 scoring answers 1-5 for accuracy and relevance. After analyzing 2,000 queries, they find: 78% of queries score ≥4, but queries requiring information from multiple document sections score an average of 2.1. Retrieved chunks have high individual relevance (avg 0.82 cosine similarity) but don't contain the full information span needed. What optimization BEST addresses this multi-span information retrieval problem?

This addresses the multi-span problem through smaller chunks with overlap (captures boundary information) combined with parent document retrieval—a proven RAG pattern where retrieving a chunk triggers fetching surrounding chunks from the same document to reconstruct broader context. This maintains granular matching (high precision) while providing the complete information span needed. The 25% overlap ensures concepts split across boundaries are captured. This is documented in Azure AI Search best practices for complex information needs.

Question 6 of 6 · Optimize generative AI systems and model performance

An e-commerce company fine-tuned GPT-4o-mini for product description generation using 5,000 human-written descriptions. Production monitoring shows the fine-tuned model generates descriptions in 1.2 seconds (p95 latency) at $0.003 per request, while the baseline GPT-4o-mini with a 5-shot prompt achieves similar quality in 2.1 seconds at $0.008 per request. The business now requires support for 12 additional product categories (only 150 examples each exist). Management wants to minimize retraining cost and time-to-production. Which approach BEST balances performance, cost, and implementation speed?

This approach uses synthetic data generation (a key AI-300 technique for addressing data scarcity) to augment the limited 150 examples per category to ~1,000, providing sufficient training signal. Including a stratified sample from original data prevents catastrophic forgetting. Lower learning rate (0.05) and fewer epochs (2) balance learning new patterns without overwriting existing knowledge. Synthetic data generation with quality filtering is the recommended Azure OpenAI approach for expanding fine-tuned model coverage with limited real data, and maintains the performance and cost advantages of fine-tuning vs prompting.