Data Preparation
Free Databricks Certified GenAI Engineer Associate practice — 6 questions on Data Preparation, with explanations. No sign-up. Full 12-question mixed test →
A data engineering team is preparing a dataset of customer support tickets for fine-tuning a generative model. The raw data contains a 'resolution_notes' column with values ranging from empty strings to 50,000+ character narratives. During initial training runs, the team observes inconsistent model performance and out-of-memory errors. Which preprocessing strategy BEST addresses this issue while preserving information quality for fine-tuning?
Semantic chunking with overlap preserves context across boundaries, filtering extreme outliers prevents OOM errors, and creating separate examples from chunks maximizes information retention while maintaining trainable batch sizes. This approach balances token limits (typically 4K-32K for modern models) with data quality.
A generative AI application requires a vector database to support retrieval-augmented generation (RAG) across 40 million technical documentation chunks. The team has already generated embeddings using a 1536-dimensional model. They need to optimize for sub-100ms p95 query latency while supporting metadata filtering on 5 categorical fields. Which Databricks-native solution BEST meets these requirements?
Databricks Vector Search provides managed ANN indexing (HNSW-based) with native Delta Sync for automatic updates, sub-100ms query performance at scale, and built-in metadata filtering. The delta-sync index ensures the vector store stays current with source Delta tables without manual intervention.
During data preparation for a domain-specific chatbot, a data scientist discovers that 15% of training examples contain personally identifiable information (PII) including email addresses, phone numbers, and account IDs embedded within conversational context. The organization's compliance policy requires PII removal, but preserving conversational structure is critical for model quality. Which approach BEST balances compliance with training effectiveness?
Entity token replacement removes PII while preserving conversational structure, sentence length, and context flow. The model learns to handle placeholder tokens that can be replaced with appropriate values at inference time. This is standard practice in PII-aware NLP preprocessing.
A team is preparing a training dataset for instruction fine-tuning a coding assistant. The raw data consists of code snippets paired with natural language descriptions, but 30% of snippets contain syntax errors, deprecated API calls, or incomplete logic. The team wants to improve dataset quality before fine-tuning. Which validation strategy provides the MOST effective quality control?
Static analysis provides deterministic, scalable validation. Linters catch syntax errors, deprecation checks verify API currency, and automated tests validate logic. This creates an objective, reproducible quality filter that directly addresses the three identified error types without requiring labeled data or manual review.
An organization is building a RAG system that requires splitting 250,000 PDF documents (technical manuals averaging 400 pages each) into semantically coherent chunks for embedding. The documents contain complex tables, multi-level numbered lists, code blocks, and cross-references. Initial tests with fixed 512-token chunking resulted in fragmented tables and broken code samples. Which chunking strategy BEST preserves semantic coherence for this document type?
Structure-aware parsing respects document semantics by treating tables and code as atomic units (preventing fragmentation), using header hierarchy for natural boundaries, and allowing variable chunk sizes to accommodate complete semantic units. The 256-2048 token range handles both brief sections and complex diagrams while overlap at transitions preserves cross-reference context.
A data preparation pipeline processes streaming user feedback for real-time model monitoring. The pipeline must detect and flag potentially toxic content before it enters the evaluation dataset. The current approach uses a pre-trained toxicity classifier that runs on each message, but processing latency has increased to 800ms per message as volume scaled to 50,000 messages/hour. Which optimization provides the MOST effective latency reduction while maintaining detection accuracy?
Micro-batching with GPU acceleration leverages parallel processing to amortize model loading overhead and exploit GPU's parallel compute. Processing 100 messages simultaneously can reduce per-message latency from 800ms to <50ms while maintaining full classifier accuracy. This is the standard approach for scaling neural inference.
Ready for the real thing?
The full course: two full-length practice tests, video lessons for every exam domain, hands-on labs and detailed explanations.
$199.99 $66.99 with code FREETEST33 — valid through August 23.
Get my $66.99 deal →