Data Ingestion and Transformation
Free AWS Certified Data Engineer - Associate practice — 6 questions on Data Ingestion and Transformation, with explanations. No sign-up. Full 12-question mixed test →
A data engineering team has three separate applications (a Lambda function, an EMR Spark streaming job, and a Kinesis Data Analytics application) all consuming from the same Kinesis Data Stream with 8 shards. During peak load, all three consumers report ReadProvisionedThroughputExceeded errors even though the producer write rate is well within limits. Which change resolves this with the LEAST operational complexity?
Without Enhanced Fan-Out (EFO), all consumers of a shard share the fixed 2 MB/s (or 5 transactions/second) read throughput. With three independent consumers competing for that same shared pool, throttling occurs regardless of shard count. Registering each consumer with EFO gives each one its own dedicated 2 MB/s per shard via HTTP/2 push, eliminating contention without needing to re-shard.
A company needs to migrate an on-premises Oracle database (2 TB) to Amazon S3 as Parquet files, and then continue replicating ongoing row-level inserts, updates, and deletes from Oracle to S3 in near real time until cutover. Which AWS service and configuration BEST meets this requirement?
AWS DMS 'Full Load and CDC' migration type performs the initial bulk load of existing data and then continuously captures ongoing changes (inserts/updates/deletes) from the source database's transaction/redo logs, applying them to the S3 target in near real time — exactly the pattern described.
An AWS Glue crawler runs nightly against an S3 data lake where the upstream producer occasionally adds new optional columns to the Parquet files but never changes or removes existing columns. The team wants the Data Catalog table to automatically pick up these new columns without the crawler ever altering the data types of existing columns or removing partitions if files are temporarily missing. Which crawler configuration should be selected?
'Add new columns only' updates the catalog schema when new fields appear but never modifies or drops existing column definitions, matching the requirement. Pairing it with 'Ignore the change and don't update the table' for deletion behavior prevents partitions from being removed from the catalog if source files are temporarily absent, avoiding accidental partition loss.
A company has an existing on-premises application built directly on the Apache Kafka Producer/Consumer API, using custom consumer group logic and exactly-once semantics via Kafka transactions. They want to move this workload to AWS with minimal rewriting of the application code. Which service should they migrate to?
Amazon MSK is a managed Apache Kafka service that is API-compatible with open-source Kafka, including consumer groups and Kafka transactions for exactly-once semantics. Applications built on the native Kafka client libraries can point to an MSK cluster with little to no code change.
During an AWS Glue Studio ETL job design session, a developer is deciding whether to use a DynamicFrame or convert immediately to a Spark DataFrame. The source data has inconsistent field types across files (a 'price' field is sometimes a string and sometimes a double) and the exact schema is not known ahead of time. Which statement correctly explains why DynamicFrame is the appropriate starting point?
A DynamicFrame is designed for semi-structured data where the schema is not fully known or is inconsistent across records. It computes a schema on the fly and supports transforms like ResolveChoice (e.g., cast, make_struct, project) specifically to handle a field that appears with multiple types across records — exactly the 'price' scenario described.
An EMR cluster running a Spark ETL job processes a 500 GB join between two large datasets. The job frequently spills to disk during the shuffle stage and produces tens of thousands of tiny output files in S3, causing downstream Athena queries to run slowly. The team wants Spark to automatically coalesce shuffle partitions to a more efficient number based on actual data size, without manually recalculating spark.sql.shuffle.partitions for every job. Which configuration should they enable?
Adaptive Query Execution (AQE), enabled via spark.sql.adaptive.enabled=true, allows Spark to inspect actual shuffle statistics at runtime and automatically coalesce small shuffle partitions into fewer, appropriately sized partitions — directly reducing the small-file problem and improving performance without manual tuning per job.
Ready for the real thing?
The full course: two full-length practice tests, video lessons for every exam domain, hands-on labs and detailed explanations.
$109.99 $34.99 with code FREETEST33 — valid through September 14.
Get my $34.99 deal →