Ingesting and processing the data
Free Google Cloud Certified - Professional Data Engineer practice — 6 questions on Ingesting and processing the data, with explanations. No sign-up. Full 12-question mixed test →
A streaming pipeline processes IoT sensor data using fixed 1-minute windows in Apache Beam on Dataflow. Some events arrive up to 10 minutes late due to unreliable field network connections. The team needs windows to emit an early result quickly, then update with a final corrected result once late data arrives, without firing on every single late element unnecessarily. Which configuration BEST meets this requirement?
Allowed lateness (withAllowedLateness) extends the window's data-driven trigger horizon so late data is still accepted and merged into the pane. Combining this with an early trigger (e.g., processing-time trigger for a fast provisional result) and a late-firing trigger produces controlled emission of an early estimate followed by a corrected final result, using accumulating mode so later panes include all prior data rather than just deltas.
A pipeline reads from Pub/Sub, processes records in Dataflow, and writes streaming inserts to a BigQuery table. The team must guarantee that no record is processed or written more than once end-to-end, even if workers restart or retries occur. Which statement correctly describes how to achieve this?
Dataflow's runner guarantees exactly-once processing of records within the pipeline itself (it dedupes retried bundles internally). For the final sink write, BigQueryIO's streaming insert API uses an insertId per row so that BigQuery's dedup logic on the streaming buffer suppresses duplicate rows from retried writes, achieving effectively-once semantics end-to-end.
A company is migrating an on-premises Oracle database to BigQuery and needs continuous low-latency replication of inserts, updates, and deletes with minimal load placed on the source database. Which approach is the current recommended solution using Google Cloud managed CDC tooling?
Datastream supports BigQuery as a direct replication destination, natively capturing change events (inserts/updates/deletes) from Oracle's redo logs with minimal source impact and automatically applying merges into BigQuery tables, removing the need for a separate Dataflow merge pipeline for the common case.
A team is migrating on-premises Spark ETL jobs that each run for about 30 minutes, several times per day, with idle time between runs. They currently use a persistent Dataproc cluster and want to minimize cost without managing cluster lifecycle logic. What should they do?
Dataproc Serverless for Spark eliminates cluster provisioning and management entirely — Google runs each batch job on transient infrastructure and bills only for the resources consumed during execution, which is ideal for short, periodic jobs with idle gaps and requires no autoscaling policy tuning.
A Cloud Composer DAG has a downstream task that must run only after a file lands in a Cloud Storage bucket, but the file arrival time is unpredictable and can occur at any point during the day. The team wants to minimize wasted compute and worker-slot usage while reacting quickly to file arrival. What is the BEST design?
Decoupling the trigger from any polling mechanism by using a GCS event notification through Pub/Sub and a Cloud Function calling the Airflow REST API creates a true event-driven trigger, minimizing latency between file arrival and DAG execution while consuming zero compute while waiting.
A streaming Dataflow pipeline ingests JSON events from Pub/Sub. Occasionally, upstream producers send malformed or schema-drifted records that cause exceptions in the parsing DoFn, and currently these exceptions cause bundle-level failures that stall the pipeline. The team needs the pipeline to keep processing valid records while capturing failed records for later inspection. What should they implement?
Catching exceptions inside the DoFn and using a multi-output ParDo (TupleTags) to split successful and failed elements into separate PCollections is the standard dead-letter pattern in Beam/Dataflow. It isolates malformed records for review or reprocessing without stalling the main pipeline, directly addressing schema drift robustness.
Ready for the real thing?
The full course: two full-length practice tests, video lessons for every exam domain, hands-on labs and detailed explanations.
$129.99 $34.99 with code FREETEST33 — valid through September 16.
Get my $34.99 deal →