Orchestrate data pipelines
Free Google Cloud Associate Data Practitioner practice — 6 questions on Orchestrate data pipelines, with explanations. No sign-up. Full 12-question mixed test →
A data engineering team runs two Cloud Composer DAGs: `ingest_orders` (hourly) and `transform_orders` (hourly, offset by 30 minutes). `transform_orders` must not start until the matching hourly run of `ingest_orders` has completed successfully, but both DAGs must remain independently deployable, and each must be able to retry on its own schedule without being merged. Which approach BEST meets this requirement?
ExternalTaskSensor lets one DAG wait on the completion state of a specific run of another DAG, matched by execution date/delta, while keeping the two DAGs fully independent in code, deployment, and retry logic.
A team wants Dataform to compile their SQLX code once per day and then execute only the tables tagged 'daily' on that schedule, without introducing Cloud Composer or any external scheduler. Which Dataform feature should they configure?
Dataform's native release configurations handle scheduled compilation, and workflow configurations schedule execution of specific tags against a release's compiled output — this is the built-in, lowest-overhead way to meet the requirement with no external orchestrator.
A pipeline must call three REST APIs in strict sequence, where each call's input depends on the prior call's JSON response, includes conditional branching based on HTTP status codes, and completes in under 5 minutes with simple retry/backoff. There is no dependency on BigQuery, Dataflow, or Dataform jobs. Which service is the BEST fit?
Cloud Workflows is purpose-built for exactly this pattern: serverless, YAML/JSON-defined sequential HTTP calls with built-in conditionals, variable passing between steps, and retry policies — without the overhead of standing up an Airflow environment.
A daily BigQuery scheduled query failed for 3 consecutive days due to a transient upstream service outage that has since been resolved. The team wants to reprocess only those 3 missed days without changing the existing daily schedule or configuration. What should they do?
BigQuery Data Transfer Service (which underlies scheduled queries) provides a manual 'Schedule backfill' capability to trigger runs for a specific historical date range without altering the existing recurring schedule.
A Cloud Composer task invokes an external API that occasionally fails due to transient throttling. The team wants the task itself to automatically retry with exponentially increasing wait times before marking the task (and therefore the DAG run) as failed. Which Airflow task parameters should be configured?
Airflow's per-task retries and retry_delay parameters, combined with retry_exponential_backoff=True, implement automatic retries with increasing wait intervals before a task is marked failed — exactly the behavior requested.
In a Cloud Composer DAG, task A runs a Dataflow job and task B runs a Dataform workflow invocation downstream of A. A third task, C, sends a pipeline-status notification to Slack and must run whenever the DAG finishes, regardless of whether A or B succeeded or failed. Which trigger_rule should task C use?
all_done triggers the task once all upstream tasks have finished executing, regardless of whether they succeeded or failed, which matches the requirement to always send a status notification.
Ready for the real thing?
The full course has two full-length practice tests, video lessons for every exam domain, hands-on labs and detailed answer explanations.
Start my full course on Udemy →