Prepare and ingest data
Free Google Cloud Associate Data Practitioner practice — 6 questions on Prepare and ingest data, with explanations. No sign-up. Full 12-question mixed test →
A retail company runs its order-management system on a self-managed MySQL database hosted on-premises. They need to replicate this data into BigQuery with minimal latency (under 5 minutes) for near real-time reporting, while keeping the on-premises database as the system of record and avoiding any custom code for change capture. Which approach BEST meets these requirements?
Datastream is Google Cloud's purpose-built change data capture (CDC) service for sources like MySQL, Oracle, and PostgreSQL. It captures inserts, updates, and deletes in near real time without custom code, and the standard pattern pairs it with the Dataflow Datastream-to-BigQuery template to continuously replicate changes into BigQuery with low latency.
A ride-sharing application publishes GPS location updates for each driver to a Pub/Sub topic. Downstream, a Dataflow job must process each driver's location updates in the exact sequence they were generated, but updates from different drivers can be processed in any order relative to each other. Which Pub/Sub configuration achieves this?
Pub/Sub guarantees ordering only within a given ordering key when message ordering is enabled on the subscription. Setting the ordering key to the driver ID ensures all updates for a single driver are delivered in publish order, while updates across different drivers can still be processed independently and in parallel, matching the stated requirement.
A manufacturing company has 300 TB of historical sensor log files stored on an on-premises NAS file server. They need a one-time bulk transfer of this data into Cloud Storage, followed by ongoing nightly incremental syncs of new files only, without writing custom scripts to track which files have already been transferred. Which service should they use?
Storage Transfer Service supports on-premises data sources through a downloadable transfer agent that reads from POSIX-compliant file systems like a NAS share. It natively supports both one-time bulk transfers and scheduled, incremental syncs that automatically detect new or changed files, eliminating the need for custom tracking scripts.
A marketing analytics team needs daily Google Ads campaign performance data loaded automatically into a BigQuery dataset for reporting, with no custom pipeline code and minimal ongoing maintenance. Which service should they configure?
BigQuery Data Transfer Service includes a built-in, managed connector for Google Ads that automatically schedules and loads campaign data into BigQuery tables with no code required — exactly matching the 'no custom pipeline' and 'minimal maintenance' requirements.
A Dataflow pipeline ingests clickstream events containing deeply nested and repeated fields (e.g., a list of product views per session) and writes the output to Cloud Storage before a separate scheduled job loads the files into BigQuery. The team wants a file format that preserves the nested/repeated schema natively, supports schema evolution as new fields are added, and loads efficiently into BigQuery. Which file format should they choose?
Avro stores an embedded schema alongside the data, natively supports nested and repeated (record and array) structures, and supports schema evolution (adding fields with defaults) in a way BigQuery understands directly during load, making it an efficient and robust choice for semi-structured, nested pipeline output destined for BigQuery.
A data analyst receives a one-time request to run a handful of exploratory SQL queries against 50 GB of Parquet files sitting in Cloud Storage. The files will never be queried again after this analysis, and the analyst wants to avoid any unnecessary data duplication or storage cost. What is the BEST approach?
A BigQuery external table lets you query data directly in Cloud Storage without copying or loading it into BigQuery-managed storage. For one-time, exploratory analysis where the data won't be reused, this avoids the storage duplication and load time that a native table would introduce, directly matching the stated goal of avoiding unnecessary cost and duplication.
Ready for the real thing?
The full course has two full-length practice tests, video lessons for every exam domain, hands-on labs and detailed answer explanations.
Start my full course on Udemy →