Designing data processing systems
Free Google Cloud Certified - Professional Data Engineer practice — 6 questions on Designing data processing systems, with explanations. No sign-up. Full 12-question mixed test →
An IoT platform ingests telemetry from 50 million devices into Cloud Bigtable using a row key of the form epoch_timestamp#device_id. Even though the cluster has 30 nodes, write throughput is bottlenecked on only a handful of nodes at any given moment. Which row key redesign BEST resolves this hotspotting while preserving the ability to query a device's recent readings efficiently?
Promoting the high-cardinality device_id to the front of the key spreads writes across the full tablet key range instead of all writes landing near the same monotonically-increasing timestamp prefix. Reversing the timestamp still lets you scan a device's most recent readings first with a simple prefix scan.
A global e-commerce company needs a single system of record for inventory that supports ACID transactions across multiple rows, SQL joins, horizontal write scalability as order volume grows, and 99.999% availability with multi-region writes. Which service should be selected?
Spanner is the only GCP database offering horizontally scalable, strongly consistent multi-row ACID transactions with SQL and a documented 99.999% multi-region SLA, making it the correct fit for global transactional inventory.
A team of BigQuery analysts with strong SQL skills but no Python or Java experience needs to build version-controlled, testable ELT transformation pipelines that run entirely inside BigQuery, with explicit dependency graphs between transformation steps. Which tool should they use?
Dataform is purpose-built for SQL-based ELT inside BigQuery: it provides dependency (DAG) management, version control integration, and built-in assertions/tests for SQL transformations without requiring general-purpose programming skills.
A 50 TB BigQuery table holds five years of order history. Nearly every query filters on order_date, and most queries also filter or aggregate by customer_region and product_category. customer_id has extremely high cardinality. Which table design BEST optimizes cost and performance?
Date partitioning lets BigQuery prune entire partitions for the near-universal order_date filter, and clustering on the two next-most-common filter columns further prunes blocks within each partition, minimizing bytes scanned.
A large enterprise has independent business units that each own and manage their own BigQuery datasets and GCS buckets under a data mesh model. Leadership wants centralized discovery, cataloging, data quality checks, and end-to-end lineage across all domains WITHOUT centralizing ownership or physically moving the data. Which GCP service should be used?
Dataplex provides a unified governance layer (catalog, data quality, lineage) that spans distributed BigQuery and GCS assets across business domains while leaving the underlying data in place and owned by its respective teams, which is exactly the data mesh governance pattern.
A company is migrating an on-premises Oracle database running heavy PL/SQL and sustaining 50,000 transactions per second. They want to minimize application rewrites by using a PostgreSQL-compatible interface, while requiring write throughput that scales horizontally beyond what a single primary instance can provide, with strong multi-row consistency. Which GCP service should they migrate to?
Spanner's PostgreSQL interface offers SQL/driver compatibility to ease migration while providing true horizontal write scalability across nodes with strong multi-row consistency, which is required at 50,000 TPS beyond any single-instance limit.
Ready for the real thing?
The full course: two full-length practice tests, video lessons for every exam domain, hands-on labs and detailed explanations.
$129.99 $34.99 with code FREETEST33 — valid through September 16.
Get my $34.99 deal →