Self-service capability for requesting BigQuery dataset creation — part1
In my previous blog posts, I have explained the scale of the data platform my team owns and manages on GCP. We manage almost all GCP data services and make them available for Data, Analytics, and Machine Learning Engineers. We also provide best practice recommendations for data engineering practices.
At the scale (100+ GCP projects with many data services and various IAM permissions granted in different environments) we operate, resource creation becomes a bottleneck for our customers. We started building self-service capabilities to alleviate this problem and make the data platform more user-friendly.
Our company uses Backstage for Internal Developer Platform (IDP) development. In the last 3 months, I designed and developed self-service templates for requesting services like these (and there are a few more to go). The templates are developed for the most frequently requested services by our customers.
- Creation of BigQuery datasets
- Creation of GCS buckets
- Granting “BigQuery Data Viewer” role on BigQuery datasets to AD groups and GCP Service Accounts
- Granting the “BigQuery Job User” role at the project level to GCP Service Accounts
- Granting the “BigQuery Read Session User” role at the project level to GCP Service Accounts
In this blog post, I will provide the details about the self-service template I developed for creating BigQuery datasets.
These are fields provided in the template.
- Domain name: This is the LoB present on our Cloud Data Platform.
- Zone name: Each domain has 3 zones (Landing, Processing, Curated).
- Purpose: Reason for creation of new BigQuery datasets.
- BigQuery dataset names: Names of the BigQuery datasets that must be created.
Once the details are entered, a PR will be submitted to the “IDP” repo with the terraform code that reads the input file and creates the BigQuery datasets accordingly.
These are the plan and apply GitHub Actions workflows that run for all 3 environments to create the resources.
Disclaimer: The posts here represent my personal views and not those of my employer or any specific vendor. Any technical advice or instructions are based on my own personal knowledge and experience.
