I’m part of a team that owns & manages the platform for an in-house built GCP native data product. This data product is designed based on Data Mesh Architecture. Data product is made up of domains. This is the architectural design of a domain. …

We provision access to GCP services (like GCS buckets, BigQuery datasets) through Terraform very frequently. Access is provisioned to both Azure AD groups (synced to GCP) and GCP service accounts. We execute code blocks like this in Terraform for access provisioning. resource “google_project_iam_member” “project_permissions_base” { member = “group:aad_group@mycompany.com” project =…

Google Cloud Platform (GCP) provides a PaaS service called Cloud SQL, which provides a cloud-based alternative to local MySQL, PostgreSQL, and SQL Server databases. For SQL Server, not all Microsoft provided features are supported on Cloud SQL. …

We have built our GCP data platform using Data Mesh principles. Each line-of-business (LoB) roughly translates to a domain on GCP. Each domain will have 3 environments, Dev, Non-Prod and Prod, and each environment will have 3 GCP projects/zones. This is the mapping between GitHub, Terraform and GCP domains. I…

Once the services/resources (like GCS bucket, BigQuery dataset) are deployed to GCP, permissions are needed to access those services/resources. We can grant access to Google Cloud services/resources by using allow policies, also known as Identity and Access Management (IAM) policies, which are attached to services/resources. A Policy is a collection…

I used Terraform to deploy these resources. GCS bucket Azure Data Lake Storage account Azure Data Factory pipeline with these components Linked Service for GCS bucket Dataset for GCS bucket Linked Service for ADLS account Dataset for ADLS storage container Pipeline with Copy Data activity to move the delimited text…

Streaming live data from on-prem SQL Server to GCP BigQuery can’t be done directly. A staging area is needed for the data before getting loaded into BigQuery. This can be achieved in different ways. Use 3rd party tools to build the pipelines for streaming the live data from on-prem SQL…

We have a pipeline in our ADF instance, which pulls data on a daily basis from 300+ tables present on a cloud data source. This pipeline gets hung for a table (it can be any table) at least once a week. …

Data is pulled is from various sources (some data sources are from on-prem and some are from other clouds like ServiceNow) into our cloud data warehouse using Azure Data Factory (ADF) pipelines. ADF pipelines run on daily basis for all data sources. There is an ADF pipeline which runs every…