We have built our GCP data platform using Data Mesh principles. Each line-of-business (LoB) roughly translates to a domain on GCP. Each domain will have 3 environments, Dev, Non-Prod and Prod, and each environment will have 3 GCP projects/zones. This is the mapping between GitHub, Terraform and GCP domains. I…

Once the services/resources (like GCS bucket, BigQuery dataset) are deployed to GCP, permissions are needed to access those services/resources. We can grant access to Google Cloud services/resources by using allow policies, also known as Identity and Access Management (IAM) policies, which are attached to services/resources. A Policy is a collection…

I used Terraform to deploy these resources. GCS bucket Azure Data Lake Storage account Azure Data Factory pipeline with these components Linked Service for GCS bucket Dataset for GCS bucket Linked Service for ADLS account Dataset for ADLS storage container Pipeline with Copy Data activity to move the delimited text…

Streaming live data from on-prem SQL Server to GCP BigQuery can’t be done directly. A staging area is needed for the data before getting loaded into BigQuery. This can be achieved in different ways. Use 3rd party tools to build the pipelines for streaming the live data from on-prem SQL…

We have a pipeline in our ADF instance, which pulls data on a daily basis from 300+ tables present on a cloud data source. This pipeline gets hung for a table (it can be any table) at least once a week. When this happens, many tables get stuck behind that…

Data is pulled is from various sources (some data sources are from on-prem and some are from other clouds like ServiceNow) into our cloud data warehouse using Azure Data Factory (ADF) pipelines. ADF pipelines run on daily basis for all data sources. There is an ADF pipeline which runs every…

We manage and maintain data warehouse on Azure SQL managed instance (We are migrating the DW to Azure Synapse Analytics). We pull data from various data sources into Azure DW. One of the data sources is ServiceNow. We maintain several ServiceNow tables in Azure database, and run nightly Azure Data…

Maintaining Azure Data Factory (ADF) pipelines for data sources having frequent schema drift is always challenging and cumbersome most of the times. We have multiple ADF pipelines pulling data from various data sources. And, metadata changes happen frequently for the tables (like new columns added or changed) in the data…

Google provides $300 free Trial period which lasts for 90 days. I activated my free trial and started looking/learning around in Google Cloud Platform (GCP) console. FYI, this is the resource hierarchy in 3 cloud service providers. Azure: Root Management group → Management groups → Subscriptions → Resource groups →…