Bharath N – Medium

May 7

Authoritative Vs non-Authoritative Terraform resources for GCP IAM policies

Once the services/resources (like GCS bucket, BigQuery dataset) are deployed to GCP, permissions are needed to access those services/resources. We can grant access to Google Cloud services/resources by using allow policies, also known as Identity and Access Management (IAM) policies, which are attached to services/resources. A Policy is a collection…

Google Cloud Platform

3 min read

Authoritative Vs non-Authoritative Terraform resources for GCP IAM policies

Mar 5

Deploy Azure Data Factory to copy data from GCP to Azure using Terraform

I used Terraform to deploy these resources. GCS bucket Azure Data Lake Storage account Azure Data Factory pipeline with these components Linked Service for GCS bucket Dataset for GCS bucket Linked Service for ADLS account Dataset for ADLS storage container Pipeline with Copy Data activity to move the delimited text…

Terraform

2 min read

Feb 28

Stream live data from on-prem SQL Server to Google Cloud SQL for SQL Server using transactional replication

Streaming live data from on-prem SQL Server to GCP BigQuery can’t be done directly. A staging area is needed for the data before getting loaded into BigQuery. This can be achieved in different ways. Use 3rd party tools to build the pipelines for streaming the live data from on-prem SQL…

Sql Server

4 min read

Nov 23, 2021

Process automation to handle long-running (hung) pipelines in Azure Data Factory (ADF)

We have a pipeline in our ADF instance, which pulls data on a daily basis from 300+ tables present on a cloud data source. This pipeline gets hung for a table (it can be any table) at least once a week. …

Azure Data Factory

2 min read

Oct 9, 2021

Hardening our Azure Data Factory (ADF) pipelines

Data is pulled is from various sources (some data sources are from on-prem and some are from other clouds like ServiceNow) into our cloud data warehouse using Azure Data Factory (ADF) pipelines. ADF pipelines run on daily basis for all data sources. There is an ADF pipeline which runs every…

Azure Data Factory

6 min read

Sep 27, 2021

Automated approach for creating ServiceNow tables in Azure database

We manage and maintain data warehouse on Azure SQL managed instance (We are migrating the DW to Azure Synapse Analytics). We pull data from various data sources into Azure DW. One of the data sources is ServiceNow. …

Servicenow

3 min read

Jul 10, 2021

Handling schema drift for tables in Azure Synapse dedicated pool

Maintaining Azure Data Factory (ADF) pipelines for data sources having frequent schema drift is always challenging and cumbersome most of the times. We have multiple ADF pipelines pulling data from various data sources. …

Azure Data Factory

4 min read

Handling schema drift for tables in Azure Synapse dedicated pool

Apr 3, 2021

Data copy from ADLS Gen2 and AWS S3 accounts into Google Storage bucket

Google provides $300 free Trial period which lasts for 90 days. I activated my free trial and started looking/learning around in Google Cloud Platform (GCP) console. FYI, this is the resource hierarchy in 3 cloud service providers. Azure: Root Management group → Management groups → Subscriptions → Resource groups →…

Azure Blob

5 min read

Data copy from ADLS Gen2 and AWS S3 accounts into Google Storage bucket

Mar 26, 2021

Migrate Azure SQL single database to AWS RDS for SQL Server using AWS DMS

In my earlier blog post, I explained the process to migrate a database from AWS RDS to Azure SQL using Azure DMS. In this blog post, you shall see the process involved in migrating a database from Azure SQL to AWS RDS for SQL Server using AWS DMS. The process…

Azure Sql Database

5 min read

Migrate Azure SQL single database to AWS RDS for SQL Server using AWS DMS

Mar 10, 2021

Run the ADF copy pipeline only when AWS S3 bucket got a file

In my earlier blog post, I explained the process for copying a file from AWS S3 bucket to ADLS Gen2 storage account using ADF. We may want the copy pipeline to run only when there is a file in AWS S3 bucket. …

Azure Logic Apps

3 min read

Run the ADF copy pipeline only when AWS S3 bucket got a file