Deploy Azure Data Factory to copy data from GCP to Azure using Terraform

I used Terraform to deploy these resources.

1. GCS bucket
2. Azure Data Lake Storage account
3. Azure Data Factory pipeline with these components

• Linked Service for GCS bucket
• Dataset for GCS bucket
• Linked Service for ADLS account
• Dataset for ADLS storage container
• Pipeline with Copy Data activity to move the delimited text file from GCS bucket to ADLS container

My Terraform code has different components.

1. Providers

Terraform relies on plugins called “providers” to interact with cloud providers, SaaS providers, and other APIs.

I included providers for both Azure and GCP in my code.

2. Variables

The Terraform language includes a few kinds of blocks for requesting or publishing named values.

I used Input Variables in my code.

3. GCP resources

Below is the code that deploys

• GCS bucket
• Service Account in GCP
• Storage Key for GCS bucket
• Storage admin role to Service Account

4. ADLS account

Below is the code that deploys

• Resource Group
• ADLS account
• Storage container in ADLS account

5. ADF components for ADLS account

Below is the code that deploys

• Azure Data Factory
• Linked Service for ADLS account
• CSV Dataset for ADLS Linked Service

6. ADF components for GCS bucket

Below is the code that deploys

• Linked Service for GCP storage bucket
• CSV Dataset for GCS bucket Linked Service

7. ADF pipeline

Below is the code that deploys COPY Data activity pipeline, to copy delimited text file from GCS storage bucket to ADLS storage container.

Disclaimer: The posts here represent my personal views and not those of my employer or any specific vendor. Any technical advice or instructions are based on my own personal knowledge and experience.