Run the ADF copy pipeline only when AWS S3 bucket got a file
In my earlier blog post, I explained the process for copying a file from AWS S3 bucket to ADLS Gen2 storage account using ADF.
We may want the copy pipeline to run only when there is a file in AWS S3 bucket. This will save the Data Integration Unit (DIU) in ADF.
We can achieve this in various ways. I achieved this using
- Azure Logic Apps
- Get Metadata and If condition in ADF
Azure Logic Apps
Azure provides many integration and automation services. Of them, my favorite are Azure Logic Apps and Azure Functions.
Azure Functions is a serverless compute service, whereas Azure Logic Apps provides serverless workflows.
For automating the ADF pipeline execution, I did the following
a. Create Logic App instance
b. Design the workflow
In the workflow, a Recurrence trigger will initiate the workflow every 4 hours.
Based on the Get S3 object metadata action’s output, the workflow will check whether there is a file in the S3 bucket or not. If the the condition is true, then the ADF copy pipeline will be executed.
Get Metadata and If condition in ADF
Azure Data Factory provides various activities for running different processes through pipelines.
For my automation, I used
a. Get Metadata (under General activity)
b. If condition (under Iteration & conditionals activity)
In Get Metadata, I’m checking whether the file exists in the AWs S3 bucket or not. Copy pipeline will be executed only if a file exists in the source.
Disclaimer: The posts here represent my personal views and not those of my employer or any specific vendor. Any technical advice or instructions are based on my own personal knowledge and experience.
