Handling schema drift for tables in Azure Synapse dedicated pool

  1. If you are using Data Flow in ADF, an option is provided to handle schema drift. When schema drift is enabled, all incoming fields are read from your source during execution and passed through the entire flow to the Sink. By default, all newly detected columns, known as drifted columns, arrive as a string data type.
  2. If you are using Serverless pool in Azure Synapse, spark code can be written to handle the schema drift. This is sample code.
  1. LookupOldWatermark: This activity gets the previous day’s watermark values for all the active tables we need to pull data for. This SQL statement is executed in this activity.

--

--

--

Databases enthusiast

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Kotlin for Java and Android Developers — Part 2

Logo Adventure for C64 Terrapin Logo

First Steps with dbt

From Prototype to Production on Heroku

Fetch API 沒有傳送 cookies

Running Kubernetes on GPU Nodes

11 Best Career Options After Engineering In Computer Science

FlutterForce — #Week 117

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Bharath N

Bharath N

Databases enthusiast

More from Medium

How to get insights from your data using Azure Databricks?

Databricks — An Introduction and Tutorial

Using Delta Lake on Data Bricks to transform Event Hub events in real time

Using Azure Databricks for Batch and Streaming Processing