Azure Data Factory, first thoughts

I have been trying out Azure Data Factory.

You can wonder if ADF is more like SSIS in the cloud or SQL Server Agent in the cloud. It does not quite feel like a pure ETL tool. Coming from SSIS on premise this is a completely different experience. My first impression is that the UI is not very intuitive. It takes a while before you realize where you need to be when solving a problem. Another thing that I found was that there is not very much about it online. Stuck with a problem? you're likely to be on your own. Googling my error message got me nothing, that's quite refreshing for me (unthinkable for a SQL Server error) 🙂

It is quite easy to copy data from A to B. In other words: you can do E L: Extract and Load. The T is not very easily done. If you want to get some data from Azure Blob storage and load it into your Azure SQL, this is a good tool. But if you want to do complex transformations? Not so sure yet...

 

 

 

BI consultant and trainer at Motion10

Leave a Reply

2 Comments

  1. This is actually a great viewpoint. Thinking in terms of the Modern Datawarehouse, it is very likely you want to move all data to the cloud first (raw as it may be) into your Data Lake. That's where ADF comes in: moving data from all sources, E-L to your Data Lake. All transformation is done later on - on-demand (because "Big Data" says schema-on-read), structured (classical EDW pattern) or even both (like the Lambda architecture).

    • Jesse Gorter

      Jesse Gorter

      I like your thinking Koos, indeed do you want to apply the business rules seperately, or even while reading. Schema on read, business logic on read too?

Next ArticleBest Practice: no more queries in frontend tools