25. Auto Create table option in Copy activity for SQL as sink in Azure data factory

In this video , we learnt how to use Auto Create table in copy activity of ADF pipeline
#adf #azuredatafactory #datafactory #azuresynapseanalytics #synapseanalytics #microsoft #cloud #cloudcomputing #dataengineer #dataengineering #dataengineeringessentials #azure #azurelearning #azurecontentannu

[@akshaykhule1906]

You saved my day, thank you and keep posting such videos

[@abhishekpathak7338]

Nicely explained. Thank you.

[@sangeethak725]

Thank you so much for ur effort.. its very useful

[@armielgonzzz]

thank you so much for this. i was wondering if 'Auto create table' can also be done with Data Flow?

[@sudhanshuranjan8526]

Can you explain how to do the same for azure delta table

[@Malhar1818]

Hi anu thanks for sharing this i need to import all the txt file remember the adls folder is having multiple file but i want only txt files into a single table in databricks how we can do that

[@RGDataJourney]

Hi Anu, recently i attended one interview they given one scenario based interview question which is here is source side every day the file format(csv, excel, json, xml etc) changing but target table should be one table only, how to load data into target table and create ADF pipeline can you explain it.

[@azurecontentannu6399]

In copy activity, You can use 'wildcard file path' option and give file path pattern as * which will pick up the file as it is irrespective of the format of the file . In sink, point the dataset to the target table.

[@dsandhya571]

@RGDataJourney Hi...could pls let me know that did u practically implemented from the given suggestion. If so kindly tell me what is the way u followed. Thanks in advance!

[@priyankadp5777]

Hi Annu..
I have a scenario... May be u can help... Scenario is to read from on prem db and write to azure sql db (upsert operation) .i am able to do this using copy activity. I have 2 new columns only in sink( create date and update date) .. create date shud be set with current date when inserting new record and update date column shud be updated only in case of updating the record(not inserting) .. how do achieve this using upsert copy activity.. i dont find any way to distinguish between new record or existinf record in copy

[@tarakram9216]

Hello mam
Are you taking any online azure data factory classes
Please reply

[@azurecontentannu6399]

Sorry not yet started any classes.. Wl inform if I do so in future.. Thankyou

[@prashanthchandrashekar2929]

Hi, thank you for this video and clear explanation.
I’m trying to replicate something similar (auto create in copy activity) but I face an error at the copy activity sink side that says cannot access the destination table- invalid object name.
I have parameterized the table name in the sink dataset and passing the appropriate @item().
I can see the expected table name in the error message.
My sink is an azure sql database too just like in the video.
A little lost over here😅
Any help would be highly appreciated.
Thanks.

[@azurecontentannu6399]

sorry but you cant use @item() in the dataset directly. First , Create a parameter named 'tablename' in dataset. Then in connection tab, in tablename, add dynamic content and select the parameter name, It wl automatically create this expression : @dataset().tablename . Now in the activity , where you are using the dataset, you need to provide the value , there use add dynamic content and write the expression using @item()

[@prashanthchandrashekar2929]

@@azurecontentannu6399 Hey, thank you for your quick reply. Yes, I have followed the same approach. But I still encounter the Invalid object name error which leaves me still thinking where I've gone wrong.😄

[@user-jz6fm1tj4b]

Nice one. Can you please share the datasets