Dear Arshad, I followed this instruction vidoe and it worked just perfect to deploy CI/CD pipeline for my large scale Synapse Data Warehouse solution which includes, many Pipelines, Data Flows, Linked Services and Datasets for my ADLS Gen2, Serverless SQL Pool and Dedicated SQL Pool, SQL Script Files (for both Serverless SQL pool and Dedicated SQL Pool), Please note that, I also follow your other valuable videos on Synapse SQL Pool and I learned a lot. Thank you so much and I appreciate it. Please continue your effort for the Azure Synapse community!
oww, cool! I would like to know how you did that, basically my challenges are 1. how to add other parameters on the ARM template if I have a lot of linked services and datasets for my pipelines 2. How to choose specific artifacts only to be released on the uat/prd
@@cliffordraygentiles I had to get around it through power shell. I added numerical values to the start of the script names and using an invoke sql command which iterated through the scripts based on the numerical value assigned. I am sure there are easier options using an ARM template but poweshell gave the most flexibility
Thank you Arshad. This is one of the best videos for synapse deployments. It was very well structured and logically flowing and the introduction section of DevOps was very useful.
This is the best explanation about Synapse CI/CD process, even better then Microsoft I only miss the configuration of OverrideParameters, I guess it is the same as we do on ADF pipelines. Tks again!
Thank you Arshad Ali, it was very helpful. Your explanation is perfect as always. You have the best Synapse Analytics channel on RUclips. It would be great if you make videos on designing Data Lakes (Delta Lake), folder structures, creating different zones etc.
The best of the best session on Synapse CICD. Could you also explain have to handle parameter overriding for other linked service, such as Azure KeyVault and Linked Storage Account.
Thank you for this clear demonstration, I learned a lot since I don't have experience on the CI/CD yet. Can you help me with the following. 1. how to add other parameters on the ARM template if I have a lot of linked services and datasets for my pipelines 2. How to choose specific artifacts only to be released on the uat/prd Keep up the good work!
Thank you for the cool demo, it is very much useful.. one quick question on the storage account when you have created 3 synapse workspaces. in my case I want to use a different storage account names for each workspace individually, does it still work without parameterising the storage account name ?
Very complete coverage good job. Curious for those of us with existing ADF devops projects, would it be wise to create a new project for synapse artifacts and not mix ADF with synapse artifacts
Hello Arshad, Thank you sooooo much for your awesome tutorial on setting up CI/CD for Azure Synapse! I have one question regards to setting the Pipeline variable for CI/CD, would love to hear your ideas and thoughts. I need to set the value as ‘extsynapsecicdadlstest’ for ‘synapse-cicd-demo-ws-test’ Workspace, but I need to change the value to ‘extsynapsecicdadlsuat’ for ‘synapse-cicd-demo-ws-uat’ Workspace. How can I do it in the ADO release pipeline so that I can update the parameter value when deploying to a different Synapse Workspace? Thanks in advance!
Hi Arshad, great video, I would like to know if the same solution will work when the test & Prod workspace are present in a separate subscriptions for each one ??
Hello good day. We have 2 existing synapse workspaces, dev and prod. After setting up git configuration on dev workspace, I noticed that all artifacts (pipelines, storage, datasets, scripts, notebooks) from prod are now existing on the master branch. How is this possible to see prod artifacts when I haven't explicitly connected prod into our dev workspace? Thanks
There is a different situation I need to handle currently. Pls help if any alternative than manual approach. The process has been implemented same as mentioned in the video. All good and happy path. There were some data issues reported and my development team had directly implemented the changes on PROD instead of Dev. Noe how to get all those changes from Prod to Dev. Pls help if any way.
@arshad Ali : I am trying to use the same process but in my workspace there are alot of notebooks, pipelines and scripts which resulted in arm workspace template exceeding more than 20 mb size which resulted in an error related to size
Very Impressive for one time migration from Dev to Test or Test to Prod. Looking for changes in the pipeline and over and over will it migrate everything based on the ARM template or will it take only the modified components. Pls let me know if there is any way we can enable or disable. Thanks
Hi Arshad Ali. Awesome content, as usual, you rock. I got it to do the same steps, however, I am stuck on one question. If, we are gonna use it for a CI/CD with Synapse, how could I parametrize Notebooks for Spark Pools? For example, one issue I faced, was the name of the Storage Account, if they are different, on our dev, it could be st_DEV but for uat can be st_UAT. If we hardcoded the name into our notebooks, to set the path, the next environments will be wrong. I tried using parameters in my pipeline and setting my cell as toggle parameters, but I think this is not the way. For example, with linked services with the same name, changing into the TemplateParameters the name of Storage Account, works well across different stages, but I search here and I couldn't find a way to use pyspark using linked service.
@@kunaldasofficial Use parameters in your notebook. You can pass a value in a synapse pipeline to a notebook (lookup activity --> notebook activity). Since there are no global variables in Synapse yet, hard copy a dev/production value in your datalake to call upon.
Hi Arshad. Is there any tutorial/documentation around how to deploy hotfixes for Synapse in production ? There is a Microsoft documentation around ADF but not for Synapse.
Hi, a quick question please. While we connect Azure Repo to Synapse workspace, is there a way I can give a custom commit message when I add new changes to Azure repos from Synapse workspace?
Hey Arshad, great demo. I get this error and unable to deploy any notebooks ##[error]Encountered with exception:Error: Failed to fetch the deployment status {"code":"400","message":"Failed Component = DataFactoryResourceProvider, ErrorCode = 400, Error = BadRequest "} any help is appreciated !
Hi Sir, can you make a video on the requirement - Developer wants to create pipelines in Synapse usng json files rather than using GUI of Synapse analytics and clicking on publish button. Rest env is same as in this video i.e Build > artifact > release
Thank you for sharing this knowledge Arshad. Can we call the source control and CICD the same? And, if the artifact is a sql script would a component overwrite its version in production DB? Thanks a lot!
i have been pushed out of my field and old coworkers destroyed my life ; harassed for 2+ years, waiting to hear from authorities, hope they will help me
![Seungmin "그렇게, 천천히, 우리(As we are)" | [Stray Kids : SKZ-PLAYER]](http://i.ytimg.com/vi/kAzmhLHePqU/mqdefault.jpg)
Dear Arshad,
I followed this instruction vidoe and it worked just perfect to deploy CI/CD pipeline for my large scale Synapse Data Warehouse solution which includes, many Pipelines, Data Flows, Linked Services and Datasets for my ADLS Gen2, Serverless SQL Pool and Dedicated SQL Pool, SQL Script Files (for both Serverless SQL pool and Dedicated SQL Pool),
Please note that, I also follow your other valuable videos on Synapse SQL Pool and I learned a lot.
Thank you so much and I appreciate it. Please continue your effort for the Azure Synapse community!
How ? this doesn't copy across serverless or dedicated pools -this just moves the scripts
oww, cool! I would like to know how you did that, basically my challenges are
1. how to add other parameters on the ARM template if I have a lot of linked services and datasets for my pipelines
2. How to choose specific artifacts only to be released on the uat/prd
@@cliffordraygentiles I had to get around it through power shell. I added numerical values to the start of the script names and using an invoke sql command which iterated through the scripts based on the numerical value assigned. I am sure there are easier options using an ARM template but poweshell gave the most flexibility
I am a regular follower of your videos and these put be in better state and built my confidence high to deliver.
Excellent and very well structured video. Very helpful. Thanks for Uploading and sharing the knowledge !
Great!... What hidden gem about Synapse on RUclips! Only 14 comments!? Good stuff!... good job! Thank you!
this is by far the best session that i had seen in synapse CICD, thank you arshad. can you please add another session for the DACPAC implementation
yes please- considering most orgs will use a serverless instance it would be good to move automate the serverless databases and deploy the scripts.
Thank you so much for very detailed explanation of Azure Synapse CI CD. This is definitely going to help many.
Arshad sir.... amazing series...sincere thanks for making the effort to educate all
Thank you Arshad. This is one of the best videos for synapse deployments. It was very well structured and logically flowing and the introduction section of DevOps was very useful.
Thanks for this video. This is one of the best video to understand synapse ci cd deployment.
this is a masterpiece, by far the best Synapse version control video out there. Again, thank you
This is the best explanation about Synapse CI/CD process, even better then Microsoft I only miss the configuration of OverrideParameters, I guess it is the same as we do on ADF pipelines. Tks again!
Hands down! This is the best explanation ever. Thank you for sharing your knowledge.
Brilliant - really well put together. Thanks Arshad.
Thank you Arshad Ali, it was very helpful. Your explanation is perfect as always. You have the best Synapse Analytics channel on RUclips.
It would be great if you make videos on designing Data Lakes (Delta Lake), folder structures, creating different zones etc.
Thank you Arshad, the content and the demo was absolutely the best!
Excellent and useful video. Thank you for sharing your experience!
Awesome! Thanx for sharing, and for the thorough walkthrough. This helped me alot in understanding both Synapse version control and Azure DevOps!
Fantastic video. Keep up the great work. Very valuable
Clear and simple explantion !! kudos
Great tutorial!
The best of the best session on Synapse CICD. Could you also explain have to handle parameter overriding for other linked service, such as Azure KeyVault and Linked Storage Account.
Thanks for the sharing. it is helpful.😀
It was wonderful. Pretty clear. Thank you.
Very nice presentation. Please create something similar to build and deploy scala jar files.
thank you, i have learned something.
Great demo!
Thank you Arshad, your explanation is very clear. Videos related to designing Data Lakehouse would be helpful to everyone.
Thank you, this video has been very helpful with Synapse!
Thank you for this clear demonstration, I learned a lot since I don't have experience on the CI/CD yet. Can you help me with the following.
1. how to add other parameters on the ARM template if I have a lot of linked services and datasets for my pipelines
2. How to choose specific artifacts only to be released on the uat/prd
Keep up the good work!
Thank you for the cool demo, it is very much useful.. one quick question on the storage account when you have created 3 synapse workspaces. in my case I want to use a different storage account names for each workspace individually, does it still work without parameterising the storage account name ?
Thank you !
Thanks so much for this video!
Very complete coverage good job. Curious for those of us with existing ADF devops projects, would it be wise to create a new project for synapse artifacts and not mix ADF with synapse artifacts
wondefull .. great job
Very good session, just please, don't forget to zoom to make small details visible. 🔍🙏
You are the best!!
Thank you
Thanks so much
Hello Arshad,
Thank you sooooo much for your awesome tutorial on setting up CI/CD for Azure Synapse! I have one question regards to setting the Pipeline variable for CI/CD, would love to hear your ideas and thoughts. I need to set the value as ‘extsynapsecicdadlstest’ for ‘synapse-cicd-demo-ws-test’ Workspace, but I need to change the value to ‘extsynapsecicdadlsuat’ for ‘synapse-cicd-demo-ws-uat’ Workspace. How can I do it in the ADO release pipeline so that I can update the parameter value when deploying to a different Synapse Workspace? Thanks in advance!
Thanks for sharing, how do we connect Azure DevOps in different tenant?
Hi Arshad, great video, I would like to know if the same solution will work when the test & Prod workspace are present in a separate subscriptions for each one ??
Hello good day. We have 2 existing synapse workspaces, dev and prod. After setting up git configuration on dev workspace, I noticed that all artifacts (pipelines, storage, datasets, scripts, notebooks) from prod are now existing on the master branch. How is this possible to see prod artifacts when I haven't explicitly connected prod into our dev workspace? Thanks
There is a different situation I need to handle currently. Pls help if any alternative than manual approach.
The process has been implemented same as mentioned in the video. All good and happy path.
There were some data issues reported and my development team had directly implemented the changes on PROD instead of Dev. Noe how to get all those changes from Prod to Dev. Pls help if any way.
@arshad Ali : I am trying to use the same process but in my workspace there are alot of notebooks, pipelines and scripts which resulted in arm workspace template exceeding more than 20 mb size which resulted in an error related to size
Very Impressive for one time migration from Dev to Test or Test to Prod. Looking for changes in the pipeline and over and over will it migrate everything based on the ARM template or will it take only the modified components. Pls let me know if there is any way we can enable or disable. Thanks
Hi Arshad Ali. Awesome content, as usual, you rock. I got it to do the same steps, however, I am stuck on one question. If, we are gonna use it for a CI/CD with Synapse, how could I parametrize Notebooks for Spark Pools? For example, one issue I faced, was the name of the Storage Account, if they are different, on our dev, it could be st_DEV but for uat can be st_UAT. If we hardcoded the name into our notebooks, to set the path, the next environments will be wrong. I tried using parameters in my pipeline and setting my cell as toggle parameters, but I think this is not the way.
For example, with linked services with the same name, changing into the TemplateParameters the name of Storage Account, works well across different stages, but I search here and I couldn't find a way to use pyspark using linked service.
@@kunaldasofficial Use parameters in your notebook. You can pass a value in a synapse pipeline to a notebook (lookup activity --> notebook activity). Since there are no global variables in Synapse yet, hard copy a dev/production value in your datalake to call upon.
Hi Arshad. Is there any tutorial/documentation around how to deploy hotfixes for Synapse in production ? There is a Microsoft documentation around ADF but not for Synapse.
Hi, a quick question please. While we connect Azure Repo to Synapse workspace, is there a way I can give a custom commit message when I add new changes to Azure repos from Synapse workspace?
Hello sir, I have a question , after successful deploy in UAT/PROD , is there anyway like date/time stamp to verify changes in upper environment ??
Hey Arshad, great demo. I get this error and unable to deploy any notebooks ##[error]Encountered with exception:Error: Failed to fetch the deployment status {"code":"400","message":"Failed Component = DataFactoryResourceProvider, ErrorCode = 400, Error = BadRequest "} any help is appreciated !
Hi Sir, can you make a video on the requirement - Developer wants to create pipelines in Synapse usng json files rather than using GUI of Synapse analytics and clicking on publish button. Rest env is same as in this video i.e Build > artifact > release
Thank you for sharing this knowledge Arshad. Can we call the source control and CICD the same? And, if the artifact is a sql script would a component overwrite its version in production DB? Thanks a lot!
i have been pushed out of my field and old coworkers destroyed my life ; harassed for 2+ years, waiting to hear from authorities, hope they will help me