Thanks, this is really helpful. What I don’t understand though is how to include the tables, views etc I’ve got within my dedicated pool (but aren’t visible in my workspace).
I can see all the contents are in json format. Even SQL scripts are appearing as json. Do you see this as a road block in doing development from an IDE like ADS or SSMS?
Potentially - I had assumed for anything following the traditional SSDT / Database project approach, you'd have a separate database project in the same repo (hence why I created Synapse in a subfolder to segregate). I think we'll end up thinking of "workspace objects" and "database objects"... maybe? Honestly haven't even begun to piece together what a full devops story across all areas looks like yet!
Hi! ¿Does anybody know if the Serverless SQL Pool is included in the CICD, if the database, table, external resources scripts can be integrated in the CICD?
New to ADF and Git, but familiar with source control in general. Can you clarify what happens when you click publish when the selected branch is something other than the main branch (EG as at 21 minutes in the video). Does it publish to the live workspace the contents based on the main branch, meaning you still need to create a pull request from development back to the main branch to get changes live, or is this done in the background?
@@AdvancingAnalytics also, I notice that Synapse ignores Spark pools when publishing to workspace_publish branch. It declares my dedicated SQL pool in the TemplateForWorkspace.json but completely forgets about the Spark pool. Do you experience the same thing?
The dedicated SQL pools still have the same accessible SQL endpoint, so I assume you can deploy SSDT projects from a git repo as with the standalone SQLDW. Technically the same approach should work for views in serverless SQL Pools but I've not tried! Simon
Hi Simon, If i recollect right in ADF, we could only publish from the Main Branch and not a feature branch. It meant that we had to move the changes first to main branch ( pull request ) and then create the ARM template in the adf_publish branch. Here I see we could publish from the Dev branch as well :) Or I am getting too old to recollect it right :(
Close, but it's whatever branch is designated as the "Collaboration Branch", in this case I set it to the Development branch, so it's only that one you can publish from. I think I messed around with the terminology a bit in the vid!
Also would like to know if we can make a selective sync while publishing..example I only want a X script to be updated and not others which might have been changed? thank you for your quick win videos!
@@sid0000009 you can only do a selective commit (ie: save X object to the branch) but publish puts everything in the collaboration branch to live. If there are pieces you don't want to publish, keep them in a separate branch!
Looking forward to video on Programmatically/Manually deploy ARM template in other environments (UAT, Prod etc.). One thing is strange that unlike ADF Microsoft has not given Import/Export ARM template functionality for Synapse.
Yeah... I tried to get a good ARM deployment demo working last week, but honestly there were too many bugs in the deployment process. I got it working with quick a few hacks and workarounds, but with the general availability rush, I'm assuming it's something that'll be fixed over the next week or so, then I'll pop a proper video together! If it's still broken next week, I guess I'll do a tour of all the bugs and how to work around :) Simon
Loving your content, loving the idea of the git integration. Keen to see it maturing compared to more data science type platforms (Databricks / Domino Data Lab / DataIku etc...)
Looks great. This gives one more reason for all those data projects to move in synapse where they get everything in one place. Now the interesting thing will be on the CI/CD part. Do you see an option to move database projects to synapse? Traditionally we build a dacpac and do the deployment. Any such approach possible here? May be we have to still maintain a separate DB project. Also, would it be possible to deploy individual changes. For example, what if one want to deploy only the pipeline changes to prod keeping other things untouched. May be we have to maintain a 'Prod' branch, do change there then do a pull.. lot of push pull merges :P Such scenarios will be interesting I guess.
Hey! Yeah, I think it's currently exactly as you said. An SSDT database project separate to the Synapse folder for deploying Dedicated SQL Pool projects via the usual dpac route. Deploying specific changes is definitely a separate branches & pull requests mechanism - same as ADF currently! Simon
Great Git is finally here 💪🏼 Great video Simon! Thanks a lot! To get it right, is the code pushed to in the main branch what’s visible in the live mode? Best regards, Marco
Nope - in order to see it in "Live" mode you need to publish code from whatever branch you have designated as the "collaboration branch". In the vid, I was using main as a separate code line that would be used to trigger a DevOps pipeline to the next environment. I'll see about doing a proper multi-environment demo soon! Simon
It is the first time that someone can clearly explain the difference between commit and publish
Thanks, this is really helpful. What I don’t understand though is how to include the tables, views etc I’ve got within my dedicated pool (but aren’t visible in my workspace).
Your channel is golden!
Really amazed by your broad understanding of analytics pipelines :D
any advice on how to deploy serverless objects?
I can see all the contents are in json format. Even SQL scripts are appearing as json. Do you see this as a road block in doing development from an IDE like ADS or SSMS?
Potentially - I had assumed for anything following the traditional SSDT / Database project approach, you'd have a separate database project in the same repo (hence why I created Synapse in a subfolder to segregate). I think we'll end up thinking of "workspace objects" and "database objects"... maybe? Honestly haven't even begun to piece together what a full devops story across all areas looks like yet!
Hey..After following all the steps my team is not able to make changes into git.How to provide them access?
Hi! ¿Does anybody know if the Serverless SQL Pool is included in the CICD, if the database, table, external resources scripts can be integrated in the CICD?
New to ADF and Git, but familiar with source control in general. Can you clarify what happens when you click publish when the selected branch is something other than the main branch (EG as at 21 minutes in the video). Does it publish to the live workspace the contents based on the main branch, meaning you still need to create a pull request from development back to the main branch to get changes live, or is this done in the background?
Great Content. Clear my concepts. Thanks.
How do we keep our "feature branch" up to date with the collaboration branch, like a "git pull"?
Yep, just create a pull request and complete it. You can create a pull request either from Synapse, or from git/DevOps directly!
@@AdvancingAnalytics also, I notice that Synapse ignores Spark pools when publishing to workspace_publish branch. It declares my dedicated SQL pool in the TemplateForWorkspace.json but completely forgets about the Spark pool. Do you experience the same thing?
Hi, Great Video. Is it possible to use Git Repo for db (SQL pool) DDL (tables, procedures, views, ...)?
The dedicated SQL pools still have the same accessible SQL endpoint, so I assume you can deploy SSDT projects from a git repo as with the standalone SQLDW. Technically the same approach should work for views in serverless SQL Pools but I've not tried!
Simon
Hi Simon, If i recollect right in ADF, we could only publish from the Main Branch and not a feature branch. It meant that we had to move the changes first to main branch ( pull request ) and then create the ARM template in the adf_publish branch. Here I see we could publish from the Dev branch as well :) Or I am getting too old to recollect it right :(
Close, but it's whatever branch is designated as the "Collaboration Branch", in this case I set it to the Development branch, so it's only that one you can publish from. I think I messed around with the terminology a bit in the vid!
Also would like to know if we can make a selective sync while publishing..example I only want a X script to be updated and not others which might have been changed? thank you for your quick win videos!
@@AdvancingAnalytics got it thanks!
@@sid0000009 you can only do a selective commit (ie: save X object to the branch) but publish puts everything in the collaboration branch to live. If there are pieces you don't want to publish, keep them in a separate branch!
Looking forward to video on Programmatically/Manually deploy ARM template in other environments (UAT, Prod etc.). One thing is strange that unlike ADF Microsoft has not given Import/Export ARM template functionality for Synapse.
Yeah... I tried to get a good ARM deployment demo working last week, but honestly there were too many bugs in the deployment process. I got it working with quick a few hacks and workarounds, but with the general availability rush, I'm assuming it's something that'll be fixed over the next week or so, then I'll pop a proper video together!
If it's still broken next week, I guess I'll do a tour of all the bugs and how to work around :)
Simon
@@AdvancingAnalytics Hi, thanks a lot for your insightful videos. I am very interested in that follow-up as well! Keep up the good work :)
Looking for synapse ci/cd
Loving your content, loving the idea of the git integration.
Keen to see it maturing compared to more data science type platforms (Databricks / Domino Data Lab / DataIku etc...)
Looks great. This gives one more reason for all those data projects to move in synapse where they get everything in one place. Now the interesting thing will be on the CI/CD part. Do you see an option to move database projects to synapse? Traditionally we build a dacpac and do the deployment. Any such approach possible here? May be we have to still maintain a separate DB project.
Also, would it be possible to deploy individual changes. For example, what if one want to deploy only the pipeline changes to prod keeping other things untouched. May be we have to maintain a 'Prod' branch, do change there then do a pull.. lot of push pull merges :P Such scenarios will be interesting I guess.
Hey!
Yeah, I think it's currently exactly as you said. An SSDT database project separate to the Synapse folder for deploying Dedicated SQL Pool projects via the usual dpac route. Deploying specific changes is definitely a separate branches & pull requests mechanism - same as ADF currently!
Simon
Great Git is finally here 💪🏼
Great video Simon! Thanks a lot!
To get it right, is the code pushed to in the main branch what’s visible in the live mode?
Best regards,
Marco
Nope - in order to see it in "Live" mode you need to publish code from whatever branch you have designated as the "collaboration branch". In the vid, I was using main as a separate code line that would be used to trigger a DevOps pipeline to the next environment. I'll see about doing a proper multi-environment demo soon!
Simon
@@AdvancingAnalytics What's about demo? Is it in plans?
Lots of talk without any visuals or clear logical flow.