Видео

como lo instalo para usar con xampp? me ayudas por favor?

How to track lineage with apply changes as in my case lineage gets breaked?

Thanks for the video!

Great videos as always❤

Nice video!

Great video, thanks!

Thanks for the video. How often is the materialized view (goldT table) refreshed in the example? Is this a configurable setting?

recently when I used <databricks-instance>#secrets/createScope , Databricks is not redirecting to Create scope screen. any idea what can be reason. I have admin privileges and all relevant privileges.

Big Thank you , Cant give much but subscribed and Liked .

Very comprehensive and basics thanks for the tutorial

Love the dog in the thumbnail!!

Thank you it is a very good explanation & I created as per your instruction & works...Thank you again

how do we manage delete? if source table is huge (size =200gb) how do we identify if some of the rows are deleted from it before inserting into target table? I dont want to delete the row in target table but just have a flag against it. The target table is in a DW which should have history of all data. CDC is not an option. Thanks

This is an amazing job. I wonder if I can use the databricks community edition for my pet project that will be live.

Superr....

Its a nice channel to learn the DE. Can you make a roadmap for data engineering for absolute beginners and what should one get grip on like tech stack, skills, use cases on projects. Thankyou in advance

Thanks a lot for creating this video. For the orchestration. Adf triggers are the only option. With Airflow orchestration we customize on the scheduling part.

Nice!

Thank you for the video! It was very helpful. I have a dozen or so pipelines that I needed to execute in batch. I was able to create a parent pipeline using a ForEach Loop that reference the info from a database table then execute all the pipelines.

in case anyone runs into the same error, maybe its a windows thing? or an update from flask but: download_name=f'{video.title}.mp3' is the code line for send_file not attachment_name

Really Thanks bro for such a informative video

In Event grid & Azure function segment, you created the topic supscription for the Azure Function. But what about the publish part? How StorageAccout container is publishing the event to that particular topic on new file upload?

As a data enthusiast this information so valuable for me. Thanks for sharing your knowledge.

Superr video as always, keep doing....

I have an etl process in place in the ADF. In our team, we wanted to implement the table and views transformation and implementation with dbt core. We were wondering if we could orchestrate the dbt with Azure. If so, then how? One of the approaches I could think of was to use Azure Managed Airflow Instance. But, will it allow us to install astronomer cosmos? I have never implemented dbt this way before, so needed to know if this would be the right approach or is there anything else you would suggest me?

Love your content always, keep rocking.....

great video, easy to understand. please pursuit with more videos.....

very useful

Always great videos...

Any clue if there is an open source alternative to Autoloader?

Superr video as always. You are back another useful content....

Superr video, waiting for your next video....

please could you provide a snapshot sample of how your reference file look like? Thank you!

I understand how it's pretty easy to work with CDC & ADF for one table, but I don't understand how I would manage my pipeline easily if I have let's say 100 tables with CDC...What's the best practice in this case ?

Do you know why the output of your DLT pipeline didnt give Streaming Tables? I wrote a similar pipeline where I read CDC from a SQL source, write into bronze using Autoloader in DLT, into a streaming table, then write into silver in SCD2 in DLT as well. Both bronze and silver tables are generated as streaming tables... In your example, you get normal Delta tables. I might confusing concepts... Also, do you have a video covering streaming tables and materialized views? Thanks!

didnt really understand what the deduplication portion had to do with the security aspect of storing sensitive data, but I learned something so I'm not gonna complain :) Great videos and good delivery. Thank you for sharing the code!

Quick Question : If a record is dropped from Source table i.e hard delete how does apply_changes handle it .

When I deploy, I check the error logs, and it seems like libraries aren't installed after deployment. Everything works fine when deploying from local host through

Thank you for another awesome episode!

Would it be simpler to use AD authentication as it guaranttes single sign on?

Hello , when I make some deletes and update I get this in "users_cdc_clean" and "user_cdc_quarantine" : "Detected a data update (for example part-00000-3795a007-fac7-4ed9-9a5a-f446b34caef9-c000.snappy.parquet) in the source table at version 3. This is currently not supported. If you'd like to ignore updates, set the option 'skipChangeCommits' to 'true'. If you would like the data update to be reflected, please restart this query with a fresh checkpoint directory" why is that? how can I manage deletes and updates?

"Efkaristo poli" , all the answers to the issue I had today.

Can we use databricks community edition for implementation?

Hi! Really good video. Very easy to follow and understand. Except if you could clarify following. If I have million devices do I have to create all of them in IOT hub? (That too manually?) I saw in your VS code you added device ID in the JSON. This added to the confusion above. Please tell me that the answer to my above question is no, and the device that you added in IoT hub is actually a device "type" and you have configured to receive messages from many device IDs for this one device type. Which of my understanding is true? 😊 Thank you so much

Thank you for sharing! Great explnation

Superr video as always

7 'eventhubs.connectionString':sc._jvm.org.apache.spark.eventhubs.EventHubsUtils.encrypt(IOT_CS), 8 "eventhubs.consumerGroup": '$Default' 9 } 10 json_schema = StructType( 11 [ 12 StructField ("DeviceID", IntegerType(), True), 13 StructField ("DeviceNumber", IntegerType(), True) 14 ] 15 ) 18 # Enable auto compaction and optimized writes in Delta TypeError: 'JavaPackage' object is not callable please help me Apostolos

Great presentation. You may want to move the mic away from you mouth to cut down on unnecessary noise.

It works with SQL Server on prem as well ?

It works with SQL Server on prem ?