The Data Engineering Experience
The Data Engineering Experience
  • Видео 13
  • Просмотров 4 960
BigQuery Introduces - Pipe Syntax
In this video I have discussed in brief the newly launched feature in BigQuery - BigQuery Pipe Syntax
Timestamps:
00:00 - Intro
02:50 - Some Examples and Usage
08:50 - Hands-on from BigQuery UI
14:14 - Advantages and Limitations
Documentation : cloud.google.com/bigquery/docs/pipe-syntax
Similar helpful resources : medium.com/google-cloud/google-pipe-syntax-modernizing-sql-without-sacrificing-its-strengths-78b6e8177648
#gcp #GoogleCloud #dataengineeringessentials #bigdata #etl #elt
#BigQuery #gcp #GoogleCloud #bigquery #pipesyntax #sql #pipe_syntax
Просмотров: 188

Видео

A Quick Guide to BigQuery Workflows
Просмотров 3773 месяца назад
In this video I have discussed in brief the newly launched feature in BigQuery - BigQuery Workflows Timestamps: 00:00 - Intro 01:05 - BigQuery Workflows 02:18 - Scheduling Details and Limitations 04:05 - Workflows - Demo in BigQuery Studio Documentation : cloud.google.com/bigquery/docs/workflows-introduction #gcp #GoogleCloud #dataengineeringessentials #bigdata #etl #elt #BigQuery #gcp #GoogleC...
GCP Data Engineer Interview Prep : Dataflow and Cloud Functions
Просмотров 6683 месяца назад
In this video, we discussed most commonly faced interview questions about the two very important GCP Data Engineering ETL Tools : Dataflow (Apache Beam) and Cloud Functions. We divided questions from each services into three levels of Difficulty : EASY, MEDIUM and HARD. Timestamps: 00:00 - Intro 00:24 - Dataflow Difficulty Level : Easy 06:15 - Dataflow Difficulty Level : Medium 11:55 - Dataflow...
GCP Data Engineer Interview Prep : BigQuery and Cloud Composer
Просмотров 2,5 тыс.3 месяца назад
In this video, we discussed most commonly faced interview questions about the two most used GCP Data Engineering services : BigQuery and Cloud Composer (Apache Airflow). We divided questions from each services into three levels of Difficulty : EASY, MEDIUM and HARD. Timestamps: 00:00 - Intro 00:55 - BigQuery Difficulty Level : Easy 05:05 - BigQuery Difficulty Level : Medium 09:25 - BigQuery Dif...
2 Useful Features of BigQuery Every Data Engineer Should Know!
Просмотров 1414 месяца назад
Unlock the Power of BigQuery with These Essential Features! 🚀 Are you looking to enhance your data analytics skills in Google BigQuery? In this video, we dive into some must-know features that can transform the way you handle and analyze data. Whether you’re a beginner or an experienced data engineer, these tips and tricks will help you optimize your workflows, manage your datasets more efficie...
String Partitioning in BigQuery
Просмотров 1674 месяца назад
In this video, we'll explore how to create a partitioned table in BigQuery using a string column using a hashing function FARM_FINGERPRINT function. This technique can optimize query performance and storage efficiency when working with large datasets. #BigQuery #SQL #DataPartitioning #GoogleCloud #DataEngineering #SQLTutorial #BigData #CloudComputing #DataAnalytics #SQLTips #DataOptimization #D...
Table Sampling in BigQuery - A powerful way to Analyze Large Datasets
Просмотров 864 месяца назад
Discover the power of table sampling in BigQuery! 🚀 Learn how to quickly and efficiently query random subsets of large datasets without scanning the entire table. #BigQuery #DataAnalysis #TableSampling #BigData #SQLTips #DataScience #GoogleCloud #TechTips #DataAnalytics #LearnSQL #CodingTips #EfficientQuerying #DataEngineering
Dynamic Pivoting in BigQuery
Просмотров 1514 месяца назад
In this video, we dive into the powerful concept of dynamic pivoting in BigQuery! Learn what pivoting is and why it's essential for transforming your data. We'll explore the advantages of dynamic pivoting over traditional methods and demonstrate how to apply it to real-world scenarios. Follow along as we break down a step-by-step guide for dynamically pivoting data in BigQuery, making your data...
Data Deduplication in BigQuery: 3 Effective Methods Explained!
Просмотров 2104 месяца назад
Learn how to remove duplicate rows from your SQL datasets with three powerful methods: SELECT DISTINCT, GROUP BY, and ROW_NUMBER() with QUALIFY. We break down each approach, explain when to use them, and provide a sample dataset for better understanding. Perfect for anyone looking to clean and optimize their data in SQL! Don't forget to like, comment, and subscribe for more data tips! #deduplic...
BigQuery Interview Questions Optimization Techniques
Просмотров 2604 месяца назад
Are you looking to optimize your BigQuery performance and save on costs? In this video, we share 10 essential tips to help you get the most out of BigQuery. Learn how to efficiently manage your datasets, choose the best data formats, utilize table previews, and leverage advanced techniques like partitioning, clustering, caching, and materialized views. Whether you're a beginner or an experience...

Комментарии

  • @10xApe
    @10xApe 5 дней назад

    Great content man, not much of that BQ info on YT.i have a multiple transformation of latge table, whats the best way to limit processing so it doesn't calk the full original table? Use temporary tabke within the workflow? Thanks 🙏

  • @harshalpatel3012
    @harshalpatel3012 13 дней назад

    Good work bro

  • @HariharanTamilarashanJeyasri
    @HariharanTamilarashanJeyasri Месяц назад

    HI BROTHER I HAVE A CONFUSED TODAY I ATTEND THE ONLY INTERVIEW FOR ROLE GCP DATA ENGINEER THEY ARE ASKING DEFENCE FIRST QUESTION FACT TABLE AND DIMENSION TABLE DIFFERENCE ACTUALLY I THINK ABOUT THESE LIKE A QUESTION I SEE IN YOUR YOUR QUESTIONS NOT LISTED I AM ALSO NOT EXPECTED THIS QUESTION SO PLEASE MAKE A SIMILAR QUESTION COULD YOU PLEASE LET ME KNOW WHICH TOPIC ARE TCP DATA MODELLING MUST FOR LEARNING THE DATA ENGINEER

  • @sravankumar6798
    @sravankumar6798 Месяц назад

    Nice work!!👏

  • @yoga_with_atharva
    @yoga_with_atharva Месяц назад

    Amazing

  • @yoga_with_atharva
    @yoga_with_atharva Месяц назад

    Can You give the answer of all question

    • @thedataengineeringexperience
      @thedataengineeringexperience Месяц назад

      I am making the video. Since I covered a wide range of questions, it is going to take a week or so :)

    • @sha2460
      @sha2460 Месяц назад

      ​@@thedataengineeringexperienceCan you please start gcp data engineering mock interview playlist series? Please. Also before that please help us with some realtime gcp data engineering projects so that we can get idea of how really projects happens in organisation. I'll be thankful for this.

  • @Brahma_atmuri
    @Brahma_atmuri Месяц назад

    Do we need to enable anything??

    • @thedataengineeringexperience
      @thedataengineeringexperience Месяц назад

      Hey! At the moment, no. You can directly start using it from the Bigquery UI itself but it has limited support. Also it might be available on limited projects so you might need to wait.

    • @Brahma_atmuri
      @Brahma_atmuri Месяц назад

      Thankyou ​@@thedataengineeringexperience

  • @sammosesmandapalli
    @sammosesmandapalli 2 месяца назад

    thank you sir, very helpful

  • @kalyanvalaja9868
    @kalyanvalaja9868 2 месяца назад

    Make a video in partition expiration and clustering in detail

  • @kalyanvalaja9868
    @kalyanvalaja9868 2 месяца назад

    Great content

  • @DrithiVarma
    @DrithiVarma 2 месяца назад

    Thank you for the awesome content which gives me direction to prepare for the interview.

  • @DrithiVarma
    @DrithiVarma 2 месяца назад

    Thanks for the awesome content!

  • @FreakONcW1
    @FreakONcW1 2 месяца назад

    Good quality stuff 🎉

  • @thirumalaip6458
    @thirumalaip6458 3 месяца назад

    Can this be a replacement to cloud composer ?

    • @thedataengineeringexperience
      @thedataengineeringexperience 3 месяца назад

      Right now I see it as a replacement of scheduled queries only. Let's hope Google Cloud adds more features to it.

  • @sammosesmandapalli
    @sammosesmandapalli 3 месяца назад

    Hir Sir, Thanks for the video. You have put lot of effort in making this video. It is very helpful for budding gcp data engineers like me. I am thankful to you for such quality content. Please keep doing videos. Thanks🙂

  • @PRIYANKAMISHRA-zq8cm
    @PRIYANKAMISHRA-zq8cm 3 месяца назад

    where are the explanation for the questions

    • @thedataengineeringexperience
      @thedataengineeringexperience 3 месяца назад

      I tried to explain the answers briefly. Detailed explanation to each question would stretch the video considerably.

    • @thedataengineeringexperience
      @thedataengineeringexperience 3 месяца назад

      Almost all the answers are easy to get if you just google them. I have just collated the questions.

  • @nitishkumarsingh5850
    @nitishkumarsingh5850 3 месяца назад

    How did you set up dark mode in Google Cloud

  • @thedataengineeringexperience
    @thedataengineeringexperience 4 месяца назад

    Official Google Cloud Documentation : cloud.google.com/bigquery/docs/user-defined-aggregates