AWS Tutorials - Methods of Building AWS Glue ETL Pipeline

AWS Glue Pipelines are responsible to ingest data in the data platform or data lake and manage data transformation lifecycle from raw to cleansed to curated state. There are many methods to build such pipelines. In this video, we talk about some of these methods and compare them for reusability, observability and development effort.

[@santospcs2011]

Thank you for the pipeline video, very insightful.

[@AWSTutorialsOnline]

Glad it was helpful!

[@hirendra83]

Excellent Tutorial. Thanks

[@AWSTutorialsOnline]

You are welcome!

[@nttazitt1300]

Very helpful tutorial, thanks.

[@AWSTutorialsOnline]

Glad it was helpful!

[@rollinOnCode]

this is super good and helpful. thank you

[@AWSTutorialsOnline]

You're very welcome!

[@nrodriguezgal148]

Excellent video and explanation.

[@AWSTutorialsOnline]

Glad it was helpful!

[@YEO19901]

Wonderful.

[@AWSTutorialsOnline]

Thanks

[@hsz7338]

As always thank you for the video. The breakdown comparison is incredibly intuitive. I am curious about your view on which approach is best in handling pipeline replay (i.e. handling pipeline failure) and CI/CD process (i.e. pipeline as code)?

[@AWSTutorialsOnline]

CICD support is available for all approaches. Replay is better in event driven approach because you can run part of pipeline based on error raised during the event.

[@adityag020]

Insightful tutorial. Can you make a practical video based on event based pipeline using Dynamodb to store metadata and configurations with retry mechanism in case if it fails?

[@AWSTutorialsOnline]

sure - coming soon :)

[@mangeshshinde2844]

Nice tutorial. Can you make some practical tutorial for event based pipeline?

[@AWSTutorialsOnline]

Yes, sure. I am getting multiple requests for that. I will do it.

[@user-lq6gc1tw2v]

Hello, good video. Maybe someone knows when use Glue workflows and when use StepFunctions?

[@AWSTutorialsOnline]

Glue workflow when you want to orchestrate Glue Job and Crawler only. StepFunction when you want to orchestrate Glue Job, Crawler plus other services as well.

[@pachappagarimohanvamsi4641]

Could you please make some practical workshop kind of thing on these approaches?

[@AWSTutorialsOnline]

Sure. will do. The Glue workflow lab is already available @ aws-dojo.com/workshoplists/workshoplist29/

[@timmyzheng6049]

Thank you for the pipeline video, very insightful. Quick question: to avoid hardcoding, can I also use DynamoDB for storing environment parameters like s3 paths / file names / business date for my ETL pipeline let's say using step functions, and what do you think is the best industry practice for storing parameters for AWS ETL pipeline?

[@AWSTutorialsOnline]

if you want to decouple the configuration, you should keep the configuration centralized in the services likes DynamoDB or Parameter Store. DD is especially good if you are going for multi-account deployment and you still want to keep the configuration centralized.

[@timmyzheng6049]

@@AWSTutorialsOnline Thank you for the response. After doing some research it seems that to pass parameters to glue job, I have to use Lambda with boto3 in step function, and as lambda can call glue job using python glue API too, does that mean there is no need to put glue job separately to step function?

[@andresmerchan6418]

Hello! Which of the three methods is more cost effective?

[@AWSTutorialsOnline]

Event based.

[@alokanand851]

Hi all,
We are using AWS Glue + PySpark to perform ETL to a destination RDS PostgreSql DB. Destination tables have columns with primary & foreign keys with UUID data type. We are failing to populate these destination UUID type columns. How can we achieve this, please suggest.

[@AWSTutorialsOnline]

I am not sure what error you are getting. ETL job has to respect table level column constraint. As long as you are doing it; there should not be a problem.

[@radhasowjanya6872]

Hello Sir..I follow all your videos. They are very useful in my project. Thank you very much.I have a quick question: Is there a possibility to add multiple SQL statements in one AWS glue Studio job? if yes can you help me with it.(use case: want to truncate the target table(Snowflake) before loading)

[@AWSTutorialsOnline]

You can multiple SQL Transform one after another to run multiple SQL statement in sequence

[@ankursinhaa2466]

I love you

[@ryany420]

awesome tutorial! I have a quetion to ask if dont mind: how shall we deal with upsert/delete in those landing/clean/curated zones? I know databricks has similar archtechture with brozne/silver/gold, but it comes with delta lake. if our destination is Redshift, should we move data into Redshift(RDBMS) in earlier stage, like before curated zone. I also send you email, hope you can help to answer. thanks heaps....

[@user-ib4pm2vw5x]

Can we do machine learning algorithm in glue job using coding

[@AWSTutorialsOnline]

Yes, you can but not recommended. You should use glue job for feature engineering but not for training the model. Model training should be done in SageMaker.