Thank you for your video. It's great. But, I have a consideration: the total number of user posts is 100000, chunk_size is 10000, so each time we run lambda job, we can pull all the data (100000) - call parallelly with 10 chunks or each call we just pull one chunk which means we need to call 10 times to get full data?
Blog at www.startdataengineering.com/post/pull-data-from-api-using-lambda-s3/
Thank you for your video. It's great. But, I have a consideration: the total number of user posts is 100000, chunk_size is 10000, so each time we run lambda job, we can pull all the data (100000) - call parallelly with 10 chunks or each call we just pull one chunk which means we need to call 10 times to get full data?
Great! Thank you very much!
hey