Good explanation. Can I use Spark streaming to read from hive table with millions of records. Say my usecase is like I have a source java program which uses Spark stream to read data from hive table and keep on posting data packets to some kafka connector. I am mainly concerned with memory usage. What is your opinion on this ?
Hello, As you mentioned on streaming source-File. Is it possible to stream a file as source and as and when new lines append to a file(csv file), is it possible to do streaming on top of that? if yes.. which streaming source method i need to call...please help
in Dstream in one batch you get only one RDD, check below link stackoverflow.com/questions/35164634/how-many-rdds-does-dstream-generate-for-a-batch-interval
this explanation was very good, thank you sir
Excellent explanation. Keep up the good work and thanks !!
Good explanation. I didn’t find batch streaming and sliding windows concepts explained like this.
And I am
Thanks.. for crystal clear explanation 👍
Excellent job you doing Lime Guru. Thanks for the effort . Expect more on Apache Spark
How to create state again and add it in rocksDB when we restarts our stream job from new checkpoint location?
Job well done. Nicely explained. This video deserves more views and likes. Kudos.👍👍👍
Good explanation 😍
Good explanation.
Good explanation. Can I use Spark streaming to read from hive table with millions of records.
Say my usecase is like I have a source java program which uses Spark stream to read data from hive table and keep on posting data packets to some kafka connector.
I am mainly concerned with memory usage. What is your opinion on this ?
Thanks too much
Nicely explain
Hello, As you mentioned on streaming source-File. Is it possible to stream a file as source and as and when new lines append to a file(csv file), is it possible to do streaming on top of that? if yes.. which streaming source method i need to call...please help
Good job
in Dstream in one batch you get only one RDD, check below link
stackoverflow.com/questions/35164634/how-many-rdds-does-dstream-generate-for-a-batch-interval