Interview QA | Spring Batch Partitioning example | Scaling and Parallel Processing | JavaTechie
HTML-код
- Опубликовано: 7 окт 2024
- This tutorial will give you complete picture about How to use spring batch Partitioning to process batch job faster with better performance
In Spring Batch, “Partitioning” is “multiple threads to process a range of data each”.
#javatechie #SpringBoot #SpringBatch
GitHub:
github.com/Jav...
Blogs:
/ javatechie
Facebook:
/ javatechie
guys if you like this video please do subscribe now and press the bell icon to not miss any update from Java Techie
Disclaimer/Policy:
--------------------------------
Note : All uploaded content in this channel is mine and its not copied from any community ,
you are free to use source code from above mentioned GitHub account
the content on this channel is good and doesn't discuss general things like what is in other channels... I loved
Thanks David ☺️.
Many thanks Basant!
As always awesome contents.
Amazing lesson brother... Keep going !
Nice video. Thank you. Always excellent ..
keep up the great work
Using SynchronizedItemStreamReader would be ideal when reading from same file using multiple threads.
This is all good, but i want to tell you that you should not take one file and split record counts while explaining Partitioning. You should have taken multiple files and pass file names from master to slave as reader param. And your Partitioning number could be based on supported core on machine.
Why we should not take single file actually I am taking single large cab and facing issue
Any reason why we should not take single file any help please
🎯 Key Takeaways for quick navigation:
00:29 Phân *vùng nghĩa là chia tập dữ liệu thành các phần và gán cho các luồng xử lý khác nhau.*
00:58 Bạn *có thể tăng hiệu suất bằng cách sử dụng nhiều luồng hơn, mỗi luồng xử lý ít dữ liệu hơn.*
06:56 PartitionHandler *định nghĩa số lượng luồng, kích thước mỗi phân vùng và bước thực thi các phân vùng đó.*
11:49 Bước *master gọi đến PartitionHandler để phân chia nhiệm vụ, các bước slave thực thi các phần được phân chia.*
20:10 Có *thể cấu hình số lượng luồng và kích thước phân vùng dựa trên yêu cầu để tối ưu hiệu suất.*
Made with HARPA AI
thank you
Saviour 🙌
Thanks for this vedio. could you please make a vedio on Graph QL Subscription with spring JPA (Data Fetching)
It's already there in my channel but I will do one remake of it
Thanks. What should be the best approach to retrieve millions of records by using spring data jpa from a view (which contains data from multiple table)? Any reference.
awsome thx
Please make video on lock while we have multiple node or multiple server how communicate each server
Sir colud u please upload splunk vedio. Monitoring purpose
You explain it very well. Thank you. if possible plz explain the scenario while we insert data into db and runtime exception occurred then what happens ? i mean it will continue next run where it stopped if yes then how constraint will work on table like PK, UNIQUE, etc? or it will rollback to previous commit ?
Yes buddy it's on my queue
@@Javatechie Hi, How to stop all the partitions? Even If one record failed to insert while we insert data into the db.
Hi There, one quick question - partitioning over single instance of server is fine, but how to deal when there are multiple instances of server where this batch application is running on.
e.g. there is a file with 1 million records and in my cloud environment there are 4 server instances where this batch application is running. how to develop the code in this cluster kind of environment for efficiently handling the record processsing using spring batch? - Thnx
Reply if u got this answer ?
Please reply me also
Thanks for this video. Batch process sometimes has an error like this => Caused by: java.io.IOException: Stream closed, org.springframework.batch.core.JobExecutionException: Partition handler returned an unsuccessful step but it gets inserted even though it gets an error.
Yes this is because who ever the thread executed first will try to close the stream that is why this error is coming so we can handle this error
Please check if you are using threadsafe reader or not. If you are not sure what you are using ping your reader. If you are nou using thread safe reader. Make that thread safe.
@@Javatechie how can we add this control? Any suggestions? I am new on multithreading.
@@akntopbas6492 implement your own reader, override the close() method of reader,
Can you create one video for tasklet and stepExecution context.
Great video! Thanks a lot!!
Just one question, if im reading a txt file from an S3 and i do not know how long it is, how can i work with the min max value inside the partitioner?
You need to use some utility method to find out min and max without knowing this value we can't apply partitioning
@@Javatechie I see, thanks!
HI BRO I WATCH YOUR VIDEOS...ONE SUGGESTION REGARDING VIDEOS YOU...... MAKE ONE VIDEO EVERY YEAR JUST TO INFORM YOUR SUBSCRIBERS SO THAT THEY HAVE BASIC UNDERSTANDING WHICH TECHNOLOGY IS CURRENTLY IN TRENDS AND WHAT ARE THE TECHNOLOGIES THAT ARE CURRENTLY USED WITH JAVA IN INDUSTRY AND TECHNOLOGY AND ITS SCOPE FROM CAREER PERSPECTIVE BECAUSE THAT WOULD REALLY HELP US TO UNDERSTAND WHAT SHOULD WE LEARN.... THE ROAD MAP FOR LEARNING TECHNOLOGIES AND PLEASE MAKE VIDEOS REGARDING PROJECT DEVELOPMENT TIPS AND TRICKS AND MISTAKES THAT WE SHOULD AVOID WHILE DEVELOPING PROJECTS AND ONCE IN A WHILE DO QnA VIDEOS TOO.....
Please give a video to save and get the fifty thousand records using jpa in a fraction of seconds
You can use same approach by extending limit of your csv file .
Here, I am not reading from files
@@manosuji1636 what's your source
My source is docx file. Now I extract this file into objects based on more criterias than I have to save it. When I save this huge record , it took more than 30 seconds. I have to sort out it
Good tutorial!! What if any Step method has some parameters in it `public Step step1(JdbcBatchItemWriter writer, JdbcCursorItemReader readerDataInitialization). How to call this method from partitionHandler()
please create video on design patterns also
Very informative video. If the csv contains more than 1000 records, how can the code will behave. How can we set max value dynamically and does this max value refers to total no of records in csv ?
Yes total row count. To make it dynamic we need to write logic to get max row count from file
Thanks for the reply. I tried the same example with StaxEventItemReader and TaskExecutor to read the xml and write into db. But it's not working. It's working only without taskexecutor. My requirement is read the xml and insert the data into DB. But the xml file is a larger one. Please suggest the solution for this to make this process faster.
When you say it's not working it means you might getting some error? What is that
One Quick Question...While partitioning u used minValue,maxValue. Why u selected these names ? Are these predefined keys for partition ?
No it's not predefined you can give any name
Sir, pls make a video about connect two micro services using interceptor
I will check this
@@Javatechie thanks for your reply sir.
Please make a video bulk pdf file processing.
Could u please start realtime project course
Hi thanks for your video. I have one question. How to stop the job(all partitions should stop) even if one record failed to insert into db. Could you please suggest me.
We need to implement transactions in that scenario or some rollback mechanism
Can someone please explain, how are we using "minValue" and "maxValue" set on ExecutionContext in this entire process?
Also how does reader know it need to read from 1 to 500 and then 501 to 1000 on different partition?
Springboot, microservice, some realtime project with money
Hi Sir, chunk size and target size should match always? I mean, we nee to pass grid size which will divide total number of rows and give exact chunk size?
Exactly
How to schedule the batch job to run in a particular frequency? Like every few minutes or every hour so that it keeps processing new records?
It's simple just use spring scheduler and specify cron expression for your frequency. You can check my spring scheduler video
@@Javatechie I tried implementing it with scheduler with Cron expression, but it just executes at that timings without checking if the previous instance of job has finished or not. Is there a more sophisticated way like how Quartz has where the scheduler skips the run when the existing instance is still running? Also which tracts the runs.
You can do something like this buddy
import org.springframework.batch.core.Job;
import org.springframework.batch.core.JobExecution;
import org.springframework.batch.core.JobParameters;
import org.springframework.batch.core.JobParametersBuilder;
import org.springframework.batch.core.launch.JobLauncher;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
@Component
public class ScheduledBatchJob {
@Autowired
private JobLauncher jobLauncher;
@Autowired
private Job job;
private JobExecution lastExecution;
@Scheduled(fixedRate = 300000) // 300000ms = 5 minutes
public void runJob() throws Exception {
if (lastExecution == null || lastExecution.isRunning()) {
System.out.println("Previous job is still running. Skipping this interval.");
return;
}
JobParameters jobParameters = new JobParametersBuilder()
.addLong("time", System.currentTimeMillis())
.toJobParameters();
lastExecution = jobLauncher.run(job, jobParameters);
}
}
@@Javatechie Thank you.
Can you please do the same one for spring boot 3, as I'm trying partitioning but my batch is not moving ahead after displaying partitions in my terminal
Hi Java techie, can you please tell me how to read all data from multiple file locations.
Thank you
Can I have two instances running at the same time? what is its behavior of it? I want to use sprintbatch but the first step is to read from a database; in addition, I also need more than one instance running at same time. so how I can avoid both instances read the same data from the source
Good observation buddy. Not sure about behaviour will check and update
May be this is not a optimal approach, we could go for taskexecutor
Yes here we used task executor , as i mentioned in video you need to increase grid size
Hi Basant, could you please update the git repo for this with springboot3 changes
How can I pass dynamic number of lines in CustomRangePartitioner? I have function to count lines in csv. Can I use JobParameters?
Hello, thanks for the video. I am using this spring partition but I am getting resultset SQL exception issue as "resultset after last row." Can you help me how to fix this ?
StepBuilderFactory is deprecated now I think
Yes it is , already I uploaded migration guide please check this Spring Batch Migration Guides | Spring Boot 2.x to 3.x | JavaTechie
ruclips.net/video/_TSjkSn2yvQ/видео.html
I am also using partitioning but partitioning based on my sorting these the ids since they are non sequential as I’m reading and deleting it after processing. I have also added a chunk listener what I have observed is that beforeChunk and afterChunk is getting called twice resulting in commit count as 2. Not sure why it is happening? Could you please let me know?
Can you share your GitHub link if it's not corporate code
will it work on multiple process? for example if i run the batch process on multiple pods.. if pod1 receives the request and execute the job. .will pod2 can automatically pickup the steps for execution?
can u explain the spring batch using tasklet
In my case i have list of object instead of csv.how can i insert the data using batch
how to handle remote partitioning.. If i need to distribute partition to different instances of my application ?
Not sure buddy I will check and update you
@@Javatechie thank you
@@Javatechie Need one help.. I am using partitioning in my use case. I have Itemreader which readers data from DB, partitioning it.. After processing I am writing back data in DB. I observed there is some data inconsistency in DB. Sometimes one of the slaveStep : partition fails or sometimes data is not committed in DB.. It is random. How Spring Batch creates Transactions. Is it transaction per Partition ? Or do we need to maintain thread synchronization ?
Question, what is the difference between taskexector from your previous vedio and partitioning ? Ideally it both runs using parrlel threads right?
here threads have control over the records. thread 1 handles from 1 to 500 and thread 2 handles 501 to 1000. in previous executor, the 10 chunks were taking and writing random data.
@@SaurabhKumar-fo6zp thanks
You will start realtime application
How to stop a job once any of the partitioned step throws an exception, as currently other partitioned steps keep running till the end and after they complete, the job stop with unsuccessful return code.
Hi , i tried this partition and geeting error - inputstream has already been read- do not use inputStreamResourse if a stream read multiple time.
I have used SimpleAsyncTaskExecuter instead of taskexecuter.
Can u please help
Hi here we are inserting in a database, if I need to call a REST service how can I proceed?
Dear All, can anybody tell me , in the reader, processor and writer, do we need to use entity class or can we create response class , so I don't want to expose entity class can anybody suggest on this. It's real-time scenario working on fist time spring batch. Kindly suggest where I need to avoid entity class exposure to public ...as I am creating stand alone application with scheduler,
sir, after implementing partitioning, data will be inserted in Table sequenctially like 1,2,3,4 and 501,502,503 etc? or it wont insert in order?
It depends but most probably it shouldn't follow order
@@Javatechie Thanks for reply but you said in the session to maintain order 1-500,500-1000, we go for partition .. if it does not follow why I will go to partition instead I will take more threads and read the data . In either of the case or won’t follow order right .. what is the main benefit of partition compare to tradition approach?
will these two threads run parally?
Yes
Hi sir, could you please help. If ids are not in sequence. What you have shown ids are in sequence manner.. Let say if ids are alphanumeric. Then how will you split the list based on grid size.
No only based on primary key we can
@@Javatechie but in practically. As per business requirement. We won't have primary key as sequence. So there should be way to split the list into k part where k is grid size.
It doesn't care about your sequence number 1 to n .it deals with row count
@@Javatechie let's say I have 10 row ids are aa01, aa02..... aa10 and min =1 max=10 grid size =3 then we need to pass 3 data blocks to reader while setting into execution context map.. Then how will you divide above 10 ids into 3 blocks. In reader you can't do query like where id >aa01& I'd
@@Javatechie ids are alphanumeric
I am getting my records processed twice. any solution for this?
I am facing one bug with that code, when file is already available in folder, it picks up and process fine with partition. However, when file is not in folder and you run the project, partition handler does not get called again. It only calls once when you run the project first time. How can we run partition handler again after hitting data save endpoint. What I want to achieve is after running the project, I will add file into folder and call endpoint for processing file into database. @Javatechie would you please show any modified solution here?
Did you get this answer ?
Both of you you should have understood how can a job will be execute I have explained all the 3 way in my part 1 video please have a look
In Spring Batch fault tolerant along with multi threading will work?
Yes why not
I have tried to run through spring boot 3.1.5 but I am seeing only the main thread name in the logs and it is not showing the task executor thread names.
Okay let me check and update you but just make sure you have defined the bean of TaskExecutor
Add Subtitles or CC please 🙏🙏🙏
It's there buddy RUclips use takes some time to sync this
Can anyone know that How to sort *priorityQueue*
Pls make one video on how implement ETag in Spring boot MS...