Can validate the request param in the api using a validation library, to only accept values in the allowed range. If the request is outside the range you can either return a response to the client telling them it's an invalid request, or send the request to the DB but with the max range on the query. Something like that. You could also write your own DB query wrappers/facades that have the limit already set there, and only use those in the application code rather than using the DB client directly.
In backend for offset and limit pagination we can use db limit and offset parameter while creating a query to fetch that many rows and this quite inefficient in certain scenarios. Keyset pagination which is quite efficient but can’t directly query a particular page. Either go forward or backward in rows with a particular limit.
This is really wonderful explanation. I know its not in scope of this series but out of curiosity would you mind throwing some light on how backing code for pagination can be implemented? Will it use some sort of sorted cache from where it returns results based on offset and max limit or is there a better way to get this done?
Clear and crisp explanation but I've one question. Do we need to hit db for every request, Is it a good idea ? If not then what are the other ways we fetch x amount to data in once(certain threshold) and serve in 20-20 chunks ?
Definitely we could fetch a threshold number of rows and warumup your cache and grab any new data only through but offset-limit pagination is quite inefficient imagine there is no cache is such case every single query of limit and offset you are hitting db grabbing all the rows and then applying limit and offset on them. Also if you are inserting new items as well there can be cases where you could have duplicate items among your list with offset-page pagination Other technique that could be useful keyset pagination where you can’t actually pass offset but can behaviour of pagination with previous and next page with a certain limit just that can’t jump to a particular page but definitely traverse them limit you have set at a time.
This is nice topic. Sometime I feel that pure REST approach is not suitable for business need. It mainly revolve around single entity but business application has complex need like to filter. Let's say for example Employee Belong to specific company and organization and there are multiple attribute associated with that and we have to pass multiple filter attribute. I feel that sometime pragmatic approach will make more sense and expose end-point like /search / filters etc will make more sense.
Could you show a sample pagination and filtering code for an e commerce site for more clarity .. Wondering how the offset limit works when the reasons body has 1000 + record s in that case for one request pay load how you return so many by setting the server side max limit 20 ?? TIA
Hi,your explanation is good. I have issue with pagination. I using one azure api to get all items. In azure portal, 500 items are there but I am getting 200 items while querying api. I applied limit and offset as well but not working. Please help.
Designing the APIs are fine, but how to implement such things? What is the right approach. Let's assume that I am building an ecommerce site and where from the database I want top 20 filtered items, how to do that in code? Thanks for the awesome content. PS: All theory and no code, makes jhony a dull boy 😉
This explanation is so valuable thank you for that , also I would like to suggest if possible to do a video demo with a limited paging server response for example : API like Zoho, or deluge code based API's , how to use the explained methods on a study case like this
Mam'm how could I understand my frontend before making backend server. If I created a registration page and login page are on different pages then it's important to make the same thing on the backed side. So, I will have to login and registration page (register, update, compare password pages) are on the different pages or same page in backed side.
hey i was asked a question where i told about offset for pagination, but interviewer told the data is added on api all the time, so when i call the next page , i might get the same data in the next offset as well coz new data was appended by time i clicked on next page, so how to handle such cases???
simple function that checks if the previous data shown != the new data being shown this time? Source: Copy-Paste Buffers in Systems use this all the time to prevent the same shit from being copied and pasted over and over again... Have you figured out a better answer for this yet?
Hi Yogita In case of pagination are all responses for all pages cached at server side at once? And then they are dispatched to client page by page as per the request? If they are cached on Server side when will the cache cleared? Thank you for all your videos 🙏
This is unlikely, it is mostly implemented by directly executing query on database, caching layer is added to improve performance, but caching entire thing in the cache defeats the purpose of caching
You did not discuss this scenario: if client request for 100 items and server max-limit is 20, how client will be able to request items 20 to 39 in the next request? How client will set its offset ? I am assuming client logic should have offset=lastoffset+numberOfItemsReceived
Using an OFFSET for huge data will still be inefficient for MILLIONS of rows as the database still needs to skip all the rows specified by offset. I expected you to give a solution to this problem in this video as the title specified "millions".
Generally, there sbould not be a payload associated with a GET request. For a POST request, filtration might not be something that is required in the usual scenarios.
Hi I have a quick question. So if i have 50 orders fetched from api and are available in dom. Now i want to implement search the order by id. In this case shall I do a internal filter in the dataset or shall I fetched filtered data from api? What would be the best approach to do it?
Fetch it from the api. You may not know what ids are there in the current data you fetched. So it can give no result too if searched by id. Also id's are indexed in databases. So fetch call is not that costly.
I have few questions on pagination. Can you explain about the page count. How to determine how many number of pages for each query. Say the query returned 90 rows, and if the results per page is 10. I need to show 9 pages (1 2 3 4 .. 9) 2. See this scenario. I have a query which returns 100 records and on top of that I need to apply some business logic and filter some data, so after filter I got 70 records. Now how pagination work when user click on page 2. (Point to be noted there might be thousands os records returned at a time for a query).
Pagination should be applied on the final records being exposed to the clients. In the case where only 70 records are accessible by the client, pagination should create only 7 pages in totality. Moreover, in these cases, the client should never know that there are actually 100 records on top of which a biz. logic filters them down to 70. Otherwise, this would be an abstraction leak.
@@UtkarshSinghPaliwal yup I agree. But my question is when I click on page two, should query the db get 100 and apply business logic again to filter and show the contents of next set of records?? If we do this there will be performance impact. One solution is to store the filtered logic and Show the 2 nd page and so on. Should we use Redis or java cache us my next question.
@@sumanthkeith1 That's a pretty decent problem statement. We need to paginate and we don't have the records to send back without applying some business logic. In this case I'd say to choose either of two things, may be not use pagination at all or cache the indexes of the records and some meta if required. I'm presuming that querying data more than once is very expensive so that logic is not an option. Now with both the cases suggested, two problems arise, one with both the paths. If no pagination used, data may not even fit in a single response in which case you may provide a download link instead. In the other case of caching, you'll have to deal with all the caching related issues like data consistency etc. So you'll have to choose either of the two evils depending on your particular costs and usage.
Is there any specific form of response code sent by the server when the limit set by the client in the query param is larger than the max limit set in the server. I was thinking if it will send either of 400 ( Bad request ) or 413 ( payload too large ), but again you get a response with valid set of items in the server. So it seems to use some specific 2XX code itself. Could you please suggest or throw some pointer on what that might be?
How do I get to know till what number should I make the pagination buttons? Because if I don't specify it then the next button will return an empty page sometime in other words the grt request will return an empty array
When you receive paginated data, the resultset should also set total records , what is the pagesize. Once you have these two data point, you know which will be the last page
Is using OFFSET a good idea to skip resources? I have heard that using OFFSET in SQL can reduce the performance of DB as if there are a large no. of records, the DB has to pull the tuples from the disk but not include them in the RESULT SET
Critical: what if you request, say first 10 (sorted by latest) orders and then 1 new order received. Now if you request next page then 1 order (1st of second page) will be repeated because thats 11th order now (not 10th anymore). How can we fix that.
This is not necessarily a technology issue, this could be an expected behavior as well. But if you want more predictable behavior use cursor based pagination
Nice explanation but only for a very primitive audience to understand. It does not contain any technical details on how it works or implemented . If you could add more details on server side services to map to your URI examples that would be helpful.
Expecting implementations of concepts as well in all videos
This video was very helpful and got more info . Thanks
Some information on how the backend queries will manage limit and offset would have been helpful to tie in the solution end to end.
Same, at moment I'm making query string and based on given filters I conditionally appending where clauses. Could you let me know your approach?
Can validate the request param in the api using a validation library, to only accept values in the allowed range. If the request is outside the range you can either return a response to the client telling them it's an invalid request, or send the request to the DB but with the max range on the query. Something like that.
You could also write your own DB query wrappers/facades that have the limit already set there, and only use those in the application code rather than using the DB client directly.
Please also make a few videos on Low Level Design and CDNs.
thanks alot!I was struggling to anwer these questions in interview .I must say it's a easy way to understand these concepts.
Wow terrific to the point with clear and clean message.
Great and clear explanation 👏👏 Thanks a lot
by your presentation we easily got the concepts, thanks to your editor😅😅
you're super underrated
In backend for offset and limit pagination we can use db limit and offset parameter while creating a query to fetch that many rows and this quite inefficient in certain scenarios.
Keyset pagination which is quite efficient but can’t directly query a particular page. Either go forward or backward in rows with a particular limit.
Thank you so much for the effort.
appreciate it
Loay from Egypt
Great. Wonderful explanation.......
This is really wonderful explanation. I know its not in scope of this series but out of curiosity would you mind throwing some light on how backing code for pagination can be implemented? Will it use some sort of sorted cache from where it returns results based on offset and max limit or is there a better way to get this done?
mam you explain awesome. Thanks
Found 2 channels useful so far .. Ur channel and another.. kudos
Plz mention the other one
@@aatiqaghazali7850 Rahul nath
greatly explained
Clear and crisp explanation but I've one question. Do we need to hit db for every request, Is it a good idea ? If not then what are the other ways we fetch x amount to data in once(certain threshold) and serve in 20-20 chunks ?
Definitely we could fetch a threshold number of rows and warumup your cache and grab any new data only through but offset-limit pagination is quite inefficient imagine there is no cache is such case every single query of limit and offset you are hitting db grabbing all the rows and then applying limit and offset on them.
Also if you are inserting new items as well there can be cases where you could have duplicate items among your list with offset-page pagination
Other technique that could be useful keyset pagination where you can’t actually pass offset but can behaviour of pagination with previous and next page with a certain limit just that can’t jump to a particular page but definitely traverse them limit you have set at a time.
Explaination is excellent please provide practical video if possible
just one word Awesome.
You did a great job with this video, thank you
Dope quality Explanation.
Your explainations are the best, right to the point, well illustrated, real world examples. Thank you.
This is nice topic. Sometime I feel that pure REST approach is not suitable for business need. It mainly revolve around single entity but business application has complex need like to filter. Let's say for example Employee Belong to specific company and organization and there are multiple attribute associated with that and we have to pass multiple filter attribute. I feel that sometime pragmatic approach will make more sense and expose end-point like /search / filters etc will make more sense.
Nice. Any video on cursor vs offset based pagination
Thanks a lot for these wonderful videos. I've become a better Engineer with the help of your videos. Keep up the awesome work 👏
Amazing explanation
Very helpful video
Could you show a sample pagination and filtering code for an e commerce site for more clarity .. Wondering how the offset limit works when the reasons body has 1000 + record s in that case for one request pay load how you return so many by setting the server side max limit 20 ?? TIA
Mam have you created video on implementation?
Thanks for posting this one
Love this video.
Can you please create a video dedicated to API Security? How do you secure an API?
amazing video thank you!
My pleasure!
Thank you for your video but there is some limitation in LIMIT and OFFSET. it will be huge cost involved for RU/s
One more way to reduce payload is sending it back as a stream, can you share use case for the same?
Could it be video/audio streaming?
Thanks
Super :)
Hi teacher thanks❤
Hi,your explanation is good.
I have issue with pagination. I using one azure api to get all items. In azure portal, 500 items are there but I am getting 200 items while querying api. I applied limit and offset as well but not working. Please help.
It's clear ..but there is more link to fetch more data in some pagination APIs
nice
Can you please cover REST API security also?
Does api works to show products which are on low prices due to price error listed on amazon or flipkart?
Designing the APIs are fine, but how to implement such things? What is the right approach. Let's assume that I am building an ecommerce site and where from the database I want top 20 filtered items, how to do that in code? Thanks for the awesome content.
PS: All theory and no code, makes jhony a dull boy 😉
Thank u
This explanation is so valuable thank you for that , also I would like to suggest if possible to do a video demo with a limited paging server response for example : API like Zoho, or deluge code based API's , how to use the explained methods on a study case like this
Mam'm how could I understand my frontend before making backend server. If I created a registration page and login page are on different pages then it's important to make the same thing on the backed side. So, I will have to login and registration page (register, update, compare password pages) are on the different pages or same page in backed side.
Thanks !!
Nic mam
hey i was asked a question where i told about offset for pagination, but interviewer told the data is added on api all the time, so when i call the next page , i might get the same data in the next offset as well coz new data was appended by time i clicked on next page, so how to handle such cases???
simple function that checks if the previous data shown != the new data being shown this time?
Source: Copy-Paste Buffers in Systems use this all the time to prevent the same shit from being copied and pasted over and over again...
Have you figured out a better answer for this yet?
Thank you so much 🔥
Hi Yogita In case of pagination are all responses for all pages cached at server side at once? And then they are dispatched to client page by page as per the request? If they are cached on Server side when will the cache cleared? Thank you for all your videos 🙏
This is unlikely, it is mostly implemented by directly executing query on database, caching layer is added to improve performance, but caching entire thing in the cache defeats the purpose of caching
You did not discuss this scenario: if client request for 100 items and server max-limit is 20, how client will be able to request items 20 to 39 in the next request? How client will set its offset ?
I am assuming client logic should have offset=lastoffset+numberOfItemsReceived
That’s implied my friend.
how does this compare to graphQL?
Using an OFFSET for huge data will still be inefficient for MILLIONS of rows as the database still needs to skip all the rows specified by offset. I expected you to give a solution to this problem in this video as the title specified "millions".
will this work for mongodb?
please cover sorting also
On the server in every request we will be fetching all the data(from database) then paginating it right, assuming no caching is done??
No. We can write the DB query which take the limit and offset numbers as arguments and fetch only the required data from the Database.
We never do that, consider server contains trillion data in table if you fetch all the data then it will go down. Read @Utkarsh comment.
What is there is no pagination.. the data is loaded based on the user scrolling the screen..
how the client side decide total page count
can we send a json data in filter if filtering data is complex?
I currently POST and json body for it currently
Generally, there sbould not be a payload associated with a GET request. For a POST request, filtration might not be something that is required in the usual scenarios.
Hi
I have a quick question.
So if i have 50 orders fetched from api and are available in dom. Now i want to implement search the order by id. In this case shall I do a internal filter in the dataset or shall I fetched filtered data from api?
What would be the best approach to do it?
Fetch it from the api. You may not know what ids are there in the current data you fetched. So it can give no result too if searched by id. Also id's are indexed in databases. So fetch call is not that costly.
I have few questions on pagination.
Can you explain about the page count. How to determine how many number of pages for each query. Say the query returned 90 rows, and if the results per page is 10. I need to show 9 pages (1 2 3 4 .. 9)
2. See this scenario. I have a query which returns 100 records and on top of that I need to apply some business logic and filter some data, so after filter I got 70 records. Now how pagination work when user click on page 2. (Point to be noted there might be thousands os records returned at a time for a query).
Pagination should be applied on the final records being exposed to the clients. In the case where only 70 records are accessible by the client, pagination should create only 7 pages in totality. Moreover, in these cases, the client should never know that there are actually 100 records on top of which a biz. logic filters them down to 70. Otherwise, this would be an abstraction leak.
@@UtkarshSinghPaliwal yup I agree. But my question is when I click on page two, should query the db get 100 and apply business logic again to filter and show the contents of next set of records?? If we do this there will be performance impact.
One solution is to store the filtered logic and Show the 2 nd page and so on.
Should we use Redis or java cache us my next question.
@@sumanthkeith1 That's a pretty decent problem statement. We need to paginate and we don't have the records to send back without applying some business logic. In this case I'd say to choose either of two things, may be not use pagination at all or cache the indexes of the records and some meta if required. I'm presuming that querying data more than once is very expensive so that logic is not an option. Now with both the cases suggested, two problems arise, one with both the paths. If no pagination used, data may not even fit in a single response in which case you may provide a download link instead. In the other case of caching, you'll have to deal with all the caching related issues like data consistency etc. So you'll have to choose either of the two evils depending on your particular costs and usage.
Is there any specific form of response code sent by the server when the limit set by the client in the query param is larger than the max limit set in the server. I was thinking if it will send either of 400 ( Bad request ) or 413 ( payload too large ), but again you get a response with valid set of items in the server. So it seems to use some specific 2XX code itself. Could you please suggest or throw some pointer on what that might be?
Probably a 422 would do.
You can throw any status code😅
U looks like NAIROBI of money heist 😀
💕
CDN can you share
How do I get to know till what number should I make the pagination buttons? Because if I don't specify it then the next button will return an empty page sometime in other words the grt request will return an empty array
When you receive paginated data, the resultset should also set total records , what is the pagesize. Once you have these two data point, you know which will be the last page
actually i learned these terms pagination, limit, offset, filters while learning DBMS and i think they're more related to that that rest Api
Is using OFFSET a good idea to skip resources? I have heard that using OFFSET in SQL can reduce the performance of DB as if there are a large no. of records, the DB has to pull the tuples from the disk but not include them in the RESULT SET
Are there any issues if we use post instead of get in this and send all the required data in requestBody rather than query params?
That will be violation of rest conventions, but if the params is too big, then you may have to use POST
Bad Practice
Critical: what if you request, say first 10 (sorted by latest) orders and then 1 new order received. Now if you request next page then 1 order (1st of second page) will be repeated because thats 11th order now (not 10th anymore). How can we fix that.
This is not necessarily a technology issue, this could be an expected behavior as well. But if you want more predictable behavior use cursor based pagination
in this video u r looking lean
Didn't work.. getting complete result
😊
Do you offer a service to integrate an API from my distributor to my website for orders and dropshipping
Nice explanation but only for a very primitive audience to understand. It does not contain any technical details on how it works or implemented . If you could add more details on server side services to map to your URI examples that would be helpful.
Sure, you can create a video on that and I will be happy to endorse it 😅
Wonderful explanation