What’s the easiest way to manage streams from mongo into snowflake using Kafka. The data structure changes and the update replace does not include data elements of the prior record. We require only the most recent update to the record no history
Joe you’re a great instructor. Starting with the context/use case and then referencing the parts is so appreciated. Your channel is wildly underrated. Also love the humor references.
I didn´t wanted to watch a 1 hour video, I think is the first time I do something like this, and wow. I learned more in one hour than in 2 days readding weird written tutorials, guides and short videos. Thank you very much.
Summary: With MongoDB you have to consider your access patterns not only for indexing but also for schema design. There are three options for child documents: 1. Embedding 2. Array of references in the parent 3. References in the children (foreign key)
Two way referencing is just a PURE GOLD!!! Now it seems obvious, but I was trying to solve similiar problem without that 2WR knowladge 🙈, and made no progress for few hours, THANKS MAN!
Really enjoyed your video. Was kinda funny when you said MongoDB has no Rules but then we have a section with Rules. hehe. Altough they are very helpful to keep in mind. Thanks for the great content!
Excellent presentation, it served super well as a foundation for the design discussions about our NoSQL schema. Although we went to with CouchDB all that you said felt relevant. Instant like!
This was really good. One thing that made me laugh is in the begining you stressed that there are no rules and then later you have "Rule 1: favour embedding" haha I just thought that was funny. Thanks for the presentation, it was helpful!
Thank you so much for posting this presentation. I appreciate the multiple recaps throughout the lecture, helped me a lot. And I appreciate the many resources.
I'm sorry, but this sounded like a sales pitch :) Understandable since you work for MongoDB. MongoDB has a predecessor, namely XML databases. MongoDB has a lot in common with XML databases. I used to work a lot with XML databases and my experience is that document databases are a * bad * fit for business domain models. You talked about cons and you didn't mention the most important con: update anomalies. But wait a minute. You mentioned this problem and came up with a beautiful solution: a reference. And even better, there is something like unbounded references. I would say, keep evolving and you end up with an SQL database :). In my experience, document databases are great when you store documents as is. For example when you have a message warehouse, or for logging/auditing. XML databases are also used by publishers to store books (DocBook and DITA for example), newspapers etcetera. But using a document database for your business model. That's a bad idea to my opinion. I have the experience to use a very performant XML database as the enterprise database for a company and it had very nasty legal consequences. It was just not the right fit . A relational model is far better. It is also much better to have a schema for you business domain model. I would get very nervous if a developer would store its documents without a schema. No rules, really? Is that a positive thing for your vital business data? Good luck!
When it says no rules. It means you define the rules that fit your business rather than let the rules define your business. you can enforce rules at the db level
I hate your vocal fry so much my god I hope this video is worth it EDIT: My opinion has not changed - your vocal fry is indeed terrible, but the video is extremely good, it's informative, concise, well presented and paced. I will be recommending it to everyone starting with mongo db. Also you seem like a really likeable guy :D
@@JoeKarlsson I know my message is confusing because I had such mixed feelings about this video, but everyone on my team loves it :D Apparently my ears are super sensitive to fry frequencies and nobody else minds it.
@@someoneelse5005 Your review was hilarious - It's a actually a glowing review. I'm considering printing it out and framing it! I'm so glad y'all enjoyed it. I'll be making more soon!
Thanks for a good video. Coming from many years with SQL I can clearly see some benefits with NoSQL. The main selling point for me is the loose data structure of the individual documents as well as query speed when everything you need is embedded in a single document. This is where NoSQL really shines over SQL Maybe I'm wrong, but I get the feeling that in most cases we still need traditional SQL-like relationship references between decoupled documents and quickly we end up with something that looks an awful lot like good old SQL.
Super thanks. Something not spoken about a lot. Modifying schemas structure in production. How can I drop a schema property that I don't need any more? Say username was unique on the user and we don't need that anymore in production. How can I go about that?
@@JoeKarlsson thanks for the response. Lost a little. To elaborate e.g. Schema({username: {required: true, unique: true}, firstName:...}) and now we don't need the username anymore. Any operations such as create collection documents without username would error.
If I want to present the elements from my database like a diagram, exist something like that? If I want to search for data it's not possible if I normalize?
There are no visual tools that show normalized diagrams on MongoDB data, however, we do have a tool called MongoDB Compass that makes it really easy to view and navigate your data. www.mongodb.com/products/compass
Instead of referencing can I keep just list of id of another document and just query those specific ids? I mean what will be the performance in that case?
Professional SQL developer here Actually the first part of your video is a little misleading, we do design sql databases with the queries in mind and We denormalize when necessary.
Thanks for the vid, now I need someone opinion. Am build a e-commerce store to sell clothing but am wonder if I should put all the product variants in the same document or reference it since as each product have more than one variants. Max properly be arround 10 variants and I only need the data to display on the home page as in "2 images,price and name". What's your guys opinion on this? And what about the query if I only need 2 images,name and retail price on the landing page.
Depends on how you will use it. Are you planning on showing all variants on your product detail page? If yes, it might be a good idea to embed it since you don't have very many sub variants. Does that help/make sense? 🥰🥰🥰
hi I got a question would it be better to keep reference on both document in one to many scenario or would it be better to just keep the reference on the one document that holds many docs ?
Fantastic Joe. Great Learning. When you talk about embedding example regarding product and parts, you favor parts. But you keep referencing to the size limit of the document. Yes its a problem, But more than that there is a bigger problem of consistency. If you embed a part and If you update the part from a screen, you have inconsistency. When you show the product you will show the old values. I am curious as to why you are not mentioning about inconsistency? I am a newbie in nosql. heavily influence by relational theory. Eager to know your thoughts
Thank you very much for the presentation! So if data I am embedding instead of referencing needs to have its own unique ID. I am used to getting these IDs for free in SQL world... is there something Mongo DB had for this or I am on my own with generating and keeping track of these IDs? Thanks!!
@@JoeKarlsson I was taking about adding an ID to a subitem inside document (embedded structure) that doesn't get an ID by default. Looks like I can generate this ID in application by doing something like BSON::ObjectId.new and it will be unique to the DB
I think it was mentioned that you may not want to embed a document so that we won't fetch extra data that we don't want. I am not sure why this count as reason to not embed. I mean, MongoDB allows the user to specify exactly which fields to load. so, if I do not want to load the embedded document from DB to the application, I can just add it to the exclusion list. however, I know that this will save network bandwidth between DB server and app server but not sure how much impact it will have on the DB server itself; since the DB server still needs to load the document from the filesystem and then send it to the app server.
Hello , I am new to mongodb. In this video there was one point that if you are embedding too much data in the document,it will increase document size and the overhead to send the document over network. But for retrieving the document, we can use projection and get only particular fields that we need. So we are decreasing the network overhead , right?. Please tell me if I am missing something here.
I worked always with mysql so its difficult. Lets say I have this card collection game where a user can have many repeated cards,, I mean I can have the same card multiple times, so I can exchange the card or sell it etc. How can I model my DB to support that functionallity?
Hi Mr.Joe, Great video and I learnt a lot. I was wondering whether you’ll be able to clear a doubt I have. In my project, users and tasks collections. I would like to save who created a task in a “createdBy” field. It’s always gonna be one value and would be selected every time one or multiple tasks are fetched. According to this video, I should embed essential data such as (userId and name) that my application would display. But I want to “refer” the user data so that fields such as name are fetched correctly. I worry if I embed the name and user updates their name, it could become costly to update in all the embeds. May I know what your approach would be? Thanks
how do i handle a situation where we have the different user fields? for example candidate and employer where candidate has email, password, resume, experience, skills and so on where employer has email, password, company size, company history and so on do i have to create everything in a single user modal? or create user plus two separate models for candidate profile and employer profile? i am company from the relational database side so it is bit confusing.
@@JoeKarlsson so those extra fields are created doesn't harm? what i mean. those extra fields might not relate to the candidate or the employer but will remain there with their account?
![Noob To Max With DRAGON REWORK In Blox Fruits [FULL MOVIE]](http://i.ytimg.com/vi/LnBMOinoOvA/mqdefault.jpg)
I'm new at using MongoDB and this video has solved many doubts I had about how to design the ideal schema for my project. Thank you very much!
This is the best thing I have ever heard - thank you!
@@JoeKarlsson have a mongodb/admin job coming up
@@DIZZLEBOI44 Nice!
What’s the easiest way to manage streams from mongo into snowflake using Kafka. The data structure changes and the update replace does not include data elements of the prior record. We require only the most recent update to the record no history
Joe you’re a great instructor. Starting with the context/use case and then referencing the parts is so appreciated. Your channel is wildly underrated. Also love the humor references.
Thank you so much. I really appreciate that.
Rule No.1 is my favorite.
=> Favor embedding unless there is a compelling reason not to.
I didn´t wanted to watch a 1 hour video, I think is the first time I do something like this, and wow. I learned more in one hour than in 2 days readding weird written tutorials, guides and short videos. Thank you very much.
Oh my gosh! That's awesome! I'm so glad it was useful for you! :)
Summary:
With MongoDB you have to consider your access patterns not only for indexing but also for schema design.
There are three options for child documents:
1. Embedding
2. Array of references in the parent
3. References in the children (foreign key)
what i learned after just this single video was 10x times more than a 200pgs doc i read. thank you
I got you :)
No one explains mongodb schema as you did. Thank you man. I finally understood MongoDB
russia is a terrorist state, btw
this one video explained more than a couple videos combined I watched before I stumbled here. :) I have subscribed.
Oh my gosh! Thank you! I'm so glad it was useful! :D
Great video, these intermediate videos are what youtube is lacking
Right? I feel the same way - lots of beginner level content, but not much for people beyond that. So glad you enjoyed this video!
Two way referencing is just a PURE GOLD!!! Now it seems obvious, but I was trying to solve similiar problem without that 2WR knowladge 🙈, and made no progress for few hours, THANKS MAN!
I am creating a bug tracker web app and this has answered my questions about how I should structure my many to many relationship, so thank you!
That's so cool! Can you send me the code? I would love to check out what you're working on!
WoW. Really enjoyed this session. Thanks for the effort
Thank you so much! I am so glad you enjoyed it!
Really enjoyed your video. Was kinda funny when you said MongoDB has no Rules but then we have a section with Rules. hehe. Altough they are very helpful to keep in mind. Thanks for the great content!
hahaha - lol - I def lied there! :P
Super helpful. I was on the fence with PostgreSQL and MongoDB. But this convinced me to go Mongo
This is the best feedback - thank you so much!
Excellent presentation, it served super well as a foundation for the design discussions about our NoSQL schema. Although we went to with CouchDB all that you said felt relevant. Instant like!
This was really good. One thing that made me laugh is in the begining you stressed that there are no rules and then later you have "Rule 1: favour embedding" haha I just thought that was funny. Thanks for the presentation, it was helpful!
LOL - you're totally right. I totally missed the irony! hahaha! :D
Thnx a lot man , this answered so many of my doubts. Deserves 10 million views, but we all know the world 😉
🙌
Thank you so much for posting this presentation. I appreciate the multiple recaps throughout the lecture, helped me a lot.
And I appreciate the many resources.
Solved all my doubts man since morning was searching for embedding design. Thanks a lot 👍
That's amazing! Thank you for sharing!
One of the best video about schema design in mongoDB thanks @Joe Karlsson. I subscribed
Thanks a squillion Joe! 👊🏽
This video is very informative, helpful and enjoyable 😊
This is my favorite comment :D
This was super helpful, Cleared many doubts I had. Thank you very much!
That's amazing! Thank you so much for watching! :D
Great informative video to get started with NoSQL when you come from a relational DB background. Thanks!
I got you! 🥰
I know it’s a bit old, but as a recent convert this was exactly what I needed 😊😊😊
better than official mongodb videos and tutorials.. thanks!
I'm sorry, but this sounded like a sales pitch :) Understandable since you work for MongoDB. MongoDB has a predecessor, namely XML databases. MongoDB has a lot in common with XML databases. I used to work a lot with XML databases and my experience is that document databases are a * bad * fit for business domain models. You talked about cons and you didn't mention the most important con: update anomalies. But wait a minute. You mentioned this problem and came up with a beautiful solution: a reference. And even better, there is something like unbounded references. I would say, keep evolving and you end up with an SQL database :). In my experience, document databases are great when you store documents as is. For example when you have a message warehouse, or for logging/auditing. XML databases are also used by publishers to store books (DocBook and DITA for example), newspapers etcetera. But using a document database for your business model. That's a bad idea to my opinion. I have the experience to use a very performant XML database as the enterprise database for a company and it had very nasty legal consequences. It was just not the right fit . A relational model is far better. It is also much better to have a schema for you business domain model. I would get very nervous if a developer would store its documents without a schema. No rules, really? Is that a positive thing for your vital business data? Good luck!
I'm using mongol for logging and analytics
When it says no rules. It means you define the rules that fit your business rather than let the rules define your business. you can enforce rules at the db level
1:20 -- Already I see you are a genius. Well done with this idea.
You deserve more subscribers! Flawless video, definitely learned a few gems from this one!
Thank you so much Nicolas - I may be biased, but I totally agree ;)
Nice Video , Every thing is crystal clear
you saved my lots of time , i am new to mongoDB from MySql background keep it up.
This is the best compliment! Thank you so much!
Love the 1960's-1970's Campagnolo Record bicycle de-exploding at around 25:20 into the video!
12:32 Embedding
17:10 Referencing
This is a very cool, well explained and fun talk on mongodb. 100% recommeded!
Thanks friend. You explain very good. from colombian, Im speak spanish and I dont have seen any video so well explained in Spanish
Just one! Love that part. Great vid thanks!
I got you
Excellent Joe!! loved you presentation. Really helped me to clarify some doubts on mongo .
I'm so glad to hear that!
Most underrated talks..awesome...👏👏👏👏
Oh my gosh! Thank you!!!
The revisions at the end was helpful. Thank you
YAY
Thank you so much! I finally understand how to design my Schema. I subscribed.
YAY! That's the best - thank you so much!
Does referencing have to use the complicated BSON _id property? Can't we just use another normal property, like, name (presuming it to be unique) ?
Thanks for making such a comprehensive video for us. Stay Blessed.
I got you! 🥰
@@JoeKarlsson started watching MongoDB University course after this video.
Your company must be proud of you. 👌🧡
I hate your vocal fry so much my god I hope this video is worth it
EDIT: My opinion has not changed - your vocal fry is indeed terrible, but the video is extremely good, it's informative, concise, well presented and paced. I will be recommending it to everyone starting with mongo db. Also you seem like a really likeable guy :D
LOL - thank you?
@@JoeKarlsson I know my message is confusing because I had such mixed feelings about this video, but everyone on my team loves it :D Apparently my ears are super sensitive to fry frequencies and nobody else minds it.
@@someoneelse5005 Your review was hilarious - It's a actually a glowing review. I'm considering printing it out and framing it! I'm so glad y'all enjoyed it. I'll be making more soon!
@@JoeKarlsson I hope so, your content is killer :D
I am new to No SQL and coming from RDBMS background. I enjoyed this session and it gave me very good idea about how to design Mondo DB.
Thanks for a good video.
Coming from many years with SQL I can clearly see some benefits with NoSQL. The main selling point for me is the loose data structure of the individual documents as well as query speed when everything you need is embedded in a single document. This is where NoSQL really shines over SQL
Maybe I'm wrong, but I get the feeling that in most cases we still need traditional SQL-like relationship references between decoupled documents and quickly we end up with something that looks an awful lot like good old SQL.
This is the best into for MongoDB. thanks a lot!
I got you
Excelent tutorial, gave me a entire panoram of mongodb
Thanks a bunch ! It was quite useful especially to see patterns at last.
I’m early into the video but I just want to say thank you. I’m new to this, so I’m trying to understand esp coming from relational
Actually quite helpful. Answered most of my questions. Thank you :)
That's awesome! So glad it clicked with you!
Thank you for this great video. Do you think nosql can completely replace sql? I feel it can if the scheme is done properly...
It depends on the used case, but for a lot of cases noSQL databases work just fine
Fantastic and straightforward presentation! Thank you sir!
Oh my gosh! Thank you so much!
Very well explained .. Joe. Super presentation .. Thanks much !!
You are so welcome - I am so glad it helped!
I'm relieved bro this is best and helpful very much thanks ....
🥰🥰🥰
Really helps doing my course project and many thannnnnnks!
Oh my gosh! So glad it's useful!
Mongodb scheme design is well explained. Thank you very much.
I got chu!
Super thanks. Something not spoken about a lot. Modifying schemas structure in production. How can I drop a schema property that I don't need any more? Say username was unique on the user and we don't need that anymore in production. How can I go about that?
Great question! You will have to write an update query to go through all the appropriate documents and update the schema to your new value.
@@JoeKarlsson thanks for the response. Lost a little. To elaborate e.g. Schema({username: {required: true, unique: true}, firstName:...}) and now we don't need the username anymore. Any operations such as create collection documents without username would error.
You really saved me. Thanks a lot!!! Subscribed
this is a really nice video!! thanks a lot for sharing your knowledge it's help me a lot
Yay! Thank you so much! I'm so glad that it's useful :D
Love the office meme. Great tutorial!
one of the best explanations, thanks from Bolivia :D
Oh my gosh! So glad it's useful!
That was great content!!! Answered many questions.
So glad it helped! :D
great!
courage bro!
it has solved many of my doubts
YES! That's awesome!
If I want to present the elements from my database like a diagram, exist something like that? If I want to search for data it's not possible if I normalize?
There are no visual tools that show normalized diagrams on MongoDB data, however, we do have a tool called MongoDB Compass that makes it really easy to view and navigate your data. www.mongodb.com/products/compass
I was looking for it too and found this free online tool:
mongo.tools
I'm newbie and I found this video is amazing, now I can do it.
This is beautifully explained. Thank you sir!
Discovered you from Twitter. Amazing talk Joe 🙏
great , I found it is very help full video to start project with schema design
I got chu ;)
43:09 - Good note
48:50 - Ref tutorial
Thank you for bringing up a really cool video!
Yay! So glad you liked it!
Dude, thanks, I have enjoyed your explanations!
Can WE have 2 schema in the same collection ? Like two type of user user1 and user2 in collection users ???
Absolutely! Google "Polymorphic Pattern" for more information
Instead of referencing can I keep just list of id of another document and just query those specific ids? I mean what will be the performance in that case?
You're an amazing teacher. Thank you.
Thank you! 🥺🥰👑
Such a amazing video .. many thanks .. great work.. cheers :)
Thank you!
Great presentation, loved it. Keep them coming😊
Professional SQL developer here Actually the first part of your video is a little misleading, we do design sql databases with the queries in mind and We denormalize when necessary.
You're right - It's not an antipattern in SQL to denormalize for performance reasons - thank you for clarifying. :D
Thanks for the vid, now I need someone opinion. Am build a e-commerce store to sell clothing but am wonder if I should put all the product variants in the same document or reference it since as each product have more than one variants. Max properly be arround 10 variants and I only need the data to display on the home page as in "2 images,price and name". What's your guys opinion on this? And what about the query if I only need 2 images,name and retail price on the landing page.
Depends on how you will use it. Are you planning on showing all variants on your product detail page? If yes, it might be a good idea to embed it since you don't have very many sub variants. Does that help/make sense? 🥰🥰🥰
I came from your blog post where this video wasn't shown properly. You might want to fix it, cheers.
Thanks for the heads up
Does reverse and cross referencing causing duplication of data
Awesome Explanation. You have nailed it
Thank you so much! Glad you enjoyed it!
Excellent video, very well explained 👍
Thanks for the explanations, very clear .☺
Very Informative . Thank you for this video.
My pleasure!
Nice resource.....could we get a complete mongodb schema playlist
at 40:35 the "timestamp" value has $date and $numberLong, what does that mean? is that variable? or function of mongodb?
15:05 Could someone please say the the term he says here? I keep hearing ass complient.
"ACID compliancy" 🤣
hi I got a question would it be better to keep reference on both document in one to many scenario or would it be better to just keep the reference on the one document that holds many docs ?
Fantastic Joe. Great Learning. When you talk about embedding example regarding product and parts, you favor parts. But you keep referencing to the size limit of the document. Yes its a problem, But more than that there is a bigger problem of consistency. If you embed a part and If you update the part from a screen, you have inconsistency. When you show the product you will show the old values. I am curious as to why you are not mentioning about inconsistency? I am a newbie in nosql. heavily influence by relational theory. Eager to know your thoughts
Thank you very much for the presentation! So if data I am embedding instead of referencing needs to have its own unique ID. I am used to getting these IDs for free in SQL world... is there something Mongo DB had for this or I am on my own with generating and keeping track of these IDs? Thanks!!
By default, MongoDB creates a unique UUID under the _id key on every single document you create. You get them for free with MongoDB too! :)
@@JoeKarlsson I was taking about adding an ID to a subitem inside document (embedded structure) that doesn't get an ID by default. Looks like I can generate this ID in application by doing something like BSON::ObjectId.new and it will be unique to the DB
@@alikkasman9853 Yes! You are exactly right :)
Absolutely amazing explanation 👌
I think it was mentioned that you may not want to embed a document so that we won't fetch extra data that we don't want.
I am not sure why this count as reason to not embed. I mean, MongoDB allows the user to specify exactly which fields to load. so, if I do not want to load the embedded document from DB to the application, I can just add it to the exclusion list.
however, I know that this will save network bandwidth between DB server and app server but not sure how much impact it will have on the DB server itself; since the DB server still needs to load the document from the filesystem and then send it to the app server.
can you record a video on schema concept during get or update API's
Hello , I am new to mongodb. In this video there was one point that if you are embedding too much data in the document,it will increase document size and the overhead to send the document over network. But for retrieving the document, we can use projection and get only particular fields that we need. So we are decreasing the network overhead , right?. Please tell me if I am missing something here.
That's true!
Thanks for the video Joe
Oh my gosh! I am so glad you enjoyed it!
I worked always with mysql so its difficult. Lets say I have this card collection game where a user can have many repeated cards,, I mean I can have the same card multiple times, so I can exchange the card or sell it etc. How can I model my DB to support that functionallity?
Hi Mr.Joe,
Great video and I learnt a lot.
I was wondering whether you’ll be able to clear a doubt I have.
In my project, users and tasks collections. I would like to save who created a task in a “createdBy” field. It’s always gonna be one value and would be selected every time one or multiple tasks are fetched.
According to this video, I should embed essential data such as (userId and name) that my application would display. But I want to “refer” the user data so that fields such as name are fetched correctly. I worry if I embed the name and user updates their name, it could become costly to update in all the embeds.
May I know what your approach would be?
Thanks
you deserve 1 million subscribers
Awesome! Great explanation
you are simply the BEST BEST BEST bru!!
I got chu
how do i handle a situation where we have the different user fields?
for example
candidate and employer
where candidate has email, password, resume, experience, skills and so on
where employer has email, password, company size, company history and so on
do i have to create everything in a single user modal?
or create user plus two separate models for candidate profile and employer profile?
i am company from the relational database side so it is bit confusing.
It depends! Do you need those every time? I would recommend you include them unless you have a good reason to separate them
@@JoeKarlsson so those extra fields are created doesn't harm? what i mean. those extra fields might not relate to the candidate or the employer but will remain there with their account?
@@testingaccount5867 Nope! I woud recommend adding them
@@JoeKarlsson we don't do this Relation. So it's something new for me.
Awesome tutorial. Thank you very much!