Thank you Aakash for giving a raw walk-through. Apart from usual documentation stuff I think the ability to google and find answers for the problems are much more important skills in the area of Data Science apart from ofcourse mathematical understanding. This walkthrough actually gave me more confidence in using pandas without worrying about typical syntax pitfalls
best project tutorial in any youtube channel. Everyone make video and cut those portion when they get stuck. On the other hand you just show the real scenario and show us how to find solution using google as well. we want similar video on data science project. Love from Bangladesh.
Do you know why I loved this video? You kept it raw and real, you are clearly portraying how a data analyst thinks and does the project, which I believe is more important than fancy coding...syntax you can get easily but analytical thinking requires the real efforts
@@shivsharma9153 open notebook through jovian run through colab with help of kaggle datset . I have written some line of code in colab .then i have close all tabs . Now how to go to that file where i have write my code.
Sir actually New York data is present in the given data set. We get the output as False if we use: 'New York' in df ['City'] And we get the output as True if we use : 'New York' in df ['City'].unique()
Yes Jeet, you are correct. We found it later but didn't update the video to show that this type of error might happen any time during working on a project. Great work on finding it!
I have seen only half the video. Couldn't stop myself from apprecating the good work. Couldn't have been done any better. Way better than the Paid education platforms
Aakash, you are one of the best teachers I have come across. Coming from a hard-core medical background and pivoting into data analytics I came across your panda's courses while preparing for my foundation in python before starting a master's in the US in analytics this Fall. Hands down you have given beginners like me a lot of handholding with your courses and videos!
@@jovianhq I went through many teachers on youtube and data camp: but truth to be told- most are ludicrously formal in their teaching methods and have a slower theoretical pace. Is there any possibility to connect with Aakash to get certain roadmap tips for a beginner who plans to venture into the US Health Business Analytics Domain?
The current dataset is updated. A high percentage of accidents occur between 3 pm to 6 pm (probably people in a hurry to get to home) Next higest percentage is 6 am to 9 am. Over 1100 cities have reported just one accident (need to investigate On Sundays, the peak occurs between 11 am and 6 pm, unlike weekdays
New York data is there, manually checked the whole data to find the city because the method you showed at 58 minutes mostly doesn't work because of the spaces, some values contain unnecessary spaces and that creates a problem.
What a great Tutorial! The df is obviously updated and I cannot find the 'New York' value in Cities, BUT there are data in cities_by_accident "cities_by_accident['New York']" and is equal to 7068
Good class of Python pandas, but in R exploratory and statistics analysis are way easier compared to Python. Example: data_frame %>% filter(City == "New York") bam!! dataset filtered. Summarize numeric data => data_frame %>% summary() !! bam!! Done!! in a totally functional way.
Both R and Python are great, you can use either one. Python is gaining more traction because it also has great packages for machine learning & deep learning.
This is a great project and I really enjoyed it. After finishing this video yesterday, I am working on other parameters to expand my analysis. I would love to see more projects from Akash. Thank you.
"New York" in df. City returns False But "New York" in df.City.uniqu() returns True. (Which I have no explanation for) And in fact, there are 4220 accident cases inside the dataset which occurred in New York inside the dataset (The dataset could be updated recently.). I don't know if it has been updated. But since the accident records fall in between the year 2016 and 2020, it would seem weird if new rows get added later on.
You are correct, there was "New York" in the dataset before as well. df.City returns a Series where if you search using the "in" operator, it will search for the indexes and not match with the values. Where as df.City.unique() creates a list and "New York" is searched within that list so you were able to find "New York".
Hi, at around 49:17 when you are checking that weather we have 'New York' data or not so in that when we are checking for the existence with, if 'New York' in df.values - it will return True And If 'New York' in df.City - False Also If 'Dublin' in df.City - False ( and for all the other cities ) So, in my preference we need to use the df.values ( it will check the whole dataset - yes might be time taking and requires unwanted computing processing as well ) Please help us to improve this part Thanks
Yes @Aman, you are correct, New York is indeed present in the dataframe. We've purposefully kept the video in it's raw format instead of editing it. This shows that it's very common to get errors like these while working on your project, one have to be very careful before making a conclusion.
@@jovianhq yes sir, thanks for the session learnt a lot from this basically for "how" to do it there is ample of resources available but "what" to do in EDA is hard to find Thanks for that
I did read your exploratory analysis file for crop production analysis… and all the agroup by queries that you had run were wrong.. plz look into it and rectify them
New York is present in the data. I think you have used the wrong function to check whether a value is present, 'New York' in df. City: Using this function will always get a false result if you check for any city. The correct function to check is 'New York' in df['City'].values
sir i try this code and his show True in 'New York' city first i see df.values df.values show my all data value in array form then i write this code 'New York' in df.values this line of code show True as a output.
idk if they have updated the dataset , but i just tried to find whether New City is there or not and if yes then the number of records ....(referring to 56:00 ) used this code - len(df[df['City']=='New York']) and got the answer.. so , New york is there in the dataset and the number of accidents is 7068
Yes, it's absolutely fine, you're not expected to know everything, and even if you know there can be a better way of implementing the same thing. So it's totally fine to google something out.
Hey, have you tried our Machine Learning course? We have covered feature Engineering/Selection and lots of other interesting topic in that course. View the course from here -> zerotogbms.com
You are correct, most probably by winter it was meant starting of winter, cause here in India Winter starts towards end of November and starting of December.
Hi I have one issue when i am trying to run a histplot in seaborn it is show a error as "module 'seaborn' has no attribute 'histplot'" i am confused like what might be the reason i tried updating the whole PIP but of no use. Can you suggest what shall be the possible solution.
Hello Sir, 1.Can you pls tell me how to handle missing observations for the following features FEATURE SUM Precipitation(in) 510527 Wind_Chill(F) 449288 Wind_Speed(mph) 128852 Humidity(%) 45506 Visibility(mi) 44206 Weather_Condition 44001 Temperature(F) 43030 Wind_Direction 41857 Pressure(in) 36270 Weather_Timestamp 30263 Airport_Code 4248 Timezone 2302 Zipcode 935 dtype: int64 I have removed "number" feature as 70% of the data of that column was missing Can we use mean/median/ mode or is there any other technique ? 2.For the univariate analysis wouldn't it be very lengthy and time consuming to study 47 features?
![Chlöe - Shake (Feat. Jeremih) [Official Visualizer]](http://i.ytimg.com/vi/aDs_1ufpfv8/mqdefault.jpg)
It was a good video, I like how he didn't cut out the part when he's stuck at some problem. ⭐⭐⭐⭐⭐
Thank you Aakash for giving a raw walk-through. Apart from usual documentation stuff I think the ability to google and find answers for the problems are much more important skills in the area of Data Science apart from ofcourse mathematical understanding. This walkthrough actually gave me more confidence in using pandas without worrying about typical syntax pitfalls
Glad it was helpful!
best project tutorial in any youtube channel. Everyone make video and cut those portion when they get stuck. On the other hand you just show the real scenario and show us how to find solution using google as well. we want similar video on data science project.
Love from Bangladesh.
Do you know why I loved this video? You kept it raw and real, you are clearly portraying how a data analyst thinks and does the project, which I believe is more important than fancy coding...syntax you can get easily but analytical thinking requires the real efforts
Hi shiv Im from non technical background i was doing the given project but when we close all tabs how to reach out to the same notebook
@@snehaldamkondwar618 Hi, where did you save it locally? Which folder
@@shivsharma9153 need to save??
@@shivsharma9153 open notebook through jovian run through colab with help of kaggle datset . I have written some line of code in colab .then i have close all tabs . Now how to go to that file where i have write my code.
@@snehaldamkondwar618 try to search locally with the name of notebook you may be able to locate it
great work jovian , we need more such types of session. Thanks a lot
Glad you liked it!
55:52 this dataset contains New York data accidents, there are around 10.000 record. I checked it using df[df['City'] == 'New York']
Yes
Thanks! Not sure why I missed it. Maybe I was using a different version of the dataset.
@@AakashNS not sure why but ["New York" in Df.City] always returns false try it with a another city it just doesn't work
@@claudiolb8552
Use:
a. >>> 'NY' in list(df.State) or
b. >>> 'New York' in list(df.City) or
c. >>> cities_by_accident['New York']
yes, new york city is there in the dataset, the state which is missing is 'Alaska'
Great course Aakash - this and everything else you have put here. Thanks for your generosity to teach this the way you have done it. Brilliant!!!
Sir actually New York data is present in the given data set.
We get the output as False if we use:
'New York' in df ['City']
And we get the output as True if we use :
'New York' in df ['City'].unique()
Yes Jeet, you are correct. We found it later but didn't update the video to show that this type of error might happen any time during working on a project. Great work on finding it!
I used - 'New York' in df.values and got True
I have seen only half the video. Couldn't stop myself from apprecating the good work. Couldn't have been done any better. Way better than the Paid education platforms
NY is in the state list. The Missing states are AK(Alaska) and HI(Hawaii). It also considers DC as state
I like the way you google for help ..Appreciate your time ..learnt new things on how to articulate our work ..thanks
Glad it was helpful!
Thank you for teaching us how to problem solve and the whole EDA process!
Happy to help!
Aakash, you are one of the best teachers I have come across.
Coming from a hard-core medical background and pivoting into data analytics I came across your panda's courses while preparing for my foundation in python before starting a master's in the US in analytics this Fall.
Hands down you have given beginners like me a lot of handholding with your courses and videos!
Thanks, I'm glad you found our course helpful! 😊 - Aakash
@@jovianhq I went through many teachers on youtube and data camp: but truth to be told- most are ludicrously formal in their teaching methods and have a slower theoretical pace.
Is there any possibility to connect with Aakash to get certain roadmap tips for a beginner who plans to venture into the US Health Business Analytics Domain?
The current dataset is updated.
A high percentage of accidents occur between 3 pm to 6 pm (probably people in a hurry to get to home)
Next higest percentage is 6 am to 9 am.
Over 1100 cities have reported just one accident (need to investigate
On Sundays, the peak occurs between 11 am and 6 pm, unlike weekdays
Interesting analysis and insights Shreyas!
Great sharing. I've been looking for a full guide like this one for a while. Thank you!
Glad you enjoyed it!
One of the best RUclips channel for data analysis learners❤️❤️
Great session on EDA. We need more such sessions on different datasets.
This is just an example, we hope the viewers will be able to make better EDA projects on different datasets after watching this video.
this EDA is so motivating to me .......we all search in google...i thought why i can't recall all of those pandas function....
Best teacher for Data scince!
Watching the master is how you learn...Thanks a lot for this...
I can't see an EDA explanation clearly in other videos... awesome 🎉
New York data is there, manually checked the whole data to find the city because the method you showed at 58 minutes mostly doesn't work because of the spaces, some values contain unnecessary spaces and that creates a problem.
This was excellent. I hope for more of these in 2023
Definitely!! Stay tuned, more interesting videos coming soon.
@@jovianhq is this the type of project that is sufficient to be included in an analytics portfolio? or does it need to be something more extensive?
This was a superb explanation of how to do EDA. Extremely helpful, Aakash!
This is best video on EDA I have ever watched! Thanks Aakash :)
Thanks for the kind feedback!
What a great Tutorial! The df is obviously updated and I cannot find the 'New York' value in Cities, BUT there are data in cities_by_accident "cities_by_accident['New York']" and is equal to 7068
Good class of Python pandas, but in R exploratory and statistics analysis are way easier compared to Python. Example: data_frame %>% filter(City == "New York") bam!! dataset filtered. Summarize numeric data => data_frame %>% summary() !! bam!! Done!! in a totally functional way.
Both R and Python are great, you can use either one. Python is gaining more traction because it also has great packages for machine learning & deep learning.
Thank you Aakash. Really helpful. Liked the way we committed mistakes and even learnt something new from it.
Glad you liked it
bro you were my senior in intermediate .ascent junior college ,vizag . got a clarity on eda btw
Glad you liked our tutorials!
concept is crystal clear by anuj bhyia.
Thanks!
Perfect video. I was looking for such video. Thank you Sir
This is a great project and I really enjoyed it. After finishing this video yesterday, I am working on other parameters to expand my analysis. I would love to see more projects from Akash. Thank you.
Hi do you know once we close all tabs how to work on it again
loved your freecodecamp course. thank you so much
56:02 The dataset does contain 'New York'
"" cities_by_accident['New York'] "" gives us the output as 10255
yes..but it does not show the values..just showing count...strange!!
Yes, the dataset does contain New York now.
Thank you for sharing this! It was really insightful to see the analysis process from start to finish. It also answered a few questions I had.
Glad it was helpful!
nice video, simple explaination and the best part was it from the scratch. loved it
Thank you Aakash, great work
Very educational video. Please keep posting such videos.
Thank you. This is one of the best video on EDA .
"New York" in df. City returns False
But "New York" in df.City.uniqu() returns True. (Which I have no explanation for)
And in fact, there are 4220 accident cases inside the dataset which occurred in New York inside the dataset (The dataset could be updated recently.). I don't know if it has been updated. But since the accident records fall in between the year 2016 and 2020, it would seem weird if new rows get added later on.
You are correct, there was "New York" in the dataset before as well. df.City returns a Series where if you search using the "in" operator, it will search for the indexes and not match with the values. Where as df.City.unique() creates a list and "New York" is searched within that list so you were able to find "New York".
You are doing great job. Thank you
Thank you!
Thank You...
looking forward to your upcoming videos
A great session indeed
Glad you liked it!
Thank you so much. Definitely a very useful video. ✌🏽
Glad it was helpful!
i liked ur naturality
This was great! Thank you for the video
My colab crashes every time i use to read this dataset. it runs for 2-3 minutes and then it get crashed. anyone can help me on this..
awesome video! big shout out from Brazil
Hey Mauricio👋, thanks!
Thank you very much ! This video was useful and easy to understand.
Among the top 100 cities in number of accidents, which states do they belong to most frequently? How do we solve this question
Thank you so much! It was very educational
This is really helpful thank you Jovian
Glad you liked it!
Hey I am getting an error while installing packages. "You may need to restart the kernel to use updated packages"
@36:30 We can also do, df.describe().shape[1]
@54:40, I got the results for new york. I did cities_by_accident.loc[“New York”]
Yes, the dataset now contains information about New York
Thank You Very Much learn lots of things through this session
Glad to hear that!
finished watching
Wow, thank you for your concise explanations.
Glad it was helpful!
For personal use:- 1:17:19
Hi, at around 49:17 when you are checking that weather we have 'New York' data or not so in that when we are checking for the existence with,
if 'New York' in df.values - it will return True
And
If 'New York' in df.City - False
Also
If 'Dublin' in df.City - False ( and for all the other cities )
So, in my preference we need to use the df.values ( it will check the whole dataset - yes might be time taking and requires unwanted computing processing as well )
Please help us to improve this part
Thanks
Yes @Aman, you are correct, New York is indeed present in the dataframe. We've purposefully kept the video in it's raw format instead of editing it. This shows that it's very common to get errors like these while working on your project, one have to be very careful before making a conclusion.
@@jovianhq yes sir, thanks for the session learnt a lot from this basically for "how" to do it there is ample of resources available but "what" to do in EDA is hard to find
Thanks for that
Really great video!
Any chance I can get a copy of the jupyter file?
Thank you so much Sir !!
In my case when i read the file the jupyter notebook is taking way too long time
What can i do?
On data preparation step my data frame is crashing continuously. What to do now?
time 1:29:38 of the video......there was no data regarding on which days accident occur so how the graph was formed
aakash is knowledeble as sky .
Can't agree more!
- Jovian Team
Big thank you for being Here 👍👍
Glad you liked it!
tysm for this. this tutorial is a blessing
You're so welcome!
Your analysis is wrong. I do and find new york City... kindly reply for further information
I did read your exploratory analysis file for crop production analysis… and all the agroup by queries that you had run were wrong.. plz look into it and rectify them
Thank you so much for all your efforts :)
Thank you for the comment! Glad you like the videos
New York is present in the data. I think you have used the wrong function to check whether a value is present, 'New York' in df. City: Using this function will always get a false result if you check for any city. The correct function to check is 'New York' in df['City'].values
it is asking permissions to use those datasets,what should i do?
sir i try this code and his show True in 'New York' city
first i see df.values
df.values show my all data value in array form
then i write this code
'New York' in df.values
this line of code show True as a output.
idk if they have updated the dataset , but i just tried to find whether New City is there or not and if yes then the number of records ....(referring to 56:00 )
used this code - len(df[df['City']=='New York']) and got the answer..
so , New york is there in the dataset and the number of accidents is 7068
You are correct! New York was indeed present in the dataset, but in the live session it got skipped due to some mistake in code.
You did a fantastic job buddy
Glad you liked it!
Great video 👏 Congrats for this awesome job
Thank you very much!
Why did u choose google collab over kaggle. I mean I would have been very easy and we could have saved 26 min.
You can use either Google Colab or Kaggle notebooks, whichever you mind more convenient!
We're now working on a Kaggle integration that will make it possible to run notebook directly from Kaggle.
At 21:03 ..I couldn't load the data ...what should I do?
Great Video
Shall i do for this as my mini project???
Previously I felt to remember all the pandas methods but you made me confident. Thank you Bhai❤❤
Glad it was helpful! Check our other courses at jovian.ai/learn
Awesome sir
thank you🙂
i have completed zero to pandas course
can you plz create content about sql integration project?
Sure we will definitely consider the topic for our upcoming courses.
at 37:59 how did we get a plot without importing matplotlib ??
Good session
df[df.City == 'New York']
^ shows New York in the dataset
Yes, the dataset seems to have changed to include New York now
After excluding the bing data, wasn't there a need to recreate the graphs and insights done before finding that bing data is faulty ?
Yup, you are correct, we should always do more research before concluding something
@@jovianhq okay 😊, thanks for the reply
My google colab tab is not opening;(
It's a good video but the dataset you uploaded that is updated one . We also want the row messy dataset which u use while handling missing values
Thank You. Unfortunately the dataset was updated from Kaggle, we don't have access to the previous version to the dataset.
Which algorithm is used
Is it Fine to Google Something while working on a project??Like How did u in the Video??
Yes, it's absolutely fine, you're not expected to know everything, and even if you know there can be a better way of implementing the same thing. So it's totally fine to google something out.
Great session
Glad you liked it!
please make vedio of feature engirreing and selection and thanks for this content
Hey, have you tried our Machine Learning course? We have covered feature Engineering/Selection and lots of other interesting topic in that course.
View the course from here -> zerotogbms.com
Please sir build neural network algorithm with manual coding without keras and use study case
You said winter months but in October to December there is autumn in USA
You are correct, most probably by winter it was meant starting of winter, cause here in India Winter starts towards end of November and starting of December.
Hi I have one issue when i am trying to run a histplot in seaborn it is show a error as "module 'seaborn' has no attribute 'histplot'" i am confused like what might be the reason i tried updating the whole PIP but of no use. Can you suggest what shall be the possible solution.
Try updating the seaborn library using the following command `pip install -U seaborn`
Hello Sir,
1.Can you pls tell me how to handle missing observations for the following features
FEATURE SUM
Precipitation(in) 510527
Wind_Chill(F) 449288
Wind_Speed(mph) 128852
Humidity(%) 45506
Visibility(mi) 44206
Weather_Condition 44001
Temperature(F) 43030
Wind_Direction 41857
Pressure(in) 36270
Weather_Timestamp 30263
Airport_Code 4248
Timezone 2302
Zipcode 935
dtype: int64
I have removed "number" feature as 70% of the data of that column was missing
Can we use mean/median/ mode or is there any other technique ?
2.For the univariate analysis wouldn't it be very lengthy and time consuming to study 47 features?
I have seen people filling median values in the empty cells
I am not able to access Temperature column.It is giving the error-'DataFrame' object has no attribute 'Temperature'.
Are you using the correct column name? 'Temperature(F)' is the column name for the Temperature column.
@@jovianhq I have tried both 'Temperature' and 'Temperature(F)' but I am getting the same error.
Can you please post your question in the Jovian Forum (jovian.ai/forum) or Jovian Discord, with relevant code/notebook link.
trying now