Bioinformatics Project from Scratch - Drug Discovery Part 1 (Data Collection and Pre-Processing)
HTML-код
- Опубликовано: 11 сен 2024
- Do you want to collect your very own novel and original dataset in biology that you can use in your Data Science Project? In this video, I will show you how to download and pre-process biological activity data from the ChEMBL database that you can use to perform Computational Drug Discovery. The dataset is comprised of compounds (molecules) that have been biologically tested for their activity towards target organism/protein of interest. This video represents Part 1 in a multi-part video series on Bioinformatics Project.
🌟 Buy me a coffee: www.buymeacoff...
⭕Code: ✅ github.com/dat...
⭕ Playlist:
Check out our other videos in the following playlists.
✅ Data Science 101: bit.ly/datapro...
✅ Data Science RUclipsr Podcast: bit.ly/datasci...
✅ Data Science Virtual Internship: bit.ly/datapro...
✅ Bioinformatics: bit.ly/dataprof...
✅ Data Science Toolbox: bit.ly/datapro...
✅ Streamlit (Web App in Python): bit.ly/datapro...
✅ Shiny (Web App in R): bit.ly/datapro...
✅ Google Colab Tips and Tricks: bit.ly/datapro...
✅ Pandas Tips and Tricks: bit.ly/datapro...
✅ Python Data Science Project: bit.ly/datapro...
✅ R Data Science Project: bit.ly/datapro...
⭕ Subscribe:
If you're new here, it would mean the world to me if you would consider subscribing to this channel.
✅ Subscribe: www.youtube.co...
⭕ Recommended Tools:
Kite is a FREE AI-powered coding assistant that will help you code faster and smarter. The Kite plugin integrates with all the top editors and IDEs to give you smart completions and documentation while you’re typing. I've been using Kite and I love it!
✅ Check out Kite: www.kite.com/g...
⭕ Recommended Books:
✅ Hands-On Machine Learning with Scikit-Learn : amzn.to/3hTKuTt
✅ Data Science from Scratch : amzn.to/3fO0JiZ
✅ Python Data Science Handbook : amzn.to/37Tvf8n
✅ R for Data Science : amzn.to/2YCPcgW
✅ Artificial Intelligence: The Insights You Need from Harvard Business Review: amzn.to/33jTdcv
✅ AI Superpowers: China, Silicon Valley, and the New World Order: amzn.to/3nghGrd
⭕ Stock photos, graphics and videos used on this channel:
✅ 1.envato.marke...
⭕ Follow us:
✅ Medium: bit.ly/chanin-m...
✅ FaceBook: / dataprofessor
✅ Website: dataprofessor.org/ (Under construction)
✅ Twitter: / thedataprof
✅ Instagram: / data.professor
✅ LinkedIn: / chanin-nantasenamat
✅ GitHub 1: github.com/dat...
✅ GitHub 2: github.com/cha...
⭕ Disclaimer:
Recommended books and tools are affiliate links that gives me a portion of sales at no cost to you, which will contribute to the improvement of this channel's contents.
#dataprofessor #bioinformatics #drugdiscovery #drugdesign #chembl #cheminformatics #bioinformaticsproject #bioinformaticproject #drug #drugs #molecule #molecules #machinelearning #lecture #dataprofessor #bigdata #QSAR #QSPR #machinelearning #datascienceproject #randomforest #decisiontree #svm #neuralnet #neuralnetwork #supportvectormachine #python #learnpython #pythonprogramming #datascience #datamining #bigdata #datascienceworkshop #dataminingworkshop #dataminingtutorial #datasciencetutorial #ai #artificialintelligence #tutorial #dataanalytics #dataanalysis #machinelearningmodel
👉Watch this video next (How to learn data science in 2021) ruclips.net/video/oR670Txwh88/видео.html
Support this Channel 👇👇👇
🌟 Buy me a coffee www.buymeacoffee.com/dataprofessor
🌟 Download Kite for FREE www.kite.com/get-kite/?
👉 Subscribe to this RUclips channel ruclips.net/user/dataprofessor
👉 Join the Newsletter of Data Professor newsletter.dataprofessor.org
👉 Blogs on Medium medium.dataprofessor.org/
Hi Professor. Where can I find the documentation for CHEMBL library?
Dear sir, Indian card does not support automatics payment please set for a prepaid plan. I am getting into trouble paying.
Do you have documentation for this project
Thanks to the discussion with Shweta in this comment section. Back in the days, 7 years ago, we manually compiled the bioactivity data of more than 2000 compounds from hundreds of research articles. The whole process took 6 months, then we spent a few more months manually curating the data, and double checking again and again for consistency. Fast forward to today, we can do the same thing in less than 10 minutes as shown in this video. I am thankful for the generosity of data providers for making these APIs as well as the various libraries such as pandas (imagine handling hundreds of Excel files and manually curating those) and scikit-learn (imagine optimizing learning parameters manually on 50 computers and via a GUI interface of data mining software such as Weka). Coding is indeed a real superpower. If you are thinking of whether to learn coding or not, my recommendation is yes! It will be one of the best decision for your career and hobby 😃
Very true. I thank you on behalf of all your subscribers for all the efforts and getting such super amazing tutorials to all of us.
@@shwetaredkar734 I'm flattered.Thanks for the kind words. Glad the tutorials were helpful.
@@DataProfessor need help sir
Dear Data Professor,
I can not even express how grateful I am for your content and dedication to your subscribers!
I come from a biological background and I am new to the bioinformatic world.
You give me motivation and great advice to continue studying in this incredible field.
Great work,
Greetings from Brazil!
Wow, thank you! Glad the contents are helpful to your journey into bioinformatics 😃
Can't wait for part 2! I know your subscribers have been asking for this series!
Thanks Ken! Inspired by subscribers and by Ken Jee’s Data Science from Scratch Series 😃
@@DataProfessor haha I am flattered! Keep up the great work!
Me too.
Yaay, thx for the video, Prof. Learned sth from my bioinformatics study, but definitely you give me more experience!
As a Bioinformatics MSc student I found this so interesting
Thanks Muhammad for watching and the kind comment! I am currently editing the video of Part 0 (Bioinformatics 101) and Part 2 will be filmed soon.😃
hi everyone
please , can anyone tell me how to get past master thesis in Bioinformatics about Drug Discovery ?
Brilliant sfuff thank you very much for sharing your knowledge and wisdom
I want to show my sincere appreciation for how you made data science so simple for me and interesting
Thanks for the kind words, this means a lot!
What an awesome explanation
I finally maneged to find channel to walk me through step by step.
I sincerely thank you
Happy to hear that, you're welcome!
This video is a treasure that I have found. Probably the first video ever on Chembl data collection. I wish this video was out when my paper was under review last year. Luckily, I could solve the reviewer query on Chembl.
Shweta, Thanks for the kind words! I’m flattered, glad it was helpful! 😃
@@DataProfessor seriously, I faced a lot many issues to solve the comment. May be I was not familiar with Chembl. But surely, this video will help many of them. Thanks for making this.
@@shwetaredkar734 Thanks for sharing. And thanks again for the kind words 😃
Wow!!! Now I can learn DS and Bioinformatics from a Thai Professional. Thanks Prof for helping get out off my SandBox prison mindset.😅😅😅
sir....thank you so much for simply sharing your knowledge.
This is incredible. Thank you so much for sharing your knowledge and experience with us!
Professor, this is an indispensable resource! You are the best!
Happy to help! Thanks for the kind words!
I watch this on May 2021, and sooo excited with this amazing project video, what a great content, thankss Prof!
Glad you enjoyed it!
Perfect lesson, thank you Professor!
Greetings from Italy!
Thanks for watching and glad it is helpful 😊
I am new to your profile. I was intrested to see about drug discovery and especially l am rescently got interested in bioinformatics. Sadly l think l chose the wrong course as I am biomedical science student in my second years and now l can see that l should have gone bioinformatics way! It was very straight forward lesson, proffessor! Thank you so much
Hi, thanks for the comment. It's okay, I was also a biomed student back in the days. You'll get a solid domain knowledge of the biomedical sciences. You can also do bioinformatics in the side.
Yess! Finally! The niche teach professor!
This was a useful, informational video! It was straightforward and very interesting to learn about. You've intrigued me to pursue a bioinformatics program and I plan on starting my own project using the same tools demonstrated, Thanks Data Professor!
Glad it was helpful!
You are so amazing data professor, thank for always sharing your expertise and knowledge
You are very welcome
I would really appreciate it if you explained your lines of code and how they work. Otherwise, I am not learning why I'm typing what I'm typing, or why it's necessary, etc.
I just watched another series by a different guy who thoroughly explained every line of code he wrote in R and I walked away with a much better understanding of why each line of code is being typed or why certain arguments were used.
Thanks for the feedback 😊
This is one only resource condensed, thanks!
Thank you for watching and support!
thank you so much professor you are a gem 💎
Thanks alot sir...please upload part 2, i m eagerly waiting for ur next video for bioinfo projects in ml😍😍😍
Thanks Ashwin, it was fun making this video, I will also release a Part 0 video (introductory video on the biology background) as well 😃
What a great video! This helps so much in getting started building my portfolio
Hello Dr Chanin,
What if the units for IC50 are different? I am trying to create a project but the imported data from the ChEMBL contains different units (uM and nM). How to solve this problem? I went through one of your comment, but could not find the code in the notebook that you mentioned. Thank you for the help!
Hi, you’ll need to convert the units to the same scale by either converting uM (micro-molar or 10^-6) to nM (nano-molar or 10^-9) or vice versa.
Great lecture. Thanks professor. Please we need second part soon 🙏😍
Ali, glad you found the video helpful. I'm also thrilled to make the next part, please stay tuned 😃
@@DataProfessor i will be
ตามมาจากช่อง Data rockie นะครับ อยากสนับสนุนให้อาจารย์เขียนหนังสือด้าน data science for bioinformatics มากเลยครับ จะรอซื้อครับ 👍
hello! can you help me? im having this kind of error when importing the libraries:
ImportError: cannot import name 'hashlib' from 'requests_cache.backends.base' (/usr/local/lib/python3.7/dist-packages/requests_cache/backends/base.py)
do you know why this happening?
this error comes when i execute:
from chembl_webresource_client.new_client import new_client
thanks for the guidance
Hi @VxS replied to your other post
Excellent and Awesome.I will share this tutorials with my friends. Sir, I have a request. Could you please make tutorial on how to collect data from BioLip, KEGG, PDBSUM etc and how to use various packages and tools such as Rcpi, Protr, PyPDI,Biopython, etc to generate descriptors? Just waiting for the second part. Many thanks!!!
Thanks Varsha for the suggestions. Currently in the making is how to use BioPython to do a protein blast. More on the molecular descriptors is also in the pipeline in forthcoming videos of the Bioinformatics Project series. 😃
@@DataProfessorthat's a great news. 👍
I am trying to do follow along, doing this with dopamine d2 receptor protein, but after pre-processing the data, i noticed that there are some duplicate values that have the same molecule_chembl_id, but different standard_value, is this normal?
Awesome Tutorial Must Watch for Beginners.
Glad you think so! Thank you for watching!
thank you professor ! Do more bioinformatics video pls
Sure thing! More to come
This is so good for self learners like me, thanks for creating this kind of bioinformatics content.
Apart from acetylcholinesterase what are other molecules for which I can create similar project ? Tried for human BRCA1 but not a great one to create the exact similar project. Please suggest a few options.
Even I'm interested to learn to build a project for HER 2 /BRCA1 / Bcr-Abl genes
Members in the Cytochrome P450 and Kinase protein family are also worth exploring
Hi, if any of my fellow bioinformaticians is watching this video. Let me know how does it help? I don't mean anything negative. The video seems great. But here you have already built a project and are running it in front of us. How do I build one of my own? Could you please teach us that? Or is this how people generally take inspiration about making projects? I am sorry I ma very new to this.
Great video data professor, waiting for part 2.. :)
Hi sir I am actually a biotechnology student and want to step into bioinformatics and have no prior knowledge about coding so do you thing this playlist is a good start and what else do you think I should do after this to understand more
You're doing such a great job sir please upload more tutorial about AI and drug discovery please upload drug design with GAN and AAE
Thanks for the suggestion!
@@DataProfessor welcome sir
At the importing Library section of the Colab, I still get a messages like "No module named 'chembl_webresource_client". Please look into that.
Have you installed this library before running the code cell.
@@DataProfessor Yeah, that day I realised the mistake is from my end.
Hi Professor, I have face a problem when following your code on Google Colab. While at the first step of copying files to Google Drive, I encounter a “ModuleNotFoundError”. And I found that this is cause by a module 'termios' which is only available on Unix system. I found that you are also using Windows OS but how can you solve the problem faced? Regarding this problem, the googled result only show me to change to unix os.
This video really awesome, thank you so much.
Thank you! This is soooooo helpful to me!!!
Sir every video of u is a new lesson for us but plz if u dont min plz zoom the vid. So that we can see the code clearly plz
Thanks for suggestion, recent videos are zoomed in, sorry for the small font size of earlier videos.
Please sir, how can I solve this error, I have Windows 8.1 and I use python 3:
ERROR: Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
I'm doing my final project for my graduation later, I thought of various ways and I didn't find any way out until I finally found your video, I'm very grateful that you made a video like this, it really helped me. May I ask a few things? how do you combine two chemical molecules so that you get a new molecular structure and then test the bioinformatics? I am very confused about that, please help me
Can I talk about this with you please?
Thank you very much..
Have you done your project by this playlist?
Hi! I have been following your tutorials from past few days. I have a question. Is it necessary to test models on the same dataset they were trained with, or can a different test dataset be used for evaluation?
Eager to see part 2 professor 😁
Wow! super step by step process explanation!
Glad it was helpful! 😃
As a high school student it has gone beyond my head now I need some sleep
are there any prerequisites to be able to do this project? like are we supposed to know some ML beforehand?
Specially thank you for your useful and practical video. It would be much more applicable if you could instruct how to execute this work with R.
Great suggestion! I'll definitely consider this for future videos. In the meantime, please refer to the GitHub and link to the research paper of one recent publication from our research group:
github.com/chaninlab/anti-sickling
pubs.rsc.org/en/content/articlelanding/2018/ra/c7ra12079f
@@DataProfessor Thank you very much for your valuable suggestions. It would be very helpful if you could advise how to download necessary data from several other databases to apply for other drug discover.
that's fantastic, thanks so much professor
Thanks for watching! Glad you liked it!
Hello sir i am not able to run the githubtools to calculating the network proximity of bladder and colorectal cancer so please 🙏 make a video on this
thanks you sir for this amazing tutorial i am msc bioinformatics student can you plz make complete video on another bioinfo project using the approach text mining and data curation.
Thanks for suggestion 😃
Hello, I would like to make an inquiry. When we obtain the dataframe with the data for the biological activity of the selected target, some numeric columns are "object", which prevents statistics from being performed. How can I change the dtype of these columns? Thank you very much!
As a college student, can i earn anything using bioinformarics by anymeans? 🤔
Hai sir thank you so much for this explanation
Hi Prof Chanin,
When I run this code in JupyterNotebook, I get stuck where we save the bioactivity data.
I run the following:
df.to_csv('C:/Users//bioactivity_data.csv')
df = pd.read_csv("bioactivity_data.csv") # to read the csv
df.head(5) # to view the data
while the I am able to access the csv on my desktop , the code doesnt run in the notebook. (still shows In [*]: at df.to_csv...
I would just like to find out if there are any suggestions as to how I should be doing this?
Hi, You could try removing the path so that you have df.to_csv('bioactivity_data.csv') and see if this works
Hi Professor can you recommend some books for self-study on the topic of bioinformatics? thank you.
Hi, there are quite a few books,the O'Reilly book "Bioinformatics Programming Using Python: Practical Programming for Biological Data" seems like a good starter to get you started with python coding bioinformatics examples.
I am having an issue trying to install the chembl resource client.
This the error that i am running into:
""ERROR: Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall. ""
Can anyone help?
EDIT: I uninstalled Anaconda and reinstalled it and its working fine.
Looking forward to part 2.
Arkadeep, thanks for the comment! Part 2 is in the making.
If i want to write a if statement where the id number in row(1) of a CSV file is == id number in my text file print the sequence that means print the information of that particular id
How to write?
Dear Professor,
Is it possible to somehow contact you, I have a small question about one thing not really about the topic but about the course that is available I am now a bit confused should I pay for it or better to not do it, since it most probably will not be much beneficial
Hi you contact me at dataprofessorofficial@gmail.com
It'll be studying computer science at university and I'm also interested in bioinformatics, but I don't have any biology background whatsoever. How should I approach bioinformatics and can I self study biology concepts and start doing bioinformatics projects?
Hi sir I am PhD scholar in Bioinformatics can u plz suggest some research topics
Hi, I made a video on how to select a research topic ruclips.net/video/zRWFR1SSBFc/видео.html
Hello, when I use df = pd.DataFrame.from_dict(res), I get this error: TypeError: Object of type DataFrame is not JSON serializable. What can I do? Thank you!
Can you recheck if you have run all cells in sequential order
@@DataProfessor My mistake! I didn't select the target correctly.
Could you please post a video on extracting various important information say chemical, ADR etc related to the drug using NLP techniques.
Thanks for the suggestion, will definitely consider this for future videos.
Dear Professor, this was an amazing tutorial and lecturer in one, thank you so much, but I have two questions, I am currently trying to create my own independent model and I have a question, the classification of active, inactive or intermediate based on the IC50 value that you used, is this a universally accepted for all molecules when making a QSAR model? What constitutes us using these parameters active 10000?
Hey there I'm getting an attribute error when setting target = new_client.target do you know the reason for this?
Hello!
I am working on a drug target interaction prediction using ML project. I wanted to know if I can use the data for my project in the same way given by you?
creating the df from the bioactivity data dictionary is taking a long time (30+min) with no luck. Any suggestions on speeding this up? i am using jupyter fyi
how to create a data folder in jupyter notebook and copy bioactivity_data csv file...please help me how do this part in jupyter notebook..help me out i got strucked at this part....in this video , how to create and copy explained colab notebooks on google drive
I cannot search for you on colab...nothing comes up.
Notebooks are available on GitHub github.com/dataprofessor so make sure to click on the GitHub tab if opening from Colab.
Hello,
In recent version of Chembl(v29) database there is a target SARS-Cov2 with Chembl Id of CHEMBL4523582; In this dataset there is a feature of Standard type containing inhibition (~98% of data) and IC50(~2% of data) as classes, I am pleased to know the difference between them or can i see those both as a same standard type....
Great video with concise explanation 👍 Is there a similar way to download sequencing data? Please make a video on that topic.
Thank you professor for this tutorial! I am very new to bioinformatics and I would like to assure one thing. For example, when I type on ChEMBL a protein like Phospholipase A2 and then select targets, all of the results displayed are Phospholipase A2 targets right? Secondly, if i choose single protein, this means that the displayed protein is targeted by phospholipase A2. Is this correct?
Thanks for what you do!
A pleasure, thanks for watching!
Sir Subscribed just now!!
Thanks, welcome aboard 😄
why did u use IC50 to work on?why dont u use all the features?
thank you for this amazing introduction. is it possible to use same approach to collect data from article repositories like elsevier, sciencedirect, scopus etc. with certain keywords and endpoints.
Similar concept applies but instead of the chembl API you can look into using the API of these article repositories.
Hi. Is there any prerequisite for fully understanding this video. I have an undergraduate biology background, with very little experience in programming. Will that be enough?
Yes absolutely, I go at the basic level that you can follow step by step in a guided way. I would recommend to also take an introductory python course via Kaggle mini-course to augment your understanding, please see this video also ruclips.net/video/ZtTt822bDNE/видео.html
Sir I tried hard to use these techniques to Autoimmunity and allergenic drug target. But I did not succeed. Please show me oath!
It's an amazing series..
Thanks for the kind words!
@@DataProfessor Can we use any other database instead of chembl???
As an aspiring data scientist, this project has been really helpful in building my portfolio with original data. Do you think building a portfolio of projects based on past academic papers is useful in today's biotech industry ?
Yes, definitely. The existing research community is based on doing cutting edge research that is based on past academic papers. The goal is to find knowledge gaps presented in the current research literature and to make incremental improvements that would lead to publishing the work itself as a research paper. Hope this helps.
Sir , where is the previous part of this video.. please provide me the link 🙏
Dear data professor and channel fans. I am a biologist and new to bioinformatics. I am trying to use your videos and code to run experiments to learn. When I open Google colab and open the data professor notebook in code/python I can't see anything.
Can you please explain to me if I need to install python in my computer to use Google colab? What other basic programs I need to be able to see the notebook and code?
Is the code still available?
If anyone can help me solve this questions to I can replicate this experiment I appreciate it very much.
Thank you.
Try and use more Of pandas’ built in method such as map or apply instead of looping over the data frame
Thanks Idan for the valuable suggestion!
Is this video recommended for someone with knowledge of only biology?
Hey dataprof!
Really excited for this amazing project.
I followed the same steps as your showed while creating the dataset. However, my findings were a little different.
when looking for missing values, I found that
- the dataset has 6 completely empty variables
- 3 features with greater than 50% to 99% missing values
- 3 features with less 50% missing values
So for the empty features, I can straight away drop the features itself. How about the features with missing values ranging from 50% to 99%.
Should I drop these features as well? Since simply applying dropna() is leaving me with an empty dataset.
let me know your thoughts.
Thanks a lot :)
Hi Utkarsh, great question! The data that is obtained in Part 1 is not yet ready for machine learning model building. It is still a raw data that needs to undergo molecular descriptor calculation that is covered in the subsequent Parts. As for missing values, once you have computed the molecular descriptors, yes you can remove those having significantly sparse data (normally we try >80% as the threshold but you are welcome to tune this threshold. Using dropna() is giving you empty data due to the sparseness of the data which leads us to rather use the low variance threshold (inverse of the % missing values as you had mentioned which is actually not missing values but rather they have 0 as the value). Hope this helps. 😃
I get an error 400;invalid request authorisation error
Lots of appreciation for this series and ur work. U are an inspiration for me.
Please guide I'm stucked onto the 4th code where we need to select target after making targets search.
I opened in colab and it's showing red exclamation.
I'm beginner and new to bioinformatics
Hi, what error message is it giving?
Sir Can you please briefly explain why we are choosing IC50?
Hi, IC50 is the bioactivity value that we use as the Y variable in model building.
very interesting. Great job!
Thanks for watching 😃
Interesting tutorial! Please, I have a question: can someone explain to me why opt for "SINGLE PROTEIN" why not "Organism" ?
Can a undergraduate student do this ? I found this really interesting. Thank you for these wonderful videos:)
Yes, absolutely, also make sure to watch the following to high-level overview to get acquainted with the topic.
2-Part Bioinformatics 101 videos as well (links below)
- Part 1 ruclips.net/video/p5iZxIT16KQ/видео.html
- Part 2 ruclips.net/video/ua08NV58Gew/видео.html
Machine Learning for Drug Discovery (Explained in 2 minutes)
ruclips.net/video/xDMzOUUnNzw/видео.html
@@DataProfessor Thanks you so much for clarifying my query. :) It helped a lot.
Sir, is these only for target types in chembl database or also for chembl compound types,tissue types, assay types, which is shown in search bar of chembl database?
Hi. We are searching for a "Target" of interest (a specific protein such as Aromatase) and the associated compounds that have been reported to bind to the target protein will be downloaded as our dataset.
Hi, I have completed Masters in Bioinformatics 12 yrs. before, With no prior practical experience. What would you suggest further for enhancing career and obtaining a job.
Hi, without prior experience may be a disadvantage if going to industry, I would suggest to find a local university and offer to do an internship or work for free to get break the gap period so that you have acquired some experience, perhaps help to complete a project. Afterwards you can start to look at other opportunities.
@@DataProfessor i needed some guidance in this area.Can you provide me your email id.Appreciate your support.
I love your work...
Thanks!
great video! thanks a lot!
Thanks for watching!
please the link of the paper in which u did this approch
Hi, here it is peerj.com/articles/2322/
Hello everyone can anyone please tell where i can find only computational based working article found in Bioinformatics?
Your are awesome man!
Hey thanks!