How to create kubeflow pipeline from scratch | Live Demo | Machine Learning | Ashutosh Tripathi
HTML-код
- Опубликовано: 17 окт 2024
- How to create kubeflow pipeline from scratch | Live Demo | Machine Learning | Ashutosh Tripathi
End to End Jupyter Notebook Explanation for Kubeflow pipeline building and executing
Topics Covered:
1. Python function needed to train and predict
2. Creating components from python functions
3. Initialise kubeflow pipeline
4. define the pipeline function and put together all the components
5. Mounting volume for component's output storage
6. Compiling pipeline and generating yaml - it can be directly uploaded to kubeflow and create experiments and runs using UI
7. Create run from pipeline function using the code
8. How to disable cache to see the each steps output on second and successive runs
Notebook link:
github.dev/Tri...
Part 2: CSV file passing between kubeflow components: • How to pass csv and da...
If you find this video helpful, don't forget to like share and subscribe. This is how you can support me.
Connect me:
LinkedIn: / ashutoshtripathiai
Instagram: / ashutoshtripathi_ai
Twitter: / ashutosh_ai
Website: ashutoshtripat...
If you want to message me directly, then connect me on LinkedIn and send a DM.
#machinelearning #kubeflow #mlops
Video on Kubeflow pipeline installation on windows:
ruclips.net/video/LSvvIt2m1Jo/видео.html
Thankyou sir iam looking for this topics from very long period
Welcome
EXCELLENT ..YOUR TUTORIAL IS VERY HELPFUL FOR LEARNING ABOUT MLOPS.......BRO.....👌👌🙌🙌🙌🙌🙏🙏🙏🙏🙏
Thank you Brother.
Great job, I can run my first pipeline from this tutorial. Thanks.
Thanks @AshutoshTripathi_AI ! Your contents are very much exciting and with full of knowledge. Can you please provide a video of full kubeflow components locally ?
Your work is really impressive. I have been following your videos and gaining a lot of knowledge. excellent work...keep it up!!!
Thank you Praveen 🙏
walking into the new week with MLOps as a new skill on my set of skills😎
That's really great....
Your way of explaining is really good.😊
Thank you
Thanks a lot Brother. One of the best videos on this concept. May you please do the same steps in GCP?
Will try to create one on GCP.
Excellent tutorial... Please make a video on Kubeflow installation.
Here is the video link on kubeflow installation locally on windows: ruclips.net/video/LSvvIt2m1Jo/видео.html
Thanks a lot for the tutorial (this one turned on my engine). One question. What if we want to create a component not from a function but from an instance of a custom resource? Assume that the instance kind is SparkApplication (with the associated operator already created under some namespace).
If you have something in a spark app, why not just create a spark pipeline?
Im actually asking, I wanna know the scenario in which you use both
Good one, Thanks.. However can you tell me about kfp module. I have installed Kubeflow in my base environment however when I did open notebook and imported kfp it is not recognised , I did tried pip install kfp and kubeflow both on my Jupyter notebook. Please put some torch on this.
Hi Ashutosh thanks for master piece video, can you help me to understand about the storage. I want to use postgreSQL or any relational database how can I interact with relational database ?
Nicely explained E2E flow
Thank you
Great way to get introduced to Kubeflow Pipeline.
Where can I get the source code for the example you have demonstarted. Kindly let us know. I would like to try it on my Kubeflow deployment.
Hi, I have updated the description of the video with the notebook link. use the link to download the kf-pipeline notebook. let me know if you face any difficulty in downloading.
do we need Kubernetes preinstalled?
Hi @Ashutosh. Evrytime i run the pipeline in the Kubeflow. I am getting logs as "This step output is taken from cache." Can you please help me?
You need to disable cache while creating the pipeline.
def some_pipeline():
# task is a target step in a pipeline
task = some_op()
task.execution_options.caching_strategy.max_cache_staleness = "P30D"
You can also refer this document
www.kubeflow.org/docs/components/pipelines/v1/overview/caching/
Hello, I am new to this whole data science concept. So my questions can be silly. Can i setup model registry and Model Tracking UI on JFrog artifactory?
As for i know jfrog is a repository manager where you can store docker images, handle CI CD. But I m not sure if we can use this for model registry. As I know it is not used for model regyand tracking purposes but need to be double checked.
Can you please make a video of how you are managing data ( for image dataset) in the bucket and accessing them in the program and kubeflow, please!
Hi Ashutosh, You are reading csv file from Git. But when i am trying to read from Local directory then it is giving me error no such directory, i am using Xeroflow for this , is there a different way to read from local directory.
Actually while running the pipeline, your local directory is not accessible from inside the pod. Hence just keep the CSV in some online repo and read it.
Hello, in terms of comparison what did you find better to use kubeflow or MLflow. It seemed like kubeflow was hanging and was using extra resources. We haven't done heavy pipeline runs and was curious to know
Kubeflow I used for pipeline creation and mlflow for model registry. Kubeflow provides registry with minio but mlflow seems more user friendly and feature rich.
@@AshutoshTripathi_AI Thank you for sharing. Yes, that was very relevant from the other MLflow video you made. It does model serving with registry very nicely.
Thanks for share!. Where is te doc 'IRIS_Classifier_pipeline1.yaml' for download?
Sir as of now I have created the MLFLOW UI which is logging all the metrics and artifacts are exactly as shown in your previous MLFLOW video and on other hand I have written the separate Kubeflow pipeline code like done in this video and my pipeline is also created successfully. But how come I can Integrate MLflow as a part of Kubeflow as both are separate as of now???
Hello Sir, very nice explanation. I have one doubt cant we use .py files rather than ipynb files? since I have an entire project. with different functionalities based on dataset.
Yes you can use .py file. Even in .ipynb file every chunk can be considered as a separate .py file
@@AshutoshTripathi_AI can we do model versioning in kubeflow if yes then how sir, can you give an idea or any possible solution.
@@datasciencewitharbaaz5221 use MLFlow for that look at my previous answer how to use MLFlow with KubeFlow pipelines.
Hi Ashutosh,
This video was very helpful. I am stuck on one point. Pls help.
What is the process if we want to execute a task, after multiple task is executed. Is there any option in .after method to add more tasks. Any help would be greatly appreciated.
Do u mean to run tasks in a sequential manner?
Yes
@@AshutoshTripathi_AI could you please help one above ask?
@@ShailendraMishra26 hi Shailendra, i replied to your above question. I did not understood what you exactly mean. Do you mean to run your task sequentially mean one after another for example if task two is dependent on output of first task then task 2 should wait for first task to finish? Is this what you are expecting?
@@AshutoshTripathi_AI Yes I want to run sequentially. But my ask is I have 3 tasks, third should be executed once other two is executed. Output of two tasks is required to run the third one. I want to check if there is any way by which I can pass multiple output parameter in after method?
Hi, thanks for the video,
Although I am not able to execute the code because the pipeline is not able to access the data file. I have tried giving absolute path as well but no luck. Can you help me
This code no more supports with latest version 2.4.0 any help?
Nice Ashutosh !
Thank you.
Hi Ashutosh thanks for your session but for me when I started pipline t-vol is showing .This step is in pending state with this message :ContainerCreating
I was trying to replicate your code , but getting this error : AttributeError: module 'kfp.components' has no attribute 'create_component_from_fun' . my kfp version 2.4.0
how to fix this
In this case please refer kubeflow official documentation of version 2.4.0 if they have changed the method name.
Thanks for sharing valuable information , I was looking for Kubeflow tutorial for long time. One thing which I am not getting clear is how to setup dashboard for kubeflow.
Ok. So do you mean the central dashboard for kubeflow where we see all the components of kubeflow like notebook server, volume, experiment, contributors..etc.....?
If yes then for this you need to deploy complete kubeflow on a kubernetes cluster.
It requires a lot of memory that is why I setup only kubeflow pipeline locally which suffices the main work for data scientists.
@@AshutoshTripathi_AI how we can setup locally, i have followed tutorial and created yaml file. Now I am stuck to upload yaml file locally and see pipeline
@@AshutoshTripathi_AI Please help on that to install locally and see pipeline
@@jilanikashif ok. I will create a video on installation soon but till then you can follow below step to install kubeflow pipeline SDK locally:
1. Install docker desktop
2. Install minikube. So just type minikube installation in Google search and open the official site. Then just follow those steps.
3. Start minikube using minikube start command
4. Type in google- kubeflow pipeline installation locally then open the kubeflow page and scroll down. There you will find there are two command which you need to execute and finally the port forwarding.
5. Once you done till this point kubeflow pipeline will be installed locally.
@@AshutoshTripathi_AI Thanks for replying and sharing knowledge, I have followed till Minikube start and its working, however for kubeflow pipeline installation it's not been working. Could you please share that page which shows command to setup in locally and port forwarding.
Hi sir, can we use both mlflow and kuber flow in a project
Kubeflow has its own artifact registry which uses minio for storage.
However if you want to use mlflow with kubeflow then you have to integrate mlflow with kubeflow.
U can use teraform to do this. I have not done this as this mainly need devops knowledge. Please refer kubeflow documentation they have some documents which u can refer.
Good 👍
🙏
Sir for example if any component fail kubeflow pipeline how i can skip and started next componet for example i have 3 componet a, b, c
b is fail i want run c even if b is fail or sucess how to achive this because when b is fail i will not move next component c so how we can do it.
If components are dependent on others then they have to run sequentially else they will run parallel without depending on others.
For sequentially execution you cant skip.
Hii Ashutosh,, I am getting error @while Kubeflow_Pipeline... Its showing like there no file directory path.. with out using S3 buckket u have any Suggestion to read the Dataset Plz,,,,
You can read it from GitHub repository. gs bucket etc
Sir I wants to become an MlOps expert can u plz,, suggest me any crack course like institute
How can I find the Session cookie, could you please share the steps in order to get session cookie. Thanks
In the browser just right click and click ok inspect option. Then click on application tab there at left side and you will see the cookies option. Then expand that and you will see the url. Just click on the url then on the body section you will find the auth_session. Just copy that long string. This is your browser auth session cookie I'd.
I did the same thing but after running my pipeline on server I am getting this error :
702 # Make the request on the httplib connection object.
- -> 703 httplib response = self. make request
704
conn,
705
method.
706
url,
707
timeout=timeout obj,
708
body=body,
709
headers=headers,
710
chunked chunked,
711
713 # If we're going to release the connection in finally:
)
then
714 # the response doesn't need to know about the connection. Otherwise
715 # it will also try to release it and we'll have a double-release
716 # mess
What should be the namespace parameter, the notebook name ??
No. Not the notebook name. By default the namespace is kubeflow. But if you are working in server deployed one then ops team might have created multiple accounts for different users. So you need to check. If u see it in the url then also u can find the namespace parameter.
As a concept kubeflow is multitenant so user accounts are segregated based on nespaces
Sir how to add Jupiter notebook in kubleflow?
Replied in ur other comment.
How we Integrate it with Kserve?
brother not working module 'kfp.components' has no attribute 'create_component_from_func'
Please check the version you are installing. It might be the case in upgraded version they have renamed it or new method came.
@@AshutoshTripathi_AI i am using the 2.0.1 may be some function name would change but there is no latest documentation out there. I am facing the problem any latest documentation is available?? I checked the Kubeflow document but it's not updated
And you are version 1.8.18 I am not able to install this specific version 1.8 using pip
@@lug__aman github.com/kubeflow/pipelines/issues/7794#issuecomment-1164986300
In kfpv2 doc is suggesting to use @component as decorator. Above function is deprecated
@@lug__aman please refer this url.
www.kubeflow.org/docs/components/pipelines/v2/pipelines/pipeline-basics/
@@AshutoshTripathi_AI i followed the document in there are some code already written in docs for the testing it has already created some function and pipelines I copy paste all things from docs for test then I compiled then it's created yaml file I simply upload in kubeflow ui which install in a cluster
But I am getting this error :
Cannot get MLMD object from meta store
Very good beta
Thank you
Can I have your githib repo link please
It is there in the description of the video.
Why it is not creating visualizations for metrics confusion metrics ?
In this video I have not covered the visualization part on kubeflow pipelines. Are you getting any error ?
@@AshutoshTripathi_AI I went through the documentations, but didnt find anything I am not getting any visualizations as it says. "No Visualizations generated, create manually." But automatically it should create righ?
@@datasciencewitharbaaz5221 i am not sure what you are doing to generate visualization. What I am thinking let me check the visualization part in kubeflow pipeline and will let u know how to generate and store.
do u take any class
I used to take it. Stopped for some time due to other work.
this code is outdated for the current version there are manny-many errors in this code you change the code according to the new version
Please refer to the official document for the updates on the newer version.
What is @ds1
Not sure. Have I mentioned it anywhere in the video? Let me know. Thanks
at 12:47
kfp.dsl contains the domain-specific language (DSL) that you can use to define and interact with pipelines and components.
You can read about it here:
www.kubeflow.org/docs/components/pipelines/v1/sdk/sdk-overview/
I am facing the following error. Does anyone know how to solve it?
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
Cell In[5], line 1
----> 1 create_step_prepare_data = kfp.components.create_component_from_func(
2 func=prepare_data,
3 base_image='python:3.7',
4 packages_to_install=['pandas==1.2.4','numpy==1.21.0']
5 )
AttributeError: module 'kfp.components' has no attribute 'create_component_from_func'
Just check which version kubeflow pipeline you are using. In older version it was not there. Try to refer kubeflow document
@@AshutoshTripathi_AI I was using the newer version, and now the error is resolved. Thanks a lot Ashutosh!
@@keerthigavenkatesh3806 good to hear.
you did not clear the data/ path
??
@@AshutoshTripathi_AI path of the data folder
First of all thanks for the video , I have a problem while creating the kfp.client () it prints : Failed to load kube config. MaxRetryError: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /apis/v1beta1/healthz (Caused by NewConnectionError(': Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it'))
Hey did you got any solution I'm also facing the same issue!!!
Hi Ashutosh thanks for your session but for me when I started pipline t-vol is showing .This step is in pending state with this message :ContainerCreating
Sir how to add Jupiter notebook in kubleflow?
For that u need to install complete kubeflow with all components which requires lot of resources. Hence what I suggest you can still install jupyter notebook with anaconda and use it o build pipeline and then connect the kubeflow pipeline as shown in the tutorial.