Creating and maintaining a conda-forge package
HTML-код
- Опубликовано: 25 авг 2024
- Conda is a powerful package manager with which you can create virtual environments and install basically any type of software on a user level. If you are in science or data science, chances are high you are using conda to manage your packages. But how can you create your own packages so that they can be installed with conda?
Community contributed conda packages tend be scattered across multiple "channels", which is not convenient for the user, and a pain to update if you have the package available elsewhere, like on Pypi. Therefore, contributing your package to a well-maintained channel like conda-forge makes the most sense.
Making these kinds of packages is not entirely trivial, which is why I decided to create this video. Production value is low and probably you can watch it sped up by a factor of 2, but I hope it is useful regardless.
In this video I discuss:
* the differences between conda and pip (01:22)
* how to creating a conda-package locally with conda-build to illustrate the build process and configuration files (09:54)
* the steps to creating a conda-forge package (20:45)
* the important parts to maintaining your conda-forge package (33:22)
A quick summary is provided at at (38:14).
Featuring: my macbook pro jet engine
LINKS:
- Conda-build documentation: docs.conda.io/...
- Conda-forge knowledge base (documentation): (conda-forge.or...
- Conda-forge staged recipes repository: github.com/con...
- Conda-forge feedstocks: github.com/con...
- Conda-forge gitter: gitter.im/cond...
Thanks, Nick, very useful
Thanks for creating this video. This really got me started with building packages.
Hi Nick, thanks for this tutorial, it's very helpful!
keep it up dude! Really nice work
Good job. A mention about using the ‘build:skip # [win]’ or similar to only build for linux would be helpful (you mentioned why it would be helpful, but not how to do it).
Best to have a look at this example github.com/conda-forge/exitwavereconstruction-feedstock/blob/main/recipe/meta.yaml and check out the docs conda-forge.org/docs/maintainer/adding_pkgs.html#build
I suggest using "conda install conda-forge::package" over "conda install -c conda-forge package". These mean different things:
1. Install this one package from conda-forge -- install dependencies and other packages from my channel settings.
2. Install this package and look for ALL packages on conda-forge first. This makes conda-forge the highest priority channel, so if you use "defaults" you will see many packages getting replaced by the same version - and again if you run another install command on the same environment without the "-c" (the same packages will get reinstalled from defaults).
In general, the packages on conda forge aim to be interoperable, so ideally all the packages in your environment should be from conda-forge. There is no guarantee packages will work together when they come from different channels, for example C and Fortran packages if they were compiled using different compilers. Conda-forge standardizes on these details and ensures that dependencies come from conda-forge during build.
@@nickcorn93 If all of the packages are coming from conda-forge, then there is no need to specify -c, --channel. If one is using -c, then they are most likely using defaults and cherry picking a few packages from conda-forge. In any case, those two invocations look very similar but act differently when conda is using strict channel priority (the default for 99% of people).
I agree that it's better to be all-in (or all out) when using conda-forge; the packages are tested very well and built using the same build-chains and configurations. It's when mixing channels that one can run into inscrutable problems. The fact that defaults and conda-forge work so well together most of the time makes the tiny inconsistencies more surprising for most users.
@@bradleykreider3358 I'm going to make a pure Python package to distribute in my own channel, however, the package needs Openbabel which is in conda-forge. Is there any way to make the meta.yaml (or any other file) to make it possible to search and install Openbabel from conda-forge upon installation?
One of the options I used is adding the "-c conda-forge" part in addition to the conda install command displayed in my conda package page. However, are there any methods to fix this without using that kind of additional part?
I'm going to make a pure Python package to distribute in my own channel, however, the package needs Openbabel which is in conda-forge. Is there any way to make the meta.yaml (or any other file) to make it possible to search and install Openbabel from conda-forge upon installation?
One of the options I used is adding the "-c conda-forge" part in addition to the conda install command displayed in my conda package page. However, are there any methods to fix this without using that kind of additional part?
i have a question im noob on this why always when can run an .py proyect y have to install some packages with conda install over again ?
I'm not sure this video is the right place for this question ;)
Thanks much Nick. I have already installed anaconda and some other libraries I need. Now I've being tried to install these 2 additional libraries: cartopy and cmocean, but unfortunately could not, using conda install in the anaconda prompt and navigator. Here is the error I get, "The current user does not have write permissions to the target environment." Is there a way to fix such installation issue? Or should I clear my anaconda environment and install conda forge? Pls any advice is highly appreciated!
Hi Lee, thanks for watching, hope it was helpful. Most likely you are using a windows OS and when you installed anaconda you chose "system wide" installation which means the environments get installed in some system folders. You can only install to those locations as an administrator, so you should start you anaconda prompt by right clicking and clicking "run as administrator". Then running "conda install -c conda-forge cartopy cmocean" should work if the environment can be resolved. If not, you must create a new virtual environment with "conda create -n name_of_virtual_environment". In any case it is good practice to install packages in a virtual environment, not in the base environment.
@@nickcorn93 thanks much Nick for this. Will try these advices. You are correct, I am using windows. You r doing great job 👏
Could you make a manual for each step, please? Thanks
Hi, thanks for watching. What do you mean for each step? This video is supposed to be an overview of the steps. The precise details of the steps will depend on the package you try to build, and the documentation will be much more helpful in this case.
@@nickcorn93 for example I'm trying to build my own python package, according to your video you have C++ library. But I did as described in the documentation and still receiving kind of exceptions. For instance: subprocess.CalledProcessError: Command '['cmd.exe', '/d', '/c', 'conda_build.bat']' returned non-zero exit status 1.
@@maniakacab when trying conda-build locally or conda-forge? Building for windows can be a huge pain, even for python packages. Problems usually result from your dependencies, which may have more than just python code inside. You might not have the right build environment installed. In any case if it is just a pure python package you made and you want to distribute it, you might just create a pip package instead. This is much easier.
@@nickcorn93 yep, I built it locally, everything is fine, at the same time I want to upload my package to conda forge. For now it compiled only for windows. There is right way to deploy to conda forge?
@@maniakacab you don't "upload" a package to conda-forge, you create a recipe. Did you watch the video? I explain these steps in ruclips.net/video/8s5aj3sjuVE/видео.html. If it does not work for whatever reason you should ask for help on the gitter channel.