Getting PyHC core projects on conda-forge

[dstansby]

👋 all, I just caught up with some of the PyHC meeting stuff around getting packages on conda-forge (for the record I'm very pro adding everything to conda-forge as well as PyPi!), and the talk about dependency issues caught my interest. I thought I'd do a quick audit of which of the core packages are already on conda-forge, and which aren't. For the ones that aren't, I manually checked their dependencies, and if they were on conda-forge.

Results below. It seems like almost all dependencies are already on conda-forge, so it shouldn't be too hard to add HAPI, kamodo, and pysat to conda-forge. For pySPEDAS it looks a bit more complicated, but definitely do-able.

Anyway, thought I'd put this out there to add a bit of data to the discussion. Pinging maintainers of the libraries that aren't on conda-forge yet in case this is useful: @rweigel @rebeccaringuette @rstoneback @supervised @drsteve

• HAPI Client ❌
  • pandas ✅
  • numpy ✅
  • kiwisolver ✅
• kamodo ❌
  • decorator ✅
  • numpy ✅
  • scipy ✅
  • sympy ✅
  • pandas ✅
  • plotly ✅
  • pytest ✅
  • hydra-core ✅
  • Flask ✅
  • flask-cors ✅
  • flask-restful ✅
  • antlr4-python3-runtime ✅
  • python-forge ✅
  • requests ✅
  • incremental ✅
  • pycapnp ✅
  • pyOpenSSL ✅
• PlasmaPy ✅
• pysat ❌
  • dask ✅
  • netCDF4 ✅
  • numpy ✅
  • pandas ✅
  • portalocker ✅
  • pytest ✅
  • scipy ✅
  • toolz ✅
  • xarray ✅
• pySPEDAS ❌
  • numpy ✅
  • requests ✅
  • cdflib ✅
  • pytplot-mpl-temp ❌
  • cdasws ❌
  • netCDF4 ✅
  • pywavelets ✅
  • astropy ✅
  • geopack ❌
  • hapiclient ❌
  • viresclient ✅
• SpacePy ❌
  • numpy ✅
  • python-dateutil ✅
  • scipy ✅
  • matplotlib ✅
  • h5py ✅
• SunPy ✅

[rweigel]

Based on https://levelup.gitconnected.com/publishing-your-python-package-on-conda-and-conda-forge-309a405740cf, I would expect that learning about the process and creating scripts to do this would take about 10-20 hours.

The far more difficult task is testing. At present, the HAPI tests are

1. Locally using ~5 versions of Python in an anaconda3 environment on a single OS.
2. After commit, a Travis CI script uses tox to test the same 5 versions in a python virtual environment on the default Travis CI versions of OS-X, Windows, and Linux using the repository version of the code base. Ideally a package would be automatically deployed to the PyPi testing repository on success and then a new process would attempt to run the tests on a clean install using the packages from PyPi, and upon success, a final version would be uploaded to PyPi. Instead, we do 3.
3. Create a Python virtual environments and run tests locally (on a single OS) after installing packages from PyPi.

I've had CS graduate students attempt to modify the Travis CI tests to use tox-conda and also automatically do the releases as described in 2. They did not succeed. The primary issue is debugging Travis CI scripts. First, one needs to strip down the existing script so that the process finishes in a reasonable amount of time. Then, one waits 5-10 minutes to see if things worked. There are always subtle issues across each OS that need to be worked out. One also has to be careful with testing because Travis only provides a certain amount of time per month for non-paid accounts.

(I've been using Travis since before Github had an equivalent and have also thought about migration.)

I would like to have a conda-forge package, but I decided to not spend much more time on improving our test and release process given my experience with past attempts. If someone donated their time to work out all of the issues I described above, I would still be reluctant to maintain a conda-forge package unless I was confident someone could help 1-2 years later if there are issues.

[dstansby]

After commit, a Travis CI script uses tox to test the same 5 versions in a python virtual environment on the default Travis CI versions of OS-X, Windows, and Linux using the repository version of the code base. Ideally a package would be automatically deployed to the PyPi testing repository on success and then a new process would attempt to run the tests on a clean install using the packages from PyPi, and upon success, a final version would be uploaded to PyPi. Instead, we do 3.

I'm not sure I follow here. tox installs the package to be tested into a fresh and isolated virtual environment. So if you're using tox to do testing (which it looks like hapiclient is) then there isn't a need to create a package, upload it to a test site, download it again, and run the tests on that because you're already testing a freshly created package. I agree that this all seems very complicated, but as far as I can see it's not needed.

Based on https://levelup.gitconnected.com/publishing-your-python-package-on-conda-and-conda-forge-309a405740cf, I would expect that learning about the process and creating scripts to do this would take about 10-20 hours.

The first section in that article is redundant if you follow the second section ("Adding your conda-forge recipe"), which is the standard way to add packages to conda-forge. That guide also doesn't mention grayskull, which is tool built to make generating a conda-forge recipe super easy for a package already on PyPI. More info is in the officical docs on adding packages to conda-forge.

Definitely appreciate that adding packages to conda-forge takes some effort - if there's anything I can help with I'd be happy to donate a bit of time to solve any issues.

[sapols]

Reopening. I appreciate the effort to keep our GitHub issues tidy! But this is still on our wishlist (just very back burnered).

[rstoneback]

pysat could almost certainly be added to conda but we don't have any funding to do so.