Temporary solution: install eccodes from PyPI in conda environment

[valeriupredoi]

Description

We get eccodes via iris-grib but our pip check test fails because the conda package does not come with its extra libraries anymore (eccodeslib, fckitlib and eckitlib) so eccodes is practically incomplete since 2.44 (inclusive) as seen by the environment (those libs are internal to the eccodes conda package, but pip thinks otherwise, it thinks they are separate dependencies, as they really are on PyPI).

The only viable solution for us is to install eccodes from PyPI inside the conda env, so that those extra deps are pulled from PyPI; then on, eccodes does get installed from conda-forge via iris-grib.

Pinning eccodes and forcing it as a conda-forge dependency does not work since the CircleCI env fails to solve. Having it as a PyPI dep in pyproject.toml does not help since we use --no-deps flag. Bit of a roadblock, proper.

Closes #2856

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🧪 The new functionality is relevant and scientifically sound
• 🛠 This pull request has a descriptive title and labels
• 🛠 All checks below this pull request were successful

---

To help with the number pull requests:

• 🙏 We kindly ask you to review two other open pull requests in this repository

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.46%. Comparing base (3338f54) to head (b0e349b).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2857   +/-   ##
=======================================
  Coverage   95.46%   95.46%           
=======================================
  Files         260      260           
  Lines       15519    15519           
=======================================
  Hits        14815    14815           
  Misses        704      704           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[bouweandela]

How about we just put a pin on eccodes to get our tests running again? Or we add the missing dependencies to our environment.yml if you're keen on using the latest version?

[valeriupredoi]

How about we just put a pin on eccodes to get our tests running again? Or we add the missing dependencies to our environment.yml if you're keen on using the latest version?

gah, no way I am starting to add all those deps, especially since we don't even depend directly on eccodes - I'll pin it to <2.44 and will ping @pp-mo see if he wants to pin it in iris-grib, so we can remove the pin ourselves

[valeriupredoi]

ah brilliant! Pinning eccodes<2.44 makes Circle lose its beans and spend forefer solving the environment - what, it was the version until just a few days ago 😆 Just gonna pin it to 2.43

[valeriupredoi]

@bouweandela this is finally working and is ready - not an elegant way of handling a problem that's not even ours 😁

[bouweandela]

I think the issue may be that eccodeslib is called eccodes on conda-forge, see conda-forge/python-eccodes-feedstock#107.

[valeriupredoi]

I think the issue may be that eccodeslib is called eccodes on conda-forge, see conda-forge/python-eccodes-feedstock#107.

it's a mess! I don't think eccodeslib is eccodes on c-f, look at the eccodes feedstock and what they pull in to build the package - it's eccodes https://github.com/conda-forge/eccodes-feedstock/blob/main/recipe/meta.yaml ; python-eccodes is just the python build of eccodes and eccodes is the whole thing - if you look at how eccodes is built on c-f you'll see it's building the whole thing, and it find libeccodes.so from inside, it's not needing to get anything else

[valeriupredoi]

no, it appears you are right - https://github.com/ecmwf/eccodes-python says it depends on eccodeslib from PyPI or on eccodes from c-f, depending how you install it; but eccodes from c-f for 2.44 onwards also depends on eccodeslib as pip sees it, so we're back to square A; they need to fix this thing, afarid we can't do much apart from this here, moreover we are going through iris-grib, that can't even support eccodes=2.44 coz their tests are failing SciTools/iris-grib#708

[valeriupredoi]

BTW you added the pip check in the wrong package feedstock (eh won't hurt tho), we need that in eccodes feedstock 🤣

[bouweandela]

Here is a mapping, hopefully that clarifies things:

Code	PyPI package	conda-forge package
https://github.com/ecmwf/eccodes	eccodeslib	eccodes
https://github.com/ecmwf/eccodes-python	eccodes	python-eccodes

BTW you added the pip check in the wrong package feedstock (eh won't hurt tho), we need that in eccodes feedstock 🤣

The eccodes conda-forge package does not seem to have any Python dependencies, so adding a pip check there wouldn't have anything to check.

[bouweandela]

Also note this message (source), which probably explains why we have started seeing this issue only now:

From version 2.43.0, the ecCodes Python bindings on PyPi will depend on the PyPi package ‘eccodeslib’ on Linux and MacOS. This package provides the binary ecCodes library. On Windows, the ecCodes Python bindings will continue to directly provide the ecCodes binary library without a dependency on eccodeslib. See below for details.

[valeriupredoi]

Yes, that's the message I just saw too. What do you propose we do for us though, this is the thing that bothers me - I reckon sticking to PyPI as I put in yesterday is the safest approach for now

[bouweandela]

What do you propose we do for us though

Wait and see if the nightly tests pick up the new build of python-eccodes

[valeriupredoi]

What do you propose we do for us though

Wait and see if the nightly tests pick up the new build of python-eccodes

it works well now! I tested with a local env, and turned on GHA here 🥳
How did you fix this with only adding a pip check in python-eccodes? Was it since it rebuilt the package to find the new dependencies? Either way - hats off, bud 🍻

[bouweandela]

How did you fix this with only adding a pip check in python-eccodes?

I also renamed the dependency of the conda package from eccodeslib to eccodes in https://github.com/conda-forge/python-eccodes-feedstock/pull/107/files#diff-f0f832c013acbec5b16dd2d7e813bbfcfd797a5a64640c6ca5191cd9b9c47bcd

[valeriupredoi]

aah! and done nicely with a repopatch too - missed that, the key ingredient - well, I had not even known about python-eccodes until I saw your PR so another +1 from me 🍻