bug: Python library conflict: python-pandas-opentrons

[danolson1]

Overview

I just upgraded my robot to version 8.3.0 and ran into a problem with a python protocol that reads an excel file. To troubleshoot this issue, I tried to re-create my OT2 robot python environment on my computer. This does not seem to be possible.

My OT2 robot has the following libraries:

Package               Version
--------------------- ------------
aiodns                1.1.1
aiohttp               3.9.3
aionotify             0.3.1
aiosignal             1.3.1
annotated-types       0.7.0
anyio                 3.7.1
asgiref               3.4.1
asttokens             2.2.1
async-timeout         4.0.3
attrs                 23.2.0
backcall              0.2.0
bleach                3.1.0
cchardet              2.1.4
cffi                  1.15.1
charset-normalizer    3.0.1
click                 8.1.2
comm                  0.1.4
decorator             5.1.1
defusedxml            0.7.1
entrypoints           0.3
et-xmlfile            1.1.0
exceptiongroup        1.2.0
executing             1.2.0
fastapi               0.100.0
fastjsonschema        2.19.1
filetype              1.2.0
frozenlist            1.4.1
greenlet              2.0.2
h11                   0.14.0
httptools             0.6.0
idna                  3.3
importlib-metadata    4.6.3
ipykernel             6.20.2
ipython               8.16.1
ipython_genutils      0.2.0
jedi                  0.18.2
Jinja2                3.1.2
jsonschema            4.17.3
jupyter-client        6.1.12
jupyter-core          4.6.1
MarkupSafe            2.1.1
matplotlib-inline     0.1.6
mistune               0.8.4
multidict             6.0.5
nbconvert             5.4.1
nbformat              5.9.2
nest-asyncio          1.5.8
notebook              6.0.3
numpy                 1.21.6
openpyxl              3.1.2
opentrons             8.3.0
opentrons-hardware    0.0.0.dev0
opentrons-shared-data 8.3.0
otupdate              8.3.0
packaging             23.2
paho-mqtt             1.6.1
pandas                1.0.5
pandocfilters         1.4.2
parso                 0.8.3
pexpect               4.8.0
pickleshare           0.7.5
pip                   22.3.1
prometheus-client     0.6.0
prompt-toolkit        3.0.36
psutil                5.9.4
ptyprocess            0.7.0
pure-eval             0.2.2
pycares               4.3.0
pycparser             2.21
pydantic              2.9.2
pydantic_core         2.23.4
pydantic-settings     2.4.0
Pygments              2.14.0
PyJWT                 2.6.0
pyrsistent            0.20.0
pyserial              3.5
python-dateutil       2.8.2
python-dotenv         1.0.1
python-multipart      0.0.6
pytz                  2022.7
pyzmq                 24.0.1
robot-server          8.3.0
Send2Trash            1.5.0
server-utils          0.0.0.dev0
setuptools            68.2.2
simplegeneric         0.8.1
six                   1.16.0
smbus                 1.1
sniffio               1.3.0
SQLAlchemy            1.4.51
stack-data            0.6.2
starlette             0.27.0
system-server         8.3.0
systemd-python        234
terminado             0.8.1
testpath              0.4.2
tornado               6.2
traitlets             5.8.0
typing_extensions     4.9.0
uvicorn               0.27.0.post1
uvloop                0.18.0
wcwidth               0.2.5
webencodings          0.5.1
websockets            9.1
wsproto               1.2.0
yarl                  1.9.4
zipp                  3.5.0

The python version is:

# python -V
Python 3.10.8

Steps to reproduce

I am trying to recreate this environment on my computer using the following yaml file:

name: ot2_v8
channels:
  - conda-forge
  - defaults
dependencies:
  - pip
  - python >= 3.10, < 3.11
  - numpy = 1.21.6 # opentrons is very specific about the version of numpy they can use
  - pandas == 1.0.5
  - openpyxl
  - jupyterlab
  - seaborn

# some packages need to be installed by pip
  - pip:
    - opentrons == 8.3.0

This results in the following error message:

LibMambaUnsatisfiableError: Encountered problems while solving:
  - package pandas-1.0.5-py36hcc50265_0 requires python_abi 3.6.* *_cp36m, but none of the providers can be installed

Could not solve for environment specs
The following packages are incompatible
├─ pandas 1.0.5  is installable with the potential options
│  ├─ pandas 1.0.5 would require
│  │  └─ python_abi 3.6.* *_cp36m, which can be installed;
│  ├─ pandas 1.0.5 would require
│  │  └─ python_abi 3.7.* *_cp37m, which can be installed;
│  ├─ pandas 1.0.5 would require
│  │  └─ python_abi 3.8.* *_cp38, which can be installed;
│  └─ pandas 1.0.5 conflicts with any installable versions previously reported;
└─ python >=3.10,<3.11  is not installable because there are no viable options
   ├─ python [3.10.0|3.10.1|...|3.10.9] would require
   │  └─ python_abi 3.10.* *_cp310, which conflicts with any installable versions previously reported;
   └─ python [3.10.0|3.10.10|...|3.10.9] conflicts with any installable versions previously reported.

The key libraries that seem to be incompatible are:
python 3.10
pandas 1.0.5
opentrons 8.3.0

Current behavior

It seems like version 8.3.0 of the robot software (which requires python >= 3.10) is not compatible with pandas version 1.0.5 (which requires python <= 3.8).

Expected behavior

The OT2 robot has a python environment that can be re-created on another computer.

It would be convenient if there was an automated way to do this, maybe a yaml file for venv or conda.

Operating system

None

System and robot setup or anything else?

I'm using an OT2 robot. I just upgraded it to app version 8.3.0.

[Mizokuiam]

Hi @danolson1
The issue you're encountering stems from a version incompatibility between pandas==1.0.5 and python>=3.10. The Opentrons robot's environment uses an older version of pandas built for Python 3.8 or lower. You're absolutely correct that simply trying to create an environment with pandas==1.0.5 and python>=3.10 will result in dependency conflicts.

Here's the breakdown and the solution:

Analysis of the Problem

The LibMambaUnsatisfiableError clearly indicates the root cause: pandas 1.0.5 is linked against an older Python ABI (Application Binary Interface). Pre-built conda packages for pandas 1.0.5 are only available for Python 3.6, 3.7, and 3.8. Your attempt to install it with Python 3.10 fails because the ABI is incompatible.

Solution

You have two primary options:

1. Upgrade pandas in your local environment: This is the recommended approach. Unless you have a very specific reason to replicate the exact robot environment (which is generally not recommended for development), upgrade pandas to a version compatible with Python 3.10. This ensures you can leverage the latest features and bug fixes.

   name: ot2_v8
   channels:
     - conda-forge
     - defaults
   dependencies:
     - pip
     - python >=3.10,<3.11  # Keep your desired Python version
     - numpy=1.21.6        # Keep numpy pinned if necessary
     - pandas              # Remove version pinning for pandas
     - openpyxl
     - jupyterlab
     - seaborn
     - pip:
       - opentrons==8.3.0

   Removing the version pinning for pandas (==1.0.5) allows conda to resolve to a compatible version. This will likely be a much more recent version of pandas.

2. Create a separate environment matching the robot (Advanced and less recommended): If you absolutely must replicate the robot's environment, you'll need to create an environment with Python 3.8. This is generally discouraged for development because you lose out on improvements and may run into other dependency issues down the line.

   name: ot2_v8_robot_replica
   channels:
     - conda-forge
     - defaults
   dependencies:
     - pip
     - python=3.8 # Specify Python 3.8
     - numpy=1.21.6
     - pandas=1.0.5 
     - openpyxl # Check if this works with Python 3.8 and pandas 1.0.5; might require version pinning
     - jupyterlab # Install jupyterlab if needed in this replica environment
     - seaborn    # Same as jupyterlab, might need adjustments for 3.8
   
   
   - pip:
     - opentrons==8.3.0 # Check compatibility with python 3.8 - unlikely, see below

   Important Caveat: Opentrons 8.3.0 likely requires Python 3.10. This means that even if you create a Python 3.8 environment with pandas 1.0.5, you won't be able to install Opentrons 8.3.0 into it. If you absolutely must replicate the robot environment for debugging purposes, you might need to downgrade Opentrons as well or use a different debugging strategy (e.g., remote debugging on the robot itself).

Best Practice: Development Environments

The ideal approach for protocol development is to maintain a more modern development environment (with updated pandas) on your computer. Focus on writing protocols that are compatible with a range of Opentrons and pandas versions by avoiding reliance on very specific version features. This makes your protocols more robust and maintainable. Use virtual environments (conda or venv) to isolate your projects and manage dependencies effectively.

[danolson1]

Initially I tried your option 1, but had errors reading Excel files due to slightly different versions of Pandas. The protocol would work when I simulated it on my environment, but then failed when I ran it on the robot.

Then I tried your option 2, but it would not install due to the dependency conflict you identified.

How does version 8.3.0 of the opentrons software use both Python 3.10 and Pandas 1.0.5?

Yes, I am using conda environments, but I would like at least one of those environments to match the robot environment exactly to ensure that the protocol I develop will work on the robot.

I occasionally do troubleshooting by debugging remotely on the robot itself, but this is only possible when the robot is not in use, since I don't want to disrupt a running protocol.