How to resolve dependency issues

When we install a package say pandas we also install various dependent packages which are not version bound.

Example: When you install pandas you implicitly install numpy too.

Recently, at work we have an application whose pandas requirement was locked at pandas==0.25.3 so whenever we build our docker image and downloaded the requirements via:

pip install -r /test-requirements.txt

We downloaded pandas==0.25.3 but as the numpy distribution was not locked pip resolved it to the latest version numpy==1.20.0 which made it incompatible with pandas dataframe such that.

import pandas as pd
df= pd.DataFrame(columns=['col_a','col_b']

resolved to the following error.

Browsing STACKOVERFLOW for the error led me to, oh I just need to update the the pandas package to the latest pandas==1.2.1 YAY!!!

I happily ran a subset of the tasks to check and put the code on staging.

I found that all my tasks had failed:

Things I had done wrong:
1) Did a major version jump for the pandas package from 0.25.3 to 1.2.1

2) The repo did not have 100% coverage so I had missed to perform some important checks on my local.

3) I had jumped to the conclusion that updating the version was the only way to resolve this issue.

If I was to encounter this issue again...

1. I would not hastily go and update the package, I would look more deeply into the issue, I would look make note of all the packages that were implicitly downloaded by pandas (the package of concern, in my case) and were updated to a different version. You can do this by running
   pip freeze in the old setup, and the new setup. (hopefully, you have a environment in staging/production which still has the stable version)

pip freeze >> old_requirements.txt (In a stable environment- likely, staging, production)
pip freeze >> new_requirements.txt (current setup causing errors)

2. Find all the packages that have been updated

diff --side-by-side --suppress-common-lines --strip-trailing-cr  --color=auto old_requirements.txt new_requirements.txt

Sample Output:

pandas==0.24.2           pandas==1.2.1
numpy==1.13.3            numpy==1.20.0

Try to decipher which package is the culprit in this case it was numpy==1.20.0 , hence lock that package to a stable version in your requirements.txt