Getting SSL certificate behind a proxy working with Python requests library from Docker

L. D. Nicolas May
4 min readDec 4, 2020

Context

At work we use a REDCap instance supported by one of our university research institutes. We often write code against the REDCap API to ETL and/or analyze clinical research data for various purposes. For ad hoc jobs, we’re mostly an R shop, and writing R against the API never gives us any trouble.

Problem

As I started building automated pipelines in Python with tools like Apache Airflow, I tried to query the REDCap data with API calls using the Python {requests} library. However, running something like this…

library(requests)r = requests.post(
"https://xxx.yyy.com/api/",
request_dict,
verify=True
)

… resulted in this error:

Traceback (most recent call last):
...
...
...
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='xxx.yyy.com', port=443): Max retries exceeded with url: /api/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1124)')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
...
...
...
requests.exceptions.SSLError: HTTPSConnectionPool(host='xxx.yyy.com', port=443): Max retries exceeded with url: /api/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1124)')))

Searching the web for this error yields many results, but one StackOverflow answer in particular led me toward a solution to my problem: https://stackoverflow.com/a/57466119/3367614

In the above StackOverflow answer, the solution is to copy-paste text from a chain of certificates into the .pem file that the Python {certifi} library uses to verify certificates. But since (1) I’ll likely screw up the copying and pasting and (2) I’m running Python scripts from a Docker containerized instance of Apache Airflow, I’d like to be able to just have the certificate chain as a file in the container volume that I pass as an argument to the requests.post() method. Using a web browser and…

--

--