Photo by Joshua Hoehne on Unsplash

Marketing Technologies

Using custom CA in Python? Here is the ‘how to’ for k8s implementations

It is a well-known industry standard to secure all the connections a system could have with TLS. It is often even used for the connections between different components of a single system, which perfectly makes sense when it comes to sophisticated systems containing several teams and different responsibility areas.

All the commonly used libraries and tools for HTTP and other connections are having built-in solutions to handle TLS connections, including verifying certificates.

PKI and CA

What is CA?

A quick recap of the CA and how a trust verification process works.

CA stands for Certificate Authority and represents the system that is in the power of trusting to services (websites / APIs / etc). At its core, it implements trust transitivity. Say, you trust the CA. CA in turn trusts a host with the APIs you need. So you can trust the APIs.

Technically host provides a certificate that is signed by CA to prove that it is trusted. The thing is that the public-private key pair of CA is created in a way, that only a private key could sign certificates, but a public key lets check the sign to be sure it is not fake. Assuming the private key is not compromised and accessible by CA only, you can verify whether the host is trusted or not.

The whole system of trust is known as PKI (public key infrastructure) which relies on initially trusted CAs that are in charge of providing trust to all the others (and hopefully checking if trust is provided to truly genuine services). The list of initially trusted CA could vary depending on the manufacturer and / or vendor. There is a default list though. Check the details here.

Sign the certificate of ones you trust | Photo by Cytonn Photography on Unsplash

Who’s signing a CA certificate? The truth is that CA is signing its certificate itself. In other words, PKI assumes that initially trusted CAs are the only ones allowed to sign their own certificates.

Why custom CA?

If you are using in-house tools and services, it is excessive to sign a certificate by public CA for each of your internal services, especially when it is not meant to be available publicly.

On the other hand, you need to check each service implemented internally before allowing access to it from other components. The PKI is perfectly designed for the task, you just assume the ‘Public’ is your organization only.

So you creating your own CA. There are variations on how to implement it, but I’ll focus on 2 particular cases when it requires an extra configuration of your applications:

1. Your custom CA is trusted by one of the core CAs, but your application has no access to the internet.
2. Your custom CA has self-signed certificates.

Implementation in Python

Assumptions:

• The process of adding your custom CA’s certificate should consider release management, so the algorithm should be optimized and applicable to be automated by CI/CD tools.
• K8s built-in CA capabilities are not used, so the CA is a separate server that is reachable from k8s worker nodes.

By changing code

The most widely spread libraries to handle HTTP interactions from my perspective are ‘urllib’ (which is actually a part of the standard Python package) and ‘requests’.

Both libraries could tackle custom certificates with slight code changes.

Here’s a piece of code for ‘urllib’:

import os
import ssl 


TIMEOUT=5

def get_ssl_context():
  if 'SSL_CERT_FILE' in os.environ:
    ssl_path = os.environ.get('SSL_CERT_FILE')
    if ssl_path is not None and ssl_path != '':
      return ssl.create_default_context(cafile=ssl_path)
  return None

with urlopen(request, timeout=TIMEOUT, context=get_ssl_context()) as response:
  ...

The implicit details you should focus on:

• You should use ‘cafile’ parameter, not a ‘capath’. Probably it is my fault, but I couldn’t succeed in using that parameter. It seems like the ‘capath’ is ignored: neither certificates are imported, nor errors are raised. I tried it with and without Path class, still not working.
• Since ‘capath’ is not available, you should have all the certificates in one file. Luckily PEM format (which is available for SSL context) allows just uniting all the certificates subsequently in one file.

Here’s a piece of code for ‘requests’:

import os
import requests


def get_ssl_path():
  if 'SSL_CERT_FILE' in os.environ:
    ssl_path = os.environ.get('SSL_CERT_FILE')
    if ssl_path is not None and ssl_path != '':
      return ssl_path
  return None

res = requests.get('https://custom-service', verify=get_ssl_path())
...

Whether it is connected to the ‘capath’ problem I’ve mentioned in ‘urllib’ or not, the ‘requests’ package is also not allowing several certificates to be defined. So the solution is still the same: unite PEM certificates.

By using an environment variable

It is worth mentioning that both, ‘urllib’ and ‘requests’ libraries, allow you to define a REQUESTS_CA_BUNDLE environment variable which will be used during HTTP requests. You should just put a path to the certification file, the same as for SSL_CERT_FILE in the examples above.

I would strongly recommend considering that opportunity carefully. Here’s what you should keep in mind:

• The environment variable actually replaces default settings. That means it will fail to request even google.com with SSL verification error.
• Package ‘urllib’ is part of a basic Python setup and is used in so many different utilities, so changing CA with the environment variable will affect them all.
• You may face one of the examples of that influence pretty fast: even the pip3 tool uses ‘urllib’. That means in case REQUESTS_CA_BUNDLE is defined you won’t be able to install any package due to the pip3 request failure.

If your application uses a limited number of internal services only and is being deployed as an isolated POD or container, the environment variable would be really helpful. In other cases, it could cause more harm than gain value.

Extra topic on certificates

If you want both, accept your custom CA certificates and the commonly trusted ones, you could define your certificate as a union of custom certificates and the default set of certificates.

A default set of certificates in Python is provided with a ‘certifi’ package which is installed as a dependency with the ‘requests’ package. The certificates are listed in a file:

<Python directory>/site-packages/certifi/cacert.pem

Thus, you only need to append your custom certificate and save it as a separate file:

#for the first time
cat <Python directory>/site-packages/certifi/cacert.pem custom-cert.cer > bundle.pem
cat custom-cert.cer > custom-cert.pem
#for the second and following cases:
cat custom-cert2.cer >> bundle.pem
cat custom-cert2.cer >> custom-cert.pem

Although there are guides and even ready-to-use Python scripts to add custom certificates to the ‘certifi’ default certificate file, I do not recommend doing it. Avoiding making changes to any 3rd party package is a best practice. In case you are working on a production application, you’ll face a package update phase regularly. Each update rewrites all the changes you undertook.

Instead, I recommend keeping a custom-cert.pem file with all custom certificates you add.

Implementation in k8s

The full algorithm is the following:

1. Implementation in the application (whether with code or with the environment variable)
2. Download the certificate in PEM format
3. Create a secret or config map with the certificate
4. Mount the certificate to POD and configure the environment variable

Let’s get through each of the remaining steps.

Download the certificate in PEM format

If you don’t have a certificate from your CA or are just in fear of those severe information security guys, here are the tips on how to get the certificate yourself.

Certificates back then | Photo by Robert Anasch on Unsplash

You need the CA certificate which is used to sign all the services you’re going to use.

To download the certificate you could use both, the browser and openssl, I will focus on the latter though.

Firstly, to grasp the certificate structure let’s take a glance at the certificate chain:

openssl s_client -showcerts -verify 5 -connect custom-service:8443 -servername custom-service  < /dev/null 

In my case service is deployed at ‘custom-service’ host with port 8443. Verify option with value 5 allows looking into the whole chain. We need the first certificate presented in the command:

Image by Dmitry Kirilovskiy

You can see that only 2 layers are presented. Depth = 1 is the last in a certificate chain — a custom CA certificate. Depth = 0 is a certificate issued for the particular service ‘custom-service’.

Afterward, all the certificates are printed. The ‘BEGIN CERTIFICATE’ and ‘END CERTIFICATE’ phrases define the start and finish of a certificate. Unfortunately, openssl couldn’t work with multiple certificates out-of-the-box. It could print only the depth=0 certificate.

Thus, let’s save all the certificates with awk (the one that had been found on the Internet):

openssl s_client -showcerts -verify 5 -connect custom-service:8443 -servername custom-service  < /dev/null | awk '/BEGIN CERTIFICATE/,/END CERTIFICATE/{ if(/BEGIN CERTIFICATE/){a++}; out="cert"a".pem"; print >out}'

As an output, we have cert1.pem and cert2.pem files. For our purpose cert1.pem is enough.

There is a single custom CA within the organization. In case several ones are used, unite all custom CAs’ certificates into a single file:

cat cert1.cer cert2.cer > bundle.pem

The extension of a certificate file could differ and doesn’t really matter for our purposes, make sure all certificates are in a PEM format though. The openssl utility will help in case transformation is needed.

Create a k8s object to store the certificate

Now we need to deliver the certificate to our POD.

Since we don’t use k8s built-in CA capabilities I believe the best option is to mount Secret or Config Map, but the Persistent Volume + Persistent Volume Claim will also work out.

Let’s see how to make it with Secret:

kubectl create secret generic custom-cert --from-file bundle.pem

Mount the certificate to POD and configure the environment variable

After the certificate is available in k8s we could mount it to a POD. Along with mounting the Secret, you should also configure the path to the certificate with the environment variable.

As mentioned above, for isolated PODs with limited HTTP destinations using the environment variable perfectly makes sense. So I’ll assign REQUESTS_CA_BUNDLE environment variable the path to the file which is mounted from secret.

So the following is to be added in a POD definition (YAML):

...
spec:
  contatiners:
  - env:
    - name: REQUESTS_CA_BUNDLE 
      value: /opt/cert/bundle.pem
    volumeMounts:
    - mountPath: /opt/cert
      name: custom-cert
      subPath: cert
      readOnly: true
    ...
  volumes:
  - name: custom-cert
    secret:
      secretName: custom-cert
      items:
      - key: bundle.pem
        path: cert/bundle.pem
...

The same is applicable in case ReplicaSet, Deployment, or DaemonSet is used.

Level Up Coding

Thanks for being a part of our community! Before you go:

• 👏 Clap for the story and follow the author 👉
• 📰 View more content in the Level Up Coding publication
• 💰 Free coding interview course ⇒ View Course
• 🔔 Follow us: Twitter | LinkedIn | Newsletter

🚀👉 Join the Level Up talent collective and find an amazing job