CI/CD practices in crypto trading (part 2)

Photo by Pierre Borthiry on Unsplash

In part1 of this journey, we designed the initial stage of the strategy verification pipeline that focused on internal behavioral checks and overall vacuum strategy profitability measurement.

These include:

• strategy class definition matches code quality rules defined by the development team (linting)
• strategy behavioral characteristics match predefined use cases (unit testing)
• strategy execution on historical data match defined profitability threshold

Continuous integration pipeline

But just to remind — why do we do all this, if we can just begin trading and see strategy profitability? Exactly because of the opposite reason — avoid trading using inefficient algorithms. Inefficient here means — an algorithm that didn’t pass check from stakeholders and subject matter experts and shows the results below the defined thresholds.

Once the feature proposal is accepted, the next stage is to promote it to a prod-like environment using a release train for a more robust verification. It is needed to ensure that “vacuum” verification meets the challenges of the real world.

Release train

The necessity of release train is to automate code promotion from multiple distributed teams to the defined environment with minimal development and cross-team agreement costs. In our case “code” can be read as “strategy”.

Also, topic sensitivity requires a human error-free promotion process avoiding sensitive data exposition and misconfiguration.

We need at least 3 environments for a new strategy to move through to be fully adopted. “Adopted” here means the strategy design that we can trust for the long run.

These environments are:

• Test: strategy is running in “dry-run” mode with no actual portfolio connected
• Sandbox: strategy is running in “live” mode with significantly reduced portfolio weight (smaller stake amount and smaller open trades limit)
• Prod: strategy is running in “live” mode with a required portfolio weight in a more stable and highly secured environment (increased resources and better cloud availability characteristics, private virtual network with limited role-based access, and zero-trust architecture in mind).

Continuous delivery pipeline

Obviously, our final goal is to have a strategy to be promoted to PROD environment for long-running execution. This can happen automatically (through monitoring actual execution during a specified timeframe) or manually.

Each environment monitors strategy effectiveness and uses this statistic for making decisions about its destiny — be promoted further or be rolled back.

Promote to test environment

Once a new strategy proposal matches all quality requirements — created pull request is available to be merged in the main branch and the “merge” button is activated.

Let’s do a merge and this will trigger the start of the next stage in this CD pipeline — deploy to test environment.

Accepting the sensitivity of this topic and the dramatic consequences of security requirements violations (possible money loss) — we would avoid using the public cloud infrastructure to bootstrap the test platform. Let’s design a solution of a zero-trust network for digital assets trading in one of the next posts, but keep it fully isolated from external access for now.

Fortunately, github actions can help with this by providing support for internal platforms to be used as executable actors. We can register our own runner in our private infrastructure that will receive execution requests once an appropriate repository event will be triggered (in our case merge code to the main production branch).

custom runner

I will use my RaspberryPi device with docker bootstrapped on it. Additionally need to be installed job runner following detailed instructions.

promote a strategy to the raspberry and run

But how to differentiate — what particular strategy configuration will be used for a particular running environment. The last thing we want is to hardcode environment-specific config and store it in the repository — this is at least insecure.

let’s follow the 12-factor methodology and use the separate configuration per environment with a single config-free codebase deployed for all. Github can help to fulfill this requirement by using environments support for public (or paid private) repositories.

After creating a test environment, we can declare environment-specific config options that will be accessed on the custom runner through environment variables. We need at least this bare minimum of config options:

• dry-run — running mode, TRUE indicates that strategy is connected to the virtualized portfolio (no actual user account is used)
• exchange-access-key — user portfolio access key
• exchange-access-secret — user portfolio access secret
• exchange-name — self explanatory

Configure environment

Promote to the sandbox environment

So far so good. We have a strategy deployed to a test environment that has been running in dry-run mode to collect its own profitability metrics. Another “good” — all diffs between environments are in their configuration but not in environment-specific strategy design.

We can now inspect running statistics to make a decision to promote it further in release train or rollback to review and improve.

As a sandbox environment is already a place where you can lose your money because it requires some portfolio to be connected, the preferred way is to avoid automatic promotion, but put responsibility for this decision to the product owner. Anytime later promotion procedure can be fully automated.

So what is a good event to start promotion? I would choose “release tag created” as an event to react to. It has next benefits:

• the new release tag indicates that the solution passed all requirements checks
• it can be secured for particular roles only (e.g. only users in “ProjectAdmin”) to put the responsibility to a specific group of people only.
• as it requires a manual act, so it indicates some state of trust for delivered code increment based on other observations

Creating release tag

Release creation event will trigger a promotion pipeline that will use current artifacts as a source for strategy runner on the sandbox environment.

Release to sandbox

We are using the same runner, the only another configuration applied: live run with valid user credentials to access exchange API.

Voila, strategies are started in a while in production-like environment and signaled about their health status in telegram.

Toolset:

• Freqtrade in docker for strategy execution
• https://github.com/features/actions to automate the delivery
• https://github.com for collaborative development model
• RaspberryPI with docker engine installed
• Telegram bot to communicate with strategies runtime

Step5: observability (monitoring and alerting)

We can already relax and observe strategy execution. It requires some time to collect historical data to prove or object its efficiency.

But what about collecting this data automatically as data points and then aggregating them to visualize strategy efficiency in some monitoring tool?

Its effectiveness can be measured as a function of time. If we can define it (e.g. balance over time, percentage of wins over time, etc.), then we can collect data points, calculate results, and visualize its profitability as a function over this data.

Also, defined business metrics can help to make decisions to move strategy backward or forward in environments stack (e.g. from “SAND” to “PROD” if results are stable during a specified timeframe). Versatile monitoring and improved health check is also a prerequisite to keep strategy runner in a healthy state.

But let’s do a pause here as the post becomes too large and oversaturated with a different aspect.