To me, the thrill of algo trading comes with figuring out a new trading strategy, implementing it, seeing the first backtest results, taking it live and see it (hopefully take money). That is the fun, exiting bit, that gets people fired up and engaged. But none of it would be possible without the right technical infrastructure. Even worse, waiting for days to finish a backtest, being unable to optimize parameters or having to make compromises left right and center when it comes to what data can be used can suck all the fun out of this trading approach.

That’s the reason why proj-h started out with a strong focus on building a technical infrastructure that enhances the actual trading and does not hinder it, and in this blog post I will introduce you to it. Anyhow, it is important to state from the get go that this is the proj-h approach and in no way or form do I want to claim that this is the best, smartest or most efficient way to do things. It’s simply what works for us. So the goal of this post is not to guide you to the promised land of algorithmic trading (since I haven’t found it myself… yet), but to simply provide transparency about this project, some food for thought and maybe even inspiration for other aspiring traders implementing their own algorithmic trading system. So with no further ado, lets jump right into the topic.

Starting with the end in mind

While it might be amazing to build a really cutting edge, high-performance trading infrastructure for the sake of building something cool, that was definitely not my approach. The goal of proj-h is to dive deep into the world of algorithmic trading, with a strong focus on the trading part. So the technical infrastructure works as an enabler but its just the means to an end. Anyhow, for the infrastructure to support anything it was instrumental to first define what the requirements are. Here I leaned heavily on my professional experience as Algorithmic Trader and came up with the following set of attributes that I think make or break a good trading system.

1. It needs to be fast. First and foremost because an algorithm should be able to react to rapidly moving market changes and not take seconds or even minutes to place or cancel an order. Second, nothing, really nothing is worse than slow backtests. It almost causes me physical pain if the backtest for a single trading day takes a couple of minutes. Fast algorithms are also key in being able to run any form of parameter optimization, which in my opinion is a crucial part of any strategy.
2. The trading engine should be the same for backtesting, paper trading and live trading. This means, that there is only one implementation of every algorithm that takes trading events from somewhere as inputs, processes them and then places orders based on that as output. Depending on if the system is running a backtest or is trading live, the inputs change and the orders that the algorithm writes are handled differently, but the core logic of the algorithm stays the same.
3. It should make life easier, not harder. This is now very personal, but since I don’t have a CS or dev ops background, for me it was very important to keep the whole infrastructure part as simple as possible. That means no crazy cloud infrastructure to take care of, no innovative new tech stack, simply using things that I am familiar with or things that I feel comfortable using.

The building blocks of proj-h

Overview

With the ideas described above in mind, the architecture shown in the figure was build. While it might look a little bit confusing on first glance, its actually quite simple and in my opinion that is the charm of it. It’s hard to find a natural entry point to dive into it, but it makes sense to start with the Algo Orchestration and Optimization tool, as this is the “brain” of the whole operation.

Algo Orchestration and Optimization Tool

This wonderfully named service is a python script is the conductor of the whole system. In a sense its like a Genie, you tell it what you want to happen and it makes sure to fulfill your wish by starting all other services as required. To do so, it naturally needs some specific information, namely which strategy you want to run, start and end time, the parameters of the strategy or if you want to optimize it and finally, if you want to run the strategy live, in paper trading or as backtest. It then takes that information, processes it, writes it to the database and starts the trading engine and the event streamer. In the optimization mode it does quite some things more, but that is a topic for a different blog post.

Trading Engine and Event Streamer

After the Algo Orchestration tool translates the user request into required operation, the trading engine comes into play. The trading engine is the heart of the whole system and this is where the individual strategies are implemented and executed. This works in the “standard” algo trading way that can be found pretty much anywhere. The engine gets trading events (this is just a fancy generic term for bars or trades or any datapoint with a timestamp, volume and price) and anytime a new event arrives, checks what the strategy wants to do. For example, a MA Crossover strategy would check for a crossover on every event and then place an order. It does so by writing the order request into the database, which is then processed somewhere else. This sounds very simple and the basic logic also is rather straightforward, the challenge here is to make this whole process as robust as possible.

As the trading engine simply processes whatever trading events that are provided to it, a service is needed that serves those events. In my case this is called the event streamer. There is a specific reason, why this is a standalone service, which is that the trading engine is completely ignorant of where the events come from on purpose, it just processes them whenever it gets them. This allows us the use the exact same piece of logic in the engine for backtesting, paper trading and live trading. Depending on what mode the algorithm is running in, the event streamer either hooks up to an exchange for live data or loads historical trades, does some preprocessing to possibly aggregate them and transform them into a uniform format and then writes them to a ZMQ queue, where the trading engine can pick them up.

Historic Trade Data Script

The historic trade data script is simply a family of tiny scripts that pick up historic data from different data sources and write them out as parquet files. This is required to get the necessary data for backtests and the only real challenge here is to first of all find a source for the required data and then make the script respect the rate limits of the data providers (mostly exchanges). This makes the runtimes quite long if a larger time period is queried, but once you have the data you (theoretically) never have to get it again.

Order Management System

Last but not least, we have the order management system. As the name implies, this is the gate to the exchanges, where the desired orders written by the trading engine are processed. So this service constantly checks the database for new orders, places them on the exchange the algo wants to place them at, checks if they get fulfilled and finally updates the database. You might ask yourself why this service is written in Python, if speed is an issue. The main reason for that is convenience. There are great Python libraries like ccxt that have multi exchange support, so instead of writing all important functions for exchange A, then again for exchange B and so on you can simply use ccxt to write one set of functions that work for all exchanges they support. If execution speed ever becomes an issue or I want to test some HFT strategies I will have to rethink that choice, but for now it works quite well.

Tech Stack Choices

Now that the core services of proj-h were introduced, it makes sense to discuss the language and technology choices that were made.

Most of smaller services are written in Python, which was a natural choice for me since I am very familiar with it and the vast amount of libraries make it very versatile and allow very rapid development. Anyhow python has one major downside in my eyes, it’s slow, no matter how much you optimize it. Hence it was clear, that the main part of proj-h’s trading system needed to be written in a faster language and here I chose Go. On the surface this might be a bit of a controversial choice, since it does not really belong to the group of fastest languages out there (I guess if you ask any AI service now what language you should use to build an algorithmic trading system it will almost exclusively tell you Rust). Anyhow, there is one major reason for me to use it, its the fastest language that I am familiar with. And with the current implementation it can handle tens of thousand trade events per second, which is more than enough for the current scope of proj-h.

A deliberate choice was made to save the historical data used for backtesting in local files in the parquet format, since polars is used to handle data wrangling in python scripts and those two technologies synergize quite well. For the communication between the event streamer and the trading engine ZeroMQ is used, an open source messaging library, as it is light weight and can handle the amount of events being send quite well. Finally, after testing InfluxDB, Clickhouse and timescaleDB the choice fell on timescale, since a relational database felt like the most elegant solution for the use case and it is quite performant.

Conclusion

The goal of this blog post is to present what is happening “under the hood” at proj-h and explain, why the system is designed as it was. At proj-h, the infrastructure and trading system are supposed to be an enabler, so when developing it the focus was always on the “why”. Building the systems behind proj-h has been an iterative process, driven by the core need for a robust and efficient algorithmic trading infrastructure and I am proud to say that this was achieved. I would also love to know about your opinions of my approach to trading infrastructure, so don't hesitate to shoot me a message at hello@proj-h.com.