Anyone who creates automated trading models is faced with a number of challenges: does it reflect the real-world, what data is available, how long does a test take, does my system test one type of model or multiple... Testing time is usually a trade-off with information detail - look at less detail the less time your test takes. A typical trading day has about 30 million trades. Are you testing one symbol, multiple symbols or the entire market at simultaneously within the context of each other? In the world of big data, new tools and techniques for processing data are emerging everyday.

For us, we needed so satisfy our user requirements with a fast testing system without losing the key characteristics of the market. As well, we wanted to leverage as many existing open source tools as possible to achieve our goals. We’ll breakdown a number of challenges from data storage, software, and hardware. On top of simulator development, we will discuss some of keys to developing a successful trading model.

This talk will cover:

• Requirements for backtesting system
• Trade-offs for speed vs details
• Data storage
• Data processing
• Hardware considerations
• Market Structure (how important is it?)
• Insights on creating a successful model