This project is a system of sub-projects that manage a large collection of securities and quotes, updates the data, tests strategies, and compares performance.

What is this project?

Ever thought you could just time the market? Well I thought I could. I thought that if you identified easily recognized cyclical patterns and created an automated approach you could make a lot of money. This project delved into the world of stocks, big data, ingestion, visualization, and a large system setup.

This was done in my Junior and Senior years of university.

Breakdown

  • Purpose: Private Project
  • Languages and Technologies: Java, Hibernate, Java Web services, XML, PostgreSQL, JUnit, JasperReports, Bash Scripting,Doman Specific Lanaguage, Big Data, GIT, Subversion, CVS
  • Methodologies used: Custom made domain specific language, simulation, statistical testing, Data mining, Machine Learning, Noise Reduction, Signal to Noise ratio, Overfitting Avoidance
  • Libraries Learned Throughout This project: Guava, Apache Wicket, Hessian, JFreeChart, Highcharts, JasperReports, Hibernate, Java concurrency patterns

Evolution

This project went over many phases of development.

  1. First this started as a simple low data approach and application that would attempt to apply a dataset. First using CVS
  2. This moved on to a second strategy being run
  3. Added a maven build file
  4. The data moved to a mysql database
  5. Moved to SVN
  6. Moved IDEs from Netbeans to Intellij
  7. Added in automated backups
  8. Built a generic strategy simiulation system that made it easier to create new strategies and backtest them
  9. Create a backtesting statistics reporting peace
  10. Added automated tools (Gitlab, Jenkins, Jacco, Checkstyle, Findbugs, PMD)
  11. Moved to GIT
  12. Got code coverage testing up
  13. Created a DSL to query and start stock strategy simulations
  14. Create a web framework to show reporting results.
  15. Automated data ingestion from an SMTP server.
  16. Created a JUnit extended testing framework

This project lasted from 2007 all the way through ~2013 as a personal project.

Pictures

Visualization of Multiple runs on individual stocks

Visualization of Multiple runs on individual stocks

Judging Strategy Performance

Judging Strategy Performance

Did it make any money?

No. At the time I didn’t have enough money to trade with an API. Also at the time, the only option to do this was InteractiveBrokers which had a 10k account minimum. What I did learn out of this project was some of the high value stocks, the downsides to algorithms and their unpredictable pain points, and some strong database and large system skills.

What else did I learn?

  • I learned the meaning of liquidity in stocks (Penny stocks and medical stocks will pop with no warning as a one off event)
  • How to do anti-joins well on large tables in Postgres. (The historical stock price table had over 20mil rows)
  • How to create a domain specific languages
  • Deep into the details of JUnit testing
  • How algorithms can be overly strict for your problem. (If you’re looking for stock drops at 5% it will look for 5%, not a close 4.99%)

Would I do this again?

At the time I think I would. I learned quite a lot here. What would I change? I would have probably wrote the strategies in a similar language to use, such as Groovy. I would have built a stronger base to run the strategies. But overall a modern overhaul would be pretty amazing.