Stock market prediction using machine learning

Stock market prediction using machine learning

Author: Newhouse Date of post: 05.06.2017

prediction - How can I go about applying machine learning algorithms to stock markets? - Quantitative Finance Stack Exchange

I have recently begun, reading and learning about machine learning. Can someone throw some light onto how to go about it or rather can anyone share their experience and few basic pointers about how to go about it or atleast start applying it to see some results from data sets? How ambitious does this sound? There seems to be a basic fallacy that someone can come along and learn some machine learning or AI algorithms, set them up as a black box, hit go, and sit back while they retire.

Learn statistics and machine learning first, then worry about how to apply them to a given problem. There is no free lunch here. Data analysis is hard work. Read "The Elements of Statistical Learning" the pdf is available for free on the website , and don't start trying to build a model until you understand at least the first 8 chapters.

Once you understand the statistics and machine learning, then you need to learn how to backtest and build a trading model, accounting for transaction costs, etc. After you have a handle on both the analysis and the finance, then it will be somewhat obvious how to apply it.

The entire point of these algorithms is trying to find a way to fit a model to data and produce low bias and variance in prediction i. Here is an example of a trading system using a support vector machine in R , but just keep in mind that you will be doing yourself a huge disservice if you don't spend the time to understand the basics before trying to apply something esoteric.

Just to add an entertaining update: I recently came across this master's thesis: It's an extensive review of different machine learning approaches compared against buy-and-hold. After almost pages, they reach the basic conclusion: My Advice to You: I have only tried genetic programming and some neural networks, and I personally think that the "learning from experience" branch seems to have the most potential.

Each one will have its strengths and weaknesses, but you may be able to combine the predictions of each algorithm into a composite prediction similar to what the winners of the NetFlix Prize did.

The general consensus amongst traders is that Artificial Intelligence is a voodoo science, you can't make a computer predict stock prices and you're sure to loose your money if you try doing it.

Nonetheless, the same people will tell you that just about the only way to make money on the stock market is to build and improve on your own trading strategy and follow it closely which is not actually a bad idea. The idea of AI algorithms is not to build Chip and let him trade for you, but to automate the process of creating strategies. It's a very tedious process and by no means is it easy: As we've heard before, a fundamental issue with AI algorithms is overfitting aka datamining bias: Apparently rats can trade too!

First the ones mentioned earlier: It means that you know you are observing only a sample of data and you want to extrapolate. You thus have to deal with in sample and out of sample issues, overfitting and so on From this viewpoint, data-mining is more focused on dead datasets ie you can see almost all the data, you have an in sample only problem than statistical learning. Because statistical learning is about working on live dataset, the applied maths that deal with them had to focus on a two scales problem:.

If you think about using statistical learning to find the parameters of a linear regression , we can model the state space like this: So we can build these dynamics: Going back to our original generic problem: The results used to prove the efficiency of statistical learning methods can be used to prove the efficiency of trading algorithms. To see that it is enough to read again the coupled dynamical system that allows to write statistical learning: Just replace minimizing a criteria by maximizing the PnL.

See for instance Optimal split of orders across liquidity pools: The statistical learning tools can be used to build iterative trading strategies most of them are iterative and prove their efficiency. The short and brutal answer is: First, because ML and Statistics is not something you can command well in one or two years.

My recommended time horizon to learn anything non-trivial is 10 years. ML not a recipe to make money, but just another means to observe reality. That's why you have statisticians focusing on Physics data analysis, on genomics, on sabermetrics etc. For the record, Jerome Friedman, co-author of ESL quoted above, is a physicist and still holds a courtesy position at SLAC.

Get a bunch of data with some companies that have defaulted, and others that haven't, with a variety of financial information and ratios. Use a machine learning method such as SVM to see if you can predict which companies will default and which will not.

Small Cap Leader - Free NASDAQ Stock Picks and Small Cap Stocks

Use that SVM in the future to short high-probability default companies and long low-probability default companies, with the proceeds of the short sales. I'm currently working on this task, to apply machine learning to stock trading.

However, the concerns raised in other answers are major obstacles. So, I'm taking a different tact. My strategy is more akin to teaching a car to drive - the machine learning is not based on the underlying data, but rather on the driver's reaction to the data.

So based on what the road looks like, the steering position of the wheel. The machine observes "correct" driving, and can very quickly mimic the driving actions.

I think this is referred to as "Supervised Learning" I'm very new to formal machine learning - taking the Stanford class on iTunes U. To apply this tact to stock trading, you take the factors that you personally consider when trading stocks price, moving average, volume, whatever and make those measures available as inputs to your machine learning algorithm.

Of course this doesn't help if you are bad at trading stocks, but it does help create an agent who can do whatever you would do. The overall idea isn't to create a millionaire black box, but rather to free up your time from watching the market closely, or to allow you to apply your strategies to more stocks that you otherwise would be able. If anyone would like to collaborate with me, please feel free to contact. People seem to think that using ML is going to circumvent the process of actually learning to trade, it doesn't.

ML can be used to refine trading ideas, but it doesn't generate them, you need to use your brain for that. One possibility worth exploring is to use the support vector machine learning tool on the Metatrader 5 platform.

Firstly, if you're not familiar with it, Metatrader 5 is a platform developed for users to implement algorithmic trading in forex and CFD markets I'm not sure if the platform can be extended to stocks and other markets. It is typically used for technical analysis based strategies i. The "Support Vector Machine Learning Tool" has been developed by one of the community of users to allow support vector machines to be applied to technical indicators and advise on trades.

A free demo version of the tool can be downloaded here if you want to investigate further. As I understand it, the tool uses historical price data to assess whether hypothetical trades in the past would have been successful. It then takes this data along with the historical values from a number of customisable indicators MACD, oscillators etc , and uses this to train a support vector machine.

A better desciption can be found at the link.

Sorry, but despite being used as a popular example in machine learning, no one has ever achieved a stock market prediction. It does not work for several reasons check random walk by Fama and quite a bit of others, rational decision making fallacy, wrong assumptions As this is not happening and you can be sure all the bank have tried it , we have good evidence, that it just does not work.

How do you think you will achieve what tens of thousands of professionals have failed to, by using the same methods they have, plus limited resources and only basic versions of their methods?

I echo much of what Shane wrote. In addition to reading ESL, I would suggest an even more fundamental study of statistics first. Beyond that, the problems I outlined in in another question on this exchange are highly relevant. In particular, the problem of datamining bias is a serious roadblock to any machine-learning based strategy. Blair Hull as an idea: Perhaps, it's better to create a stock-screener classification , rather than trying to predict stock price non-deterministic regression problem.

Making money that way seems to be much easier, and we can use SVCs and reinforcement learning to achieve the same. I am presently trying to build the same using CAN SLIM model of stock picking, and identifying small-cap stocks that can give multibagger returns in years time. Any other tips or pointers will be gladly appreciated. Look at the seasonal chart for crude oil over 30 years from seasonalcharts.

Traders know what to do even without using machine learning. By posting your answer, you agree to the privacy policy and terms of service. By subscribing, you agree to the privacy policy and terms of service. Sign up or log in to customize your list.

Stack Exchange Inbox Reputation and Badges.

Stock Price Prediction With Big Data and Machine Learning - Eugene Zhulenev

Questions Tags Users Badges Unanswered. Quantitative Finance Stack Exchange is a question and answer site for finance professionals and academics. Join them; it only takes a minute: Here's how it works: Anybody can ask a question Anybody can answer The best answers are voted up and rise to the top. How can I go about applying machine learning algorithms to stock markets? I am not very sure, if this question fits in here.

Also, do mention about standard algorithms that should be tried or looked at while doing this. Tal Fishman 8, 4 39 A good starting point is this blog: I don't think that this makes much reference to machine learning algorithms.

I honestly don't think that this question fits here. Actually it does ok, it depends on the definition of machine learning but e. Shane, great answer below but I also think this is a great question here since I'm sure every quant here at sometime pondered about this at some time.

Unlike the 'develop strategy' link, this is more generic and widely helpful judging from the votes too. My advice to you: Shane 7, 3 40 As a shameless plug, I recently started a guided tour of the above book on my blog if you want to follow along statalgo.

stock market prediction using machine learning

I will be reproducing the major analysis from the book using R. To be frank, in some way, I was someone who was trying to do what you mentioned at the start! Jase As one of the authors of the mentioned master's thesis I can quote my own work and say: It might be like probability theory: This [could be] due to its practical application in gambling.

Here are some resources that you might want to look into: There are several ways to minimize overfitting: Use a validation set: Use online machine learning: The assumption is that various algorithms may have overfit the data in some area, but the "correct" combination of their predictions will have better predictive power. Lirik 7 I don't have such an attitude, but when I talk to various people I get that vibe. I must have missed it.

stock market prediction using machine learning

Perhaps it was actually: Two aspects of statistical learning are useful for trading 1. Because statistical learning is about working on live dataset, the applied maths that deal with them had to focus on a two scales problem: To give you pointers on such mathematical results: Volume 9, Number 1 , A Stochastic Approximation Method, by Herbert Robbins and Sutton Monro, in Ann.

Volume 22, Number 3 , So, study Statistics and Finance for a few years. Go your own way. JD Long 4. Just because you know machine learning and statistics, it does not imply that you know how to apply it to finance. Mike Aug 10 '11 at One basic application is predicting financial distress. Neil McGuigan 4 There is a saying "Picking pennies up in front of steam rollers".

You're doing the equivalent of selling an out-of-the-money put. In this case, you'll make tiny profits for years, then get totally cleaned out when the market melts down every 10 years or so. There is also an equivalent strategy that buys out-of-the-money puts: See Talab's The Black Swan. This is thoroughly incomplete as this information may already be in the current price.

You need another layer to this system to test whether the information is or is not in the price. This is much the same as the approach I'm taking with Neural Nets and blogging about at dekalogblog.

Craig 1 6. There can be patterns in the data, e. Remember that international companies have spent hundreds of billions of dollars and man hours on the very best and brightest artificial intelligence minds over the last 40 years. I've spoken to some of the towers of mind responsible for the alphas over at Citadel and Goldman Sachs, and the hubris from novices to think they can put together an algorithm that will go toe to toe with them, and win, is almost as silly as a child telling you he's going to jump to the moon.

Good luck kid, and watch out for the space martians. Not to say new champions can't be made, but the odds are against you. Just an aside regarding your "most compelling" reason: I'm not sure what you mean by a "stock market prediction" index futures?

You can try this course on Udactiy https: Omorhefere imoloame 21 2. Sohil Gupta 11 1. GIF and the seasonal chart for heating oil over 29 years: GIF Traders know what to do even without using machine learning. Sign up or log in StackExchange. Sign up using Facebook. Sign up using Email and Password. Post as a guest Name. In it, you'll get: The week's top questions and answers Important community announcements Questions that need answers.

Quantitative Finance Stack Exchange works best with JavaScript enabled.

stock market prediction using machine learning

I wouldn't say the general consensus is that it's voodoo science. MathOverflow Mathematics Cross Validated stats Theoretical Computer Science Physics Chemistry Biology Computer Science Philosophy more 3. Meta Stack Exchange Stack Apps Area 51 Stack Overflow Talent.

inserted by FC2 system