Stock Data Analysis with Python (Second Edition)
This article is originally published at https://ntguardian.wordpress.com
Introduction
This is a lecture for MATH 4100/CS 5160: Introduction to Data Science, offered at the University of Utah, introducing time series data analysis applied to finance. This is also an update to my earlier blog posts on the same topic (this one combining them together). I strongly advise referring to this blog post instead of the previous ones (which I am not altering for the sake of preserving a record). The code should work as of July 7th, 2018. (And sorry for some of the formatting; WordPress.com’s free version doesn’t play nice with code or tables.)
Advanced mathematics and statistics have been present in finance for some time. Prior to the 1980s, banking and finance were well-known for being “boring”; investment banking was distinct from commercial banking and the primary role of the industry was handling “simple” (at least in comparison to today) financial instruments, such as loans. Deregulation under the Regan administration, coupled with an influx of mathematical talent, transformed the industry from the “boring” business of banking to what it is today, and since then, finance has joined the other sciences as a motivation for mathematical research and advancement. For example one of the biggest recent achievements of mathematics was the derivation of the Black-Scholes formula, which facilitated the pricing of stock options (a contract giving the holder the right to purchase or sell a stock at a particular price to the issuer of the option). That said, bad statistical models, including the Black-Scholes formula, hold part of the blame for the 2008 financial crisis.
In recent years, computer science has joined advanced mathematics in revolutionizing finance and trading, the practice of buying and selling of financial assets for the purpose of making a profit. In recent years, trading has become dominated by computers; algorithms are responsible for making rapid split-second trading decisions faster than humans could make (so rapidly, the speed at which light travels is a limitation when designing systems). Additionally, machine learning and data mining techniques are growing in popularity in the financial sector, and likely will continue to do so. For example, high-frequency trading (HFT) is a branch of algorithmic trading where computers make thousands of trades in short periods of time, engaging in complex strategies such as statistical arbitrage and market making. While algorithms may outperform humans, the technology is still new and playing an increasing role in a famously turbulent, high-stakes arena. HFT was responsible for phenomena such as the 2010 flash crash and a 2013 flash crash prompted by a hacked Associated Press tweet about an attack on the White House.
This lecture, however, will not be about how to crash the stock market with bad mathematical models or trading algorithms. Instead, I intend to provide you with basic tools for handling and analyzing stock market data with Python. We will be using stock data as a first exposure to time series data, which is data considered dependent on the time it was observed (other examples of time series include temperature data, demand for energy on a power grid, Internet server load, and many, many others). I will also discuss moving averages, how to construct trading strategies using moving averages, how to formulate exit strategies upon entering a position, and how to evaluate a strategy with backtesting.
DISCLAIMER: THIS IS NOT FINANCIAL ADVICE!!! Furthermore, I have ZERO experience as a trader (a lot of this knowledge comes from a one-semester course on stock trading I took at Salt Lake Community College)! This is purely introductory knowledge, not enough to make a living trading stocks. People can and do lose money trading stocks, and you do so at your own risk!
Preliminaries
I will be using two packages, quandl and pandas_datareader, which are not installed with Anaconda if you are using it. To install these packages, run the following at the appropriate command prompt:
conda install quandl conda install pandas-datareader
Getting and Visualizing Stock Data
Getting Data from Quandl
Before we analyze stock data, we need to get it into some workable format. Stock data can be obtained from Yahoo! Finance, Google Finance, or a number of other sources. These days I recommend getting data from Quandl, a provider of community-maintained financial and economic data. (Yahoo! Finance used to be the go-to source for good quality stock data, but the API was discontinued in 2017 and reliable data can no longer be obtained: see this question/answer on StackExchange for more details.)
By default the get()
function in quandl will return a pandas DataFrame
containing the fetched data.
import pandas as pd import quandl import datetime # We will look at stock prices over the past year, starting at January 1, 2016 start = datetime.datetime(2016,1,1) end = datetime.date.today() # Let's get Apple stock data; Apple's ticker symbol is AAPL # First argument is the series we want, second is the source ("yahoo" for Yahoo! Finance), third is the start date, fourth is the end date s = "AAPL" apple = quandl.get("WIKI/" + s, start_date=start, end_date=end) type(apple)
pandas.core.frame.DataFrame
apple.head()
Open | High | Low | Close | Volume | |
---|---|---|---|---|---|
Date | |||||
2016-01-04 | 102.61 | 105.368 | 102.00 | 105.35 | 67649387.0 |
2016-01-05 | 105.75 | 105.850 | 102.41 | 102.71 | 55790992.0 |
2016-01-06 | 100.56 | 102.370 | 99.87 | 100.70 | 68457388.0 |
2016-01-07 | 98.68 | 100.130 | 96.43 | 96.45 | 81094428.0 |
2016-01-08 | 98.55 | 99.110 | 96.76 | 96.96 | 70798016.0 |
Ex-Dividend | Split Ratio | Adj. Open | Adj. High | Adj. Low | Adj. Close | Adj. Volume |
---|---|---|---|---|---|---|
0.0 | 1.0 | 99.136516 | 101.801154 | 98.547165 | 101.783763 | 67649387.0 |
0.0 | 1.0 | 102.170223 | 102.266838 | 98.943286 | 99.233131 | 55790992.0 |
0.0 | 1.0 | 97.155911 | 98.904640 | 96.489269 | 97.291172 | 68457388.0 |
0.0 | 1.0 | 95.339552 | 96.740467 | 93.165717 | 93.185040 | 81094428.0 |
0.0 | 1.0 | 95.213952 | 95.754996 | 93.484546 | 93.677776 | 70798016.0 |
Let’s briefly discuss this. Open is the price of the stock at the beginning of the trading day (it need not be the closing price of the previous trading day), high is the highest price of the stock on that trading day, low the lowest price of the stock on that trading day, and close the price of the stock at closing time. Volume indicates how many stocks were traded. Adjusted prices (such as the adjusted close) is the price of the stock that adjusts the price for corporate actions. While stock prices are considered to be set mostly by traders, stock splits (when the company makes each extant stock worth two and halves the price) and dividends (payout of company profits per share) also affect the price of a stock and should be accounted for.
Visualizing Stock Data
Now that we have stock data we would like to visualize it. I first demonstrate how to do so using the matplotlib package. Notice that the apple
DataFrame
object has a convenience method, plot()
, which makes creating plots easier.
import matplotlib.pyplot as plt # Import matplotlib # This line is necessary for the plot to appear in a Jupyter notebook %matplotlib inline # Control the default size of figures in this Jupyter notebook %pylab inline pylab.rcParams['figure.figsize'] = (15, 9) # Change the size of plots apple["Adj. Close"].plot(grid = True) # Plot the adjusted closing price of AAPL
Populating the interactive namespace from numpy and matplotlib
A linechart is fine, but there are at least four variables involved for each date (open, high, low, and close), and we would like to have some visual way to see all four variables that does not require plotting four separate lines. Financial data is often plotted with a Japanese candlestick plot, so named because it was first created by 18th century Japanese rice traders. Such a chart can be created with matplotlib, though it requires considerable effort.
I have made a function you are welcome to use to more easily create candlestick charts from pandas data frames, and use it to plot our stock data. (Code is based off this example, and you can read the documentation for the functions involved here.)
from matplotlib.dates import DateFormatter, WeekdayLocator,\ DayLocator, MONDAY from mpl_finance import candlestick_ohlc def pandas_candlestick_ohlc(dat, stick = "day", adj = False, otherseries = None): """ :param dat: pandas DataFrame object with datetime64 index, and float columns "Open", "High", "Low", and "Close", likely created via DataReader from "yahoo" :param stick: A string or number indicating the period of time covered by a single candlestick. Valid string inputs include "day", "week", "month", and "year", ("day" default), and any numeric input indicates the number of trading days included in a period :param adj: A boolean indicating whether to use adjusted prices :param otherseries: An iterable that will be coerced into a list, containing the columns of dat that hold other series to be plotted as lines This will show a Japanese candlestick plot for stock data stored in dat, also plotting other series if passed. """ mondays = WeekdayLocator(MONDAY) # major ticks on the mondays alldays = DayLocator() # minor ticks on the days dayFormatter = DateFormatter('%d') # e.g., 12 # Create a new DataFrame which includes OHLC data for each period specified by stick input fields = ["Open", "High", "Low", "Close"] if adj: fields = ["Adj. " + s for s in fields] transdat = dat.loc[:,fields] transdat.columns = pd.Index(["Open", "High", "Low", "Close"]) if (type(stick) == str): if stick == "day": plotdat = transdat stick = 1 # Used for plotting elif stick in ["week", "month", "year"]: if stick == "week": transdat["week"] = pd.to_datetime(transdat.index).map(lambda x: x.isocalendar()[1]) # Identify weeks elif stick == "month": transdat["month"] = pd.to_datetime(transdat.index).map(lambda x: x.month) # Identify months transdat["year"] = pd.to_datetime(transdat.index).map(lambda x: x.isocalendar()[0]) # Identify years grouped = transdat.groupby(list(set(["year",stick]))) # Group by year and other appropriate variable plotdat = pd.DataFrame({"Open": [], "High": [], "Low": [], "Close": []}) # Create empty data frame containing what will be plotted for name, group in grouped: plotdat = plotdat.append(pd.DataFrame({"Open": group.iloc[0,0], "High": max(group.High), "Low": min(group.Low), "Close": group.iloc[-1,3]}, index = [group.index[0]])) if stick == "week": stick = 5 elif stick == "month": stick = 30 elif stick == "year": stick = 365 elif (type(stick) == int and stick >= 1): transdat["stick"] = [np.floor(i / stick) for i in range(len(transdat.index))] grouped = transdat.groupby("stick") plotdat = pd.DataFrame({"Open": [], "High": [], "Low": [], "Close": []}) # Create empty data frame containing what will be plotted for name, group in grouped: plotdat = plotdat.append(pd.DataFrame({"Open": group.iloc[0,0], "High": max(group.High), "Low": min(group.Low), "Close": group.iloc[-1,3]}, index = [group.index[0]])) else: raise ValueError('Valid inputs to argument "stick" include the strings "day", "week", "month", "year", or a positive integer') # Set plot parameters, including the axis object ax used for plotting fig, ax = plt.subplots() fig.subplots_adjust(bottom=0.2) if plotdat.index[-1] - plotdat.index[0] < pd.Timedelta('730 days'): weekFormatter = DateFormatter('%b %d') # e.g., Jan 12 ax.xaxis.set_major_locator(mondays) ax.xaxis.set_minor_locator(alldays) else: weekFormatter = DateFormatter('%b %d, %Y') ax.xaxis.set_major_formatter(weekFormatter) ax.grid(True) # Create the candelstick chart candlestick_ohlc(ax, list(zip(list(date2num(plotdat.index.tolist())), plotdat["Open"].tolist(), plotdat["High"].tolist(), plotdat["Low"].tolist(), plotdat["Close"].tolist())), colorup = "black", colordown = "red", width = stick * .4) # Plot other series (such as moving averages) as lines if otherseries != None: if type(otherseries) != list: otherseries = [otherseries] dat.loc[:,otherseries].plot(ax = ax, lw = 1.3, grid = True) ax.xaxis_date() ax.autoscale_view() plt.setp(plt.gca().get_xticklabels(), rotation=45, horizontalalignment='right') plt.show() pandas_candlestick_ohlc(apple, adj=True, stick="month")
With a candlestick chart, a black candlestick indicates a day where the closing price was higher than the open (a gain), while a red candlestick indicates a day where the open was higher than the close (a loss). The wicks indicate the high and the low, and the body the open and close (hue is used to determine which end of the body is the open and which the close). Candlestick charts are popular in finance and some strategies in technical analysis use them to make trading decisions, depending on the shape, color, and position of the candles. I will not cover such strategies today.
We may wish to plot multiple financial instruments together; we may want to compare stocks, compare them to the market, or look at other securities such as exchange-traded funds (ETFs). Later, we will also want to see how to plot a financial instrument against some indicator, like a moving average. For this you would rather use a line chart than a candlestick chart. (How would you plot multiple candlestick charts on top of one another without cluttering the chart?)
Below, I get stock data for some other tech companies and plot their adjusted close together.
microsoft, google = (quandl.get("WIKI/" + s, start_date=start, end_date=end) for s in ["MSFT", "GOOG"]) # Below I create a DataFrame consisting of the adjusted closing price of these stocks, first by making a list of these objects and using the join method stocks = pd.DataFrame({"AAPL": apple["Adj. Close"], "MSFT": microsoft["Adj. Close"], "GOOG": google["Adj. Close"]}) stocks.head()
AAPL | GOOG | MSFT | |
---|---|---|---|
Date | |||
2016-01-04 | 101.783763 | 741.84 | 52.181598 |
2016-01-05 | 99.233131 | 742.58 | 52.419653 |
2016-01-06 | 97.291172 | 743.62 | 51.467434 |
2016-01-07 | 93.185040 | 726.39 | 49.677262 |
2016-01-08 | 93.677776 | 714.47 | 49.829617 |
stocks.plot(grid = True)
What’s wrong with this chart? While absolute price is important (pricy stocks are difficult to purchase, which affects not only their volatility but your ability to trade that stock), when trading, we are more concerned about the relative change of an asset rather than its absolute price. Google’s stocks are much more expensive than Apple’s or Microsoft’s, and this difference makes Apple’s and Microsoft’s stocks appear much less volatile than they truly are (that is, their price appears to not deviate much).
One solution would be to use two different scales when plotting the data; one scale will be used by Apple and Microsoft stocks, and the other by Google.
stocks.plot(secondary_y = ["AAPL", "MSFT"], grid = True)
A “better” solution, though, would be to plot the information we actually want: the stock’s returns. This involves transforming the data into something more useful for our purposes. There are multiple transformations we could apply.
One transformation would be to consider the stock’s return since the beginning of the period of interest. In other words, we plot:
This will require transforming the data in the stocks
object, which I do next. Notice that I am using a lambda function, which allows me to pass a small function defined quickly as a parameter to another function or method (you can read more about lambda functions here).
python # df.apply(arg) will apply the function arg to each column in df, and return a DataFrame with the result # Recall that lambda x is an anonymous function accepting parameter x; in this case, x will be a pandas Series object stock_return = stocks.apply(lambda x: x / x[0]) stock_return.head() - 1
AAPL | GOOG | MSFT | |
---|---|---|---|
Date | |||
2016-01-04 | 0.000000 | 0.000000 | 0.000000 |
2016-01-05 | -0.025059 | 0.000998 | 0.004562 |
2016-01-06 | -0.044139 | 0.002399 | -0.013686 |
2016-01-07 | -0.084480 | -0.020827 | -0.047993 |
2016-01-08 | -0.079639 | -0.036895 | -0.045073 |
stock_return.plot(grid = True).axhline(y = 1, color = "black", lw = 2)
This is a much more useful plot. We can now see how profitable each stock was since the beginning of the period. Furthermore, we see that these stocks are highly correlated; they generally move in the same direction, a fact that was difficult to see in the other charts.
Alternatively, we could plot the change of each stock per day. One way to do so would be to plot the percentage increase of a stock when comparing day to day , with the formula:
But change could be thought of differently as:
These formulas are not the same and can lead to differing conclusions, but there is another way to model the growth of a stock: with log differences.
(Here, is the natural log, and our definition does not depend as strongly on whether we use or .) The advantage of using log differences is that this difference can be interpreted as the percentage change in a stock but does not depend on the denominator of a fraction. Additionally, log differences have a desirable property: the sum of the log differences can be interpreted as the total change (as a percentage) over the period summed (which is not a property of the other formulations; they will overestimate growth). Log differences also more cleanly correspond to how stock prices are modeled in continuous time.
We can obtain and plot the log differences of the data in stocks
as follows:
# Let's use NumPy's log function, though math's log function would work just as well import numpy as np stock_change = stocks.apply(lambda x: np.log(x) - np.log(x.shift(1))) # shift moves dates back by 1. stock_change.head()
AAPL | GOOG | MSFT | |
---|---|---|---|
Date | |||
2016-01-04 | NaN | NaN | NaN |
2016-01-05 | -0.025379 | 0.000997 | 0.004552 |
2016-01-06 | -0.019764 | 0.001400 | -0.018332 |
2016-01-07 | -0.043121 | -0.023443 | -0.035402 |
2016-01-08 | 0.005274 | -0.016546 | 0.003062 |
stock_change.plot(grid = True).axhline(y = 0, color = "black", lw = 2)
Which transformation do you prefer? Looking at returns since the beginning of the period make the overall trend of the securities in question much more apparent. Changes between days, though, are what more advanced methods actually consider when modelling the behavior of a stock. so they should not be ignored.
We often want to compare the performance of stocks to the performance of the overall market. SPY, which is the ticker symbol for the SPDR S&P 500 exchange-traded mutual fund (ETF), is a fund that attempts only to imitate the composition of the S&P 500 stock index, and thus represents the value in “the market.”
SPY data is not available for free from Quandl, so I will get this data from Yahoo! Finance. (I don’t have a choice.)
Below I get data for SPY and compare its performance to the performance of our stocks.
#import pandas_datareader.data as web # Going to get SPY from Yahoo! (I know I said you shouldn't but I didn't have a choice) #spyder = web.DataReader("SPY", "yahoo", start, end) # Didn't work #spyder = web.DataReader("SPY", "google", start, end) # Didn't work either # If all else fails, read from a file, obtained from here: http://www.nasdaq.com/symbol/spy/historical spyderdat = pd.read_csv("/home/curtis/Downloads/HistoricalQuotes.csv") # Obviously specific to my system; set to # location on your machine spyderdat = pd.DataFrame(spyderdat.loc[:, ["open", "high", "low", "close", "close"]].iloc[1:].as_matrix(), index=pd.DatetimeIndex(spyderdat.iloc[1:, 0]), columns=["Open", "High", "Low", "Close", "Adj Close"]).sort_index() spyder = spyderdat.loc[start:end] stocks = stocks.join(spyder.loc[:, "Adj Close"]).rename(columns={"Adj Close": "SPY"}) stocks.head()
AAPL | GOOG | MSFT | SPY | |
---|---|---|---|---|
Date | ||||
2016-01-04 | 101.783763 | 741.84 | 52.181598 | 201.0192 |
2016-01-05 | 99.233131 | 742.58 | 52.419653 | 201.3600 |
2016-01-06 | 97.291172 | 743.62 | 51.467434 | 198.8200 |
2016-01-07 | 93.185040 | 726.39 | 49.677262 | 194.0500 |
2016-01-08 | 93.677776 | 714.47 | 49.829617 | 191.9230 |
stock_return = stocks.apply(lambda x: x / x[0]) stock_return.plot(grid = True).axhline(y = 1, color = "black", lw = 2)
stock_change = stocks.apply(lambda x: np.log(x) - np.log(x.shift(1))) stock_change.plot(grid=True).axhline(y = 0, color = "black", lw = 2)
Classical Risk Metrics
From what we have so far we can already compute informative metrics for our stocks, which can be considered some measure of risk.
First, we will want to annualize our returns, thus computing the annual percentage rate (APR). This helps us keep returns on a common time scale.
stock_change_apr = stock_change * 252 * 100 # There are 252 trading days in a year; the 100 converts to percentages stock_change_apr.tail()
AAPL | GOOG | MSFT | SPY | |
---|---|---|---|---|
Date | ||||
2018-03-21 | -577.463148 | -157.285338 | -176.499833 | NaN |
2018-03-22 | -359.355133 | -984.592233 | -743.873619 | NaN |
2018-03-23 | -589.663945 | -669.637836 | -743.366326 | NaN |
2018-03-26 | 1168.762361 | 768.649993 | 1839.012005 | NaN |
2018-03-27 | -654.582257 | -1178.241231 | -1185.615651 | NaN |
Some of these numbers look initially like nonsense, but that’s okay for now.
The metrics I want are:
- The average return
- Volatility (the standard deviation of returns)
- and
- The Sharpe ratio
The first two metrics are largely self-explanatory, but the latter two need explaining.
First, the risk-free rate, which I denote by , is the rate of return on a risk-free financial asset. This asset exists only in theory but often yields on low-risk instruments like 3-month U.S. Treasury Bills can be viewed as being virtually risk-free and thus their yields can be used to approximate the risk-free rate. I get the data for these instruments below.
tbill = quandl.get("FRED/TB3MS", start_date=start, end_date=end) tbill.tail()
Value | |
---|---|
Date | |
2018-02-01 | 1.57 |
2018-03-01 | 1.70 |
2018-04-01 | 1.76 |
2018-05-01 | 1.86 |
2018-06-01 | 1.90 |
tbill.plot()
rrf = tbill.iloc[-1, 0] # Get the most recent Treasury Bill rate rrf
1.8999999999999999
Now, a linear regression model is a model of the following form:
is an error process. Another way to think of this process model is:
is the predicted value of given . In other words, a linear regression model tells you how and are related, and how values of can be used to predict values of . is the intercept of the model and is the slope. In particular, would be the predicted value of if were zero, and gives how much changes when changes by one unit.
There is an easy way to compute and given the sample means and and sample standard deviations and and the correlation between and $y$, denoted with :
In finance, we use and like so:
is the return of a financial asset (a stock) and is the excess return, or return exceeding the risk-free rate of return. is the return of the market at time . Then and can be interpreted like so:
- is average excess return over the market.
- is how much a stock moves in relation to the market. If then the stock generally moves in the same direction as the market, while when the stock moves strongly in response to the market the stock is less responsive to the market.
Below I get a pandas Series
that contains how much each stock is correlated with SPY (our approximation of the market).
smcorr = stock_change_apr.drop("SPY", 1).corrwith(stock_change_apr.SPY) # Since RRF is constant it doesn't change the # correlation so we can ignore it in our # calculation smcorr
AAPL 0.547184 GOOG 0.592740 MSFT 0.671356 dtype: float64
Then I compute and .
sy = stock_change_apr.drop("SPY", 1).std() sx = stock_change_apr.SPY.std() sy
AAPL 339.921782 GOOG 312.319468 MSFT 329.308164 dtype: float64
sx # Standard deviation for x
164.60477271861888
ybar = stock_change_apr.drop("SPY", 1).mean() - rrf xbar = stock_change_apr.SPY.mean() - rrf ybar
AAPL 19.769035 GOOG 11.766893 MSFT 22.362806 dtype: float64
xbar
14.962934571070926
beta = smcorr * sy / sx alpha = ybar - beta * xbar beta
AAPL 1.129978 GOOG 1.124658 MSFT 1.343114 dtype: float64
alpha
AAPL 2.861252 GOOG -5.061295 MSFT 2.265881 dtype: float64
The Sharpe ratio is another popular risk metric, defined below:
Here is the volatility of the stock. We want the sharpe ratio to be large. A large Sharpe ratio indicates that the stock's excess returns are large relative to the stock's volatilitly. Additionally, the Sharpe ratio is tied to a statistical test (the -test) to determine if a stock earns more on average than the risk-free rate; the larger this ratio, the more likely this is to be the case.
Your challenge now is to compute the Sharpe ratio for each stock listed here, and interpret it. Which stock seems to be the better investment according to the Sharpe ratio?
sharpe = (ybar - rrf)/sy sharpe
AAPL 0.052568 GOOG 0.031592 MSFT 0.062139 dtype: float64
(xbar - rrf)/sx
0.079359391318507888
Moving Averages
Charts are very useful. In fact, some traders base their strategies almost entirely off charts (these are the "technicians", since trading strategies based off finding patterns in charts is a part of the trading doctrine known as technical analysis). Let's now consider how we can find trends in stocks.
A -day moving average is, for a series and a point in time , the average of the past days: that is, if denotes a moving average process, then:
Moving averages smooth a series and helps identify trends. The larger is, the less responsive a moving average process is to short-term fluctuations in the series . The idea is that moving average processes help identify trends from "noise". Fast moving averages have smaller and more closely follow the stock, while slow moving averages have larger , resulting in them responding less to the fluctuations of the stock and being more stable.
pandas provides functionality for easily computing moving averages. I demonstrate its use by creating a 20-day (one month) moving average for the Apple data, and plotting it alongside the stock.
apple["20d"] = np.round(apple["Adj. Close"].rolling(window = 20, center = False).mean(), 2) pandas_candlestick_ohlc(apple.loc['2016-01-04':'2016-12-31',:], otherseries = "20d", adj=True)
Notice how late the rolling average begins. It cannot be computed until 20 days have passed. This limitation becomes more severe for longer moving averages. Because I would like to be able to compute 200-day moving averages, I'm going to extend out how much AAPL data we have. That said, we will still largely focus on 2016.
start = datetime.datetime(2010,1,1) apple = quandl.get("WIKI/AAPL", start_date=start, end_date=end) apple["20d"] = np.round(apple["Adj. Close"].rolling(window = 20, center = False).mean(), 2) pandas_candlestick_ohlc(apple.loc['2016-01-04':'2016-12-31',:], otherseries = "20d", adj=True)
You will notice that a moving average is much smoother than the actua stock data. Additionally, it’s a stubborn indicator; a stock needs to be above or below the moving average line in order for the line to change direction. Thus, crossing a moving average signals a possible change in trend, and should draw attention.
Traders are usually interested in multiple moving averages, such as the 20-day, 50-day, and 200-day moving averages. It’s easy to examine multiple moving averages at once.
apple["50d"] = np.round(apple["Adj. Close"].rolling(window = 50, center = False).mean(), 2) apple["200d"] = np.round(apple["Adj. Close"].rolling(window = 200, center = False).mean(), 2) pandas_candlestick_ohlc(apple.loc['2016-01-04':'2016-12-31',:], otherseries = ["20d", "50d", "200d"], adj=True)
The 20-day moving average is the most sensitive to local changes, and the 200-day moving average the least. Here, the 200-day moving average indicates an overall bearish trend: the stock is trending downward over time. The 20-day moving average is at times bearish and at other times bullish, where a positive swing is expected. You can also see that the crossing of moving average lines indicate changes in trend. These crossings are what we can use as trading signals, or indications that a financial security is changind direction and a profitable trade might be made.
Trading Strategy
Our concern now is to design and evaluate trading strategies.
Any trader must have a set of rules that determine how much of her money she is willing to bet on any single trade. For example, a trader may decide that under no circumstances will she risk more than 10% of her portfolio on a trade. Additionally, in any trade, a trader must have an exit strategy, a set of conditions determining when she will exit the position, for either profit or loss. A trader may set a target, which is the minimum profit that will induce the trader to leave the position. Likewise, a trader may have a maximum loss she is willing to tolerate; if potential losses go beyond this amount, the trader will exit the position in order to prevent any further loss. We will suppose that the amount of money in the portfolio involved in any particular trade is a fixed proportion; 10% seems like a good number.
Here, I will be demonstrating a moving average crossover strategy. We will use two moving averages, one we consider “fast”, and the other “slow”. The strategy is:
- Trade the asset when the fast moving average crosses over the slow moving average.
- Exit the trade when the fast moving average crosses over the slow moving average again.
A trade will be prompted when the fast moving average crosses from below to above the slow moving average, and the trade will be exited when the fast moving average crosses below the slow moving average later.
We now have a complete strategy. But before we decide we want to use it, we should try to evaluate the quality of the strategy first. The usual means for doing so is backtesting, which is looking at how profitable the strategy is on historical data. For example, looking at the above chart’s performance on Apple stock, if the 20-day moving average is the fast moving average and the 50-day moving average the slow, this strategy does not appear to be very profitable, at least not if you are always taking long positions.
Let’s see if we can automate the backtesting task. We first identify when the 20-day average is below the 50-day average, and vice versa.
apple['20d-50d'] = apple['20d'] - apple['50d'] apple.tail()
Open | High | Low | Close | Volume | |
---|---|---|---|---|---|
Date | |||||
2018-03-21 | 175.04 | 175.09 | 171.26 | 171.270 | 35247358.0 |
2018-03-22 | 170.00 | 172.68 | 168.60 | 168.845 | 41051076.0 |
2018-03-23 | 168.39 | 169.92 | 164.94 | 164.940 | 40248954.0 |
2018-03-26 | 168.07 | 173.10 | 166.44 | 172.770 | 36272617.0 |
2018-03-27 | 173.68 | 175.15 | 166.92 | 168.340 | 38962839.0 |
Ex-Dividend | Split Ratio | Adj. Open | Adj. High | Adj. Low | Adj. Close |
---|---|---|---|---|---|
0.0 | 1.0 | 175.04 | 175.09 | 171.26 | 171.270 |
0.0 | 1.0 | 170.00 | 172.68 | 168.60 | 168.845 |
0.0 | 1.0 | 168.39 | 169.92 | 164.94 | 164.940 |
0.0 | 1.0 | 168.07 | 173.10 | 166.44 | 172.770 |
0.0 | 1.0 | 173.68 | 175.15 | 166.92 | 168.340 |
Adj. Volume | 20d | 50d | 200d | 20d-50d |
---|---|---|---|---|
35247358.0 | 176.94 | 172.57 | 162.68 | 4.37 |
41051076.0 | 176.76 | 172.46 | 162.75 | 4.30 |
40248954.0 | 176.23 | 172.27 | 162.81 | 3.96 |
36272617.0 | 175.92 | 172.22 | 162.91 | 3.70 |
38962839.0 | 175.41 | 172.05 | 162.98 | 3.36 |
We will refer to the sign of this difference as the regime; that is, if the fast moving average is above the slow moving average, this is a bullish regime (the bulls rule), and a bearish regime (the bears rule) holds when the fast moving average is below the slow moving average. I identify regimes with the following code.
# np.where() is a vectorized if-else function, where a condition is checked for each component of a vector, and the first argument passed is used when the condition holds, and the other passed if it does not apple["Regime"] = np.where(apple['20d-50d'] > 0, 1, 0) # We have 1's for bullish regimes and 0's for everything else. Below I replace bearish regimes's values with -1, and to maintain the rest of the vector, the second argument is apple["Regime"] apple["Regime"] = np.where(apple['20d-50d'] < 0, -1, apple["Regime"]) apple.loc['2016-01-04':'2016-12-31',"Regime"].plot(ylim = (-2,2)).axhline(y = 0, color = "black", lw = 2)
apple["Regime"].plot(ylim = (-2,2)).axhline(y = 0, color = "black", lw = 2)
apple["Regime"].value_counts()
1 1323 -1 694 0 53 Name: Regime, dtype: int64
The last line above indicates that for 1005 days the market was bearish on Apple, while for 600 days the market was bullish, and it was neutral for 54 days.
Trading signals appear at regime changes. When a bullish regime begins, a buy signal is triggered, and when it ends, a sell signal is triggered. Likewise, when a bearish regime begins, a sell signal is triggered, and when the regime ends, a buy signal is triggered (this is of interest only if you ever will short the stock, or use some derivative like a stock option to bet against the market).
It's simple to obtain signals. Let indicate the regime at time , and the signal at time . Then:
, with indicating "sell", indicating "buy", and no action. We can obtain signals like so:
# To ensure that all trades close out, I temporarily change the regime of the last row to 0 regime_orig = apple.loc[:, "Regime"].iloc[-1] apple.loc[:, "Regime"].iloc[-1] = 0 apple["Signal"] = np.sign(apple["Regime"] - apple["Regime"].shift(1)) # Restore original regime data apple.loc[:, "Regime"].iloc[-1] = regime_orig apple.tail()
/home/curtis/anaconda3/lib/python3.6/site-packages/pandas/core/indexing.py:194: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy self._setitem_with_indexer(indexer, value)
Open | High | Low | Close | Volume | |
---|---|---|---|---|---|
Date | |||||
2018-03-21 | 175.04 | 175.09 | 171.26 | 171.270 | 35247358.0 |
2018-03-22 | 170.00 | 172.68 | 168.60 | 168.845 | 41051076.0 |
2018-03-23 | 168.39 | 169.92 | 164.94 | 164.940 | 40248954.0 |
2018-03-26 | 168.07 | 173.10 | 166.44 | 172.770 | 36272617.0 |
2018-03-27 | 173.68 | 175.15 | 166.92 | 168.340 | 38962839.0 |
Ex-Dividend | Split Ratio | Adj. Open | Adj. High | Adj. Low | Adj. Close |
---|---|---|---|---|---|
0.0 | 1.0 | 175.04 | 175.09 | 171.26 | 171.270 |
0.0 | 1.0 | 170.00 | 172.68 | 168.60 | 168.845 |
0.0 | 1.0 | 168.39 | 169.92 | 164.94 | 164.940 |
0.0 | 1.0 | 168.07 | 173.10 | 166.44 | 172.770 |
0.0 | 1.0 | 173.68 | 175.15 | 166.92 | 168.340 |
Adj. Volume | 20d | 50d | 200d | 20d-50d | Regime | Signal |
---|---|---|---|---|---|---|
35247358.0 | 176.94 | 172.57 | 162.68 | 4.37 | 1 | 0.0 |
41051076.0 | 176.76 | 172.46 | 162.75 | 4.30 | 1 | 0.0 |
40248954.0 | 176.23 | 172.27 | 162.81 | 3.96 | 1 | 0.0 |
36272617.0 | 175.92 | 172.22 | 162.91 | 3.70 | 1 | 0.0 |
38962839.0 | 175.41 | 172.05 | 162.98 | 3.36 | 1 | -1.0 |
apple["Signal"].plot(ylim = (-2, 2))
apple["Signal"].value_counts()
0.0 2014 -1.0 28 1.0 27 Name: Signal, dtype: int64
We would buy Apple stock 23 times and sell Apple stock 23 times. If we only go long on Apple stock, only 23 trades will be engaged in over the 6-year period, while if we pivot from a long to a short position every time a long position is terminated, we would engage in 23 trades total. (Bear in mind that trading more frequently isn’t necessarily good; trades are never free.)
You may notice that the system as it currently stands isn’t very robust, since even a fleeting moment when the fast moving average is above the slow moving average triggers a trade, resulting in trades that end immediately (which is bad if not simply because realistically every trade is accompanied by a fee that can quickly erode earnings). Additionally, every bullish regime immediately transitions into a bearish regime, and if you were constructing trading systems that allow both bullish and bearish bets, this would lead to the end of one trade immediately triggering a new trade that bets on the market in the opposite direction, which again seems finnicky. A better system would require more evidence that the market is moving in some particular direction. But we will not concern ourselves with these details for now.
Let’s now try to identify what the prices of the stock is at every buy and every sell.
apple.loc[apple["Signal"] == 1, "Close"]
Date 2010-03-16 224.450 2010-06-18 274.074 2010-08-16 247.640 2010-09-20 283.230 2011-05-12 346.570 2011-07-14 357.770 2011-12-28 402.640 2012-06-25 570.765 2013-05-17 433.260 2013-07-31 452.530 2013-10-16 501.114 2014-03-11 536.090 2014-03-12 536.610 2014-03-24 539.190 2014-04-25 571.940 2014-10-28 106.740 2015-02-05 119.940 2015-04-28 130.560 2015-10-27 114.550 2016-03-10 101.170 2016-06-23 96.100 2016-06-30 95.600 2016-07-25 97.340 2016-12-21 117.060 2017-08-02 157.140 2017-11-01 166.890 2018-03-08 176.940 Name: Close, dtype: float64
apple.loc[apple["Signal"] == -1, "Close"]
Date 2010-06-11 253.5100 2010-07-22 259.0240 2010-08-17 251.9700 2011-03-30 348.6300 2011-03-31 348.5075 2011-05-27 337.4100 2011-11-17 377.4100 2012-05-09 569.1800 2012-10-17 644.6136 2013-06-26 398.0700 2013-10-04 483.0300 2014-01-28 506.5000 2014-03-17 526.7400 2014-04-22 531.6990 2014-10-17 97.6700 2015-01-05 106.2500 2015-04-16 126.1700 2015-06-25 127.5000 2015-06-26 126.7500 2015-12-18 106.0300 2016-05-05 93.2400 2016-06-27 92.0400 2016-07-11 96.9800 2016-11-15 107.1100 2017-06-27 143.7400 2017-10-03 154.4800 2018-02-06 163.0300 2018-03-27 168.3400 Name: Close, dtype: float64
# Create a DataFrame with trades, including the price at the trade and the regime under which the trade is made. apple_signals = pd.concat([ pd.DataFrame({"Price": apple.loc[apple["Signal"] == 1, "Adj. Close"], "Regime": apple.loc[apple["Signal"] == 1, "Regime"], "Signal": "Buy"}), pd.DataFrame({"Price": apple.loc[apple["Signal"] == -1, "Adj. Close"], "Regime": apple.loc[apple["Signal"] == -1, "Regime"], "Signal": "Sell"}), ]) apple_signals.sort_index(inplace = True) apple_signals
Price | Regime | Signal | |
---|---|---|---|
Date | |||
2010-03-16 | 28.844953 | 1 | Buy |
2010-06-11 | 32.579568 | -1 | Sell |
2010-06-18 | 35.222329 | 1 | Buy |
2010-07-22 | 33.288194 | -1 | Sell |
2010-08-16 | 31.825192 | 0 | Buy |
2010-08-17 | 32.381657 | -1 | Sell |
2010-09-20 | 36.399003 | 1 | Buy |
2011-03-30 | 44.803814 | 0 | Sell |
2011-03-31 | 44.788071 | -1 | Sell |
2011-05-12 | 44.539075 | 1 | Buy |
2011-05-27 | 43.361888 | -1 | Sell |
2011-07-14 | 45.978431 | 1 | Buy |
2011-11-17 | 48.502445 | -1 | Sell |
2011-12-28 | 51.744852 | 1 | Buy |
2012-05-09 | 73.147563 | -1 | Sell |
2012-06-25 | 73.351258 | 1 | Buy |
2012-10-17 | 83.195498 | -1 | Sell |
2013-05-17 | 56.878472 | 1 | Buy |
2013-06-26 | 52.258721 | -1 | Sell |
2013-07-31 | 59.408242 | 1 | Buy |
2013-10-04 | 63.831819 | -1 | Sell |
2013-10-16 | 66.221597 | 1 | Buy |
2014-01-28 | 67.325247 | -1 | Sell |
2014-03-11 | 71.682490 | 0 | Buy |
2014-03-12 | 71.752021 | 1 | Buy |
2014-03-17 | 70.432269 | -1 | Sell |
2014-03-24 | 72.097002 | 1 | Buy |
2014-04-22 | 71.095354 | -1 | Sell |
2014-04-25 | 76.476120 | 1 | Buy |
2014-10-17 | 92.387441 | -1 | Sell |
2014-10-28 | 100.966883 | 1 | Buy |
2015-01-05 | 100.937944 | -1 | Sell |
2015-02-05 | 114.390004 | 1 | Buy |
2015-04-16 | 120.331722 | -1 | Sell |
2015-04-28 | 124.518583 | 1 | Buy |
2015-06-25 | 122.104986 | 0 | Sell |
2015-06-26 | 121.386721 | -1 | Sell |
2015-10-27 | 110.198438 | 1 | Buy |
2015-12-18 | 102.440744 | -1 | Sell |
2016-03-10 | 98.271427 | 1 | Buy |
2016-05-05 | 91.122295 | -1 | Sell |
2016-06-23 | 93.917337 | 1 | Buy |
2016-06-27 | 89.949550 | -1 | Sell |
2016-06-30 | 93.428693 | 1 | Buy |
2016-07-11 | 94.777350 | -1 | Sell |
2016-07-25 | 95.129174 | 1 | Buy |
2016-11-15 | 105.787035 | -1 | Sell |
2016-12-21 | 115.614138 | 1 | Buy |
2017-06-27 | 143.159139 | -1 | Sell |
2017-08-02 | 156.504989 | 1 | Buy |
2017-10-03 | 154.480000 | -1 | Sell |
2017-11-01 | 166.890000 | 1 | Buy |
2018-02-06 | 163.030000 | -1 | Sell |
2018-03-08 | 176.940000 | 1 | Buy |
2018-03-27 | 168.340000 | 1 | Sell |
# Let's see the profitability of long trades apple_long_profits = pd.DataFrame({ "Price": apple_signals.loc[(apple_signals["Signal"] == "Buy") & apple_signals["Regime"] == 1, "Price"], "Profit": pd.Series(apple_signals["Price"] - apple_signals["Price"].shift(1)).loc[ apple_signals.loc[(apple_signals["Signal"].shift(1) == "Buy") & (apple_signals["Regime"].shift(1) == 1)].index ].tolist(), "End Date": apple_signals["Price"].loc[ apple_signals.loc[(apple_signals["Signal"].shift(1) == "Buy") & (apple_signals["Regime"].shift(1) == 1)].index ].index }) apple_long_profits
End Date | Price | Profit | |
---|---|---|---|
Date | |||
2010-03-16 | 2010-06-11 | 28.844953 | 3.734615 |
2010-06-18 | 2010-07-22 | 35.222329 | -1.934135 |
2010-09-20 | 2011-03-30 | 36.399003 | 8.404812 |
2011-05-12 | 2011-05-27 | 44.539075 | -1.177188 |
2011-07-14 | 2011-11-17 | 45.978431 | 2.524014 |
2011-12-28 | 2012-05-09 | 51.744852 | 21.402711 |
2012-06-25 | 2012-10-17 | 73.351258 | 9.844240 |
2013-05-17 | 2013-06-26 | 56.878472 | -4.619751 |
2013-07-31 | 2013-10-04 | 59.408242 | 4.423577 |
2013-10-16 | 2014-01-28 | 66.221597 | 1.103650 |
2014-03-12 | 2014-03-17 | 71.752021 | -1.319753 |
2014-03-24 | 2014-04-22 | 72.097002 | -1.001648 |
2014-04-25 | 2014-10-17 | 76.476120 | 15.911321 |
2014-10-28 | 2015-01-05 | 100.966883 | -0.028939 |
2015-02-05 | 2015-04-16 | 114.390004 | 5.941719 |
2015-04-28 | 2015-06-25 | 124.518583 | -2.413598 |
2015-10-27 | 2015-12-18 | 110.198438 | -7.757693 |
2016-03-10 | 2016-05-05 | 98.271427 | -7.149132 |
2016-06-23 | 2016-06-27 | 93.917337 | -3.967788 |
2016-06-30 | 2016-07-11 | 93.428693 | 1.348657 |
2016-07-25 | 2016-11-15 | 95.129174 | 10.657861 |
2016-12-21 | 2017-06-27 | 115.614138 | 27.545001 |
2017-08-02 | 2017-10-03 | 156.504989 | -2.024989 |
2017-11-01 | 2018-02-06 | 166.890000 | -3.860000 |
2018-03-08 | 2018-03-27 | 176.940000 | -8.600000 |
Let’s now create a simulated portfolio of $1,000,000, and see how it would behave, according to the rules we have established. This includes:
- Investing only 10% of the portfolio in any trade
- Exiting the position if losses exceed 20% of the value of the trade.
When simulating, bear in mind that:
- Trades are done in batches of 100 stocks.
- Our stop-loss rule involves placing an order to sell the stock the moment the price drops below the specified level. Thus we need to check whether the lows during this period ever go low enough to trigger the stop-loss. Realistically, unless we buy a put option, we cannot guarantee that we will sell the stock at the price we set at the stop-loss, but we will use this as the selling price anyway for the sake of simplicity.
- Every trade is accompanied by a commission to the broker, which should be accounted for. I do not do so here.
Here’s how a backtest may look:
# We need to get the low of the price during each trade. tradeperiods = pd.DataFrame({"Start": apple_long_profits.index, "End": apple_long_profits["End Date"]}) apple_long_profits["Low"] = tradeperiods.apply(lambda x: min(apple.loc[x["Start"]:x["End"], "Adj. Low"]), axis = 1) apple_long_profits
End Date | Price | Profit | Low | |
---|---|---|---|---|
Date | ||||
2010-03-16 | 2010-06-11 | 28.844953 | 3.734615 | 25.606402 |
2010-06-18 | 2010-07-22 | 35.222329 | -1.934135 | 30.791939 |
2010-09-20 | 2011-03-30 | 36.399003 | 8.404812 | 35.341333 |
2011-05-12 | 2011-05-27 | 44.539075 | -1.177188 | 42.335061 |
2011-07-14 | 2011-11-17 | 45.978431 | 2.524014 | 45.367990 |
2011-12-28 | 2012-05-09 | 51.744852 | 21.402711 | 51.471117 |
2012-06-25 | 2012-10-17 | 73.351258 | 9.844240 | 72.688768 |
2013-05-17 | 2013-06-26 | 56.878472 | -4.619751 | 51.942335 |
2013-07-31 | 2013-10-04 | 59.408242 | 4.423577 | 59.001273 |
2013-10-16 | 2014-01-28 | 66.221597 | 1.103650 | 65.972629 |
2014-03-12 | 2014-03-17 | 71.752021 | -1.319753 | 69.932180 |
2014-03-24 | 2014-04-22 | 72.097002 | -1.001648 | 68.371743 |
2014-04-25 | 2014-10-17 | 76.476120 | 15.911321 | 75.409086 |
2014-10-28 | 2015-01-05 | 100.966883 | -0.028939 | 99.652062 |
2015-02-05 | 2015-04-16 | 114.390004 | 5.941719 | 112.949876 |
2015-04-28 | 2015-06-25 | 124.518583 | -2.413598 | 117.651750 |
2015-10-27 | 2015-12-18 | 110.198438 | -7.757693 | 102.228192 |
2016-03-10 | 2016-05-05 | 98.271427 | -7.149132 | 89.752692 |
2016-06-23 | 2016-06-27 | 93.917337 | -3.967788 | 89.421814 |
2016-06-30 | 2016-07-11 | 93.428693 | 1.348657 | 92.158220 |
2016-07-25 | 2016-11-15 | 95.129174 | 10.657861 | 94.230069 |
2016-12-21 | 2017-06-27 | 115.614138 | 27.545001 | 113.342546 |
2017-08-02 | 2017-10-03 | 156.504989 | -2.024989 | 149.160000 |
2017-11-01 | 2018-02-06 | 166.890000 | -3.860000 | 154.000000 |
2018-03-08 | 2018-03-27 | 176.940000 | -8.600000 | 164.940000 |
# Now we have all the information needed to simulate this strategy in apple_adj_long_profits cash = 1000000 apple_backtest = pd.DataFrame({"Start Port. Value": [], "End Port. Value": [], "End Date": [], "Shares": [], "Share Price": [], "Trade Value": [], "Profit per Share": [], "Total Profit": [], "Stop-Loss Triggered": []}) port_value = .1 # Max proportion of portfolio bet on any trade batch = 100 # Number of shares bought per batch stoploss = .2 # % of trade loss that would trigger a stoploss for index, row in apple_long_profits.iterrows(): batches = np.floor(cash * port_value) // np.ceil(batch * row["Price"]) # Maximum number of batches of stocks invested in trade_val = batches * batch * row["Price"] # How much money is put on the line with each trade if row["Low"] < (1 - stoploss) * row["Price"]: # Account for the stop-loss share_profit = np.round((1 - stoploss) * row["Price"], 2) stop_trig = True else: share_profit = row["Profit"] stop_trig = False profit = share_profit * batches * batch # Compute profits # Add a row to the backtest data frame containing the results of the trade apple_backtest = apple_backtest.append(pd.DataFrame({ "Start Port. Value": cash, "End Port. Value": cash + profit, "End Date": row["End Date"], "Shares": batch * batches, "Share Price": row["Price"], "Trade Value": trade_val, "Profit per Share": share_profit, "Total Profit": profit, "Stop-Loss Triggered": stop_trig }, index = [index])) cash = max(0, cash + profit) apple_backtest
End Date | End Port. Value | Profit per Share | Share Price | Shares | |
---|---|---|---|---|---|
2010-03-16 | 2010-06-11 | 1.012698e+06 | 3.734615 | 28.844953 | 3400.0 |
2010-06-18 | 2010-07-22 | 1.007282e+06 | -1.934135 | 35.222329 | 2800.0 |
2010-09-20 | 2011-03-30 | 1.029975e+06 | 8.404812 | 36.399003 | 2700.0 |
2011-05-12 | 2011-05-27 | 1.027268e+06 | -1.177188 | 44.539075 | 2300.0 |
2011-07-14 | 2011-11-17 | 1.032820e+06 | 2.524014 | 45.978431 | 2200.0 |
2011-12-28 | 2012-05-09 | 1.073486e+06 | 21.402711 | 51.744852 | 1900.0 |
2012-06-25 | 2012-10-17 | 1.087267e+06 | 9.844240 | 73.351258 | 1400.0 |
2013-05-17 | 2013-06-26 | 1.078490e+06 | -4.619751 | 56.878472 | 1900.0 |
2013-07-31 | 2013-10-04 | 1.086452e+06 | 4.423577 | 59.408242 | 1800.0 |
2013-10-16 | 2014-01-28 | 1.088218e+06 | 1.103650 | 66.221597 | 1600.0 |
2014-03-12 | 2014-03-17 | 1.086239e+06 | -1.319753 | 71.752021 | 1500.0 |
2014-03-24 | 2014-04-22 | 1.084736e+06 | -1.001648 | 72.097002 | 1500.0 |
2014-04-25 | 2014-10-17 | 1.107012e+06 | 15.911321 | 76.476120 | 1400.0 |
2014-10-28 | 2015-01-05 | 1.106983e+06 | -0.028939 | 100.966883 | 1000.0 |
2015-02-05 | 2015-04-16 | 1.112331e+06 | 5.941719 | 114.390004 | 900.0 |
2015-04-28 | 2015-06-25 | 1.110400e+06 | -2.413598 | 124.518583 | 800.0 |
2015-10-27 | 2015-12-18 | 1.102642e+06 | -7.757693 | 110.198438 | 1000.0 |
2016-03-10 | 2016-05-05 | 1.094778e+06 | -7.149132 | 98.271427 | 1100.0 |
2016-06-23 | 2016-06-27 | 1.090413e+06 | -3.967788 | 93.917337 | 1100.0 |
2016-06-30 | 2016-07-11 | 1.091897e+06 | 1.348657 | 93.428693 | 1100.0 |
2016-07-25 | 2016-11-15 | 1.103621e+06 | 10.657861 | 95.129174 | 1100.0 |
2016-12-21 | 2017-06-27 | 1.128411e+06 | 27.545001 | 115.614138 | 900.0 |
2017-08-02 | 2017-10-03 | 1.126994e+06 | -2.024989 | 156.504989 | 700.0 |
2017-11-01 | 2018-02-06 | 1.124678e+06 | -3.860000 | 166.890000 | 600.0 |
2018-03-08 | 2018-03-27 | 1.119518e+06 | -8.600000 | 176.940000 | 600.0 |
Start Port. Value | Stop-Loss Triggered | Total Profit | Trade Value |
---|---|---|---|
1.000000e+06 | 0.0 | 12697.691096 | 98072.841239 |
1.012698e+06 | 0.0 | -5415.577333 | 98622.521053 |
1.007282e+06 | 0.0 | 22692.991110 | 98277.306914 |
1.029975e+06 | 0.0 | -2707.531638 | 102439.873355 |
1.027268e+06 | 0.0 | 5552.830218 | 101152.549241 |
1.032820e+06 | 0.0 | 40665.151235 | 98315.218526 |
1.073486e+06 | 0.0 | 13781.935982 | 102691.760672 |
1.087267e+06 | 0.0 | -8777.527400 | 108069.096937 |
1.078490e+06 | 0.0 | 7962.438409 | 106934.835757 |
1.086452e+06 | 0.0 | 1765.839598 | 105954.555657 |
1.088218e+06 | 0.0 | -1979.628917 | 107628.031714 |
1.086239e+06 | 0.0 | -1502.472160 | 108145.503103 |
1.084736e+06 | 0.0 | 22275.849051 | 107066.568572 |
1.107012e+06 | 0.0 | -28.938709 | 100966.883069 |
1.106983e+06 | 0.0 | 5347.546691 | 102951.003221 |
1.112331e+06 | 0.0 | -1930.878038 | 99614.866549 |
1.110400e+06 | 0.0 | -7757.693367 | 110198.437846 |
1.102642e+06 | 0.0 | -7864.045388 | 108098.569555 |
1.094778e+06 | 0.0 | -4364.566368 | 103309.070918 |
1.090413e+06 | 0.0 | 1483.522558 | 102771.562745 |
1.091897e+06 | 0.0 | 11723.647322 | 104642.091188 |
1.103621e+06 | 0.0 | 24790.501098 | 104052.724175 |
1.128411e+06 | 0.0 | -1417.492367 | 109553.492367 |
1.126994e+06 | 0.0 | -2316.000000 | 100134.000000 |
1.124678e+06 | 0.0 | -5160.000000 | 106164.000000 |
apple_backtest["End Port. Value"].plot()
Our portfolio’s value grew by 13% in about six years. Considering that only 10% of the portfolio was ever involved in any single trade, this is not bad performance.
Notice that this strategy never lead to our rule of never allowing losses to exceed 20% of the trade’s value being invoked. For the sake of simplicity, we will ignore this rule in backtesting.
A more realistic portfolio would not be betting 10% of its value on only one stock. A more realistic one would consider investing in multiple stocks. Multiple trades may be ongoing at any given time involving multiple companies, and most of the portfolio will be in stocks, not cash. Now that we will be investing in multiple stops and exiting only when moving averages cross (not because of a stop-loss), we will need to change our approach to backtesting. For example, we will be using one pandas DataFrame
to contain all buy and sell orders for all stocks being considered, and our loop above will have to track more information.
I have written functions for creating order data for multiple stocks, and a function for performing the backtesting.
def ma_crossover_orders(stocks, fast, slow): """ :param stocks: A list of tuples, the first argument in each tuple being a string containing the ticker symbol of each stock (or however you want the stock represented, so long as it's unique), and the second being a pandas DataFrame containing the stocks, with a "Close" column and indexing by date (like the data frames returned by the Yahoo! Finance API) :param fast: Integer for the number of days used in the fast moving average :param slow: Integer for the number of days used in the slow moving average :return: pandas DataFrame containing stock orders This function takes a list of stocks and determines when each stock would be bought or sold depending on a moving average crossover strategy, returning a data frame with information about when the stocks in the portfolio are bought or sold according to the strategy """ fast_str = str(fast) + 'd' slow_str = str(slow) + 'd' ma_diff_str = fast_str + '-' + slow_str trades = pd.DataFrame({"Price": [], "Regime": [], "Signal": []}) for s in stocks: # Get the moving averages, both fast and slow, along with the difference in the moving averages s[1][fast_str] = np.round(s[1]["Close"].rolling(window = fast, center = False).mean(), 2) s[1][slow_str] = np.round(s[1]["Close"].rolling(window = slow, center = False).mean(), 2) s[1][ma_diff_str] = s[1][fast_str] - s[1][slow_str] # np.where() is a vectorized if-else function, where a condition is checked for each component of a vector, and the first argument passed is used when the condition holds, and the other passed if it does not s[1]["Regime"] = np.where(s[1][ma_diff_str] > 0, 1, 0) # We have 1's for bullish regimes and 0's for everything else. Below I replace bearish regimes's values with -1, and to maintain the rest of the vector, the second argument is apple["Regime"] s[1]["Regime"] = np.where(s[1][ma_diff_str] < 0, -1, s[1]["Regime"]) # To ensure that all trades close out, I temporarily change the regime of the last row to 0 regime_orig = s[1].loc[:, "Regime"].iloc[-1] s[1].loc[:, "Regime"].iloc[-1] = 0 s[1]["Signal"] = np.sign(s[1]["Regime"] - s[1]["Regime"].shift(1)) # Restore original regime data s[1].loc[:, "Regime"].iloc[-1] = regime_orig # Get signals signals = pd.concat([ pd.DataFrame({"Price": s[1].loc[s[1]["Signal"] == 1, "Adj. Close"], "Regime": s[1].loc[s[1]["Signal"] == 1, "Regime"], "Signal": "Buy"}), pd.DataFrame({"Price": s[1].loc[s[1]["Signal"] == -1, "Adj. Close"], "Regime": s[1].loc[s[1]["Signal"] == -1, "Regime"], "Signal": "Sell"}), ]) signals.index = pd.MultiIndex.from_product([signals.index, [s[0]]], names = ["Date", "Symbol"]) trades = trades.append(signals) trades.sort_index(inplace = True) trades.index = pd.MultiIndex.from_tuples(trades.index, names = ["Date", "Symbol"]) return trades def backtest(signals, cash, port_value = .1, batch = 100): """ :param signals: pandas DataFrame containing buy and sell signals with stock prices and symbols, like that returned by ma_crossover_orders :param cash: integer for starting cash value :param port_value: maximum proportion of portfolio to risk on any single trade :param batch: Trading batch sizes :return: pandas DataFrame with backtesting results This function backtests strategies, with the signals generated by the strategies being passed in the signals DataFrame. A fictitious portfolio is simulated and the returns generated by this portfolio are reported. """ SYMBOL = 1 # Constant for which element in index represents symbol portfolio = dict() # Will contain how many stocks are in the portfolio for a given symbol port_prices = dict() # Tracks old trade prices for determining profits # Dataframe that will contain backtesting report results = pd.DataFrame({"Start Cash": [], "End Cash": [], "Portfolio Value": [], "Type": [], "Shares": [], "Share Price": [], "Trade Value": [], "Profit per Share": [], "Total Profit": []}) for index, row in signals.iterrows(): # These first few lines are done for any trade shares = portfolio.setdefault(index[SYMBOL], 0) trade_val = 0 batches = 0 cash_change = row["Price"] * shares # Shares could potentially be a positive or negative number (cash_change will be added in the end; negative shares indicate a short) portfolio[index[SYMBOL]] = 0 # For a given symbol, a position is effectively cleared old_price = port_prices.setdefault(index[SYMBOL], row["Price"]) portfolio_val = 0 for key, val in portfolio.items(): portfolio_val += val * port_prices[key] if row["Signal"] == "Buy" and row["Regime"] == 1: # Entering a long position batches = np.floor((portfolio_val + cash) * port_value) // np.ceil(batch * row["Price"]) # Maximum number of batches of stocks invested in trade_val = batches * batch * row["Price"] # How much money is put on the line with each trade cash_change -= trade_val # We are buying shares so cash will go down portfolio[index[SYMBOL]] = batches * batch # Recording how many shares are currently invested in the stock port_prices[index[SYMBOL]] = row["Price"] # Record price old_price = row["Price"] elif row["Signal"] == "Sell" and row["Regime"] == -1: # Entering a short pass # Do nothing; can we provide a method for shorting the market? #else: #raise ValueError("I don't know what to do with signal " + row["Signal"]) pprofit = row["Price"] - old_price # Compute profit per share; old_price is set in such a way that entering a position results in a profit of zero # Update report results = results.append(pd.DataFrame({ "Start Cash": cash, "End Cash": cash + cash_change, "Portfolio Value": cash + cash_change + portfolio_val + trade_val, "Type": row["Signal"], "Shares": batch * batches, "Share Price": row["Price"], "Trade Value": abs(cash_change), "Profit per Share": pprofit, "Total Profit": batches * batch * pprofit }, index = [index])) cash += cash_change # Final change to cash balance results.sort_index(inplace = True) results.index = pd.MultiIndex.from_tuples(results.index, names = ["Date", "Symbol"]) return results # Get more stocks (microsoft, google, facebook, twitter, netflix, amazon, yahoo, ge, qualcomm, ibm, hp) = (quandl.get("WIKI/" + s, start_date=start, end_date=end) for s in ["MSFT", "GOOG", "FB", "TWTR", "NFLX", "AMZN", "YHOO", "GE", "QCOM", "IBM", "HPQ"])
signals = ma_crossover_orders([("AAPL", apple), ("MSFT", microsoft), ("GOOG", google), ("FB", facebook), ("TWTR", twitter), ("NFLX", netflix), ("AMZN", amazon), ("YHOO", yahoo), ("GE", ge), ("QCOM", qualcomm), ("IBM", ibm), ("HPQ", hp)], fast = 20, slow = 50) signals
/home/curtis/anaconda3/lib/python3.6/site-packages/pandas/core/indexing.py:194: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy self._setitem_with_indexer(indexer, value)
Price | Regime | Signal | ||
---|---|---|---|---|
Date | Symbol | |||
2010-03-16 | AAPL | 28.844953 | 1.0 | Buy |
AMZN | 131.790000 | 1.0 | Buy | |
GE | 14.129260 | 1.0 | Buy | |
HPQ | 19.921951 | 1.0 | Buy | |
IBM | 105.460506 | 1.0 | Buy | |
MSFT | 23.978839 | -1.0 | Sell | |
NFLX | 10.090000 | 1.0 | Buy | |
QCOM | 32.235226 | -1.0 | Sell | |
YHOO | 16.360000 | -1.0 | Sell | |
2010-03-17 | YHOO | 16.500000 | 1.0 | Buy |
2010-03-24 | MSFT | 24.207442 | 1.0 | Buy |
2010-04-01 | QCOM | 34.929069 | 1.0 | Buy |
2010-05-07 | QCOM | 30.161131 | -1.0 | Sell |
2010-05-10 | HPQ | 18.684203 | -1.0 | Sell |
2010-05-17 | YHOO | 16.270000 | -1.0 | Sell |
2010-05-19 | AMZN | 124.590000 | -1.0 | Sell |
GE | 13.495907 | -1.0 | Sell | |
MSFT | 23.161072 | -1.0 | Sell | |
2010-05-20 | IBM | 102.001194 | -1.0 | Sell |
2010-06-11 | AAPL | 32.579568 | -1.0 | Sell |
2010-06-18 | AAPL | 35.222329 | 1.0 | Buy |
2010-06-29 | IBM | 103.064049 | 1.0 | Buy |
2010-06-30 | IBM | 101.737540 | -1.0 | Sell |
2010-07-07 | IBM | 104.637735 | 1.0 | Buy |
2010-07-20 | IBM | 104.266971 | -1.0 | Sell |
2010-07-22 | AAPL | 33.288194 | -1.0 | Sell |
2010-07-27 | QCOM | 32.585294 | 1.0 | Buy |
2010-07-28 | IBM | 105.815940 | 1.0 | Buy |
2010-07-29 | NFLX | 14.002857 | -1.0 | Sell |
2010-08-02 | HPQ | 18.129988 | 1.0 | Buy |
… | … | … | … | … |
2017-11-01 | AAPL | 166.890000 | 1.0 | Buy |
2017-12-06 | NFLX | 185.300000 | -1.0 | Sell |
2017-12-15 | HPQ | 20.920000 | -1.0 | Sell |
2017-12-26 | FB | 175.990000 | -1.0 | Sell |
2018-01-03 | FB | 184.670000 | 1.0 | Buy |
2018-01-09 | NFLX | 209.310000 | 1.0 | Buy |
2018-01-11 | HPQ | 22.410000 | 1.0 | Buy |
2018-01-18 | QCOM | 68.050000 | -1.0 | Sell |
2018-01-19 | QCOM | 68.040000 | 1.0 | Buy |
2018-02-06 | AAPL | 163.030000 | -1.0 | Sell |
2018-02-21 | IBM | 153.960000 | -1.0 | Sell |
QCOM | 63.400000 | -1.0 | Sell | |
2018-02-22 | HPQ | 21.390000 | -1.0 | Sell |
2018-02-23 | FB | 183.290000 | -1.0 | Sell |
2018-02-27 | GOOG | 1118.290000 | -1.0 | Sell |
2018-03-08 | AAPL | 176.940000 | 1.0 | Buy |
2018-03-09 | HPQ | 24.650000 | 1.0 | Buy |
2018-03-14 | GOOG | 1149.490000 | 1.0 | Buy |
2018-03-23 | GOOG | 1021.570000 | -1.0 | Sell |
2018-03-27 | AAPL | 168.340000 | 1.0 | Sell |
AMZN | 1497.050000 | 1.0 | Sell | |
FB | 152.190000 | -1.0 | Buy | |
GE | 13.440000 | -1.0 | Buy | |
GOOG | 1005.100000 | -1.0 | Buy | |
HPQ | 21.770000 | 1.0 | Sell | |
IBM | 151.910000 | -1.0 | Buy | |
MSFT | 89.470000 | 1.0 | Sell | |
NFLX | 300.690000 | 1.0 | Sell | |
QCOM | 54.840000 | -1.0 | Buy | |
TWTR | 28.070000 | 1.0 | Sell |
511 rows × 3 columns
bk = backtest(signals, 1000000) bk
End Cash | Portfolio Value | Profit per Share | ||
---|---|---|---|---|
Date | Symbol | |||
2010-03-16 | AAPL | 9.019272e+05 | 1.000000e+06 | 0.000000 |
AMZN | 8.096742e+05 | 1.000000e+06 | 0.000000 | |
GE | 7.107693e+05 | 1.000000e+06 | 0.000000 | |
HPQ | 6.111596e+05 | 1.000000e+06 | 0.000000 | |
IBM | 5.162451e+05 | 1.000000e+06 | 0.000000 | |
MSFT | 5.162451e+05 | 1.000000e+06 | 0.000000 | |
NFLX | 4.163541e+05 | 1.000000e+06 | 0.000000 | |
QCOM | 4.163541e+05 | 1.000000e+06 | 0.000000 | |
YHOO | 4.163541e+05 | 1.000000e+06 | 0.000000 | |
2010-03-17 | YHOO | 3.173541e+05 | 1.000000e+06 | 0.000000 |
2010-03-24 | MSFT | 2.181036e+05 | 1.000000e+06 | 0.000000 |
2010-04-01 | QCOM | 1.203022e+05 | 1.000000e+06 | 0.000000 |
2010-05-07 | QCOM | 2.047534e+05 | 9.866498e+05 | -4.767938 |
2010-05-10 | HPQ | 2.981744e+05 | 9.804610e+05 | -1.237749 |
2010-05-17 | YHOO | 3.957944e+05 | 9.790810e+05 | -0.230000 |
2010-05-19 | AMZN | 4.830074e+05 | 9.740410e+05 | -7.200000 |
GE | 5.774787e+05 | 9.696076e+05 | -0.633354 | |
MSFT | 6.724391e+05 | 9.653174e+05 | -1.046370 | |
2010-05-20 | IBM | 7.642402e+05 | 9.622041e+05 | -3.459312 |
2010-06-11 | AAPL | 8.750107e+05 | 9.749017e+05 | 3.734615 |
2010-06-18 | AAPL | 7.799105e+05 | 9.749017e+05 | 0.000000 |
2010-06-29 | IBM | 6.871528e+05 | 9.749017e+05 | 0.000000 |
2010-06-30 | IBM | 7.787166e+05 | 9.737079e+05 | -1.326510 |
2010-07-07 | IBM | 6.845426e+05 | 9.737079e+05 | 0.000000 |
2010-07-20 | IBM | 7.783829e+05 | 9.733742e+05 | -0.370764 |
2010-07-22 | AAPL | 8.682610e+05 | 9.681520e+05 | -1.934135 |
2010-07-27 | QCOM | 7.737637e+05 | 9.681520e+05 | 0.000000 |
2010-07-28 | IBM | 6.785293e+05 | 9.681520e+05 | 0.000000 |
2010-07-29 | NFLX | 8.171576e+05 | 1.006889e+06 | 3.912857 |
2010-08-02 | HPQ | 7.174427e+05 | 1.006889e+06 | 0.000000 |
… | … | … | … | … |
2017-11-01 | AAPL | 1.297792e+05 | 2.153164e+06 | 0.000000 |
2017-12-06 | NFLX | 3.336092e+05 | 2.149600e+06 | -3.240000 |
2017-12-15 | HPQ | 5.700052e+05 | 2.170395e+06 | 1.840267 |
2017-12-26 | FB | 8.339902e+05 | 2.244450e+06 | 49.370000 |
2018-01-03 | FB | 6.123862e+05 | 2.244450e+06 | 0.000000 |
2018-01-09 | NFLX | 4.030762e+05 | 2.244450e+06 | 0.000000 |
2018-01-11 | HPQ | 1.789762e+05 | 2.244450e+06 | 0.000000 |
2018-01-18 | QCOM | 4.511762e+05 | 2.301960e+06 | 14.377402 |
2018-01-19 | QCOM | 2.266442e+05 | 2.301960e+06 | 0.000000 |
2018-02-06 | AAPL | 4.222802e+05 | 2.297328e+06 | -3.860000 |
2018-02-21 | IBM | 6.378242e+05 | 2.309716e+06 | 8.848221 |
QCOM | 8.470442e+05 | 2.294404e+06 | -4.640000 | |
2018-02-22 | HPQ | 1.060944e+06 | 2.284204e+06 | -1.020000 |
2018-02-23 | FB | 1.280892e+06 | 2.282548e+06 | -1.380000 |
2018-02-27 | GOOG | 1.504550e+06 | 2.316306e+06 | 168.790000 |
2018-03-08 | AAPL | 1.274528e+06 | 2.316306e+06 | 0.000000 |
2018-03-09 | HPQ | 1.045283e+06 | 2.316306e+06 | 0.000000 |
2018-03-14 | GOOG | 8.153852e+05 | 2.316306e+06 | 0.000000 |
2018-03-23 | GOOG | 1.019699e+06 | 2.290722e+06 | -127.920000 |
2018-03-27 | AAPL | 1.238541e+06 | 2.279542e+06 | -8.600000 |
AMZN | 1.537951e+06 | 2.377684e+06 | 490.710000 | |
FB | 1.537951e+06 | 2.377684e+06 | -32.480000 | |
GE | 1.537951e+06 | 2.377684e+06 | -16.213194 | |
GOOG | 1.537951e+06 | 2.377684e+06 | -144.390000 | |
HPQ | 1.740412e+06 | 2.350900e+06 | -2.880000 | |
IBM | 1.740412e+06 | 2.350900e+06 | 6.798221 | |
MSFT | 2.026716e+06 | 2.451672e+06 | 31.491454 | |
NFLX | 2.327406e+06 | 2.543052e+06 | 91.380000 | |
QCOM | 2.327406e+06 | 2.543052e+06 | -13.200000 | |
TWTR | 2.683895e+06 | 2.683895e+06 | 11.090000 |
Share Price | Shares | Start Cash | Total Profit | Trade Value | Type |
---|---|---|---|---|---|
28.844953 | 3400.0 | 1.000000e+06 | 0.0 | 98072.841239 | Buy |
131.790000 | 700.0 | 9.019272e+05 | 0.0 | 92253.000000 | Buy |
14.129260 | 7000.0 | 8.096742e+05 | 0.0 | 98904.822860 | Buy |
19.921951 | 5000.0 | 7.107693e+05 | 0.0 | 99609.756314 | Buy |
105.460506 | 900.0 | 6.111596e+05 | 0.0 | 94914.455453 | Buy |
23.978839 | 0.0 | 5.162451e+05 | 0.0 | 0.000000 | Sell |
10.090000 | 9900.0 | 5.162451e+05 | 0.0 | 99891.000000 | Buy |
32.235226 | 0.0 | 4.163541e+05 | 0.0 | 0.000000 | Sell |
16.360000 | 0.0 | 4.163541e+05 | 0.0 | 0.000000 | Sell |
16.500000 | 6000.0 | 4.163541e+05 | 0.0 | 99000.000000 | Buy |
24.207442 | 4100.0 | 3.173541e+05 | 0.0 | 99250.512998 | Buy |
34.929069 | 2800.0 | 2.181036e+05 | 0.0 | 97801.393965 | Buy |
30.161131 | 0.0 | 1.203022e+05 | -0.0 | 84451.168198 | Sell |
18.684203 | 0.0 | 2.047534e+05 | -0.0 | 93421.012620 | Sell |
16.270000 | 0.0 | 2.981744e+05 | -0.0 | 97620.000000 | Sell |
124.590000 | 0.0 | 3.957944e+05 | -0.0 | 87213.000000 | Sell |
13.495907 | 0.0 | 4.830074e+05 | -0.0 | 94471.347126 | Sell |
23.161072 | 0.0 | 5.774787e+05 | -0.0 | 94960.396545 | Sell |
102.001194 | 0.0 | 6.724391e+05 | -0.0 | 91801.074363 | Sell |
32.579568 | 0.0 | 7.642402e+05 | 0.0 | 110770.532335 | Sell |
35.222329 | 2700.0 | 8.750107e+05 | 0.0 | 95100.288159 | Buy |
103.064049 | 900.0 | 7.799105e+05 | 0.0 | 92757.644524 | Buy |
101.737540 | 0.0 | 6.871528e+05 | -0.0 | 91563.785641 | Sell |
104.637735 | 900.0 | 7.787166e+05 | 0.0 | 94173.961584 | Buy |
104.266971 | 0.0 | 6.845426e+05 | -0.0 | 93840.274319 | Sell |
33.288194 | 0.0 | 7.783829e+05 | -0.0 | 89878.124302 | Sell |
32.585294 | 2900.0 | 8.682610e+05 | 0.0 | 94497.352787 | Buy |
105.815940 | 900.0 | 7.737637e+05 | 0.0 | 95234.345561 | Buy |
14.002857 | 0.0 | 6.785293e+05 | 0.0 | 138628.285714 | Sell |
18.129988 | 5500.0 | 8.171576e+05 | 0.0 | 99714.934419 | Buy |
… | … | … | … | … | … |
166.890000 | 1200.0 | 3.300472e+05 | 0.0 | 200268.000000 | Buy |
185.300000 | 0.0 | 1.297792e+05 | -0.0 | 203830.000000 | Sell |
20.920000 | 0.0 | 3.336092e+05 | 0.0 | 236396.000000 | Sell |
175.990000 | 0.0 | 5.700052e+05 | 0.0 | 263985.000000 | Sell |
184.670000 | 1200.0 | 8.339902e+05 | 0.0 | 221604.000000 | Buy |
209.310000 | 1000.0 | 6.123862e+05 | 0.0 | 209310.000000 | Buy |
22.410000 | 10000.0 | 4.030762e+05 | 0.0 | 224100.000000 | Buy |
68.050000 | 0.0 | 1.789762e+05 | 0.0 | 272200.000000 | Sell |
68.040000 | 3300.0 | 4.511762e+05 | 0.0 | 224532.000000 | Buy |
163.030000 | 0.0 | 2.266442e+05 | -0.0 | 195636.000000 | Sell |
153.960000 | 0.0 | 4.222802e+05 | 0.0 | 215544.000000 | Sell |
63.400000 | 0.0 | 6.378242e+05 | -0.0 | 209220.000000 | Sell |
21.390000 | 0.0 | 8.470442e+05 | -0.0 | 213900.000000 | Sell |
183.290000 | 0.0 | 1.060944e+06 | -0.0 | 219948.000000 | Sell |
1118.290000 | 0.0 | 1.280892e+06 | 0.0 | 223658.000000 | Sell |
176.940000 | 1300.0 | 1.504550e+06 | 0.0 | 230022.000000 | Buy |
24.650000 | 9300.0 | 1.274528e+06 | 0.0 | 229245.000000 | Buy |
1149.490000 | 200.0 | 1.045283e+06 | 0.0 | 229898.000000 | Buy |
1021.570000 | 0.0 | 8.153852e+05 | -0.0 | 204314.000000 | Sell |
168.340000 | 0.0 | 1.019699e+06 | -0.0 | 218842.000000 | Sell |
1497.050000 | 0.0 | 1.238541e+06 | 0.0 | 299410.000000 | Sell |
152.190000 | 0.0 | 1.537951e+06 | -0.0 | 0.000000 | Buy |
13.440000 | 0.0 | 1.537951e+06 | -0.0 | 0.000000 | Buy |
1005.100000 | 0.0 | 1.537951e+06 | -0.0 | 0.000000 | Buy |
21.770000 | 0.0 | 1.537951e+06 | -0.0 | 202461.000000 | Sell |
151.910000 | 0.0 | 1.740412e+06 | 0.0 | 0.000000 | Buy |
89.470000 | 0.0 | 1.740412e+06 | 0.0 | 286304.000000 | Sell |
300.690000 | 0.0 | 2.026716e+06 | 0.0 | 300690.000000 | Sell |
54.840000 | 0.0 | 2.327406e+06 | -0.0 | 0.000000 | Buy |
28.070000 | 0.0 | 2.327406e+06 | 0.0 | 356489.000000 | Sell |
511 rows × 9 columns
bk["Portfolio Value"].groupby(level = 0).apply(lambda x: x[-1]).plot()
A more realistic portfolio that can invest in any in a list of twelve (tech) stocks has a final growth of about 100%. How good is this? While on the surface not bad, we will see we could have done better.
Benchmarking
Backtesting is only part of evaluating the efficacy of a trading strategy. We would like to benchmark the strategy, or compare it to other available (usually well-known) strategies in order to determine how well we have done.
Whenever you evaluate a trading system, there is one strategy that you should always check, one that beats all but a handful of managed mutual funds and investment managers: buy and hold SPY. The efficient market hypothesis claims that it is all but impossible for anyone to beat the market. Thus, one should always buy an index fund that merely reflects the composition of the market.By buying and holding SPY, we are effectively trying to match our returns with the market rather than beat it.
I look at the profits for simply buying and holding SPY.
#spyder = web.DataReader("SPY", "yahoo", start, end) spyder = spyderdat.loc[start:end] spyder.iloc[[0,-1],:]
Open | High | Low | Close | Adj Close | |
---|---|---|---|---|---|
date | |||||
2010-01-04 | 112.37 | 113.39 | 111.51 | 113.33 | 113.33 |
2018-01-29 | 285.93 | 286.43 | 284.50 | 284.68 | 284.68 |
batches = 1000000 // np.ceil(100 * spyder.loc[:,"Adj Close"].iloc[0]) # Maximum number of batches of stocks invested in trade_val = batches * batch * spyder.loc[:,"Adj Close"].iloc[0] # How much money is used to buy SPY final_val = batches * batch * spyder.loc[:,"Adj Close"].iloc[-1] + (1000000 - trade_val) # Final value of the portfolio final_val
2507880.0
# We see that the buy-and-hold strategy beats the strategy we developed earlier. I would also like to see a plot. ax_bench = (spyder["Adj Close"] / spyder.loc[:, "Adj Close"].iloc[0]).plot(label = "SPY") ax_bench = (bk["Portfolio Value"].groupby(level = 0).apply(lambda x: x[-1]) / 1000000).plot(ax = ax_bench, label = "Portfolio") ax_bench.legend(ax_bench.get_lines(), [l.get_label() for l in ax_bench.get_lines()], loc = 'best') ax_bench
Buying and holding SPY performs about as well as our trading system, at least how we currently set it up, and we haven’t even accounted for how expensive our more complex strategy is in terms of fees. Given both the opportunity cost and the expense associated with the active strategy, we should not use it.
What could we do to improve the performance of our system? For starters, we could try diversifying. All the stocks we considered were tech companies, which means that if the tech industry is doing poorly, our portfolio will reflect that. We could try developing a system that can also short stocks or bet bearishly, so we can take advantage of movement in any direction. We could seek means for forecasting how high we expect a stock to move. Whatever we do, though, must beat this benchmark; otherwise there is an opportunity cost associated with our trading system.
Other benchmark strategies exist, and if our trading system beat the “buy and hold SPY” strategy, we may check against them. Some such strategies include:
- Buy SPY when its closing monthly price is aboves its ten-month moving average.
- Buy SPY when its ten-month momentum is positive. (Momentum is the first difference of a moving average process, or .)
(I first read of these strategies here.) The general lesson still holds: don’t use a complex trading system with lots of active trading when a simple strategy involving an index fund without frequent trading beats it. This is actually a very difficult requirement to meet.
As a final note, suppose that your trading system did manage to beat any baseline strategy thrown at it in backtesting. Does backtesting predict future performance? Not at all. Backtesting has a propensity for overfitting, so just because backtesting predicts high growth doesn’t mean that growth will hold in the future. There are strategies for combatting overfitting, such as walk-forward analysis and holding out a portion of a dataset (likely the most recent part) as a final test set to determine if a strategy is profitable, followed by “sitting on” a strategy that managed to survive these two filters and seeing if it remains profitable in current markets.
Conclusion
While this lecture ends on a depressing note, keep in mind that the efficient market hypothesis has many critics. My own opinion is that as trading becomes more algorithmic, beating the market will become more difficult. That said, it may be possible to beat the market, even though mutual funds seem incapable of doing so (bear in mind, though, that part of the reason mutual funds perform so poorly is because of fees, which is not a concern for index funds).
This lecture is very brief, covering only one type of strategy: strategies based on moving averages. Many other trading signals exist and employed. Additionally, we never discussed in depth shorting stocks, currency trading, or stock options. Stock options, in particular, are a rich subject that offer many different ways to bet on the direction of a stock. You can read more about derivatives (including stock options and other derivatives) in the book Derivatives Analytics with Python: Data Analysis, Models, Simulation, Calibration and Hedging, which is available from the University of Utah library.
Another resource (which I used as a reference while writing this lecture) is the O’Reilly book Python for Finance, also available from the University of Utah library.
If you were interested in investigating algorithmic trading, where would you go from here? I would not recommend using the code I wrote above for backtesting; there are better packages for this task. Python has some libraries for algorithmic trading, such as pyfolio (for analytics), zipline (for backtesting and algorithmic trading), and backtrader (also for backtesting and trading). zipline seems to be popular likely because it is used and developed by quantopian, a “crowd-sourced hedge fund” that allows users to use their data for backtesting and even will license profitable strategies from their authors, giving them a cut of the profits. However, I prefer backtrader and have written blog posts on using it. It is likely the more complicated between the two but that’s the cost of greater power. I am a fan of its design. I also would suggest learning R, since it has many packages for analyzing financial data (moreso than Python) and it’s surprisingly easy to use R functions in Python (as I demonstrate in this post).
You can read more about using R and Python for finance on my blog.
Remember that it is possible (if not common) to lose money in the stock market. It’s also true, though, that it’s difficult to find returns like those found in stocks, and any investment strategy should take investing in it seriously. This lecture is intended to provide a starting point for evaluating stock trading and investments, and, more generally, analyzing temporal data, and I hope you continue to explore these ideas.
I have created a video course published by Packt Publishing entitled Training Your Systems with Python Statistical Modeling, the third volume in a four-volume set of video courses entitled, Taming Data with Python; Excelling as a Data Analyst. This course discusses how to use Python for machine learning. The course covers classical statistical methods, supervised learning including classification and regression, clustering, dimensionality reduction, and more! The course is peppered with examples demonstrating the techniques and software on real-world data and visuals to explain the concepts presented. Viewers get a hands-on experience using Python for machine learning. If you are starting out using Python for data analysis or know someone who is, please consider buying my course or at least spreading the word about it. You can buy the course directly or purchase a subscription to Mapt and watch it there.
If you like my blog and would like to support it, spread the word (if not get a copy yourself)! Also, stay tuned for future courses I publish with Packt at the Video Courses section of my site.
Thanks for visiting r-craft.org
This article is originally published at https://ntguardian.wordpress.com
Please visit source website for post related comments.