Author: cfsmith

Cointegration, Correlation, and Log Returns

Co-Author: Eric Kammers

I recently created a Twitter account for the blog where I will curate and comment on content I find interesting related to finance, data science, and data visualization. Please follow me at @Quantoisseur (see the embedded stream on the sidebar). Enjoy the post!


The differences between correlation and cointegration can often be confusing. While there are some helpful explanations online, I wasn’t satisfied with the visual examples. When looking at a plot of an actual pair of symbols where the correlation and cointegration test results differ, it can be difficult to pinpoint which portions of the time series are responsible for these separate properties. To solve this, I decided to produce some basic examples with sinusoidal functions so I could solidify my understanding of these concepts.

First, let’s highlight the difference between cointegration and correlation. Correlation is more familiar to most of us, especially outside of the financial industry. Correlation is a measure of how well two variables move in tandem together over time. Two common correlation measures are Pearson’s product-moment coefficient and Spearman’s ranks-order coefficient. Both coefficients range from -1, perfect negative correlation, to 0, no correlation, to 1, perfect positive correlation. Positive correlation means that the variables move in tandem in the same direction while negative correlation means that they move in tandem but in opposite directions. When calculating correlation, we look at returns rather than price because returns are normalized across differently priced assets. The main difference between the two correlation coefficients is that the Spearman coefficient measures the monotonic relationship between two variables, while the Pearson coefficient measures their linear relationship. Figure 1 below shows how the different coefficients behave when two variables exhibit either a linear or nonlinear relationship. Notice how the Spearman coefficient remains 1 for both scenarios since the relationship in both cases is perfectly monotonic.

fig1
Figure 1: Pearson vs Spearman for Nonlinear and Linear Functions

Based on the distributions of the data, these coefficients can behave differently which I will explore with additional examples later in this post. Here are some resources for further clarification on the Pearson and Spearman coefficients.

Now, cointegration tests do not measure how well two variables move together, but rather whether the difference between their means remains constant. Often, variables with high correlation will also be cointegrated, and vice versa, but this isn’t always the case. In contrast to correlation, when testing for cointegration we use prices rather than returns since we’re more interested in the trend between the variables’ means over time than in the individual price movements. There are multiple cointegration tests, but in this case, I’ll be using the Augmented Dicky-Fuller test to evaluate the stationarity of the residuals from the linear model created with the pair’s price series.

Second, using log returns for financial calculations is, in many cases, preferable to using simple returns. There are many resources online explaining the advantages and disadvantages of using log returns. We will not dive into this topic too much, but some of the advantages are due to assuming a log normal distribution which makes them easier to work with and gives them convenient properties like time-additivity. Figure 2 below shows the relationship between log and simple returns.

fig2
Figure 2: Relationship between Simple and Log Returns

Furthermore, correlation is a second moment calculation meaning that it is only appropriate if higher moments are insignificant. Using log returns is better so we can ensure the higher moments are negligible and avoid having to use copulas.

Now with this framework, we can introduce some visual examples. Figure 3 below will be our baseline example which we will adjust in a variety of ways to examine how the values in the table react. In this figure, the red and green series are identical but are oscillating around different mean prices. The difference between the means of the variables is static over time which is why ADF test confirms their cointegration. The price, simple returns, and log returns correlations are all 1, perfectly positively correlated.

fig3
Figure 3: Baseline Example, Perfect Cointegration and Correlation

By phase shifting the green price series as seen in Figure 4 below, all the correlation coefficients now indicate a lack of correlation between the series. As expected, the pair remains cointegrated.

fig4
Figure 4: Perfect Cointegration but No Correlation

I now put the pair back in sync and the red series is adjusted as seen in Figure 5. The pair isn’t cointegrated anymore since the difference between their means fluctuates over time. The returns correlation coefficients agree that the series are strongly correlated while the price only supports a weak correlation.

Figure_Update_Same
Figure 5: No Cointegration but Strong Returns Correlation

In the above example, the Pearson and Spearman coefficients begin to diverge but now we’ll look at an example where they differ significantly. Since the Spearman coefficient is based on the rank-order of the variables and not the actual distance between them, it is known to be more resilient to large deviations and outliers. We can test this by adding an anomaly, possibly a data outage, to the top series by randomly choosing a period of 25 data points to set equal to 1. The effect can be observed in the table accompanying Figure 6 below. The Spearman coefficient supports strong positive correlation while the Pearson coefficient claims there is little to no correlation.

fig6
Figure 6: Outliers Effect on Pearson and Spearman Coefficient Calculations

The final example we will look at it is a situation where the returns are not strongly correlated but the prices are. Instinctively, I think I would side with the returns correlation results in Figure 7.

fig7
Figure 7: High Price Correlation but Low Returns Correlation

One aspect of these correlation tests we have been overlooking, is the distributions of the variables. In these sinusoidal examples, neither simple nor log returns are normally distributed. It is often advertised that the Pearson correlation coefficient requires the data to be normally distributed. One counter argument is the distribution only needs to be symmetric, not necessarily normal. The Spearman coefficient is a nonparametric statistic and thus does not require a normal distribution. In many of the previous examples, the two coefficients are functionally the same despite the odd distribution of the log returns. In Figure 8 below, we take our basic series and add random noise to one of them which creates a more normal distribution. The normality of these log returns are tested with the Shapiro-Wilk normality test. As seen in the right histogram, our basic sinusoidal wave’s log returns reject the null hypothesis that they are normally distributed. In the left histogram, the noisy wave’s log returns fail to reject the null hypothesis.

fig8

fig9.png
Figure 8: Distributions and Price Series when Noise is Added to One Series

Despite changing one variable’s distribution, the Pearson and Spearman coefficients remain about the same. Additionally, as seen in Figure 9 below, normalizing both variable’s distributions does not cause the coefficients to differ.

fig10.png
Figure 9: Noise Added to Both Price Series

These distribution examples do not fully support a side of the debate but I’m not convinced that the Pearson coefficient strictly requires normality.

Playing around with these examples was very helpful for my understanding of cointegration, correlation, and log returns. It is now very clear to me why returns, particularly log returns, are used when calculating correlation and why price is used to test for cointegration. The choice between using the Pearson or Spearman correlation coefficient is slightly more difficult but it can’t hurt to look at both and see how it impacts your data decisions!

The code to generate all the figures in this post can be found here.


Eric Kammers is a recent graduate of the University of Washington (2017) where he studied Industrial & Systems Engineering. He is actively seeking opportunities that will add value to his current skill-set. He is a strong-willed, self-driven individual who has the urge for life-time learning. He loves mathematics and statistics, especially applying their methods to practical problems in data science and engineering. LinkedIn: https://www.linkedin.com/in/ekammers/

Redesigning Data Visualization Atrocities #1

Hello all, for a while I’ve been wanting to diversify the content on my blog to include general data science and visualization samples. I finally got a burst of motivation this weekend when I was casually studying for the GRE and came across a figure in a practice test that, in my opinion, is an example of a failed visualization. Figure 1 below shows the original figure which was followed by a series of basic data interpretation questions.

Capture
Figure 1: Original Figure

Now before we dive into this, I’m sure the GRE purposefully uses unhelpful visualizations to ensure test takers understand the components of a graph (legend, axes, etc.) well enough to extract the necessary information via brute force inspection. Either way, we’re going to make this figure more accessible and visualizing assisting. There are 3 serious issues that I have with it.

  1. The date ranges (top series) and individual months (bottom series) are seemingly unrelated but are plotted on the same graph using a broken y-axis with a discontinuous scale. The two parts of the graph are pretty well distinguished but to avoid associating their patterns/trends with each other, it would be recommended to separate these plots.
  2. The only way to understand the time period for each line is by looking at the markers and legend. Besides their filling, these markers have no connection to their corresponding time periods.
  3. The x-axis contains the different charitable causes where their order is irrelevant. By using a line plot with markers, it gives you the impression that order is important and that the upward trend, for example, could matter.

To solve the first problem, the plots are separated and placed side-by-side in Figure 2 below. Though not a huge improvement, the y-axes are now appropriate and the date ranges are clearly separated from the individual months.

1
Figure 2: Enhanced Figure – Y-Axes Fixed

Now to fix the second problem, we drop the meaningless markers and adopt a grayscale where more recent dates are darker than earlier dates. It still requires you to refer to the legend but once you understand the direction of the grayscale, the figure becomes much more digestible as seen below.

2
Figure 3: Enhanced Figure – Markers Fixed

Based on the dates in the figure, I assume adding color was not an option at the time of its creation but we’re going to go ahead and spice it up with some color now. Finally, to fix the third problem, the x-axis is now time, increasing left to right, and the individual lines represent the different charitable causes. The figure is immediately more intuitive as now trends along the x-axis have meaning. As seen in Figure 4 below, in the date range plot, your attention is immediately drawn to the increase in disaster relief between the 2nd and 3rd date range. In the monthly plot, the uniqueness of the disaster relief and child safety lines compared to the others is quickly realized.

3
Figure 4: Enhanced Figure – Independent Variable

Now that we have fixed my issues with the original figure, we have a clean visualization. There are still some improvements we can make though. Having to track the individual lines and values can be visually straining. It is much easier on our eyes to compare 2-D areas so by switching to a stacked bar chart, Figure 5 becomes even more accessible. The significant changes in private donations to disaster relief are still very prominent in the new figure. Unfortunately, by switching chart types, we lose information on the exact donation values for each cause in exchange for gaining information on the total amount of private donations.

4
Figure 5: Enhanced Figure – Stacked Bar Chart

Depending on the purpose of the figure, the trade-off between individual donation values and the total donation values may or may not be worth it. A somewhat happy medium when working with a stacked bar chart, with this many categories, is to use a radar chart instead. As seen in Figure 6 below, the individual donation values are available and by considering the area of the series, the total donation values can be extrapolated. Additionally, a monochromatic scale can be used to highlight time component of the series.

5
Figure 6: Enhanced Figure – Radar Chart

I hope this example highlighted the importance of choosing an appropriate visualization based on your audience and the aspects of the data that you want emphasized. Especially when working with a small data set like this, the details matter. Thanks for reading!

Twitter and StockTwits Sentiment Data Open-Close

Hello all, last week I wrote a guest post featured on Dr. Ernest Chan’s blog which highlighted some of my research while working with QTS Capital Management on social media sentiment analysis and its place in financial models. The focus of this research was on how to derive sentiment signals from the labeled StockTwits data. This proved to be possible but not as statistically significant as using natural language processing on all of the StockTwits data, like we do at Social Market Analytics.  In addition to performing sentiment analysis on StockTwits, we also use Twitter as a source. In some cases we find the Twitter data to outperform StockTwits. In the Figure below, the same open to close simulation is ran with the Twitter data and results in a 4.8 Sharpe Ratio.

BPM

(Enlarge Figure)

Please contact me if you have any questions. Thanks!

Cointegrated ETF Pairs Part II

Update 5/17: As discussed in the comments, the reason the results are so exaggerated is because it is missing portfolio rebalancing to account for the changing hedge ratio. It would be interesting to try an adaptive hedge ratio that requires only weekly or monthly rebalancing to see how legitimately profitable this type of strategy could be.

Welcome back! This week’s post will backtest a basic mean reverting strategy on a cointegrated ETF pair time series constructed using the methods described in part I. Since the EWA (Australia) – EWC (Canada) pair was found to be more naturally cointegrated, I decided to run the rolling linear regression model (EWA chosen as the dependent variable) with a lookback window of 21 days on this pair to create the spread below.

1
Figure 1: Cointegrated Pair, EWA & EWC

With the adaptive hedge ratio, the spread looks well suited to backtest a mean reverting strategy on. Before that, we should check what the minimum capital required to trade this spread is. Though everyone has a different margin requirement, I thought it would be useful to walkthrough how you would calculate the capital required. In this example we assume our broker allows a margin of 50%. We first will compute the daily ratio between the pair, EWC/EWA. This ratio represents the amount of EWA shares for each share of EWC that must be owned to have an equal dollar move for every 1% move. The ratio fluctuates daily but has a mean of 1.43. This makes sense because EWC, on average, trades at higher price. We then multiply these ratios by the rolling beta. Then for reference, we can fix the held EWC shares to 100 and multiply the previous values (ratios*rolling beta) by 100 to determine the amount of EWA shares that would be held. The amount of capital required to hold this spread can then be calculated with the equation: margin*abs((EWC price * 100) + (EWA price * calculated shares)). This is plotted for our example below.

newnew
Figure 2: Required Capital

From this plot we can see that the series has a max value of $5,466 which is not a relatively large required capital. I hypothesize that the less cointegrated a pair is, the higher the minimum capital will be (try the EWZ-IGE pair).

We can now go ahead and backtest the figure 1 time series! A common mean reversal strategy uses Bollinger Bands, where we enter positions when the price deviates past a Z-score/standard deviation threshold from the mean. The exit signals can be determined from the half-life of its mean reversion or it can be based on the Z-score. To avoid look-ahead bias, I calculated the mean, standard deviation, and Z-score with a rolling 50-day window. Unfortunately, this window had to be chosen with data-snooping bias but was a reasonable choice. This backtest will also ignore transaction costs and other spread execution nuances but should still reasonably reflect the strategy’s potential performance. I decided on the following signals:

  • Enter Long/Close Short: Z-Score < -1
  • Close Long/Enter Short: Z-Score > 1

This is a standard Bollinger Bands strategy and results were encouraging.

3

4

Though it made a relatively small amount of trades over 13 years, it boasts an impressive 2.7 Sharpe Ratio with 97% positive trades. Below on the left we can see the strategy’s performance vs. SPY (using very minimal leverage) and on the right the positions/trades are shown.

new2

Overall, this definitely supports the potential of trading cointegrated ETF pairs with Bollinger Bands. I think it would be interesting to explore a form of position sizing based on either market volatility or the correlation between the ETF pair and another symbol/ETF. This concludes my analysis of cointegrated ETF pairs for now.

Acknowledgments: Thank you to Brian Peterson and Ernest Chan for explaining how to calculate the minimum capital required to trade a spread. Additionally, all of my blog posts have been edited prior to being published by Karin Muggli, so a huge thank you to her!

Note: I’m currently looking for a full-time quantitative research/trading position beginning summer/fall 2017. I’m currently a senior at the University of Washington, majoring in Industrial and Systems Engineering and minoring in Applied Mathematics. I also have taken upper level computer science classes and am proficient in a variety of programming languages. Resume: https://www.pdf-archive.com/2017/01/31/coltonsmith-resume-g/. LinkedIn: https://www.linkedin.com/in/coltonfsmith. Please let me know of any open positions that would be a good fit for me. Thanks!

Full Code:

detach("package:dplyr", unload=TRUE)
require(quantstrat)
require(IKTrading)
require(DSTrading)
require(knitr)
require(PerformanceAnalytics)
require(quantstrat)
require(tseries)
require(roll)
require(ggplot2)

# Full test
initDate="1990-01-01"
from="2003-01-01"
to="2015-12-31"

## Create "symbols" for Quanstrat
## adj1 = EWA (Australia), adj2 = EWC (Canada)

## Get data
getSymbols("EWA", from=from, to=to)
getSymbols("EWC", from=from, to=to)
dates = index(EWA)

adj1 = unclass(EWA$EWA.Adjusted)
adj2 = unclass(EWC$EWC.Adjusted)

## Ratio (EWC/EWA)
ratio = adj2/adj1

## Rolling regression
window = 21
lm = roll_lm(adj2,adj1,window)

## Plot beta
rollingbeta <- fortify.zoo(lm$coefficients[,2],melt=TRUE)
ggplot(rollingbeta, ylab="beta", xlab="time") + geom_line(aes(x=Index,y=Value)) + theme_bw()

## Calculate the spread
sprd <- vector(length=3273-21)
for (i in 21:3273) {
sprd[i-21] = (adj1[i]-rollingbeta[i,3]*adj2[i]) + 98.86608 ## Make the mean 100
}
plot(sprd, type="l", xlab="2003 to 2016", ylab="EWA-hedge*EWC")

## Find minimum capital
hedgeRatio = ratio*rollingbeta$Value*100
spreadPrice = 0.5*abs(adj2*100+adj1*hedgeRatio)
plot(spreadPrice, type="l", xlab="2003 to 2016", ylab="0.5*(abs(EWA*100+EWC*calculatedShares))")

## Combine columns and turn into xts
close = sprd
date = as.data.frame(dates[22:3273])
data = cbind(date, close)
dfdata = as.data.frame(data)
xtsData = xts(dfdata, order.by=as.Date(dfdata$date))
xtsData$close = as.numeric(xtsData$close)
xtsData$dum = vector(length = 3252)
xtsData$dum = NULL
xtsData$dates.22.3273. = NULL

## Add SMA, moving stdev, and z-score
rollz<-function(x,n){
avg=rollapply(x, n, mean)
std=rollapply(x, n, sd)
z=(x-avg)/std
return(z)
}

## Varying the lookback has a large affect on the data
xtsData$zScore = rollz(xtsData,50)
symbols = 'xtsData'

## Backtest
currency('USD')
Sys.setenv(TZ="UTC")
stock(symbols, currency="USD", multiplier=1)

#trade sizing and initial equity settings
tradeSize <- 10000
initEq <- tradeSize

strategy.st <- portfolio.st <- account.st <- "EWA_EWC"
rm.strat(portfolio.st)
rm.strat(strategy.st)
initPortf(portfolio.st, symbols=symbols, initDate=initDate, currency='USD')
initAcct(account.st, portfolios=portfolio.st, initDate=initDate, currency='USD',initEq=initEq)
initOrders(portfolio.st, initDate=initDate)
strategy(strategy.st, store=TRUE)

#SIGNALS
add.signal(strategy = strategy.st,
name="sigFormula",
arguments = list(label = "enterLong",
formula = "zScore < -1", cross = TRUE), label = "enterLong") add.signal(strategy = strategy.st, name="sigFormula", arguments = list(label = "exitLong", formula = "zScore > 1",
cross = TRUE),
label = "exitLong")

add.signal(strategy = strategy.st,
name="sigFormula",
arguments = list(label = "enterShort",
formula = "zScore > 1",
cross = TRUE),
label = "enterShort")

add.signal(strategy = strategy.st,
name="sigFormula",
arguments = list(label = "exitShort",
formula = "zScore < -1",
cross = TRUE),
label = "exitShort")

#RULES
add.rule(strategy = strategy.st,
name = "ruleSignal",
arguments = list(sigcol = "enterLong",
sigval = TRUE,
orderqty = 15,
ordertype = "market",
orderside = "long",
replace = FALSE,
threshold = NULL),
type = "enter")

add.rule(strategy = strategy.st,
name = "ruleSignal",
arguments = list(sigcol = "exitLong",
sigval = TRUE,
orderqty = "all",
ordertype = "market",
orderside = "long",
replace = FALSE,
threshold = NULL),
type = "exit")

add.rule(strategy = strategy.st,
name = "ruleSignal",
arguments = list(sigcol = "enterShort",
sigval = TRUE,
orderqty = -15,
ordertype = "market",
orderside = "short",
replace = FALSE,
threshold = NULL),
type = "enter")

add.rule(strategy = strategy.st,
name = "ruleSignal",
arguments = list(sigcol = "exitShort",
sigval = TRUE,
orderqty = "all",
ordertype = "market",
orderside = "short",
replace = FALSE,
threshold = NULL),
type = "exit")

#apply strategy
t1 <- Sys.time()
out <- applyStrategy(strategy=strategy.st,portfolios=portfolio.st)
t2 <- Sys.time()
print(t2-t1)

#set up analytics
updatePortf(portfolio.st)
dateRange <- time(getPortfolio(portfolio.st)$summary)[-1]
updateAcct(portfolio.st,dateRange)
updateEndEq(account.st)

#Stats
tStats <- tradeStats(Portfolios = portfolio.st, use="trades", inclZeroDays=FALSE)
tStats[,4:ncol(tStats)] <- round(tStats[,4:ncol(tStats)], 2)
print(data.frame(t(tStats[,-c(1,2)])))

#Averages
(aggPF <- sum(tStats$Gross.Profits)/-sum(tStats$Gross.Losses))
(aggCorrect <- mean(tStats$Percent.Positive))
(numTrades <- sum(tStats$Num.Trades))
(meanAvgWLR <- mean(tStats$Avg.WinLoss.Ratio))

#portfolio cash PL
portPL <- .blotter$portfolio.EWA_EWC$summary$Net.Trading.PL

## Sharpe Ratio
(SharpeRatio.annualized(portPL, geometric=FALSE))

## Performance vs. SPY
instRets <- PortfReturns(account.st)
portfRets <- xts(rowMeans(instRets)*ncol(instRets), order.by=index(instRets))

cumPortfRets <- cumprod(1+portfRets)
firstNonZeroDay <- index(portfRets)[min(which(portfRets!=0))]
getSymbols("SPY", from=firstNonZeroDay, to="2015-12-31")
SPYrets <- diff(log(Cl(SPY)))[-1]
cumSPYrets <- cumprod(1+SPYrets)
comparison <- cbind(cumPortfRets, cumSPYrets)
colnames(comparison) <- c("strategy", "SPY")
chart.TimeSeries(comparison, legend.loc = "topleft", colorset = c("green","red"))

## Chart Position
rets <- PortfReturns(Account = account.st)
rownames(rets) <- NULL
charts.PerformanceSummary(rets, colorset = bluefocus)