How to acquire free historical Tick and Bar data for algo trading and backtesting in 2020! [Stocks, Forex, and Crypto Currency]

jack

jack

Administrator
Staff member
I noticed that many guides or lists of tick data sources were out of date, broken, or otherwise not working, so I decided to make an updated list with modern sources for use as a resource with the algo trading community. I will also keep this up to date as new sources appear and old sources break.

Below are the guides for downloading tick or bar level data for various markets which can be used for backtesting and analysis.

Note: Typical format for tick and bar data is in CVS files. You might need to adjust or re-arrange said files as needed to fit into your desired analytics and/or backtesting program.


Table of Contents:
  1. Stocks (this post)
  2. Forex (Darwinex, Pepperstone / TrueFX, Dukascopy)
  3. Crypto Tick and Bar Data


--------------------------


Stocks [Daily Bars]

Yahoo Finance - US and International stock exchanges - On a stock by stock basis, you can download decades of historical daily bar data by doing the following steps:
  1. Go to https://finance.yahoo.com
  2. In the search field, type in your desired symbol or company name and select the appropriate result
  3. Click the 'Historical Data' tab
  4. Set the date range to 'MAX'
  5. Hit the 'Apply' button
  6. Click the 'Download Data' link just below the 'Apply' button
  7. Enjoy your daily bar data

Google Finance - Rest in peace.. :( Google nuked their finance site back to the stone age many years ago so this service is pretty useless now (Honestly Google, what were you thinking? Google Finance was awesome back in 2012 and prior.) Google also used to have an API but that too got turned off. I included an entry for Google here because many people will be searching around the net and find references to using Google Finance's API but be unsure why it's not working as expected.

Cautious mention - QuantConnect - tick and minute bar data on stocks and forex. I'm currently reviewing QuantConnect as a potential source. They do offer a lot of free data but since their ecosystem was built around offering up crowd sourced strategies to prop firms I'm hesitant to recommend them until I've wrapped my head around their offering and can say I trust their platform.
 
Last edited:
Forex [Tick Data] (Since tick data is available from multiple sources without cost, I will only mention bar data at the end.*)



Darwinex -
Darwinex is easy but you'll need to have a live account. Anyone can signup for an account (live) and head over to this page here once logged in: Historical Tick Data Download. From there, just click the "Request FTP Access" button to have your FTP login details generated for you. Darwinex provides tick level data all major and minor currency pairs, as well as index and commodity CFDs, with most of them going back to mid-late 2017. You don't need to deposit to have a live account at Darwinex, you just need to have a verified account with them, so there's little downside to enabling access to this high quality source of tick data. (As a side benefit, you can also get access to historical price data of their Darwin Exchange, so you can model investing in other traders and model their strategy returns, which is pretty cool.)​
​
​

Pepperstone / Integral / TrueFX -
Pepperstone relies on price feeds and liquidity very similar to Integral's offering, so similar that for a while Pepperstone used to publish Integral's data on their own FTP for easy customer access. Unfortunately, they stopped offering this FTP access some time ago (thus why most online tutorials referencing them are out of date) and the source of data that Pepperstone currently recommends is Integral's TrueFX data service offering. A free account at TrueFX.com will get you access to tick data on most major and minor currency pairs from the past year only (Currently only 2019 YTD is available for free.) I encouraged Pepperstone to bring back their historical data access to regain access to years prior to 2019 and if they ever open back up then I'll update this spot here with access info.​
​
​

Dukascopy -
This source is a staple in the forex tick data world. Dukascopy Bank has been around for a long time and has offered historical tick data on their products with dates going back into the mid-early 2000's. Like the other sources, you'll need signup to their site, but they do not require a live account to access data. To access, all you need to do is visit the Dukascopy Bank Historical Data Feed page, and use the historical download tool provided (with registering or logging in when prompted before it will let you download.) The download tool makes it easy to select what data you want, but note that all non-forex listings here (like stocks) are CFDs on Duka's platform, NOT the underlying stock data that trades on a stock exchange. (If you're interested in Dukascopy but need to trade under Euro zone regulations, you can find their Europe subsidiary here with different trading conditions.)​

*Regarding Bar type data - Downscale it from the tick data above as needed. However, if you really only want daily bar data, you can find it from Oanda's website, or others pretty easily.
 
Last edited:
Last edited:
hi, thanks for the great article.

well, i' ve downloaded phyton and opened cmd, typed in "pip install bitmex" and got the error below.

'pip' is not recognized as an internal or external command,
operable program or batch file.

can you help me?
 
ciskeceli said:
hi, thanks for the great article.

well, i' ve downloaded phyton and opened cmd, typed in "pip install bitmex" and got the error below.

'pip' is not recognized as an internal or external command,
operable program or batch file.

can you help me?
Click to expand...
Sounds like you might have unchecked adding python to your system's PATH during the installation. Try uninstalling Python, then reinstall making sure to "install for all users" and "add to PATH" when given the options.
 
Hi, thanks for the help!

I see BTCUSD listed on Binance but it's only letting me pull BTCUSDT, it says that BTCUSD is an invalid symbol. Any idea how to fix this issue? Thanks!
 
Hi Jack,

thank you very much for the information, really useful!

I have a question: do you know how I can extract the full symbols list from Yahoo finance in order to avoid downloading stock info one by one? I would like to backtest the European market and I didn't find a way to obtain the ticker list from every european market.

Thanks in advance and good luck with the trading!
 
MettoCHief said:
Hey Jack, is it possible to change the tick data into daily data? Thank you!
Click to expand...
Yes, it's possible to do this via Pandas (that the script already uses.) After the line that sets the timestamp as your index you'd then use a resample function.

Read up on how to use panda's resample function in the official pandas docs. It includes examples. Link here:

pandas.DataFrame.resample — pandas 2.3.3 documentation

pandas.pydata.org pandas.pydata.org

Good luck.
 
jack said:
Yes, it's possible to do this via Pandas (that the script already uses.) After the line that sets the timestamp as your index you'd then use a resample function.

Read up on how to use panda's resample function in the official pandas docs. It includes examples. Link here:

pandas.DataFrame.resample — pandas 2.3.3 documentation

pandas.pydata.org pandas.pydata.org

Good luck.
Click to expand...
Thanks! Going try it out! Do you know if and how it is possible to get historical data of orderbooks? Like bid and ask prices

BR
 
jack said:
Yes, it's possible to do this via Pandas (that the script already uses.) After the line that sets the timestamp as your index you'd then use a resample function.

Read up on how to use panda's resample function in the official pandas docs. It includes examples. Link here:

pandas.DataFrame.resample — pandas 2.3.3 documentation

pandas.pydata.org pandas.pydata.org

Good luck.
Click to expand...
Hey Jack, I was not able to get it done. Maybe u could help me out one more time.
Thanks!
BR
 
jack said:
Where are you getting stuck? It's a well documented process .. like an extra line or two. What did you try?
Click to expand...
Sorry I'm not so familiar with scripting in Python...newbie on that. Tried data.resample('1140ms').sum()
getting error that Im using a Range Index instead of datetimeindex
 
MettoCHief said:
Sorry I'm not so familiar with scripting in Python...newbie on that. Tried data.resample('1140ms').sum()
getting error that Im using a Range Index instead of datetimeindex
Click to expand...


I believe in teaching, and encourage people to figure it out themselves after pointing them in the right direction.. but that said, this one turns out to be more than just an extra line given you gotta declare a conversion key for the aggregate. So harder than the 'one liner' I suggested earlier.

Here's an edited script that will work for converting to "daily" bars, what you first asked.

Python: 
import pandas as pd
from binance.client import Client
import datetime
import os

# YOUR API KEYS HERE
api_key = ""    #Enter your own API-key here
api_secret = "" #Enter your own API-secret here

bclient = Client(api_key=api_key, api_secret=api_secret)

start_date = datetime.datetime.strptime('1 Mar 2021', '%d %b %Y')
today = datetime.datetime.today()

conversion = {'open' : 'first', 'high' : 'max', 'low' : 'min', 'close' : 'last', 'volume' : 'sum'}

def binanceBarExtractor(symbol):
    print('working on: {}'.format(symbol))
    filename = '{}_MinuteBars.csv'.format(symbol)

    klines = bclient.get_historical_klines(symbol, Client.KLINE_INTERVAL_1MINUTE, start_date.strftime("%d %b %Y %H:%M:%S"), today.strftime("%d %b %Y %H:%M:%S"), 1000)
    data = pd.DataFrame(klines, columns = ['timestamp', 'open', 'high', 'low', 'close', 'volume', 'close_time', 'quote_av', 'trades', 'tb_base_av', 'tb_quote_av', 'ignore' ])
    data['timestamp'] = pd.to_datetime(data['timestamp'], unit='ms')
    data['volume'] = pd.to_numeric(data['volume'], errors='coerce')
    
    data.set_index('timestamp', inplace=True)

    data = data.resample('D').agg(conversion)
    
    data.to_csv(filename)
    print('finished!')


if __name__ == '__main__':
    for sym in ['BTCUSDT',]:
        binanceBarExtractor(sym)
 
jack said:
I believe in teaching, and encourage people to figure it out themselves after pointing them in the right direction.. but that said, this one turns out to be more than just an extra line given you gotta declare a conversion key for the aggregate. So harder than the 'one liner' I suggested earlier.

Here's an edited script that will work for converting to "daily" bars, what you first asked.

Python: 
import pandas as pd
from binance.client import Client
import datetime
import os

# YOUR API KEYS HERE
api_key = ""    #Enter your own API-key here
api_secret = "" #Enter your own API-secret here

bclient = Client(api_key=api_key, api_secret=api_secret)

start_date = datetime.datetime.strptime('1 Mar 2021', '%d %b %Y')
today = datetime.datetime.today()

conversion = {'open' : 'first', 'high' : 'max', 'low' : 'min', 'close' : 'last', 'volume' : 'sum'}

def binanceBarExtractor(symbol):
    print('working on: {}'.format(symbol))
    filename = '{}_MinuteBars.csv'.format(symbol)

    klines = bclient.get_historical_klines(symbol, Client.KLINE_INTERVAL_1MINUTE, start_date.strftime("%d %b %Y %H:%M:%S"), today.strftime("%d %b %Y %H:%M:%S"), 1000)
    data = pd.DataFrame(klines, columns = ['timestamp', 'open', 'high', 'low', 'close', 'volume', 'close_time', 'quote_av', 'trades', 'tb_base_av', 'tb_quote_av', 'ignore' ])
    data['timestamp'] = pd.to_datetime(data['timestamp'], unit='ms')
    data['volume'] = pd.to_numeric(data['volume'], errors='coerce')
   
    data.set_index('timestamp', inplace=True)

    data = data.resample('D').agg(conversion)
   
    data.to_csv(filename)
    print('finished!')


if __name__ == '__main__':
    for sym in ['BTCUSDT',]:
        binanceBarExtractor(sym)
Click to expand...
Wow amazing! Thank you for your help! I appreciate it!
Best Regards!
 
You must log in or register to reply here.
Back
Top