download Intra Day Data R Programming

Why get the data from IEX?

There are three reasons why you should get your intraday data from IEX:

It's free. IEX is the only platform that lets you regularly download large amounts of intraday OHCL (open, high, close, low) data for free. Usually, intraday data is expensive if you want to obtain large data sets.
IEX has a great API. The IEX API is well documented, easy to understand, and above all it supports asynchronous requests flawlessly.
You can paper-trade on Alpaca. Alpaca is the best API for algorithmic trading that I have found. They offer free, unlimited paper trading for IEX ticker symbols. Hence, you can directly put your strategies that you developed with the IEX data to practice.

What are alternative API services for financial data?

Let me mention thre e other services that I have tested and that you may consider as alternative sources for intraday OHCL data. All of these may be interesting for particular reasons.

World Trading Data: If you want to collect intraday OHCL data from other exchanges e.g. NASDAQ, NYSE and also non-US exchanges, this is a very cheap option. Example: With the $16/month plan, you can get 2-min resolution data for ca. 50k stocks. However, it will take you frequent downloads (1 session per day for 2 weeks every other 2 weeks) and your downloads will be quite slow since World Trading Data is not very supportive of asynchronous requests in my experience.
Tiingo : If you want to collect historic 1-min intraday data from IEX since approx. 2017, Tiingo is the cheapest option. It will only cost you ca. $10 in total since Tiingo has very generous API call limits. Note: the IEX API does not allow you to access intraday data more than 30 calendar days in the past. Therefore, Tiingo may be interesting if you want to get historic data for a longer time period quickly. In contrast to that, the IEX API is great for regular and completely free downloads.
Alphavantage : If you are looking to regularly collect intraday OHCL data for other exchanges e.g. NASDAQ, NYSE and don't need a large amount of ticker symbols, then Alphavantage may be sufficient to get this data for free. Alphavantage covers a very large amount of tickers. However, the API call limits for the free plan are rather limited (5 API calls per minute, 500 calls per day).

What is the purpose of the script?

The script introduced below is what I personally use to collect 1-min intraday data from IEX. You may want to adjust the script for your own purposes. Maybe however, it's just what you are looking for. I wrote the script with the following three purposes in mind:

Collect the data for later processing e.g. backtesting and training machine learning models. (Not: to feed your life algorithms with data)
Store the data locally e.g. on your computer or maybe even in a cloud storage folder on your computer. (Not: to put the data into a database)
Make the download simple e.g. I want to be able to run the script any time with python download_IEX.py(Not: having to pass any parameters or having to do the download on specific days)

How does the script work?

Whenever you execute the script with python download_IEX.py, you are starting a new download session (for all dates in the past for which data is available and not downloaded yet). Each download session works as follows:

Initializing log: The script initiates a new log file for the current session (see init_logging()). E.g. if you conduct your session on October 26, you will find the corresponding log file in script/log/20191026.log. Essentially, each log file contains the printouts that you could see in the terminal when executing the script.
Fetching dates: The script fetches all dates for which downloads have to be done (see get_dates()). For this, it checks the existing folders in the output directory and takes a note of the dates for which data has been downloaded already. The required dates are then computed as all dates that are 30 days or less in the past and not yet in the output folder.
Conducting download: The script then conducts a separate asynchronous download session for each of the fetched dates from step 2. For each date, an asynchronous download session is prepared (see asyncio_prep()), e.g. all available tickers from the IEX API are fetched. Then, this session is executed with the function download_tickers_asynchronous(), which asynchronously fetches and writes the data for the individual tickers (see get_csv() and write_csv()).

How do I set up the script?

To set up the script, you need to complete the following three steps.

Download the repository to your computer

You can find the Github repository here. Download the repository by going to Clone or download and then on Download zip . Then, place the downloaded file IEX_historical-prices-master.zip in a folder of your choice on your computer and extract the zip directory.

You should now have a folder (with a name of your choice) that contains the following files:

2. Get your free API key from IEX

Go to the IEX Cloud website and register for free. Then find your free API token under API Tokens at SECRET (see red square in the screenshot below).

3. Set up your script/config.json file

Open a new file in a text editor of your choice, e.g. Visual Studio Code. Then, put the following into the first three lines of the new file and replace YOUR_TOKEN with the SECRET token from the previous step.

          {          "TOKEN": "YOUR_TOKEN"          }

Now, click Save as and save the new file with the name config.json in the script directory. Your folder should now look like this:

How do I start the download?

After setting up the script as described above, you can open a new terminal at the script folder and execute the script withpython download_IEX.py. If all required packages are installed (see the imports at the beginning of download_IEX.py), the script will start downloading the IEX intraday data.

In the terminal, you will see the following for each downloaded file:
the timestamp, the ticker symbol, the index of the ticker symbol (among all available IEX tickers) and the date for which the ticker was downloaded:

How do I interrupt the download?

You can interrupt the download simply by interrupting the script that's being executed. (If you execute the script in the terminal, control+c does the trick on Mac.) Later on, you can just resume the script by restarting it with python download_IEX.py but you will have to restart the download for the date for which data was currently being downloaded when you interrupted the script.

Importantly, by default no files or folders are deleted when the script starts. Therefore, you might want to manually delete the folder for the date with incomplete downloads. That way, the script will do a fresh start for the mentioned date when you restart it with python download_IEX.py.

How often should I do the download?

If you want to use this script for regularly downloading IEX data, you will want to run it at least every four weeks. The reason is that according to the documentation, the API lets you download data for "30 trailing calendar days" (see IEX documentation). This means that if you were to only do a download session e.g. every eight weeks, you will lack data for some days.

However, I was actually able to get data for more than 30 trailing calendar days. Probably the documentation meant to say "30 trailing calendar week days". In any case, it's safe to execute the script every four weeks.

Note that whenever you run the script, you won't need to set any parameters since it automatically detects for which dates you have downloaded data already and for which dates you can download data from IEX. Therefore, you could also execute the script e.g. every two weeks or have irregular breaks between your download sessions (sometimes two weeks, sometimes more).

How do I make use of the downloaded files?

You can see that the script automatically creates a folder structure by year, ISO calendar week and date. For every date, the script performs a separate asynchronous download session. This means that you will have folders for each trading date that contain separate csv files: one csv file for each ticker.

Essentially, the script gives you the raw data that comes from IEX. How you process this data later on is up to you. E.g. you might want to append all files for the same ticker to get one single file per ticker symbol.

Important notes:

When the asynchronous download session for one specific day is completed, the folder for this day is zipped automatically to save space. Hence, you may want to unzip the folders before processing the raw data.
Each date folder contains three data folders DONE, ERROR, and NONE. You will want to use the files in the folder DONE because these files contain actual intraday data for valid ticker symbols. The files in the folder ERROR are mostly test tickers from IEX (see this GitHub issue) and the files in the folder NONE correspond to tickers that did not have any trades that day (see this GitHub issue).

How will the downloaded data look like?

In the DONE folder of a given date, you will find one csv file for every ticker:

And each of these files contains the same OHCL information. Here is an example for Apple (AAPL):

Reference

Source code on Github
IEX Cloud documentation
API for paper-trading IEX tickers

Thanks for reading and looking forward to your feedback!

Posted by: issacregenera.blogspot.com

Source: https://towardsdatascience.com/how-to-download-all-historic-intraday-ohcl-data-from-iex-with-python-asynchronously-via-api-b5b04a31b187

Cari Blog Ini

jodydoer