Before (or after) diving into the code, if you want to understand the context behind this project, see READMEQuant.md

• Run this script: ./ishares/direct_downloader.py

etf_landing_page = 'https://www.ishares.com/us/products/239707/ishares-russell-1000-etf'
start_date = '2019-12-31'
end_date = datetime.now().strftime('%Y-%m-%d')
output_file = './data/direct_downloader/IWB_holdings_20220823.csv'

• etf_downloader.py provides functions to get individual etf holdings from ishares.com (wherein each call/event is a etf_ticker holding_date pair).

• sync_etf_downloader.pyinvokes/queues a series of new events keeping local/cloud directories in sync with iShares.com
• Supports local or S3 file-store (see outputpath input parameter)
• Supports multiprocessing (see parameter in config.py)
• Polite, under-the-radar requests (various throttles and perameters available in config.py)

./examples/ishares_to_df.py scrape a specific url:

from ishares.etf_downloader import get_etf_holdings_df, get_etf_index

# index/metadata of all etfs on ishares product line
etf_index = get_etf_index()

# get holdings directly from iShares.com (R3000 on 2020-07-15)
df_ishares = get_etf_holdings_df(ticker='IWV',holdings_date='2020-07-15')

For more comprehensive data projects, you can run sync_etf_downlaoder.py for full holding histories (back to 2006) of multiple etfs. If you define outputpath=='s3' and run the athena table ddl, you can run sql queries across different dates:

from ishares.utils import s3_helpers, athena_helpers

# list available files in s3 bucket
etf_files = s3_helpers.list_keys(Bucket='etf-holdings')

# get a single file from s3
df_small = s3_helpers.get_etf_holdings('IWV', '2020-06-30')

# query a lot of data with Athena
df_big = athena_helpers.query('select * from qcdb.ishares_holdings '
                              'where etf=\'IWV\' '
                              'and '
                              'asofdate between date \'2020-01-31\' and date \'2020-06-30\'')

can be run anytime to keep etf holdings up to date. If overwrite=True, the entire etf holdings history is downloaded

usage: batch_etf_downloader.py [-h] --outputpath OUTPUTPATH [--overwrite]
                               tickers [tickers ...]

batch ETL of iShares ETF holdings, supporting multiple funds and dates; option
to update or overwrite existing extracts

positional arguments:
  tickers               List of ETF Tickers (space-delimeted) you wish to
                        download; full list of tickers located here:
                        ./ishares/data/ishares-etf-index.csv

optional arguments:
  -h, --help            show this help message and exit
  --outputpath OUTPUTPATH
                        where to send output on local machine; if
                        outputpath=='s3', output is uploaded to the
                        Aws.OUPUT_BUCKET variable defined in config.py
  --overwrite           Overwrite holdings that have already been downloaded
                        to S3 (otherwise it's an update)

S3 naming convention: <config.Aws.OUPUT_BUCKET>/type=holdings/state=formatted/etf=IWF/asofdate=2006-09-29.csv
Local naming convention: ./ishares/output/type=holdings/state=formatted/etf=IWF/asofdate=2006-09-29.csv

S3 naming convention: <config.Aws.OUPUT_BUCKET>/type=holdings/state=formatted/etf=IWF/asofdate=2006-09-29.csv
Local naming convention: ./ishares/output/type=holdings/state=formatted/etf=IWF/asofdate=2006-09-29.csv

1. ishares/build_etf_master_index.py - run this periodically to refresh the ETF universe