Part 1 - Historical Data Collection

In [1]:

from quantrocket.ibg import start_gateways
start_gateways(wait=True)

Out[1]:

{'ibg1': {'status': 'running'}}

Collect CL futures chain¶

Next, we need to collect contract details for all available CL futures. CL is included in QuantRocket's free sample data, so we can collect the contract details by specifying the "FREE" country:

In [2]:

from quantrocket.master import collect_ibkr_listings
collect_ibkr_listings(countries="FREE")

Out[2]:

{'status': 'success', 'msg': 'successfully loaded IBKR FREE securities'}

Define universe of CL futures¶

Next we define a universe of CL futures for easy reference. To do so, download a CSV of CL futures from the securities master database:

In [3]:

from quantrocket.master import download_master_file
download_master_file("cl_futures.csv", exchanges="NYMEX", sec_types="FUT", symbols="CL")

Then upload the CSV to create the "cl-fut" universe:

In [4]:

from quantrocket.master import create_universe
create_universe("cl-fut", infilepath_or_buffer="cl_futures.csv")

Out[4]:

{'code': 'cl-fut', 'provided': 209, 'inserted': 209, 'total_after_insert': 209}

Define rollover rules¶

For the purpose of defining calendar spreads, we must define rollover rules to specify which contract should be considered the front month and the various back months. Example rules are defined in quantrocket.master.rollover.yml, where we specify to rollover 10 business days before expiration. See the usage guide for more rollover rule options.

The master service looks for this file in the codeload directory, so move it there to install it:

In [5]:

# move file over unless it already exists
![ -e /codeload/quantrocket.master.rollover.y*ml ] && echo 'oops, the file already exists!' || mv quantrocket.master.rollover.yml /codeload/

Collect historical data¶

Next we collect 1-min historical data with the following parameters:

bar_type: The BID_ASK bar type provides the average bid and ask over the period of the bar.
outside_rth: We opt to include data from outside regular trading hours so that our moving averages and Bollinger Bands aren't jumpy.
shard: We shard/partition the database by month, resulting in a separate database per month (see the usage guide for more on sharding).
start_date: We enforce a start date of 2.5 years ago. IBKR only provides historical data for futures that expired less than 2 years ago, but the IBKR API will sometimes unsuccessfully look for data much earlier than that, which slows down data collection.

In [6]:

from quantrocket.history import create_ibkr_db
import pandas as pd

start_date = (pd.Timestamp.today() - pd.Timedelta(days=365*2.5)).date().isoformat()

create_ibkr_db("cl-1min-bbo", 
              universes="cl-fut", 
              bar_size="1 min", 
              bar_type="BID_ASK", 
              outside_rth=True,
              shard="month",
              start_date=start_date
             )

Out[6]:

{'status': 'successfully created quantrocket.v2.history.cl-1min-bbo.sqlite'}

Then we collect the data. Be prepared for intraday data collection to take some time (perhaps a day or so depending on several variables).

In [7]:

from quantrocket.history import collect_history
collect_history("cl-1min-bbo")

Out[7]:

{'status': 'the historical data will be collected asynchronously'}

Monitor flightlog for completion:

quantrocket.history: INFO [cl-1min-bbo] Collecting history from IBKR for 144 securities in cl-1min-bbo
...
quantrocket.history: INFO [cl-1min-bbo] Saved 22664 total records for 50 total securities to quantrocket.v2.history.cl-1min-bbo.sqlite

Note that it is normal to see warning messages like the following:

quantrocket.history: WARNING [cl-1min-bbo] IBKR reports CLZ8 FUT (sid QF000000023508) cannot be found in their system, this is commonly due to a stock delisting or switching exchanges or a derivative contract expiring, see http://qrok.it/h/err/200 for more help (error code 200: No security definition has been found for the request)

The CL futures chain collected earlier contains contracts that precede the approximately 2-year period for which IBKR provides historical data. The warning message reports that IBKR does not have historical data for this particular contract.

Historical Data Collection¶

Collect CL futures chain¶

Define universe of CL futures¶

Define rollover rules¶

Collect historical data¶

Next Up¶