Release Notes

Changes since the initial release of QuantRocket are documented here.

Version numbers

The format of QuantRocket version numbers is MAJOR.MINOR.PATCH (for example 1.3.1). As new features are added, typically the minor version is incremented, with the major version occasionally being incremented for large redesigns or backwards-incompatible changes. Patch versions are incremented for bug fixes and small updates in between successive minor version releases.

Due to QuantRocket's microservice architecture, each service/container has its own version number. To ensure compatibility between containers, all services/containers in your deployment should share the same major and minor version. They need not share the same patch version. For example, it's okay to mix quantrocket/history:1.3.2 with quantrocket/master:1.3.0 but not with quantrocket/master:1.2.2.

Some services/containers have version numbers with more than three numeric components to reflect the version number of an important piece of underlying software. For example, quantrocket/ibg version numbers have the format 1.3.973.0, where 973 reflects the version of IB Gateway embedded in the container.

How to update

There are two options for updating your versions: a full update or a partial update. Use a full update when updating to a new major or minor version (for example, from 1.2.x to 1.3.x). Use a partial update when updating to a new patch version (for example, from 1.3.1 to 1.3.2).

It's a good idea to make a backup of your docker-compose.yml before upgrading so that it's easy to roll back your deployment if needed.

Full update

At any time, the Configuration Wizard can be used to generate a YAML file containing the latest versions of each image. You can update your entire deployment by generating a new YAML file to replace your existing YAML file. This ensures that all service dependencies are satisfied and that the behavior of your deployment matches the website documentation.

Partial update

For a partial update, modify your existing YAML file to reference the new version of the image. For example, to update to version 1.3.0 of the history service, modify the relevant section of your YAML to read:

history:
    image: 'quantrocket/history:1.3.0'

Deploy the update

Whether you choose a full or partial update, you can deploy the changes as normal using Docker Compose:

$ docker-compose -p quantrocket up -d

You can reclaim disk space by removing old images that are no longer needed:

$ docker image prune --all

Release 1.3

Highlights

  • Short sale data. You can collect and query the number of shortable shares and associated borrow fees. See the usage guide
  • internal redesign of historical data collection, providing improved speed and reliability, especially for collecting small bar sizes such as 1-minute bars. See more details below in the quantrocket/history section.
  • addition of QGrid, providing Excel-like sorting and filtering of DataFrames. See the usage guide.
  • minute backtests now support making daily history calls (data.history(sid, 'price', 10, '1d') in addition to minute history calls (data.history(sid, 'price', 10, '1m'). This is the same behavior as on Quantopian. Minute bars are rolled up to daily bars during bundle ingestion to support this behavior.
  • supporting for ingesting minute databases into Zipline that don't fit in memory

API deprecations

  • For API endpoints related to data collection, the term "fetch" has been deprecated and replaced by the term "collect". The purpose is to improve the clarity of the API. The deprecated commands and functions are still available for backwards compatibility but will now print a warning message to remind you to switch over.
 CLIPython API
fundamentalfetch-financials -> collect-financials
fetch-estimates -> collect-estimates
fetch_reuters_financials -> collect_reuters_financials
fetch_reuters_estimates -> collect_reuters_estimates
historyfetch -> collectfetch_history -> collect_history
cancel_history_requests -> cancel_collections
masterfetch-calendar -> collect-calendarfetch_listings -> collect_listings
fetch_option_chains -> collect_option_chains
fetch_calendar -> collect_calendar
  • Not specifying a calendar with quantrocket zipline ingest (which causes the default NYSE calendar to be used) is deprecated and will now print a warning message. Associating data with the wrong calendar leads to confusing error messages; therefore it is best to be explicit.

quantrocket/account

1.3.0

2018-09-14

  • retrieve all exchange rates directly from ECB due to reliability problems with exchangeratesapi.io

quantrocket/countdown

1.3.0

2018-09-14

  • update to quantrocket-client==0.45.1

quantrocket/db

1.3.0

2018-09-14

  • add option to view details when listing databases
  • support for managing sharded databases stored in subdirectories
  • fix a bug that prevented pulling a single database from S3 if it didn't already exist locally

quantrocket/fundamental

1.3.0

2018-09-14

  • short sale data: collect and query number of shortable shares and associated borrow fees
  • performance improvement: query results are now cached for up to 24 hours. Running the same query will use the cache, yielding a faster response.
  • reliability improvements for Reuters fundamental data collection
  • update API and logging to reflect replacement of the term "fetch" with "collect". Commands and functions such as fetch-financials are now deprecated and renamed collect-financials. However, the deprecated commands and functions will continue to be available for backwards compatibility.

quantrocket/history

1.3.2

2018-09-25

  • reduce memory footprint of intraday data collection by using odo to write data to disk
  • add warning if insufficient disk space for estimated final database size

1.3.1

2018-09-18

This is a bug fix release.

  • Guarantee that query results are always sorted by date. Previously, in rare cases query results might be only approximately sorted by date. Both Moonshot and Zipline expect query results to be sorted by date.

1.3.0

2018-09-14

  • internal redesign of historical data collection, with the following benefits:
    • fewer timeouts, especially for small bar sizes such as 1-minute
    • total database size is estimated in addition to total runtime
    • faster startup time, especially for small bar sizes
    • new IB Gateways that appear during data collection are now discovered and used
    • improved reliability, including adaptive timeouts and better retry logic
    • more informative reporting via the detailed logs, including processing stages, response time report, and completion report
  • add support for sharding by both time and conid, to maximize query flexibility
  • update API and logging to reflect replacement of the term "fetch" with "collect". Commands and functions such as fetch are now deprecated and renamed collect. However, the deprecated commands and functions will continue to be available for backwards compatibility.

quantrocket/houston

1.3.0

2018-09-14

  • update to openresty 1.13.6.2 and noVNC 1.0.0
  • add tips to noVNC menu in IB Gateway GUI

quantrocket/ibg

1.3.973.0

2018-09-14

  • remove command for accessing IB logfile via Docker. Starting in IB Gateway 973, logfiles are stored in a binary format and must be viewed/exported through the GUI.
  • stability improvements

quantrocket/jupyter

1.3.0

2018-09-14

  • update to Anaconda 5.2.0
  • update to pyfolio 0.9.0
  • add QGrid 1.1.1
  • update to quantrocket-client 0.45.1 (view changes)
  • update to moonshot 0.6.0 (view changes)

quantrocket/master

1.3.0

2018-09-14

  • improve ETF identification for European ETFs
  • update API and logging to reflect replacement of the term "fetch" with "collect". Commands and functions such as fetch_listings are now deprecated and renamed collect_listings. However, the deprecated commands and functions will continue to be available for backwards compatibility.

quantrocket/moonshot

1.3.0

2018-09-14

  • update to moonshot 0.6.0 (view changes)
  • update to Anaconda 5.2.0
  • update to TA-Lib 0.4.16
  • update to quantrocket-client==0.45.1 (view changes)

quantrocket/satellite

1.3.0

2018-09-14

  • update to quantrocket-client==0.45.1

quantrocket/zipline

1.3.1

2018-09-17

This is a bug fix release.

  • update to quantrocket-zipline-extensions 1.3.3: fixes bug to allow daily roll-up of minute bars to work with multi-security bundles (view changes)

1.3.0

2018-09-14

  • update to zipline 1.3
  • minute backtests now support making daily history calls (data.history(sid, 'price', 10, '1d') in addition to minute history calls (data.history(sid, 'price', 10, '1m'). This is the same behavior as on Quantopian. Minute bars are rolled up to daily bars during bundle ingestion to support this behavior.
  • add option to filter history database by date range, universes, or conids before ingestion
  • redesign minute bar ingestion to reduce memory usage by loading one security at a time
  • more flexible options for removing ingested data
  • update to quantrocket-zipline-extensions 1.3.2 (view changes)
  • Not specifying a calendar with quantrocket zipline ingest (which causes the default NYSE calendar to be used) is deprecated and will now print a warning message. Associating data with the wrong calendar leads to confusing error messages; therefore it is best to be explicit.

Release 1.2

quantrocket/account

1.2.1

2018-08-24

  • update exchangeratesapi.io URL after upstream change

1.2.0

2018-05-30

  • use exchangeratesapi.io instead of fixer.io, and get full rates history from ECB website
  • add account portfolio endpoint

quantrocket/codeload

1.2.0

2018-05-30

  • add optional OVERWRITE_EXISTING env var to overwrite existing files (default is to ignore existing)

quantrocket/countdown

1.2.0

2018-05-30

  • log cron mail to logspout
  • update to quantrocket-client==0.40.3

quantrocket/db

1.2.0

2018-05-30

  • disable s3 pull on launch if env var S3PULL_ON_LAUNCH=false

quantrocket/fundamental

1.2.0

2018-05-30

  • include restatements by default in query financials endpoint, and always use FiscalPeriodEndDate for date filters (to simplify)
  • include delisted securities when returning fundamentals (unless universe does not contain any)

quantrocket/history

1.2.2

2018-07-12

1.2.1

2018-06-05

  • fix memory leak that caused memory usage to grow indefinitely during long-running data collection

1.2.0

2018-05-30

  • add option to override universes when collecting history
  • switch to primary exchange on error "BEST queries are not supported for this contract"
  • allow collecting history availability only, and allow querying availability
  • add support for time-sharded databases
  • cumulative daily calculations for intraday databases
  • include delisted securities when returning history (unless universe does not contain any)
  • add delist_missing option
  • add between_times param to create-db endpoint

quantrocket/houston

1.2.0

2018-05-30

  • remove support for Docker Cloud after service discontinuation
  • don't log jupyter access log because too noisy

quantrocket/ibg

973.1

2018-08-01

  • fix an issue preventing connecting a live account

973.0

2018-07-27

  • update to IB Gateway version 973
  • automatically restart IB Gateway if unresponsive for 60 seconds (doesn't apply if 2-factor authentication is enabled; number of seconds can be set with environment variable IBG_FORCE_RESTART)
  • stability and reliability improvements

quantrocket/jupyter

1.2.2

2018-07-11

  • update to quantrocket-client==0.41.0 with support for segmented backtests (view changes)
  • update to moonshot==0.5.3 (view changes)

1.2.1

2018-06-18

Starting with this version, the versioning scheme for quantrocket/jupyter is changed to mirror other services. (Previously the versioning scheme mirrored the underlying Anaconda version.)

  • update to quantrocket-client==0.40.7 (view changes)
  • update to alphalens==3.2.0

quantrocket/launchpad

1.2.1

2018-07-24

  • allow sec types IND and FOP in quantrocket.launchpad.permissions.yml

quantrocket/license-service

1.2.1

2018-07-19

  • support for multiple accounts which are denominated in different base currencies

quantrocket/master

1.2.1

2018-06-12

  • save tick size rules to securities master database at the time listings are collected
  • improve error handling of trading calendar endpoints

1.2.0

2018-05-30

  • allow querying DateDelisted
  • change query logic to AND universes and exchange/currencies/etc instead of ORing them
  • add endpoint for rounding prices to tick sizes
  • add endpoints for collecting and querying trading hours

quantrocket/moonshot

1.2.2

2018-07-11

1.2.1

2018-06-18

1.2.0

2018-05-30

  • add trade endpoint
  • better handling of passing on-the-fly params
  • return CSV by default instead of PDF in backtest and paramscan endpoints
  • add codeload to path instead of codeload_moonshot (for parity with Jupyter)

quantrocket/satellite

1.2.0

2018-05-30

  • update to quantrocket-client==0.40.3

Release 1.1

quantrocket/account

1.1.1

2018-03-12

  • better type inference with JSON output: integers are now returned as integers instead of as floats
  • exchange rates from a base currency to itself are now stored as a convenience

quantrocket/countdown

1.1.1

2018-03-20

  • update to quantrocket-client==0.31.0

quantrocket/db

1.1.1

2018-03-20

  • improve handling of S3 pulls on service launch to speed up launch time

quantrocket/fundamental

1.1.1

2018-03-20

  • better type inference with JSON output: integers are now returned as integers instead of as floats

quantrocket/history

1.1.1

2018-03-14

  • improve handling of IB response timeouts

    Previously when the IB API failed to respond to a request after 30 seconds, the history service would log the failure and continue with the next request for the same security. Since non-responses from the IB API are usually temporary, and since the history service fills data from back to forwards (old to new), this behavior could lead to gaps in the data. Now, after a non-response from the IB API, the history service will skip to the next security to avoid creating data gaps.

  • make runtime estimates more informative by including average response time and number of remaining requests

  • shorten request durations for intraday bars to speed up response times

quantrocket/houston

1.1.2

2018-03-20

  • redirect root path to JupyterLab; requires quantrocket/jupyter:4.4.0.9 or higher

quantrocket/ibg

972.0

2018-05-30

  • update to IB Gateway version 972

971.0

2018-05-04

  • update to IB Gateway version 971

970.2

2018-05-02

  • add credentials endpoint for setting/getting IB credentials on running container

970.0

2018-03-13

  • update to IB Gateway version 970

quantrocket/jupyter

4.4.0.10

2018-05-30

  • update to quantrocket-client==0.40.3
  • update to moonshot==0.5.2

4.4.0.9

2018-03-16

  • update from classic Jupyter to JupyterLab
  • add QuantRocket command line interface (CLI) to Jupyter terminal. With this release it is no longer necessary to install the QuantRocket client separately.
  • add custom command to open IB Gateway GUI from Jupyter interface
  • automatically packagify codeload directory and subdirectories and make them importable using import codeload

    Any folders under the codeload directory which are valid Python package names (lowercase letters and underscores) and which contain one or more .py files will automatically be packagified, that is, an __init__.py file will be created if it doesn't already exist. Packages can be imported from the top-level codeload package using standard Python dot notation. For example, a function called my_function located in codeload/moonshot/utils.py can be imported in Jupyter notebooks as follows:

    from codeload.moonshot.utils import my_function
  • update to Alphalens 0.3.0

quantrocket/master

1.1.1

2018-03-16

  • add Estonia, Latvia, Lithuania exchanges
  • better type inference with JSON output: integers are now returned as integers instead of as floats

quantrocket/moonshot

1.1.1

2018-03-20

  • update to Moonchart 0.3.1 (fixes bug with plotting gross returns)

quantrocket/zipline

1.1.1.11

1.1.1.9

  • better handling of minute bar ingestion

1.1.1.8