Software Update

How to update

It's a good idea to make a backup of your docker-compose.yml before updating so that it's easy to roll back your deployment if needed.

To update to the latest version of QuantRocket, download the latest Docker Compose file.

For local deployments:

$ cd /path/to/quantrocket/folder
$ curl 'https://www.quantrocket.com/composefiles/latest/local/docker-compose.yml' -o docker-compose.yml

For cloud deployments:

$ cd /path/to/quantrocket/folder
$ curl 'https://www.quantrocket.com/composefiles/latest/cloud/docker-compose.yml' -o docker-compose.yml

(You can also download the Compose file from the downloads page.)

Then deploy the changes using Docker Compose:

$ docker-compose -p quantrocket up -d

You can reclaim disk space by removing old images that are no longer needed:

$ docker image prune --all

Versioning scheme

The format of QuantRocket version numbers is MAJOR.MINOR.PATCH (for example 1.3.1). As new features are added, typically the minor version is incremented, with the major version occasionally being incremented for large redesigns or backwards-incompatible changes. Patch versions are incremented for bug fixes and small updates in between successive minor version releases.

The QuantRocket version number is indicated in the Compose file, for example: x-quantrocket-version: '1.4.1'.

Due to QuantRocket's microservice architecture, each service/container has its own version number, for example: quantrocket/master:1.4.0. These version numbers will share the same major and minor version as the Compose file's x-quantrocket-version, but the patch versions can differ.

Some services/containers have version numbers with more than three numeric components to reflect the version number of an important piece of underlying software. For example, quantrocket/ibg version numbers have the format 1.3.973.0, where 973 reflects the version of IB Gateway embedded in the container.


Release Notes

Release 1.4.1

2018-11-23

This is a patch release that fixes a breaking change to container naming conventions introduced by Docker Compose version 1.23.0. See this community post for more details.


Release 1.4.0

2018-11-19

Highlights

  • Sharadar premium data integration. Sharadar provides harmonized point-in-time fundamentals and end of day prices for 14,000 US companies, active and delisted. Fully integrated with QuantRocket and Moonshot. See the usage guide, data guide, and pricing.
  • Codeload Library. This release introduces support for cloning sample code into your deployment from QuantRocket's browsable Codeload library.
  • Set credentials and configuration within JupyterLab. With this release, credentials and configuration that were previously set via the Configuration Wizard can now be set or edited from within JupyterLab using the CLI or Python API. This includes IB credentials, license key, S3 credentials, Papertrail integration, and more. See the release notes below for details.

API changes

  • For the CLI, quantrocket master listings has been deprecated in favor of quantrocket master collect. This change is to bring parity with the data collection terminology of other commands (such as quantrocket history collect). The deprecated command is still available but will print a warning.
  • quantrocket master collect/quantrocket.master.collect_listings now support multiple exchanges using the exchanges parameter. The exchange parameter (singular) is deprecated and will print a deprecation warning.
  • Delisted securities are now included by default in the master file (quantrocket master get/download_master_file). They can optionally be excluded with the --exclude-delisted/exclude_delisted parameter. Previously, delisted securities were excluded from the master file by default, but this diverged from QuantRocket's standard behavior of returning all query results by default.

quantrocket/codeload:1.4.0

quantrocket/countdown:1.4.0

quantrocket/db:1.4.0

quantrocket/flightlog:1.4.0

quantrocket/fundamental:1.4.0

quantrocket/history:1.4.0

quantrocket/ibg:1.4.973.0

quantrocket/license-service:1.4.0

quantrocket/master:1.4.0

  • Add API endpoints for collecting and querying Sharadar listings and for translating between IB and Sharadar domains
  • include delisted securities by default in query results
  • reliability improvements to IB listings collection

quantrocket/zipline:1.4.0

  • Update from Python 3.5 to 3.6

Release 1.3

Highlights

  • Short sale data. You can collect and query the number of shortable shares and associated borrow fees. See the usage guide
  • internal redesign of historical data collection, providing improved speed and reliability, especially for collecting small bar sizes such as 1-minute bars. See more details below in the quantrocket/history section.
  • addition of QGrid, providing Excel-like sorting and filtering of DataFrames. See the usage guide.
  • minute backtests now support making daily history calls (data.history(sid, 'price', 10, '1d') in addition to minute history calls (data.history(sid, 'price', 10, '1m'). This is the same behavior as on Quantopian. Minute bars are rolled up to daily bars during bundle ingestion to support this behavior.
  • supporting for ingesting minute databases into Zipline that don't fit in memory

API deprecations

  • For API endpoints related to data collection, the term "fetch" has been deprecated and replaced by the term "collect". The purpose is to improve the clarity of the API. The deprecated commands and functions are still available for backwards compatibility but will now print a warning message to remind you to switch over.
 CLIPython API
fundamentalfetch-financials -> collect-financials
fetch-estimates -> collect-estimates
fetch_reuters_financials -> collect_reuters_financials
fetch_reuters_estimates -> collect_reuters_estimates
historyfetch -> collectfetch_history -> collect_history
cancel_history_requests -> cancel_collections
masterfetch-calendar -> collect-calendarfetch_listings -> collect_listings
fetch_option_chains -> collect_option_chains
fetch_calendar -> collect_calendar
  • Not specifying a calendar with quantrocket zipline ingest (which causes the default NYSE calendar to be used) is deprecated and will now print a warning message. Associating data with the wrong calendar leads to confusing error messages; therefore it is best to be explicit.

quantrocket/account

1.3.0

2018-09-14

  • retrieve all exchange rates directly from ECB due to reliability problems with exchangeratesapi.io

quantrocket/countdown

1.3.0

2018-09-14

  • update to quantrocket-client==0.45.1

quantrocket/db

1.3.0

2018-09-14

  • add option to view details when listing databases
  • support for managing sharded databases stored in subdirectories
  • fix a bug that prevented pulling a single database from S3 if it didn't already exist locally

quantrocket/fundamental

1.3.0

2018-09-14

  • short sale data: collect and query number of shortable shares and associated borrow fees
  • performance improvement: query results are now cached for up to 24 hours. Running the same query will use the cache, yielding a faster response.
  • reliability improvements for Reuters fundamental data collection
  • update API and logging to reflect replacement of the term "fetch" with "collect". Commands and functions such as fetch-financials are now deprecated and renamed collect-financials. However, the deprecated commands and functions will continue to be available for backwards compatibility.

quantrocket/history

1.3.2

2018-09-25

  • reduce memory footprint of intraday data collection by using odo to write data to disk
  • add warning if insufficient disk space for estimated final database size

1.3.1

2018-09-18

This is a bug fix release.

  • Guarantee that query results are always sorted by date. Previously, in rare cases query results might be only approximately sorted by date. Both Moonshot and Zipline expect query results to be sorted by date.

1.3.0

2018-09-14

  • internal redesign of historical data collection, with the following benefits:
    • fewer timeouts, especially for small bar sizes such as 1-minute
    • total database size is estimated in addition to total runtime
    • faster startup time, especially for small bar sizes
    • new IB Gateways that appear during data collection are now discovered and used
    • improved reliability, including adaptive timeouts and better retry logic
    • more informative reporting via the detailed logs, including processing stages, response time report, and completion report
  • add support for sharding by both time and conid, to maximize query flexibility
  • update API and logging to reflect replacement of the term "fetch" with "collect". Commands and functions such as fetch are now deprecated and renamed collect. However, the deprecated commands and functions will continue to be available for backwards compatibility.

quantrocket/houston

1.3.0

2018-09-14

  • update to openresty 1.13.6.2 and noVNC 1.0.0
  • add tips to noVNC menu in IB Gateway GUI

quantrocket/ibg

1.3.973.0

2018-09-14

  • remove command for accessing IB logfile via Docker. Starting in IB Gateway 973, logfiles are stored in a binary format and must be viewed/exported through the GUI.
  • stability improvements

quantrocket/jupyter

1.3.0

2018-09-14

  • update to Anaconda 5.2.0
  • update to pyfolio 0.9.0
  • add QGrid 1.1.1
  • update to quantrocket-client 0.45.1 (view changes)
  • update to moonshot 0.6.0 (view changes)

quantrocket/master

1.3.0

2018-09-14

  • improve ETF identification for European ETFs
  • update API and logging to reflect replacement of the term "fetch" with "collect". Commands and functions such as fetch_listings are now deprecated and renamed collect_listings. However, the deprecated commands and functions will continue to be available for backwards compatibility.

quantrocket/moonshot

1.3.0

2018-09-14

  • update to moonshot 0.6.0 (view changes)
  • update to Anaconda 5.2.0
  • update to TA-Lib 0.4.16
  • update to quantrocket-client==0.45.1 (view changes)

quantrocket/satellite

1.3.0

2018-09-14

  • update to quantrocket-client==0.45.1

quantrocket/zipline

1.3.1

2018-09-17

This is a bug fix release.

  • update to quantrocket-zipline-extensions 1.3.3: fixes bug to allow daily roll-up of minute bars to work with multi-security bundles (view changes)

1.3.0

2018-09-14

  • update to zipline 1.3
  • minute backtests now support making daily history calls (data.history(sid, 'price', 10, '1d') in addition to minute history calls (data.history(sid, 'price', 10, '1m'). This is the same behavior as on Quantopian. Minute bars are rolled up to daily bars during bundle ingestion to support this behavior.
  • add option to filter history database by date range, universes, or conids before ingestion
  • redesign minute bar ingestion to reduce memory usage by loading one security at a time
  • more flexible options for removing ingested data
  • update to quantrocket-zipline-extensions 1.3.2 (view changes)
  • Not specifying a calendar with quantrocket zipline ingest (which causes the default NYSE calendar to be used) is deprecated and will now print a warning message. Associating data with the wrong calendar leads to confusing error messages; therefore it is best to be explicit.

Release 1.2

quantrocket/account

1.2.1

2018-08-24

  • update exchangeratesapi.io URL after upstream change

1.2.0

2018-05-30

  • use exchangeratesapi.io instead of fixer.io, and get full rates history from ECB website
  • add account portfolio endpoint

quantrocket/codeload

1.2.0

2018-05-30

  • add optional OVERWRITE_EXISTING env var to overwrite existing files (default is to ignore existing)

quantrocket/countdown

1.2.0

2018-05-30

  • log cron mail to logspout
  • update to quantrocket-client==0.40.3

quantrocket/db

1.2.0

2018-05-30

  • disable s3 pull on launch if env var S3PULL_ON_LAUNCH=false

quantrocket/fundamental

1.2.0

2018-05-30

  • include restatements by default in query financials endpoint, and always use FiscalPeriodEndDate for date filters (to simplify)
  • include delisted securities when returning fundamentals (unless universe does not contain any)

quantrocket/history

1.2.2

2018-07-12

1.2.1

2018-06-05

  • fix memory leak that caused memory usage to grow indefinitely during long-running data collection

1.2.0

2018-05-30

  • add option to override universes when collecting history
  • switch to primary exchange on error "BEST queries are not supported for this contract"
  • allow collecting history availability only, and allow querying availability
  • add support for time-sharded databases
  • cumulative daily calculations for intraday databases
  • include delisted securities when returning history (unless universe does not contain any)
  • add delist_missing option
  • add between_times param to create-db endpoint

quantrocket/houston

1.2.0

2018-05-30

  • remove support for Docker Cloud after service discontinuation
  • don't log jupyter access log because too noisy

quantrocket/ibg

973.1

2018-08-01

  • fix an issue preventing connecting a live account

973.0

2018-07-27

  • update to IB Gateway version 973
  • automatically restart IB Gateway if unresponsive for 60 seconds (doesn't apply if 2-factor authentication is enabled; number of seconds can be set with environment variable IBG_FORCE_RESTART)
  • stability and reliability improvements

quantrocket/jupyter

1.2.2

2018-07-11

  • update to quantrocket-client==0.41.0 with support for segmented backtests (view changes)
  • update to moonshot==0.5.3 (view changes)

1.2.1

2018-06-18

Starting with this version, the versioning scheme for quantrocket/jupyter is changed to mirror other services. (Previously the versioning scheme mirrored the underlying Anaconda version.)

  • update to quantrocket-client==0.40.7 (view changes)
  • update to alphalens==3.2.0

quantrocket/launchpad

1.2.1

2018-07-24

  • allow sec types IND and FOP in quantrocket.launchpad.permissions.yml

quantrocket/license-service

1.2.1

2018-07-19

  • support for multiple accounts which are denominated in different base currencies

quantrocket/master

1.2.1

2018-06-12

  • save tick size rules to securities master database at the time listings are collected
  • improve error handling of trading calendar endpoints

1.2.0

2018-05-30

  • allow querying DateDelisted
  • change query logic to AND universes and exchange/currencies/etc instead of ORing them
  • add endpoint for rounding prices to tick sizes
  • add endpoints for collecting and querying trading hours

quantrocket/moonshot

1.2.2

2018-07-11

1.2.1

2018-06-18

1.2.0

2018-05-30

  • add trade endpoint
  • better handling of passing on-the-fly params
  • return CSV by default instead of PDF in backtest and paramscan endpoints
  • add codeload to path instead of codeload_moonshot (for parity with Jupyter)

quantrocket/satellite

1.2.0

2018-05-30

  • update to quantrocket-client==0.40.3

Release 1.1

quantrocket/account

1.1.1

2018-03-12

  • better type inference with JSON output: integers are now returned as integers instead of as floats
  • exchange rates from a base currency to itself are now stored as a convenience

quantrocket/countdown

1.1.1

2018-03-20

  • update to quantrocket-client==0.31.0

quantrocket/db

1.1.1

2018-03-20

  • improve handling of S3 pulls on service launch to speed up launch time

quantrocket/fundamental

1.1.1

2018-03-20

  • better type inference with JSON output: integers are now returned as integers instead of as floats

quantrocket/history

1.1.1

2018-03-14

  • improve handling of IB response timeouts

    Previously when the IB API failed to respond to a request after 30 seconds, the history service would log the failure and continue with the next request for the same security. Since non-responses from the IB API are usually temporary, and since the history service fills data from back to forwards (old to new), this behavior could lead to gaps in the data. Now, after a non-response from the IB API, the history service will skip to the next security to avoid creating data gaps.

  • make runtime estimates more informative by including average response time and number of remaining requests

  • shorten request durations for intraday bars to speed up response times

quantrocket/houston

1.1.2

2018-03-20

  • redirect root path to JupyterLab; requires quantrocket/jupyter:4.4.0.9 or higher

quantrocket/ibg

972.0

2018-05-30

  • update to IB Gateway version 972

971.0

2018-05-04

  • update to IB Gateway version 971

970.2

2018-05-02

  • add credentials endpoint for setting/getting IB credentials on running container

970.0

2018-03-13

  • update to IB Gateway version 970

quantrocket/jupyter

4.4.0.10

2018-05-30

  • update to quantrocket-client==0.40.3
  • update to moonshot==0.5.2

4.4.0.9

2018-03-16

  • update from classic Jupyter to JupyterLab
  • add QuantRocket command line interface (CLI) to Jupyter terminal. With this release it is no longer necessary to install the QuantRocket client separately.
  • add custom command to open IB Gateway GUI from Jupyter interface
  • automatically packagify codeload directory and subdirectories and make them importable using import codeload

    Any folders under the codeload directory which are valid Python package names (lowercase letters and underscores) and which contain one or more .py files will automatically be packagified, that is, an __init__.py file will be created if it doesn't already exist. Packages can be imported from the top-level codeload package using standard Python dot notation. For example, a function called my_function located in codeload/moonshot/utils.py can be imported in Jupyter notebooks as follows:

    from codeload.moonshot.utils import my_function
  • update to Alphalens 0.3.0

quantrocket/master

1.1.1

2018-03-16

  • add Estonia, Latvia, Lithuania exchanges
  • better type inference with JSON output: integers are now returned as integers instead of as floats

quantrocket/moonshot

1.1.1

2018-03-20

  • update to Moonchart 0.3.1 (fixes bug with plotting gross returns)

quantrocket/zipline

1.1.1.11

1.1.1.9

  • better handling of minute bar ingestion

1.1.1.8