timescaledb/sql/updates

Extension updates

This directory contains "modfiles" (SQL scripts) with modifications that are applied when updating from one version of the extension to another.

The actual update scripts are compiled from modfiles by concatenating them with the current source code (which should come at the end of the resulting update script). Update scripts can "jump" several versions by using multiple modfiles in order. There are two types of modfiles:

• Transition modfiles named <from>-<to>.sql, where from and to indicate the (adjacent) versions transitioning between. Transition modfiles are concatenated to form the lineage from an origin version to any later version.
• Origin modfiles named .sql, which are included only in update scripts that origin at the particular version given in the name. So, for instance, 0.7.0.sql is only included in the script moving from 0.7.0 to the current version, but not in, e.g., the update script for 0.4.0 to the current version. These files typically contain fixes for bugs that are specific to the origin version, but are no longer present in the transition modfiles.

To ensure that this update process works, there are a few principles to consider.

1. Modfiles should, in most cases, only contain ALTER or DROP commands that change or remove objects. In some cases, modifications of metadata are also necessary.
2. DROP FUNCTION needs to be idempotent. In most cases that means commands should have an IF EXISTS clause. The reason is that some modfiles might try to, e.g., DROP functions that aren't present because they only exist in an intermediate version of the database, which is skipped over.
3. Modfiles cannot rely on objects or functions that are present in a previous version of the extension. This is because a particular modfile should work when upgrading from any previous version of the extension, where those functions or objects aren't present yet.
4. The creation of new metadata tables need to be part of modfiles, similar to ALTERs of such tables. Otherwise, later modfiles cannot rely on those tables being present.
5. When creating a new aggregate, the CREATE statement should be added to both aggregate.sql AND an update file. aggregate.sql is run once when TimescaleDB is installed so adding a definition in an update file is the only way to ensure that upgrading users get the new function.

Notes on post_update.sql We use a special config var (timescaledb.update_script_stage ) to notify that dependencies have been setup and now timescaledb specific queries can be enabled. This is useful if we want to, for example, modify objects that need timescaledb specific syntax as part of the extension update). The scripts in post_update.sql are executed as part of the ALTER EXTENSION stmt.

Note that modfiles that contain no changes need not exist as a file. Transition modfiles must, however, be listed in the CMakeLists.txt file in the parent directory for an update script to be built for that version.

Extension downgrades

You can enable the generation of a downgrade file by setting GENERATE_DOWNGRADE_SCRIPT to ON, for example:

./bootstrap -DGENERATE_DOWNGRADE_SCRIPT=ON

To support downgrades to previous versions of the extension, it is necessary to execute CMake from a Git repository since the generation of a downgrade script requires access to the previous version files that are used to generate an update script. In addition, we only generate a downgrade script to the immediate preceeding version and not to any other preceeding versions.

The source and target versions are found in be found in the file version.config file in the root of the source tree, where version is the source version and downgrade_to_version is the target version. Note that we have a separate field for the downgrade.

A downgrade file consists of:

• A prolog that is retrieved from the target version.
• A version-specific piece of code that exists on the source version.
• An epilog that is retrieved from the target version.

The prolog consists of the files mentioned in the PRE_UPDATE_FILES variable in the target version of cmake/ScriptFiles.cmake.

The version-specific code is found in the source version of the file sql/updates/reverse-dev.sql.

The epilog consists of the files in variables SOURCE_FILES, SET_POST_UPDATE_STAGE, POST_UPDATE_FILES, and UNSET_UPDATE_STAGE in that order.

For downgrades to work correctly, some rules need to be followed:

1. If you add new objects in sql/updates/latest-dev.sql, you need to remove them in the version-specific downgrade file. The sql/updates/pre-update.sql in the target version do not know about objects created in the source version, so they need to be dropped explicitly.
2. Since sql/updates/pre-update.sql can be executed on a later version of the extension, it might be that some objects have been removed and do not exist. Hence DROP calls need to use IF NOT EXISTS.

Note that, in contrast to update scripts, downgrade scripts are not built by composing several downgrade scripts into a more extensive downgrade script. The downgrade scripts are intended to be use only in special cases and are not intended to be use to move up and down between versions at will, which is why we only generate a downgrade script to the immediately preceeding version.

When releasing a new version

When releasing a new version, please rename the file reverse-dev.sql to <version>--<downgrade_to_version>.sql and add that name to REV_FILES variable in the sql/CMakeLists.txt. This will allow generation of downgrade scripts for any version in that list, but it is currently not added.