Test the ABI compatibility of postgres across minor version updates
wrt timescale. Namely, compile the extension on an early minor version
of postgres and test it on the latest snapshot of the major branch.
Previously we just checked that we could upgrade from 0.1.0 until
latest but this missed some issues upgrading from intermediate
versions to latest. Now we check that all released versions can
successfully update.
Previously, for timezones w/o tz. The range_end and range_start were
defined as UTC, but the constraints on the table were written as as
the local time at the time of chunk creation. This does not work well
if timezones change over the life of the hypertable.
This change removes the dependency on local time for all timestamp
partitioning. Namely, the range_start and range_end remain as UTC
but the constraints are now always written in UTC too. Since old
constraints correctly describe the data currently in the chunks, the
update script to handle this change changes range_start and range_end
instead of the constraints.
Fixes#300.
The C source files can now be formatted with pgindent using
a new CMake build target.
The location of pgindent and dependencies are auto-discovered
by CMake or can be overridden on the command line.
The extension now works with PostgreSQL 10, while
retaining compatibility with version 9.6.
PostgreSQL 10 has numerous internal changes to functions and
APIs, which necessitates various glue code and compatibility
wrappers to seamlessly retain backwards compatiblity with older
versions.
Test output might also differ between versions. In particular,
the psql client generates version-specific output with `\d` and
EXPLAINs might differ due to new query optimizations. The test
suite has been modified as follows to handle these issues. First,
tests now use version-independent functions to query system
catalogs instead of using `\d`. Second, changes have been made to
the test suite to be able to verify some test outputs against
version-dependent reference files.
Moving the build system to CMake allows easy cross-platform
compiles, dependency checks, and more. In particular, CMake
allows us to easily build on Windows, and Visual Studio now
has native CMake support.
This is part of the ongoing effort to simplify the metadata tables and
removing any triggers on them that cause side effects.
This change includes the following:
- Remove the on_change_hypertable() trigger on the hypertable catalog
table.
- Remove the TRUNCATE blocking triggers on all metadata tables. If
we think such blocking is important, we should do this in an
event trigger or the processUtility hook.
- Put all SQL files in a single load_order.txt instead of splitting
across three distinct files. Now all SQL files are included in
update scripts as well for simplicity and consistency.
- As a result of removing triggers and related functions, the
setup_main() and restore_timescaledb() functions are no longer
needed. This also further simplifies the database restore process
as calling restore_timescaledb() is no longer needed (or possible).
- Refactor create_hypertable_row() to do more validation before
allocating a new hypertable ID. This avoids incrementing the serial
ID unnecessarily in case some validations fail.
This change refactors the chunk index handling to make better use
of standard PostgreSQL catalog information, while removing the
hypertable_index metadata table and associated triggers, including
those on the chunk_index table. The chunk_index table itself is
also simplified.
A benefit of this refactoring is that indexes are no longer
created using string mangling to construct the CREATE INDEX command
for a chunk, based on the string definition of the hypertable
index. Instead, indexes are created in C using proper index-related
internal data structures.
Chunk indexes can now also be renamed and are added in the parent
index tablespace. Changing tablespace on a hypertable index also
recurses to chunks, as expected. Default indexes that are added when
creating a hypertable use the hypertable's tablespace.
Creating Hypertable indexes with the CONCURRENTLY modifier is
currently blocked, due to unclear semantics regarding concurrent
creation over many tables, including how to deal with snapshots.
- Save an intermediate build image to avoid having to download APK
packages for every build
- Add debug symbols and assertions to extension build
- Install git and gdb by default for easier debugging
The update test script compares a TimescaleDB instance, which is
updated to the latest version from an older version, to a cleanly
installed one at the latest version. It does this by inserting the
same data on both and comparing the data and metadata tables. This
comparison breaks if the new version has changed some behavior (e.g.,
chunk indexes might use a different naming scheme), because the
updated version will retain the metadata related to the old behavior
while the latest version will create metadata according to the new
behavior.
To fix this issue, the update script now performs a database backup on
the old version and restores it on the cleanly installed version of
TimescaleDB. This retains the metadata in accordance with the old
behavior, although the new version will create new metadata according
to the new behavior. This has the further benefit of testing the
restore process during upgrades, which is probably the way people will
move to a new version of TimescaleDB if they do not update the
extension with the data in place.
The capture errors related to broken update scripts, the update test
now includes an integrity test for some metadata tables.
This PR add support for primary-key, foreign-key, unique, and exclusion constraints.
Previously supported are CHECK and NOT NULL constraints. Now, foreign key
constraints where a hypertable references a plain table is support
(while vice versa, with a plain table references a hypertable, is still not).
Using pg_dump/pg_restore for a database with hypertables will back
up all internal tables too, which is sometimes not desired. These
scripts allow one to backup regular PostgreSQL tables separately
and then pick and choose which hypertables to backup and restore.
This also provides a forward-compatible way of backing up and
restoring hypertables, since data is dumped to CSV and restored
by re-creating and re-inserting the data.
To allow upgrading versions older than the most recent one,
update scripts from older versions need to be added to the
extension. This is now done by including all files in the
file extra_extension_files.txt to the extension. Currently
that file needs to be manually updated when a new release
is built.
Also includes some update infrastructure changes. Main difference
is that sql/updates/ scripts are now split into pre- and post- files.
This is needed because table updates need to happen before reloading
functions (some functions checked for accessing right schema at
definition time). Meanwhile trigger updates need to happen after
since triggers must refer to valid functions at definition time.
SQL code is now split into setup, functions, and init files to
allow a subset to be run when the extension is updated. During
build, an update script is now also generated.
Release Docker builds are now provided in a separate repository,
https://github.com/timescale/timescaledb-docker.
Tests and development builds for Docker are now provided
by two new scripts in the scripts directory:
- `docker-build.sh` to build a development image from current sources.
- `docker-run-tests.sh` to run tests for current sources through a Docker container.
Clean up the table schema to get rid of legacy tables and functionality
that makes it more difficult to provide an upgrade path.
Notable changes:
* Get rid of legacy tables and code
* Simplify directory structure for SQL code
* Simplify table hierarchy: remove root table and make chunk tables
* inherit directly from main table
* Change chunk table suffix from _data to _chunk
* Simplify schema usage: _timescaledb_internal for internal functions.
* _timescaledb_catalog for metadata tables.
* Remove postgres_fdw dependency
* Improve code comments in sql code
This PR removes the need to run setup_timescaledb. It also fixes
pg_dump and pg_restore. Previously, the database would restore in
a broken state because trigger functions were never attached to
meta tables (since setup_timescaledb() was not run). However, attaching
those triggers at extension creation also causes problems since the data
copy happens after extension creation but we don't want triggers fired
on the data restored (that could cause duplicate rows, for example).
The solution to this chicken-and-egg problem in this PR is to have
a special configuration (GUC) variable `timescaledb.restoring` that,
if 'on', would prevent the extension from attaching triggers at
extension creation. Then, after restoration, you'd call
restore_timescaledb() to attach the triggers and unset the GUC above.
This procedure is documented in the README as part of this PR.
This commit adds an example how to run the Docker image so the
data is persisted between restarts based on our docker-run.sh
script. It also fixes a typo in the docker-run.sh script where
the default DATA_DIR was not correctly set.
The dblink extension is blacklisted by some cloud-hosting providers and
is an unnecessary dependency for single-node operation. Since we don't plan
to use dblink to implement clustering this PR removes the dependency.
If a user attempts to setup a database while not connecting using
the network, port is NULL and thus fails constraint checks. Instead,
we now use the default Postgres port of 5432 when this happens.
Also, setup_db() is renamed to setup_timescaledb() for clarity in
the presence of other extensions.
Currently TimescaleDB does not close chunks mid-insert, so large
batches will overfill a chunk. This commit adds a script to
split large CSV files into smaller batches to allow reasonable
closing of chunks.
Since create_hypertable() allows you to optionally specify a
partitioning column, it makes sense to default to one partition when
no column is specified and asking for the number of partitions when a
column is specified and the number of partitions is not (instead of
defaulting to one).
This patch also changes the order and type of partitioning-related
input arguments to create_hypertable() so that the number of
partitions can easily be specified alongside the partitioning column
and without type casting.
Setting up single node is now:
```
CREATE EXTENSION IF NOT EXISTS iobeamdb CASCADE;
select setup_single_node();
```
To setup a cluster do (on meta node):
```
CREATE EXTENSION IF NOT EXISTS iobeamdb CASCADE;
select set_meta();
```
on data node:
```
CREATE EXTENSION IF NOT EXISTS iobeamdb CASCADE;
select join_cluster('metadb', 'metahost');
```
This assumes that the commands are issued by the same user on both the
meta node and the data node. Otherwise the data node also has to
specify the user name to use when connecting to the meta node.
- Directory structure now matches common practices
- Regression tests now run with pg_regress via the PGXS infrastructure.
- Unit tests do not integrate well with pg_regress and have to be run
separately.
- Docker functionality is separate from main Makefile. Run with
`make -f docker.mk` to build and `make -f docker.mk run` to run
the database in a container.
Allowing database deployments that use the same database for both the
meta and node roles simplifies operations when iobeamdb is deployed on
a single-node. It allows the database to operate without any
cross-network communication (either through dblink or postgres_fdw) and
thus offers stronger consistency guarantees.
