Replace hardcoded database name from regression tests with :TEST_DBNAME
Remove creation of database single_2 from test runner and add it to
bgw_launcher and loader test since no other tests used those
use SQL comments in test scripts
Fixes get_create_command to produce a valid create_hypertable
call when the column name is a keyword
Add test.execute_sql helper function to test support functions
TimescaleDB has always supported functions on closed (space)
dimension, i.e., for hash partitioning. However, functions have not
been supported on open (time) dimensions, instead requiring columns to
have a supported time type (e.g, integer or timestamp). This restricts
the tables that can be time partitioned. Tables with custom "time"
types, which can be transformed by a function expression into a
supported time type, are not supported.
This change generalizes partitioning so that both open and closed
dimensions can have an associated partitioning function that
calculates a dimensional value. Fortunately, since we already support
functions on closed dimensions, the changes necessary to support this
on any dimension are minimal. Thus, open dimensions now support an
(optional) partitioning function that transforms the input type to a
supported time type (e.g., integer or timestamp type). Any indexes on
such dimensional columns become expression indexes.
Tests have been added for chunk expansion and the hashagg and sort
transform optimizations on tables that are using a time partitioning
function.
Currently, not all of these optimizations are well supported, but this
could potentially be fixed in the future.
Previously, the pg_dump test was broken because it is not possible to reference psql variables
from inside bash commands run through psql. This is fixed by hardcoding the username passed to
the bash commands inside the test.
Also, we changed the insert block trigger preventing inserts into hypertable to a non-internal
trigger, because internal triggers are not dumped by pg_dump. We need to dump the trigger so that
it is already in place after a pg_restore, to prevent users from accidentally inserting rows into
a hypertable while timescaledb_restoring=on.
A hypertable's root table should never have any tuples, but it can
acquire tuples by accident if the TimescaleDB extension is not
preloaded or `timescaledb.restoring` is set to ON.
To avoid the above issue, a hypertable's root table now has a
(internal) trigger that generates an error when tuples are
inserted. This preserves the integrity of the hypertable even when
restoring or the extension is not preloaded.
An internal trigger has the advantage of being mostly transparent to
users (e.g., it doesn't show up with \d) and it is not inherited by
chunks, so it needs no special handling to avoid adding it to chunks.
The blocking trigger is added in the update script and if it is
detected that the hypertable's root table has data in the process, the
update will fail with an error and instructions on how to fix.
This adds a simple check to enforce that partitioning functions
are IMMUTABLE. This is a requirement since a partitioning function
must always return the same value given the same input.
This fixes an issue with the `if_not_exists` option to add_dimension()
that caused the command to fail on a non-empty table. When this option
is set, the command should only fail if the dimension does not exists
and the table is non-empty.
Tables can now hold existing data, which is optionally migrated from
the main table to chunks when create_hypertable() is called.
The data migration is similar to the COPY path, with the single
difference that the inserted/copied tuples come from an existing table
instead of being read from a file. After the data has been migrated,
the main table is truncated.
One potential downside of this approach is that all of this happens in
a single transaction, which means that the table is blocked while
migration is ongoing, preventing inserts by other transactions.
The functions for adding and updating dimensions have been refactored
in C to:
- improve usage of proper error codes
- make messages that better conform with the PostgreSQL standard.
- improve security by avoiding that lots of code run under SECURITY DEFINER
A new if_not_exists option has also been added to add_dimension() and
a the number of partitions can now be set using the new
set_number_partitions() function.
A bug in the validation of smallint time intervals has been fixed. The
previous code didn't check for intervals > 0 and smallint intervals
accepted values up to UINT16_MAX instead of INT16_MAX.
A hypertable's associated schema is used to create and store internal
data tables (chunks). A hypertable creates tables in that schema,
typically with full superuser permissions, regardless of whether the
hypertable's owner or the current user have permissions for the schema.
If the schema doesn't exist, the hypertable will create it when
creating the first chunk, even though the user or table owner does
not have permissions to create schemas in the database.
This change adds proper permissions checks to create_hypertable() so
that users cannot create hypertables with a custom associated schema
unless they have the proper permissions on the schema or the database.
Chunks are also no longer created with internal schema permissions if
the associated schema is something different from the internal schema.
Attaching tablespaces to hypertables is now handled
in native code, with improved permissions checking and
caching of tablespaces in the Hypertable data object.
The user should be able to add time dimensions using INTERVAL when
the column type is TIMESTAMP/TIMESTAMPTZ/DATE, so this change adds
that support.
Additionally it adds some additional tests and checks for
add_dimension, e.g., a nice error when the table is not a
hypertable.
This change reduces the usage of SECURITY DEFINER on SQL
functions and fixes related permissions issues. It also
properly checks hypertable permissions relative the current_user
instead of the session_user, which otherwise breaks SET ROLE,
among other things.
The extension now works with PostgreSQL 10, while
retaining compatibility with version 9.6.
PostgreSQL 10 has numerous internal changes to functions and
APIs, which necessitates various glue code and compatibility
wrappers to seamlessly retain backwards compatiblity with older
versions.
Test output might also differ between versions. In particular,
the psql client generates version-specific output with `\d` and
EXPLAINs might differ due to new query optimizations. The test
suite has been modified as follows to handle these issues. First,
tests now use version-independent functions to query system
catalogs instead of using `\d`. Second, changes have been made to
the test suite to be able to verify some test outputs against
version-dependent reference files.
Add check that time dimensions are set as NOT NULL in the
main table that a hypertable is created from. If it is not
set, the constraint will be added.
This commit moves a lot of test setup logic to runner.sh. Also
passes the right commands to the regression infrastructure to create
appropriate users and run tests as a regular user.
All regression tests will now use a non-superuser unless superuser is
necessary. This PR is a meant to prevent things like issue #226.
This PR also fixes some more permission bugs found during this testing.
Using pg_dump/pg_restore for a database with hypertables will back
up all internal tables too, which is sometimes not desired. These
scripts allow one to backup regular PostgreSQL tables separately
and then pick and choose which hypertables to backup and restore.
This also provides a forward-compatible way of backing up and
restoring hypertables, since data is dumped to CSV and restored
by re-creating and re-inserting the data.
Previously the default chunk time in microseconds was too large
for a SMALLINT or INTEGER field. Now, we only assign a default
value if the type is TIMESTAMP or TIMESTAMPTZ. Integer timestamps,
such as SMALLINT, INTEGER, and BIGINT, need to be explicitly set
since only the user knows what units the numbers represent.
Further, we check to make sure the chunk time interval is not too
large for SMALLINT and INTEGER so as to avoid confusing problems
later when the user goes to insert.
A new public-facing API `add_dimension(table, column, ...)`
makes it possible to add additional dimensions (partitioning
columns) to a hypertable.
Currently, new dimension can only be added to empty tables.
The code has also been refactored with a corresponding
internal function that is called by both `add_dimension()`
and `create_hypertable()`.
Previously create_hypertable() would throw an error when called on
an already existing hypertable. This can now be skipped by setting
if_not_exists argument to true.
Previously, each test set their own (although mostly the same)
configuration for log output and error verbosity. This is now set
globally in the test runner so that tests only need to set these
configuration parameters if they need to override the defaults. The
log verbosity is also reduced so that errors aren't generated with the
line number of the source file that output the error. Line numbers in
the output can break tests when upgrading to a new PostgreSQL version
that outputs a different line number.
Since create_hypertable() allows you to optionally specify a
partitioning column, it makes sense to default to one partition when
no column is specified and asking for the number of partitions when a
column is specified and the number of partitions is not (instead of
defaulting to one).
This patch also changes the order and type of partitioning-related
input arguments to create_hypertable() so that the number of
partitions can easily be specified alongside the partitioning column
and without type casting.
Setting up single node is now:
```
CREATE EXTENSION IF NOT EXISTS iobeamdb CASCADE;
select setup_single_node();
```
To setup a cluster do (on meta node):
```
CREATE EXTENSION IF NOT EXISTS iobeamdb CASCADE;
select set_meta();
```
on data node:
```
CREATE EXTENSION IF NOT EXISTS iobeamdb CASCADE;
select join_cluster('metadb', 'metahost');
```
This assumes that the commands are issued by the same user on both the
meta node and the data node. Otherwise the data node also has to
specify the user name to use when connecting to the meta node.
- Directory structure now matches common practices
- Regression tests now run with pg_regress via the PGXS infrastructure.
- Unit tests do not integrate well with pg_regress and have to be run
separately.
- Docker functionality is separate from main Makefile. Run with
`make -f docker.mk` to build and `make -f docker.mk run` to run
the database in a container.
