When creating a hypertable, we create default indexes on the partition
column, which requires that the column be a part of the table's primary
or composite key. The error message shown to the user in this scenario
did not indicate this correctly. This adds a hint to make this clearer.
Closes#2907.
We now allow FKs to hypertables but the initial check when creating
the hypertable was not adjusted so tables with pre-existing FKs
would not be allowed to be changed into hypertable while creating
the FK constraint afterwards succeeded.
When creating a hypertable with default indexes and migrating data by
setting `migrate_data=>true` is not creating the default indexes on
chunks.
Fixed it by changing the internal order or operations by first create
the default indexes and after it migrate the data.
Fixes#7190
Foreign keys pointing to hypertables are not supported. Creating a
hypertable from a table referenced by foreign key succeeds, but it
leaves the referencing (child) table in a broken state, failing on every
insert with a `violates foreign key constraint` error.
To prevent this scenario, if a foreign key reference to the table exists
before converting it to a hypertable, the following error will be
raised:
```
cannot have FOREIGN KEY contraints to hypertable "<table_name>"
HINT: Remove all FOREIGN KEY constraints to table "<table_name>"
before making it a hypertable.
```
Fixes#6452
- Updated show_chunks, drop_chunks APIs to get the affected
chunks using chunk creation time metadata based on the
"date/time/interval" like boundary specified for the INTEGER
columns.
- We honor "integer_now" function if it's specified so as to keep
backwards compatibility with the existing behavior
Co-authored-by: Dipesh Pandit <dipesh@timescale.com>
- Added creation_time attribute to timescaledb catalog table "chunk".
Also, updated corresponding view timescaledb_information.chunks to
include chunk_creation_time attribute.
- A newly created chunk is assigned the creation time during chunk
creation to handle new partition range for give dimension (Time/
SERIAL/BIGSERIAL/INT/...).
- In case of an already existing chunk, the creation time is updated as
part of running upgrade script. The current timestamp (now()) at the
time of upgrade has been assigned as chunk creation time.
- Similarly, downgrade script is updated to drop the attribute
creation_time from catalog table "chunk".
- All relevant queries/views/test output have been updated accordingly.
Co-authored-by: Nikhil Sontakke <nikhil@timescale.com>
When an invalid function oid is passed to set_integer_now_func, it finds
out that the function oid is invalid but before throwing the error, it
calls ReleaseSysCache on an invalid tuple causing a segfault. Fixed that
by removing the invalid call to ReleaseSysCache.
Fixes#6037
To increase schema security we do not want to mix our own internal
objects with user objects. Since chunks are created in the
_timescaledb_internal schema our internal functions should live in
a different dedicated schema. This patch make the necessary
adjustments for the following functions:
- get_partition_for_key(val anyelement)
- get_partition_hash(val anyelement)
To increase schema security we do not want to mix our own internal
objects with user objects. Since chunks are created in the
_timescaledb_internal schema our internal functions should live in
a different dedicated schema. This patch make the necessary
adjustments for the following functions:
- to_unix_microseconds(timestamptz)
- to_timestamp(bigint)
- to_timestamp_without_timezone(bigint)
- to_date(bigint)
- to_interval(bigint)
- interval_to_usec(interval)
- time_to_internal(anyelement)
- subtract_integer_from_now(regclass, bigint)
To increase schema security we do not want to mix our own internal
objects with user objects. Since chunks are created in the
_timescaledb_internal schema our internal functions should live in
a different dedicated schema. This patch make the necessary
adjustments for the get_create_command function.
When TidRangeScan is child of ChunkAppend or ConstraintAwareAppend node, an
error is reported as "invalid child of chunk append: Node (26)". This patch
fixes the issue by recognising TidRangeScan as a valid child.
Fixes: #4872
Consider a hypertable which has a foreign key constraint on a
referenced table, which is a parititioned table. In such case, foreign
key constraint is duplicated for each of the partitioned table. When
we insert into a hypertable, we end up checking the foreign key
constraint multiple times, which obviously leads to foreign key
constraint violation. Instead we only check foreign key constraint of
the parent table of the partitioned table.
Fixes#4684
The schema of base table on which hypertables are created, should define
columns with proper data types. As per postgres best practices Wiki
(https://wiki.postgresql.org/wiki/Don't_Do_This), one should not define
columns with CHAR, VARCHAR, VARCHAR(N), instead use TEXT data type.
Similarly instead of using timestamp, one should use timestamptz.
This patch reports a WARNING to end user when creating hypertables,
if underlying parent table, has columns of above mentioned data types.
Fixes#4335
This change allows to create new dimensions even with
existing chunks.
It does not modify any existing data or do migration,
instead it creates full-range (-inf/inf) dimension slice for
existing chunks in order to be compatible with newly created
dimension.
All new chunks created after this will follow logic of the new
dimension and its partitioning.
Fix: #2818
Replace hardcoded database name from regression tests with :TEST_DBNAME
Remove creation of database single_2 from test runner and add it to
bgw_launcher and loader test since no other tests used those
use SQL comments in test scripts
Fixes get_create_command to produce a valid create_hypertable
call when the column name is a keyword
Add test.execute_sql helper function to test support functions
TimescaleDB has always supported functions on closed (space)
dimension, i.e., for hash partitioning. However, functions have not
been supported on open (time) dimensions, instead requiring columns to
have a supported time type (e.g, integer or timestamp). This restricts
the tables that can be time partitioned. Tables with custom "time"
types, which can be transformed by a function expression into a
supported time type, are not supported.
This change generalizes partitioning so that both open and closed
dimensions can have an associated partitioning function that
calculates a dimensional value. Fortunately, since we already support
functions on closed dimensions, the changes necessary to support this
on any dimension are minimal. Thus, open dimensions now support an
(optional) partitioning function that transforms the input type to a
supported time type (e.g., integer or timestamp type). Any indexes on
such dimensional columns become expression indexes.
Tests have been added for chunk expansion and the hashagg and sort
transform optimizations on tables that are using a time partitioning
function.
Currently, not all of these optimizations are well supported, but this
could potentially be fixed in the future.
Previously, the pg_dump test was broken because it is not possible to reference psql variables
from inside bash commands run through psql. This is fixed by hardcoding the username passed to
the bash commands inside the test.
Also, we changed the insert block trigger preventing inserts into hypertable to a non-internal
trigger, because internal triggers are not dumped by pg_dump. We need to dump the trigger so that
it is already in place after a pg_restore, to prevent users from accidentally inserting rows into
a hypertable while timescaledb_restoring=on.
A hypertable's root table should never have any tuples, but it can
acquire tuples by accident if the TimescaleDB extension is not
preloaded or `timescaledb.restoring` is set to ON.
To avoid the above issue, a hypertable's root table now has a
(internal) trigger that generates an error when tuples are
inserted. This preserves the integrity of the hypertable even when
restoring or the extension is not preloaded.
An internal trigger has the advantage of being mostly transparent to
users (e.g., it doesn't show up with \d) and it is not inherited by
chunks, so it needs no special handling to avoid adding it to chunks.
The blocking trigger is added in the update script and if it is
detected that the hypertable's root table has data in the process, the
update will fail with an error and instructions on how to fix.
This adds a simple check to enforce that partitioning functions
are IMMUTABLE. This is a requirement since a partitioning function
must always return the same value given the same input.
This fixes an issue with the `if_not_exists` option to add_dimension()
that caused the command to fail on a non-empty table. When this option
is set, the command should only fail if the dimension does not exists
and the table is non-empty.
Tables can now hold existing data, which is optionally migrated from
the main table to chunks when create_hypertable() is called.
The data migration is similar to the COPY path, with the single
difference that the inserted/copied tuples come from an existing table
instead of being read from a file. After the data has been migrated,
the main table is truncated.
One potential downside of this approach is that all of this happens in
a single transaction, which means that the table is blocked while
migration is ongoing, preventing inserts by other transactions.
The functions for adding and updating dimensions have been refactored
in C to:
- improve usage of proper error codes
- make messages that better conform with the PostgreSQL standard.
- improve security by avoiding that lots of code run under SECURITY DEFINER
A new if_not_exists option has also been added to add_dimension() and
a the number of partitions can now be set using the new
set_number_partitions() function.
A bug in the validation of smallint time intervals has been fixed. The
previous code didn't check for intervals > 0 and smallint intervals
accepted values up to UINT16_MAX instead of INT16_MAX.
A hypertable's associated schema is used to create and store internal
data tables (chunks). A hypertable creates tables in that schema,
typically with full superuser permissions, regardless of whether the
hypertable's owner or the current user have permissions for the schema.
If the schema doesn't exist, the hypertable will create it when
creating the first chunk, even though the user or table owner does
not have permissions to create schemas in the database.
This change adds proper permissions checks to create_hypertable() so
that users cannot create hypertables with a custom associated schema
unless they have the proper permissions on the schema or the database.
Chunks are also no longer created with internal schema permissions if
the associated schema is something different from the internal schema.
Attaching tablespaces to hypertables is now handled
in native code, with improved permissions checking and
caching of tablespaces in the Hypertable data object.
The user should be able to add time dimensions using INTERVAL when
the column type is TIMESTAMP/TIMESTAMPTZ/DATE, so this change adds
that support.
Additionally it adds some additional tests and checks for
add_dimension, e.g., a nice error when the table is not a
hypertable.
This change reduces the usage of SECURITY DEFINER on SQL
functions and fixes related permissions issues. It also
properly checks hypertable permissions relative the current_user
instead of the session_user, which otherwise breaks SET ROLE,
among other things.
The extension now works with PostgreSQL 10, while
retaining compatibility with version 9.6.
PostgreSQL 10 has numerous internal changes to functions and
APIs, which necessitates various glue code and compatibility
wrappers to seamlessly retain backwards compatiblity with older
versions.
Test output might also differ between versions. In particular,
the psql client generates version-specific output with `\d` and
EXPLAINs might differ due to new query optimizations. The test
suite has been modified as follows to handle these issues. First,
tests now use version-independent functions to query system
catalogs instead of using `\d`. Second, changes have been made to
the test suite to be able to verify some test outputs against
version-dependent reference files.
Add check that time dimensions are set as NOT NULL in the
main table that a hypertable is created from. If it is not
set, the constraint will be added.
This commit moves a lot of test setup logic to runner.sh. Also
passes the right commands to the regression infrastructure to create
appropriate users and run tests as a regular user.
All regression tests will now use a non-superuser unless superuser is
necessary. This PR is a meant to prevent things like issue #226.
This PR also fixes some more permission bugs found during this testing.
Using pg_dump/pg_restore for a database with hypertables will back
up all internal tables too, which is sometimes not desired. These
scripts allow one to backup regular PostgreSQL tables separately
and then pick and choose which hypertables to backup and restore.
This also provides a forward-compatible way of backing up and
restoring hypertables, since data is dumped to CSV and restored
by re-creating and re-inserting the data.
Previously the default chunk time in microseconds was too large
for a SMALLINT or INTEGER field. Now, we only assign a default
value if the type is TIMESTAMP or TIMESTAMPTZ. Integer timestamps,
such as SMALLINT, INTEGER, and BIGINT, need to be explicitly set
since only the user knows what units the numbers represent.
Further, we check to make sure the chunk time interval is not too
large for SMALLINT and INTEGER so as to avoid confusing problems
later when the user goes to insert.
A new public-facing API `add_dimension(table, column, ...)`
makes it possible to add additional dimensions (partitioning
columns) to a hypertable.
Currently, new dimension can only be added to empty tables.
The code has also been refactored with a corresponding
internal function that is called by both `add_dimension()`
and `create_hypertable()`.
Previously create_hypertable() would throw an error when called on
an already existing hypertable. This can now be skipped by setting
if_not_exists argument to true.
Previously, each test set their own (although mostly the same)
configuration for log output and error verbosity. This is now set
globally in the test runner so that tests only need to set these
configuration parameters if they need to override the defaults. The
log verbosity is also reduced so that errors aren't generated with the
line number of the source file that output the error. Line numbers in
the output can break tests when upgrading to a new PostgreSQL version
that outputs a different line number.
Since create_hypertable() allows you to optionally specify a
partitioning column, it makes sense to default to one partition when
no column is specified and asking for the number of partitions when a
column is specified and the number of partitions is not (instead of
defaulting to one).
This patch also changes the order and type of partitioning-related
input arguments to create_hypertable() so that the number of
partitions can easily be specified alongside the partitioning column
and without type casting.
Setting up single node is now:
```
CREATE EXTENSION IF NOT EXISTS iobeamdb CASCADE;
select setup_single_node();
```
To setup a cluster do (on meta node):
```
CREATE EXTENSION IF NOT EXISTS iobeamdb CASCADE;
select set_meta();
```
on data node:
```
CREATE EXTENSION IF NOT EXISTS iobeamdb CASCADE;
select join_cluster('metadb', 'metahost');
```
This assumes that the commands are issued by the same user on both the
meta node and the data node. Otherwise the data node also has to
specify the user name to use when connecting to the meta node.
- Directory structure now matches common practices
- Regression tests now run with pg_regress via the PGXS infrastructure.
- Unit tests do not integrate well with pg_regress and have to be run
separately.
- Docker functionality is separate from main Makefile. Run with
`make -f docker.mk` to build and `make -f docker.mk run` to run
the database in a container.
