If `drop_chunks` has been executed on a hypertable that has a
continuous aggregate defined, the chunks will be removed and marked as
dropped in `_timescaledb_catalog.chunk` but the lines will not be
removed. This will cause problems for the update script since it is
missing a check to only process chunks that are not dropped and will
try to cast the chunk name into a `REGCLASS` for a table that does not
exist.
This commit fixes this by adding a check that the chunk is not dropped
and also fixes the update test to not count dropped chunks when
comparing with the chunk index since the chunk index does not count
dropped chunks.
Fixes#2791
The update scripts now test extension updates for a multi-node setup
with one access node and one data node. Currently the following is
tested:
* A distributed hypertable is created in the database where the update
process is normally tested.
* A new database representing a data node is created. This database is
updated and restored in the same way as the other database. However,
no integrity or other validation checks are currently run on the
datanode database; only the ability to update the extension is
tested.
Some tests contain code that only work if the build is based on a Git
clone, which caused these tests to fail when Git was not available.
This commit splits out those tests and only enabling them if Git is
found and Git information can be retrieved.
Add a test for updating statistics on chunks compressed before
statistics where kept. This patch also sets metadata for previously
compressed chunks so running ANALYZE on them will set the correct
tuplecount.
The command `get_git_commit` shows quite a little information and
especially if mistakes are made it is hard to verify what the binary is
actually based on.
This commit extends the function to provide some more information:
specifically the SHA of the HEAD regardless of whether there is a tag
on it or not and the time for the HEAD commit.
```
postgres=# select * from _timescaledb_internal.get_git_commit();
commit_tag | commit_hash | commit_time
--------------------------+-------------+------------------------
1.7.4-10-g09b0b778-dirty | 09b0b778 | 2020-09-13 17:50:38+02
(1 row)
```
If git is not installed, the function `get_git_commit` will return an
error indicating that no git information is available. If some of the
fields are available, they will be emitted and the remaining fields
will be NULL.
Fixes#2457
The constraint test fails when running with sanitizers (ASAN and
UBSAN) due to a tablespace conflict. Make sure tablespaces are cleaned
up prior to recreating.
This resolves an issue when using both constraints with index
tablespaces AND constraints that have no indexes on a single hypertable.
chunk_constraint_add_table_constraint() was attempting to add
`USING INDEX TABLESPACE` to all constraints when a given hypertable had
any constraint configured to use a tablespace, resulting in a SYNTAX
error and blocking the creation of new chunks.
The solution here is to limit index tablespace lookups to only the
constraint types which use indexes: primary key and unique so that only
those constraits will have `USING INDEX TABLESPACE` prepended when
necessary.
Change database names to be unique over the test suite by adding the
test database name in front of the created database names in the test.
This will allow the test to be executed in parallel with other tests
since it will not have conflicting databases in the same cluster.
Previously, there were a few directories created for tablespaces, but
this commit changes that to create one directory for each test where
the tablespace can be put. This is done by using a directory prefix for
each tablespace and each test should then create a subdirectory under
that prefix for the tablespace. The commit keeps variables for the old
tablespace paths around so that old tests work while transitioning to
the new system.
In #2514 a a race condition between inserts and `drop_chunks` is fixed
and this commit will repair the dimension slices table by
re-constructing missing dimension slices from the corresponding
constraint expressions.
Closes#1986
When upgrading from 1.7, it's possible to have retention policies
which overlap with continuous aggregates. These make use of the
cascade_to_materializations parameter to avoid invalidating the
aggregate.
In 2.0 there is no equivalent behavior to prevent the retention from
disrupting the aggregate. So during the 2.0 upgrade, check for any
running retention policies that are dropping chunks still used by a
continuous aggregate and suspend them (scheduled=>false). This will
also print a notice informing the user of what happened and how to
resume the retention policy if that's what they truly want.
Fixes#2530
Fix a check for a compatible chunk time interval type when creating a
hypertable with a custom time type.
Previously, the check allowed `Interval` type intervals for any
dimension type that is not an integer type, including custom time
types. The check is now changed so that it only accepts an `Interval`
for timestamp and date type dimensions.
A number of related error messages are also cleaned up so that they
are more consistent and conform to the error style guide.
The database must know the valid time range of a custom time type,
similar to how it knows the time ranges of officially supported time
types. However, the only way to "know" the valid time range of a
custom time type is to assume it is the same as the one of a supported
time type.
A previous commit tried to make such assumptions by finding an
appropriate cast from the custom time type to a supported time
type. However, this fails in case there are multiple casts available
that each could return a different type and range.
This change restricts the choice of valid time ranges to only that of
the bigint time type.
Fixes#2523
Removes unlrelated column schedule_interval from
timescaledb_information.continuous_aggregates view and simplifies it.
Renames argument cagg in refresh_continuous_aggregate into
continuous_aggregate as in add_continuous_aggregate_policy.
Part of #2521
Since handling of telemetry differs between timescaledb versions
when telemetry is disabled via environment variable we ignore the
scheduled flag in the post update diff.
Renaming the parameter `hypertable_or_cagg` in functions `drop_chunks`
and `show_chunks` to `relation` and changing parameter name from
`main_table` to `hypertable` or `relation` depending on context.
This change will add an invalidation to the
materialization_invalidation_log for any region earlier than the
ignore_invalidation_older_than parameter when updating a continuous
aggregate to 2.0. This is needed as we do not record invalidations
in this region prior to 2.0 and there is no way to ensure the
aggregate is up to date within this range.
Fixes#2450
This patch removes enterprise license support and moves
move_chunk() function under community license (TSL).
Licensing validation code been reworked and simplified.
Previously used timescaledb.license_key guc been renamed to
timescaledb.license.
This change also makes testing code more strict against
used license. Apache test suite now can test only apache-licensed
functions.
Fixes#2359
This patch changes the update test to use the same checks
between clean / updated install and dumped/restored install.
Previously only a small subset of the checks would be run against
the updated instance and most of the tests would only run against
the dumped and restored container.
As part of the 2.0 continous aggregate changes, we are removing the
continuous_aggs_completed_threshold table. However, this may result
in currently running aggregates being considered complete even if
their completed threshold hadn't reached the invalidation threshold.
This change fixes this by adding an entry to the invalidation log
for any such aggregates.
Fixes#2314
This patch fixes the format strings used to construct object names
in tests. The format strings used in those tests would break when
object names are involved that require quoting.
The recently added test for hypertable detection used compression
which is not available in ApacheOnly tests so we move that test
to regresscheck-t. Additionally we move the other test in
plan_hypertable_cache to plan_expand_hypertable to reduce the number
of tests.
Rename the `refresh_interval` field in
`timescaledb_information.continuous_aggregate` view to match the
parameter name in `add_continuous_aggregate_policy`.
Removes options refresh_lag, max_interval_per_job and
ignore_invalidation_older_than from continuous aggregate creation with
CREATE MATERIALIZED VIEW, since they are not related this statement
any more. They are already replaced with the corresponding options in
add_continuous_aggregate_policy.
This commit removes only options, while the options are still stored
in the catalog and need to be removed from there in a separate PR.
When a hypertable was referenced in a subquery that was not already
in our hypertable cache we would fail to detect it as hypertable
leading to transparent decompression not working for that hypertable.
When the extension is updated to 2.0, we need to migrate
existing ignore_invalidation_older_than settings to the new
continuous aggregate policy framework.
ignore_invalidation_older_than setting is mapped to start_interval
of the refresh policy.If the default value is used, it is mapped
to NULL start_interval, otherwise it is converted to an
interval value.
When a constraint is backed by an index like a unique constraint
or a primary key constraint the constraint can be renamed by either
ALTER TABLE RENAME CONSTRAINT or by ALTER INDEX RENAME. Depending
on the command used to rename different internal metadata tables
would be adjusted leading to corrupt metadata. This patch makes
ALTER TABLE RENAME CONSTRAINT and ALTER INDEX RENAME adjust the
same metadata tables.
This change simplifies the name of the functions for adding and
removing a continuous aggregate policy. The functions are renamed
from:
- `add_refresh_continuous_aggregate_policy`
- `remove_refresh_continuous_aggregate_policy`
to
- `add_continuous_aggregate_policy`
- `remove_continuous_aggregate_policy`
Fixes#2320
This commit will add support for `WITH NO DATA` when creating a
continuous aggregate and will refresh the continuous aggregate when
creating it unless `WITH NO DATA` is provided.
All test cases are also updated to use `WITH NO DATA` and an additional
test case for verifying that both `WITH DATA` and `WITH NO DATA` works
as expected.
Closes#2341
With the new continuous aggregate API, some of
the parameters used to create a continuous agg are
now obsolete. Remove refresh_lag, max_interval_per_job
and ignore_invalidation_older_than information from
timescaledb_information.continuous_aggregates.
If a tablespace is attached to a hypertable the tablespace of the
hypertable is not set, but if the tablespace is set it is also
attached. A similar situation occurs if tablespaces are detached.
This means that if a hypertable is created with a tablespace and then
all tablespaces are detached, the chunks will still be put in the
tablespace of the hypertable.
With this commit, attaching a tablespace to a hypertable will set the
tablespace of the hypertable if it does not already have one. Detaching
a tablespace from a hypertable will set the tablespace to the default
tablespace if the tablespace being detached is the tablespace for the
hypertable.
If `detach_tablespace` is called with only a tablespace name, it will
be detached from all tables it is attached to. This commit ensures that
the tablespace for the hypertable is set to the default tablespace if
it was set to the tablespace being detached.
Fixes#2299
This patch changes the scheduler to ignore telemetry jobs when
telemetry is disabled. With this change telemetry jobs will no
longer use background worker resources when telemetry is disabled.
Time types, like date and timestamps, have limits that aren't the same
as the underlying storage type. For instance, while a timestamp is
stored as an `int64` internally, its max supported time value is not
`INT64_MAX`. Instead, `INT64_MAX` represents `+Infinity` and the
actual largest possible timestamp is close to `INT64_MAX` (but not
`INT64_MAX-1` either). The same applies to min values.
Unfortunately, time handling code does not check for these boundaries;
in most cases, overflow handling when, e.g., bucketing, are checked
against the max integer values instead of type-specific boundaries. In
other cases, overflows simply throw errors instead of clamping to the
boundary values, which makes more sense in many situations.
Using integer time suffers from similar issues. To take one example,
simply inserting a valid `smallint` value close to the max into a
table with a `smallint` time column fails:
```
INSERT INTO smallint_table VALUES ('32765', 1, 2.0);
ERROR: value "32770" is out of range for type smallint
```
This happens because the code that adds dimensional constraints always
checks for overflow against `INT64_MAX` instead of the type-specific
max value. Therefore, it tries to create a chunk constraint that ends
at `32770`, which is outside the allowed range of `smallint`.
The resolve these issues, several time-related utility functions have
been implemented that, e.g., return type-specific range boundaries,
and perform saturated addition and subtraction while clamping to
supported boundaries.
Fixes#2292
Tablespaces are created cluster-wide, which means that tests that
create tablespaces cannot run together with other tests that create the
same tablespaces. This commit make those tests into solo tests to avoid
collisions with other tablespace-creating tests and also fix a test.
This change renames function to approximate_row_count() and adds
support for regular tables. Return a row count estimate for a
table instead of a table list.
Support add and remove continuous agg policy functions
Integrate policy execution with refresh api for continuous
aggregates
The old api for continuous aggregates adds a job automatically
for a continuous aggregate. This is an explicit step with the
new API. So remove this functionality.
Refactor some of the utility functions so that the code can be shared
by multiple policies.
