If a bad value is given to `alter_job` or `add_job` for a configuration
parameter, no error will be given but the job will fail to execute.
This commit adds checks of the configuration parameters to the
functions so that an error is given immediately when calling it. The
commit factors out the extraction of parameters from the configuration
from the execution functions into a separate functions and calls them
from `alter_job` and `add_job` as well as when executing the job. Only
non-custom job checks are done.
The commit also moves a few functions that were only used in TSL code
from the `src/` directory to the `tsl/src/` directory and also removes
a redundant permission check and does a minor refactoring of the
`job_execute` function so that an active snapshot is always created
regardless of whether a transaction is open or not. The corresponding
code in the individual policy functions are removed since they are not
needed.
Closes#2607
Function refresh_continuous_aggregate, which takes a continuous
aggregate and a chunk, is added. It refreshes the continuous aggregate
on the given chunk if there are invalidations. The function can be
used in a transaction, e.g., together with following drop_chunks. This
allows users to create a user defined action to refresh and drop
chunks. Therefore, the refresh on drop is removed from drop_chunks.
Tests are updated to no longer use continuous aggregate options that
will be removed, such as `refresh_lag`, `max_interval_per_job` and
`ignore_invalidation_older_than`. `REFRESH MATERIALIZED VIEW` has also
been replaced with `CALL refresh_continuous_aggregate()` using ranges
that try to replicate the previous refresh behavior.
The materializer test (`continuous_aggregate_materialize`) has been
removed, since this tested the "old" materializer code, which is no
longer used without `REFRESH MATERIALIZED VIEW`. The new API using
`refresh_continuous_aggregate` already allows manual materialization
and there are two previously added tests (`continuous_aggs_refresh`
and `continuous_aggs_invalidate`) that cover the new refresh path in
similar ways.
When updated to use the new refresh API, some of the concurrency
tests, like `continuous_aggs_insert` and `continuous_aggs_multi`, have
slightly different concurrency behavior. This is explained by
different and sometimes more conservative locking. For instance, the
first transaction of a refresh serializes around an exclusive lock on
the invalidation threshold table, even if no new threshold is
written. The previous code, only took the heavier lock once, and if, a
new threshold was written. This new, and stricter locking, means that
insert processes that read the invalidation threshold will block for a
short time when there are concurrent refreshes. However, since this
blocking only occurs during the first transaction of the refresh
(which is quite short), it probably doesn't matter too much in
practice. The relaxing of locks to improve concurrency and performance
can be implemented in the future.
This commit will add support for `WITH NO DATA` when creating a
continuous aggregate and will refresh the continuous aggregate when
creating it unless `WITH NO DATA` is provided.
All test cases are also updated to use `WITH NO DATA` and an additional
test case for verifying that both `WITH DATA` and `WITH NO DATA` works
as expected.
Closes#2341
The invalidation log initialization did not adjust
the timestamp to internal format. Use ts_get_now_internal
in all places that write entries to the invalidation log.
We change the syntax for defining continuous aggregates to use `CREATE
MATERIALIZED VIEW` rather than `CREATE VIEW`. The command still creates
a view, while `CREATE MATERIALIZED VIEW` creates a table. Raise an
error if `CREATE VIEW` is used to create a continuous aggregate and
redirect to `CREATE MATERIALIZED VIEW`.
In a similar vein, `DROP MATERIALIZED VIEW` is used for continuous
aggregates and continuous aggregates cannot be dropped with `DROP
VIEW`.
Continuous aggregates are altered using `ALTER MATERIALIZED VIEW`
rather than `ALTER VIEW`, so we ensure that it works for `ALTER
MATERIALIZED VIEW` and gives an error if you try to use `ALTER VIEW` to
change a continuous aggregate.
Note that we allow `ALTER VIEW ... SET SCHEMA` to be used with the
partial view as well as with the direct view, so this is handled as a
special case.
Fixes#2233
Co-authored-by: =?UTF-8?q?Erik=20Nordstr=C3=B6m?= <erik@timescale.com>
Co-authored-by: Mats Kindahl <mats@timescale.com>
To drop a continuous aggregate it was necessary to use the `CASCADE`
keyword, which would then cascade to the materialized hypertable. Since
this can cascade the drop to other objects that are dependent on the
continuous aggregate, this could accidentally drop more objects than
intended.
This commit fixes this by removing the check for `CASCADE` and adding
the materialized hypertable to the list of objects to drop.
Fixestimescale/timescaledb-private#659
We added a timescaledb.ignore_invalidation_older_than parameter for
continuous aggregatess. This parameter accept a time-interval (e.g. 1
month). if set, it limits the amount of time for which to process
invalidation. Thus, if
timescaledb.ignore_invalidation_older_than = '1 month'
then any modifications for data older than 1 month from the current
timestamp at insert time will not cause updates to the continuous
aggregate. This limits the amount of work that a backfill can trigger.
This parameter must be >= 0. A value of 0 means that invalidations are
never processed.
When recording invalidations for the hypertable at insert time, we use
the maximum ignore_invalidation_older_than of any continuous agg attached
to the hypertable as a cutoff for whether to record the invalidation
at all. When materializing a particular continuous agg, we use that
aggs ignore_invalidation_older_than cutoff. However we have to apply
that cutoff relative to the insert time not the materialization
time to make it easier for users to reason about. Therefore,
we record the insert time as part of the invalidation entry.
Previously, refresh_lag in continuous aggs was calculated
relative to the maximum timestamp in the table. Change the
semantics so that it is relative to now(). This is more
intuitive.
Requires an integer_now function applied to hypertables
with integer-based time dimensions.
When explicit JOIN syntax is used the from list will contain a
JoinExpr instead of a list of RangeTblRef which was not detected
properly by cagg_validate_query. This patch adds a check for
RangeTblRef to the validation code.
Add indexes for materialization table created by continuous aggregates.
This behavior can be turned on/off by using timescaledb.create_group_indexes parameter
of the WITH clause when the continuous agg is created.
Tests include:
1. UPDATEs get logged to the invalidation log.
2. the invalidation log is not written to on ABORT.
3. custom time-partitioning functions are not supported
(also updates error message for this case).
