With the new continuous aggregate API, some of
the parameters used to create a continuous agg are
now obsolete. Remove refresh_lag, max_interval_per_job
and ignore_invalidation_older_than information from
timescaledb_information.continuous_aggregates.
This maintenance release contains bugfixes since the 1.7.3 release. We deem it
high priority for upgrading if TimescaleDB is deployed with replicas (synchronous
or asynchronous).
In particular the fixes contained in this maintenance release address an issue with
running queries on compressed hypertables on standby nodes.
**Bugfixes**
* #2340 Remove tuple lock on select path
The function `cagg_watermark` returns the time threshold at which
materialized data ends and raw query data begins in a real-time
aggregation query (union view).
The watermark is simply the completed threshold of the continuous
aggregate materializer. However, since the completed threshold will no
longer exist with the new continuous aggregates, the watermark
function has been changed to return the end of the last bucket in the
materialized hypertable.
In most cases, the completed threshold is the same as the end of the
last materialized bucket. However, there are situations when it is
not; for example, when there is a filter in the view query some
buckets might not be materialized because no data matched the
filter. The completed threshold would move ahead regardless. For
instance, if there is only data from "device_2" in the raw hypertable
and the aggregate has a filter `device=1`, there will be no buckets
materialized although the completed threshold moves forward. Therefore
the new watermark function might sometimes return a lower watermark
than the old function. A similar situation explains the different
output in one of the union view tests.
This change renames function to approximate_row_count() and adds
support for regular tables. Return a row count estimate for a
table instead of a table list.
Support add and remove continuous agg policy functions
Integrate policy execution with refresh api for continuous
aggregates
The old api for continuous aggregates adds a job automatically
for a continuous aggregate. This is an explicit step with the
new API. So remove this functionality.
Refactor some of the utility functions so that the code can be shared
by multiple policies.
We change the syntax for defining continuous aggregates to use `CREATE
MATERIALIZED VIEW` rather than `CREATE VIEW`. The command still creates
a view, while `CREATE MATERIALIZED VIEW` creates a table. Raise an
error if `CREATE VIEW` is used to create a continuous aggregate and
redirect to `CREATE MATERIALIZED VIEW`.
In a similar vein, `DROP MATERIALIZED VIEW` is used for continuous
aggregates and continuous aggregates cannot be dropped with `DROP
VIEW`.
Continuous aggregates are altered using `ALTER MATERIALIZED VIEW`
rather than `ALTER VIEW`, so we ensure that it works for `ALTER
MATERIALIZED VIEW` and gives an error if you try to use `ALTER VIEW` to
change a continuous aggregate.
Note that we allow `ALTER VIEW ... SET SCHEMA` to be used with the
partial view as well as with the direct view, so this is handled as a
special case.
Fixes#2233
Co-authored-by: =?UTF-8?q?Erik=20Nordstr=C3=B6m?= <erik@timescale.com>
Co-authored-by: Mats Kindahl <mats@timescale.com>
This maintenance release contains bugfixes since the 1.7.2 release. We deem it high
priority for upgrading.
In particular the fixes contained in this maintenance release address issues in compression,
drop_chunks and the background worker scheduler.
**Bugfixes**
* #2059 Improve infering start and stop arguments from gapfill query
* #2067 Support moving compressed chunks
* #2068 Apply SET TABLESPACE for compressed chunks
* #2090 Fix index creation with IF NOT EXISTS for existing indexes
* #2092 Fix delete on tables involving hypertables with compression
* #2164 Fix telemetry installed_time format
* #2184 Fix background worker scheduler memory consumption
* #2222 Fix `negative bitmapset member not allowed` in decompression
* #2255 Propagate privileges from hypertables to chunks
* #2256 Fix segfault in chunk_append with space partitioning
* #2259 Fix recursion in cache processing
* #2261 Lock dimension slice tuple when scanning
**Thanks**
* @akamensky for reporting an issue with drop_chunks and ChunkAppend with space partitioning
* @dewetburger430 for reporting an issue with setting tablespace for compressed chunks
* @fvannee for reporting an issue with cache invalidation
* @nexces for reporting an issue with ChunkAppend on space-partitioned hypertables
* @PichetGoulu for reporting an issue with index creation and IF NOT EXISTS
* @prathamesh-sonpatki for contributing a typo fix
* @sezaru for reporting an issue with background worker scheduler memory consumption
This change removes, simplifies, and unifies code related to
`drop_chunks` and `show_chunks`. As a result of prior changes to
`drop_chunks`, e.g., making table relid mandatory and removing
cascading options, there's an opportunity to clean up and simplify the
rather complex code for dropping and showing chunks.
In particular, `show_chunks` is now consistent with `drop_chunks`; the
relid argument is mandatory, a continuous aggregate can be used in
place of a hypertable, and the input time ranges are checked and
handled in the same way.
Unused code is also removed, for instance, code that cascaded drop
chunks to continuous aggregates remained in the code base while the
option no longer exists.
This patch adds functionality to schedule arbitrary functions
or procedures as background jobs.
New functions:
add_job(
proc REGPROC,
schedule_interval INTERVAL,
config JSONB DEFAULT NULL,
initial_start TIMESTAMPTZ DEFAULT NULL,
scheduled BOOL DEFAULT true
)
Add a job that runs proc every schedule_interval. Proc can
be either a function or a procedure implemented in any language.
delete_job(job_id INTEGER)
Deletes the job.
run_job(job_id INTEGER)
Execute a job in the current session.
This patch changes the signature from cagg_watermark(oid) to
cagg_watermark(int). Since this is an API breaking change it couldn't
be done in an earlier release.
When a continuous aggregate is refreshed, it also needs to move the
invalidation threshold in case the refresh window stretches beyond the
current threshold. The new invalidation threshold must be set in its
own transaction during the refresh, which can only be done if the
refresh command is a procedure.
This patch adds policies to the update test to ensure their
configuration is properly migrated during updates. This patch
also fixes the inconsistent background job application_name
and adjusts them in the update script.
This patch changes the application name for background worker jobs
to include the job_id which makes the application name unique and
allows joining against pg_stat_activity to get a list of currently
running background worker processes. This change also makes
identifying misbehaving jobs easier from the postgres log as the
application name can be included in the log line.
Since the minimum version we can update from on PG11 is 1.1.0 we
can get rid of all the previous versions update files as they
are not a valid update source for any current version.
The parameter `cascade_to_materialization` is removed from
`drop_chunks` and `add_drop_chunks_policy` as well as associated tables
and test functions.
Fixes#2137
This change replaces the add_drop_chunks_policy function with
add_retention_policy. This also renames the older_than parameter
of that function as retention_window. Likewise, the
remove_drop_chunks_policy is also being renamed
remove_retention_policy.
Fixes#2119
This change adds a new refresh function called
`refresh_continuous_aggregate` that allows refreshing a continuous
aggregate over a given window of data, called the "refresh window".
This is the first step in a larger overhaul of the continuous
aggregate feature with the goal of cleaning up the API and separating
policy from the core functionality.
Currently, the refresh function does a brute-force refresh of a window
and it bypasses the whole invalidation framework. Future updates
intend to integrate with this framework (with modifications) to
optimize refreshes. An exclusive lock is take on the continuous
aggregate's internal materialized hypertable in order to protect
against concurrent refreshing. However, as this serializes refreshes,
we might want to relax this locking in the future to allow, e.g.,
concurrent refreshes of non-overlapping windows.
The new refresh functionality includes basic tests for bad input and
refreshing across different windows. Unfortunately, a bug in the
optimization code for `time_bucket` causes timestamps to overflow the
allowed MAX time. Therefore, refresh windows that are close to the MAX
allowed size are not yet supported or tested.
This patch adds a proc_name, proc_schema, hypertable_id index to
bgw_job. 3 functions using the new index are added as well:
ts_bgw_job_find_by_proc
ts_bgw_job_find_by_hypertable_id
ts_bgw_job_find_by_proc_and_hypertable_id
These functions are required for migrating the existing policies
to store their configuration in bgw_job directly.
timescaledb_information.chunks view shows metadata
related to chunks.
timescaledb_information.dimensions shows metadata
related to hypertable's dimensions.
This change fixes the stats collecting code to also return the slot
collation fields for PG12. This fixes a bug (#2093) where running an
ANALYZE in PG12 would break queries on distributed tables.
This maintenance release contains bugfixes since the 1.7.1 release. We deem it medium
priority for upgrading.
In particular the fixes contained in this maintenance release address bugs in continuous
aggregates, drop_chunks and compression.
**Features**
* #1877 Add support for fast pruning of inlined functions
**Bugfixes**
* #1908 Fix drop_chunks with unique constraints when cascade_to_materializations is false
* #1915 Check for database in extension_current_state
* #1918 Unify chunk index creation
* #1932 Change compression locking order
* #1938 Fix gapfill locf treat_null_as_missing
* #1982 Check for disabled telemetry earlier
* #1984 Fix compression bit array left shift count
* #1997 Add checks for read-only transactions
* #2002 Reset restoring gucs rather than explicitly setting 'off'
* #2028 Fix locking in drop_chunks
* #2031 Enable compression for tables with compound foreign key
* #2039 Fix segfault in create_trigger_handler
* #2043 Fix segfault in cagg_update_view_definition
* #2046 Use index tablespace during chunk creation
* #2047 Better handling of chunk insert state destruction
* #2049 Fix handling of PlaceHolderVar in DecompressChunk
* #2051 Fix tuple concurrently deleted error with multiple continuous aggregates
**Thanks**
* @akamensky for reporting an issue with telemetry and an issue with drop_chunks
* @darko408 for reporting an issue with decompression
* @dmitri191 for reporting an issue with failing background workers
* @eduardotsj for reporting an issue with indexes not inheriting tablespace settings
* @fourseventy for reporting an issue with multiple continuous aggregrates
* @fvannee for contributing optimizations for pruning inlined functions
* @jflambert for reporting an issue with failing telemetry jobs
* @nbouscal for reporting an issue with compression jobs locking referenced tables
* @nicolai6120 for reporting an issue with locf and treat_null_as_missing
* @nomanor for reporting an issue with expression index with table references
* @olernov for contributing a fix for compressing tables with compound foreign keys
* @werjo for reporting an issue with drop_chunks and unique constraints
If a new chunk is created as part of an insert and drop_chunks runs
concurrently with the insert, there is a risk of a race. This is a test
for this.
Add locks for dimension slice tuples
If a dimension slice tuple is found while adding new chunk constraints
as part of a chunk creation it is not locked prior to adding the chunk
constraint. Hence a concurrently executing `drop_chunks` can find a
dimension slice unused (because there is no chunk constraint that
references it) and subsequently remove it. The insert will the continue
to add the chunk constraint with a reference to a now non-existent
dimension slice.
This commit fixes this by locking the dimension slice tuple with a
share lock when creating chunks and locking the dimension slice with an
exclusive lock prior to scanning for existing chunk constraints.
The commit also contains a script that repair the `dimension_slice`
table if it is broken by extracting information about dimension slices
that are mentioned in `chunk_constraint` table but not present in
`dimension_slice` table and re-create the rows from the constraints on
the chunks.
If a tablespace is provided for an index on a hypertable, it will be
also used for the index on new chunks. This is done when constraints
are created on new chunk from the hypertable constraints.
Fixes#903
Setting the `timescaledb.restoring` guc explicitly to 'off'
for the db meant that the setting got exported in `pg_dumpall`
and some other cases where that setting would then conflict
with the setting set by the pre_restore function causing it to
be overridden and causing errors on restore. This changes to
`RESET` so that instead it will take the system default and not
be dumped separately as an override.
The `drop_chunks` function is refactored to make table name mandatory
for the function. As a result, the function was also refactored to
accept the `regclass` type instead of table name plus schema name and
the parameters were reordered to match the order for `show_chunks`.
The commit also refactor the code to pass the hypertable structure
between internal functions rather than the hypertable relid and moving
error checks to the PostgreSQL function. This allow the internal
functions to avoid some lookups and use the information in the
structure directly and also give errors earlier instead of first
dropping chunks and then error and roll back the transaction.
This change refactors how connections are handled during remote
transactions. In particular, the connection cache now stays consistent
during transactions, even during rollbacks. Previously, the connection
cache was replaced on every rollback, even if the rollback was
intentional (i.e, not due to an error). This made it hard to debug
connections since the cache became completely empty.
Connections could also be left in the cache in a bad state after
failed transactions. This has been fixed by moving connection checks
to the cache and tying transaction state changes to each
connection. This ensures that such checks are done in one canonical
place instead of being spread out throughout the code.
Given how tightly coupled a remote transaction is with its connection,
it might make sense to remove the separate remote transaction store
and instead put this information in each connection. This is left to a
future change, however.
In addition to the above changes, this commit includes:
* Showing transaction depth and invalidation in the transaction store
* Invalidation on individual connections instead of replacing the
whole cache
* Closing of connections to a local database that is being dropped to
prevent "in use" errors.
* Ability to add callbacks to async requests that are executed when a
response is received. This is used by remote transactions to mark
connections as having successfully completed a transaction. Thus, on
errors, it is easy to detect connections that are in bad states.
* Error checks on each connection instead of having global error
tracking for each remote transaction. This change removes the global
error state for distributed transactions.
The connection cache for remote transactions can now be examined using
a function that shows all connections in the cache. This allows easier
debugging and validation both in tests and on live systems.
In particular, we'd like to know that connections are in good state
post commit or rollback and that we don't leave bad connections in the
cache.
The remote transaction test (`remote_txn`) has been updated to show
the connection cache as remote transactions are
executed. Unfortunately, the whole cache is replaced on every
(sub-)transaction rollback, which makes it hard to debug the
connection state of a particular remote transaction. Further, some
connections are left in the cache in a bad state after, e.g.,
connection loss.
These issues will be fixed with an upcoming change.
This patch changes the update script generation to not use
scratch files and removes the sql fragments to set and unset
the post_update_stage from CMakeLists.txt and puts them into
dedicated files.
Replace EXECUTE PROCEDURE with EXECUTE FUNCTION because the former
is deprecated in PG11+. Unfortunately some test output will still
have EXECUTE PROCEDURE because pg_get_triggerdef in PG11 still
generates a definition with EXECUTE PROCEDURE.
This commit removes the `cascade` option from the function
`drop_chunks` and `add_drop_chunk_policy`, which will now never cascade
drops to dependent objects. The tests are fixed accordingly and
verbosity turned up to ensure that the dependent objects are printed in
the error details.
Implements SQL function set_replication_factor, which changes
replication factor of a distributed hypertable. The change of the
replication factor doesn't affect existing chunks. Newly created
chunks are replicated according to new replication factor.
This change adds a new parameter to the detach_data_node and
delete_data_node functions that will allow the user to automatically
shrink their space dimension to match the number of nodes.
This change adds a new command to return a subset of the column
stats for a hypertable (column width, percent null, and percent
distinct). As part of the execution of this command on an access
node, these stats will be collected for distributed chunks and
updated on the access node.
The `postgres` database might not exists on a data node, but
`template1` will always exist so if a connection using `postgres`
fails, we use `template1` as a secondary database.
This is similar to how `connectMaintenanceDatabase` in the PostgreSQL
code base works.
A new function, `get_chunk_relstats()`, allows fetching relstats
(basically `pg_class.{relpages,reltuples`) from remote chunks on data
nodes and writing it to the `pg_class` entry for the corresponding
local chunk. The function expects either a chunk or a hypertable as
input and returns the relstats for the given chunk or all chunks for
the given hypertable, respectively.
Importing relstats as described is useful as part of a distributed
ANALYZE/VACUUM that won't require fetching all data into the access
node for local sampling (like the current implemention does).
In a future change, this function will be called as part of a local
ANALYZE on the access node that runs ANALYZE on all data nodes
followed by importing of the resulting relstats for the analyzed
chunks.
