timescale_information.job_stats view was missing information about
jobs running user actions, since they don't have associated
hypertables. This commit fixes that the view includes user jobs in
addition to the predefined policies, which were already covered as all
of them reference a hypertable.
Add the hypertable's schema and name to the continuous aggregates view
in the information schema, since these fields where missing.
The new fields use the same names and order in the view to be
consistent with other information views that reference the hypertable.
Fixes#2653
This release candidate contains bugfixes since the previous release
candidate, as well as additional minor features including support for
"user-mapping" authentication between access/data nodes and an
experimental API for refreshing continuous aggregates on individual
chunks.
**Minor Features**
* #2627 Add optional user mappings support
* #2635 Add API to refresh continuous aggregate on chunk
**Bugfixes**
* #2560 Fix SCHEMA DROP CASCADE with continuous aggregates
* #2593 Set explicitly all lock parameters in alter_job
* #2604 Fix chunk creation on hypertables with foreign key constraints
* #2610 Support analyze of internal compression table
* #2612 Optimize internal cagg_watermark function
* #2613 Refresh correct partial during refresh on drop
* #2617 Fix validation of available extensions on data node
* #2619 Fix segfault in decompress_chunk for chunks with dropped columns
* #2620 Fix DROP CASCADE for continuous aggregate
* #2625 Fix subquery errors when using AsyncAppend
* #2626 Fix incorrect total_table_pages setting for compressed scan
* #2628 Stop recursion in cache invalidation
Add an optional password parameter to `add_data_node` so that users
that don't have a password in a `passfile` on the access node can add
data nodes using password authentication. Together with user mappings,
this allows full multinode configuration without relying on passwords
or certificates provided in external/on-disk files.
While wasswords can be provided in the database via a user mapping
object, such a mapping is created on a per-server basis and requires
the foreign server to exist prior to creating the mapping. When adding
a data node, however, bootstrapping and/or validation of the data node
happens at the same time as the server object is created, which means
no user mapping can be created prior to adding the data
node. Therefore, the password must be provided as an argument to add
data node instead of via a user mapping.
Fortunately, using a function parameter might be preferred to a user
mapping since the (plaintext) password won't be stored in the
database. A user mapping for the user that created the data node can
optionally be added after the data node has been added. But it might
be desirable to only create user mappings for unprivileged users that
will mostly interact only with specific distributed hypertables.
This release candidate contains bugfixes since the previous release candidate.
**Minor Features**
* #2520 Support non-transactional distibuted_exec
**Bugfixes**
* #2307 Overflow handling for refresh policy with integer time
* #2503 Remove error for correct bootstrap of data node
* #2507 Fix validation logic when adding a new data node
* #2510 Fix outer join qual propagation
* #2514 Lock dimension slices when creating new chunk
* #2515 Add if_attached argument to detach_data_node()
* #2517 Fix member access within misaligned address in chunk_update_colstats
* #2525 Fix index creation on hypertables with dropped columns
* #2543 Pass correct status to lock_job
* #2544 Assume custom time type range is same as bigint
* #2563 Fix DecompressChunk path generation
* #2564 Improve continuous aggregate datatype handling
* #2568 Change use of ssl_dir GUC
* #2571 Make errors and messages conform to style guide
* #2577 Exclude compressed chunks from ANALYZE/VACUUM
In #2514 a a race condition between inserts and `drop_chunks` is fixed
and this commit will repair the dimension slices table by
re-constructing missing dimension slices from the corresponding
constraint expressions.
Closes#1986
When upgrading from 1.7, it's possible to have retention policies
which overlap with continuous aggregates. These make use of the
cascade_to_materializations parameter to avoid invalidating the
aggregate.
In 2.0 there is no equivalent behavior to prevent the retention from
disrupting the aggregate. So during the 2.0 upgrade, check for any
running retention policies that are dropping chunks still used by a
continuous aggregate and suspend them (scheduled=>false). This will
also print a notice informing the user of what happened and how to
resume the retention policy if that's what they truly want.
Fixes#2530
The `modification_time` column is hard to maintain with any level of
consistency over merges and splits of invalidation ranges so this
commit removes it from the invalidation log entries for both
hypertables and continuous aggregates. If the modification time is
needed in the future, we need to re-introduce it in a manner that can
maintain it over both merges and splits.
THe function `ts_get_now_internal` is also removed since it is not used
any more.
Part of #2521
Removes unlrelated column schedule_interval from
timescaledb_information.continuous_aggregates view and simplifies it.
Renames argument cagg in refresh_continuous_aggregate into
continuous_aggregate as in add_continuous_aggregate_policy.
Part of #2521
Before this commit, executions of `distributed_exec` was always
transactional and this could not be disabled. This cause problems when
executing statements that cannot be executed inside a transaction, such
as `CREATE DATABASE`.
With this commit, we introduce a parameter `transactional` to
`distributed_exec` that allow non-transactional executions on data
nodes. The parameter defaults to `TRUE`. Generates an error if
`distributed_exec` is non-transactional and inside a transaction.
It also changes `distributed_exec` to be a procedure, which better
matches it's usage (it returns `VOID`).
Closes#1660
This change makes detach_data_node() function consistent with
other data node management functions by adding missing
if_attach argument.
The function will not show an error in case if data node is not
attached and if_attached is set to true.
Issue: #2506
This release adds major new features and bugfixes since the 1.7.4 release.
We deem it moderate priority for upgrading.
This release adds the long-awaited support for distributed hypertables to
TimescaleDB. With 2.0, users can create distributed hypertables across
multiple instances of TimescaleDB, configured so that one instance serves
as an access node and multiple others as data nodes. All queries for a
distributed hypertable are issued to the access node, but inserted data
and queries are pushed down across data nodes for greater scale and
performance.
This release also adds support for user-defined actions allowing users to
define actions that are run by the TimescaleDB automation framework.
In addition to these major new features, the 2.0 branch introduces _breaking_ changes
to APIs and existing features, such as continuous aggregates. These changes are not
backwards compatible and might require changes to clients and/or scripts that rely on
the previous APIs. Please review our updated documentation and do proper testing to
ensure compatibility with your existing applications.
The noticeable breaking changes in APIs are:
- Redefined functions for policies
- A continuous aggregate is now created with `CREATE MATERIALIZED VIEW`
instead of `CREATE VIEW` and automated refreshing requires adding a policy
via `add_continuous_aggregate_policy`
- Redesign of informational views, including new (and more general) views for
information about policies and user-defined actions
This release candidate is upgradable, so if you are on a previous release (e.g., 1.7.4)
you can upgrade to the release candidate and later expect to be able to upgrade to the
final 2.0 release. However, please carefully consider your compatibility requirements
_before_ upgrading.
**Major Features**
* #1923 Add support for distributed hypertables
* #2006 Add support for user-defined actions
* #2435 Move enterprise features to community
* #2437 Update Timescale License
**Minor Features**
* #2011 Constify TIMESTAMPTZ OP INTERVAL in constraints
* #2105 Support moving compressed chunks
**Bugfixes**
* #1843 Improve handling of "dropped" chunks
* #1886 Change ChunkAppend leader to use worker subplan
* #2116 Propagate privileges from hypertables to chunks
* #2263 Fix timestamp overflow in time_bucket optimization
* #2270 Fix handling of non-reference counted TupleDescs in gapfill
* #2325 Fix rename constraint/rename index
* #2370 Fix detection of hypertables in subqueries
* #2376 Fix caggs width expression handling on int based hypertables
* #2416 Check insert privileges to create chunk
* #2428 Allow owner change of continuous aggregate
* #2436 Propagate grants in continuous aggregates
Renaming the parameter `hypertable_or_cagg` in functions `drop_chunks`
and `show_chunks` to `relation` and changing parameter name from
`main_table` to `hypertable` or `relation` depending on context.
This change will add an invalidation to the
materialization_invalidation_log for any region earlier than the
ignore_invalidation_older_than parameter when updating a continuous
aggregate to 2.0. This is needed as we do not record invalidations
in this region prior to 2.0 and there is no way to ensure the
aggregate is up to date within this range.
Fixes#2450
This patch removes enterprise license support and moves
move_chunk() function under community license (TSL).
Licensing validation code been reworked and simplified.
Previously used timescaledb.license_key guc been renamed to
timescaledb.license.
This change also makes testing code more strict against
used license. Apache test suite now can test only apache-licensed
functions.
Fixes#2359
Commit 8e1e6036 changed chunk compression to disable autovacuum
on compressed chunks but did not apply the setting to chunks
compressed before that change. So this patch changes chunks
compressed with previous version to disable autovacuum as well.
As part of the 2.0 continous aggregate changes, we are removing the
continuous_aggs_completed_threshold table. However, this may result
in currently running aggregates being considered complete even if
their completed threshold hadn't reached the invalidation threshold.
This change fixes this by adding an entry to the invalidation log
for any such aggregates.
Fixes#2314
This change removes the catalog options `refresh_lag`,
`max_interval_per_job` and `ignore_invalidation_older_than`, which are
no longer used.
Closes#2396
When rebuilding the bgw_job table the update script wouldnt remember
the state of the sequence and reset it back to the default leading
to failed job inserts until the sequence catches up.
When the extension is updated to 2.0, we need to migrate
existing ignore_invalidation_older_than settings to the new
continuous aggregate policy framework.
ignore_invalidation_older_than setting is mapped to start_interval
of the refresh policy.If the default value is used, it is mapped
to NULL start_interval, otherwise it is converted to an
interval value.
This change renames function to approximate_row_count() and adds
support for regular tables. Return a row count estimate for a
table instead of a table list.
Support add and remove continuous agg policy functions
Integrate policy execution with refresh api for continuous
aggregates
The old api for continuous aggregates adds a job automatically
for a continuous aggregate. This is an explicit step with the
new API. So remove this functionality.
Refactor some of the utility functions so that the code can be shared
by multiple policies.
This change removes, simplifies, and unifies code related to
`drop_chunks` and `show_chunks`. As a result of prior changes to
`drop_chunks`, e.g., making table relid mandatory and removing
cascading options, there's an opportunity to clean up and simplify the
rather complex code for dropping and showing chunks.
In particular, `show_chunks` is now consistent with `drop_chunks`; the
relid argument is mandatory, a continuous aggregate can be used in
place of a hypertable, and the input time ranges are checked and
handled in the same way.
Unused code is also removed, for instance, code that cascaded drop
chunks to continuous aggregates remained in the code base while the
option no longer exists.
This patch adds functionality to schedule arbitrary functions
or procedures as background jobs.
New functions:
add_job(
proc REGPROC,
schedule_interval INTERVAL,
config JSONB DEFAULT NULL,
initial_start TIMESTAMPTZ DEFAULT NULL,
scheduled BOOL DEFAULT true
)
Add a job that runs proc every schedule_interval. Proc can
be either a function or a procedure implemented in any language.
delete_job(job_id INTEGER)
Deletes the job.
run_job(job_id INTEGER)
Execute a job in the current session.
This patch adds policies to the update test to ensure their
configuration is properly migrated during updates. This patch
also fixes the inconsistent background job application_name
and adjusts them in the update script.
This patch changes the application name for background worker jobs
to include the job_id which makes the application name unique and
allows joining against pg_stat_activity to get a list of currently
running background worker processes. This change also makes
identifying misbehaving jobs easier from the postgres log as the
application name can be included in the log line.
The parameter `cascade_to_materialization` is removed from
`drop_chunks` and `add_drop_chunks_policy` as well as associated tables
and test functions.
Fixes#2137
This change replaces the add_drop_chunks_policy function with
add_retention_policy. This also renames the older_than parameter
of that function as retention_window. Likewise, the
remove_drop_chunks_policy is also being renamed
remove_retention_policy.
Fixes#2119
This patch adds a proc_name, proc_schema, hypertable_id index to
bgw_job. 3 functions using the new index are added as well:
ts_bgw_job_find_by_proc
ts_bgw_job_find_by_hypertable_id
ts_bgw_job_find_by_proc_and_hypertable_id
These functions are required for migrating the existing policies
to store their configuration in bgw_job directly.
This maintenance release contains bugfixes since the 1.7.1 release. We deem it medium
priority for upgrading.
In particular the fixes contained in this maintenance release address bugs in continuous
aggregates, drop_chunks and compression.
**Features**
* #1877 Add support for fast pruning of inlined functions
**Bugfixes**
* #1908 Fix drop_chunks with unique constraints when cascade_to_materializations is false
* #1915 Check for database in extension_current_state
* #1918 Unify chunk index creation
* #1932 Change compression locking order
* #1938 Fix gapfill locf treat_null_as_missing
* #1982 Check for disabled telemetry earlier
* #1984 Fix compression bit array left shift count
* #1997 Add checks for read-only transactions
* #2002 Reset restoring gucs rather than explicitly setting 'off'
* #2028 Fix locking in drop_chunks
* #2031 Enable compression for tables with compound foreign key
* #2039 Fix segfault in create_trigger_handler
* #2043 Fix segfault in cagg_update_view_definition
* #2046 Use index tablespace during chunk creation
* #2047 Better handling of chunk insert state destruction
* #2049 Fix handling of PlaceHolderVar in DecompressChunk
* #2051 Fix tuple concurrently deleted error with multiple continuous aggregates
**Thanks**
* @akamensky for reporting an issue with telemetry and an issue with drop_chunks
* @darko408 for reporting an issue with decompression
* @dmitri191 for reporting an issue with failing background workers
* @eduardotsj for reporting an issue with indexes not inheriting tablespace settings
* @fourseventy for reporting an issue with multiple continuous aggregrates
* @fvannee for contributing optimizations for pruning inlined functions
* @jflambert for reporting an issue with failing telemetry jobs
* @nbouscal for reporting an issue with compression jobs locking referenced tables
* @nicolai6120 for reporting an issue with locf and treat_null_as_missing
* @nomanor for reporting an issue with expression index with table references
* @olernov for contributing a fix for compressing tables with compound foreign keys
* @werjo for reporting an issue with drop_chunks and unique constraints
If a new chunk is created as part of an insert and drop_chunks runs
concurrently with the insert, there is a risk of a race. This is a test
for this.
Add locks for dimension slice tuples
If a dimension slice tuple is found while adding new chunk constraints
as part of a chunk creation it is not locked prior to adding the chunk
constraint. Hence a concurrently executing `drop_chunks` can find a
dimension slice unused (because there is no chunk constraint that
references it) and subsequently remove it. The insert will the continue
to add the chunk constraint with a reference to a now non-existent
dimension slice.
This commit fixes this by locking the dimension slice tuple with a
share lock when creating chunks and locking the dimension slice with an
exclusive lock prior to scanning for existing chunk constraints.
The commit also contains a script that repair the `dimension_slice`
table if it is broken by extracting information about dimension slices
that are mentioned in `chunk_constraint` table but not present in
`dimension_slice` table and re-create the rows from the constraints on
the chunks.
The `drop_chunks` function is refactored to make table name mandatory
for the function. As a result, the function was also refactored to
accept the `regclass` type instead of table name plus schema name and
the parameters were reordered to match the order for `show_chunks`.
The commit also refactor the code to pass the hypertable structure
between internal functions rather than the hypertable relid and moving
error checks to the PostgreSQL function. This allow the internal
functions to avoid some lookups and use the information in the
structure directly and also give errors earlier instead of first
dropping chunks and then error and roll back the transaction.
This commit removes the `cascade` option from the function
`drop_chunks` and `add_drop_chunk_policy`, which will now never cascade
drops to dependent objects. The tests are fixed accordingly and
verbosity turned up to ensure that the dependent objects are printed in
the error details.
The timescale clustering code so far has been written referring to the
remote databases as 'servers'. This terminology is a bit overloaded,
and in particular we don't enforce any network topology limitations
that the term 'server' would suggest. In light of this we've decided
to change to use the term 'node' when referring to the different
databases in a distributed database. Specifically we refer to the
frontend as an 'access node' and to the backends as 'data nodes',
though we may omit the access or data qualifier where it's unambiguous.
As the vast bulk of the code so far has been written for the case where
there was a single access node, almost all instances of 'server' were
references to data nodes. This change has updated the code to rename
those instances.
This functionality enables users to block or allow creation of new
chunks on a data node for one or more hypertables. Use cases for this
include the ability to block new chunks when a data node is running
low on disk space or to affect chunk distribution across data nodes.
Sometimes blocking data nodes for new chunks can make a hypertable
under-replicated. For that case an additional argument `force => true`
can be supplied to force blocking new chunks.
Here are some examples.
Block for a specific hypertable:
`SELECT * FROM block_new_chunks_on_server('server_1', 'disttable');`
Block for all hypertables on the server:
`SELECT * FROM block_new_chunks_on_server('server_1', force =>true);`
Unblock:
`SELECT * FROM allow_new_chunks_on_server('server_1', true);`
This change adds the `force` argument to `detach_server` as well. If
detaching or blocking new chunks will make a hypertable
under-replicated then `force => true` needs to used.
This commit adds the ability to resolve whether or not 2PC
transactions have been committed or aborted and also adds a heal
function to resolve transactions that have been prepared but not
committed or rolled back.
This commit also removes the server id of the primary key on the
remote_txn table and adds another index. This was done because the
`remote_txn_persistent_record_exists` should not rely on the server
being contacted but should rather just check for the existance of the
id. This makes the resolution safe to setups where two frontend server
definitions point to the same database. While this may not be a
properly configured setup, it's better if the resolution process is
robust to this case.
