First step to remove the re-aggregation for Continuous Aggregates
is to remove the `chunk_id` from the materialization hypertable.
Also added new metadata column named `finalized` to `continuous_cagg`
catalog table in order to store information about the new following
finalized version of Continuous Aggregates that will not need the
partials anymore. This flag is important to maintain backward
compatibility with previous Continuous Aggregate implementation that
requires the `chunk_id` to refresh data properly.
Improve the performance of metadata scanning during hypertable
expansion.
When a hypertable is expanded to include all children chunks, only the
chunks that match the query restrictions are included. To find the
matching chunks, the planner first scans for all matching dimension
slices. The chunks that reference those slices are the chunks to
include in the expansion.
This change optimizes the scanning for slices by avoiding repeated
open/close of the dimension slice metadata table and index.
At the same time, related dimension slice scanning functions have been
refactored along the same line.
An index on the chunk constraint metadata table is also changed to
allow scanning on dimension_slice_id. Previously, dimension_slice_id
was the second key in the index, which made scans on this key less
efficient.
Reorganize the code and fix minor bug that was not computing the size
of FSM, VM and INIT forks of the parent hypertable.
Fixed the bug by exposing the `ts_relation_size` function to the SQL
level to encapsulate the logic to compute `heap`, `indexes` and `toast`
sizes.
This release is medium priority for upgrade. We recommend that you
upgrade at the next available opportunity.
This release adds major new features since the 2.5.2 release,
including:
Compression in continuous aggregates Experimental support for timezones
in continuous aggregates Experimental support for monthly buckets in
continuous aggregates It also includes several bug fixes. Telemetry
reports are switched to a new format, and now include more detailed
statistics on compression, distributed hypertables and indexes.
**Features**
* #3768 Allow ALTER TABLE ADD COLUMN with DEFAULT on compressed
hypertable
* #3769 Allow ALTER TABLE DROP COLUMN on compressed hypertable
* #3943 Optimize first/last
* #3945 Add support for ALTER SCHEMA on multi-node
* #3949 Add support for DROP SCHEMA on multi-node
**Bugfixes**
* #3808 Properly handle max_retries option
* #3863 Fix remote transaction heal logic
* #3869 Fix ALTER SET/DROP NULL contstraint on distributed hypertable
* #3944 Fix segfault in add_compression_policy
* #3961 Fix crash in EXPLAIN VERBOSE on distributed hypertable
* #4015 Eliminate float rounding instabilities in interpolate
* #4019 Update ts_extension_oid in transitioning state
* #4073 Fix buffer overflow in partition scheme
**Improvements**
Query planning performance is improved for hypertables with a large
number of chunks.
**Thanks**
* @fvannee for reporting a first/last memory leak
* @mmouterde for reporting an issue with floats and interpolate
This patch locks down search_path in extension install and update
scripts to only contain pg_catalog, this requires that any reference
in those scripts is fully qualified. Additionally we add explicit
create commands to all update scripts for objects added to the
public schema. This change will make update scripts fail if a
function with identical signature already exists when installing
or upgrading instead reusing the existing object.
TimescaleDB was vulnerable to a privilege escalation attack in
the extension installation script. An attacker could precreate
objects normally owned by the extension and get those objects
used in the installation script since the script would only try
to create them if they did not already exist. Thanks to Pedro
Gallegos for reporting the problem.
This patch changes the schema, table and function creation to fail
and abort the installation when the object already exists instead
of using the existing object.
Security: CVE-2022-24128
Refactor the telemetry function and format to include stats broken
down on common relation types. The types include:
- Tables
- Partitioned tables
- Hypertables
- Distributed hypertables
- Continuous aggregates
- Materialized views
- Views
and for each of these types report (when applicable):
- Total number of relations
- Total number of children/chunks
- Total data volume (broken into heap, toast, and indexes).
- Compression stats
- PG stats, like reltuples
The telemetry function has also been refactored to return `jsonb`
instead of `text`. This makes it easier to query and manipulate the
resulting JSON format, and also gives cleaner output.
Closes#3932
Enable ALTER MATERIALIZED VIEW (timescaledb.compress)
This enables compression on the underlying materialized
hypertable. The segmentby and orderby columns for
compression are based on the GROUP BY clause and time_bucket
clause used while setting up the continuous aggregate.
timescaledb_information.continuous_aggregate view defn
change
Add support for compression policy on continuous
aggregates
Move code from job.c to policy_utils.c
Add support functions to check compression
policy validity for continuous aggregates.
When deleting a data node, it is often convenient to be able to also
drop the database on the data node so that the node can be added again
using the same database name. However, dropping the database is
optional since it should be possible to delete a data node even if it
is no longer responding.
With the new functionality, a data node's database can be dropped as
follows:
```sql
SELECT delete_data_node('dn1', drop_database=>true);
```
Note that the default behavior is still to not drop the database in
order to be compatible with the old behavior. Enabling the option also
makes the function non-transactional, since dropping a database is not
transactional. Therefore, it is not possible to use this option in a
transaction block.
Closes#3876
This patch allows using time_bucket_ng("N month", ...) in CAGGs. Users can also
specify years, or months AND years. CAGGs on top of distributed hypertables
are supported as well.
When executing `recompress_chunk` and a query at the same time, a
deadlock can be generated because the chunk relation and the chunk
index and the compressed and uncompressd chunks are locked in different
orders. In particular, when `recompress_chunk` is executing, it will
first decompress the chunk and as part of that lock the uncompressed
chunk index in AccessExclusive mode and when trying to compress the
chunk again it will try to lock the uncompressed chunk in
AccessExclusive as part of truncating it.
Note that `decompress_chunk` and `compress_chunk` lock the relations in
the same order and the issue arises because the procedures are combined
inth a single transaction.
To avoid the deadlock, this commit rewrites the `recompress_chunk` to
be a procedure and adds a commit between the decompression and
compression. Committing the transaction after the decompress will allow
reads and inserts to proceed by working on the uncompressed chunk, and
the compression part of the procedure will take the necessary locks in
strict order, thereby avoiding a deadlock.
In addition, the isolation test is rewritten so that instead of adding
a waitpoint in the PL/SQL function, we implement the isolation test by
taking a lock on the compressed table after the decompression.
Fixes#3846
Commit fffd6c2350f5b3237486f3d49d7167105e72a55b fixes problem related
to PortalContext using PL/pgSQL procedure to execute the policy.
Unfortunately this new implementation introduced a problem when we use
INTEGER and not BIGINT for the time dimension.
Fixed it by dealing correclty with the integer types: SMALLINT, INTEGER
and BIGINT.
Also refatored the policy compression procedure replacing the two
procedures `policy_compression_{interval|integer}` by a simple
`policy_compression_execute` casting dimension type dynamically.
Fixes#3773
This release adds major new features since the 2.4.2 release.
We deem it moderate priority for upgrading.
This release includes these noteworthy features:
* Continuous Aggregates for Distributed Hypertables
* Support for PostgreSQL 14
* Experimental: Support for timezones in `time_bucket_ng()`, including
the `origin` argument
This release also includes several bug fixes.
**Features**
* #3034 Add support for PostgreSQL 14
* #3435 Add continuous aggregates for distributed hypertables
* #3505 Add support for timezones in `time_bucket_ng()`
**Bugfixes**
* #3580 Fix memory context bug executing TRUNCATE
* #3592 Allow alter column type on distributed hypertable
* #3598 Improve evaluation of stable functions such as now() on access
node
* #3618 Fix execution of refresh_caggs from user actions
* #3625 Add shared dependencies when creating chunk
* #3626 Fix memory context bug executing TRUNCATE
* #3627 Schema qualify UDTs in multi-node
* #3638 Allow owner change of a data node
* #3654 Fix index attnum mapping in reorder_chunk
* #3661 Fix SkipScan path generation with constant DISTINCT column
* #3667 Fix compress_policy for multi txn handling
* #3673 Fix distributed hypertable DROP within a procedure
* #3701 Allow anyone to use size utilities on distributed hypertables
* #3708 Fix crash in get_aggsplit
* #3709 Fix ordered append pathkey check
* #3712 Fix GRANT/REVOKE ALL IN SCHEMA handling
* #3717 Support transparent decompression on individual chunks
* #3724 Fix inserts into compressed chunks on hypertables with caggs
* #3727 Fix DirectFunctionCall crash in distributed_exec
* #3728 Fix SkipScan with varchar column
* #3733 Fix ANALYZE crash with custom statistics for custom types
* #3747 Always reset expr context in DecompressChunk
**Thanks**
* @binakot and @sebvett for reporting an issue with DISTINCT queries
* @hardikm10, @DavidPavlicek and @pafiti for reporting bugs on TRUNCATE
* @mjf for reporting an issue with ordered append and JOINs
* @phemmer for reporting the issues on multinode with aggregate queries and evaluation of now()
* @abolognino for reporting an issue with INSERTs into compressed hypertables that have cagg
* @tanglebones for reporting the ANALYZE crash with custom types on multinode
This PR removes the C code that executes the compression
policy. Instead we use a PL/pgSQL procedure to execute
the policy.
PG13.4 and PG12.8 introduced some changes
that require PortalContexts while executing transactions.
The compression policy procedure compresses chunks in
multiple transactions. We have seen some issues with snapshots
and portal management in the policy code (due to the
PG13.4 code changes). SPI API has transaction-portal management
code. However, the compression policy code does not use SPI
interfaces. But it is fairly easy to just convert this into
a PL/pgSQL procedure (which calls SPI) rather than replicating
portal managment code in C to manage multiple txns in the
compression policy.
This PR also disallows decompress_chunk, compress_chunk and
recompress_chunk in txn read only mode.
Fixes#3656
Update version numbers and add 2.4.2 to update tests.
We have to put the DROP FUNCTION back in latest-dev because
2.4.2 did not include the commit which removed the function
definitions.
This release contains bug fixes since the 2.4.1 release.
We deem it high priority to upgrade.
**Bugfixes**
* #3437 Rename on all continuous aggregate objects
* #3469 Use signal-safe functions in signal handler
* #3520 Modify compression job processing logic
* #3527 Fix time_bucket_ng behaviour with origin argument
* #3532 Fix bootstrap with regresschecks disabled
* #3574 Fix failure on job execution by background worker
* #3590 Call cleanup functions on backend exit
**Thanks**
* @jankatins for reporting a crash with background workers
* @LutzWeischerFujitsu for reporting an issue with bootstrap
When renaming a column on a continuous aggregate, only the user view
column was renamed. This commit changes this by applying the rename on
the materialized table, the user view, the direct view, and the partial
view, as well as the column name in the dimension table.
Since this also changes some of the table definitions, we need to
perform the same rename in the update scripts as well, which is done by
comparing the direct view and the user view to decide what columns that
require a rename and then executing that rename on the direct view,
partial view, and materialization table, as well as updating the column
name in the dimension table.
When renaming columns in tables with indexes, the column in the table
is renamed but not the column in the index, which keeps the same name.
When restoring from a dump, however, the column name of the table is
used, which create a diff in the update test. For that reason, we
change the update tests to not list index definitions as part of the
comparison. The existance of the indexes is still tracked and compared
since the indexes for a hypertable is listed as part of the output.
If a downgrade does not revert some changes, this will cause a diff in
the downgrade test. Since the rename is benign and not easy to revert,
this will cause test failure. Instead, we add a file to do extra
actions during a clean-rerun to prevent these diffs. In this case,
applying the same renames as the update script.
Fixes#3405
This release adds new experimental features since the 2.3.1 release.
The experimental features in this release are:
* APIs for chunk manipulation across data nodes in a distributed
hypertable setup. This includes the ability to add a data node and move
chunks to the new data node for cluster rebalancing.
* The `time_bucket_ng` function, a newer version of `time_bucket`. This
function supports years, months, days, hours, minutes, and seconds.
We’re committed to developing these experiments, giving the community
a chance to provide early feedback and influence the direction of
TimescaleDB’s development. We’ll travel faster with your input!
Please create your feedback as a GitHub issue (using the
experimental-schema label), describe what you found, and tell us the
steps or share the code snip to recreate it.
This release also includes several bug fixes.
PostgreSQL 11 deprecation announcement
Timescale is working hard on our next exciting features. To make that
possible, we require functionality that is available in Postgres 12 and
above. Postgres 11 is not supported with TimescaleDB 2.4.
**Experimental Features**
* #3293 Add timescaledb_experimental schema
* #3302 Add block_new_chunks and allow_new_chunks API to experimental
schema. Add chunk based refresh_continuous_aggregate.
* #3211 Introduce experimental time_bucket_ng function
* #3366 Allow use of experimental time_bucket_ng function in continuous aggregates
* #3408 Support for seconds, minutes and hours in time_bucket_ng
* #3446 Implement cleanup for chunk copy/move.
**Bugfixes**
* #3401 Fix segfault for RelOptInfo without fdw_private
* #3411 Verify compressed chunk validity for compressed path
* #3416 Fix targetlist names for continuous aggregate views
* #3434 Remove extension check from relcache invalidation callback
* #3440 Fix remote_tx_heal_data_node to work with only current database
**Thanks**
* @fvannee for reporting an issue with hypertable expansion in functions
* @amalek215 for reporting an issue with cache invalidation during pg_class vacuum full
* @hardikm10 for reporting an issue with inserting into compressed chunks
* @dberardo-com and @iancmcc for reporting an issue with extension updates after renaming columns of continuous aggregates.
A chunk copy/move operation is carried out in stages and it can
fail in any of them. We track the last completed stage in the
"chunk_copy_operation" catalog table. In case of failure, a
"chunk_copy_cleanup" function can be invoked to bring the chunk back
to its original state on the source datanode and all transient objects
like replication slot, publication, subscription, empty chunk, metadata
updates, etc are cleaned up.
Includes test case changes for each and every stage induced failure.
To avoid confusion between chunk copy activity and chunk copy operation
this patch also consistently uses "operation" everywhere now instead of
"activity"
Remove copy_chunk_data() function and code needed to support it,
such as the 'transactional' argument.
Rework copy chunk logic using separate stages.
Introduce copy_chunk() API function as an internal wrapper for
the move_chunk().
The building blocks required for implementing end-to-end copy/move
chunk functionality have now been wrapped in a procedure.
A procedure is required because multiple transactions are needed to
carry out the activity across the access node and the involved two data
nodes.
The following steps are encapsulated in this procedure
1) Create an empty chunk table on the destination data node
2) Copy the data from the src data node chunk to this newly created
destination node chunk. This is done via inbuilt PostgreSQL logical
replication functionality
3) Attach this chunk to the hypertable on the dst data node
4) Remove this chunk from the src data node to complete the move if
requested
A new catalog table "chunk_copy_activity" has been added to track
the progress of the above stages. A unique id gets assigned to each
activity and it is updated with the completed stages as things
progress.
The `create_chunk` API has been extended to allow creating a chunk
from an existing relational table. The table is turned into a chunk by
attaching it to the root hypertable via inheritance.
The purpose of this functionality is to allow copying a chunk to
another node. First, the chunk table and data is copied. After that,
the `create_chunk` can be executed to make the new table part of the
hypertable.
Currently, the relational table used to create the chunk has to match
the hypertable in terms of constraints, triggers, etc. PostgreSQL
itself enforces the existence of same-named CHECK constraints, but no
enforcement currently exists for other objects, including triggers
UNIQUE, PRIMARY KEY, or FOREIGN KEY constraints. Such enforcement can
be implemented in the future, if deemed necessary. Another option is
to automatically add all the required objects (triggers, constraints)
based on the hypertable equivalents. However, that might also lead to
duplicate objects in case some of them exist on the table prior to
creating the chunk.
Add 2.3.1 to the update tests and update the downgrade target for the
downgrade version. This commit also fixes two issues that were fixed in
the release branch:
1. Drop `_timescaledb_internal.refresh_continuous_aggregate`
2. Update the changelog to match the release branch.
**Bugfixes**
* #3279 Add some more randomness to chunk assignment
* #3288 Fix failed update with parallel workers
* #3300 Improve trigger handling on distributed hypertables
* #3304 Remove paths that reference parent relids for compressed chunks
* #3305 Fix pull_varnos miscomputation of relids set
* #3310 Generate downgrade script
* #3314 Fix heap buffer overflow in hypertable expansion
* #3317 Fix heap buffer overflow in remote connection cache.
* #3327 Make aggregate in caggs fully qualified
* #3327 Make aggregates in caggs fully qualified
* #3336 Fix pg_init_privs objsubid handling
* #3345 Fix SkipScan distinct column identification
* #3355 Fix heap buffer overflow when renaming compressed hypertable columns.
* #3367 Improve DecompressChunk qual pushdown
* #3377 Fix bad use of repalloc
**Thanks**
* @db-adrian for reporting an issue when accessing cagg view through postgres_fdw
* @fncaldas and @pgwhalen for reporting an issue accessing caggs when public is not in search_path
* @fvannee, @mglonnro and @ebreijo for reporting an issue with the upgrade script
* @fvannee for reporting a performance regression with SkipScan
This release adds major new features since the 2.2.1 release. We deem
it moderate priority for upgrading.
This release adds support for inserting data into compressed chunks
and improves performance when inserting data into distributed
hypertables. Distributed hypertables now also support triggers and
compression policies.
The bug fixes in this release address issues related to the handling
of privileges on compressed hypertables, locking, and triggers with
transition tables.
**Features**
* #3116 Add distributed hypertable compression policies
* #3162 Use COPY when executing distributed INSERTs
* #3199 Add GENERATED column support on distributed hypertables
* #3210 Add trigger support on distributed hypertables
* #3230 Support for inserts into compressed chunks
**Bugfixes**
* #3213 Propagate grants to compressed hypertables
* #3229 Use correct lock mode when updating chunk
* #3243 Fix assertion failure in decompress_chunk_plan_create
* #3250 Fix constraint triggers on hypertables
* #3251 Fix segmentation fault due to incorrect call to chunk_scan_internal
* #3252 Fix blocking triggers with transition tables
**Thanks**
* @yyjdelete for reporting a crash with decompress_chunk and identifying the bug in the code
* @fabriziomello for documenting the prerequisites when compiling against PostgreSQL 13
The view uses cached information from compression_chunk_size to
report the size of compressed chunks. Since compressed chunks
can be modified, we call pg_relation_size on the compressed chunk
while reporting the size
The view also incorrectly used the hypertable's reltoastrelid to
calculate toast bytes. It has been changed to use the chunk's
reltoastrelid.
Grants and revokes where not propagated to compressed hypertables, if
the hypertable had a compressed hypertable, meaning that `pg_dump` and
`pg_restore` would not be able to dump nor restore the compressed part
of a hypertable unless they were owners and/or was a superuser.
This commit fixes this by propagating grants and revokes on a
hypertable to the associated compressed hypertable, if one exists,
which will then include the compressed hypertable and the associated
chunks in the grant and revoke execution.
It also adds code to fix the permissions of compressed hypertables and
all associated chunks in an update and adds an update test to check
that the permissions match.
Fixes#3209
This release adds major new features since the 2.1.1 release.
We deem it moderate priority for upgrading.
This release adds the Skip Scan optimization, which significantly
improves the performance of queries with DISTINCT ON. This
optimization is not yet available for queries on distributed
hypertables.
This release also adds a function to create a distributed
restore point, which allows performing a consistent restore of a
multi-node cluster from a backup.
The bug fixes in this release address issues with size and stats
functions, high memory usage in distributed inserts, slow distributed
ORDER BY queries, indexes involving INCLUDE, and single chunk query
planning.
**PostgreSQL 11 deprecation announcement**
Timescale is working hard on our next exciting features. To make that
possible, we require functionality that is unfortunately absent on
PostgreSQL 11. For this reason, we will continue supporting PostgreSQL
11 until mid-June 2021. Sooner to that time, we will announce the
specific version of TimescaleDB in which PostgreSQL 11 support will
not be included going forward.
**Major Features**
* #2843 Add distributed restore point functionality
* #3000 SkipScan to speed up SELECT DISTINCT
**Bugfixes**
* #2989 Refactor and harden size and stats functions
* #3058 Reduce memory usage for distributed inserts
* #3067 Fix extremely slow multi-node order by queries
* #3082 Fix chunk index column name mapping
* #3083 Keep Append pathkeys in ChunkAppend
**Thanks**
* @BowenGG for reporting an issue with indexes with INCLUDE
* @fvannee for reporting an issue with ChunkAppend pathkeys
* @pedrokost and @RobAtticus for reporting an issue with size
functions on empty hypertables
* @phemmer and @ryanbooz for reporting issues with slow
multi-node order by queries
* @stephane-moreau for reporting an issue with high memory usage during
single-transaction inserts on a distributed hypertable.
Fix a number of issues with size and stats functions:
* Return `0` size instead of `NULL` in several functions when
hypertables have no chunks (e.g., `hypertable_size`,
`hypertable_detailed_size`).
* Return `NULL` when functions are called on non-hypertables instead
of simply failing with generic error `query returned no rows`.
* Include size of "root" hypertable, which can have non-zero size
indexes and other objects even if the root table holds no data.
* Make `hypertable_detailed_size` include one additional row for
storage size of objects on the access node. While the access node
stores no data, the empty hypertable may still take up some disk
space.
* Improve test coverage for all size utility functions. In particular,
add tests on regular tables as well as empty and compressed
hypertables.
* Several size utility functions that were defined as `PL/pgSQL`
functions have been converted to simple `SQL` functions since they
ran only a single SQL query.
The `dist_util` test is moved to the solo test group because,
otherwise, it gives different size output when run in parallel vs. in
isolation.
Fixes#2871
This maintenance release contains bugfixes since the 2.0.1 release. We
deem it high priority for upgrading.
The bug fixes in this release address issues with joins, the status of
background jobs, and disabling compression. It also includes
enhancements to continuous aggregates, including improved validation
of policies and optimizations for faster refreshes when there are a
lot of invalidations.
**Minor features**
* #2926 Optimize cagg refresh for small invalidations
**Bugfixes**
* #2850 Set status for backend in background jobs
* #2883 Fix join qual propagation for nested joins
* #2884 Add GUC to control join qual propagation
* #2885 Fix compressed chunk check when disabling compression
* #2908 Fix changing column type of clustered hypertables
* #2942 Validate continuous aggregate policy
**Thanks**
* @zeeshanshabbir93 for reporting the issue with full outer joins
* @Antiarchitect for reporting the issue with slow refreshes of
* @diego-hermida for reporting the issue about being unable to disable
compression
* @mtin for reporting the issue about wrong job status
The check for existence of compressed chunks when disabling
compression would not ignore dropped chunks making it impossible
to disable compression on hypertables with continuous aggregates
that had dropped chunks.
This patch ignores dropped chunks in this check and also sets
compressed_chunk_id to NULL in the metadata for deleted chunks.
This maintenance release contains bugfixes since the 2.0.0 release. We deem it
high priority for upgrading.
In particular the fixes contained in this maintenance release address issues
in continuous aggregates, compression, JOINs with hypertables and when
upgrading from previous versions.
**Bugfixes**
* #2772 Always validate existing database and extension
* #2780 Fix config enum entries for remote data fetcher
* #2806 Add check for dropped chunk on update
* #2828 Improve cagg watermark caching
* #2842 Do not mark job as started when setting next_start field
* #2845 Fix continuous aggregate privileges during upgrade
* #2851 Fix nested loop joins that involve compressed chunks
* #2860 Fix projection in ChunkAppend nodes
* #2861 Remove compression stat update from update script
* #2865 Apply volatile function quals at decompresschunk node
* #2866 Avoid partitionwise planning of partialize_agg
* #2868 Fix corruption in gapfill plan
* #2874 Fix partitionwise agg crash due to uninitialized memory
**Thanks**
* @alex88 for reporting an issue with joined hypertables
* @brian-from-quantrocket for reporting an issue with extension update and dropped chunks
* @dhodyn for reporting an issue when joining compressed chunks
* @markatosi for reporting a segfault with partitionwise aggregates enabled
* @PhilippJust for reporting an issue with add_job and initial_start
* @sgorsh for reporting an issue when using pgAdmin on windows
* @WarriorOfWire for reporting the bug with gapfill queries not being
able to find pathkey item to sort
Copy ACL privileges (grants) from the query view (user-facing object)
to the internal objects (e.g., materialized hypertable, direct view,
and partial view) when updating the extension to the new version. A
previous change added such propagation of privileges when executing
`GRANT` statements, but didn't apply it retrospectively to the
internal objects of existing continuous aggregates.
Having the right permissions on internal objects is also necessary for
the watermark function used by real-time aggregation since it queries
the materialized hypertable directly.
The update script copies the ACL information from the user-facing view
of every continuous aggregate to its internal objects (including the
materialized hypertable and its chunks). This is done by direct insert
into `pg_class` instead of executing a `GRANT` statement in the update
script, since the latter will record the grant/ACL as an init
privilege (i.e., the system believes the GRANT is for an extension
object). The init privilege will prevent this ACL from being included
in future dump files, since `pg_dump` only includes non-init
privileges as it believes such privileges will be recreated with the
extension.
Fixes#2825
With this release, we are officially moving TimescaleDB 2.0 to GA,
concluding several release candidates.
TimescaleDB 2.0 adds the much-anticipated support for distributed
hypertables (multi-node TimescaleDB), as well as new features and
enhancements to core functionality to give users better clarity and
more control and flexibility over their data.
Multi-node architecture: In particular, with TimescaleDB 2.0, users
can now create distributed hypertables across multiple instances of
TimescaleDB, configured so that one instance serves as an access node
and multiple others as data nodes. All queries for a distributed
hypertable are issued to the access node, but inserted data and queries
are pushed down across data nodes for greater scale and performance.
Multi-node TimescaleDB can be self managed or, for easier operation,
launched within Timescale's fully-managed cloud services.
This release also adds:
* Support for user-defined actions, allowing users to define,
customize, and schedule automated tasks, which can be run by the
built-in jobs scheduling framework now exposed to users.
* Significant changes to continuous aggregates, which now separate the
view creation from the policy. Users can now refresh individual
regions of the continuous aggregate materialized view, or schedule
automated refreshing via policy.
* Redesigned informational views, including new (and more general)
views for information about hypertable's dimensions and chunks,
policies and user-defined actions, as well as support for multi-node
TimescaleDB.
* Moving all formerly enterprise features into our Community Edition,
and updating Timescale License, which now provides additional (more
permissive) rights to users and developers.
Some of the changes above (e.g., continuous aggregates, updated
informational views) do introduce breaking changes to APIs and are not
backwards compatible. While the update scripts in TimescaleDB 2.0 will
upgrade databases running TimescaleDB 1.x automatically, some of these
API and feature changes may require changes to clients and/or upstream
scripts that rely on the previous APIs. Before upgrading, we recommend
reviewing upgrade documentation at docs.timescale.com for more details.
**Major Features**
TimescaleDB 2.0 moves the following major features to GA:
* #1923 Add support for distributed hypertables
* #2006 Add support for user-defined actions
* #2125#2221 Improve Continuous Aggregate API
* #2084#2089#2098#2417 Redesign informational views
* #2435 Move enterprise features to community
* #2437 Update Timescale License
**Previous Release Candidates**
* #2702 Release Candidate 4 (December 2, 2020)
* #2637 Release Candidate 3 (November 12, 2020)
* #2554 Release Candidate 2 (October 20, 2020)
* #2478 Release Candidate 1 (October 1, 2020)
**Minor Features**
Since the last release candidate 4, there are several minor
improvements:
* #2746 Optimize locking for create chunk API
* #2705 Block tableoid access on distributed hypertable
* #2730 Do not allow unique index on compressed hypertables
* #2764 Bootstrap data nodes with versioned extension
**Bugfixes**
Since the last release candidate 4, there are several bugfixes:
* #2719 Support disabling compression on distributed hypertables
* #2742 Fix compression status in chunks view for distributed chunks
* #2751 Fix crash and cancel when adding data node
* #2763 Fix check constraint on hypertable metadata table
**Thanks**
Thanks to all contributors for the TimescaleDB 2.0 release:
* @airton-neto for reporting a bug in executing some queries with UNION
* @nshah14285 for reporting an issue with propagating privileges
* @kalman5 for reporting an issue with renaming constraints
* @LbaNeXte for reporting a bug in decompression for queries with
subqueries
* @semtexzv for reporting an issue with continuous aggregates on
int-based hypertables
* @mr-ns for reporting an issue with privileges for creating chunks
* @cloud-rocket for reporting an issue with setting an owner on
continuous aggregate
* @jocrau for reporting a bug during creating an index with transaction
per chunk
* @fvannee for reporting an issue with custom time types
* @ArtificialPB for reporting a bug in executing queries with
conditional ordering on compressed hypertable
* @dutchgecko for reporting an issue with continuous aggregate datatype
handling
* @lambdaq for suggesting to improve error message in continuous
aggregate creation
* @francesco11112 for reporting memory issue on COPY
* @Netskeh for reporting bug on time_bucket problem in continuous
aggregates
* @mr-ns for reporting the issue with CTEs on distributed hypertables
* @akamensky for reporting an issue with recursive cache invalidation
* @ryanbooz for reporting slow queries with real-time aggregation on
continuous aggregates
* @cevian for reporting an issue with disabling compression on
distributed hypertables
The `replication_factor` is set to `-1` on hypertables that are
created on data nodes as part of a larger distributed
hypertable. However, the check constraint on the hypertable metadata
table doesn't allow such values, causing update scripts to fail when
this check constraint is recreated as part of updating to version
`2.0.0-rc4`.
The reason it is possible to insert violating rows is because check
constraints aren't validated when inserting data using PostgreSQL's
internal catalog functions (in C). Therefore, the violating row can
exist until one tries to update a data node to `2.0.0-rc4`, at which
point the update script tries to recreate the `hypertable` metadata
table due to other changes that were made to the table.
This change fixes the check constraint to account for `-1` as a valid
value, and also changes the update scripts to account for the new
check constraint so that updates to the latest version will no longer
fail.
This release candidate contains bugfixes since the previous release
candidate, as well as additional minor features. It improves
validation of configuration changes for background jobs, adds support
for gapfill on distributed tables, contains improvements to the memory
handling for large COPY, and contains improvements to compression for
distributed hypertables.
**Minor Features**
* #2689 Check configuration in alter_job and add_job
* #2696 Support gapfill on distributed hypertable
* #2468 Show more information in get_git_commit
* #2678 Include user actions into job stats view
* #2664 Fix support for complex aggregate expression
* #2672 Add hypertable to continuous aggregates view
* #2662 Save compression metadata settings on access node
* #2707 Introduce additional db for data node bootstrapping
**Bugfixes**
* #2688 Fix crash for concurrent drop and compress chunk
* #2666 Fix timeout handling in async library
* #2683 Fix crash in add_job when given NULL interval
* #2698 Improve memory handling for remote COPY
* #2555 Set metadata for chunks compressed before 2.0
Add a test for updating statistics on chunks compressed before
statistics where kept. This patch also sets metadata for previously
compressed chunks so running ANALYZE on them will set the correct
tuplecount.
1. Add compression_state column for hypertable catalog
by renaming compressed column for the hypertable catalog
table. compression_state is a tri-state column.
This column indicates if the hypertable has
compression enabled (value = 1) or if it is an internal
compression table (value = 2).
2. Save compression settings on access node when compression
is turned on for a distributed hypertable
For a distributed hypertable, that has compression enabled,
compression_state is set. We don't create any internal tables
on the access node.
Fixes#2660
The command `get_git_commit` shows quite a little information and
especially if mistakes are made it is hard to verify what the binary is
actually based on.
This commit extends the function to provide some more information:
specifically the SHA of the HEAD regardless of whether there is a tag
on it or not and the time for the HEAD commit.
```
postgres=# select * from _timescaledb_internal.get_git_commit();
commit_tag | commit_hash | commit_time
--------------------------+-------------+------------------------
1.7.4-10-g09b0b778-dirty | 09b0b778 | 2020-09-13 17:50:38+02
(1 row)
```
If git is not installed, the function `get_git_commit` will return an
error indicating that no git information is available. If some of the
fields are available, they will be emitted and the remaining fields
will be NULL.
Fixes#2457
timescale_information.job_stats view was missing information about
jobs running user actions, since they don't have associated
hypertables. This commit fixes that the view includes user jobs in
addition to the predefined policies, which were already covered as all
of them reference a hypertable.
Add the hypertable's schema and name to the continuous aggregates view
in the information schema, since these fields where missing.
The new fields use the same names and order in the view to be
consistent with other information views that reference the hypertable.
Fixes#2653
This release candidate contains bugfixes since the previous release
candidate, as well as additional minor features including support for
"user-mapping" authentication between access/data nodes and an
experimental API for refreshing continuous aggregates on individual
chunks.
**Minor Features**
* #2627 Add optional user mappings support
* #2635 Add API to refresh continuous aggregate on chunk
**Bugfixes**
* #2560 Fix SCHEMA DROP CASCADE with continuous aggregates
* #2593 Set explicitly all lock parameters in alter_job
* #2604 Fix chunk creation on hypertables with foreign key constraints
* #2610 Support analyze of internal compression table
* #2612 Optimize internal cagg_watermark function
* #2613 Refresh correct partial during refresh on drop
* #2617 Fix validation of available extensions on data node
* #2619 Fix segfault in decompress_chunk for chunks with dropped columns
* #2620 Fix DROP CASCADE for continuous aggregate
* #2625 Fix subquery errors when using AsyncAppend
* #2626 Fix incorrect total_table_pages setting for compressed scan
* #2628 Stop recursion in cache invalidation
