Rewrite approximate_row_count to SQL instead of PLpgSQL and remove
superfluous JOINs against pg_namespace. Adjust tuple calculation
for PG14 since in PG14 reltuples for partitioned tables is the sum
of it's children so we need to exclude those from calculation to
not doublecount.
This patch marks time_bucket_ng() as IMMUTABLE. Two exceptions are:
- time_bucket_ng(interval, timestamptz) timestamptz
- time_bucket_ng(interval, timestamptz, timestamptz) timestamptz
... due to their implementation, see the comments. These two overloaded
versions were introduced only for backward compatibility with time_bucket()
and are not needed for building continuous aggregates.
This patch adds a version of time_bucket_ng() with 'origin' argument.
It doesn't address any other known issues. E.g. volatility of the function
will be changed in another patch. The error messages are going to be improved
when the feature gets a little more stable.
This patch adds support of timezones in time_bucket_ng(). The 'origin'
argument can't be used with timezones yet. This will be implemented in
a separate pull request.
This release contains bug fixes since the 2.4.0 release. We deem it
high priority to upgrade.
The release fixes continous aggregate refresh for postgres 12.8 and
13.4, a crash with ALTER TABLE commands and a crash with continuous
aggregates with HAVING clause.
**Bugfixes**
* #3430 Fix havingqual processing for continuous aggregates
* #3468 Disable tests by default if tools are not found
* #3462 Fix crash while tracking alter table commands
* #3489 Fix continuous agg bgw job failure for PG 12.8 and 13.4
* #3494 Improve error message when adding data nodes
**Thanks**
* @brianbenns for reporting a segfault with continuous aggregates
* @brianbenns for reporting a segfault with continuous aggregates
* @usego for reporting an issue with continuous aggregate refresh on PG 13.4
This release adds new experimental features since the 2.3.1 release.
The experimental features in this release are:
* APIs for chunk manipulation across data nodes in a distributed
hypertable setup. This includes the ability to add a data node and move
chunks to the new data node for cluster rebalancing.
* The `time_bucket_ng` function, a newer version of `time_bucket`. This
function supports years, months, days, hours, minutes, and seconds.
We’re committed to developing these experiments, giving the community
a chance to provide early feedback and influence the direction of
TimescaleDB’s development. We’ll travel faster with your input!
Please create your feedback as a GitHub issue (using the
experimental-schema label), describe what you found, and tell us the
steps or share the code snip to recreate it.
This release also includes several bug fixes.
PostgreSQL 11 deprecation announcement
Timescale is working hard on our next exciting features. To make that
possible, we require functionality that is available in Postgres 12 and
above. Postgres 11 is not supported with TimescaleDB 2.4.
**Experimental Features**
* #3293 Add timescaledb_experimental schema
* #3302 Add block_new_chunks and allow_new_chunks API to experimental
schema. Add chunk based refresh_continuous_aggregate.
* #3211 Introduce experimental time_bucket_ng function
* #3366 Allow use of experimental time_bucket_ng function in continuous aggregates
* #3408 Support for seconds, minutes and hours in time_bucket_ng
* #3446 Implement cleanup for chunk copy/move.
**Bugfixes**
* #3401 Fix segfault for RelOptInfo without fdw_private
* #3411 Verify compressed chunk validity for compressed path
* #3416 Fix targetlist names for continuous aggregate views
* #3434 Remove extension check from relcache invalidation callback
* #3440 Fix remote_tx_heal_data_node to work with only current database
**Thanks**
* @fvannee for reporting an issue with hypertable expansion in functions
* @amalek215 for reporting an issue with cache invalidation during pg_class vacuum full
* @hardikm10 for reporting an issue with inserting into compressed chunks
* @dberardo-com and @iancmcc for reporting an issue with extension updates after renaming columns of continuous aggregates.
A chunk copy/move operation is carried out in stages and it can
fail in any of them. We track the last completed stage in the
"chunk_copy_operation" catalog table. In case of failure, a
"chunk_copy_cleanup" function can be invoked to bring the chunk back
to its original state on the source datanode and all transient objects
like replication slot, publication, subscription, empty chunk, metadata
updates, etc are cleaned up.
Includes test case changes for each and every stage induced failure.
To avoid confusion between chunk copy activity and chunk copy operation
this patch also consistently uses "operation" everywhere now instead of
"activity"
The internal function `copy_chunk_data` was removed as part of
refactoring and is no longer necessary to remove in the downgrade
script since the function was never part of a release.
A new view in the experimental schema shows information related to
chunk replication. The view can be used to learn the replication
status of a chunk while also providing a way to easily find nodes to
move or copy chunks between in order to ensure a fully replicated
multi-node cluster.
Tests have been added to illustrate the potential usage.
Remove copy_chunk_data() function and code needed to support it,
such as the 'transactional' argument.
Rework copy chunk logic using separate stages.
Introduce copy_chunk() API function as an internal wrapper for
the move_chunk().
The building blocks required for implementing end-to-end copy/move
chunk functionality have now been wrapped in a procedure.
A procedure is required because multiple transactions are needed to
carry out the activity across the access node and the involved two data
nodes.
The following steps are encapsulated in this procedure
1) Create an empty chunk table on the destination data node
2) Copy the data from the src data node chunk to this newly created
destination node chunk. This is done via inbuilt PostgreSQL logical
replication functionality
3) Attach this chunk to the hypertable on the dst data node
4) Remove this chunk from the src data node to complete the move if
requested
A new catalog table "chunk_copy_activity" has been added to track
the progress of the above stages. A unique id gets assigned to each
activity and it is updated with the completed stages as things
progress.
Add internal copy_chunk_data() function which implements a way
to copy chunk data between data nodes using logical
replication.
This patch prepared together with @nikkhils.
The `create_chunk` API has been extended to allow creating a chunk
from an existing relational table. The table is turned into a chunk by
attaching it to the root hypertable via inheritance.
The purpose of this functionality is to allow copying a chunk to
another node. First, the chunk table and data is copied. After that,
the `create_chunk` can be executed to make the new table part of the
hypertable.
Currently, the relational table used to create the chunk has to match
the hypertable in terms of constraints, triggers, etc. PostgreSQL
itself enforces the existence of same-named CHECK constraints, but no
enforcement currently exists for other objects, including triggers
UNIQUE, PRIMARY KEY, or FOREIGN KEY constraints. Such enforcement can
be implemented in the future, if deemed necessary. Another option is
to automatically add all the required objects (triggers, constraints)
based on the hypertable equivalents. However, that might also lead to
duplicate objects in case some of them exist on the table prior to
creating the chunk.
This function drops a chunk on a specified data node. It then removes
the metadata about the datanode, chunk association on the access node.
This function is meant for internal use as part of the "move chunk"
functionality.
If only one chunk replica remains then this function refuses to drop it
to avoid data loss.
Creates a table for chunk replica on the given data node. The table
gets the same schema and name as the chunk. The created chunk replica
table is not added into metadata on the access node or data node.
The primary goal is to use it during copy/move chunk.
Adds an internal API function to create an empty chunk table according
the given hypertable for the given chunk table name and dimension
slices. This functions creates a chunk table inheriting from the
hypertable, so it guarantees the same schema. No TimescaleDB's
metadata is updated.
To be able to create the chunk table in a tablespace attached to the
hyeprtable, this commit allows calculating the tablespace id without
the dimension slice to exist in the catalog.
If there is already a chunk, which collides on dimension slices, the
function fails to create the chunk table.
The function will be used internally in multi-node to be able to
replicate a chunk from one data node to another.
As a future replacement for time_bucket(), time_bucket_ng()
should support seconds, minutes, and hours. This patch adds
this support. The implementation is the same as for
time_bucket(). Timezones are not yet supported.
Add 2.3.1 to the update tests and update the downgrade target for the
downgrade version. This commit also fixes two issues that were fixed in
the release branch:
1. Drop `_timescaledb_internal.refresh_continuous_aggregate`
2. Update the changelog to match the release branch.
**Bugfixes**
* #3279 Add some more randomness to chunk assignment
* #3288 Fix failed update with parallel workers
* #3300 Improve trigger handling on distributed hypertables
* #3304 Remove paths that reference parent relids for compressed chunks
* #3305 Fix pull_varnos miscomputation of relids set
* #3310 Generate downgrade script
* #3314 Fix heap buffer overflow in hypertable expansion
* #3317 Fix heap buffer overflow in remote connection cache.
* #3327 Make aggregate in caggs fully qualified
* #3327 Make aggregates in caggs fully qualified
* #3336 Fix pg_init_privs objsubid handling
* #3345 Fix SkipScan distinct column identification
* #3355 Fix heap buffer overflow when renaming compressed hypertable columns.
* #3367 Improve DecompressChunk qual pushdown
* #3377 Fix bad use of repalloc
**Thanks**
* @db-adrian for reporting an issue when accessing cagg view through postgres_fdw
* @fncaldas and @pgwhalen for reporting an issue accessing caggs when public is not in search_path
* @fvannee, @mglonnro and @ebreijo for reporting an issue with the upgrade script
* @fvannee for reporting a performance regression with SkipScan
During an update, it is not possible to run the downgrade scripts until
the release has been tagged, but the update scripts can be run. This
means that we need to split the previous version into two different
fields: one for running the update tests and one for running the
downgrade tests.
From PG12 on, CREATE OR REPLACE is supported for aggregates,
therefore, since we have dropped support for PG11, we can avoid
going through the rigamarole of having our aggregates in a separate
file from the functions we define to support them. Nor do we need to
handle aggregates separately from other functions as their creation
is now idempotent.
This function is IMMUTABLE when it doesn't accept timestamptz arguments,
and STABLE otherwise. See the comments in sql/time_bucket_ng.sql for
more details.
This commit add functions and code to generate a downgrade script from
the current version to the previous version. This requires execution
from a Git repository since it retrieves the prolog and epilog for the
"downgrade" file from the version given by `update_from_version` in the
`version.config` file.
The commit adds several CMake functions that simplifies the composition
of script files, but these are not used to generate the update scripts.
A potential improvement is to use the scripts to also generate the
update scripts.
This commit supports generating a downgrade script from the
current version to the previous version. Other versions are handled
using a variable containing file names of reverse update
scripts and the source and target version is extracted from the file
names, which is assumed to be of the form
`<source-version>--<target-version>.sql`.
In addition to adding support for generating downgrade scripts, the
commit adds a downgrade test file that tests a release in a similar way
to the update script and adds it as a workflow.
This patch adds time_bucket_ng() function to the experimental
schema. The "ng" part stands for "next generation". Unlike
current time_bucket() implementation the _ng version will support
months, years and timezones.
Current implementation doesn't claim to be complete. For instance,
it doesn't support timezones yet. The reasons to commit it in it's
current state are 1) to shorten the feedback loop 2) to start
experimenting with monthly buckets are soon as possible,
3) to reduce the unnecessary work of rebasing and resolving
conflicts 4) to make the work easier to the reviewers
The post-update script was handling preserving initprivs for newly
added catalog tables and views. However, newly added catalog sequences
need separate handling otherwise update tests start failing. We also
now grant privileges for all future sequences in the update tests.
In passing, default the PG_VERSION in the update tests to 12 since we
don't work with PG11 anymore.
Add a workflow to check that CMake files are correctly formatted as
well as a custom target to format CMake files in the repository. This
commit also runs the formatting on all CMake files in the repository.
Since we are only interested in entries with classoid pg_class
our queries should reflect that. Without these restrictions
objects that have entries for multiple classoids can cause the
extension update to fail.
When executing "ALTER EXTENSION timescaledb UPDATE TO ..." it will fail
if parallel workers spawn for the update itself. Disable parallel
execution during the update.
pg_init_privs can have multiple entries per relation if the relation
has per column privileges. An objsubid different from 0 means that
the entry is for a column privilege. Since we do not currently
restore column privileges we have to ignore those rows otherwise
the update script will fail when tables with column privileges are
present.
When querying continuous aggregate views with a search_path not
including public the query will fail cause the function reference
in the finalize call is not fully qualified.
This can surface when querying caggs through postgres_fdw which
resets search_path to contain only pg_catalog.
Fixes#1919Fixes#3326
This release adds major new features since the 2.2.1 release. We deem
it moderate priority for upgrading.
This release adds support for inserting data into compressed chunks
and improves performance when inserting data into distributed
hypertables. Distributed hypertables now also support triggers and
compression policies.
The bug fixes in this release address issues related to the handling
of privileges on compressed hypertables, locking, and triggers with
transition tables.
**Features**
* #3116 Add distributed hypertable compression policies
* #3162 Use COPY when executing distributed INSERTs
* #3199 Add GENERATED column support on distributed hypertables
* #3210 Add trigger support on distributed hypertables
* #3230 Support for inserts into compressed chunks
**Bugfixes**
* #3213 Propagate grants to compressed hypertables
* #3229 Use correct lock mode when updating chunk
* #3243 Fix assertion failure in decompress_chunk_plan_create
* #3250 Fix constraint triggers on hypertables
* #3251 Fix segmentation fault due to incorrect call to chunk_scan_internal
* #3252 Fix blocking triggers with transition tables
**Thanks**
* @yyjdelete for reporting a crash with decompress_chunk and identifying the bug in the code
* @fabriziomello for documenting the prerequisites when compiling against PostgreSQL 13
After inserts go into a compressed chunk, the chunk is marked as
unordered.This PR adds a new function recompress_chunk that
compresses the data and sets the status back to compressed. Further
optimizations for this function are planned but not part of this PR.
This function can be invoked by calling
SELECT recompress_chunk(<chunk_name>).
recompress_chunk function is automatically invoked by the compression
policy job, when it sees that a chunk is in unordered state.
The view uses cached information from compression_chunk_size to
report the size of compressed chunks. Since compressed chunks
can be modified, we call pg_relation_size on the compressed chunk
while reporting the size
The view also incorrectly used the hypertable's reltoastrelid to
calculate toast bytes. It has been changed to use the chunk's
reltoastrelid.
When creating a constraint trigger on a hypertable the command
succeeds and the constraint trigger works correctly for existing
chunks but any chunk creation after that would fail with an
error, because the constraint trigger was treated like a normal
constraint. But since pg_get_constraintdef does not return a SQL
command for constraint triggers the corresponding command that was
created would throw an error.
Fixes#3235
The timescaledb_information.chunks view used to return NULL in the
is_compressed field for distributed chunks. This changes the view to
always return true or false, depending on the chunk status flags in the
Access Node. This is a hint and cannot be used as a source of truth for
the compression state of the actual chunks in the Data Nodes.
Grants and revokes where not propagated to compressed hypertables, if
the hypertable had a compressed hypertable, meaning that `pg_dump` and
`pg_restore` would not be able to dump nor restore the compressed part
of a hypertable unless they were owners and/or was a superuser.
This commit fixes this by propagating grants and revokes on a
hypertable to the associated compressed hypertable, if one exists,
which will then include the compressed hypertable and the associated
chunks in the grant and revoke execution.
It also adds code to fix the permissions of compressed hypertables and
all associated chunks in an update and adds an update test to check
that the permissions match.
Fixes#3209
