This release contains bug fixes since the 2.12.0 release.
We recommend that you upgrade at the next available opportunity.
**Bugfixes**
* #6113 Fix planner distributed table count
* #6117 Avoid decompressing batches using an empty slot
* #6123 Fix concurrency errors in OSM API
* #6142 Do not throw an error when deprecation GUC cannot be read
**Thanks**
* @symbx for reporting a crash when selecting from empty hypertables
So far, the parallel query plans of the agg_partials_pushdown test were
not deterministic since multiple workers were used and the parallel
session leader also participated in query processing. This makes it
impossible to predict which worker would process how many tuples. With
this patch, the number of parallel workers for the agg_partials_pushdown
test has been reduced to 1 and the parallel_leader_participation has
been disabled to ensure deterministic query plans.
This moves EXPLAIN utility functions into apache to make it available
to all custom nodes. Since these are copies of static postgres functions
there is no reason for them to be TSL licensed.
When the continuous_agg_invalidation trigger is called with an invalid
hypertable id (e.g., caused by an inconsistent backup restore), the
hypertable lookup returns NULL. This case was not handled properly so
far. This patch adds proper checks to the function.
Reset the flag to control PlaceHolderInfo creation because we're copying
the entire content of the (PlannerInfo *)root data structure when
building the first/last path.
https://github.com/postgres/postgres/commit/b3ff6c74
The trigger `continuous_agg_invalidation_trigger` receive the hypertable
id as parameter as the following example:
Triggers:
ts_cagg_invalidation_trigger AFTER INSERT OR DELETE OR UPDATE ON
_timescaledb_internal._hyper_3_59_chunk
FOR EACH ROW EXECUTE FUNCTION
_timescaledb_functions.continuous_agg_invalidation_trigger('3')
The problem is in the compatibility layer using PL/pgSQL code there's no
way to passdown the parameter from the wrapper trigger function created
to the underlying trigger function in another schema.
To solve this we simple create a new function in the deprecated
`_timescaledb_internal` schema pointing to the function trigger and
inside the C code we emit the WARNING message if the function is called
from the deprecated schema.
When continuous_agg_invalidation_trigger is called without any argument,
fcinfo->context in continuous_agg_trigfn is NULL and it should not be
dereferenced.
The multinode tests in regresscheck-shared were already disabled
by default and removing them allows us to skip setting up the
multinode environment in regresscheck-shared. This database
is also used for sqlsmith which will make sqlsmith runs more
targetted. Additionally this will be a step towards running
regresscheck-shared unmodified against our cloud.
The initial commit to adjust RELOPT_DEADREL for PG16 also freed
the RelOptInfo but even though we dont want them to be referenced
again since they are internal we must not free them because the
lower plan nodes still reference them leading to segfaults in the
executor when trying to access those reloptinfo. With this change
most of the compression tests pass on PG16.
This patch also gets rid of RELOPT_DEADREL for earlier PG versions
and always removes the entry from simple_rel_array.
This commit fixes two issues with the osm range update API:
1. Deadlock between two concurrent range updates.
Two transactions concurrently attempting to update the range
of the OSM chunk by calling hypertable_osm_range_update would
deadlock because the dimension slice tuple was first locked with
a FOR KEY SHARE lock, then a FOR UPDATE lock was requested before
proceeding with the dimension slice tuple udpate.
This commit fixes the deadlock by taking FOR UPDATE lock on the
tuple from the start, before proceeding to update it.
2. Tuple concurrently updated error for hypertable tuple.
When one session tries to update the range of the OSM chunk and another
enables compression on the hypertable, the update failed with tuple
concurrently updated error. This commit fixes this by first locking the
hypertable tuple with a FOR UPDATE lock before proceeding to UPDATE it.
Isolation tests for OSM range API are also added.
- Added creation_time attribute to timescaledb catalog table "chunk".
Also, updated corresponding view timescaledb_information.chunks to
include chunk_creation_time attribute.
- A newly created chunk is assigned the creation time during chunk
creation to handle new partition range for give dimension (Time/
SERIAL/BIGSERIAL/INT/...).
- In case of an already existing chunk, the creation time is updated as
part of running upgrade script. The current timestamp (now()) at the
time of upgrade has been assigned as chunk creation time.
- Similarly, downgrade script is updated to drop the attribute
creation_time from catalog table "chunk".
- All relevant queries/views/test output have been updated accordingly.
Co-authored-by: Nikhil Sontakke <nikhil@timescale.com>
When connected to TimescaleDB with the timescaledb.disable_load=on GUC
setting, calling these functions causes an error to be thrown.
By specifying missing_ok=true, we prevent this situation from causing an
error for the user.
There are certain modifications/commands that should not be allowed
in our update/downgrade scripts. For example, when adding or dropping
columns to timescaledb catalog tables, the right way to do this is to
drop and recreate the table with the desired definition instead of doing
ALTER TABLE ... ADD/DROP COLUMN. This is required to ensure consistent
attribute numbers across versions.
This workflow detects this and some other incorrect catalog table
modifications and fails with an error in that case.
Fixes#6049
Users can set a NULL value as column name and cause a segfault
which would crash the instance. This change checks for NULL
values and errors out with appropriate message.
When dropping a table, we get notices if that causes to cancel
background jobs and thus fail tests. By stopping the workers
before dropping the table, we should reduce this flakiness.
Isolation test `cagg_insert` is flaky because refresh steps can
complete in any order. Adding constraints so that completion is
reported in the same order in all test runs.
Fixes#5331
The current hypertable creation interface is heavily focused on a time
column, but since hypertables are focused on partitioning of not only
time columns, we introduce a more generic API that support different
types of keys for partitioning.
The new interface introduced new versions of create_hypertable,
add_dimension, and a replacement function `set_partitioning_interval`
that replaces `set_chunk_time_interval`. The new functions accept an
instance of dimension_info that can be constructed using constructor
functions `by_range` and `by_hash`, allowing a more versatile and
future-proof API.
For examples:
SELECT create_hypertable('conditions', by_range('time'));
SELECT add_dimension('conditions', by_hash('device'));
The old API remains, but will eventually be deprecated.
We now build APT packages for Debian 10,11,12 and Ubuntu packages
for 20.04 and 22.04 for both amd64 and arm64.
We also build Ubuntu packages for 23.04 for amd64 but not for arm64
because pgdg packages for postgres is not available for arm64 on
Ubuntu 23.04.
Update the location and name of .perltidyrc in the README
After the recent change (244b3e637c)
the location and name of perltitdyrc has changed.
Signed-off-by: noctarius aka Christoph Engelbert <me@noctarius.com>
When running COPY command into a compressed hypertable, we
could end up using an empty slot for filtering compressed batches.
This happens when a previously created copy buffer for a chunk
does not contain any new tuples for inserting. The fix is to
verify slots before attempting to do anything else.
Historically creating a Continuous Aggregate make it realtime by default
but it confuse users specially when using `WITH NO DATA` option. Also is
well known that realtime Continuous Aggregates can potentially lead to
issues with Hierarchical and Data Tiering.
Improved the UX by making Continuous Aggregates non-realtime by default.
PG13 introduced an option to DROP DATABASE statement to terminate all
existing connections to the target database. Now that our minor
supported version is PG13 make sense to use it on regression tests in
order to avoid potential flaky tests.
The check for a distributed hypertable was done after ht had been
changed to the compressed hypertable potentially leading to miscount
or even segfault when the cache lookup for the compressed hypertable
returned NULL.
The bgw_launcher test changes the tablespace for a database. However,
this requires that the database is used and no BGW are accessing the
database. This PR ensures that the background workers are stopped before
the tablespace is changed.
The CONTRIBUTING.md file states that our CI does not run regression
checks on Windows, which is outdated information. In fact, we do execute
regression checks on Windows in our current CI. This pull request
removes the outdated section.
One of the test cases changes chunk status in order to fail
the background job in order to ensure jobs continue to be
scheduled even after such a scenario. The problem lies in
the race condition between compression policy and the status
update which both update the catalog and that causes random
test failures. The solution is to add the compression policy
after updating the status, eliminating the possibility of
race condition.
In 683e2bcf189f478ef51d2bc9b409eabf483df18a the default
schedule_interval of the compression policy was changed. This PR adjusts
the test output in the optional multi-node tests.
The test compression_bgw cannot be executed multiple times because a
role was created and not properly deleted. This PR adds the needed
teardown step to the test.
Calling ts_scan_iterator_next will advance internal scanner data
structures. It is therefore not safe to use the iterator after
ts_scan_iterator_next returned NULL and expect to be able to access
the previous tuple. Under certain circumstances this may point the
iterator to tuples not normally visible.
By default, the compression policy is scheduled for every
chunk_time_interval / 2 in the current implementation, equal to three
days and twelve hours with our default settings. This schedule interval
was sufficient for previous versions of TimescaleDB. However, with the
introduction of features like mutable compression and ON CONFLICT .. DO
UPDATE queries, regular DML operations decompress data. To ensure that
modified data is compressed earlier, this patch reduces the schedule
interval of the compression policy to run at least every 12 hours.
Our nightly windows tests have not been running for several months,
causing the status badge in our readme to remain in a failing state
without ever turning green. This pull request enables the daily
scheduled windows builds once again.
This patch stops the background worker in the bgw_launcher test teardown
before the database is dropped. The current version of the test is flaky
because sometimes the database cannot be dropped due to BGW activity.
The regression test bgw_db_scheduler tests if the BGW scheduler starts a
background job after it is created. However, this only happens if the
previous instance of the BGW scheduler is no longer active. If the old
BGW scheduler instance is still running, it will pick up the job and
prevent it from starting in the current test.
The current version of the BGW scheduler mock does not support multiple
background workers. If a second worker was registered, the reference to
the previously registered worker gets lost. So, the implementation of
the mock scheduler only waits for the latest registered worker before
updating the mocked current time, which causes a race condition. If the
previous workers are still active and logging messages after this point,
the messages are logged with the updated mocked time. This leads to a
non-deterministic behavior and test failures. For example:
msg_no | mock_time | application_name | msg
---------+-----------+-------------------------+----------------------
- 1 | 0 | Retention Policy [1002] | job 1002 [...]
[...]
+ 1 | 1000000 | Retention Policy [1002] | job 1002 [...]
With the current version of our docs, the first and third URL in our
extension installation message pointed to the same page. This PR removes
one of the duplicates.
This release contains performance improvements for compressed hypertables
and continuous aggregates and bug fixes since the 2.11.2 release.
We recommend that you upgrade at the next available opportunity.
This release moves all internal functions from the _timescaleb_internal
schema into the _timescaledb_functions schema. This separates code from
internal data objects and improves security by allowing more restrictive
permissions for the code schema. If you are calling any of those internal
functions you should adjust your code as soon as possible. This version
also includes a compatibility layer that allows calling them in the old
location but that layer will be removed in 2.14.0.
**PostgreSQL 12 support removal announcement**
Following the deprecation announcement for PostgreSQL 12 in TimescaleDB 2.10,
PostgreSQL 12 is not supported starting with TimescaleDB 2.12.
Currently supported PostgreSQL major versions are 13, 14 and 15.
PostgreSQL 16 support will be added with a following TimescaleDB release.
**Features**
* #5137 Insert into index during chunk compression
* #5150 MERGE support on hypertables
* #5515 Make hypertables support replica identity
* #5586 Index scan support during UPDATE/DELETE on compressed hypertables
* #5596 Support for partial aggregations at chunk level
* #5599 Enable ChunkAppend for partially compressed chunks
* #5655 Improve the number of parallel workers for decompression
* #5758 Enable altering job schedule type through `alter_job`
* #5805 Make logrepl markers for (partial) decompressions
* #5809 Relax invalidation threshold table-level lock to row-level when refreshing a Continuous Aggregate
* #5839 Support CAgg names in chunk_detailed_size
* #5852 Make set_chunk_time_interval CAggs aware
* #5868 Allow ALTER TABLE ... REPLICA IDENTITY (FULL|INDEX) on materialized hypertables (continuous aggregates)
* #5875 Add job exit status and runtime to log
* #5909 CREATE INDEX ONLY ON hypertable creates index on chunks
**Bugfixes**
* #5860 Fix interval calculation for hierarchical CAggs
* #5894 Check unique indexes when enabling compression
* #5951 _timescaledb_internal.create_compressed_chunk doesn't account for existing uncompressed rows
* #5988 Move functions to _timescaledb_functions schema
* #5788 Chunk_create must add an existing table or fail
* #5872 Fix duplicates on partially compressed chunk reads
* #5918 Fix crash in COPY from program returning error
* #5990 Place data in first/last function in correct mctx
* #5991 Call eq_func correctly in time_bucket_gapfill
* #6015 Correct row count in EXPLAIN ANALYZE INSERT .. ON CONFLICT output
* #6035 Fix server crash on UPDATE of compressed chunk
* #6044 Fix server crash when using duplicate segmentby column
* #6045 Fix segfault in set_integer_now_func
* #6053 Fix approximate_row_count for CAggs
* #6081 Improve compressed DML datatype handling
* #6084 Propagate parameter changes to decompress child nodes
**Thanks**
* @ajcanterbury for reporting a problem with lateral joins on compressed chunks
* @alexanderlaw for reporting multiple server crashes
* @lukaskirner for reporting a bug with monthly continuous aggregates
* @mrksngl for reporting a bug with unusual user names
* @willsbit for reporting a crash in time_bucket_gapfill
The paths filter in github workflows will trigger when at least
one of the pathes match unless everything else has been explicitly
excluded. For the ignored workflows we want it to only trigger
when only those files explicitly specified are changed and nothing
else.
Introduce version number for OsmCallbacks struct
Add a callback for cascading drop chunks to OSM and
integrate with drop_chunks
Add backward compatibility for OsmCallbacks
