In certain scenarios, when generating buckets with
monthly buckets and different timezones, gapfill
would create timestamps which don't align with
time_bucket and thus potentially generating multiple
rows for an individual month. Instead of relying on
previous timestamp to generate the next one, now
we generate them always from the start point
which will make us align with time_bucket buckets.
EXPLAIN ANALYZE for compressed DML would error out with `bogus varno`
error because we would modify the original expressions of the plan
that were still referenced in nodes instead of adjusting copies and
using those copies in our internal scans.
When defining a GUC Postgres do a cross-check between the initial value
of the C declaration associated to a GUC and its actual boot value in
assert-enabled builds.
Previous PR #6206 didn't fixed it entirely and this happen just on PG16.
postgres/postgres@a73952b7
In #6188 we introduced a new GUC `timescaledb.bgw_log_level` to control
background workers log level, but if we missed to set the default value
into the global variable leading to an assertion when checking for
default values when defining it in the `_guc_init` process.
Fix it by properly initialize the global variable to the
`WARNING` that is the default for `log_min_messages`.
To avoid having to spam the log with unecessary background worker
progress messages a GUC for controlling the background worker log level
is introduced. This can be set using `ALTER SYSTEM` plus a call of
`pg_reload_conf()` and will set the `log_min_messages` for the
background workers only (scheduler and all job runners).
It also adds some additional job progress messages at debug level.
Fixes#6161
When a background worker is canceled by 'select delete_job()' the
following log message is printed:
NOTICE: cancelling the background worker for job
However, this log message is not deterministic and breaks our CI checks.
So, this patch removes the log message from the test output to make the
regression checks more deterministic.
This patch stores the current catalog version in an internal
table to allow us to verify catalog and code version match.
When doing dump/restore people occasionally report very unusual
errors and during investigation it is discovered that they loaded
a dump from an older version and run it with a later code version.
This allows to detect mismatches between installed code version
and loaded dump version. The version number in the metadata table
will be kept updated in upgrade and downgrade scripts.
This patch adds tracking number of batches and tuples that needed
to be decompressed as part of DML operations on compressed hypertables.
These will be visible in EXPLAIN ANALYZE output like so:
QUERY PLAN
Custom Scan (HypertableModify) (actual rows=0 loops=1)
Batches decompressed: 2
Tuples decompressed: 25
-> Insert on decompress_tracking (actual rows=0 loops=1)
-> Custom Scan (ChunkDispatch) (actual rows=2 loops=1)
-> Values Scan on "*VALUES*" (actual rows=2 loops=1)
(6 rows)
When the regression tests output differs between PG versions we create a
template test file (*.sql.in) and add several different output files for
each supported version.
Over time we support new PG version and deprecate older ones, for
example nowadays we're working on PG16 support and removed PG12 so we
need to check if we still need template files.
The jobs framework contains two functions for loading jobs
(ts_bgw_job_get_scheduled and bgw_job_from_tupleinfo). However, the
ts_bgw_job_get_scheduled function did not properly populate the
fixed_schedule field, which caused some bugs in the calculation of the
next execution time for a job. This PR ensures the fixed_schedule field
is populated properly.
This release contains bug fixes since the 2.12.0 release.
We recommend that you upgrade at the next available opportunity.
**Bugfixes**
* #6113 Fix planner distributed table count
* #6117 Avoid decompressing batches using an empty slot
* #6123 Fix concurrency errors in OSM API
* #6142 Do not throw an error when deprecation GUC cannot be read
**Thanks**
* @symbx for reporting a crash when selecting from empty hypertables
So far, the parallel query plans of the agg_partials_pushdown test were
not deterministic since multiple workers were used and the parallel
session leader also participated in query processing. This makes it
impossible to predict which worker would process how many tuples. With
this patch, the number of parallel workers for the agg_partials_pushdown
test has been reduced to 1 and the parallel_leader_participation has
been disabled to ensure deterministic query plans.
This moves EXPLAIN utility functions into apache to make it available
to all custom nodes. Since these are copies of static postgres functions
there is no reason for them to be TSL licensed.
When the continuous_agg_invalidation trigger is called with an invalid
hypertable id (e.g., caused by an inconsistent backup restore), the
hypertable lookup returns NULL. This case was not handled properly so
far. This patch adds proper checks to the function.
Reset the flag to control PlaceHolderInfo creation because we're copying
the entire content of the (PlannerInfo *)root data structure when
building the first/last path.
https://github.com/postgres/postgres/commit/b3ff6c74
The trigger `continuous_agg_invalidation_trigger` receive the hypertable
id as parameter as the following example:
Triggers:
ts_cagg_invalidation_trigger AFTER INSERT OR DELETE OR UPDATE ON
_timescaledb_internal._hyper_3_59_chunk
FOR EACH ROW EXECUTE FUNCTION
_timescaledb_functions.continuous_agg_invalidation_trigger('3')
The problem is in the compatibility layer using PL/pgSQL code there's no
way to passdown the parameter from the wrapper trigger function created
to the underlying trigger function in another schema.
To solve this we simple create a new function in the deprecated
`_timescaledb_internal` schema pointing to the function trigger and
inside the C code we emit the WARNING message if the function is called
from the deprecated schema.
When continuous_agg_invalidation_trigger is called without any argument,
fcinfo->context in continuous_agg_trigfn is NULL and it should not be
dereferenced.
The multinode tests in regresscheck-shared were already disabled
by default and removing them allows us to skip setting up the
multinode environment in regresscheck-shared. This database
is also used for sqlsmith which will make sqlsmith runs more
targetted. Additionally this will be a step towards running
regresscheck-shared unmodified against our cloud.
The initial commit to adjust RELOPT_DEADREL for PG16 also freed
the RelOptInfo but even though we dont want them to be referenced
again since they are internal we must not free them because the
lower plan nodes still reference them leading to segfaults in the
executor when trying to access those reloptinfo. With this change
most of the compression tests pass on PG16.
This patch also gets rid of RELOPT_DEADREL for earlier PG versions
and always removes the entry from simple_rel_array.
This commit fixes two issues with the osm range update API:
1. Deadlock between two concurrent range updates.
Two transactions concurrently attempting to update the range
of the OSM chunk by calling hypertable_osm_range_update would
deadlock because the dimension slice tuple was first locked with
a FOR KEY SHARE lock, then a FOR UPDATE lock was requested before
proceeding with the dimension slice tuple udpate.
This commit fixes the deadlock by taking FOR UPDATE lock on the
tuple from the start, before proceeding to update it.
2. Tuple concurrently updated error for hypertable tuple.
When one session tries to update the range of the OSM chunk and another
enables compression on the hypertable, the update failed with tuple
concurrently updated error. This commit fixes this by first locking the
hypertable tuple with a FOR UPDATE lock before proceeding to UPDATE it.
Isolation tests for OSM range API are also added.
- Added creation_time attribute to timescaledb catalog table "chunk".
Also, updated corresponding view timescaledb_information.chunks to
include chunk_creation_time attribute.
- A newly created chunk is assigned the creation time during chunk
creation to handle new partition range for give dimension (Time/
SERIAL/BIGSERIAL/INT/...).
- In case of an already existing chunk, the creation time is updated as
part of running upgrade script. The current timestamp (now()) at the
time of upgrade has been assigned as chunk creation time.
- Similarly, downgrade script is updated to drop the attribute
creation_time from catalog table "chunk".
- All relevant queries/views/test output have been updated accordingly.
Co-authored-by: Nikhil Sontakke <nikhil@timescale.com>
When connected to TimescaleDB with the timescaledb.disable_load=on GUC
setting, calling these functions causes an error to be thrown.
By specifying missing_ok=true, we prevent this situation from causing an
error for the user.
There are certain modifications/commands that should not be allowed
in our update/downgrade scripts. For example, when adding or dropping
columns to timescaledb catalog tables, the right way to do this is to
drop and recreate the table with the desired definition instead of doing
ALTER TABLE ... ADD/DROP COLUMN. This is required to ensure consistent
attribute numbers across versions.
This workflow detects this and some other incorrect catalog table
modifications and fails with an error in that case.
Fixes#6049
Users can set a NULL value as column name and cause a segfault
which would crash the instance. This change checks for NULL
values and errors out with appropriate message.
When dropping a table, we get notices if that causes to cancel
background jobs and thus fail tests. By stopping the workers
before dropping the table, we should reduce this flakiness.
Isolation test `cagg_insert` is flaky because refresh steps can
complete in any order. Adding constraints so that completion is
reported in the same order in all test runs.
Fixes#5331
The current hypertable creation interface is heavily focused on a time
column, but since hypertables are focused on partitioning of not only
time columns, we introduce a more generic API that support different
types of keys for partitioning.
The new interface introduced new versions of create_hypertable,
add_dimension, and a replacement function `set_partitioning_interval`
that replaces `set_chunk_time_interval`. The new functions accept an
instance of dimension_info that can be constructed using constructor
functions `by_range` and `by_hash`, allowing a more versatile and
future-proof API.
For examples:
SELECT create_hypertable('conditions', by_range('time'));
SELECT add_dimension('conditions', by_hash('device'));
The old API remains, but will eventually be deprecated.
We now build APT packages for Debian 10,11,12 and Ubuntu packages
for 20.04 and 22.04 for both amd64 and arm64.
We also build Ubuntu packages for 23.04 for amd64 but not for arm64
because pgdg packages for postgres is not available for arm64 on
Ubuntu 23.04.
Update the location and name of .perltidyrc in the README
After the recent change (244b3e637c)
the location and name of perltitdyrc has changed.
Signed-off-by: noctarius aka Christoph Engelbert <me@noctarius.com>
When running COPY command into a compressed hypertable, we
could end up using an empty slot for filtering compressed batches.
This happens when a previously created copy buffer for a chunk
does not contain any new tuples for inserting. The fix is to
verify slots before attempting to do anything else.
Historically creating a Continuous Aggregate make it realtime by default
but it confuse users specially when using `WITH NO DATA` option. Also is
well known that realtime Continuous Aggregates can potentially lead to
issues with Hierarchical and Data Tiering.
Improved the UX by making Continuous Aggregates non-realtime by default.
PG13 introduced an option to DROP DATABASE statement to terminate all
existing connections to the target database. Now that our minor
supported version is PG13 make sense to use it on regression tests in
order to avoid potential flaky tests.
The check for a distributed hypertable was done after ht had been
changed to the compressed hypertable potentially leading to miscount
or even segfault when the cache lookup for the compressed hypertable
returned NULL.
The bgw_launcher test changes the tablespace for a database. However,
this requires that the database is used and no BGW are accessing the
database. This PR ensures that the background workers are stopped before
the tablespace is changed.
The CONTRIBUTING.md file states that our CI does not run regression
checks on Windows, which is outdated information. In fact, we do execute
regression checks on Windows in our current CI. This pull request
removes the outdated section.
One of the test cases changes chunk status in order to fail
the background job in order to ensure jobs continue to be
scheduled even after such a scenario. The problem lies in
the race condition between compression policy and the status
update which both update the catalog and that causes random
test failures. The solution is to add the compression policy
after updating the status, eliminating the possibility of
race condition.
In 683e2bcf189f478ef51d2bc9b409eabf483df18a the default
schedule_interval of the compression policy was changed. This PR adjusts
the test output in the optional multi-node tests.
