Add 2.9.0 to update test scripts and adjust downgrade scripts for
2.9.0. Additionally adjust CHANGELOG to sync with the actual release
CHANGELOG and add PG15 to CI.
Issue occurs in extended query protocol mode only where every
query goes through PREPARE and EXECUTE phase. First time ANALYZE
is executed, a list of relations to be vaccumed is extracted and
saved in a list. This list is referenced in parsetree node. Once
execution of ANALYZE is complete, this list is cleaned up, however
reference to the same is not cleaned up in parsetree node. When
second time ANALYZE is executed, segfault happens as we access an
invalid memory location.
Fixed the issue by restoring the actual value in parsetree node
once ANALYZE completes its execution.
Fixes#4857
The code we inherited from postgres expects that if we have a const null
or false clause, it's going to be the single one, but that's not true
for runtime chunk exclusion because we don't try to fold such
restrictinfos after evaluating the mutable functions. Fix it to also
work for multiple restrictinfos.
On PG15 CustomScan by default is not projection capable, thus wraps this
node in Result node. THis change in PG15 causes tests result files which
have EXPLAIN output to fail. This patch fixes the plan outputs.
Fixes#4833
We have a rare condition where a debug build asserts on more than one
job with the same job id. Since it is hard to create a reproduction,
this commit adds a printout for those conditions and print out all the
jobs with that job id in the postgres log.
Part of #4863
Version 15 pg_dump program does not log any messages with log level <
PG_LOG_WARNING to stdout. Whereas this check is not present in version
14, thus we see corresponding tests fail with missing log information.
This patch fixes by supressing those log information, so that the tests
pass on all versions of postgresql.
Fixes#4832
The changes from e555eea lead to flakiness. They are a leftover of
earlier version and probably not needed anymore.
The original version is also still flaky on Windows, so use linear
regression to tell if the memory usage is increasing.
Verified to still fail on 2.7.x
In 8375b9aa536a619a5ac2644e0dae3c25880a4ead, a patch was added to handle
chunks closes during an ongoing copy operation. However, this patch
introduces a performance regression. All MultiInsertBuffers are deleted
after they are flushed. In this PR, the performance regression is fixed.
The most commonly used MultiInsertBuffers survive flushing.
The 51259b31c4c62b87228b059af0bbf28caa143eb3 commit changes the way the
per-tuple context is used. Since this commit, more objects are stored in
this context. The size of the context was used to set the tuple size to
PG < 14. The extra objects in the context lead to wrong (very large)
results and flushes almost after every tuple read.
The cache synchronization introduced in
296601b1d7aba7f23aea3d47c617e2d6df81de3e is reverted. With the current
implementation, `MAX_PARTITION_BUFFERS` survive the flash. If
`timescaledb.max_open_chunks_per_insert` is lower than
`MAX_PARTITION_BUFFERS` , a buffer flush would be performed after each
tuple read.
Currently, the next start of a scheduled background job is
calculated by adding the `schedule_interval` to its finish
time. This does not allow scheduling jobs to execute at fixed
times, as the next execution is "shifted" by the job duration.
This commit introduces the option to execute a job on a fixed
schedule instead. Users are expected to provide an initial_start
parameter on which subsequent job executions are aligned. The next
start is calculated by computing the next time_bucket of the finish
time with initial_start origin.
An `initial_start` parameter is added to the compression, retention,
reorder and continuous aggregate `add_policy` signatures. By passing
that upon policy creation users indicate the policy will execute on
a fixed schedule, or drifting schedule if `initial_start` is not
provided.
To allow users to pick a drifting schedule when registering a UDA,
an additional parameter `fixed_schedule` is added to `add_job`
to allow users to specify the old behavior by setting it to false.
Additionally, an optional TEXT parameter, `timezone`, is added to both
add_job and add_policy signatures, to address the 1-hour shift in
execution time caused by DST switches. As internally the next start of
a fixed schedule job is calculated using time_bucket, the timezone
parameter allows using timezone-aware buckets to calculate
the next start.
Recently we fixed a DDL error (#4739) after upgrading to 2.8.0 version
that surprisly the CI upgrade/dowgrade tests didn't complained during
the development of the feature (#4552).
Fixed it by adding an specific query in the `post.catalog.sql` script to
make sure we'll check all the constraints of our internal tables and
catalog.
Consider a hypertable which has a foreign key constraint on a
referenced table, which is a parititioned table. In such case, foreign
key constraint is duplicated for each of the partitioned table. When
we insert into a hypertable, we end up checking the foreign key
constraint multiple times, which obviously leads to foreign key
constraint violation. Instead we only check foreign key constraint of
the parent table of the partitioned table.
Fixes#4684
Use DROP DATABASE WITH(FORCE) to drop the database in pg_dump test
since occasionally there would still be connections to the database
leading to test failures. Unfortunately PG12 does not support that
syntax so we have to drop without that option on PG12.
The schema of base table on which hypertables are created, should define
columns with proper data types. As per postgres best practices Wiki
(https://wiki.postgresql.org/wiki/Don't_Do_This), one should not define
columns with CHAR, VARCHAR, VARCHAR(N), instead use TEXT data type.
Similarly instead of using timestamp, one should use timestamptz.
This patch reports a WARNING to end user when creating hypertables,
if underlying parent table, has columns of above mentioned data types.
Fixes#4335
In some cases, entire hypertables can be excluded
at runtime. Some Examples:
WHERE col @> ANY(subselect)
if the subselect returns empty set
WHERE col op (subselect)
if the op is a strict operator and
the subselect returns empty set.
When qual clauses are not on partition columns, we use
the old chunk exclusion, otherwise we try hypertable exclusion.
Hypertable exclusion is executed once per hypertable.
This is cheaper than the chunk exclusion
that is once-per-chunk.
Do not allocate various temporary data in PortalContext, such as the
hyperspace point corresponding to the row, or the intermediate data
required for chunk lookup.
This patch allows bucketing by month for time_bucket with date,
timestamp or timestamptz. When bucketing by month the interval
must only contain month components. When using origin together
with bucketing by month only the year and month components are
honoured.
To bucket by month we get the year and month of a date and convert
that to the nth month since origin. This allows us to treat month
bucketing similar to int bucketing. During this process we ignore
the day component and therefore only support bucketing by full months.
Make truncating a uncompressed chunk drop the data for the case where
they reside in a corresponding compressed chunk.
Generate invalidations for Continuous Aggregates after TRUNCATE, so
as to have consistent refresh operations on the materialization
hypertable.
Fixes#4362
When executing `get_partition_{hash|for_key}` inside an IMMUTABLE
function we're getting the following error:
`ERROR: unsupported expression argument node type 112`
This error is because the underlying `resolve_function_argtype` was not
dealing with `T_Param` node type.
Fixed it by dealing properly with `T_Param` node type returning the
`paramtype` for the argument type.
Fixes#4575
When working on a fix for #4555 discovered that executing
`{GRANT|REVOKE} .. ON ALL TABLES IN SCHEMA` in an empty schema lead to
an assertion because we change the way that command is executed by
collecting all objects involved and processing one by one.
Fixed it by executing the previous process utility hook just when the
list of target objects is not empty.
Fixes#4581
Add a new metadata table `dimension_partition` which explicitly and
statefully details how a space dimension is split into partitions, and
(in the case of multi-node) which data nodes are responsible for
storing chunks in each partition. Previously, partition and data nodes
were assigned dynamically based on the current state when creating a
chunk.
This is the first in a series of changes that will add more advanced
functionality over time. For now, the metadata table simply writes out
what was previously computed dynamically in code. Future code changes
will alter the behavior to do smarter updates to the partitions when,
e.g., adding and removing data nodes.
The idea of the `dimension_partition` table is to minimize changes in
the partition to data node mappings across various events, such as
changes in the number of data nodes, number of partitions, or the
replication factor, which affect the mappings. For example, increasing
the number of partitions from 3 to 4 currently leads to redefining all
partition ranges and data node mappings to account for the new
partition. Complete repartitioning can be disruptive to multi-node
deployments. With stateful mappings, it is possible to split an
existing partition without affecting the other partitions (similar to
partitioning using consistent hashing).
Note that the dimension partition table expresses the current state of
space partitions; i.e., the space-dimension constraints and data nodes
to be assigned to new chunks. Existing chunks are not affected by
changes in the dimension partition table, although an external job
could rewrite, move, or copy chunks as desired to comply with the
current dimension partition state. As such, the dimension partition
table represents the "desired" space partitioning state.
Part of #4125
This change allows to create new dimensions even with
existing chunks.
It does not modify any existing data or do migration,
instead it creates full-range (-inf/inf) dimension slice for
existing chunks in order to be compatible with newly created
dimension.
All new chunks created after this will follow logic of the new
dimension and its partitioning.
Fix: #2818
In the session timescaledb_post_restore() was called the value for
timescaledb.restoring might not be changed because the reset_val
for the GUC was still on. We have to use explicit SET in this
session to adjust the GUC.
When triggering chunk creation on a hypertable with non-default
statistics targets by a user different from the hypertable owner
the chunk creation will fail with a permission error. This patch
changes the chunk table creation to run the attribute modification
as the table owner.
Fixes#4474
Executing an IMMUTABLE function that has parameters and exception
handling block multiple times in the same transaction causes a null
pointer segfault when try to reset a non-initialized ts_baserel_info.
Fixed it by preventing to reset a non-initialized `ts_baserel_info`.
Fixes#4489
Timescale 2.7 released a new version of Continuous Aggregate (#4269)
that allows users efectivelly create and use indexes in the
materialization hypertable. The boring part of it is that users should
discover what is the associated materialization hypertable to issue a
`CREATE INDEX` statement.
Improved it by allowing users to easily create indexes in the
materialization hypertable by simple executing a `CREATE INDEX` direct
in the Continuous Aggregate.
Example:
`CREATE INDEX name_of_the_index ON continuous_agregate (column);`
The copy test is flaky because some test data is generated
dynamically based on the current date. This patch changes the data
generation to a time series with fixed dates.
This commit contains extends our telemetry system with function call
telemetry. It gathers function call-counts from all queries, and send
back counts for those functions that are built in or from our related
extensions.
A number of TimescaleDB functions internally call `AlterTableInternal`
to modify tables or indexes. For instance, `compress_chunk` and
`attach_tablespace` act as DDL commands to modify
hypertables. However, crashes occur when these functions are called
via `SELECT * INTO FROM <function_name>` or the equivalent `CREATE
TABLE AS` statement. The crashes happen because these statements are
considered process utility commands and therefore sets up an event
trigger context for collecting commands. However, the event trigger
context is not properly set up to record alter table statements in
this code path, thus causing the crashes.
To prevent crashes, wrap `AlterTableInternal` with the event trigger
functions to properly initialize the event trigger context.
This patch solves a crash in the multi-buffer copy optimization,
which was introduced in commit
bbb2f414d2090efd2d8533b464584157860ce49a.
This patch handles closed chunks (e.g., caused by timescaledb.max_open_
chunks_per_insert) properly. The problem is addressed by:
1) Re-reading the ChunkInsertState before the data is stored, which
ensures that the underlying table is open.
2) A TSCopyMultiInsertBuffer is deleted after the data of the buffer
is flushed. So, operations like table_finish_bulk_insert are
executed and the associated chunk can properly be closed.
This patch adds a test for attnum consistency to our update scripts.
When attnum between fresh install and updated install is different
the updated installation will not be able to correctly process affected
catalog tables.
Following work started by #4294 to improve performance of Continuous
Aggregates by removing the re-aggregation in the user view.
This PR get rid of `partialize_agg` and `finalize_agg` aggregate
functions and store the finalized aggregated (plain) data in the
materialization hypertable.
Because we're not storing partials anymore and removed the
re-aggregation, now is be possible to create indexes on aggregated
columns in the materialization hypertable in order to improve the
performance even more.
Also removed restrictions on types of aggregates users can perform
with Continuous Aggregates:
* aggregates with DISTINCT
* aggregates with FILTER
* aggregates with FILTER in HAVING clause
* aggregates without combine function
* ordered-set aggregates
* hypothetical-set aggregates
By default new Continuous Aggregates will be created using this new
format, but the previous version (with partials) will be supported.
Users can create the previous style by setting to `false` the storage
paramater named `timescaledb.finalized` during the creation of the
Continuous Aggregate.
Fixes#4233
