Postgres knows whether a given aggregate is parallel-safe, and creates
parallel aggregation plans based on that. The `partialize_agg` is a
wrapper we use to perform partial aggregation on data nodes. It is a
pure function that produces serialized aggregation state as a result.
Being pure, it doesn't influence parallel safety. This means we don't
need to mark it parallel-unsafe to artificially disable the parallel
plans for partial aggregation. They will be chosen as usual based on
the parallel-safety of the underlying aggregate function.
The copy test is flaky because some test data is generated
dynamically based on the current date. This patch changes the data
generation to a time series with fixed dates.
When dealing with Intervals with month component timezone changes
can result in multiple day differences in the outcome of these
calculations due to different month lengths. When dealing with
months we add a 7 day safety buffer.
For all these calculations it is fine if we exclude less chunks
than strictly required for the operation, additional exclusion
with exact values will happen in the executor. But under no
circumstances must we exclude too much cause there would be
no way for the executor to get those chunks back.
The initial patch to use now() expressions during planner hypertable
expansion only supported intervals with no day or month component.
This patch adds support for intervals with day component.
If the interval has a day component then the calculation needs
to take into account daylight saving time switches and thereby a
day would not always be exactly 24 hours. We mitigate this by
adding a safety buffer to account for these dst switches when
dealing with intervals with day component. These calculations
will be repeated with exact values during execution.
Since dst switches seem to range between -1 and 2 hours we set
the safety buffer to 4 hours.
This patch also refactors the tests since the previous tests
made it hard to tell the feature was working after the constified
values have been removed from the plans.
Currently coccinelle and shellcheck get run an additional time
for every merged commit to master. This patch adjusts the config
so they are only run on pull request or on push to prerelease_test
instead of push to any branch, similar to how all the other
workflows are set up.
This commit contains extends our telemetry system with function call
telemetry. It gathers function call-counts from all queries, and send
back counts for those functions that are built in or from our related
extensions.
Compilers are not smart enough to check that `conn` is initialized
inside the loop so not initializing it gives an error. Added an
initializer to the auto variable to get rid of the error.
Commit 35ea80ff added an optimization to enable expressions with
now() to be used during plan-time chunk exclusion by constifying
the now() expression. The added constified constraints were left
in the plan even though they were only required during the
hypertable explansion. This patch marks those constified constraints
and removes them once they are no longer required.
The multi-buffer copy optimization creates a multi-insert buffer per
ChunkInsertState. However, chunks can be closed. When
ts_chunk_dispatch_get_chunk_insert_state is called for a closed chunk
again, a new ChunkInsertState is returned. So far, also a new
multi-insert buffer has been created. Therefore, multiple batch
operations could be executed per chunk, which reduces the efficiency
of the optimization.
This patch introduces an HTAB that maps from the chunk_id to the
multi-insert buffer. Even when a chunk is closed, all tuples for
a chunk are stored in the same buffer.
This release adds major new features since the 2.6.1 release.
We deem it moderate priority for upgrading.
This release includes these noteworthy features:
* Optimize continuous aggregate query performance and storage
* The following query clauses and functions can now be used in a continuous
aggregate: FILTER, DISTINCT, ORDER BY as well as [Ordered-Set Aggregate](https://www.postgresql.org/docs/current/functions-aggregate.html#FUNCTIONS-ORDEREDSET-TABLE)
and [Hypothetical-Set Aggregate](https://www.postgresql.org/docs/current/functions-aggregate.html#FUNCTIONS-HYPOTHETICAL-TABLE)
* Optimize now() query planning time
* Improve COPY insert performance
* Improve performance of UPDATE/DELETE on PG14 by excluding chunks
This release also includes several bug fixes.
If you are upgrading from a previous version and were using compression
with a non-default collation on a segmentby-column you should recompress
those hypertables.
**Features**
* #4045 Custom origin's support in CAGGs
* #4120 Add logging for retention policy
* #4158 Allow ANALYZE command on a data node directly
* #4169 Add support for chunk exclusion on DELETE to PG14
* #4209 Add support for chunk exclusion on UPDATE to PG14
* #4269 Continuous Aggregates finals form
* #4301 Add support for bulk inserts in COPY operator
* #4311 Support non-superuser move chunk operations
* #4330 Add GUC "bgw_launcher_poll_time"
* #4340 Enable now() usage in plan-time chunk exclusion
**Bugfixes**
* #3899 Fix segfault in Continuous Aggregates
* #4225 Fix TRUNCATE error as non-owner on hypertable
* #4236 Fix potential wrong order of results for compressed hypertable with a non-default collation
* #4249 Fix option "timescaledb.create_group_indexes"
* #4251 Fix INSERT into compressed chunks with dropped columns
* #4255 Fix option "timescaledb.create_group_indexes"
* #4259 Fix logic bug in extension update script
* #4269 Fix bad Continuous Aggregate view definition reported in #4233
* #4289 Support moving compressed chunks between data nodes
* #4300 Fix refresh window cap for cagg refresh policy
* #4315 Fix memory leak in scheduler
* #4323 Remove printouts from signal handlers
* #4342 Fix move chunk cleanup logic
* #4349 Fix crashes in functions using AlterTableInternal
* #4358 Fix crash and other issues in telemetry reporter
**Thanks**
* @abrownsword for reporting a bug in the telemetry reporter and testing the fix
* @jsoref for fixing various misspellings in code, comments and documentation
* @yalon for reporting an error with ALTER TABLE RENAME on distributed hypertables
* @zhuizhuhaomeng for reporting and fixing a memory leak in our scheduler
Postgres changes the internal state format for numeric aggregates
which we materialize in caggs in PG14. This will invalidate the
affected columns when upgrading from PG13 to PG14. This patch
adds a warning to the update script when we encounter this
configuration.
The deadlock_recompress_chunk isolation test uses syntax not
supported by earlier versions of the isolation tester leading
to a parse failure when processing that file on PG13.2.
This patch skips that particular test on PG13.2.
Commit 7b9d8673 added an isolation test for telemetry but did not
add it to the ignore list for earlier postgres versions. PG14 changed
the output format for isolation tests which got backported to PG12
and PG13 which makes the output of earlier PG12 and PG13 different
from the latest one so we ignore isolation tests on those earlier
versions.
With recent refactorings the telemetry test seems to trigger the
same use-after-free bug that got triggered by tablespace test so
we skip that test on PG12.0 as well.
Make the following changes to the telemetry reporter background worker:
- Add a read lock to the current relation that the reporter collects
stats for. This lock protects against concurrent deletion of the
relation, which could lead to errors that would prevent the reporter
from completing its report.
- Set an active snapshot in the telemetry background process for use
when scanning a relation for stats collection.
- Reopen the scan iterator when collecting chunk compression stats for
a relation instead of keeping it open and restarting the scan. The
previous approach seems to cause crashes due to memory corruption of
the scan state. Unfortunately, the exact cause has not been
identified, but the change has been verified to work on a live
running instance (thanks to @abrownsword for the help with
reproducing the crash and testing fixes).
Fixes#4266
Remove noop ternary operator in cagg_rebuild_view_definition. We
return if finalized is true on line 2475 so the NIL would never
be reached in the ternary operator. Found by coverity.
Multinode queries use _timescaledb_internal.chunks_in to specify
the chunks from which to select data. The chunk id in
regresscheck-shared is not stable and may differ depending on
execution order leading to flaky tests.
The table metrics_dist1 was only used by a single test and therefore
should not be part of shared_setup but instead be created in the
test that actually uses it. This reduces executed time of
regresscheck-shared when that test is not run.
A number of TimescaleDB functions internally call `AlterTableInternal`
to modify tables or indexes. For instance, `compress_chunk` and
`attach_tablespace` act as DDL commands to modify
hypertables. However, crashes occur when these functions are called
via `SELECT * INTO FROM <function_name>` or the equivalent `CREATE
TABLE AS` statement. The crashes happen because these statements are
considered process utility commands and therefore sets up an event
trigger context for collecting commands. However, the event trigger
context is not properly set up to record alter table statements in
this code path, thus causing the crashes.
To prevent crashes, wrap `AlterTableInternal` with the event trigger
functions to properly initialize the event trigger context.
Add a new stage "complete" in the "chunk_copy_operation" to
indicate successful move/copy chunk operations. Make the
"cleanup_copy_chunk_operation" procedure more robust and make it only
delete chunk operation entries from the catalog without doing any other
unwanted cleanup if called on successful operations.
This patch solves a crash in the multi-buffer copy optimization,
which was introduced in commit
bbb2f414d2090efd2d8533b464584157860ce49a.
This patch handles closed chunks (e.g., caused by timescaledb.max_open_
chunks_per_insert) properly. The problem is addressed by:
1) Re-reading the ChunkInsertState before the data is stored, which
ensures that the underlying table is open.
2) A TSCopyMultiInsertBuffer is deleted after the data of the buffer
is flushed. So, operations like table_finish_bulk_insert are
executed and the associated chunk can properly be closed.
This patch fixes the constify_now optimization to ignore Vars of
different level. Previously this could potentially lead to an
assertion failure cause the varno of that varno might be bigger
than the number of entries in the rangetable. Found by sqlsmith.
This patch adds a test for attnum consistency to our update scripts.
When attnum between fresh install and updated install is different
the updated installation will not be able to correctly process affected
catalog tables.
While we do filter out chunk ids and hypertable ids from the test
output, the output was still unstable when those ids switch between
single and double digit as that changes the length of the query
decorator in EXPLAIN output. This patch removes this decorator
entirely from all shared test output.
Following work started by #4294 to improve performance of Continuous
Aggregates by removing the re-aggregation in the user view.
This PR get rid of `partialize_agg` and `finalize_agg` aggregate
functions and store the finalized aggregated (plain) data in the
materialization hypertable.
Because we're not storing partials anymore and removed the
re-aggregation, now is be possible to create indexes on aggregated
columns in the materialization hypertable in order to improve the
performance even more.
Also removed restrictions on types of aggregates users can perform
with Continuous Aggregates:
* aggregates with DISTINCT
* aggregates with FILTER
* aggregates with FILTER in HAVING clause
* aggregates without combine function
* ordered-set aggregates
* hypothetical-set aggregates
By default new Continuous Aggregates will be created using this new
format, but the previous version (with partials) will be supported.
Users can create the previous style by setting to `false` the storage
paramater named `timescaledb.finalized` during the creation of the
Continuous Aggregate.
Fixes#4233
The non-superuser needs to have REPLICATION privileges atleast. A
new function "subscription_cmd" has been added to allow running
subscription related commands on datanodes. This function implicitly
upgrades to the bootstrapped superuser and then performs subscription
creation/alteration/deletion commands. It only accepts subscriptions
related commands and errors out otherwise.
Commit 3b35da76 changed the setup script for regresscheck-shared
to no longer be usable directly by the sqlsmith workflow. This
patch set TEST_DBNAME at the top of the script so it is easier
to use the script outside of regression check environment.
Bump PG versions to 12.11, 13.7 and 14.3 in update tests. The update
test was skipped with the previous PG version bump because upstream
docker images were not yet available which we rely on in the update
test.
This implements an optimization to allow now() expression to be
used during plan time chunk exclusions. Since now() is stable it
would not normally be considered for plan time chunk exclusion.
To enable this behaviour we convert `column > now()` expressions
into `column > const AND column > now()`. Assuming that time
always moves forward this is safe even for prepared statements.
This optimization works for SELECT, UPDATE and DELETE.
On hypertables with many chunks this can lead to a considerable
speedup for certain queries.
The following expressions are supported:
- column > now()
- column >= now()
- column > now() - Interval
- column > now() + Interval
- column >= now() - Interval
- column >= now() + Interval
Interval must not have a day or month component as those depend
on timezone settings.
Some microbenchmark to show the improvements, I did best of five
for all of the queries.
-- hypertable with 1k chunks
-- with optimization
select * from metrics1k where time > now() - '5m'::interval;
Time: 3.090 ms
-- without optimization
select * from metrics1k where time > now() - '5m'::interval;
Time: 145.640 ms
-- hypertable with 5k chunks
-- with optimization
select * from metrics5k where time > now() - '5m'::interval;
Time: 4.317 ms
-- without optimization
select * from metrics5k where time > now() - '5m'::interval;
Time: 775.259 ms
-- hypertable with 10k chunks
-- with optimization
select * from metrics10k where time > now() - '5m'::interval;
Time: 4.853 ms
-- without optimization
select * from metrics10k where time > now() - '5m'::interval;
Time: 1766.319 ms (00:01.766)
-- hypertable with 20k chunks
-- with optimization
select * from metrics20k where time > now() - '5m'::interval;
Time: 6.141 ms
-- without optimization
select * from metrics20k where time > now() - '5m'::interval;
Time: 3321.968 ms (00:03.322)
Speedup with 1k chunks: 47x
Speedup with 5k chunks: 179x
Speedup with 10k chunks: 363x
Speedup with 20k chunks: 540x
