In some cases, entire hypertables can be excluded
at runtime. Some Examples:
WHERE col @> ANY(subselect)
if the subselect returns empty set
WHERE col op (subselect)
if the op is a strict operator and
the subselect returns empty set.
When qual clauses are not on partition columns, we use
the old chunk exclusion, otherwise we try hypertable exclusion.
Hypertable exclusion is executed once per hypertable.
This is cheaper than the chunk exclusion
that is once-per-chunk.
Do not allocate various temporary data in PortalContext, such as the
hyperspace point corresponding to the row, or the intermediate data
required for chunk lookup.
This patch allows bucketing by month for time_bucket with date,
timestamp or timestamptz. When bucketing by month the interval
must only contain month components. When using origin together
with bucketing by month only the year and month components are
honoured.
To bucket by month we get the year and month of a date and convert
that to the nth month since origin. This allows us to treat month
bucketing similar to int bucketing. During this process we ignore
the day component and therefore only support bucketing by full months.
Make truncating a uncompressed chunk drop the data for the case where
they reside in a corresponding compressed chunk.
Generate invalidations for Continuous Aggregates after TRUNCATE, so
as to have consistent refresh operations on the materialization
hypertable.
Fixes#4362
When executing `get_partition_{hash|for_key}` inside an IMMUTABLE
function we're getting the following error:
`ERROR: unsupported expression argument node type 112`
This error is because the underlying `resolve_function_argtype` was not
dealing with `T_Param` node type.
Fixed it by dealing properly with `T_Param` node type returning the
`paramtype` for the argument type.
Fixes#4575
When working on a fix for #4555 discovered that executing
`{GRANT|REVOKE} .. ON ALL TABLES IN SCHEMA` in an empty schema lead to
an assertion because we change the way that command is executed by
collecting all objects involved and processing one by one.
Fixed it by executing the previous process utility hook just when the
list of target objects is not empty.
Fixes#4581
Add a new metadata table `dimension_partition` which explicitly and
statefully details how a space dimension is split into partitions, and
(in the case of multi-node) which data nodes are responsible for
storing chunks in each partition. Previously, partition and data nodes
were assigned dynamically based on the current state when creating a
chunk.
This is the first in a series of changes that will add more advanced
functionality over time. For now, the metadata table simply writes out
what was previously computed dynamically in code. Future code changes
will alter the behavior to do smarter updates to the partitions when,
e.g., adding and removing data nodes.
The idea of the `dimension_partition` table is to minimize changes in
the partition to data node mappings across various events, such as
changes in the number of data nodes, number of partitions, or the
replication factor, which affect the mappings. For example, increasing
the number of partitions from 3 to 4 currently leads to redefining all
partition ranges and data node mappings to account for the new
partition. Complete repartitioning can be disruptive to multi-node
deployments. With stateful mappings, it is possible to split an
existing partition without affecting the other partitions (similar to
partitioning using consistent hashing).
Note that the dimension partition table expresses the current state of
space partitions; i.e., the space-dimension constraints and data nodes
to be assigned to new chunks. Existing chunks are not affected by
changes in the dimension partition table, although an external job
could rewrite, move, or copy chunks as desired to comply with the
current dimension partition state. As such, the dimension partition
table represents the "desired" space partitioning state.
Part of #4125
This change allows to create new dimensions even with
existing chunks.
It does not modify any existing data or do migration,
instead it creates full-range (-inf/inf) dimension slice for
existing chunks in order to be compatible with newly created
dimension.
All new chunks created after this will follow logic of the new
dimension and its partitioning.
Fix: #2818
In the session timescaledb_post_restore() was called the value for
timescaledb.restoring might not be changed because the reset_val
for the GUC was still on. We have to use explicit SET in this
session to adjust the GUC.
When triggering chunk creation on a hypertable with non-default
statistics targets by a user different from the hypertable owner
the chunk creation will fail with a permission error. This patch
changes the chunk table creation to run the attribute modification
as the table owner.
Fixes#4474
Executing an IMMUTABLE function that has parameters and exception
handling block multiple times in the same transaction causes a null
pointer segfault when try to reset a non-initialized ts_baserel_info.
Fixed it by preventing to reset a non-initialized `ts_baserel_info`.
Fixes#4489
Timescale 2.7 released a new version of Continuous Aggregate (#4269)
that allows users efectivelly create and use indexes in the
materialization hypertable. The boring part of it is that users should
discover what is the associated materialization hypertable to issue a
`CREATE INDEX` statement.
Improved it by allowing users to easily create indexes in the
materialization hypertable by simple executing a `CREATE INDEX` direct
in the Continuous Aggregate.
Example:
`CREATE INDEX name_of_the_index ON continuous_agregate (column);`
The copy test is flaky because some test data is generated
dynamically based on the current date. This patch changes the data
generation to a time series with fixed dates.
This commit contains extends our telemetry system with function call
telemetry. It gathers function call-counts from all queries, and send
back counts for those functions that are built in or from our related
extensions.
A number of TimescaleDB functions internally call `AlterTableInternal`
to modify tables or indexes. For instance, `compress_chunk` and
`attach_tablespace` act as DDL commands to modify
hypertables. However, crashes occur when these functions are called
via `SELECT * INTO FROM <function_name>` or the equivalent `CREATE
TABLE AS` statement. The crashes happen because these statements are
considered process utility commands and therefore sets up an event
trigger context for collecting commands. However, the event trigger
context is not properly set up to record alter table statements in
this code path, thus causing the crashes.
To prevent crashes, wrap `AlterTableInternal` with the event trigger
functions to properly initialize the event trigger context.
This patch solves a crash in the multi-buffer copy optimization,
which was introduced in commit
bbb2f414d2090efd2d8533b464584157860ce49a.
This patch handles closed chunks (e.g., caused by timescaledb.max_open_
chunks_per_insert) properly. The problem is addressed by:
1) Re-reading the ChunkInsertState before the data is stored, which
ensures that the underlying table is open.
2) A TSCopyMultiInsertBuffer is deleted after the data of the buffer
is flushed. So, operations like table_finish_bulk_insert are
executed and the associated chunk can properly be closed.
This patch adds a test for attnum consistency to our update scripts.
When attnum between fresh install and updated install is different
the updated installation will not be able to correctly process affected
catalog tables.
Following work started by #4294 to improve performance of Continuous
Aggregates by removing the re-aggregation in the user view.
This PR get rid of `partialize_agg` and `finalize_agg` aggregate
functions and store the finalized aggregated (plain) data in the
materialization hypertable.
Because we're not storing partials anymore and removed the
re-aggregation, now is be possible to create indexes on aggregated
columns in the materialization hypertable in order to improve the
performance even more.
Also removed restrictions on types of aggregates users can perform
with Continuous Aggregates:
* aggregates with DISTINCT
* aggregates with FILTER
* aggregates with FILTER in HAVING clause
* aggregates without combine function
* ordered-set aggregates
* hypothetical-set aggregates
By default new Continuous Aggregates will be created using this new
format, but the previous version (with partials) will be supported.
Users can create the previous style by setting to `false` the storage
paramater named `timescaledb.finalized` during the creation of the
Continuous Aggregate.
Fixes#4233
This implements an optimization to allow now() expression to be
used during plan time chunk exclusions. Since now() is stable it
would not normally be considered for plan time chunk exclusion.
To enable this behaviour we convert `column > now()` expressions
into `column > const AND column > now()`. Assuming that time
always moves forward this is safe even for prepared statements.
This optimization works for SELECT, UPDATE and DELETE.
On hypertables with many chunks this can lead to a considerable
speedup for certain queries.
The following expressions are supported:
- column > now()
- column >= now()
- column > now() - Interval
- column > now() + Interval
- column >= now() - Interval
- column >= now() + Interval
Interval must not have a day or month component as those depend
on timezone settings.
Some microbenchmark to show the improvements, I did best of five
for all of the queries.
-- hypertable with 1k chunks
-- with optimization
select * from metrics1k where time > now() - '5m'::interval;
Time: 3.090 ms
-- without optimization
select * from metrics1k where time > now() - '5m'::interval;
Time: 145.640 ms
-- hypertable with 5k chunks
-- with optimization
select * from metrics5k where time > now() - '5m'::interval;
Time: 4.317 ms
-- without optimization
select * from metrics5k where time > now() - '5m'::interval;
Time: 775.259 ms
-- hypertable with 10k chunks
-- with optimization
select * from metrics10k where time > now() - '5m'::interval;
Time: 4.853 ms
-- without optimization
select * from metrics10k where time > now() - '5m'::interval;
Time: 1766.319 ms (00:01.766)
-- hypertable with 20k chunks
-- with optimization
select * from metrics20k where time > now() - '5m'::interval;
Time: 6.141 ms
-- without optimization
select * from metrics20k where time > now() - '5m'::interval;
Time: 3321.968 ms (00:03.322)
Speedup with 1k chunks: 47x
Speedup with 5k chunks: 179x
Speedup with 10k chunks: 363x
Speedup with 20k chunks: 540x
The setup scripts for upgrade/downgrade tests of Continuous Aggregates
has too many duplicated code for pre-2.0 tests. Refactor it a bit
removing the duplicated code by using `\if \else \endif` psql
meta-commands.
Also added a properly `round` function to all functions that returns
`float8` in SQL scripts because in rare cases it lead to flaky tests.
This is part of #4269.
This commit backports the Postgres multi-buffer / bulk insert
optimization into the timescale copy operator. If the target chunk
allows it (e.g., if no triggers are defined on the hypertable or the
chunk is not compressed), the data is stored in in-memory buffers
first and then flushed to the chunks in bulk operations.
Implements: #4080
Add an internal api to drop a single chunk.
This function drops the storage and metadata
associated with the chunk.
Note that chunk dependencies are not affected.
e.g. Continuous aggs are not updated when this chunk
is dropped.
First step to remove the re-aggregation for Continuous Aggregates
is to remove the `chunk_id` from the materialization hypertable.
Also added new metadata column named `finalized` to `continuous_cagg`
catalog table in order to store information about the new following
finalized version of Continuous Aggregates that will not need the
partials anymore. This flag is important to maintain backward
compatibility with previous Continuous Aggregate implementation that
requires the `chunk_id` to refresh data properly.
When postgres expands an inheritance tree it also adds the
parent hypertable as child relation. Since for a hypertable
the parent will never have any data we can mark this
relation as dummy relation so it gets ignored in later
steps. This is only relevant for code paths that use the
postgres inheritance code as we don't include the hypertable
as child when expanding the hypertable ourself.
This is similar to 3c40f924 which did the same adjustment for DELETE.
This patch also moves the marking into get_relation_info_hook so
it happens a bit earlier and prevents some additional cycles.
Commit 3c40f924 accidently broke DELETE statement triggers on PG14 that
were only defined on the hypertable itself. This patch fixes the issue
and also makes the trigger test no longer pg version specific.
Change truncate test to ignore warnings about potentially orphaned
files when dropping the test database. This seems to happen quite
frequently on appveyor causing the test to be flaky.
When postgres expands an inheritance tree it also adds the
parent hypertable as child relation. Since for a hypertable the
parent will never have any data we can mark this relation as
dummy relation so it gets ignored in later steps. This is only
relevant for code paths that use the postgres inheritance code
as we don't include the hypertable as child when expanding the
hypertable ourself.
Add the missing variables to the finalization view of Continuous
Aggregates and the corresponding columns to the materialization table.
Cover the case of targets that contain Aggref nodes and Var nodes
that are outside of the Aggref nodes at the same time.
Stop rebuilding the Continuous Aggregate view with ALTER MATERIALIZED
VIEW. Attempt to repair the view at post-update time instead, and fail
gracefully if it is not possible to do so without raw hypertable schema
or data modifications.
Stop rebuilding the Continuous Aggregate view when switching realtime
aggregation on and off. Instead, manipulate the User View by either:
1. removing the UNION ALL right-hand side and the WHERE clause when
disabling realtime aggregation
2. adding the Direct View to the right of a UNION ALL operator and
defining WHERE clauses with the relevant watermark checks when
enabling realtime aggregation
Fixes#3898
Stop throwing error "must be owner of hypertable" when a user with
TRUNCATE privilege on the hypertable attempts to TRUNCATE.
Previously we had a check that required TRUNCATE to only be
performed by the table owner, not taking into account the user's
TRUNCATE privilege, which is sufficient to allow this operation.
Fixes#4183
This patch changes the extension function list to include the
signature as well since functions with different signature are
separate objects in postgres. This also changes the list to include
all functions. Even though functions in internal schemas are not
considered public API they still need be treated the same as functions
in other schemas with regards to extension upgrade/downgrade.
This patch also moves the test to regresscheck-shared since we do
not dedicated database to run these tests.
Reorganize the code and fix minor bug that was not computing the size
of FSM, VM and INIT forks of the parent hypertable.
Fixed the bug by exposing the `ts_relation_size` function to the SQL
level to encapsulate the logic to compute `heap`, `indexes` and `toast`
sizes.
Add option `USE_TELEMETRY` that can be used to exclude telemetry from
the compile.
Telemetry-specific SQL is moved, which is only included when extension
is compiled with telemetry and the notice is changed so that the
message about telemetry is not printed when Telemetry is not compiled
in.
The following code is not compiled in when telemetry is not used:
- Cross-module functions for telemetry.
- Checks for telemetry job in job execution.
- GUC variables `telemetry_level` and `telemetry_cloud`.
Telemetry subsystem is not included when compiling without telemetry,
which requires some functions to be moved out of the telemetry
subsystem:
- Metadata handling is moved out of the telemetry module since it is
used not only with telemetry.
- UUID functions are moved into a separate module instead of being
part of the telemetry subsystem.
- Telemetry functions are either added or removed when updating from a
previous version.
Tests are updated to:
- Not use telemetry functions to get UUID or Metadata and instead use
the moved UUID and metadata functions.
- Not include telemetry information in tests that do not require it.
- Configuration files do not set telemetry variables when telemetry is
not compiled in.
- Replaced usage of telemetry functions in non-telemetry tests with
other sources of same information.
Fixes#3931
Smoke tests where missing critical files and some tests had changed
since last run and did not handle update smoke tests, so fixing all
necessary issues.
Chunk scan performance during querying is improved by avoiding
repeated open and close of relations and indexes when joining chunk
information from different metadata tables.
When executing a query on a hypertable, it is expanded to include all
its children chunks. However, during the expansion, the chunks that
don't match the query constraints should also be excluded. The
following changes are made to make the scanning and exclusion more
efficient:
* Ensure metadata relations and indexes are only opened once even
though metadata for multiple chunks are scanned. This avoids doing
repeated open and close of tables and indexes for each chunk
scanned.
* Avoid interleaving scans of different relations, ensuring better
data locality, and having, e.g., indexes warm in cache.
* Avoid unnecessary scans that repeat work already done.
* Ensure chunks are locked in a consistent order (based on Oid).
To enable the above changes, some refactoring was necessary. The chunk
scans that happen during constraint exclusion are moved into separate
source files (`chunk_scan.c`) for better structure and readability.
Some test outputs are affected due to the new ordering of chunks in
append relations.
Make the Scanner module more flexible by allowing optional control
over when the scanned relation is opened and closed. Relations can
then remain open over multiple scans, which can improve performance
and efficiency.
Closes#2173
We cache the Chunk structs in RelOptInfo private data. They are later
used to estimate the chunk sizes, check which data nodes they belong
to, et cetera. Looking up the chunks is expensive, so this change
speeds up the planning.
This patch locks down search_path in extension install and update
scripts to only contain pg_catalog, this requires that any reference
in those scripts is fully qualified. Additionally we add explicit
create commands to all update scripts for objects added to the
public schema. This change will make update scripts fail if a
function with identical signature already exists when installing
or upgrading instead reusing the existing object.
TimescaleDB was vulnerable to a privilege escalation attack in
the extension installation script. An attacker could precreate
objects normally owned by the extension and get those objects
used in the installation script since the script would only try
to create them if they did not already exist. Thanks to Pedro
Gallegos for reporting the problem.
This patch changes the schema, table and function creation to fail
and abort the installation when the object already exists instead
of using the existing object.
Security: CVE-2022-24128
If the ANY construct contains a singleton NULL then the logic in
"dimension_values_create_from_array" barfs causing a crash. Fix it
appropriately in the caller "hypertable_restrict_info_add_expr"
function.
Refactor the telemetry function and format to include stats broken
down on common relation types. The types include:
- Tables
- Partitioned tables
- Hypertables
- Distributed hypertables
- Continuous aggregates
- Materialized views
- Views
and for each of these types report (when applicable):
- Total number of relations
- Total number of children/chunks
- Total data volume (broken into heap, toast, and indexes).
- Compression stats
- PG stats, like reltuples
The telemetry function has also been refactored to return `jsonb`
instead of `text`. This makes it easier to query and manipulate the
resulting JSON format, and also gives cleaner output.
Closes#3932
