This patch changes the extension function list to include the
signature as well since functions with different signature are
separate objects in postgres. This also changes the list to include
all functions. Even though functions in internal schemas are not
considered public API they still need be treated the same as functions
in other schemas with regards to extension upgrade/downgrade.
This patch also moves the test to regresscheck-shared since we do
not dedicated database to run these tests.
Reorganize the code and fix minor bug that was not computing the size
of FSM, VM and INIT forks of the parent hypertable.
Fixed the bug by exposing the `ts_relation_size` function to the SQL
level to encapsulate the logic to compute `heap`, `indexes` and `toast`
sizes.
Add option `USE_TELEMETRY` that can be used to exclude telemetry from
the compile.
Telemetry-specific SQL is moved, which is only included when extension
is compiled with telemetry and the notice is changed so that the
message about telemetry is not printed when Telemetry is not compiled
in.
The following code is not compiled in when telemetry is not used:
- Cross-module functions for telemetry.
- Checks for telemetry job in job execution.
- GUC variables `telemetry_level` and `telemetry_cloud`.
Telemetry subsystem is not included when compiling without telemetry,
which requires some functions to be moved out of the telemetry
subsystem:
- Metadata handling is moved out of the telemetry module since it is
used not only with telemetry.
- UUID functions are moved into a separate module instead of being
part of the telemetry subsystem.
- Telemetry functions are either added or removed when updating from a
previous version.
Tests are updated to:
- Not use telemetry functions to get UUID or Metadata and instead use
the moved UUID and metadata functions.
- Not include telemetry information in tests that do not require it.
- Configuration files do not set telemetry variables when telemetry is
not compiled in.
- Replaced usage of telemetry functions in non-telemetry tests with
other sources of same information.
Fixes#3931
Smoke tests where missing critical files and some tests had changed
since last run and did not handle update smoke tests, so fixing all
necessary issues.
Chunk scan performance during querying is improved by avoiding
repeated open and close of relations and indexes when joining chunk
information from different metadata tables.
When executing a query on a hypertable, it is expanded to include all
its children chunks. However, during the expansion, the chunks that
don't match the query constraints should also be excluded. The
following changes are made to make the scanning and exclusion more
efficient:
* Ensure metadata relations and indexes are only opened once even
though metadata for multiple chunks are scanned. This avoids doing
repeated open and close of tables and indexes for each chunk
scanned.
* Avoid interleaving scans of different relations, ensuring better
data locality, and having, e.g., indexes warm in cache.
* Avoid unnecessary scans that repeat work already done.
* Ensure chunks are locked in a consistent order (based on Oid).
To enable the above changes, some refactoring was necessary. The chunk
scans that happen during constraint exclusion are moved into separate
source files (`chunk_scan.c`) for better structure and readability.
Some test outputs are affected due to the new ordering of chunks in
append relations.
Make the Scanner module more flexible by allowing optional control
over when the scanned relation is opened and closed. Relations can
then remain open over multiple scans, which can improve performance
and efficiency.
Closes#2173
We cache the Chunk structs in RelOptInfo private data. They are later
used to estimate the chunk sizes, check which data nodes they belong
to, et cetera. Looking up the chunks is expensive, so this change
speeds up the planning.
This patch locks down search_path in extension install and update
scripts to only contain pg_catalog, this requires that any reference
in those scripts is fully qualified. Additionally we add explicit
create commands to all update scripts for objects added to the
public schema. This change will make update scripts fail if a
function with identical signature already exists when installing
or upgrading instead reusing the existing object.
TimescaleDB was vulnerable to a privilege escalation attack in
the extension installation script. An attacker could precreate
objects normally owned by the extension and get those objects
used in the installation script since the script would only try
to create them if they did not already exist. Thanks to Pedro
Gallegos for reporting the problem.
This patch changes the schema, table and function creation to fail
and abort the installation when the object already exists instead
of using the existing object.
Security: CVE-2022-24128
If the ANY construct contains a singleton NULL then the logic in
"dimension_values_create_from_array" barfs causing a crash. Fix it
appropriately in the caller "hypertable_restrict_info_add_expr"
function.
Refactor the telemetry function and format to include stats broken
down on common relation types. The types include:
- Tables
- Partitioned tables
- Hypertables
- Distributed hypertables
- Continuous aggregates
- Materialized views
- Views
and for each of these types report (when applicable):
- Total number of relations
- Total number of children/chunks
- Total data volume (broken into heap, toast, and indexes).
- Compression stats
- PG stats, like reltuples
The telemetry function has also been refactored to return `jsonb`
instead of `text`. This makes it easier to query and manipulate the
resulting JSON format, and also gives cleaner output.
Closes#3932
To support tests with different configuration options, we split the
tests into *test configurations*. Each test configuration NAME will have
- A configuration template file `NAME.conf.in` that is used to run the
suite of tests.
- A variable `TEST_FILES_<NAME>` listing the test files available for
that test suite.
- A variable `SOLO_TESTS_<NAME>` that lists the tests that need to be
run as solo tests.
The code to generate test schedules is then factored out into a
separate file and used for each configuration.
This patch changes DELETE handling for hypertables to have the
postgres ModifyTable node be wrapped in a custom HypertableModify
node. By itself this does not change DELETE handling for hypertables
but instead enables subsequent patches to implement e.g. chunk
exclusion for DELETE or DELETE on compressed chunks.
Since PG 14 codepath for INSERT is different from previous versions
this PR will only change the plan for PG14+. DELETE handling for
distributed hypertables is not changed as part of this patch.
Starting with PG15, default permissions on the public schema is
restricted for any non-superuser non-owner. This causes test failures
since tables can no longer be created without explicitly adding
permissions, so we remove grant when bootstrapping the data nodes and
instead grant permissions to the users in the regression tests. This
keeps the default permissions on data nodes, but allow regression tests
to run.
Fixes#3957
Reference: https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=b073c3cc
Commit 97c2578ffa6b08f733a75381defefc176c91826b overcomplicated the
`invalidate_add_entry` API by adding parameters related to the remote
function call for multi-node on materialization hypertables.
Refactored it simplifying the function interface and adding a new
function to deal with materialization hypertables on multi-node
environment.
Fixes#3833
This patch optimizes how first()/last() initialize the compare
function. Previously the compare function would be looked up for
every transition function call but since polymorphic types are
resolved at parse time for a specific aggregate instance the compare
function will not change during its lifetime. Additionally this
patch also fixes a memory leak when using first()/last() on
pointer types.
These changes lead to a roughly 2x speed improvement for first()/
last() and make the memory usage near-constant.
Fixes#3935
Enable ALTER MATERIALIZED VIEW (timescaledb.compress)
This enables compression on the underlying materialized
hypertable. The segmentby and orderby columns for
compression are based on the GROUP BY clause and time_bucket
clause used while setting up the continuous aggregate.
timescaledb_information.continuous_aggregate view defn
change
Add support for compression policy on continuous
aggregates
Move code from job.c to policy_utils.c
Add support functions to check compression
policy validity for continuous aggregates.
The time_bucket comparison transformation code assumed the value and
the width of the time_bucket comparison expression were both Const.
But this was not validated only asserted. This can lead to wrong
query results. Found by sqlsmith.
Calling `CREATE DATABASE` we cannot have any backends connected to the
source database, except the current one. Since background workers
connect to any database that has installed the extension, they can
linger and cause a flaky error.
The test changed to just check that the extension can be implicitly
installed, but this require waiting for backends to be unregistered
from `procArray` in `procarray.c`, which the function
`pg_terminate_backend` does not handle properly and hence results in a
flaky test. This happen because `pg_terminate_backend` only signals the
process to terminate, but does not wait for it to actually terminate.
Instead, we add an explicit function that checks the `procArray` and
waits for it to not have any backends for a database and uses that to
wait for termination.
Since the number of buckets of the histogram function is an argument
to the function call it is possible to initialize the histogram
state with a lower number than actually needed in further calls
leading to a segfault. This patch changes the memory access to use
the number the state was initialized with instead of the number
passed to the call. This also changes the function to error when
the passed number differs from the initialized state.
This patch fixes a typo in the error message of CREATE INDEX when
using timescaledb.transaction_per_chunk.
Co-authored-by: Sven Klemm <sven@timescale.com>
Fix the "GRANT/REVOKE ALL IN SCHEMA" handling uniformly across
single-node and multi-node.
Even thought this is a SCHEMA specific activity, we decided to
include the chunks even if they are part of another SCHEMA. So
they will also end up getting/resetting the same privileges.
Includes test case changes for both single-node and multi-node use
cases.
PG14 introduced new `ALTER TABLE` sub-commands:
* `.. ALTER COLUMN .. SET COMPRESSION`: handled it properly on
`process_utility` hook code and added related regression tests
* `.. DETACH PARTITION .. {CONCURRENTLY | FINALIZE}`: handled it
properly on `process_utility` hook code but there's no need to add
regression tests because we don't rely to native partitioning in
hypertables.
Closes#3643
Since custom types are hashable in PG14 the partition test will be
different on PG14. Since the only difference was testing whether
creating hypertable with custom type paritition throws errors
without partitioning function that specific test got moved to ddl
tests which already is pg version specific.
VACUUM VERBOSE is the source for flaky tests and we don't gain much
by including the verbose output in the test. Additionally removing
the verbose option prevents us from having to make the vacuum tests
pg-version specific as PG14 slightly changes the formatting of the
VACUUM VERBOSE output.
With memoize enabled PG14 append tests produce a very different
plan compared to previous PG versions. To make comparing plans
between PG versions easier we disable memoize for PG14.
PG14 also modified how EXTRACT is shown in EXPLAIN output
so any query using EXTRACT will have different EXPLAIN output
between PG14 and previous versions.
Inside the `process_truncate` function is created a new relations list
removing the distributed hypertables and this new list is assigned to
the current statement relation list. This new list is allocated into
the `PortalContext` that is destroyed at the end of the statement
execution.
The problem arise on the subsequent `TRUNCATE` call because the
compiled plpgsql code is cached into another memory context and the
elements of the relations inside this cache is pointing to an invalid
memory area because the `PortalContext` is destroyed.
Fixed it by allocating the new relations list to the same memory
context of the original list.
Fixes#3580, fixes#3622, fixes#3182
Using now() in regression tests will result in flaky tests as this
can result in creating different number of chunks depending on
alignment of now() relative to chunk boundaries.
When a new chunk is created, the ACL is copied from the hypertable, but
the shared dependencies are not set at all. Since there is no shared
dependency, a `DROP OWNED BY` will not find the chunk and revoke the
privileges for the user from the chunk. When the user is later dropped,
the ACL for the chunk will contain a non-existent user.
This commit fixes this by adding shared dependencies of the hypertable
to the chunk when the chunk is created.
Fixes#3614
Update version numbers and add 2.4.2 to update tests.
We have to put the DROP FUNCTION back in latest-dev because
2.4.2 did not include the commit which removed the function
definitions.
When renaming a column on a continuous aggregate, only the user view
column was renamed. This commit changes this by applying the rename on
the materialized table, the user view, the direct view, and the partial
view, as well as the column name in the dimension table.
Since this also changes some of the table definitions, we need to
perform the same rename in the update scripts as well, which is done by
comparing the direct view and the user view to decide what columns that
require a rename and then executing that rename on the direct view,
partial view, and materialization table, as well as updating the column
name in the dimension table.
When renaming columns in tables with indexes, the column in the table
is renamed but not the column in the index, which keeps the same name.
When restoring from a dump, however, the column name of the table is
used, which create a diff in the update test. For that reason, we
change the update tests to not list index definitions as part of the
comparison. The existance of the indexes is still tracked and compared
since the indexes for a hypertable is listed as part of the output.
If a downgrade does not revert some changes, this will cause a diff in
the downgrade test. Since the rename is benign and not easy to revert,
this will cause test failure. Instead, we add a file to do extra
actions during a clean-rerun to prevent these diffs. In this case,
applying the same renames as the update script.
Fixes#3405
Rewrite approximate_row_count to SQL instead of PLpgSQL and remove
superfluous JOINs against pg_namespace. Adjust tuple calculation
for PG14 since in PG14 reltuples for partitioned tables is the sum
of it's children so we need to exclude those from calculation to
not doublecount.
This patch adds a version of time_bucket_ng() with 'origin' argument.
It doesn't address any other known issues. E.g. volatility of the function
will be changed in another patch. The error messages are going to be improved
when the feature gets a little more stable.
This patch adds support of timezones in time_bucket_ng(). The 'origin'
argument can't be used with timezones yet. This will be implemented in
a separate pull request.
In this specific case, when we create a hypertable then
we add a "not-null" constraint to the "time" column if it
does not exist. That is done via an internal ALTER TABLE
subcommand in dimension_add_not_null_on_column function. If the
currentEventTriggerState structure is enabled then it's necessary to
set up the command tracking appropriately, otherwise crash ensues. This
has been fixed via this now.
Includes test changes.
If the names for entries in the targetlist for the direct and partial
views of the continuous aggregate does not match the attribute names in
the tuple descriptor for the result tuple of the user view an error
will be generated. This commit fixes this by setting the targetlist
resource names of the columns in the direct and partial view to the
corresponding attribute name of the user view relation tuple
descriptor.
Fixes#3051Fixes#3405
As a future replacement for time_bucket(), time_bucket_ng()
should support seconds, minutes, and hours. This patch adds
this support. The implementation is the same as for
time_bucket(). Timezones are not yet supported.
In nested function invocations hypertable expansion would not work
correctly and a hypertable would not be expanded by timescaledb
code but by postgres table inheritance leading to fdw_private
not being properly initialized.
Fixes#3391
Rework debug waitpoint functionality to produce an error in
case if the waitpoint is enabled.
This update introduce a controlled way to simulate errors
scenarios during testing.
Move the timestamp_limits and with_clause_parser test to
regresscheck-shared since those tests don't need a private
database incurring less overhead to run those tests.
Also add missing ORDER BY clauses to some of the queries
in timestamp_limits to make the output more stable.
This patch adds time_bucket_ng() function to the experimental
schema. The "ng" part stands for "next generation". Unlike
current time_bucket() implementation the _ng version will support
months, years and timezones.
Current implementation doesn't claim to be complete. For instance,
it doesn't support timezones yet. The reasons to commit it in it's
current state are 1) to shorten the feedback loop 2) to start
experimenting with monthly buckets are soon as possible,
3) to reduce the unnecessary work of rebasing and resolving
conflicts 4) to make the work easier to the reviewers
The post-update script was handling preserving initprivs for newly
added catalog tables and views. However, newly added catalog sequences
need separate handling otherwise update tests start failing. We also
now grant privileges for all future sequences in the update tests.
In passing, default the PG_VERSION in the update tests to 12 since we
don't work with PG11 anymore.
Add a workflow to check that CMake files are correctly formatted as
well as a custom target to format CMake files in the repository. This
commit also runs the formatting on all CMake files in the repository.
Combine dist_hypertable_pg12 test since all the tests in that file
can run on all supported PG versions now.
We also rename the views test to information_views to make it clearer
what the test is about and rename the join test to pg_join since
this is the postgres join test ported to hypertables.
With the removal of PG11 support we can use the same template for
the rowsecurity test. We still need to keep the output version
specific since the plan output differs between PG12 and PG13.
Triggers on distributed hypertables can now be renamed due to having
the rename statement forwarded to data nodes. This also applies to
other objects on such tables, like constraints and indexes, since they
share the same DDL "rename" machinery. Tests are added for these
cases.
For convenience, trigger functions on distributed hypertables will now
also be created automatically on the data nodes. In other words, it is
no longer necessary to pre-create the trigger function on all data
nodes.
This change also fixes an issue with statement-level triggers, which
weren't properly propagated to data nodes during `CREATE TRIGGER`.
Fixes#3238
