Bump year in copyright information to 2022 and adjust same scripts
to reference NOTICE that didn't have the reference yet.
This patch also removes orphaned test/expected/utils.out.
Enable ALTER MATERIALIZED VIEW (timescaledb.compress)
This enables compression on the underlying materialized
hypertable. The segmentby and orderby columns for
compression are based on the GROUP BY clause and time_bucket
clause used while setting up the continuous aggregate.
timescaledb_information.continuous_aggregate view defn
change
Add support for compression policy on continuous
aggregates
Move code from job.c to policy_utils.c
Add support functions to check compression
policy validity for continuous aggregates.
This patch adds a test to detect changes in minimum required pg versions
for our packages. We build timescaledb against latest postgres version
and then run the tests with that library loaded into previous postgres
version. This patch also changes the ABI test for maximum compatible
postgres version to use the same format and removes the old ABI test
code.
The header file `gssapi/gssapi.h` is present in Alpine package
`krb5-dev` but for some reason it was not installed prior to building,
causing upgrade and downgrade test failures.
Writing a shell script correctly can be hard even for a skilled
programmer. shellcheck is a static analysis tool that helps catch
common errors in shell scripts. We now have 36 executable scripts in
our repository, for which shellcheck reports 126 errors (calculated
like find . -type f -executable -exec bash -c '[ "$(file --brief
--mime-type "$1")" == "text/x-shellscript" ]' sh {} \; -exec shellcheck
-f gcc --exclude=SC2086 {} \; | cut -d: -f1 | sort | uniq | wc -l).
This commit fixes these warnings and adds a GitHub actions workflow
that runs shellcheck on all the executable shell scripts in the
repository. The warning SC2086: Double quote to prevent globbing and
word splitting is disabled globally, because it has little practical
consequences, sometimes leads to false positives, and is general is too
widespread because people forget to quote.
This patch adds a workflow to run coccinelle scripts against our
codebase. This can be used to find errors in the code base.
The initial patch adds a check to find unbalanced heap_form_tuple
heap_freetuple occurances.
https://coccinelle.gitlabpages.inria.fr/website/
Skip the chunk_utils test on PG12 instead of ignoring it because
since the test segfaults it might actually cancel other tests
even though it doesn't run in parallel.
This patch implements various improvements to the memory tests.
Memory tests are split off into separate workflow to allow running
them separately from prerelease tests by pushing to branch
memory_test. The scripts are adjusted to no longer build docker
images to run the tests but instead running the tests directly.
The python script to monitor memory usage was improved and now
flushes after printing to make output appear consistently in CI.
This patch also changes the memory test to run on PG 12,13 and 14
instead of only 12.
Add support for continuous aggregates for distributed hypertables by
allowing a continuous aggregate to read from a distributed hypertable
so that the continuous aggregate is on the access node while the
hypertable data is on the data nodes.
For distributed hypertables, both the hypertable and continuous
aggregate invalidation log are kept on the data nodes and the refresh
window is computed at refresh time on each data node. Since the
continuous aggregate materialization hypertable is not present on the
data nodes, the invalidation log was extended to allow using a
non-local hypertable id on the data nodes. This means that you cannot
create continuous aggregates on the data nodes since those could clash
with continuous aggregates on the access node.
Some utility statements added entries to the invalidation logs
directly (truncating chunks and hypertables, as well as dropping
individual chunks), so to handle this case, internal functions were
added to allow logging invalidation on the data nodes from the access
node.
The commit also includes some fixes to memory context usage that
caused crashes for invalidation triggers and also disable per data
node queries during refresh since that would otherwise generate an
exception.
Fixes#3435
Co-authored-by: Mats Kindahl <mats@timescale.com>
Postgres 12.0 has a use-after-free bug in the event trigger code
leading to a failing chunk_utils test in our nightly CI runs.
This patch ignores this test for runs on PG 12.0.
https://github.com/postgres/postgres/commit/93f726c0
To improve remote query push down, do the following:
* Import changes to remote cost estimates from PostgreSQL 14
`postgres_fdw`. The cost estimations for distributed (remote)
queries are originally based on the `postgres_fdw` code in
PG11. However, fixes and changes have been applied in never
PostgreSQL versions, which improves, among other things, costing of
sorts and having clauses.
* Increase the cost of transferring tuples. This penalizes doing
grouping/aggregation on the AN since it requires transferring more
tuples, leading to the planner preferring the push-down plans.
* As a result of the above, the improved costing also makes
distributed queries behave similar across all currently supported
PostgreSQL versions for our test cases.
* Enable `dist_query` tests on PG14 (since it now passes).
* Update the `dist_partial_agg` test to use additional ordering
columns so that there is no diff in the test output due to ordering
of input to the `first` and `last` functions.
The plan output of the dist_partial_agg test is different on PG14
so we need to make it PG version specific. On PG14 sorts are pushed
down more often leading to better plans in some cases.
This also updates the dist_hypertable-14 test output which differs
to previous PG versions due to some renumbering of relation aliases.
PG14 introduced new `ALTER TABLE` sub-commands:
* `.. ALTER COLUMN .. SET COMPRESSION`: handled it properly on
`process_utility` hook code and added related regression tests
* `.. DETACH PARTITION .. {CONCURRENTLY | FINALIZE}`: handled it
properly on `process_utility` hook code but there's no need to add
regression tests because we don't rely to native partitioning in
hypertables.
Closes#3643
Since custom types are hashable in PG14 the partition test will be
different on PG14. Since the only difference was testing whether
creating hypertable with custom type paritition throws errors
without partitioning function that specific test got moved to ddl
tests which already is pg version specific.
VACUUM VERBOSE is the source for flaky tests and we don't gain much
by including the verbose output in the test. Additionally removing
the verbose option prevents us from having to make the vacuum tests
pg-version specific as PG14 slightly changes the formatting of the
VACUUM VERBOSE output.
With memoize enabled PG14 append tests produce a very different
plan compared to previous PG versions. To make comparing plans
between PG versions easier we disable memoize for PG14.
PG14 also modified how EXTRACT is shown in EXPLAIN output
so any query using EXTRACT will have different EXPLAIN output
between PG14 and previous versions.
The previous PR enabling tests on PG14 did not actually require
tests to pass but with INSERT support merged and most of the tests
passing it makes sense to require tests to pass to not introduce
regression and explicitly not require the currently known failing
tests.
Update version numbers and add 2.4.2 to update tests.
We have to put the DROP FUNCTION back in latest-dev because
2.4.2 did not include the commit which removed the function
definitions.
When renaming a column on a continuous aggregate, only the user view
column was renamed. This commit changes this by applying the rename on
the materialized table, the user view, the direct view, and the partial
view, as well as the column name in the dimension table.
Since this also changes some of the table definitions, we need to
perform the same rename in the update scripts as well, which is done by
comparing the direct view and the user view to decide what columns that
require a rename and then executing that rename on the direct view,
partial view, and materialization table, as well as updating the column
name in the dimension table.
When renaming columns in tables with indexes, the column in the table
is renamed but not the column in the index, which keeps the same name.
When restoring from a dump, however, the column name of the table is
used, which create a diff in the update test. For that reason, we
change the update tests to not list index definitions as part of the
comparison. The existance of the indexes is still tracked and compared
since the indexes for a hypertable is listed as part of the output.
If a downgrade does not revert some changes, this will cause a diff in
the downgrade test. Since the rename is benign and not easy to revert,
this will cause test failure. Instead, we add a file to do extra
actions during a clean-rerun to prevent these diffs. In this case,
applying the same renames as the update script.
Fixes#3405
Since the isolationtester got modified to improve the query result
output which got backported in to PG12 and PG13 we have to adjust
all isolationtester output.
https://github.com/postgres/postgres/commit/4a05406
If tools like `pg_regress` or `pg_isolation_regress` are not found, an
error is generated telling you that you need to disable the tests
explicitly using `REGRESS_CHECKS`. This is an inconvenience for the
user since in most cases you just want to build the system with what
you have.
This commit changes that by automatically disabling regression tests,
isolation tests, TAP tests, and formatting targets when tools cannot be
found, which means that users do not normally have to disable tests
explicitly. A notice is still printed that the targets are not added.
In addition, the commit adds an option `REQUIRE_ALL_TESTS` for the rare
case when you want to make sure that all tests are indeed executed (for
example, in CI runs), which by default is off.
Add 2.3.1 to the update tests and update the downgrade target for the
downgrade version. This commit also fixes two issues that were fixed in
the release branch:
1. Drop `_timescaledb_internal.refresh_continuous_aggregate`
2. Update the changelog to match the release branch.
During an update, it is not possible to run the downgrade scripts until
the release has been tagged, but the update scripts can be run. This
means that we need to split the previous version into two different
fields: one for running the update tests and one for running the
downgrade tests.
This commit add functions and code to generate a downgrade script from
the current version to the previous version. This requires execution
from a Git repository since it retrieves the prolog and epilog for the
"downgrade" file from the version given by `update_from_version` in the
`version.config` file.
The commit adds several CMake functions that simplifies the composition
of script files, but these are not used to generate the update scripts.
A potential improvement is to use the scripts to also generate the
update scripts.
This commit supports generating a downgrade script from the
current version to the previous version. Other versions are handled
using a variable containing file names of reverse update
scripts and the source and target version is extracted from the file
names, which is assumed to be of the form
`<source-version>--<target-version>.sql`.
In addition to adding support for generating downgrade scripts, the
commit adds a downgrade test file that tests a release in a similar way
to the update script and adds it as a workflow.
Change sanitizer test to run on PG12 and make it use the same
infrastructure as the other linux regression tests.
Co-authored-by: Sven Klemm <sven@timescale.com>
The post-update script was handling preserving initprivs for newly
added catalog tables and views. However, newly added catalog sequences
need separate handling otherwise update tests start failing. We also
now grant privileges for all future sequences in the update tests.
In passing, default the PG_VERSION in the update tests to 12 since we
don't work with PG11 anymore.
Add a workflow to check that CMake files are correctly formatted as
well as a custom target to format CMake files in the repository. This
commit also runs the formatting on all CMake files in the repository.
Install the Perl prerequisites when building the image for ABI tests,
including the `prove` binary.
Although the ABI tests currently don't run TAP tests, CMake still
failed the configuration since it expects the prerequisites to be
installed unless it is run with `-DTAP_CHECKS=off`.
CMake now detects if the necessary prerequisites for running TAP
tests are installed on the local system. This includes perl
installation and other dependencies, such as the IPC::Run module and
prove binary.
GitHub removed default installation of clang-9 from the ubuntu
images but those are required to build postgres with llvm so we have
to explicitly install the required packages.
https://github.com/actions/virtual-environments/pull/3381
