This is a *temporary* change to our workflows to ignore the test
because it is flaky and requires many jobs to be restarted before a
commit can be merged. Does not address the issue that causes the
flakiness, which needs to be investigated.
We don't do this currently, so some queries return the wrong ordering
of rows if there is an index on the compressed chunk. The fix only
works for the newly created chunks. We could add a migration that
corrects the old compressed chunks, but it seems to be too heavy and
not to lend itself well to automation -- we'll have to recreate the
indexes if there are any. So the old chunks continue to return a wrong
result.
Add the missing variables to the finalization view of Continuous
Aggregates and the corresponding columns to the materialization table.
Cover the case of targets that contain Aggref nodes and Var nodes
that are outside of the Aggref nodes at the same time.
Stop rebuilding the Continuous Aggregate view with ALTER MATERIALIZED
VIEW. Attempt to repair the view at post-update time instead, and fail
gracefully if it is not possible to do so without raw hypertable schema
or data modifications.
Stop rebuilding the Continuous Aggregate view when switching realtime
aggregation on and off. Instead, manipulate the User View by either:
1. removing the UNION ALL right-hand side and the WHERE clause when
disabling realtime aggregation
2. adding the Direct View to the right of a UNION ALL operator and
defining WHERE clauses with the relevant watermark checks when
enabling realtime aggregation
Fixes#3898
Due to a security vulnerability in git recent git versions now check
ownership of files and error out if it doesnt match current user.
Since we bind mount the source checkout into the build container
the user of the checkout is unlikely to match the user inside the
container. This patch configures git to skip the owner check for
the bind-mounted directory.
https://github.blog/2022-04-12-git-security-vulnerability-announced/
Add option `USE_TELEMETRY` that can be used to exclude telemetry from
the compile.
Telemetry-specific SQL is moved, which is only included when extension
is compiled with telemetry and the notice is changed so that the
message about telemetry is not printed when Telemetry is not compiled
in.
The following code is not compiled in when telemetry is not used:
- Cross-module functions for telemetry.
- Checks for telemetry job in job execution.
- GUC variables `telemetry_level` and `telemetry_cloud`.
Telemetry subsystem is not included when compiling without telemetry,
which requires some functions to be moved out of the telemetry
subsystem:
- Metadata handling is moved out of the telemetry module since it is
used not only with telemetry.
- UUID functions are moved into a separate module instead of being
part of the telemetry subsystem.
- Telemetry functions are either added or removed when updating from a
previous version.
Tests are updated to:
- Not use telemetry functions to get UUID or Metadata and instead use
the moved UUID and metadata functions.
- Not include telemetry information in tests that do not require it.
- Configuration files do not set telemetry variables when telemetry is
not compiled in.
- Replaced usage of telemetry functions in non-telemetry tests with
other sources of same information.
Fixes#3931
Smoke tests where missing critical files and some tests had changed
since last run and did not handle update smoke tests, so fixing all
necessary issues.
TimescaleDB was vulnerable to a privilege escalation attack in
the extension installation script. An attacker could precreate
objects normally owned by the extension and get those objects
used in the installation script since the script would only try
to create them if they did not already exist. Thanks to Pedro
Gallegos for reporting the problem.
This patch changes the schema, table and function creation to fail
and abort the installation when the object already exists instead
of using the existing object.
Security: CVE-2022-24128
On MacOS Monterey with LLVM 13.0.1 when executing export_prefix_check.sh
we get an error:
/usr/local/opt/llvm/bin/nm: error: : unknown argument '-defined-only',
did you mean '--defined-only'?
This patch adds a better check of which flag does `nm` expect.
This patch adds coccinelle scripts for detecting use-after-free
bugs in relation to ts_cache_release.
This will find code using the following pattern:
Use of the hypertable from cache after releasing the cache
and use of the dimension of the hypertable from a cache
after the cache was released.
Change the prefix for continuous aggregate tests from
continuous_aggs_ to cagg_. This is similar to commit 6a8c2b66
which did this adjustment for isolation tests because we were
running into length limitations for the spec name. This patch
adjusts the remaining tests to be consistent with the naming
used in isolation tests.
Upstream changed the explain output for the memoize node to include
an additional `Cache Mode` line. While we could adjust our test
runner to always ignore that line this would prevent us from testing
the cache mode in future tests.
6c32c09777
Bump year in copyright information to 2022 and adjust same scripts
to reference NOTICE that didn't have the reference yet.
This patch also removes orphaned test/expected/utils.out.
Enable ALTER MATERIALIZED VIEW (timescaledb.compress)
This enables compression on the underlying materialized
hypertable. The segmentby and orderby columns for
compression are based on the GROUP BY clause and time_bucket
clause used while setting up the continuous aggregate.
timescaledb_information.continuous_aggregate view defn
change
Add support for compression policy on continuous
aggregates
Move code from job.c to policy_utils.c
Add support functions to check compression
policy validity for continuous aggregates.
This patch adds a test to detect changes in minimum required pg versions
for our packages. We build timescaledb against latest postgres version
and then run the tests with that library loaded into previous postgres
version. This patch also changes the ABI test for maximum compatible
postgres version to use the same format and removes the old ABI test
code.
The header file `gssapi/gssapi.h` is present in Alpine package
`krb5-dev` but for some reason it was not installed prior to building,
causing upgrade and downgrade test failures.
Writing a shell script correctly can be hard even for a skilled
programmer. shellcheck is a static analysis tool that helps catch
common errors in shell scripts. We now have 36 executable scripts in
our repository, for which shellcheck reports 126 errors (calculated
like find . -type f -executable -exec bash -c '[ "$(file --brief
--mime-type "$1")" == "text/x-shellscript" ]' sh {} \; -exec shellcheck
-f gcc --exclude=SC2086 {} \; | cut -d: -f1 | sort | uniq | wc -l).
This commit fixes these warnings and adds a GitHub actions workflow
that runs shellcheck on all the executable shell scripts in the
repository. The warning SC2086: Double quote to prevent globbing and
word splitting is disabled globally, because it has little practical
consequences, sometimes leads to false positives, and is general is too
widespread because people forget to quote.
This patch adds a workflow to run coccinelle scripts against our
codebase. This can be used to find errors in the code base.
The initial patch adds a check to find unbalanced heap_form_tuple
heap_freetuple occurances.
https://coccinelle.gitlabpages.inria.fr/website/
Skip the chunk_utils test on PG12 instead of ignoring it because
since the test segfaults it might actually cancel other tests
even though it doesn't run in parallel.
This patch implements various improvements to the memory tests.
Memory tests are split off into separate workflow to allow running
them separately from prerelease tests by pushing to branch
memory_test. The scripts are adjusted to no longer build docker
images to run the tests but instead running the tests directly.
The python script to monitor memory usage was improved and now
flushes after printing to make output appear consistently in CI.
This patch also changes the memory test to run on PG 12,13 and 14
instead of only 12.
Add support for continuous aggregates for distributed hypertables by
allowing a continuous aggregate to read from a distributed hypertable
so that the continuous aggregate is on the access node while the
hypertable data is on the data nodes.
For distributed hypertables, both the hypertable and continuous
aggregate invalidation log are kept on the data nodes and the refresh
window is computed at refresh time on each data node. Since the
continuous aggregate materialization hypertable is not present on the
data nodes, the invalidation log was extended to allow using a
non-local hypertable id on the data nodes. This means that you cannot
create continuous aggregates on the data nodes since those could clash
with continuous aggregates on the access node.
Some utility statements added entries to the invalidation logs
directly (truncating chunks and hypertables, as well as dropping
individual chunks), so to handle this case, internal functions were
added to allow logging invalidation on the data nodes from the access
node.
The commit also includes some fixes to memory context usage that
caused crashes for invalidation triggers and also disable per data
node queries during refresh since that would otherwise generate an
exception.
Fixes#3435
Co-authored-by: Mats Kindahl <mats@timescale.com>
Postgres 12.0 has a use-after-free bug in the event trigger code
leading to a failing chunk_utils test in our nightly CI runs.
This patch ignores this test for runs on PG 12.0.
https://github.com/postgres/postgres/commit/93f726c0
To improve remote query push down, do the following:
* Import changes to remote cost estimates from PostgreSQL 14
`postgres_fdw`. The cost estimations for distributed (remote)
queries are originally based on the `postgres_fdw` code in
PG11. However, fixes and changes have been applied in never
PostgreSQL versions, which improves, among other things, costing of
sorts and having clauses.
* Increase the cost of transferring tuples. This penalizes doing
grouping/aggregation on the AN since it requires transferring more
tuples, leading to the planner preferring the push-down plans.
* As a result of the above, the improved costing also makes
distributed queries behave similar across all currently supported
PostgreSQL versions for our test cases.
* Enable `dist_query` tests on PG14 (since it now passes).
* Update the `dist_partial_agg` test to use additional ordering
columns so that there is no diff in the test output due to ordering
of input to the `first` and `last` functions.
The plan output of the dist_partial_agg test is different on PG14
so we need to make it PG version specific. On PG14 sorts are pushed
down more often leading to better plans in some cases.
This also updates the dist_hypertable-14 test output which differs
to previous PG versions due to some renumbering of relation aliases.
PG14 introduced new `ALTER TABLE` sub-commands:
* `.. ALTER COLUMN .. SET COMPRESSION`: handled it properly on
`process_utility` hook code and added related regression tests
* `.. DETACH PARTITION .. {CONCURRENTLY | FINALIZE}`: handled it
properly on `process_utility` hook code but there's no need to add
regression tests because we don't rely to native partitioning in
hypertables.
Closes#3643
Since custom types are hashable in PG14 the partition test will be
different on PG14. Since the only difference was testing whether
creating hypertable with custom type paritition throws errors
without partitioning function that specific test got moved to ddl
tests which already is pg version specific.
VACUUM VERBOSE is the source for flaky tests and we don't gain much
by including the verbose output in the test. Additionally removing
the verbose option prevents us from having to make the vacuum tests
pg-version specific as PG14 slightly changes the formatting of the
VACUUM VERBOSE output.
With memoize enabled PG14 append tests produce a very different
plan compared to previous PG versions. To make comparing plans
between PG versions easier we disable memoize for PG14.
PG14 also modified how EXTRACT is shown in EXPLAIN output
so any query using EXTRACT will have different EXPLAIN output
between PG14 and previous versions.
The previous PR enabling tests on PG14 did not actually require
tests to pass but with INSERT support merged and most of the tests
passing it makes sense to require tests to pass to not introduce
regression and explicitly not require the currently known failing
tests.
