The test runner used to use a lockfile to decide whether
initial cluster setup has to be done. Unfortunately
checking for existance and creating the lockfile are 2 distinct
operations leading to race conditions. This patch changes the
runner to use a directory instead because with a directory
the operation is atomic.
Setting the `timescaledb.restoring` guc explicitly to 'off'
for the db meant that the setting got exported in `pg_dumpall`
and some other cases where that setting would then conflict
with the setting set by the pre_restore function causing it to
be overridden and causing errors on restore. This changes to
`RESET` so that instead it will take the system default and not
be dumped separately as an override.
This change ensures that API functions and DDL operations
which modify data respects read-only transaction state
set by default_transaction_read_only option.
This patch makes TSL_MODULE_PATHNAME available when executing
testsupport.sql in the regression test runner. This fixes an
error that happened in the test runner that was suppressed
because it happens before the actual test run.
The sql tests still had version checks and would run EXPLAIN
with different parameters depending on postgres version. Since
all supported postgres versions now support all the parameters
we use we can safely remove the version check.
When fail-fast is true a single failing job will cancel
the entire workflow which is not desirable for scheduled runs
and the code style tests. This patch changes fail-fast
to false for the code style tests and the scheduled i386 tests.
It also changes the different code style check steps to always
run.
The telemetry code still had code to handle the version format used
by postgres before PG10 which is dead code now since PG11 is the
minimum version. This patch removes that code path.
The compression_ddl test had a permutation that depended on the
PGISOLATIONTIMEOUT to cancel the test leading to unreasonably long
running and flaky test. This patch changes the test to set
lock_timeout instead to cancel the blocking much earlier.
In slower environments the isolation tests are extremely flaky
because lock_timeout is only 50ms this patch changes lock_timeout
to 500ms for the isolation tests leading to much more reliable tests
in those environments.
With replicated chunks, the function to import column stats would
experience errors when updating `pg_statistics`, since it tried to
write identical stats from several replica chunks.
This change fixes this issue by filtering duplicate stats rows
received from data nodes.
In the future, this could be improved by only requesting stats from
"primary" chunks on each data node, thus avoiding duplicates without
having to filter the result. However, this would complicate the
function interface as it would require sending a list of chunks
instead of just getting the stats for all chunks in a hypertable.
The "invalidation" column was accidentally set as the "processing"
column in `show_connection_cache`. This made the output of these
columns show the wrong values. This change fixes this issue.
Replaces now() with mocked timestamp value in
continuous_aggs_materialize test, so the test doesn't produce
unexpected result when now() is close to a chunk border.
PostgreSQL 12 is preinstalled, while 11 is not. To unify different
paths of PG 11 and 12 binaries, this commit implements workaround by
forcing installation of PostgreSQL 12, so it is in the same path as
PostgreSQL 11.
Function `ts_hypertable_create_from_info` will error if the hypertable
already exists unless the if-not-exists flag is not set. If we reach
this point, either if-not-exists-flag was set or the hypertable did not
exist and was created above.
In `ts_hypertable_create_from_info`, a call to
`ts_dimension_info_validate` with `space_dim_info` will be made if (and
only if) the table did not exist. The function does not only validate
the dimension, it also set `dimension_id` field.
If the table already existed, `created` will be false and the
`dimension_id` will be set to the invalid OID, which means that
`ts_hypertable_check_partitioning` will crash since it it expect a
proper OID to be passed.
This commit fixes that by checking if the hypertable exists prior to
calling `ts_hypertable_create_from_info` in
`ts_hypertable_create_internal` and aborting with an error if the
hypertable already exists.
Fixes#1987
The `drop_chunks` function is refactored to make table name mandatory
for the function. As a result, the function was also refactored to
accept the `regclass` type instead of table name plus schema name and
the parameters were reordered to match the order for `show_chunks`.
The commit also refactor the code to pass the hypertable structure
between internal functions rather than the hypertable relid and moving
error checks to the PostgreSQL function. This allow the internal
functions to avoid some lookups and use the information in the
structure directly and also give errors earlier instead of first
dropping chunks and then error and roll back the transaction.
Since we want to run additional test configurations when triggered
by a push to prerelease_test or by cron but github actions don't
allow a dynamic matrix via yaml configuration we generate the matrix
with a python script. While we could always have the full matrix
and have if checks on every step that would make the actual checks
harder to browse because the workflow will have lots of entries and
only by navigating into the individual jobs would t sit be visible
if a job was actually run.
Additionally we set fail-fast to true for pull requests and false
for other event types.
The async library for remote connections has multiple ways to deal
with errors. For instance, errors can both be thrown internally and be
returned as port of an `AsyncResult`. In other cases, it is possible
to pass in an "elevel" to avoid throwing errors, but this only works
for some functions and not for all errors.
This change tries to make error handling more consistent by avoiding
throwing errors while waiting for results, instead returning any error
as `AsyncResponseError`. This makes error handling more predictible
and the caller has to explicitly check if the returned response is an
error or valid result without having to try-catch any errors.
A number of issues with potentially leaking PQresult:s have also been
fixed.
Since the connection cache is no longer replaced on a transaction
rollback, it is not necessary to pin the connection cache (this wasn't
done correctly in some places in any case, e.g.,
`data_node_get_connection`).
This change refactors how connections are handled during remote
transactions. In particular, the connection cache now stays consistent
during transactions, even during rollbacks. Previously, the connection
cache was replaced on every rollback, even if the rollback was
intentional (i.e, not due to an error). This made it hard to debug
connections since the cache became completely empty.
Connections could also be left in the cache in a bad state after
failed transactions. This has been fixed by moving connection checks
to the cache and tying transaction state changes to each
connection. This ensures that such checks are done in one canonical
place instead of being spread out throughout the code.
Given how tightly coupled a remote transaction is with its connection,
it might make sense to remove the separate remote transaction store
and instead put this information in each connection. This is left to a
future change, however.
In addition to the above changes, this commit includes:
* Showing transaction depth and invalidation in the transaction store
* Invalidation on individual connections instead of replacing the
whole cache
* Closing of connections to a local database that is being dropped to
prevent "in use" errors.
* Ability to add callbacks to async requests that are executed when a
response is received. This is used by remote transactions to mark
connections as having successfully completed a transaction. Thus, on
errors, it is easy to detect connections that are in bad states.
* Error checks on each connection instead of having global error
tracking for each remote transaction. This change removes the global
error state for distributed transactions.
The connection cache for remote transactions can now be examined using
a function that shows all connections in the cache. This allows easier
debugging and validation both in tests and on live systems.
In particular, we'd like to know that connections are in good state
post commit or rollback and that we don't leave bad connections in the
cache.
The remote transaction test (`remote_txn`) has been updated to show
the connection cache as remote transactions are
executed. Unfortunately, the whole cache is replaced on every
(sub-)transaction rollback, which makes it hard to debug the
connection state of a particular remote transaction. Further, some
connections are left in the cache in a bad state after, e.g.,
connection loss.
These issues will be fixed with an upcoming change.
If `bits_used` is not exactly 64, a shift will be attempted, even when
`bits_used > 64`. According to the standard "[...] if the value of the
right operand is negative or is greater or equal to the number of bits
in the promoted left operand, the behavior is undefined."
Hence we change the code to return `PG_UINT64_MAX` if a request to
shift more bits than the number of bits in the type is requested,
otherwise we perform the shift.
If telemetry is disabled, it is checked inside the `ts_telemetry_main`
function, which starts to execute after a background worker has been
scheduled to execute the function. This means that any errors occurring
when starting the job will trigger errors and generate lines in the
log.
This commit moves the check to before a background worker is scheduled
for executing the job and allow the telemetry job to trivially succeed
without actually invoking the worker, hence no errors will be
generated for the telemetry job if it is disabled.
Also, the privacy test result changes and it is updated to check that
the telemetry job cannot leak information regardless of whether
telemetry is on or off.
Fixes#1934#1788
This patch adjusts the export prefix check to exclude some symbols
present when using older toolchain and also includes gcov symbols
so export prefix check can be used on coverage enabled builds.
This patch adds a slack notification on failed i386 tests.
Unfortunately most slack actions don't work with the 32bit
image so our options for better styling/costumisation are
somewhat limited.
GCC 7.5 warns that invalidation_range in continuous aggregate
materialization may be used uninitialized.
Older clang versons warn that res in calculate_next_start_on_failure
may be used uninitialized.
This patch changes the code to always initialize those two variables.
This release adds new functionality on distributed hypertables,
including (but not limited to) basic LIMIT pushdown, manual chunk
compression, table access methods storage options, SERIAL columns,
and altering of the replication factor.
This release only supports PG11 and PG12. Thus, PG9.6 and PG10
are no longer supported.
Note that the 2.0 major release will introduce breaking changes
to user functions and APIs. In particular, this beta removes the
cascade parameter from drop_chunks and changes the names of
certain GUC parameters. Expect additional breaking changes to be
introduced up until the 2.0 release.
**For beta releases**, upgrading from an earlier version of the
extension (including previous beta releases) is not supported.
**Features**
* #1877 Add support for fast pruning of inlined functions
* #1922 Cleanup GUC names
* #1923 Add repartition option on detach/delete_data_node
* #1923 Allow ALTER TABLE SET on distributed hypertable
* #1923 Allow SERIAL columns for distributed hypertables
* #1923 Basic LIMIT push down support
* #1923 Implement altering replication factor
* #1923 Support compression on distributed hypertables
* #1923 Support storage options for distributed hypertables
* #1941 Change default prefix for distributed tables
* #1943 Support table access methods for distributed hypertables
* #1952 Remove cascade option from drop_chunks
* #1955 Remove support for PG9.6 and PG10
**Bugfixes**
* #1915 Check for database in extension_current_state
* #1918 Unify chunk index creation
* #1923 Fix insert batch size calculation for prepared statements
* #1923 Fix port conversion issue in add_data_node
* #1932 Change compression locking order
* #1938 Fix gapfill locf treat_null_as_missing
**Thanks**
* @dmitri191 for reporting an issue with failing background workers
* @fvannee for optimizing pruning of inlined functions
* @nbouscal for reporting an issue with compression jobs locking referenced tables
* @nicolai6120 for reporting an issue with locf and treat_null_as_missing
* @nomanor for reporting an issue with expression index with table references
If a hypertable is created with an index on it and a continuous
aggregate is further defined on the hypertable, it will create an
internal dependency between the chunks of the hypertable and the chunks
of the continuous aggregate.
When dropping chunks with `cascade_to_materialization` set to `FALSE`,
this will generate an error since the delete is not cascaded to the
internal dependencies.
This commit fixes this by collecting the internal dependencies and
using `performMultipleDelete` rather than `performDelete` to delete
several objects as one operation.
Fixes#1889
Enabling clang-tidy on builds not using clang compiler will not
compile due to incompatible compiler flags, so this patch
changes the current behaviour to only enable clang-tidy when
using clang compiler. You can still overwrite the behaviour
by passing -DLINTER=ON when not using clang.
This patch changes the update script generation to not use
scratch files and removes the sql fragments to set and unset
the post_update_stage from CMakeLists.txt and puts them into
dedicated files.
Inlineable functions used to be slow to plan, because the query
preprocessing function could not find the relations inside the
functions, as they haven't been inlined yet at that point.
This commit adds a separate check in the get_relation_info_hook
to optimize pruning of hypertables.
When using time_bucket_gapfill with the treat_null_as_missing
option we did not properly set the number of valid values when
generating our own virtual tuples leading to "cannot extract
attribute from empty tuple slot" when the number of values got
reset. This patch changes the gapfill code to always set
the number of valid values when generating virtual tuples.
This patch removes code support for PG9.6 and PG10. In addition to
removing PG96 and PG10 macros the following changes are done:
remove HAVE_INT64_TIMESTAMP since this is always true on PG10+
remove PG_VERSION_SUPPORTS_MULTINODE
Replace EXECUTE PROCEDURE with EXECUTE FUNCTION because the former
is deprecated in PG11+. Unfortunately some test output will still
have EXECUTE PROCEDURE because pg_get_triggerdef in PG11 still
generates a definition with EXECUTE PROCEDURE.
This commit removes the `cascade` option from the function
`drop_chunks` and `add_drop_chunk_policy`, which will now never cascade
drops to dependent objects. The tests are fixed accordingly and
verbosity turned up to ensure that the dependent objects are printed in
the error details.
