This change implements deduplication and merging of invalidation
entries for continuous aggregates in order to reduce the number of
reduntant entries in the continuous aggregate invalidation
log. Merging is done both when copying over entries from the
hypertable to the continuous aggregate invalidation log and when
cutting already existing invalidations in the latter log. Doing this
merging in both steps helps reduce the number of invalidations also
for the continuous aggregates that don't get refreshed by the active
refresh command.
Merging works by scanning invalidations in order of the lowest
modified value, and given this ordering it is possible to merge the
current and next entry into one large entry if they are
overlapping. This can continue until the current and next invalidation
are disjoint or there are no more invalidations to process.
Note, however, that only the continuous aggregate that gets refreshed
will be fully deduplicated. Some redundant entries might exist for
other aggregates since their entries in the continuous aggregate log
aren't cut against the refresh window.
Full deduplication for the refreshed continuous aggregate is only
possible if the continuous aggregate invalidation log is processed
last, since that also includes "old" entries. Therefore, this change
also changes the ordering of how the logs are processed. This also
makes it possible to process the hypertable invalidation log in the
first transaction of the refresh.
This patch separates the ordered append join tests from the other
ordered append tests. It also adds additional constraints to some
queries to speed them up. These changes result in a 3x speedup for
regresscheck-shared.
The invalidation threshold governs the window of data from the head of
a hypertable that shouldn't be subject to invalidations in order to
reduce write amplification during inserts on the hypertable.
When a continuous aggregate is refreshed, the invalidation threshold
must be moved forward (or initialized if it doesn't previously exist)
whenever the refresh window stretches beyond the current threshold.
Tests for setting the invalidation threshold are also added, including
new isolation tests for concurrency.
When a continuous aggregate is refreshed, it also needs to move the
invalidation threshold in case the refresh window stretches beyond the
current threshold. The new invalidation threshold must be set in its
own transaction during the refresh, which can only be done if the
refresh command is a procedure.
The calculation of the max-size refresh window for integer-based
continuous aggregates used the range of 64-bit integers for all
integer types, while the max ranges for 16- and 32-bit integers are
lower. This change adds the missing range boundaries.
This patch adds policies to the update test to ensure their
configuration is properly migrated during updates. This patch
also fixes the inconsistent background job application_name
and adjusts them in the update script.
This patch changes the update test to run on PG 11.8 and 12.3 instead
of 11.0 and 12.0. This patch also adds additional diagnostic output
when errors occur during the update test.
This patch changes the application name for background worker jobs
to include the job_id which makes the application name unique and
allows joining against pg_stat_activity to get a list of currently
running background worker processes. This change also makes
identifying misbehaving jobs easier from the postgres log as the
application name can be included in the log line.
The with_clause_parser and continuous_aggs_drop_chunks tests were
not referenced in the CMakeLists leading to those tests never being
run. This patch adds them to the appropriate file and adjusts the
output.
This patch changes the telemetry code to always send the installed_time
timestamp AS ISO8601. Previously it was depending on local settings
leading to timestamps not processable by the telemetry receiver.
This patch changes how the scheduler handles memory contexts.
Previously only memory allocated during transactions would get
freed and everything else remained allocated.
The scheduler now uses 2 memory contexts for its operation: scheduler_mctx
for long-lived objects and scratch_mctx for short-lived objects.
After every iteration of the scheduling main loop scratch_mctx gets
reset. Special care needs to be taken in regards to memory contexts
since StartTransactionCommand creates and switches to a transaction
memory context which gets deleted on CommitTransactionCommand which
switches CurrentMemoryContext back to TopMemoryContext. So operations
wrapped in Start/CommitTransactionCommit will not happen in scratch_mctx
but will get freed on CommitTransactionCommand.
This change adds intitial support for invalidation processing when
refreshing a continuous aggregate. Note that, currently, invalidations
are only cleared during a refresh, but not yet used to optimize
refreshes. There are two steps to this processing:
1. Invalidations are moved from hypertable invalidation log to the
cagg invalidation log
2. The cagg invalidation entries are then processed for the continuous
aggregate that gets refreshed.
The second step involves finding all invalidations that overlap with
the given refresh window and then either deleting them or cutting
them, depending on how they overlap.
Currently, the "invalidation threshold" is not moved up during a
refresh. This would only be required if the refresh window crosses
that threshold and will be addressed in a future change.
This change adds the ability to set a snapshot to use with scans
executed with the Scanner module. The Scanner uses SnapshotSelf by
default, but this isn't appropriate for certain scans that, e.g.,
don't want to see their own changes. An option to keep the lock
on the scanned relation after the scan is also added.
To drop a continuous aggregate it was necessary to use the `CASCADE`
keyword, which would then cascade to the materialized hypertable. Since
this can cascade the drop to other objects that are dependent on the
continuous aggregate, this could accidentally drop more objects than
intended.
This commit fixes this by removing the check for `CASCADE` and adding
the materialized hypertable to the list of objects to drop.
Fixestimescale/timescaledb-private#659
A recent change changed the macos build to run in release mode
which also changed postgres to be built without assertions.
Since we inherit the assertion setting from postgres this leads
to assertions being disabled for our code as well.
With assertions disabled clang errors on detecting null pointer
dereferences so this patch turns assertions for macos back on.
Since the postgres build is cached this took not effect immediately
and remained unnoticed in the CI run against the PR introducing the
change.
This patch changes pg_regress to sort the test list when tests
are controlled with either TESTS or SKIPS. This makes it more
consistent with the unfiltered test run which gets a sorted
list from cmake.
Since the minimum version we can update from on PG11 is 1.1.0 we
can get rid of all the previous versions update files as they
are not a valid update source for any current version.
The parameter `cascade_to_materialization` is removed from
`drop_chunks` and `add_drop_chunks_policy` as well as associated tables
and test functions.
Fixes#2137
If the access node is adding itself as a data node using `add_data_node`
it will deadlock since transactions will be opened on both the access
node and data node both trying to update the metadata.
This commit fixes this by updating `set_dist_id` to check if the UUID
being added as `dist_uuid` is the same as the `uuid` of the node. If
that is the case, it raises an error.
Fixes#2133
This change replaces the add_drop_chunks_policy function with
add_retention_policy. This also renames the older_than parameter
of that function as retention_window. Likewise, the
remove_drop_chunks_policy is also being renamed
remove_retention_policy.
Fixes#2119
Adds a test to call detach_tablespaces on a distributed hypertable.
Since no tablespaces can be attached to distributed hyperatbles, the
test detaches 0 tablespaces. Also a test to detach tablespaces on a
data node is added.
This change adds an new isolation test for concurrent refreshing on a
continuous aggregate. Although a bucket (group) in a continuous
aggregate should be unique on the GROUP BY columns there is no unique
constraint on the materialized hypertable to protect against duplicate
buckets. Therefore, concurrent refreshes can result in duplicate rows
in the materialized hypertable although such duplicates should not be
possible by the underlying query's definition.
This change adds a new refresh function called
`refresh_continuous_aggregate` that allows refreshing a continuous
aggregate over a given window of data, called the "refresh window".
This is the first step in a larger overhaul of the continuous
aggregate feature with the goal of cleaning up the API and separating
policy from the core functionality.
Currently, the refresh function does a brute-force refresh of a window
and it bypasses the whole invalidation framework. Future updates
intend to integrate with this framework (with modifications) to
optimize refreshes. An exclusive lock is take on the continuous
aggregate's internal materialized hypertable in order to protect
against concurrent refreshing. However, as this serializes refreshes,
we might want to relax this locking in the future to allow, e.g.,
concurrent refreshes of non-overlapping windows.
The new refresh functionality includes basic tests for bad input and
refreshing across different windows. Unfortunately, a bug in the
optimization code for `time_bucket` causes timestamps to overflow the
allowed MAX time. Therefore, refresh windows that are close to the MAX
allowed size are not yet supported or tested.
A function to lookup the name of a chunk constraint returned a pointer
to string without first copying the string into a safe memory
context. This probably worked by chance because everything in the scan
function ran in the current memory context, including the deforming of
the tuple. However, returning pointers to data in deformed tuples can
easily cause memory corruption with the introduction of other changes
(such as improved memory management).
This memory issue is fixed by explicitly reallocating the string in
the memory context that should be used for any returned data.
Changes are also made to avoid unnecessarily deforming tuples multiple
times in the same scan function.
This change makes the Scanner code agnostic to the underlying storage
implementation of the tables it scans. This also fixes a bug that made
it impossible to use non-heap table access methods on a
hypertable. The bug existed because a check is made for existing data
before a table is made into a hypertable. And, since this check reads
data from the table using the Scanner, it must be able to read the
data irrespective of the underlying storage.
As a result of the more generic scan interface, resource management is
also improved by delivering tuples in reference-counted tuple table
slots. A backwards-compatibility layer is used for PG11, which maps
all table access functions to the heap equivalents.
Whenever chunks are created, no privileges are added to the chunks.
For accesses that go through the hypertable permission checks are
ignored so reads and writes will succeed anyway. However, for direct
accesses to the chunks, permission checks are done, which creates
problems for, e.g., `pg_dump`.
This commit fixes this by propagating `GRANT` and `REVOKE` statements
to the chunks when executed on the hypertable, and whenever new chunks
are created, privileges are copied from the hypertable.
This commit do not propagate privileges for distributed hypertables,
this is in a separate commit.
