This change fixes a number of typos and issues with inconsistent
formatting for compression-related code. A couple of other fixes for
variable names, etc. have also been applied.
The gapfill test assumed 1 day is always <= 24 hours which is not
true during DST switch leading to a failing test when run in that
time. This PR fixes the test to have reproducable output even
when run during DST switch.
There was a race condition between the post_restore function
restarting the background worker and the setting of the
restoring flag to "off". If the worker started before the
change to the restoring flag had been committed, it would not
see the change and then die because the worker should exit
when the db is in a restoring state. This modifies the
post_restore function to use a restart instead of a start
so that it waits on the commit to start up. It also adds
logic to the entrypoint to reload config changes caused
by an `ALTER DATABASE SET` command. These changes are
normally only seen at connection startup but we have to
wait until after our lock on the modifying transaction is
released to know whether we should adopt them.
When a MergeAppendPath has children that do not produce sorted
output a Sort node will be injected during plan creation, those
plans would trigger an error about invalid child nodes in
ConstraintAwareAppend. This PR makes ConstraintAwareAppend handle
those plans correctly.
When we copy the invalidation logs for individual continuous
aggregates, the lowest-value was globally overwritten. Fix this
so that the change is specific to a continuous aggregate.
This bug could result in missing invalidations.
This PR improves the scheduling of jobs when the number of
jobs exceeds the amount of background workers. Previously,
this was not a case the scheduler handled well.
The basic strategy we employ to handle this case better is to
use a job's next_start field to create a priority for jobs.
More concretely, jobs are scheduled in increasing order of
next_start. If the scheduler runs out of bgw's it waits to
until bgws become available and then retries again, also
in increasing next_start order.
The first change this PR implements is start jobs in order
of increasing next_start. We also make sure that if we run
out of BGWs, the scheduler will try again in START_RETRY_MS
(1 second by default).
This PR also needed to change the logic of what happens when
a job fails to start because BGWs have run out. Previously,
such jobs were marked as failed and their next_start was reset
using the regular post-failure backoff logic. But, this means
that a job looses its priority every time we run out of BGWs.
Thus, we changed this logic so that next_start does not change
when we encounter this situation.
There are actually 2 ways to run out of BGWs:
1) We run out of the timescale limit on BGWs - in this case
the job is simply put back into the scheduled state, and it
will be retried in START_RETRY_MS. The job is not marked
started or failed. This is the common error.
2) We run out of PostgreSQL workers. We won't know if this
failed until we try to start the worker, by which time the
job must be in the started state. Thus if we run into this
error we must mark the job as failed. But we don't change
next_start. To do this we create a new job result type
called JOB_FAILURE_TO_START.
When the index for a chunk was created the attnos for the index
predicate were not adjusted leading to insert errors on hypertables
with dropped columns that had indexes with predicates.
This PR adjust index predicate attnos when creating the chunk index
to match the chunk attno.
Function `timescaledb_CopyFrom` created a variable `errcallback` to
save away the previous error callback and restoring it afterwards.
However, if `ccstate->cstate` was false, the variable would not be set
and the later restore of the error callback would use a random value.
This commit fixes the issue by initializing `errcallback` to all zeroes
and checking if a callback has been saved away before restoring it.
This change will check sql commands which start a background worker
on a hypertable to verify that the table owner has permission to
log into the database. This is necessary, as background workers for
these commands will run with the permissions of the table owner, and
thus immediately fail if unable to log in.
When trying to compress a chunk that had a column of datatype
interval delta-delta compression would be selected for the column
but our delta-delta compression does not support interval and
would throw an errow when trying to compress a chunk.
This PR changes the compression selected for interval to dictionary
compression.
To make tests more stable and to remove some repeated code in the
tests this PR changes the test runner to stop background workers.
Individual tests that need background workers can still start them
and this PR will only stop background workers for the initial database
for the test, behaviour for additional databases created during the
tests will not change.
The log level used for continuous aggregate materialization messages
was INFO which is for requested information. Since there is no way to
control the behaviour externally INFO is a suboptimal choice because
INFO messages cannot be easily suppressed leading to irreproducable
test output. Even though time can be mocked to make output consistent
this is only available in debug builds.
This patch changes the log level of those messages to LOG, so
clients can easily control the ouput by setting client_min_messages.
Error message used to specify that interval must be defined in terms
of days or smaller, which was confusing because we really meant any
fixed interval (e.g., weeks, days, hours, minutes, etc.), but not an
interval that is not of fixed duration (e.g., months or years).
Direct function calls to PG functions can throw
errors for bad inputs. If these are not handled
when called by the scheduler, the process dies
and background workers cannot be scheduled.
This change fixes the `plan_expand_hypertable` test, which was broken
and never ran the output comparison due to prematurely ending in an
uncaught error. The test appeared to succeeded, however, since also
the broken test "expected" files were committed to the repo.
Fixing the test revealed that the query output with our optimizations
enabled is incorrect for outer joins (i.e., the output from the query
differed from regular PostgreSQL). Restriction clauses were too
aggressively pushed down to outer relations, leading to chunk
exclusion when it shouldn't happen. This incorrect behavior has also
been fixed.
REFRESH MATERIALIZED VIEW statements sometimes crash
because the generated plan has a HashAgg node
instead of a Partial Aggregate node when executing
INSERT INTO <materialization_table>
SELECT * FROM <view>
Where the view stmt itself is along these lines
SELECT time_bucket('1 day'::interval, cpu."time") ,
......
_timescaledb_internal.partialize_agg(avg(...))
FROM <table>
As of this change, a number of enterprise features are accessible to
community users. These features include:
* reorder
* policies around reorder and drop chunks
The move chunks feature is still covered by enterprise. Gapfill no
longer warns about expired enterprise license.
Tests have been updated to reflect these changes.
This commit updates the README with updated resources, including Timescale
Cloud, new support options, release notes, and others. For readability, it
also moves instructions for building from source to a separate readme file.
Fix order by queries on compressed hypertables that
have char segment by column.
The segment by var column for decompressed chunks should be
created after setting the typmod and collation ids. Otherwise, we
get failures with char datatypes while decompressing.
Fixes#1650
When doing a migrate of table data to a hypertable during a call of
`create_hypertable`, any error (including an interrupt signal) would
cause the signal handling to call `CopyFromErrorCallback` which would
read garbage data because a heap scan descriptor is read as if it was a
`CopyState` structure.
This commit fixes that by not adding a new context to the error context
stack when migrating data, which will ensure that the error context
strack does not contain garbage data.
Fixes#1651
The GapFill node was not fully reset on a ReScan, so if there was a
GapFill within a NestedLoop, only the first iteration would return
results. This commit fixes this issues.
Refactors multiple implementations of finding hypertables in cache
and failing with different error messages if not found. The
implementations are replaced with calling functions, which encapsulate
a single error message. This provides the unified error message and
removes need for copy-paste.
When drop_chunks is called with cascade_to_materialization = true,
the materialized data is deleted from the materialization
hypertable, but the chunks are not dropped. This fix drops chunks
if possible and deletes the data only if the materialized chunk
cannot be dropped (which is the case if the materialzied chunk
contains data from multiple raw chunks and some of the raw chunks
are not dropped).
Fixes#1644
TEST_SPINWAIT_ITERS is used as maximum number of iterations to wait
for background worker results. Each iteration has a 0.1 second sleep
so a setting of 10 makes the test wait 1 second for the desired result.
This PR changes the iterations to 100 to wait at most 10 seconds
which should help with some flaky bgw tests.
Postgres CFLAGS includes -Wdeclaration-after-statement which leads
to problems when compiling with -Werror since we aim for C99 and allow
that so we strip this flag from PG_CFLAGS before adding postgres flags
to our own
When using `COPY TO` on a hypertable (which copies from the hypertable
to some other destination), nothing will be printed and nothing will be
copied. Since this can be potentially confusing for users, this commit
print a notice when an attempt is made to copy from a hypertable
directly (not using a query) to some other destination.
This release adds major new features and bugfixes since the 1.5.1 release.
We deem it moderate priority for upgrading.
The major new feature in this release allows users to keep the aggregated
data in a continuous aggregate while dropping the raw data with drop_chunks.
This allows users to save storage by keeping only the aggregates.
The semantics of the refresh_lag parameter for continuous aggregates has
been changed to be relative to the current timestamp instead of the maximum
value in the table. This change requires that an integer_now func be set on
hypertables with integer-based time columns to use continuous aggregates on
this table.
We added a timescaledb.ignore_invalidation_older_than parameter for continuous
aggregatess. This parameter accept a time-interval (e.g. 1 month). if set,
it limits the amount of time for which to process invalidation. Thus, if
timescaledb.ignore_invalidation_older_than = '1 month', then any modifications
for data older than 1 month from the current timestamp at modification time may
not cause continuous aggregate to be updated. This limits the amount of work
that a backfill can trigger. By default, all invalidations are processed.
**Major Features**
* #1589 Allow drop_chunks while keeping continuous aggregates
**Minor Features**
* #1568 Add ignore_invalidation_older_than option to continuous aggs
* #1575 Reorder group-by clause for continuous aggregates
* #1592 Improve continuous agg user messages
**Bugfixes**
* #1565 Fix partial select query for continuous aggregate
* #1591 Fix locf treat_null_as_missing option
* #1594 Fix error in compression constraint check
* #1603 Add join info to compressed chunk
* #1606 Fix constify params during runtime exclusion
* #1607 Delete compression policy when drop hypertable
* #1608 Add jobs to timescaledb_information.policy_stats
* #1609 Fix bug with parent table in decompression
* #1624 Fix drop_chunks for ApacheOnly
* #1632 Check for NULL before dereferencing variable
**Thanks**
* @optijon for reporting an issue with locf treat_null_as_missing option
* @acarrera42 for reporting an issue with constify params during runtime exclusion
* @ChristopherZellermann for reporting an issue with the compression constraint check
* @SimonDelamare for reporting an issue with joining hypertables with compression
The worker assignment for parallel queries behaves differently
from the 64 bit builds leading to differences in plan output
and failing tests. This commit ignores the results of the parallel
tests for 32 bit builds.
The drop_chunks function would call continuous_agg_drop_chunks_by_chunk_id
even for hypertables without continuous aggregates leading to license
errors when compiled as Apache-only. This PR skips the call when the
hypertable has no continuous aggregates.
Descending into subplans during constification of params
seems unsafe and has led to bugs. Turning this off seems
to be safe and not regress any tested optimizations.
In the future we may want to find a way to optimize this
case as well.
Fixes#1598.
This change fixes an assertion-based crash that happened when using
the `partialize_agg` function together with HAVING clauses. For
instance,
```
SELECT time_bucket('1 day', time), device,
__timescaledb_internal.partialize_agg(avg(temp))
GROUP BY 1, 2
HAVING avg(temp) > 3;
```
would crash because the HAVING clause's aggregate didn't have its
`Aggref` node set to partial aggregate mode.
Regular partial aggregations executed by the planner (i.e., those not
induced by the `partialize_agg` function) have their HAVING aggs
transparently moved to the target list during planning so that the
finalize node can use it when applying the final filter on the
resulting groups. However, it doesn't make much sense to transparently
do that when partializing with `partialize_agg` since it would be odd
to return more columns than requested by the user. Therefore, the
caller would have to do that manually. This, in fact, is also done
when materializing continuous aggregates.
For this reason, HAVING clauses with `partialize_agg` are blocked,
except in cases where the planner transparently reduces the HAVING
expression to a simple filter (e.g., `HAVING device > 3`).
Apart from fixing this issue, this change also refectors some of the
related code and adds tests for some error cases.
Previously we could have a dangling policy and job referring
to a now-dropped hypertable.
We also block changing the compression options if a policy exists.
Fixes#1570
The constraint check previously assumed that the col_meta
offset for a column was equal to that columns attribute
offset. This is incorrect in the presence of dropped columns.
Fixed to match on column names.
Fixes#1590
