This patch adds two new fields to the telemetry report,
`stats_by_job_type` and `errors_by_sqlerrcode`. Both report results
grouped by job type (different types of policies or
user defined action).
The patch also adds a new field to the `bgw_job_stats` table,
`total_duration_errors` to separate the duration of the failed runs
from the duration of successful ones.
Postgres source code define the macro `OidIsValid()` to check if the Oid
is valid or not (comparing against the `InvalidOid` type). See
`src/include/c.h` in Postgres source three.
Changed all direct comparisons against `InvalidOid` for the `OidIsValid`
call and add a coccinelle check to make sure the future changes will use
it correctly.
Trying to resume a failed Continuous Aggregate raise an exception that
the migration plan already exists, but this is wrong and the expected
behaviour should be resume the migration and continue from the last
failed step.
This change introduces a new option to the compression procedure which
decouples the uncompressed chunk interval from the compressed chunk
interval. It does this by allowing multiple uncompressed chunks into one
compressed chunk as part of the compression procedure. The main use-case
is to allow much smaller uncompressed chunks than compressed ones. This
has several advantages:
- Reduce the size of btrees on uncompressed data (thus allowing faster
inserts because those indexes are memory-resident).
- Decrease disk-space usage for uncompressed data.
- Reduce number of chunks over historical data.
From a UX point of view, we simple add a compression with clause option
`compress_chunk_time_interval`. The user should set that according to
their needs for constraint exclusion over historical data. Ideally, it
should be a multiple of the uncompressed chunk interval and so we throw
a warning if it is not.
Version 15 pg_dump program does not log any messages with log level <
PG_LOG_WARNING to stdout. Whereas this check is not present in version
14, thus we see corresponding tests fail with missing log information.
This patch fixes by supressing those log information, so that the tests
pass on all versions of postgresql.
Fixes#4832
Look for the binary with exact version before looking for the
generic name to prevent failure when clang-format is lower then
required version but clang-format-14 exists.
After data is tiered using OSM, we cannot insert data into the same
range. Need a callback that can be invoked by timescaledb to check
for range overlaps before creating a new chunk
It changed the type of Var.varno from Index to int. I'm starting to
wonder if it was a good idea to enable this warning, but maybe we can
give it the last try.
Get rid of `GETSTRUCT` to fill the form data and use `heap_deform_tuple`
instead. This is necessary specially when you have variable lenght
fields and/or fields that accept NULL values. This refactoring will be
specially usefull in a following PR for Nested Continuous Aggregates
where we'll add a new metadata to the catalog that can accept NULL
values.
Also refactor the rename view and schema code paths improving the
readability and maintainability.
The PR #3899 introduced a new function named `tsl_cagg_try_repair` to
try to fix buggy Continuous Aggregates that lead to segfault on a select
query. It added an INFO message when skipping the check for Continuous
Aggregate that don't have partials and it's annoying during the
upgrade/downgrade the extension specially if you have many Continuous
Aggregate.
Remove this message because it's completely useless.
The telemetry code that reads the postgres version was not updated
when support for older postgres version was dropped so we introduce
a new macro PG_MAJOR_MIN which is the oldest pg major version we
support.
This PR adds a Coccinelle test for ereport(..) calls that use the PG
12.3+ syntax
(postgres/postgres@a867154516). We had
some of these calls in the past, which we had to fix afterward (
see #4733, #4809, #4871). This Coccinelle patch detects such calls and
reports them in the CI run.
Previous commit #4664 introduced the ability to execute
background jobs on a fixed schedule.
This commit updates our telemetry data to include the number
of jobs scheduled to execute on a fixed schedule vs the number
registered to execute on a drifting schedule.
The changes from e555eea lead to flakiness. They are a leftover of
earlier version and probably not needed anymore.
The original version is also still flaky on Windows, so use linear
regression to tell if the memory usage is increasing.
Verified to still fail on 2.7.x
To avoid untrusted logged data to use `set-state` and `set-output`
workflow commands without the intention of the workflow author GitHub
have introduced a new set of environment files to manage state and
output.
This commit changes the existing uses of `set-output` to use the new
environment files instead.
See https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
A new health check function _timescaledb_internal.health() returns the
health and status of the database instance, including any configured
data nodes (in case the instance is an access node).
Since the function returns also the health of the data nodes, it tries
hard to avoid throwing errors. An error will fail the whole function
and therefore not return any node statuses, although some of the nodes
might be healthy.
The health check on the data nodes is a recursive (remote) call to the
same function on those nodes. Unfortunately, the check will fail with
an error if a connection cannot be established to a node (or an error
occurs on the connection), which means the whole function call will
fail. This will be addressed in a future change by returning the error
in the function result instead.
Since e555eea9dbc05f4c09cf0d7e23b814054a459d19 the function
TSCopyMultiInsertInfoIsEmpty is no longer used. This patch removes the
unused code from src/copy.c.
In 8375b9aa536a619a5ac2644e0dae3c25880a4ead, a patch was added to handle
chunks closes during an ongoing copy operation. However, this patch
introduces a performance regression. All MultiInsertBuffers are deleted
after they are flushed. In this PR, the performance regression is fixed.
The most commonly used MultiInsertBuffers survive flushing.
The 51259b31c4c62b87228b059af0bbf28caa143eb3 commit changes the way the
per-tuple context is used. Since this commit, more objects are stored in
this context. The size of the context was used to set the tuple size to
PG < 14. The extra objects in the context lead to wrong (very large)
results and flushes almost after every tuple read.
The cache synchronization introduced in
296601b1d7aba7f23aea3d47c617e2d6df81de3e is reverted. With the current
implementation, `MAX_PARTITION_BUFFERS` survive the flash. If
`timescaledb.max_open_chunks_per_insert` is lower than
`MAX_PARTITION_BUFFERS` , a buffer flush would be performed after each
tuple read.
Dimension partition lookups use binary search to find the partition to
place a chunk in. However, in the code, an array of partitions might
not be sorted because the sort happened on a copy of the array instead
of the main array. This change fixes the issue to ensure the array is
sorted and binary search works properly.
For some unexpected conditions, we have a check and an error that is
generated. Since this always generate an error, it is more difficult to
find the bug if the error is generated rather than an assert fired
generating a core dump. Similarly, some asserts can occur in production
builds and will lead to strange situations triggering a crash. For
those cases we should instead generate an error.
This commit introduces a macro `Ensure` that will result in an assert
in debug builds, but an error message in release build. This macro
should only be used for conditions that should not occur during normal
runtime, but which can happen is odd corner-cases in release builds and
therefore warrants an error message.
It also replaces some existing checks with such errors to demonstrate
usage.
External components like timescaledb_cloudutils might want to add
additional options or do additional ssl related processing. They can
do so by implementing a hook and then assigning it to a timescaledb
variable to allow timescale to invoke it as appropriate.
When TimescaleDB 2.9.0 compiled against PG15, many tests fail due to
permissions on default public schema not getting propagated to data
nodes, although connected user has required GRANTS on access nodes.
This patch fixes failing tests by explicitly granting required permissions
for connected user or role after data nodes are added.
Currently, the next start of a scheduled background job is
calculated by adding the `schedule_interval` to its finish
time. This does not allow scheduling jobs to execute at fixed
times, as the next execution is "shifted" by the job duration.
This commit introduces the option to execute a job on a fixed
schedule instead. Users are expected to provide an initial_start
parameter on which subsequent job executions are aligned. The next
start is calculated by computing the next time_bucket of the finish
time with initial_start origin.
An `initial_start` parameter is added to the compression, retention,
reorder and continuous aggregate `add_policy` signatures. By passing
that upon policy creation users indicate the policy will execute on
a fixed schedule, or drifting schedule if `initial_start` is not
provided.
To allow users to pick a drifting schedule when registering a UDA,
an additional parameter `fixed_schedule` is added to `add_job`
to allow users to specify the old behavior by setting it to false.
Additionally, an optional TEXT parameter, `timezone`, is added to both
add_job and add_policy signatures, to address the 1-hour shift in
execution time caused by DST switches. As internally the next start of
a fixed schedule job is calculated using time_bucket, the timezone
parameter allows using timezone-aware buckets to calculate
the next start.
Timescale 2.7 released a new version of Continuous Aggregate (#4269)
that store the final aggregation state instead of the byte array of
the partial aggregate state, offering multiple opportunities of
optimizations as well a more compact form.
This new version also removes the unecessary `chunk_id` column from the
materialization hypertable and consequently the re-aggregation in the
user view. It means the user view that query the materialization
hypertable don't have a GROUP BY clause anymore that was problematic
for query performance.
Before 2.7 when users turn compression ON we infer compression options
`segmentby` and `orderby` based on the GROUP BY clause and time bucket
respectively. With the new version without a GROUP BY clause in the user
view the inferetion for the 'segmentby' compression option stopped to
work.
Fixed it by changing the code to the compression on the new version of
Continuous Aggregate (aka finals form) behave the same as the old
version.
Fix#4816
