Different num_chunks values reported by
timescaledb_information.hypertables and
timescaledb_information.chunks.
View definition of hypertables was
not filtering dropped and osm_chunks.
Fixes#5338
This patch backports following:
1. Refactor ExecInsert/Delete/Update
Backported commit 25e777cf8e547d7423d2e1e9da71f98b9414d59e
2. Backport all MERGE related interfaces and its implementations.
Backported commit 7103ebb7aae8ab8076b7e85f335ceb8fe799097c
The `bucket_info` variable is initialized by `caggtimebucketinfo_init`
function called inside the following branch:
`if (rte->relkind == RELKIND_RELATION || rte->relkind == RELKIND_VIEW)`
If for some reason we don't enter in this branch then the `bucket_info`
will not be initialized leading to an uninitialized variable when
returning `bucket_info` at the end of the `cagg_validate_query`
function.
Fixed it by initializing with zeros the `bucket_info` variable when
declaring it.
Found by coverity scan.
This small patch adds support for continuous aggregates to the
`hypertable_detailed_size` (and with that `hypertable_size`).
It adds an additional check to see if a continuous aggregate exists
if a hypertable with the given regclass name isn't found.
This patch adds the functionality that is needed to perform distributed,
parallel joins on reference tables on access nodes. This code allows the
pushdown of a join if:
* (1) The setting "ts_guc_enable_per_data_node_queries" is enabled
* (2) The outer relation is a distributed hypertable
* (3) The inner relation is marked as a reference table
* (4) The join is a left join or an inner join
This PR introduces a timeout argument and a new logic to the
timescale_internal.ping_data_node() function which allows
to handle io timeouts for nodes being unresponsive.
Fix#5312
NameData is a fixed-size type of 64 bytes. Using strlcpy to copy data
into a NameData struct can cause problems because any data that follows
the initial null-terminated string will also be part of the data.
While running TimescaleDB under valgrind I've found
two cases of incorrect memory handling.
Case 1: When creating timescaledb extension, during the insertion of
metadata there is some junk in memory that is not zeroed
before writing there.
Changes in metadata.c fix this.
Case 2: When executing GRANT smth ON ALL TABLES IN SCHEMA some_schema
and deconstructing this statement into granting to individual tables,
process of copying names of those tables is wrong.
Currently, you aren't copying the data itself, but an address to data
on a page in some buffer. There's a problem - when the page in this
buffer changes, copied address would lead to wrong data.
Changes in process_utility.c fix this by allocating memory and then
copying needed relname there.
Fixes#5311
The upgrade and downgrade tests are running PostgreSQL in Docker
containers. The function 'wait_for_pg' is used to determine if
PostgreSQL is ready to accept connections. In contrast to the upgrade
tests, the downgrade tests use more relaxed timeout values. The upgrade
tests sometimes fail because PostgreSQL cannot accept connections within
the configured time range. This patch applies the more relaxed timeout
values also to the upgrade script.
When executing functions, SPI assumes that `TopTransactionContext` is
used for atomic execution contexts and `PortalContext` is used for
non-atomic contexts. Since jobs need to be able to commit and start
transactions, they are executing in a non-atomic context hence
`PortalContext` will be used, but `PortalContext` is not set when
starting the job. This is not a problem for PL/PgSQL executor, but for
other executors (such as PL/Python) it would be.
This commit fixes the issue by setting the `PortalContext` variable to
the portal context created for the portal and restores it (to NULL)
after execution.
Fixes#5326
The content of bgw_main is copied into BackgroundWorker.
bgw_function_name, which has a length of BGW_MAXLEN. This patch ensures
that the size of these fields has the same length.
This release contains new features and bug fixes since the 2.9.3
release.
This release is high priority for upgrade. We strongly recommend that
you upgrade as soon as possible.
**Features**
* #5241 Allow RETURNING clause when inserting into compressed chunks
* #5245 Manage life-cycle of connections via memory contexts
* #5246 Make connection establishment interruptible
* #5253 Make data node command execution interruptible
* #5243 Enable real-time aggregation for continuous aggregates with
joins
* #5262 Extend enabling compression on a continuous aggregrate with
'compress_segmentby' and 'compress_orderby' parameters
**Bugfixes**
* #4926 Fix corruption when inserting into compressed chunks
* #5218 Add role-level security to job error log
* #5214 Fix use of prepared statement in async module
* #5290 Compression can't be enabled on continuous aggregates when
segmentby/orderby columns need quotation
* #5239 Fix next_start calculation for fixed schedules
To don't make developers confused the right name for Continuous
Aggregates on top of another Continuous Aggregates is `Hierarchical
Continuous Aggregates`, so changed the usage of term `nested` for
`hierarchical`.
Using `strlcpy` to copy variables holding PostgreSQL names can cause
issues since names are fixed-size types of length 64. This means that
any data that follows the initial null-terminated string will also be
part of the data.
Instead of using `const char*` for PostgreSQL names, use `NameData`
type for PostgreSQL names and use `namestrcpy` to copy them rather
than `strlcpy`.
As of upstream a73952b795 (16devel), custom GUCs have their initialized
values checked against the configured boot value. If they differ, an
assertion is tripped. So I've gone through the list and aligned the two
sides, adding defaults where they did not previously exist for
consistency.
To allow for dynamic boot values, the new assertion logic allows the
initialized value to be zero, in which case any boot value will be
accepted. ts_guc_max_open_chunks_per_insert relies on the value of
work_mem, so I've removed its (explicit) initializer entirely.
TextDatumGetCString() was made typesafe in upstream HEAD (16devel), so
now the compiler catches this. As Tom puts it in ac50f84866:
"TextDatumGetCString(PG_GETARG_TEXT_P(x))" is formally wrong: a text*
is not a Datum. Although this coding will accidentally fail to fail on
all known platforms, it risks leaking memory if a detoast step is needed,
unlike "TextDatumGetCString(PG_GETARG_DATUM(x))" which is what's used
elsewhere.
Just noticed abysmal INSERT performance when experimenting with one of
our customers' data set, and turns out my cache sizes were
misconfigured, leading to constant hypertable chunk cache thrashing.
Show a warning to detect this misconfiguration. Also use more generous
defaults, we're not supposed to run on a microwave (unlike Postgres).
We were using mismatched compiler and gcov, which led to either
segfaults or errors like "GCOV_TAG_COUNTER_ARCS mismatch". Add some
cmake code that tries to find the gcov that matches the compiler.
This should hopefully fix some of the mysterious missing coverage
problems that we've been experiencing for a while.
This patch fixes several issues with next_start calculation.
- Previously, the offset was added twice in some cases.
This is fixed by this patch.
- Additionally, schedule intervals with month components
were not handled correctly.
Internally, time_bucket with origin is used to calculate
the next start. However, in the case of month intervals, the
timestamp calculated for a bucket is always aligned on the first
day of the month, regardless of origin.
Therefore, previously the result was aligned with origin by adding
the difference between origin and its respective time bucket.
This difference was computed as a fixed length interval in terms
of days and time. That computation led to incorrect computation of
next start occasionally, for example when a job should be executed on
the last day of a month.
That is fixed by adding an appropriate interval of months to
initial_start and letting Postgres handle this computation properly.
Fixes#5216
The hypertable_compression test had on implicit reliance on the
ordering of tuples when querying the materialized results.
This patch makes the ordering explicit in this test.
1) Simplify the path generation for the parameterized data node scans.
1) Adjust the data node scan cost if it's an index scan, instead of always
treating it as a sequential scan.
1) Hard-code the grouping estimation for distributed hypertable, instead
of using the totally bogus per-column ndistinct value.
1) Add the GUC to disable parameterized
data node scan.
1) Add more tests.
This release contains bug fixes since the 2.9.2 release.
This release is high priority for upgrade. We strongly recommend that you
upgrade as soon as possible.
**Bugfixes**
* #4804 Skip bucketing when start or end of refresh job is null
* #5108 Fix column ordering in compressed table index not following the order of a multi-column segment by definition
* #5187 Don't enable clang-tidy by default
* #5255 Fix year not being considered as a multiple of day/month in hierarchical continuous aggregates
* #5259 Lock down search_path in SPI calls
A previous change accidentally broke the dist_hypertable test so that
it prematurely exited. This change restores the test so that it
executes properly.
The function to execute remote commands on data nodes used a blocking
libpq API that doesn't integrate with PostgreSQL interrupt handling,
making it impossible for a user or statement timeout to cancel a
remote command.
Refactor the remote command execution function to use a non-blocking
API and integrate with PostgreSQL signal handling via WaitEventSets.
Partial fix for #4958.
Refactor remote command execution function
A follow-up for the review comments in the previous PR.
1. Create one backport PR per one source PR, even with multiple commits.
1. Add a comment to the source PR if we fail to backport it for some
reason.
Previously all intervals were converted to seconds using "epoch"
with date_part. However, this treats a year as 365.25 days to
account for leap years, leading to the unexpected situation that
a year is not a multiple of a day or a month.
Fixed by treating month-only intervals as multiples of 30 days.
Fixes#5231
1. Find the latest release branch
1. For each commit in main and not in release branch (compared by
title), find the corresponding PR.
1. If the PR fixes an issue labeled "bug", and neither the PR nor the
issue are labeled "no-backport", cherry-pick the commits from the PR
onto the release branch, and create a PR with these changes.
We should not broadly make functions security definer as that increases
the attack surface for our extension. Especially for our telemetry we
should strive to only run it with the minimum required privileges.
For an unknown reason, pip started to install an older version of
prospector which is incompatible with the current pylint. Require the
new prospector version explicitly.
