Calling `ts_dist_cmd_invoke_on_data_nodes_using_search_path()` function
without an active transaction allows connection invalidation event
happen between applying `search_path` and the actual command
execution, which leads to an error.
This change introduces a way to ignore connection cache invalidations
using `remote_connection_cache_invalidation_ignore()` function.
This work is based on @nikkhils original fix and the problem research.
Fix#4022
The constify code constifying TIMESTAMPTZ expressions when doing
chunk exclusion did not account for daylight saving time switches
leading to different calculation outcomes when timezone changes.
This patch adds a 4 hour safety buffer to any such calculations.
The code added to support VIEWs did not account for the fact that
varno could be from a different nesting level and therefore not
be present in the current range table.
Allow planner chunk exclusion in subqueries. When we decicde on
whether a query may benefit from constifying now and encounter a
subquery peek into the subquery and check if the constraint
references a hypertable partitioning column.
Fixes#4524
This patch adjusts the operator logic for valid space dimension
constraints to no longer look for an exact match on both sides
of the operator but instead allow mismatched datatypes.
Previously a constraint like `col = value` would require `col`
and `value` to have matching datatype with this change `col` and
`value` can be different datatype as long as they have equality
operator in btree family.
Mismatching datatype can happen commonly when using int8 columns
and comparing them with integer literals. Integer literals default
to int4 so the datatypes would not match unless special care has
been taken in writing the constraints and therefore the optimization
would never apply in those cases.
Since we do not use our own hypertable expansion for SELECT FOR UPDATE
queries we need to make sure to add the extra information necessary to
get hashed space partitions with the native postgres inheritance
expansion working.
This patch adds a new time_bucket_gapfill function that
allows bucketing in a specific timezone.
You can gapfill with explicit timezone like so:
`SELECT time_bucket_gapfill('1 day', time, 'Europe/Berlin') ...`
Unfortunately this introduces an ambiguity with some previous
call variations when an untyped start/finish argument was passed
to the function. Some queries might need to be adjusted and either
explicitly name the positional argument or resolve the type ambiguity
by casting to the intended type.
This patch changes get_git_commit to always return the full hash.
Since different git versions do not agree on the length of the
abbreviated hash this made the length flaky. To make the length
consistent change it to always be the full hash.
When a query has multiple distributed hypertables the row-by-by
fetcher cannot be used. This patch changes the fetcher selection
logic to throw a better error message in those situations.
Previously the following error would be produced in those situations:
unexpected PQresult status 7 when starting COPY mode
The gapfill mechanism to detect an aggregation group change was
using datumIsEqual to compare the group values. datumIsEqual does
not detoast values so when one value is toasted and the other value
is not it will not return the correct result. This patch changes
the gapfill code to use the correct equal operator for the type
of the group column instead of datumIsEqual.
This patch fixes the param handling in prepared statements for generic
plans in ChunkAppend making those params usable in chunk exclusion.
Previously those params would not be resolved and therefore not used
for chunk exclusion.
Fixes#3719
When executing multinode queries that initialize row-by-row fetcher
but never execute it the node cleanup code would hit an assertion
checking the state of the fetcher. Found by sqlsmith.
The "empty" bytea value in a column of a distributed table when
selected was being returned as "null". The actual value on the
datanodes was being stored appropriately but just the return code path
was converting it into "null" on the AN. This has been handled via the
use of PQgetisnull() function now.
Fixes#3455
This patch transforms constraints on hash-based space partitions to make
them usable by postgres constraint exclusion.
If we have an equality condition on a space partitioning column, we add
a corresponding condition on get_partition_hash on this column. These
conditions match the constraints on chunks, so postgres' constraint
exclusion is able to use them and exclude the chunks.
The following transformations are done:
device_id = 1
becomes
((device_id = 1) AND (_timescaledb_internal.get_partition_hash(device_id) = 242423622))
s1 = ANY ('{s1_2,s1_2}'::text[])
becomes
((s1 = ANY ('{s1_2,s1_2}'::text[])) AND
(_timescaledb_internal.get_partition_hash(s1) = ANY ('{1583420735,1583420735}'::integer[])))
These transformations are not visible in EXPLAIN output as we remove
them again after hypertable expansion is done.
For certain inserts on a distributed hypertable, e.g., involving CTEs
and upserts, plans can be generated that weren't properly handled by
the DataNodeCopy and DataNodeDispatch execution nodes. In particular,
the nodes expect ChunkDispatch as a child node, but PostgreSQL can
sometimes insert a Result node above ChunkDispatch, causing the crash.
Further, behavioral changes in PG14 also caused the DataNodeCopy node
to sometimes wrongly believe a RETURNING clause was present. The check
for returning clauses has been updated to fix this issue.
Fixes#4339
Postgres knows whether a given aggregate is parallel-safe, and creates
parallel aggregation plans based on that. The `partialize_agg` is a
wrapper we use to perform partial aggregation on data nodes. It is a
pure function that produces serialized aggregation state as a result.
Being pure, it doesn't influence parallel safety. This means we don't
need to mark it parallel-unsafe to artificially disable the parallel
plans for partial aggregation. They will be chosen as usual based on
the parallel-safety of the underlying aggregate function.
When dealing with Intervals with month component timezone changes
can result in multiple day differences in the outcome of these
calculations due to different month lengths. When dealing with
months we add a 7 day safety buffer.
For all these calculations it is fine if we exclude less chunks
than strictly required for the operation, additional exclusion
with exact values will happen in the executor. But under no
circumstances must we exclude too much cause there would be
no way for the executor to get those chunks back.
The initial patch to use now() expressions during planner hypertable
expansion only supported intervals with no day or month component.
This patch adds support for intervals with day component.
If the interval has a day component then the calculation needs
to take into account daylight saving time switches and thereby a
day would not always be exactly 24 hours. We mitigate this by
adding a safety buffer to account for these dst switches when
dealing with intervals with day component. These calculations
will be repeated with exact values during execution.
Since dst switches seem to range between -1 and 2 hours we set
the safety buffer to 4 hours.
This patch also refactors the tests since the previous tests
made it hard to tell the feature was working after the constified
values have been removed from the plans.
Commit 35ea80ff added an optimization to enable expressions with
now() to be used during plan-time chunk exclusion by constifying
the now() expression. The added constified constraints were left
in the plan even though they were only required during the
hypertable explansion. This patch marks those constified constraints
and removes them once they are no longer required.
The table metrics_dist1 was only used by a single test and therefore
should not be part of shared_setup but instead be created in the
test that actually uses it. This reduces executed time of
regresscheck-shared when that test is not run.
This patch fixes the constify_now optimization to ignore Vars of
different level. Previously this could potentially lead to an
assertion failure cause the varno of that varno might be bigger
than the number of entries in the rangetable. Found by sqlsmith.
Commit 3b35da76 changed the setup script for regresscheck-shared
to no longer be usable directly by the sqlsmith workflow. This
patch set TEST_DBNAME at the top of the script so it is easier
to use the script outside of regression check environment.
This implements an optimization to allow now() expression to be
used during plan time chunk exclusions. Since now() is stable it
would not normally be considered for plan time chunk exclusion.
To enable this behaviour we convert `column > now()` expressions
into `column > const AND column > now()`. Assuming that time
always moves forward this is safe even for prepared statements.
This optimization works for SELECT, UPDATE and DELETE.
On hypertables with many chunks this can lead to a considerable
speedup for certain queries.
The following expressions are supported:
- column > now()
- column >= now()
- column > now() - Interval
- column > now() + Interval
- column >= now() - Interval
- column >= now() + Interval
Interval must not have a day or month component as those depend
on timezone settings.
Some microbenchmark to show the improvements, I did best of five
for all of the queries.
-- hypertable with 1k chunks
-- with optimization
select * from metrics1k where time > now() - '5m'::interval;
Time: 3.090 ms
-- without optimization
select * from metrics1k where time > now() - '5m'::interval;
Time: 145.640 ms
-- hypertable with 5k chunks
-- with optimization
select * from metrics5k where time > now() - '5m'::interval;
Time: 4.317 ms
-- without optimization
select * from metrics5k where time > now() - '5m'::interval;
Time: 775.259 ms
-- hypertable with 10k chunks
-- with optimization
select * from metrics10k where time > now() - '5m'::interval;
Time: 4.853 ms
-- without optimization
select * from metrics10k where time > now() - '5m'::interval;
Time: 1766.319 ms (00:01.766)
-- hypertable with 20k chunks
-- with optimization
select * from metrics20k where time > now() - '5m'::interval;
Time: 6.141 ms
-- without optimization
select * from metrics20k where time > now() - '5m'::interval;
Time: 3321.968 ms (00:03.322)
Speedup with 1k chunks: 47x
Speedup with 5k chunks: 179x
Speedup with 10k chunks: 363x
Speedup with 20k chunks: 540x
The general idea is to have two types of fetcher: "fast" and "general
purpose". We use the row-by-row fetcher as the "fast" one. This commit
removes support of text protocol in this fetcher, because it's only
needed for some niche types that don't have a binary serialization, and
is also slower than binary one. Because the row-by-row fetcher now only
understands binary protocol, we must check that the binary
serialization is actually available for the participating data types.
If not, we have to revert to using the cursor fetcher unless row-by-row
was explicitly requested by the user. This happens at execution time,
precisely, at creation of TupleFactory, because that's when we look up
the conversion functions.
The rest of the commit is removing the text protocol support
from row-by-row, plus EXPLAIN changes (we don't know the fetcher type
at the planning stage anymore, so not showing it).
Allow the calls of time_bucket_gapfill to be executed at the
data nodes for improved query performance. With this, time_bucket_gapfill
is pushed to data nodes in the following conditions,
1. when only one data node has all the chunks
2. when space dimension does not overlap across data nodes
3. when group-by matches space dimension
This patch changes the extension function list to include the
signature as well since functions with different signature are
separate objects in postgres. This also changes the list to include
all functions. Even though functions in internal schemas are not
considered public API they still need be treated the same as functions
in other schemas with regards to extension upgrade/downgrade.
This patch also moves the test to regresscheck-shared since we do
not dedicated database to run these tests.
Change the prefix for continuous aggregate tests from
continuous_aggs_ to cagg_. This is similar to commit 6a8c2b66
which did this adjustment for isolation tests because we were
running into length limitations for the spec name. This patch
adjusts the remaining tests to be consistent with the naming
used in isolation tests.
When interpolating float values the result of the calculation
might be unstable for certain values when y0 and y1 are equal.
This patch short circuits the formula and returns y0 immediately
when y0 and y1 are identical.
Fixes#1528
This patch fixes subtract_integer_from_now on 32-bit platforms,
improves error handling and adds some basic tests.
subtract_integer_from_now would trigger an assert when called
on a hypertable without integer time dimension (found by sqlsmith).
Additionally subtract_integer_from_now would segfault when called
on a hypertable without partitioning dimensions.
When getting the next tuple from the subplan gapfill would apply
the projection to it which was incorrect since the subplan already
did the projection and the projection for the gapfill tuple has to
be done when the tuple is handed to the parent node.
Fixes#3834
When a query has a filter that only needs to be evaluated once per
query it will be represented as a Result node with the filter
condition on the Result node and the actual query as child of the
result node. find_data_node_scan_state_child did not consider
Result node as valid node to contain a DataNodeScan node leading
to a `unexpected child node of Append or MergeAppend: 62` for
queries that had one-time filter with a subquery.
The code in cursor_fetcher_rewind asserted that there always
is an associated request which is not true if EOF was reached
already. Found by sqlsmith.
Fixes#3786
The row-by-row fetcher is more efficient, so we want to use it when we
can -- that is, when the have to read only one table from the data
node, without interleaving it with anything else. This patch adds an
option of choosing the fetcher type automatically. It detects the
simplest case of only one distributed table in the entire query, and
enables row-by-row fetcher. For other cases, the cursor fetcher is
used.
Previously, we would push DISTINCT ON down to the data nodes even when
the pathkeys of the resulting paths on the data nodes were not
compatible with the given DISTINCT ON columns. This commit disables
pushdown when the sorting is not compatible.
Fixes#3784
This patch adds support for transparent decompression in queries
on individual chunks.
This is required for distributed hypertables with compression
when enable_per_data_node_queries is set to false. Without
this functionality queries on distributed hypertables with
compression would not return data for compressed chunks as
the generated FDW queries would target individual chunks.
Fixes#3714
PostgreSQL 14 introduced new `Memoize Node` that serve as a cache of
results from parameterized nodes.
We should make sure it will work correctly together with ChunckAppend
custom node over hypertables (compressed and uncompressed).
Closes#3684
With memoize enabled PG14 append tests produce a very different
plan compared to previous PG versions. To make comparing plans
between PG versions easier we disable memoize for PG14.
PG14 also modified how EXTRACT is shown in EXPLAIN output
so any query using EXTRACT will have different EXPLAIN output
between PG14 and previous versions.
