In some cases, entire hypertables can be excluded
at runtime. Some Examples:
WHERE col @> ANY(subselect)
if the subselect returns empty set
WHERE col op (subselect)
if the op is a strict operator and
the subselect returns empty set.
When qual clauses are not on partition columns, we use
the old chunk exclusion, otherwise we try hypertable exclusion.
Hypertable exclusion is executed once per hypertable.
This is cheaper than the chunk exclusion
that is once-per-chunk.
This changes ChunkAppend to always keep the pathkeys from the original
path because the original path was either a MergeAppendPath and this
will become an ordered append or the original path is an AppendPath
and since we do not reorder children the order will be kept intact.
For the AppendPath case with pathkeys it was most likely an Append
with only a single child. We could skip the ChunkAppend path creation
if there is only a single child but we decided earlier that ChunkAppend
would be beneficial for this query so we treat it the same as if it
had multiple children.
Fixes#3030
When the targetlist of a ChunkAppend node was modified after plan
creation the ChunkAppend node would not properly project because
input and output targetlist where pointing to the same list, so
any modification to the output targetlist would carry over to the
input targetlist. This patch decouples input targetlist from
output targetlist to prevent this from happening.
Change our GUC names to use enable-prefix for all boolean GUCs
similar to postgres GUC names.
This patch renames disable_optimizations to enable_optimizations and
constraint_aware_append to enable_constraint_aware_append and removes
optimize_non_hypertables.
The queries to produce test data for space partitioned
hypertables in the append test did not have an explicit
ORDER BY clause leading to a different ordering for the
chunks created on PG12.
Descending into subplans during constification of params
seems unsafe and has led to bugs. Turning this off seems
to be safe and not regress any tested optimizations.
In the future we may want to find a way to optimize this
case as well.
Fixes#1598.
The intermediate expressions used for runtime exclusions would
only be freed at the end of a query leading to excessive memory
usage for nested loop joins with many loops. This patch changes
runtime exclusion to free intermediate expressions as soon as
they are no longer needed.
Some of the append test queries relied on ANALYZE not having run yet
to produce the desired plan. This patch changes the test to
generate more test data and runs ANALYZE explicitly so the tests
are reproducible.
Pushing down limit in ChunkApppend is not safe in the JOIN case
because the decision about required tuples can happen above
ChunkAppend node. This patch disables LIMIT pushdown for all
JOINs.
Output postgres error log after ARM test failure
The append test is flaky on ARM and sometimes switches to use a
Sort -> ChunkAppend instead of MergeAppend. This patch disables
ChunkAppend for this test since we want to test interaction
with MergeAppend in the test.
On ARM the continuous_aggs_bgw test is flaky and fails quite often,
so this patch changes CI to ignore the result of that test on ARM.
Do not push down limit to subplan when the query has operations
that modify the number of tuples returned. Even if we know a
hard limit overall, it doesn't apply if the query has any
grouping/aggregation operations, or SRFs in the tlist.
When ordered append tried to push down targetlist to child paths
it assumed childs would be scans on rels which is not true for
space partitioning where children might be MergeAppend nodes.
This patch also no longer applies the ordered append optimization
to partial paths because its not safe to do so.
This patch also adds more tests for space partitioned hypertables.
ChunkAppend replaced the list of child plans on the plan node with
a local copy which corrupted the child plans when the plan was
turned into a generic plan.
The RangeTblEntry index used during planning might be different from
the index used during execution due to flattening. This patch changes
ConstraintAwareAppend to handle queries where the index changed between
planning and execution.
When the column a Param references is not a partitioning column
the constraint is not useful for excluding chunks so we skip
enabling runtime exclusion for those cases.
This patch makes TimescaleDB use ChunkAppend in places where it
used to used to use ConstraintAwareAppend before.
ConstraintAwareAppend will still be used for MergeAppend nodes
that cannot be changed to Ordered Append or when ChunkAppend is
disabled.
When a query on a hypertable is identified as benefitting from
execution exclusion Append nodes will be replaced by ChunkAppend
nodes.
This will enable the use of runtime exclusion for joins, lateral
joins, subqueries and correlated subqueries.
This patch adds a new ChunkAppend node. This node combines the
functionality of ConstraintAwareAppend and Append and additionally
adds support for runtime chunk exclusion.
This patch only changes the ordered append plans to the new node.
The patch also adds support for space partitioned hypertables
to ordered append and for hypertables with indexes not on all
chunks.
Runtime chunk exclusion will allow chunk exclusion to exclude
chunks for JOINs, LATERAL JOINs and correlated subqueries.
Cross datatype comparisons between DATE/TIMESTAMP/TIMESTAMPTZ
are not immutable which prevents their usage for chunk exclusion.
Unfortunately estimate_expression_value will not estimate those
expressions which makes them unusable for execution time chunk
exclusion with constraint aware append.
To circumvent this we inject casts and use an operator
with the same datatype on both sides when constifying the
restrictinfo. This allows estimate_expression_value
to evaluate those expressions and makes them accessible for
execution time chunk exclusion.
The following transformations are done:
TIMESTAMP OP TIMESTAMPTZ => TIMESTAMP OP (TIMESTAMPTZ::TIMESTAMP)
TIMESTAMPTZ OP DATE => TIMESTAMPTZ OP (DATE::TIMESTAMPTZ)
No transformation is required for TIMESTAMP OP DATE because
those operators are marked immutable.
