The queries to produce test data for space partitioned
hypertables in the append test did not have an explicit
ORDER BY clause leading to a different ordering for the
chunks created on PG12.
Descending into subplans during constification of params
seems unsafe and has led to bugs. Turning this off seems
to be safe and not regress any tested optimizations.
In the future we may want to find a way to optimize this
case as well.
Fixes#1598.
The intermediate expressions used for runtime exclusions would
only be freed at the end of a query leading to excessive memory
usage for nested loop joins with many loops. This patch changes
runtime exclusion to free intermediate expressions as soon as
they are no longer needed.
Some of the append test queries relied on ANALYZE not having run yet
to produce the desired plan. This patch changes the test to
generate more test data and runs ANALYZE explicitly so the tests
are reproducible.
Pushing down limit in ChunkApppend is not safe in the JOIN case
because the decision about required tuples can happen above
ChunkAppend node. This patch disables LIMIT pushdown for all
JOINs.
Output postgres error log after ARM test failure
The append test is flaky on ARM and sometimes switches to use a
Sort -> ChunkAppend instead of MergeAppend. This patch disables
ChunkAppend for this test since we want to test interaction
with MergeAppend in the test.
On ARM the continuous_aggs_bgw test is flaky and fails quite often,
so this patch changes CI to ignore the result of that test on ARM.
Do not push down limit to subplan when the query has operations
that modify the number of tuples returned. Even if we know a
hard limit overall, it doesn't apply if the query has any
grouping/aggregation operations, or SRFs in the tlist.
When ordered append tried to push down targetlist to child paths
it assumed childs would be scans on rels which is not true for
space partitioning where children might be MergeAppend nodes.
This patch also no longer applies the ordered append optimization
to partial paths because its not safe to do so.
This patch also adds more tests for space partitioned hypertables.
ChunkAppend replaced the list of child plans on the plan node with
a local copy which corrupted the child plans when the plan was
turned into a generic plan.
The RangeTblEntry index used during planning might be different from
the index used during execution due to flattening. This patch changes
ConstraintAwareAppend to handle queries where the index changed between
planning and execution.
When the column a Param references is not a partitioning column
the constraint is not useful for excluding chunks so we skip
enabling runtime exclusion for those cases.
This patch makes TimescaleDB use ChunkAppend in places where it
used to used to use ConstraintAwareAppend before.
ConstraintAwareAppend will still be used for MergeAppend nodes
that cannot be changed to Ordered Append or when ChunkAppend is
disabled.
When a query on a hypertable is identified as benefitting from
execution exclusion Append nodes will be replaced by ChunkAppend
nodes.
This will enable the use of runtime exclusion for joins, lateral
joins, subqueries and correlated subqueries.
This patch adds a new ChunkAppend node. This node combines the
functionality of ConstraintAwareAppend and Append and additionally
adds support for runtime chunk exclusion.
This patch only changes the ordered append plans to the new node.
The patch also adds support for space partitioned hypertables
to ordered append and for hypertables with indexes not on all
chunks.
Runtime chunk exclusion will allow chunk exclusion to exclude
chunks for JOINs, LATERAL JOINs and correlated subqueries.
Cross datatype comparisons between DATE/TIMESTAMP/TIMESTAMPTZ
are not immutable which prevents their usage for chunk exclusion.
Unfortunately estimate_expression_value will not estimate those
expressions which makes them unusable for execution time chunk
exclusion with constraint aware append.
To circumvent this we inject casts and use an operator
with the same datatype on both sides when constifying the
restrictinfo. This allows estimate_expression_value
to evaluate those expressions and makes them accessible for
execution time chunk exclusion.
The following transformations are done:
TIMESTAMP OP TIMESTAMPTZ => TIMESTAMP OP (TIMESTAMPTZ::TIMESTAMP)
TIMESTAMPTZ OP DATE => TIMESTAMPTZ OP (DATE::TIMESTAMPTZ)
No transformation is required for TIMESTAMP OP DATE because
those operators are marked immutable.
