This PR adds test infrastructure for running tests with shared tables.
This allows having hypertables with specific configurations usable for
all tests. Since these tests also don't require creating a new database
for each test case some of the overhead of the normal tests is removed.
While this will lead to much faster query tests some tests will still
require their own database to test things, but most queres could be moved
to this infrastructure to improve test coverage and speed them up.
Ordered append for space partitioned hypertable would lead to an
error when the ORDER BY clause was not a column reference on PG 9.6
and PG 10. This patch fixes ordered append for space partitioned
hypertable and allows arbitary expressions to be used in the
ORDER BY clause.
With ordered append, chunk exclusion occur only along the primary open
"time" dimension, failing to exclude chunks along additional
partitioning dimensions. For instance, a query on a two-dimensional
table "hyper" (time, device), such as
```
SELECT * FROM hyper
WHERE time > '2019-06-11 12:30'
AND device = 1
ORDER BY time;
```
would only exclude chunks based on the "time" column restriction, but
not the "device" column restriction. This causes an unnecessary number
of chunks to be included in the query plan.
The reason this happens is because chunk exclusion during ordered
append is based on pre-sorting the set of slices in the primary
dimension to determine ordering. This is followed by a scan for
chunks slice-by-slice in the order of the sorted slices. Since those
scans do not include the restrictions in other dimensions, chunks that
would otherwise not match are included in the result.
This change fixes this issue by using the "regular" chunk scan that
account for multi-dimensional restrictions. This is followed by a sort
of the resulting chunks along the primary "time" dimension.
While this, sometimes, means sorting a larger set than the initial
slices in the primary "time" dimension, the resulting chunk set is
smaller instead. Sorting chunks also allows doing secondary ordering
on chunk ID for those chunks that belong to the same "time"
slice. While this additional ordering is not required for correct
tuple ordering, it gives slightly nicer EXPLAIN output since chunks
are also ordered by ID.
The initial implementation for Ordered Append required the ORDER BY
clause expression to match the time partitioning column.
This patch loosens that restriction and will apply the Ordered
Append optimization for queries with ORDER BY time_bucket and
date_trunc as well.
The Order display of the ChunkAppend node used the output of
the node instead of the input of the node when resolving the
targetlist index to display the order information leading to
incorrect display when the Sort column was not passed through
or the position changed.
This patch makes TimescaleDB use ChunkAppend in places where it
used to used to use ConstraintAwareAppend before.
ConstraintAwareAppend will still be used for MergeAppend nodes
that cannot be changed to Ordered Append or when ChunkAppend is
disabled.
When a query on a hypertable is identified as benefitting from
execution exclusion Append nodes will be replaced by ChunkAppend
nodes.
This will enable the use of runtime exclusion for joins, lateral
joins, subqueries and correlated subqueries.
The initial support for space partitions for ordered append
only supported 1 space dimension. This patch removes that restriction
and supports an arbitrary amount of space partitions.
When doing a JOIN with a hypertable ordered append required the
ORDER BY clause to have the time partitioning column as first
expression. This patch loosens that restriction.
If the ORDER BY clause of a query does not reference our hypertable
but has an equality condition on the time partitioning column
of our hypertable we will do an ordered append because
it allows the planner to skip the Sort step for MergeJoin.
This patch adds a new ChunkAppend node. This node combines the
functionality of ConstraintAwareAppend and Append and additionally
adds support for runtime chunk exclusion.
This patch only changes the ordered append plans to the new node.
The patch also adds support for space partitioned hypertables
to ordered append and for hypertables with indexes not on all
chunks.
Runtime chunk exclusion will allow chunk exclusion to exclude
chunks for JOINs, LATERAL JOINs and correlated subqueries.
The comparison to check whether the column references
matches the time dimension column assumed that the reference
would be to the hypertable which might not be true for JOINs.
This patch adds a check to make sure the reference in the
ORDER BY is actually a reference to the hypertable before
checking the column is the time dimension.
Ordered append used to only consider startup cost of the first child.
This patch changes the cost for ordered append to also include the total
cost of the first child
Constraint aware append used some planner data structures in the
executor which cannot be serialized so they will not be available
in parallel workers, this patch refactors this and marks the
constraint aware append node as parallel safe when the subpath is
parallel safe. The additional function executions in the test output
are because the constify is run once per chunk now, previously it run
only once and the result was copied to the restrictinfo of every chunk
and then adjusted for the chunk, but since the adjustment for the chunk
has to happen in the planner because it needs access to appendrelinfo
we now create per chunk restrictinfo in the planner and constify each of
these in the executor.
On PostgreSQL < 12 now() is not parallel safe and using now() in a
query will prevent parallelism. As a workaround transaction_timestamp()
or CURRENT_TIMESTAMP can be used which will not prevent parallelism.
This optimization will replace the MergeAppendPath for queries on
hypertables ordered by the time partitioning column and with a
LIMIT clause with an ordered AppendPath. This optimization will
remove the need for last point queries to access every chunk of
a hypertable.
This commit also adds struct TimescaleDBPrivate which is stored
in RelOptInfo->fdw_private to store TimescaleDB-specific plan state
between different planner hook invocations in the plan.
We needed to add TimescaleDBPrivate to store a flag indicating
whether or not to use ordered append between different parts of
the planner.
