If a hypertable uses a non default tablespace, the compressed hypertable
and its corresponding toast table and index is still created in the
default tablespace.
This PR fixes this unexpected behavior and creates the compressed
hypertable and its toast table and index in the same tablespace as
the hypertable.
Fixes#5520
This patch adds an optimization to the DecompressChunk node. If the
query 'order by' and the compression 'order by' are compatible (query
'order by' is equal or a prefix of compression 'order by'), the
compressed batches of the segments are decompressed in parallel and
merged using a binary heep. This preserves the ordering and the sorting
of the result can be prevented. Especially LIMIT queries benefit from
this optimization because only the first tuples of some batches have to
be decompressed. Previously, all segments were completely decompressed
and sorted.
Fixes: #4223
Co-authored-by: Sotiris Stamokostas <sotiris@timescale.com>
On compressed hypertables 3 schema levels are in use simultaneously
* main - hypertable level
* chunk - inheritance level
* compressed chunk
In the build_scankeys method all of them appear - as slot have their
fields represented as a for a row of the main hypertable.
Accessing the slot by the attribut numbers of the chunks may lead to
indexing mismatches if there are differences between the schemes.
Fixes: #5577
Since telemetry job has a special code path to be able to be used both
from Apache code and from TSL code, trying to execute the telemetry job
with run_job() will fail.
This code will allow run_job() to be used with the telemetry job to
trigger a send of telemetry data. You have to belong to the group that
owns the telemetry job (or be the owner of the telemetry job) to be
able to use it.
Closes#5605
There were no permission checks when calling run_job(), so it was
possible to execute any job regardless of who owned it. This commit
adds such checks.
The internal `cagg_rebuild_view_definition` function was trying to cast
a pointer to `RangeTblRef` but it actually is a `RangeTblEntry`.
Fixed it by using the already existing `direct_query` data struct to
check if there are JOINs in the CAgg to be repaired.
When updating or deleting tuples from a compressed chunk, we first
need to decompress the matching tuples then proceed with the operation.
This optimization reduces the amount of data decompressed by using
compressed metadata to decompress only the affected segments.
This block was removed by accident, in order to support this we
need to ensure the uniqueness in the compressed data which is
something we should do in the future thus removing this block.
Inserting multiple rows into a compressed chunk could have bypassed
constraint check in case the table had segment_by columns.
Decompression is narrowed to only consider candidates by the actual
segment_by value.
Because of caching - decompression was skipped for follow-up rows of
the same Chunk.
Fixes#5553
When inserting into a compressed chunk with constraints present,
we need to decompress relevant tuples in order to do speculative
inserting. Usually we used segment by column values to limit the
amount of compressed segments to decompress. This change expands
on that by also using segment metadata to further filter
compressed rows that need to be decompressed.
This was previously disabled as no data resided on the
uncompressed chunk once it was compressed, but this is not
the case anymore with partially compressed chunks, so we
enable indexscan for the uncompressed chunk again.
Fixes#5432
Co-authored-by: Ante Kresic <ante.kresic@gmail.com>
Instead of using a user name to register the owner of a job, we use
regrole. This allows renames to work properly since the underlying OID
does not change when the owner name changes.
We add a check when calling `DROP ROLE` that there is no job with that
owner and generate an error if there is.
Our cost model should be self-consistent, and the relative values for
the remote tuple and startup costs should reflect their real cost,
relative to costs of other operations like CPU tuple cost.
For example, now remote costs are set even lower than the parallel tuple
and startup cost. Contrary to that, their real world cost is going to be
an order of magnitude higher or more, because parallel tuples are sent
through shared memory, and remote tuples are sent over the network.
Increasing these costs leads to query plan improvements, e.g. we start
to favor the GROUP BY pushdown in some cases.
The start_scheduled_jobs function mistakenly sorts the scheduled_jobs
list in-place. As a result, when the ts_update_scheduled_jobs_list
function compares the updated list of scheduled jobs with the existing
scheduled jobs list, it is comparing a list that is sorted by job_id to
one that is sorted by next_start time. Fix that by properly copying the
scheduled_jobs list into a new list and use that for sorting.
Fixes#5537
With recent changes, we enabled analyze on uncompressed chunk tables
for compressed chunks. This change includes analyzing the compressed
chunks table when analyzing the hypertable and its chunks,
enabling us to remove the generating stats when compressing chunks.
In case of joins in the continuous aggregates, pass the required
structs to the new rte created. These values are required by the
planner to finally query the materialized view.
Fixes#5433
Commit 16fdb6ca5e introduced `timescaledb_experimental.policies` view
to expose the Continuous Aggregate policies but the current JOINS over
our catalog are not accurate.
Fixed it by properly JOIN the underlying catalog tables to expose the
correct information without duplicates about the Continuous Aggregate
policies.
Fixes#5492
When refreshing from the beginning (window_start=NULL) of a
Continuous Aggregate with variable time bucket we were getting a
`timestamp out of range` error.
Fixed it by setting `-Infinity` when passing `window_start=NULL` when
refreshing a Continuous Aggregate with variable time bucket.
Fixes#5474, #5534
Several error messages for continuous aggregates are not following the
error message style guidelines at
https://www.postgresql.org/docs/current/error-style-guide.html
In particular, they do not write the hints and detailed messages as
full sentences.
This is mostly a cosmetic change. When only 1 child is present there
is no need for ordered append. In this situation we might still
benefit from a ChunkAppend node here due to runtime chunk exclusion
when we have non-immutable constraints, so we still add the ChunkAppend
node in that situation even with only 1 child.
These tests try to verify that changing physical layout
of chunks (either compressed or uncompressed) should
yield consistent results. They also verify index mapping
on compressed chunks is handled correctly.
Decompression produces records which have all the decompressed data
set, but it also retains the fields which are used internally during
decompression.
These didn't cause any problem - unless an operation is being done
with the whole row - in which case all the fields which have ended up
being non-null can be a potential segfault source.
Fixes#5458#5411
This patch does following:
1. Executor changes to parse qual ExprState to check if SEGMENTBY
column is specified in WHERE clause.
2. Based on step 1, we build scan keys.
3. Executor changes to do heapscan on compressed chunk based on
scan keys and move only those rows which match the WHERE clause
to staging area aka uncompressed chunk.
4. Mark affected chunk as partially compressed.
5. Perform regular UPDATE/DELETE operations on staging area.
6. Since there is no Custom Scan (HypertableModify) node for
UPDATE/DELETE operations on PG versions < 14, we don't support this
feature on PG12 and PG13.
Refactor the code path that handles remote distributed COPY. The
main changes include:
* Use a hash table to lookup data node connections instead of a list.
* Refactor the per-data node buffer code that accumulates rows into
bigger CopyData messages.
* Reduce the default number of rows in a CopyData message to 100. This
seems to improve throughput, probably striking a better balance
between message overhead and latency.
* The number of rows to send in each CopyData message can now be
changed via a new foreign data wrapper option.
The joins could be between a continuous aggregate and hypertable,
continuous aggregate and a regular Postgres table,
and continuous aggregate and a regular Postgres view.
This patch introduces a C-function to perform the recompression at
a finer granularity instead of decompressing and subsequently
compressing the entire chunk.
This improves performance for the following reasons:
- it needs to sort less data at a time and
- it avoids recreating the decompressed chunk and the heap
inserts associated with that by decompressing each segment
into a tuplesort instead.
If no segmentby is specified when enabling compression or if an
index does not exist on the compressed chunk then the operation is
performed as before, decompressing and subsequently
compressing the entire chunk.
When calling the `cagg_watermark` function to get the watermark of a
Continuous Aggregate we execute a `SELECT MAX(time_dimension)` query
in the underlying materialization hypertable.
The problem is that a `SELECT MAX(time_dimention)` query can be
expensive because it will scan all hypertable chunks increasing the
planning time for a Realtime Continuous Aggregates.
Improved it by creating a new catalog table to serve as a cache table
to store the current Continous Aggregate watermark in the following
situations:
- Create CAgg: store the minimum value of hypertable time dimension
data type;
- Refresh CAgg: store the last value of the time dimension materialized
in the underlying materialization hypertable (or the minimum value of
materialization hypertable time dimension data type if there's no
data materialized);
- Drop CAgg Chunks: the same as refresh cagg.
Closes#4699, #5307
During the compression autovacuum use to be disabled for uncompressed
chunk and enable after decompression. This leads to postgres
maintainence issue. Let's not disable autovacuum for uncompressed
chunk anymore. Let postgres take care of the stats in its natural way.
Fixes#309
This patch allows unique constraints on compressed chunks. When
trying to INSERT into compressed chunks with unique constraints
any potentially conflicting compressed batches will be decompressed
to let postgres do constraint checking on the INSERT.
With this patch only INSERT ON CONFLICT DO NOTHING will be supported.
For decompression only segment by information is considered to
determine conflicting batches. This will be enhanced in a follow-up
patch to also include orderby metadata to require decompressing
less batches.
This patch changes the way user-defined FDW options (e.g., startup
costs, per-tuple costs) are handled. So far, these values were retrieved
in apply_fdw_and_server_options() but reset to default values afterward.
The copy fetcher fetches tuples in batches. When the last element in the
batch is the file trailer, the trailer was not handled correctly. The
existing logic did not perform a PQgetCopyData in that case. Therefore
the state of the fetcher was not set to EOF and the copy operation was
not correctly finished at this point.
Fixes: #5323
WHERE clause with SEGMENTBY column of type text/bytea
non-equality operators are not pushed down to Seq Scan
node of compressed chunk. This patch fixes this issue.
Fixes#5286
Make `partialize_agg()` support parallel query execution. To make this
work, the finalize node need combine the individual partials from each
parallel worker, but the final step that turns the resulting partial
into the finished aggregate should not happen. Thus, in the case of
distributed hypertables, each data node can run a parallel query to
compute a partial, and the access node can later combine and finalize
these partials into the final aggregate. Esssentially, there will be
one combine step (minus final) on each data node, and then another one
plus final on the access node.
To implement this, the finalize aggregate plan is simply modified to
elide the final step, and to reserialize the partial. It is only
possible to do this at the plan stage; if done at the path stage, the
PostgreSQL planner will hit assertions that assume that the node has
certain values (e.g., it doesn't expect combine Paths to skip the
final step).
Previously we used date_part("epoch", interval) and integer division
internally to determine whether the top cagg's interval is a
multiple of its parent's.
This led to precision loss and wrong results
in the case of intervals with sub-second components.
Fixed by using the `ts_interval_value_to_internal` function to convert
intervals to appropriate integer representation for division.
Fixes#5277
When a Continuous Aggregate is created the `chunk_interval_size` is
defined my the `chunk_interval_size` of the original hypertable
multiplied by a fixed factor of 10.
The problem is currently when we create a Hierarchical Continuous
Aggregate the same factor is applied and it lead to an exponential
`chunk_interval_size`.
Fixed it by just copying the `chunk_interval_size` from the base
Continuous Aggregate for an Hierachical Continuous Aggreagate.
Fixes#5382
Renamed:
tsl/test/sql/size_utils.sql
tsl/test/expected/size_utils.out
To:
tsl/test/sql/size_utils_tsl.sql
tsl/test/expected/size_utils_tsl.out
because conflicting with test/sql/size_utils.sql
