This patch deprecates the recompress_chunk procedure as all that
functionality is covered by compress_chunk now. This patch also adds a
new optional boolean argument to compress_chunk to force applying
changed compression settings to existing compressed chunks.
Add a CMake option to add the PostgreSQL source directory as a system
include for the build system. This will ensure that the PG source
include directory will end up in a generated `compile_commands.json`,
which can be used by language servers (e.g., clangd) to navigate from
the TimescaleDB source directly to the PostgreSQL source.
If the comparison function for qsort is non-transitive, there is a risk
of out-of-bounds access. Subtraction of integers can lead to overflows,
so instead use a real comparison function.
Log progress of the compression/decompression APIs into the postgresql
log file. In case of issues in the field, we really do not have much
idea about which stage the compression function is stuck at. Hopefully
we will have a better idea now with these log messages in place. We
want to keep things simple and enable logging of the progress by
default for these APIs. We can re-look if the users complaint about the
chattiness later.
The default for the function definition of decompress_chunk was already
changed in a previous patch. This patch also adjusts the internal
default, but since there was no way this could have been called it does
not affect the user-facing API.
The recompression policy only recompresses unordered chunks which
will be rarely created but does not deal with partial chunks which
are the result of DML operations into compressed chunks.
In #4770 we changed the behavior of compression policies to continue
compressing chunks even if a failure happens in one of them.
The problem is if a failure happens in one or all the chunks the job is
marked as successful, and this is wrong because a failure happens so if
an exception occurs when compressing chunks we'll raise an exception at
the end of the procedure in order to mark the job as failed.
Fix missing alignment handling, and also buffer size problems. We were
not properly reporting the failures in the compression_algo tests, so
didn't notice these problems. Add part of tests from "vector text
predicates" PR to improve coverage.
If a lot of chunks are involved then the current pl/pgsql function
to compute the size of each chunk via a nested loop is pretty slow.
Additionally, the current functionality makes a system call to get the
file size on disk for each chunk everytime this function is called.
That again slows things down. We now have an approximate function which
is implemented in C to avoid the issues in the pl/pgsql function.
Additionally, this function also uses per backend caching using the
smgr layer to compute the approximate size cheaply. The PG cache
invalidation clears off the cached size for a chunk when DML happens
into it. That size cache is thus able to get the latest size in a
matter of minutes. Also, due to the backend caching, any long running
session will only fetch latest data for new or modified chunks and can
use the cached data (which is calculated afresh the first time around)
effectively for older chunks.
Issues due to OpenSSL 3.2 seem to be resolved via components
installing 3.1.x. So, remove our temporary changes.
We also disable uploading of test stats till the next minor release
of PG on macOS.
This patch changes those functions to no longer error by default when
the chunk is not the expected state, instead a warning is raised.
This is in preparation for changing compress_chunk to forward to
the appropriate operation depending on the chunk state.
When inserting into a compressed chunk with an ON CONFLICT clause, and
there is a concurrent recompression of the chunk, an error can occur
due to a failed lookup of the compressed chunk.
The error happens because the chunk information is re-read from an
outdated cache holding a compressed chunk ID that no longer exists
after the chunk was recompressed (recompression effectively removes
the compressed chunk and creates it again).
Fix this issue by ensuring that the insert path only uses up-to-date
chunk metadata read _after_ the lock on the chunk was taken.
Foreign keys pointing to hypertables are not supported. Creating a
hypertable from a table referenced by foreign key succeeds, but it
leaves the referencing (child) table in a broken state, failing on every
insert with a `violates foreign key constraint` error.
To prevent this scenario, if a foreign key reference to the table exists
before converting it to a hypertable, the following error will be
raised:
```
cannot have FOREIGN KEY contraints to hypertable "<table_name>"
HINT: Remove all FOREIGN KEY constraints to table "<table_name>"
before making it a hypertable.
```
Fixes#6452
Refactor the compression to code to move the column map code
in a separate function and change tsl_get_compressed_chunk_index_for_recompression
to not initialize RowCompressor anymore.
The watermark function for CAggs is declared as STABLE since the value
of the function changes after every CAgg refresh. The function
volatility prevents the planner from replacing the function invocation
with a constant value and executing plan time chunk exclusion. This
leads to high planning times on hypertables with many chunks.
This PR replaces the function invocation with a constant value to allow
plan time exclusion of chunks. We perform the replacement at plan time
instead of changing the function volatility to IMMUTABLE, because we
want to control the constification. Only queries that access the
underlying hypertable in a query (i.e., no queries like SELECT
cagg_watermark(...) without any FROM condition) are rewritten. This is
done to make sure that the query is properly invalidated when the
underlying table changes (e.g., the watermark is updated) and the query
is replanned on the subsequent execution.
Fixes: #6105, #6321
Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
This change introduces a GUC to limit the amount of tuples
that can be decompressed during an INSERT, UPDATE, or
DELETE on a compressed hypertable. Its used to prevent
running out of disk space if you try to decompress
a lot of data by accident.
The chunk-wise aggregation pushdown code creates a copy of the existing
paths; it copies the existing paths to create new ones with pushed-down
aggregates. However, the copy_merge_append_path function behaves
differently than other copy functions (e.g., copy_append_path); it
resets the pathtarget on the copy. This leads to a wrong pathlist and
crashes. This PR fixes the wrong pathtarget by setting it after the path
is copied.
This patch removes the restrictions preventing changing compression
settings when compressed chunks exist. Compression settings can now
be changed at any time and will not affect existing compressed chunks
but any subsequent compress_chunk will apply the changed settings.
A decompress_chunk/compress_chunk cycle will also compress the chunk
with the new settings.
Logging and caching related tables from the timescaledb extension
should not be dumped using pg_dump. Our scripts specify a few such
unwanted tables. Apart from being unnecessary, the "job_errors" had
some restricted permissions causing additional problems in pg_dump.
We now don't include such tables for dumping.
Fixes#5449
Previously when using BETWEEN ... AND additional constraints in a
WHERE clause, the BETWEEN was not handled correctly because it was
wrapped in a BoolExpr node, which prevented plantime exclusion.
The flattening of such expressions happens in `eval_const_expressions`
which gets called after our constify_now code.
This commit fixes the handling of this case to allow chunk exclusion to
take place at planning time.
Also, makes sure we use our mock timestamp in all places in tests.
Previously we were using a mix of current_timestamp_mock and now(),
which was returning unexpected/incorrect results.
The commit #6513 removed some restricted chunk operations, enabling
adding constraints to OSM chunks directly. This operation is blocked
on OSM chunks. The present commit ensures that adding a constraint
directly on an OSM chunk is blocked.
Segmentwise recompression grabbed an AccessExclusiveLock on
the compressed chunk index. This would block all read operations
on the chunk which involved said index. Reducing the lock to
ExclusiveLock would allow reads, removing blocks from other
ongoing operations.
This patch changes the dump configuration for
_timescaledb_catalog.metadata to include all entries. To allow loading
logical dumps with this configuration an insert trigger is added that
turns uniqueness conflicts into updates to not block the restore.
Since #6505, the changelog script tries to access the
secrets.ORG_AUTOMATION_TOKEN. However, accessing secrets is not possible
for PRs. This PR changes the token to the default access token, which
is available in PRs and provides read access to the issue API.
We couldn't use parameterized index scan by segmentby column on
compressed chunk, effectively making joins on segmentby unusable. We
missed this when bulk-updating test references for PG16. The underlying
reason is the incorrect creation of equivalence members for segmentby
columns of compressed chunk tables. This commit fixes it.
Since the optional time_bucket arguments like offset, origin and
timezone shift the output by at most bucket width we can optimize
these similar to how we optimize the other time_bucket constraints.
Fixes#4825
