The cmake call to "find_package(OpenSSL)" on windows may set the
variable OPENSSL_LIBRARIES to a non-standard list which might include
non path items such as "debug", "optimized". These non-standard
elements in the list would cause the link failure with error
"LNK1104".
Fix:
Check and retain only valid paths in OPENSSL_LIBRARIES list.
closes#407
With recent changes, we enabled analyze on uncompressed chunk tables
for compressed chunks. This change includes analyzing the compressed
chunks table when analyzing the hypertable and its chunks,
enabling us to remove the generating stats when compressing chunks.
In case of joins in the continuous aggregates, pass the required
structs to the new rte created. These values are required by the
planner to finally query the materialized view.
Fixes#5433
Commit 16fdb6ca5e introduced `timescaledb_experimental.policies` view
to expose the Continuous Aggregate policies but the current JOINS over
our catalog are not accurate.
Fixed it by properly JOIN the underlying catalog tables to expose the
correct information without duplicates about the Continuous Aggregate
policies.
Fixes#5492
When refreshing from the beginning (window_start=NULL) of a
Continuous Aggregate with variable time bucket we were getting a
`timestamp out of range` error.
Fixed it by setting `-Infinity` when passing `window_start=NULL` when
refreshing a Continuous Aggregate with variable time bucket.
Fixes#5474, #5534
Several error messages for continuous aggregates are not following the
error message style guidelines at
https://www.postgresql.org/docs/current/error-style-guide.html
In particular, they do not write the hints and detailed messages as
full sentences.
This is mostly a cosmetic change. When only 1 child is present there
is no need for ordered append. In this situation we might still
benefit from a ChunkAppend node here due to runtime chunk exclusion
when we have non-immutable constraints, so we still add the ChunkAppend
node in that situation even with only 1 child.
This patch drops the following internal SQL functions which were
unused:
_timescaledb_internal.is_main_table(regclass);
_timescaledb_internal.is_main_table(text, text);
_timescaledb_internal.hypertable_from_main_table(regclass);
_timescaledb_internal.main_table_from_hypertable(integer);
_timescaledb_internal.time_literal_sql(bigint, regtype);
While executing compression operations in parallel with
inserting into chunks (both operations which can potentially
change the chunk status), we could get into situations where
the chunk status would end up inconsistent. This change re-reads
the chunk status after locking the chunk to make sure it can
decompress data when handling ON CONFLICT inserts correctly.
Verify that insertion into compressed chunks does not block
each other if the chunks is already partially compressed. Also
check that using the RETURNING clause works the same.
These tests try to verify that changing physical layout
of chunks (either compressed or uncompressed) should
yield consistent results. They also verify index mapping
on compressed chunks is handled correctly.
Commit 8afdddc2da added the first step for deprecating the old format
of Continuous Aggregate but just for PostgreSQL 15 and later versions.
During the extension update we emit a message about the deprecation but
this has being emited even if the user is using PostgreSQL versions
before 15.
Fixed it by emiting the WARNING just when PostgreSQL version is greater
or equal to 15.
This patch moves the support functions for histogram, first and last
into the _timescaledb_functions schema. Since we alter the schema
of the existing functions in upgrade scripts and do not change the
aggregates this should work completely transparently for any user
objects using those aggregates.
Commit 57fde383b3dddd0b52263218e65a0135981c2d34 changed the
messaging but did not format the error hint correctly.
This patch fixes the error hint.
Fixes#5490
Now we look them up again at execution time, which adds up for tables
with a large number of chunks.
This gives about 15% speedup (100 mcs) on a small query on a table from
tests with 50 chunks:
`select id, ts, value from metric_compressed order by id, ts limit 100;`
Decompression produces records which have all the decompressed data
set, but it also retains the fields which are used internally during
decompression.
These didn't cause any problem - unless an operation is being done
with the whole row - in which case all the fields which have ended up
being non-null can be a potential segfault source.
Fixes#5458#5411
Currently internal user objects like chunks and our functions
live in the same schema making locking down that schema hard.
This patch adds a new schema _timescaledb_functions that is meant
to be the schema used for timescaledb internal functions to
allow separation of code and chunks or other user objects.
This patch does following:
1. Executor changes to parse qual ExprState to check if SEGMENTBY
column is specified in WHERE clause.
2. Based on step 1, we build scan keys.
3. Executor changes to do heapscan on compressed chunk based on
scan keys and move only those rows which match the WHERE clause
to staging area aka uncompressed chunk.
4. Mark affected chunk as partially compressed.
5. Perform regular UPDATE/DELETE operations on staging area.
6. Since there is no Custom Scan (HypertableModify) node for
UPDATE/DELETE operations on PG versions < 14, we don't support this
feature on PG12 and PG13.
Refactor the code path that handles remote distributed COPY. The
main changes include:
* Use a hash table to lookup data node connections instead of a list.
* Refactor the per-data node buffer code that accumulates rows into
bigger CopyData messages.
* Reduce the default number of rows in a CopyData message to 100. This
seems to improve throughput, probably striking a better balance
between message overhead and latency.
* The number of rows to send in each CopyData message can now be
changed via a new foreign data wrapper option.
OSM chunks have their own fdw_private which conflicts with checks in
the MergeAppend code path causing segfaults. This commit fixes this by
returning early when there is an OSM chunk in the MergeAppendPath.
Add isolation test case to check that the chunk object created during
chunk copy/move operation on the destination datanode always has
superuser credentials till the end of the operation.
The joins could be between a continuous aggregate and hypertable,
continuous aggregate and a regular Postgres table,
and continuous aggregate and a regular Postgres view.
There is a bug in `width_bucket()` causing an overflow and subsequent
NaN value as a result of dividing with `+inf`. The NaN value is
interpreted as an integer and hence generates an index out of range for
the buckets.
This commit fixes this by generating an error rather than
segfaulting for bucket indexes that are out of range.
During chunk creation, the chunk's dimensional CHECK constraints are
created via an "upcall" to PL/pgSQL code. However, creating
dimensional constraints in PL/pgSQL code sometimes fails, especially
during high-concurrency inserts, because PL/pgSQL code scans metadata
using a snapshot that might not see the same metadata as the C
code. As a result, chunk creation sometimes fail during constraint
creation.
To fix this issue, implement dimensional CHECK-constraint creation in
C code. Other constraints (FK, PK, etc.) are still created via an
upcall, but should probably also be rewritten in C. However, since
these constraints don't depend on recently updated metadata, this is
left to a future change.
Fixes#5456
This patch introduces a C-function to perform the recompression at
a finer granularity instead of decompressing and subsequently
compressing the entire chunk.
This improves performance for the following reasons:
- it needs to sort less data at a time and
- it avoids recreating the decompressed chunk and the heap
inserts associated with that by decompressing each segment
into a tuplesort instead.
If no segmentby is specified when enabling compression or if an
index does not exist on the compressed chunk then the operation is
performed as before, decompressing and subsequently
compressing the entire chunk.
There is a security loophole in current core Postgres, due to which
it's possible for a non-superuser to gain superuser access by attaching
dependencies like expression indexes, triggers, etc. before logical
replication commences.
To avoid this, we now ensure that the chunk objects that get created
for the subscription are done so as a superuser. This avoids malicious
dependencies by regular users.
When calling the `cagg_watermark` function to get the watermark of a
Continuous Aggregate we execute a `SELECT MAX(time_dimension)` query
in the underlying materialization hypertable.
The problem is that a `SELECT MAX(time_dimention)` query can be
expensive because it will scan all hypertable chunks increasing the
planning time for a Realtime Continuous Aggregates.
Improved it by creating a new catalog table to serve as a cache table
to store the current Continous Aggregate watermark in the following
situations:
- Create CAgg: store the minimum value of hypertable time dimension
data type;
- Refresh CAgg: store the last value of the time dimension materialized
in the underlying materialization hypertable (or the minimum value of
materialization hypertable time dimension data type if there's no
data materialized);
- Drop CAgg Chunks: the same as refresh cagg.
Closes#4699, #5307
During the compression autovacuum use to be disabled for uncompressed
chunk and enable after decompression. This leads to postgres
maintainence issue. Let's not disable autovacuum for uncompressed
chunk anymore. Let postgres take care of the stats in its natural way.
Fixes#309
No functional changes, mostly just reshuffles the code to prepare for
batch decompression.
Also removes unneeded repeated column value stores and ExecStoreTuple,
to save 3-5% execution time on some queries.
Invalidate the catalog snapshot in the scanner to ensure that any
lookups into `pg_catalog` uses a snapshot that is consistent with the
snapshot used to scan TimescaleDB metadata.
This fixes an issue where a chunk could be looked up without having a
proper relid filled in, causing an assertion failure
(`ASSERT_IS_VALID_CHUNK`). When a chunk is scanned and found (in
`chunk_tuple_found()`), the Oid of the chunk table is filled in using
`get_relname_relid()`, which could return InvalidOid due to use of a
different snapshot when scanning `pg_class`. Calling
`InvalidateCatalogSnapshot()` before starting the metadata scan in
`Scanner` ensures the pg_catalog snapshot used is refreshed.
Due to the difficulty of reproducing this MVCC issue, no regression or
isolation test is provided, but it is easy to hit this bug when doing
highly concurrent COPY:s into a distributed hypertable.
The functions `PQconndefaults` and `PQmakeEmptyPGresult` calls
`malloc` and can return NULL if it fails to allocate memory for the
defaults and the empty result. It is checked with an `Assert`, but this
will be removed in production builds.
Replace the `Assert` with an checks to generate an error in production
builds rather than trying to de-reference the pointer and cause a
crash.
When adding new status values we must make sure to add special
handling for these values to the downgrade script as previous
versions will not know how to deal with those.
This patch allows unique constraints on compressed chunks. When
trying to INSERT into compressed chunks with unique constraints
any potentially conflicting compressed batches will be decompressed
to let postgres do constraint checking on the INSERT.
With this patch only INSERT ON CONFLICT DO NOTHING will be supported.
For decompression only segment by information is considered to
determine conflicting batches. This will be enhanced in a follow-up
patch to also include orderby metadata to require decompressing
less batches.
Reindexing a relation requires AccessExclusiveLock which prevents
queries on that chunk. This patch changes decompress_chunk to update
the index during decompression instead of reindexing. This patch
does not change the required locks as there are locking adjustments
needed in other places to make it safe to weaken that lock.
