This PR introduces a new SQL function to associate a
hypertable or continuous agg with a custom job. If
this dependency is setup, the job is automatically
deleted when the hypertable/cagg is dropped.
Add _timescaledb_internal.attach_osm_table_chunk.
This treats a pre-existing foreign table as a
hypertable chunk by adding dummy metadata to the
catalog tables.
The numeric format changed between PG13 and PG14 to include infinities.
As a result the serialized partial state of numeric aggregates also
changed format.
If a user that has stored partials (e.g. by using Continuous
Aggregates) upgrades to PG14 then the partial state deserialization
will lead to errors due to the mismatch with the PG14 code.
Repair the deserialization process on the fly by appending zeroed
plus-infinity and minus-infinity counts for the numeric aggregate state
to use.
Fixes#4427
The "empty" bytea value in a column of a distributed table when
selected was being returned as "null". The actual value on the
datanodes was being stored appropriately but just the return code path
was converting it into "null" on the AN. This has been handled via the
use of PQgetisnull() function now.
Fixes#3455
Switch update/downgrade test to use 14.4. This has been split off
the CI PR switching the rest of CI to 14.4 because it usually takes
more then a week for the upstream docker images to become available.
The commit-msg hook was counting the end-of-line characters in the 50-
and 72-character limit, so commit messages had to be wrapped slightly
shorter in order to avoid the hook complaint. Strip the EOL characters
during the check instead.
Since we use universal newline handling during the file read, stripping
LF ('\n') should be enough to handle both Windows and Unix development
environments.
This git hook is copied automatically during CMake, and breaks commits
immediately afterwards if Python 2 isn't installed. Since Python 2 has
been end-of-life for a while now, take this chance to upgrade.
We parse apart the output of `pg_config --configure` to see whether or
not OpenSSL is enabled. This is a bit brittle; it has broken in the past
with the change from --with-openssl to --with-ssl, and upstream is
currently testing a change to the build system (Meson, which doesn't
rely on autotools during configuration) that is likely to further
interfere with this approach.
As an alternative, we can just look at the header files to get this
information. USE_OPENSSL is not defined in pg_config.h if SSL support
is not compiled in.
We mandate that a superuser or a user with REPLICATION privileges can
invoke copy_chunk or move_chunk procedures. Internally, many stages
are carried out to complete the activity and different stages need
different user permissions. To keep things uniform we now switch to the
bootstrap superuser in each stage. Care is taken to ensure that the
original hypertable ownership is retained on the new chunk post the
move operation.
The compression code had 2 files using the same header guard. This
patch renames the file with floating point helper functions to
float_utils.h and renames the other file to compression/api since
that more clearly reflects the purpose of the functions.
In `src/ts_catalog/catalog.c` we explicit define some constraints and
indexes names into `catalog_table_index_definitions` array, but in our
pre-install SQL script for schema definition we don't, so let's be more
explicit here and prevent future surprises.
If a superuser is used to invoke attach_data_node on a hypertable then
we need to ensure that the object created on this data node has the
same original ownership permissions.
Fixes#4433
Add a parameter `drop_remote_data` to `detach_data_node()` which
allows dropping the hypertable on the data node when detaching
it. This is useful when detaching a data node and then immediately
attaching it again. If the data remains on the data node, the
re-attach will fail with an error complaining that the hypertable
already exists.
The new parameter is analogous to the `drop_database` parameter of
`delete_data_node`. The new parameter is `false` by default for
compatibility and ensures that a data node can be detached without
requiring communicating with the data node (e.g., if the data node is
not responding due to a failure).
Closes#4414
Timescale 2.7 released a new version of Continuous Aggregate (#4269)
that allows users efectivelly create and use indexes in the
materialization hypertable. The boring part of it is that users should
discover what is the associated materialization hypertable to issue a
`CREATE INDEX` statement.
Improved it by allowing users to easily create indexes in the
materialization hypertable by simple executing a `CREATE INDEX` direct
in the Continuous Aggregate.
Example:
`CREATE INDEX name_of_the_index ON continuous_agregate (column);`
Refactor make_partfunc_call to only accept fnoid and rettype instead
of PartitioningFunc which also gets rid of a coverity warning about
that parameter being passed by value instead of reference because
that parameter was longer than 128 bytes.
Previous pull request #4269 introduced new format for Continuous
Aggregates and we also added regression tests for the `deprecated`
version to make sure it will keep working until we decide to completely
deprecate and remove the old version.
Unfortunately for some deprecated continous aggregates regression tests
we miss to set properly the flag `timescaledb.finalized=false`.
Fixed it by properly setting the `timecaledb.finalized=false` during
the continuous aggregate creation.
Don't keep the chunk constraints while searching. The number of
candidate chunks can be very large, so keeping these constraints is a
lot of work and uses a lot of memory. For finding the matching chunk,
it is enough to track the number of dimensions that matched a given
chunk id. After finding the chunk id, we can look up only the matching
chunk data with the usual function.
This saves some work when doing INSERTs.
This patch transforms constraints on hash-based space partitions to make
them usable by postgres constraint exclusion.
If we have an equality condition on a space partitioning column, we add
a corresponding condition on get_partition_hash on this column. These
conditions match the constraints on chunks, so postgres' constraint
exclusion is able to use them and exclude the chunks.
The following transformations are done:
device_id = 1
becomes
((device_id = 1) AND (_timescaledb_internal.get_partition_hash(device_id) = 242423622))
s1 = ANY ('{s1_2,s1_2}'::text[])
becomes
((s1 = ANY ('{s1_2,s1_2}'::text[])) AND
(_timescaledb_internal.get_partition_hash(s1) = ANY ('{1583420735,1583420735}'::integer[])))
These transformations are not visible in EXPLAIN output as we remove
them again after hypertable expansion is done.
Commit dcb7dcc5 removed the constified intermediate values used
during hypertable expansion but only did so completely for PG14.
For PG12 and PG13 some constraints remained in the plan.
Add a parameter `schedule_interval` to retention and
compression policies to allow users to define the schedule
interval. Fall back to previous default if no value is
specified.
Fixes#3806
This patch consolidates all shared memory requests in a
shmem_request_hook. While there are no strict requirements when
to request shared memory for PG < 15 in PG 15 it has to happen
in the shmem_request_hook otherwise the request will be blocked.
https://github.com/postgres/postgres/commit/4f2400cb
Previously we created one index per segmentby column, of the form
`(col, _ts_meta_sequence_num)`.
Compressed data is ordered by segmentby, then by orderby within
the segments and lastly by the sequence number of the batch. So if
segmentby columns are missing from the index, that index cannot be
used to produce ordered output, requiring an extra sort step.
A composite index containing all segmentby columns removes the
additional sort step and gives better plans.
Fixes#4314
For certain inserts on a distributed hypertable, e.g., involving CTEs
and upserts, plans can be generated that weren't properly handled by
the DataNodeCopy and DataNodeDispatch execution nodes. In particular,
the nodes expect ChunkDispatch as a child node, but PostgreSQL can
sometimes insert a Result node above ChunkDispatch, causing the crash.
Further, behavioral changes in PG14 also caused the DataNodeCopy node
to sometimes wrongly believe a RETURNING clause was present. The check
for returning clauses has been updated to fix this issue.
Fixes#4339
PG13 added an implementation of find_em_expr_for_rel to postgres
core code. Which is removed again in PG15. This patch adjusts
our macros to deal with the removal in PG15.
https://github.com/postgres/postgres/commit/f3dd9fe1
When enabling compression on a hypertable, the orderby option can be
omitted, which will set the default value of "time DESC".
However previously, when executing the same command twice not setting
orderby, the second time we would get an error that orderby was
previously set and must be specified.
For example when executing
`alter table set (timescaledb.compress, timescaledb.segmentby = '..')`
The reason for that error was that it's unclear if no orderby means
leave as is, or if it means set the default value.
But in the case where orderby is already set to the default value,
there is no ambiguity and both cases are equivalent, so the default
value can be reset without giving an error.
Fixes#4331
Non-functional change.
Parameters to workers were passed in as a serialized string, which then
needs to be serialized and deserialized using dedicated functions.
This commit refactors code to pass parameters to workers as a struct,
which is then just copied into the `bgw_extra` field of
`BackgroundWorker`. The struct contains simple values and can therefore
be copied using memcpy(3c).
Postgres knows whether a given aggregate is parallel-safe, and creates
parallel aggregation plans based on that. The `partialize_agg` is a
wrapper we use to perform partial aggregation on data nodes. It is a
pure function that produces serialized aggregation state as a result.
Being pure, it doesn't influence parallel safety. This means we don't
need to mark it parallel-unsafe to artificially disable the parallel
plans for partial aggregation. They will be chosen as usual based on
the parallel-safety of the underlying aggregate function.
The copy test is flaky because some test data is generated
dynamically based on the current date. This patch changes the data
generation to a time series with fixed dates.
When dealing with Intervals with month component timezone changes
can result in multiple day differences in the outcome of these
calculations due to different month lengths. When dealing with
months we add a 7 day safety buffer.
For all these calculations it is fine if we exclude less chunks
than strictly required for the operation, additional exclusion
with exact values will happen in the executor. But under no
circumstances must we exclude too much cause there would be
no way for the executor to get those chunks back.
The initial patch to use now() expressions during planner hypertable
expansion only supported intervals with no day or month component.
This patch adds support for intervals with day component.
If the interval has a day component then the calculation needs
to take into account daylight saving time switches and thereby a
day would not always be exactly 24 hours. We mitigate this by
adding a safety buffer to account for these dst switches when
dealing with intervals with day component. These calculations
will be repeated with exact values during execution.
Since dst switches seem to range between -1 and 2 hours we set
the safety buffer to 4 hours.
This patch also refactors the tests since the previous tests
made it hard to tell the feature was working after the constified
values have been removed from the plans.
