Dimension partition lookups use binary search to find the partition to
place a chunk in. However, in the code, an array of partitions might
not be sorted because the sort happened on a copy of the array instead
of the main array. This change fixes the issue to ensure the array is
sorted and binary search works properly.
For some unexpected conditions, we have a check and an error that is
generated. Since this always generate an error, it is more difficult to
find the bug if the error is generated rather than an assert fired
generating a core dump. Similarly, some asserts can occur in production
builds and will lead to strange situations triggering a crash. For
those cases we should instead generate an error.
This commit introduces a macro `Ensure` that will result in an assert
in debug builds, but an error message in release build. This macro
should only be used for conditions that should not occur during normal
runtime, but which can happen is odd corner-cases in release builds and
therefore warrants an error message.
It also replaces some existing checks with such errors to demonstrate
usage.
External components like timescaledb_cloudutils might want to add
additional options or do additional ssl related processing. They can
do so by implementing a hook and then assigning it to a timescaledb
variable to allow timescale to invoke it as appropriate.
When TimescaleDB 2.9.0 compiled against PG15, many tests fail due to
permissions on default public schema not getting propagated to data
nodes, although connected user has required GRANTS on access nodes.
This patch fixes failing tests by explicitly granting required permissions
for connected user or role after data nodes are added.
Currently, the next start of a scheduled background job is
calculated by adding the `schedule_interval` to its finish
time. This does not allow scheduling jobs to execute at fixed
times, as the next execution is "shifted" by the job duration.
This commit introduces the option to execute a job on a fixed
schedule instead. Users are expected to provide an initial_start
parameter on which subsequent job executions are aligned. The next
start is calculated by computing the next time_bucket of the finish
time with initial_start origin.
An `initial_start` parameter is added to the compression, retention,
reorder and continuous aggregate `add_policy` signatures. By passing
that upon policy creation users indicate the policy will execute on
a fixed schedule, or drifting schedule if `initial_start` is not
provided.
To allow users to pick a drifting schedule when registering a UDA,
an additional parameter `fixed_schedule` is added to `add_job`
to allow users to specify the old behavior by setting it to false.
Additionally, an optional TEXT parameter, `timezone`, is added to both
add_job and add_policy signatures, to address the 1-hour shift in
execution time caused by DST switches. As internally the next start of
a fixed schedule job is calculated using time_bucket, the timezone
parameter allows using timezone-aware buckets to calculate
the next start.
Timescale 2.7 released a new version of Continuous Aggregate (#4269)
that store the final aggregation state instead of the byte array of
the partial aggregate state, offering multiple opportunities of
optimizations as well a more compact form.
This new version also removes the unecessary `chunk_id` column from the
materialization hypertable and consequently the re-aggregation in the
user view. It means the user view that query the materialization
hypertable don't have a GROUP BY clause anymore that was problematic
for query performance.
Before 2.7 when users turn compression ON we infer compression options
`segmentby` and `orderby` based on the GROUP BY clause and time bucket
respectively. With the new version without a GROUP BY clause in the user
view the inferetion for the 'segmentby' compression option stopped to
work.
Fixed it by changing the code to the compression on the new version of
Continuous Aggregate (aka finals form) behave the same as the old
version.
Fix#4816
This name better reflects its characteristics, and I'm thinking about
resurrecting the old row-by-row fetcher later, because it can be useful
for parameterized queries.
INSERT into compressed hypertable with number of open chunks greater
than ts_guc_max_open_chunks_per_insert causes segementation fault.
New row which needs to be inserted into compressed chunk has to be
compressed. Memory required as part of compressing a row is allocated
from RowCompressor::per_row_ctx memory context. Once row is compressed,
ExecInsert() is called, where memory from same context is used to
allocate and free it instead of using "Executor State". This causes
a corruption in memory.
Fixes: #4778
Include OOM kill event logs into error printout. Previously these
would only be visible by inspecting the postgres log and looking
for killed by signal 9.
The only configuration we're missing is the newline for braces after
case labels. The rest of the differences looks like bugs/omissions of
the version 8 that we use now.
Require clang-format-14 in cmake and use it in the CI check. We can't
support versions earlier than 14 because they have some
formatting differences that can't be configured.
Group the incoming rows into batches on access node before COPYing to
data nodes. This gives 2x-5x speedup on various COPY queries to
distributed hypertables.
Also fix the text format passthrough, and prefer text transfer format
for text input to be able to use this passthrough. It saves a lot of
CPU on the access node.
By default pg_isready only waits for 3 seconds before giving up
which is occasionally not enough in the windows tests. This patch
bumps the timeout up to 30 seconds which should be plenty to have
the cluster start up under all circumstances.
Recently we fixed a DDL error (#4739) after upgrading to 2.8.0 version
that surprisly the CI upgrade/dowgrade tests didn't complained during
the development of the feature (#4552).
Fixed it by adding an specific query in the `post.catalog.sql` script to
make sure we'll check all the constraints of our internal tables and
catalog.
Update scripts should not use ADD/DROP/RENAME and always rebuild
catalog tables to ensure the objects are identical between new
install, upgrade and downgrade.
Changed queries to use LATERAL join on size functions and views instead
of CTEs and it eliminate a lot of unnecessary projections and give a
chance for the planner to push-down predicates.
Closes#4775
The optimization that constifies certain now() expressions before
hypertable expansion did not apply to CURRENT_TIMESTAMP even
though it is functionally similar to now(). This patch extends the
optimization to CURRENT_TIMESTAMP.
This release is a patch release. We recommend that you upgrade at the
next available opportunity.
**Bugfixes**
* #4454 Keep locks after reading job status
* #4658 Fix error when querying a compressed hypertable with compress_segmentby on an enum column
* #4671 Fix a possible error while flushing the COPY data
* #4675 Fix bad TupleTableSlot drop
* #4676 Fix a deadlock when decompressing chunks and performing SELECTs
* #4685 Fix chunk exclusion for space partitions in SELECT FOR UPDATE queries
* #4694 Change parameter names of cagg_migrate procedure
* #4698 Do not use row-by-row fetcher for parameterized plans
* #4711 Remove support for procedures as custom checks
* #4712 Fix assertion failure in constify_now
* #4713 Fix Continuous Aggregate migration policies
* #4720 Fix chunk exclusion for prepared statements and dst changes
* #4726 Fix gapfill function signature
* #4737 Fix join on time column of compressed chunk
* #4738 Fix error when waiting for remote COPY to finish
* #4739 Fix continuous aggregate migrate check constraint
* #4760 Fix segfault when INNER JOINing hypertables
* #4767 Fix permission issues on index creation for CAggs
**Thanks**
* @boxhock and @cocowalla for reporting a segfault when JOINing hypertables
* @carobme for reporting constraint error during continuous aggregate migration
* @choisnetm, @dustinsorensen, @jayadevanm and @joeyberkovitz for reporting a problem with JOINs on compressed hypertables
* @daniel-k for reporting a background worker crash
* @justinpryzby for reporting an error when compressing very wide tables
* @maxtwardowski for reporting problems with chunk exclusion and space partitions
* @yuezhihan for reporting GROUP BY error when having compress_segmentby on an enum column
The test was flaky because the scheduler was launching the scheduled
jobs, therefore their next_start field could differ depending on
whether they had finished executing.
Fixed by not selecting the next_start field in alter_job calls,
when it is not of interest.
Fixes#4719
This patch adds a membership check to the 'Waiting for Engineering'
workflow. The check ensures that the workflow is only triggered by
external users. Without this check, when someone from the database
engineering team responded to an issue, the "need-more-info" flag was
removed and the issue was erroneously moved to the "waiting for
engineering" column.
When a compression_policy is executed by a background worker, the policy
should continue to execute even if compressing or decompressing one of
the chunks fails.
Fixes: #4610
Calling `ts_dist_cmd_invoke_on_data_nodes_using_search_path()` function
without an active transaction allows connection invalidation event
happen between applying `search_path` and the actual command
execution, which leads to an error.
This change introduces a way to ignore connection cache invalidations
using `remote_connection_cache_invalidation_ignore()` function.
This work is based on @nikkhils original fix and the problem research.
Fix#4022
Change the pgspot check to use downgrade_to_version instead of
update_from_version.
We use downgrade_to_version instead of update_from_version because
when the release PR for a new version has been merged to main
but the version is not tagged yet update_from will not exist yet.
In all other situations update_from_version and downgrade_to_version
should point to the same version.
The materialised hypertable resides in the _timescaledb.internal schema
which resulted in permission error at the time of manual index creation
by non super user. To solve this, it now switches to timescaledb user
before index creation of CAgg.
Fixes#4735
This commit gives more visibility into job failures by making the
information regarding a job runtime error available in an extension
table (`job_errors`) that users can directly query.
This commit also adds an infromational view on top of the table for
convenience.
To prevent the `job_errors` table from growing too large,
a retention job is also set up with a default retention interval
of 1 month. The retention job is registered with a custom check
function that requires that a valid "drop_after" interval be provided
in the config field of the job.
Depending on the statistics target, running ANALYZE on a chunk before
compression can cause a lot of random IO operations for chunks that
are bigger than the number of pages ANALYZE needs to read. By moving
that operation after the heap is loaded into memory for sorting,
we increase the chance of hitting cache and reducing disk operations
necessary to execute compression jobs.
