This commit gives more visibility into job failures by making the
information regarding a job runtime error available in an extension
table (`job_errors`) that users can directly query.
This commit also adds an infromational view on top of the table for
convenience.
To prevent the `job_errors` table from growing too large,
a retention job is also set up with a default retention interval
of 1 month. The retention job is registered with a custom check
function that requires that a valid "drop_after" interval be provided
in the config field of the job.
Depending on the statistics target, running ANALYZE on a chunk before
compression can cause a lot of random IO operations for chunks that
are bigger than the number of pages ANALYZE needs to read. By moving
that operation after the heap is loaded into memory for sorting,
we increase the chance of hitting cache and reducing disk operations
necessary to execute compression jobs.
When compressing larger chunks, compression sort tends to use
temporary files since memory limits (`work_mem`) are usually
pretty small to fit all the data into memory. On the other hand,
using `maintenance_work_mem` makes more sense since its generally
safer to use a larger value without impacting general resource usage.
When reading the job status table `bgw_job_stat` and after that
updating it, locks where released after the read, allowing a competing
session to update the job status and trigger a concurrent update error
either in a session doing the update or in the scheduler. Since the
scheduler does not recover after aborting with an error, this caused
the background worker subsystem to stop and not start new jobs.
This commit fixes this by upgrading `RowExclusiveLock` to
`ShareRowExclusiveLock` to ensure that not two sessions tries to update
the row at the same time, remove an initial speculative lock
that are taken when a job status row can be added, and also keeps the
lock until the end of the transaction to prevent other sessions to
update. Since these updating transactions are short, it should not
cause other threads to block long.
Fixes#4293
Consider a hypertable which has a foreign key constraint on a
referenced table, which is a parititioned table. In such case, foreign
key constraint is duplicated for each of the partitioned table. When
we insert into a hypertable, we end up checking the foreign key
constraint multiple times, which obviously leads to foreign key
constraint violation. Instead we only check foreign key constraint of
the parent table of the partitioned table.
Fixes#4684
To allow perltidy to be used from editors and IDEs, the perltidyrc file
is moved from `scripts/perltidyrc` to `.perltidyrc`. This will allow
editors and IDEs to quickly find the style file in the root of the
repository, where `perltidy` normally searches for the style file.
The workflow is then updated to use the new location, and the two
options `--backup-file-extension` and `--backup-and-modify-in-place`
are moved to the workflow file to allow editors and IDEs to use
whatever method they find useful to process the file.
After migrate a Continuous Aggregate from the old format to the new
using `cagg_migrate` procedure we end up with the following problems:
* Refresh policy is not copied from the OLD to the NEW cagg;
* Compression setting is not copied from the OLD to the NEW cagg.
Fixed it by properly copying the refresh policy and setting the
`timescaledb.compress=true` flag to the new CAGG.
Fix#4710
The constify code constifying TIMESTAMPTZ expressions when doing
chunk exclusion did not account for daylight saving time switches
leading to different calculation outcomes when timezone changes.
This patch adds a 4 hour safety buffer to any such calculations.
Instances upgraded to 2.8.0 will end up with a wrong check constraint
in catalog table `continuous_aggregate_migrate_plan_step`.
Fixed it by removing and adding the constraint with the correct checks.
Fix#4727
In af8e3c6b12035dd45bc7bf61e9420489cfa56eaa, an GitHub action to assign
issues to a project column has been introduced. However, this action
does not work with organization-wide projects. This patch replaces the
used project management action with a more recent one.
This patch fixes a deadlock between chunk decompression and SELECT
queries executed in parallel. The change in
a608d7db614c930213dee8d6a5e9d26a0259da61 requests an AccessExclusiveLock
for the decompressed chunk instead of the compressed chunk, resulting in
deadlocks.
In addition, an isolation test has been added to test that SELECT
queries on a chunk that is currently decompressed can be executed.
Fixes#4605
Very recent compilers will warn about function pointers with empty
argument list. While currently in C func() means a function with
an unspecified argument list the next version of the C standard will
change this to mean func(void).
Since PG 12.3, the syntax of the ereport() function has changed
slightly. The change 97a603fe5c5ba25aa6e4e596fd7da7a401051d7b introduces
an ereport() call that only works using newer PG versions. This PR
changes the ereport() call to a PG 12.0 compatible syntax.
The patch #4714 introduces the extraction of the running query from a
coredump by calling gdb with two commands. These commands are separated
by a newline character. This character may not always be correctly
converted to a new line before being processed by gdb, resulting in the
following error:
Invalid character '\' in expression.
This PR ensures the proper handling of the newline character.
Fix the 'Waiting for Engineering' job. Issues that contain the label
'need-more-info' that receive an issue_comment are automatically
moved to the 'Waiting for Engineering' column on the bug board and
the 'need-more-info' is removed.
Consider a compressed hypertable has many columns (like more than 600 columns).
In call to compress_chunk(), the compressed tuple size exceeds, 8K which causes
error as "row is too big: size 10856, maximum size 8160."
This patch estimates the tuple size of compressed hypertable and reports a
warning when compression is enabled on hypertable. Thus user gets aware of
this warning before calling compress_chunk().
Fixes#4398
The code added to support VIEWs did not account for the fact that
varno could be from a different nesting level and therefore not
be present in the current range table.
Use DROP DATABASE WITH(FORCE) to drop the database in pg_dump test
since occasionally there would still be connections to the database
leading to test failures. Unfortunately PG12 does not support that
syntax so we have to drop without that option on PG12.
Don't include the used database ports into test output as this
will lead to failing tests when running against a local instance
or against a preconfigured cloud instance.
Using multiple different configurations in a single target will
not work when running against a local instance or when running
against a preconfigured cloud instance. With recent adjustments
to the test cleanup this should not be needed anymore and if we
really need different configuration we should make it a separate
target to make it compatible with instances configured outside
of pg_regress.
This patch changes the regression test schedule generation to only
completely rebuild the schedule when TESTS is specified. In all
other cases the existing schedule is amended. This means any
parallelity present in the cmake generated schedule will be kept.
Previously the pgspot call used in CI did not properly check the
update script. Unsafe function creations for functions with changed
signatures could go undetected in the update script.
Patch #4425 introduced regression test failures, namely, a crash
in function `ts_bgw_job_update_by_id`. The failures are due to the
COMMIT statement in the custom check procedure. This patch removes
that particular test case from bgw_custom.
The out of background worker test in bgw_db_scheduler is flaky and
fails very often, especially in the 32bit environment and on windows.
This patch removes that specific test from bgw_db_scheduler. If we
want to test this specific part of the scheduler this should be
better rewritten in an isolation test.
Removed the underline character prefix '_' from the parameter names of
the procedure `cagg_migrate`. The new signature is:
cagg_migrate(
IN cagg regclass,
IN override boolean DEFAULT false,
IN drop_old boolean DEFAULT false
)
Timescale 2.8 released a migration path from the old format of
Continuous Aggregate to the new format (#4552).
Unfortunately it lacks of proper tests when a non-superuser execute the
migration. For a non-superuser execute the migration properly it
requires SELECT/INSERT/UPDATE permissions in some catalog objects:
* _timescaledb_catalog.continuous_agg_migrate_plan
* _timescaledb_catalog.continuous_agg_migrate_plan_step
* _timescaledb_catalog.continuous_agg_migrate_plan_step_step_id_seq
Improved the regression tests to cover the lack of permissions in the
catalog objects for non-superusers.
Allow planner chunk exclusion in subqueries. When we decicde on
whether a query may benefit from constifying now and encounter a
subquery peek into the subquery and check if the constraint
references a hypertable partitioning column.
Fixes#4524
The schema of base table on which hypertables are created, should define
columns with proper data types. As per postgres best practices Wiki
(https://wiki.postgresql.org/wiki/Don't_Do_This), one should not define
columns with CHAR, VARCHAR, VARCHAR(N), instead use TEXT data type.
Similarly instead of using timestamp, one should use timestamptz.
This patch reports a WARNING to end user when creating hypertables,
if underlying parent table, has columns of above mentioned data types.
Fixes#4335
We don't have to look up the dimension slices for dimensions for which
we don't have restrictions.
Also sort chunks by ids before looking up the metadata, because this
gives more favorable table access patterns (closer to sequential).
This fixes a planning time regression introduced in 2.7.
This patch adjusts the operator logic for valid space dimension
constraints to no longer look for an exact match on both sides
of the operator but instead allow mismatched datatypes.
Previously a constraint like `col = value` would require `col`
and `value` to have matching datatype with this change `col` and
`value` can be different datatype as long as they have equality
operator in btree family.
Mismatching datatype can happen commonly when using int8 columns
and comparing them with integer literals. Integer literals default
to int4 so the datatypes would not match unless special care has
been taken in writing the constraints and therefore the optimization
would never apply in those cases.
Since we do not use our own hypertable expansion for SELECT FOR UPDATE
queries we need to make sure to add the extra information necessary to
get hashed space partitions with the native postgres inheritance
expansion working.
The primary key for compression_chunk_size was defined as chunk_id,
compressed_chunk_id but other places assumed chunk_id is actually
unique and would error when it was not. Since it makes no sense
to have multiple entries per chunk since that reference would be
to a no longer existing chunk the primary key is changed to chunk_id
only with this patch.
This patch adds a new time_bucket_gapfill function that
allows bucketing in a specific timezone.
You can gapfill with explicit timezone like so:
`SELECT time_bucket_gapfill('1 day', time, 'Europe/Berlin') ...`
Unfortunately this introduces an ambiguity with some previous
call variations when an untyped start/finish argument was passed
to the function. Some queries might need to be adjusted and either
explicitly name the positional argument or resolve the type ambiguity
by casting to the intended type.
