This release contains bug fixes since the 2.9.0 release.
This release is high priority for upgrade. We strongly recommend that you
upgrade as soon as possible.
**Bugfixes**
* #5072 Fix CAgg on CAgg bucket size validation
* #5101 Fix enabling compression on caggs with renamed columns
* #5106 Fix building against PG15 on Windows
* #5117 Fix postgres server restart on background worker exit
* #5121 Fix privileges for job_errors in update script
This patch changes INSERTs into compressed chunks to no longer
be immediately compressed but stored in the uncompressed chunk
instead and later merged with the compressed chunk by a separate
job.
This greatly simplifies the INSERT-codepath as we no longer have
to rewrite the target of INSERTs and on-the-fly compress leading
to a roughly 2x improvement on INSERT rate into compressed chunk.
Additionally this improves TRIGGER-support for INSERTs into
compressed chunks.
This is a necessary refactoring to allow UPSERT/UPDATE/DELETE on
compressed chunks in follow-patches.
This commit enables PG15 in the following workflows:
- Regression Linux
- ABI test
- Memory tests
- Coverity
- SQLSmith
- Additional cron tests
Co-authored-by: Bharathy Satish <bharathy@timescale.com>
The bucket size of a Continuous Aggregate should be greater or equal to
the parent Continuous Aggregate because there are many cases where you
actually want to roll up on another dimension.
When defining compression segment by parameter using multiple columns,
the parameter ordering is not respected for index creation.
This patch fixes the issue by maintaining the same order in which user
has defined columns in segment by clause.
Fixes#5104
Use rand() instead of random() cause the latter is not available
on Windows and postgres stopped backporting it with PG15.
Ideally we switch to the crypto functions added in PG15 but that
requires a bit more work and this is the minimal change required
to get it to build against PG15 on Windows.
Add 2.9.0 to update test scripts and adjust downgrade scripts for
2.9.0. Additionally adjust CHANGELOG to sync with the actual release
CHANGELOG and add PG15 to CI.
On caggs with realtime aggregation changing the column name does
not update all the column aliases inside the view metadata.
This patch changes the code that creates the compression
configuration for caggs to get the column name from the materialization
hypertable instead of the view internals.
Fixes#5100
When CAggs on CAggs was introduced in commit 3749953 the regression
tests was splited into 6 (six) different test suites.
Simplyfied it grouping tests and reduced to just 2 (two) different test
suites. It save resources and time because each suite test spawn it own
Postgres instance.
The last minor versions for PG14 (14.6) and PG15 (15.1) were unlisted by
chocolatey maintainers due to some issues.
Fixed it by hardcoding the 14.5 until the packages become available
again.
When CAggs migration was introduced in commit e34218ce the regression
tests was splited into 6 (six) different test suites.
Simplyfied it grouping tests and reduced to just 2 (two) different test
suites. It save resources and time because each suite test spawn it own
Postgres instance.
The tree contains a lot of design and architecture documents, but they
are not linked together, so this commits adds a few additional README
and build a basic structure for the documentation.
It allows to override tuplesort with indexscan
if compression setting keys matches with Index keys.
Moreover this feature has Enable/Disable Toggle.
To Disable from the client use the following command,
SET timescaledb.enable_compression_indexscan = 'OFF'
Planner can decide to use BitmapHeapScans exclusively when
scanning compressed chunks. Since we filter out such scans
due to previously reported issues, this can lead to no
query plan devised when querying compressed chunks. This
change allows Bitmap scans on compressed chunks only when
it is not parameterized.
Fixes#5090
The cursor_fetcher_rewind method assumes that the data node cursor is
rewind either after eof or when there is an associated request. But the
rewind can also occur once the server has generated required number of
rows by joining the relation being scanned with another regular
relation. In this case, the fetch would not have reached eof and there
will be no associated requests as the rows would have been already
loaded into the cursor causing the assertion in cursor_fetcher_rewind
to fail. Fixed that by removing the Assert and updating
cursor_fetcher_rewind to discard the response only if there is an
associated request.
Fixes#5053
The new permissions checks to fix CVE-2022-29187 in Git caused some
issues in our CI pipeline. This patch adds the checkout directory to
Git's "safe.directory" setting.
When popping the hypertable cache stack, it might happen that the
hypertable cache was invalidated between the push and the pop. In that
case, the baserel cache can contain invalid entries pointing to the now
popped hypertable cache, so we reset the baserel cache.
Fixes#4795
Issue occurs in extended query protocol mode only where every
query goes through PREPARE and EXECUTE phase. First time ANALYZE
is executed, a list of relations to be vaccumed is extracted and
saved in a list. This list is referenced in parsetree node. Once
execution of ANALYZE is complete, this list is cleaned up, however
reference to the same is not cleaned up in parsetree node. When
second time ANALYZE is executed, segfault happens as we access an
invalid memory location.
Fixed the issue by restoring the actual value in parsetree node
once ANALYZE completes its execution.
Fixes#4857
When the sanitizer is active, the tests require a lot of memory. If they
are run in large parallel groups, out-of-memory situations can occur.
This patch reduces the size of parallel groups to 5 when the sanitizer
is active.
This patch contains two changes to the Sanitizer checks:
(1) All logfiles of the Sanitizer will be uploaded to the
CI database.
(2) The Sanitizer checks are executed on every PR.
The `dist_hypertable` test needs a lot of memory, especially when the
sanitizer is enabled. This patch runs this test as a `SOLO_TEST`. This
ensures that PostgreSQL does not run into an out-of-memory situation.
So far, we have treated the 'log_path' setting of the sanitizer like a
file. In fact, this value is used as a prefix for the created log file.
Since we expected the exact file name when uploading the sanitizer
output, this file was not found and we lost the messages of the
sanitizer. This PR changes the behavior. We now treat the setting as a
prefix and upload all files created in a new sanitizer output folder.
In remote_txn, testcases which kill remote processes on data nodes,
tend to rollback transactions and as part of the process, WARNINGS/ERROR
are reported to client. Client however reports WARNINGS/ERROR in different
order intermittently. This is an issue specific to psql utility. Placing
psql in gdb and trying to diagnise the problem does not reproduce the issue.
This patch fixes the tests by not reporting WARNINGS.
Fixes#4837
Ensure the COPY fetcher implementation reads data until EOF with
`PQgetCopyData()`. Also ensure the malloc'ed copy data is freed with
`PQfreemem()` if an error is thrown in the processing loop.
Previously, the COPY fetcher didn't read until EOF, and instead
assumed EOF when the COPY file trailer is received. Since EOF wasn't
reached, it required terminating the COPY with an extra call to the
(deprecated) `PQendcopy()` function.
Still, there are cases when a COPY needs to be prematurely terminated,
for example, when querying with a LIMIT clause. Therefore, distinguish
between "normal" end (when receiving EOF) and forceful end (cancel the
ongoing query).
This release adds major new features since the 2.8.1 release.
We deem it moderate priority for upgrading.
This release includes these noteworthy features:
* Hierarchical Continuous Aggregates (aka Continuous Aggregate on top of another Continuous Aggregate)
* Improve `time_bucket_gapfill` function allowing specifying timezone to bucket
* Use `alter_data_node()` to change the data node configuration. This function introduces the option to configure the availability of the data node.
This release also includes several bug fixes.
**Features**
* #4476 Batch rows on access node for distributed COPY
* #4567 Exponentially backoff when out of background workers
* #4650 Show warnings when not following best practices
* #4664 Introduce fixed schedules for background jobs
* #4668 Hierarchical Continuous Aggregates
* #4670 Add timezone support to time_bucket_gapfill
* #4678 Add interface for troubleshooting job failures
* #4718 Add ability to merge chunks while compressing
* #4786 Extend the now() optimization to also apply to CURRENT_TIMESTAMP
* #4820 Support parameterized data node scans in joins
* #4830 Add function to change configuration of a data nodes
* #4966 Handle DML activity when datanode is not available
* #4971 Add function to drop stale chunks on a datanode
**Bugfixes**
* #4663 Don't error when compression metadata is missing
* #4673 Fix now() constification for VIEWs
* #4681 Fix compression_chunk_size primary key
* #4696 Report warning when enabling compression on hypertable
* #4745 Fix FK constraint violation error while insert into hypertable which references partitioned table
* #4756 Improve compression job IO performance
* #4770 Continue compressing other chunks after an error
* #4794 Fix degraded performance seen on timescaledb_internal.hypertable_local_size() function
* #4807 Fix segmentation fault during INSERT into compressed hypertable
* #4822 Fix missing segmentby compression option in CAGGs
* #4823 Fix a crash that could occur when using nested user-defined functions with hypertables
* #4840 Fix performance regressions in the copy code
* #4860 Block multi-statement DDL command in one query
* #4898 Fix cagg migration failure when trying to resume
* #4904 Remove BitmapScan support in DecompressChunk
* #4906 Fix a performance regression in the query planner by speeding up frozen chunk state checks
* #4910 Fix a typo in process_compressed_data_out
* #4918 Cagg migration orphans cagg policy
* #4941 Restrict usage of the old format (pre 2.7) of continuous aggregates in PostgreSQL 15.
* #4955 Fix cagg migration for hypertables using timestamp without timezone
* #4968 Check for interrupts in gapfill main loop
* #4988 Fix cagg migration crash when refreshing the newly created cagg
**Thanks**
* @jflambert for reporting a crash with nested user-defined functions.
* @jvanns for reporting hypertable FK reference to vanilla PostgreSQL partitioned table doesn't seem to work
* @kou for fixing a typo in process_compressed_data_out
* @xvaara for helping reproduce a bug with bitmap scans in transparent decompression
* @byazici for reporting a problem with segmentby on compressed caggs
* @tobiasdirksen for requesting Continuous aggregate on top of another continuous aggregate
* @xima for reporting a bug in Cagg migration
Not specifying alpine version causes libssl version
to change, which in turn cause error in downgrade tests
as well as ABI tests.
This commit also fixes shellcheck failures.
Some failing windows tests are addd to ignore list.
Co-authored-by: Lakshmi Narayanan Sreethar <lakshmi@timescale.com>
Co-authored-by: Alexander Kuzmenkov <akuzmenkov@timescale.com>
Signed-off-by: Sachin <sachin@timescale.com>
RelationGetSmgr was backported by upstream to the STABLE branches
but is not yet available in any released version so we cannot use
pg version to determine presence of RelationGetSmgr.
The telemetry process runs on a regular basis and usually does not make
a lot of noise, but in a few particular cases, it writes entries to the
log unnecessarily.
If the telemetry server cannot be contacted, it will print a warning in
the log that the server cannot be contacted. Since it is nothing wrong
with the system and the telemetry process will try to re-connect at a
later time, it is unnecessary to print as a warning.
If the telemetry response is malformed, a warning is printed. This is
also unnecessary since there is nothing wrong with the system, there is
nothing the user can do about it, and this warning can be largely
ignored.
If the hard-coded telemetry scheme is incorrect, a warning will be
printed. This should not normally happen, and if it happens on a
running server, there is nothing that can be done to eliminate the
error message and the message is unnecessary.
When the telemetry job exits, a standard termination message is
printed in the log. Although harmless, it is mostly confusing and
provide no value to the user.
If the telemetry process is attempting to connect, or is connected, to
the telemetry server, the telemetry server will wait until the
connection gets a timeout before shutting down. This is unnecessary
since there is no critical problem in aborting the connection and doing
a direct shutdown.
This commit turns those warnings into notices and installs a signal
handler so that the telemetry job exits silently and abort any
outstanding connections.
Fixes#4028
rd_smgr should not be accessed directly but RelationGetSmgr should
be used instead. Accessing it directly can lead to segfaults when
parallel relcache flushes are happening.
f10f0ae420
When trying to create cagg on top of any relation that is a neither
a hypertable nor a continuous aggregate the command would segfault.
This patch changes the code to handle this case gracefully and error
out when trying to create a cagg on top of a relation that is not
supported. Found by coverity.
Commit 3749953e introduce Hierarchical Continuous Aggregates (aka
Continuous Aggregate on top of another Continuous Aggregate) but it
lacks of some basic validations.
Validations added during the creation of a Hierarchical Continuous
Aggregate:
* Forbid create a continuous aggregate with fixed-width bucket on top of
a continuous aggregate with variable-width bucket.
* Forbid incompatible bucket widths:
- should not be equal;
- bucket width of the new continuous aggregate should be greater than
the source continuous aggregate;
- bucket width of the new continuous aggregate should be multiple of
the source continuous aggregate.
The sqlsmith and coverity workflows used the cache postgres build
but could not produce a build by themselves and therefore relied
on other workflows to produce the cached binaries. This patch
changes those workflows to use normal postgres packages instead
of custom built postgres to remove that dependency.
This patch changes the code that blocks frozen chunk
modifications to no longer use triggers but to use custom
node instead. Frozen chunks is a timescaledb internal object
and should therefore not be protected by TRIGGER which is
external and creates several hazards. TRIGGERs created to
protect internal state contend with user-created triggers.
The trigger created to protect frozen chunks does not work
well with our restoring GUC which we use when restoring
logical dumps. Thirdly triggers are not functional for any
internal operations but are only working in code paths that
explicitly added trigger support.
Apply date_trunc to last_successful_finish.
Commit 20cdd9ca3ed0c2d62779c4fc61d278a489b4460a mostly fixed
the flakiness, but date_trunc was not
applied to the last_successful_finish
so we still got some flaky runs.
If a datanode goes down for whatever reason then DML activity to
chunks residing on (or targeted to) that DN will start erroring out.
We now handle this by marking the target chunk as "stale" for this
DN by changing the metadata on the access node. This allows us to
continue to do DML to replicas of the same chunk data on other DNs
in the setup. This obviously will only work for chunks which have
"replication_factor" > 1. Note that for chunks which do not have
undergo any change will continue to carry the appropriate DN related
metadata on the AN.
This means that such "stale" chunks will become underreplicated and
need to be re-balanced by using the copy_chunk functionality by a micro
service or some such process.
Fixes#4846
