The view uses cached information from compression_chunk_size to
report the size of compressed chunks. Since compressed chunks
can be modified, we call pg_relation_size on the compressed chunk
while reporting the size
The view also incorrectly used the hypertable's reltoastrelid to
calculate toast bytes. It has been changed to use the chunk's
reltoastrelid.
GitHub only has the last 3 clang versions installed by default in
the virtual environment and no longer installed clang-9 when they
added clang-12. Since we need clang-9 to build postgres with
llvm support we need to explicitly install the required package.
https://github.com/actions/virtual-environments/pull/3381
Use subclassing to inherit from `PostgresNode` and create a hierarchy
containing `AccessNode` and `DataNode` to simplify creating tests with
multiple nodes.
Also, two new functions are added:
`TimescaleNode::create`: Creates the new node by calling
`get_new_node`, `init` and `start` in that order.
`AccessNode::add_data_node`: Adds a new data node to the access node.
Also rewrite the test to use the new hierarchy.
Add a run of `perltidy` to make sure that all the Perl code is
correctly formatted. Also fix the formatting of one file that did not
follow the style in `scripts/perltidyrc`.
chunk_scan_internal is called with 2 callback functions, a
filter function and a tuple_found function that
are invoked with the same argument. ts_chunk_set_compressed
and ts_chunk_clear_compressed use chunk_tuple_dropped_filter
but do not pass in the required ChunkStubScanCtx* as argument.
Instead an argument of type int* is passed in. This causes
a segmentation fault.
Add a new chunk filter function that is compatible with
the tuple_found function used by ts_chunk_clear_compressed_chunk
and ts_chunk_set_compressed_chunk
Fixes#3158
When creating a constraint trigger on a hypertable the command
succeeds and the constraint trigger works correctly for existing
chunks but any chunk creation after that would fail with an
error, because the constraint trigger was treated like a normal
constraint. But since pg_get_constraintdef does not return a SQL
command for constraint triggers the corresponding command that was
created would throw an error.
Fixes#3235
GitHub removed default installation of clang-9 from the ubuntu
images but those are required to build postgres with llvm so we have
to explicitly install the required packages.
https://github.com/actions/virtual-environments/pull/3381
decompress_chunk_plan_create used get_actual_clauses to extract
RestrictInfo clauses and adds them as quals. This function is only
supposed to be used when none of the RestrictInfos are pseudoconstant
leading to an assertion failure when the query has pseudoconstant
quals.
Fixes#3241
The TAP testing PG framework meets our requirements for doing
multinode/stand-alone-binaries testing for scenarios which need
individual multiple PG instances (multi-node testing is a prime
example).
This commit adds the necessary wrappers to allow the use of the TAP
testing framework in timescale code base. The README in "test/perl"
directory gives a fair idea of how to write tap tests. A simple tap
test has been added to provide a reference point for developers to
write new ones.
One can go to "build/tsl/test" and invoke "make checkprove" to see
the tap test in action.
Also includes changes for enabling github CI to run these taptests by
installing the requisite dependencies.
Includes changes to license checking scripts to handle new *.pl and
*.pm files.
Also added a new scripts/perltidyrc to aid in formatting of these
files.
The timescaledb_information.chunks view used to return NULL in the
is_compressed field for distributed chunks. This changes the view to
always return true or false, depending on the chunk status flags in the
Access Node. This is a hint and cannot be used as a source of truth for
the compression state of the actual chunks in the Data Nodes.
Change CI tests to run against pg 11.12, 12.7 and 13.3.
The latest postgres versions adjusts the event_trigger test to
output the list of currently defined event triggers making the
test output different when timescaledb is installed since we
define our own event triggers. Due to this change we have to
ignore the result of this test in our CI.
https://github.com/postgres/postgres/commit/f3c45378
Grants and revokes where not propagated to compressed hypertables, if
the hypertable had a compressed hypertable, meaning that `pg_dump` and
`pg_restore` would not be able to dump nor restore the compressed part
of a hypertable unless they were owners and/or was a superuser.
This commit fixes this by propagating grants and revokes on a
hypertable to the associated compressed hypertable, if one exists,
which will then include the compressed hypertable and the associated
chunks in the grant and revoke execution.
It also adds code to fix the permissions of compressed hypertables and
all associated chunks in an update and adds an update test to check
that the permissions match.
Fixes#3209
A new custom plan/executor node is added that implements distributed
INSERT using COPY in the backend (between access node and data
nodes). COPY is significantly faster than the existing method that
sets up prepared INSERT statements on each data node. With COPY,
tuples are streamed to data nodes instead of batching them in order to
"fill" a configured prepared statement. A COPY also avoids the
overhead of having to plan the statement on each data node.
Using COPY doesn't work in all situations, however. Neither ON
CONFLICT nor RETURNING clauses work since COPY lacks support for
them. Still, RETURNING is possible if one knows that the tuples aren't
going to be modified by, e.g., a trigger. When tuples aren't modified,
one can return the original tuples on the access node.
In order to implement the new custom node, some refactoring has been
performed to the distributed COPY code. The basic COPY support
functions have been moved to the connection module so that switching
in and out of COPY_IN mode is part of the core connection
handling. This allows other parts of the code to manage the connection
mode, which is necessary when, e.g., creating a remote chunk. To
create a chunk, the connection needs to be switched out of COPY_IN
mode so that regular SQL statements can be executed again.
Partial fix for #3025.
This change makes it possible to create and use triggers on
distributed hypertables. The DDL for `CREATE TRIGGER` and `DROP
TRIGGER` are now forwarded to data nodes. Previously, `CREATE TRIGGER`
worked, but didn't forward the DDL command.
Note that triggers exist only on the data nodes and on the data node's
root hypertable. Triggers need to execute on the data nodes where data
is stored; in particular, row-based "before" triggers might modify the
tuple. On the access node, chunks are foreign tables and such tables
do not support triggers.
Also note that statement-level triggers execute once on each node
involved, including the access node. So, if, e.g., two rows are
inserted into two different data nodes in the same statement, a
statement-level insert trigger will fire once on the access node and
once on each data node.
An issue with `RETURNING` handling for distributed insert is also
fixed (in the `DataNodeDispatch` executor node). The fix ensures that
the modified tuple is returned when a trigger modifies the tuple on a
data node.
Fixes#3201
Add support for deparsing `GENERATED` columns when creating
distributed hypertables, including support for generating values
"on-the-fly" when a `RETURNING` clause is used with an `INSERT`
statement.
Previously, using generated columns on a hypertable led to errors or
had unpredictible results.
In the definition of "Timescale Data Manipulation Interfaces", the description of
Data Manipulation Language was accidentally and erroneously given the acronym
"DDL" as a parenthetical, when it should have been (and was meant to be) given
the proper acronym of "DML". No intent is changed with this correction.
This maintenance release contains bugfixes since the 2.2.0 release. We
deem it high priority for upgrading.
This release extends Skip Scan to multinode by enabling the pushdown
of `DISTINCT` to data nodes. It also fixes a number of bugs in the
implementation of Skip Scan, in distributed hypertables, in creation
of indexes, in compression, and in policies.
**Features**
* #3113 Pushdown "SELECT DISTINCT" in multi-node to allow use of Skip
Scan
**Bugfixes**
* #3101 Use commit date in `get_git_commit()`
* #3102 Fix `REINDEX TABLE` for distributed hypertables
* #3104 Fix use after free in `add_reorder_policy`
* #3106 Fix use after free in `chunk_api_get_chunk_stats`
* #3109 Copy recreated object permissions on update
* #3111 Fix `CMAKE_BUILD_TYPE` check
* #3112 Use `%u` to format Oid instead of `%d`
* #3118 Fix use after free in cache
* #3123 Fix crash while using `REINDEX TABLE CONCURRENTLY`
* #3135 Fix SkipScan path generation in `DISTINCT` queries
with expressions
* #3146 Fix SkipScan for IndexPaths without pathkeys
* #3147 Skip ChunkAppend if AppendPath has no children
* #3148 Make `SELECT DISTINCT` handle non-var targetlists
* #3151 Fix `fdw_relinfo_get` assertion failure on `DELETE`
* #3155 Inherit `CFLAGS` from PostgreSQL
* #3169 Fix incorrect type cast in compression policy
* #3183 Fix segfault in calculate_chunk_interval
* #3185 Fix wrong datatype in integer based retention policy
**Thanks**
* @Dead2, @dv8472 and @einsibjarni for reporting an issue with
multinode queries and views
* @hperez75 for reporting an issue with Skip Scan
* @nathanloisel for reporting an issue with compression on hypertables
with integer-based timestamps
* @xin-hedera for fixing an issue with compression on hypertables with
integer-based timestamps
Block calling calculate_chunk_interval on distributed hypertables
to prevent segfault when finding min and max as distributed chunks
have no tableam since they are foreign tables.
1) Quell a compiler warning about uninitialized use of "varno"
variable.
2) Fix ts_chunk_get_by_name to return early if rogue schema/table args
are passed in.
We started inheriting CFLAGS from postgresql via ba51adc3. Although we
remove "-W" flags, we still end up accepting other options like "-L"
from postgresql. This can cause compilation to fail with compilers
like Clang which enable reporting of errors on unused command line
arguments.
This patch conditionally adds -Wno-unused-command-line-argument in
our cmake files to avoid unhelpful unused-command-line-argument
warnings/errors.
This patch adds coredump support for the 32bit CI runs. This patch
switches 32bit run to the debian base image and install postgres
from PGDG instead of using the alpine image so we don't have to
compile postgres ourselves and can just install the debugsymbol
package for postgres.
There are quite a few compiler flags that need to be identical
between postgresql and timescaledb otherwise there will be
seemingly unexplainable failures during execution.
We inherit all CFLAGS from postgresql except those starting
with -W as we have our own set of warning options.
The code in fdw_relinfo_get assumed rel->fdw_private and
rel->fdw_private->fdw_relation_info were always allocated
which is not true for DELETEs for instance.
Tables, indexes, and sequences that are recreated as part of an update
does not propagate permissions to the recreated object. This commit
fixes that by saving away the permissions in `pg_class` temporarily and
then copying them back into the `pg_class` table.
If internal objects are created or re-created, they get the wrong
initial privileges, which result in privileges not being dumped when
using `pg_dump`. Save away the privileges before starting the update
and restore them afterwards to make sure that the privileges are
maintained over the update.
For new objects, we use the initial privileges of the `chunk` metadata
table, which should always have correct initial privileges.
Fixes#3078
The chunks of the datanodes seem to contain very few rows. That can
cause PostgreSQL to choose a SeqScan plan in some cases. Add a few
more rows to make it choose IndexScan all of the time.
The update smoke test script can be used to test that an extension
works correctly. It will run the normal update tests, but do it against
a single server.
It also makes some minor refactorings of the update tests to make it
possible to run the tests against different versions of the server and
also support the smoke tests where some features cannot be tested.
The current SELECT DISTINCT pushdown code assumed that the targetlist
will always contain references to column attributes of the target
table.
So, even a simple "SELECT DISTINCT 1 from table;" causes a segmentation
fault because the "varno" field is not assigned. Fix this oversight.
Issue reported by @svenklemm
Fixestimescale/timescaledb-private#920
Since postgres has optimizations for Append paths without children
we do not replace those paths with ChunkAppend. While this will
prevent ChunkAppendPaths without children during planning, in the
executor it is still possible for ChunkAppend to have no children
as run-time exclusion may remove all of them.
The SkipScan code assumed IndexPaths on ordered indexes always
have pathkeys which is not true when the pathkeys are not useful
for the query, leading to a segfault for those queries.
This patch fixes the flaky pg_dump test by switching off the current
database before using it as template. It also disables background
job scheduling instead of stopping them before creating the new
database as stopping bgw jobs might not happen immediately.
Fixes#2847
Checking code formatting and other simple code style test is good to
perform on PRs to feature branches, not only the master branch. This
commit removes the limitation to perform tests only on PR to the
master.
Construct "SELECT DISTINCT target_list" or "SELECT DISTINCT ON (col1,
col..) target_list" statement to push down the DISTINCT clause to the
remote side.
We only allow references to basic "Vars" or constants in the DISTINCT
exprs
So, "SELECT DISTINCT col1" is fine but "SELECT DISTINCT 2*col1" is not.
"SELECT DISTINCT col1, 'const1', NULL, col2" which is a mix of column
references and constants is also supported. Everything else is not
supported.
This pushdown also needs to work when
timescaledb.enable_per_data_node_queries is disabled.
All existing test cases in which "SELECT DISTINCT" is now being pushed
down have been modified. New test cases have been added to check that
the remote side uses "Skip Scans" as is suitable in some cases.
Since we run 20 tests in parallel in the tsl tests multiple
multinode tests might exhaust the number of connections the
test instance allowed. This patch changes the number of connections
to 200 for regresscheck-t.
