To increase schema security we do not want to mix our own internal
objects with user objects. Since chunks are created in the
_timescaledb_internal schema our internal functions should live in
a different dedicated schema. This patch make the necessary
adjustments for the following functions:
- chunk_constraint_add_table_constraint(_timescaledb_catalog.chunk_constraint)
- chunk_drop_replica(regclass,name)
- chunk_index_clone(oid)
- chunk_index_replace(oid,oid)
- create_chunk_replica_table(regclass,name)
- drop_stale_chunks(name,integer[])
- health()
- hypertable_constraint_add_table_fk_constraint(name,name,name,integer)
- process_ddl_event()
- wait_subscription_sync(name,name,integer,numeric)
Removed the PG12 specific macros and all the now, dead code. Also updated
the testcases which had workarounds in place to make them compatible
with PG12.
CONCURRENTLY option of the REINDEX TABLE is not supported and
leads to a crash when using with hypertable. Block it and print
the error message.
Fix#3122
When mapping index column references from the hypertable index
to the chunk index the name of the index column was used instead
of the column name of the hypertables. While those names are
identical most of time unless modified this is not true for indexes
with INCLUDE on a column that is also a key column. Additionally
this fixes a bug where expression indexes would never get their
column references adjusted.
Fixes#3056
When creating an index with IF NOT EXISTS we still tried to
create indexes for all the chunks but since no index was created
on the parent table the meta data did not have the object id of
the main table index leading to an error when trying to open
the main table index. This patch adjusts the logic to check
for IF NOT EXISTS and does an early return when no index was
created on the parent table.
This patch changes chunk index creation to use the same functions
for creating index in one transaction and using multiple transactions.
The single transaction index creation used to adjust the original
stmt and adjusted it for the chunk which lead to problems with table
references not being adjusted properly for the chunk.
This change includes a major refactoring to support PostgreSQL
12. Note that many tests aren't passing at this point. Changes
include, but are not limited to:
- Handle changes related to table access methods
- New way to expand hypertables since expansion has changed in
PostgreSQL 12 (more on this below).
- Handle changes related to table expansion for UPDATE/DELETE
- Fixes for various TimescaleDB optimizations that were affected by
planner changes in PostgreSQL (gapfill, first/last, etc.)
Before PostgreSQL 12, planning was organized something like as
follows:
1. construct add `RelOptInfo` for base and appendrels
2. add restrict info, joins, etc.
3. perform the actual planning with `make_one_rel`
For our optimizations we would expand hypertables in the middle of
step 1; since nothing in the query planner before `make_one_rel` cared
about the inheritance children, we didn’t have to be too precises
about where we were doing it.
However, with PG12, and the optimizations around declarative
partitioning, PostgreSQL now does care about when the children are
expanded, since it wants as much information as possible to perform
partition-pruning. Now planning is organized like:
1. construct add RelOptInfo for base rels only
2. add restrict info, joins, etc.
3. expand appendrels, removing irrelevant declarative partitions
4. perform the actual planning with make_one_rel
Step 3 always expands appendrels, so when we also expand them during
step 1, the hypertable gets expanded twice, and things in the planner
break.
The changes to support PostgreSQL 12 attempts to solve this problem by
keeping the hypertable root marked as a non-inheritance table until
`make_one_rel` is called, and only then revealing to PostgreSQL that
it does in fact have inheritance children. While this strategy entails
the least code change on our end, the fact that the first hook we can
use to re-enable inheritance is `set_rel_pathlist_hook` it does entail
a number of annoyances:
1. this hook is called after the sizes of tables are calculated, so we
must recalculate the sizes of all hypertables, as they will not
have taken the chunk sizes into account
2. the table upon which the hook is called will have its paths planned
under the assumption it has no inheritance children, so if it's a
hypertable we have to replan it's paths
Unfortunately, the code for doing these is static, so we need to copy
them into our own codebase, instead of just using PostgreSQL's.
In PostgreSQL 12, UPDATE/DELETE on inheritance relations have also
changed and are now planned in two stages:
- In stage 1, the statement is planned as if it was a `SELECT` and all
leaf tables are discovered.
- In stage 2, the original query is planned against each leaf table,
discovered in stage 1, directly, not part of an Append.
Unfortunately, this means we cannot look in the appendrelinfo during
UPDATE/DELETE planning, in particular to determine if a table is a
chunk, as the appendrelinfo is not at the point we wish to do so
initialized. This has consequences for how we identify operations on
chunks (sometimes for blocking and something for enabling
functionality).
When the index for a chunk was created the attnos for the index
predicate were not adjusted leading to insert errors on hypertables
with dropped columns that had indexes with predicates.
This PR adjust index predicate attnos when creating the chunk index
to match the chunk attno.
Creating an expression index on a hypertable with dropped columns
would lead to an error on the next INSERT into the hypertable
because the attribute numbers in the expression were not properly
adjusted. This patch fixes the adjusting of the attribute numbers
for expression indexes.
This patch does refactoring necessary to support execution of DDL
commands on remote servers.
Basically it extends cross module api with ddl_command_start,
ddl_command_end and sql_drop functions.
Variable hypertables_list added to ProcessUtilityArg. It is used
to keep a list of found hypertables during Utility/DDL statement
parsing. This information and information gathered from other
hook functions will be used to distinct distributed hypertables
and forward DDL commands to any remote servers associated with
them.
Replace hardcoded database name from regression tests with :TEST_DBNAME
Remove creation of database single_2 from test runner and add it to
bgw_launcher and loader test since no other tests used those
use SQL comments in test scripts
The extension now works with PostgreSQL 10, while
retaining compatibility with version 9.6.
PostgreSQL 10 has numerous internal changes to functions and
APIs, which necessitates various glue code and compatibility
wrappers to seamlessly retain backwards compatiblity with older
versions.
Test output might also differ between versions. In particular,
the psql client generates version-specific output with `\d` and
EXPLAINs might differ due to new query optimizations. The test
suite has been modified as follows to handle these issues. First,
tests now use version-independent functions to query system
catalogs instead of using `\d`. Second, changes have been made to
the test suite to be able to verify some test outputs against
version-dependent reference files.
When creating an index on a chunk based on a index on a hypertable, it
is necessary to adjust the attribute numbers of referenced columns in
case the hypertable and the chunk have different number of attributes
(this can happen, e.g., due to removing columns on the hypertable that
aren't inherited by a newly created chunk). This adjustment was
handled for regular indexes, but not expression indexes, causing an
error to be raised. This change adds the proper handling of expression
indexes.
This change fixes two things that were overlooked in a prior
refactoring of chunk index handling.
First, column attribute numbers of a hypertable might not match a
chunk if, e.g., a column on the hypertable has been removed. In such
circumstances, indexes created on chunks based on a corresponding
hypertable index need to account for differences in column attribute
numbers. This change ensures that column attributes are always
translated to match the chunk an index is created on.
Second, ShareLock was acquired by mistake on each hypertable index
when recursing these indexes to chunks, potentially causing
deadlocks. ShareLock should only be taken on the heap relation that an
index is created on. This is now fixed. Further, locking during index
creation has been cleaned up so that it is easier to overview the
locks taken on various relations.
This commit moves a lot of test setup logic to runner.sh. Also
passes the right commands to the regression infrastructure to create
appropriate users and run tests as a regular user.
All regression tests will now use a non-superuser unless superuser is
necessary. This PR is a meant to prevent things like issue #226.
This PR also fixes some more permission bugs found during this testing.
This change refactors the chunk index handling to make better use
of standard PostgreSQL catalog information, while removing the
hypertable_index metadata table and associated triggers, including
those on the chunk_index table. The chunk_index table itself is
also simplified.
A benefit of this refactoring is that indexes are no longer
created using string mangling to construct the CREATE INDEX command
for a chunk, based on the string definition of the hypertable
index. Instead, indexes are created in C using proper index-related
internal data structures.
Chunk indexes can now also be renamed and are added in the parent
index tablespace. Changing tablespace on a hypertable index also
recurses to chunks, as expected. Default indexes that are added when
creating a hypertable use the hypertable's tablespace.
Creating Hypertable indexes with the CONCURRENTLY modifier is
currently blocked, due to unclear semantics regarding concurrent
creation over many tables, including how to deal with snapshots.
