We've decided to adopt the ts_ prefix on all exported C functions in
order to avoid having symbol conflicts with future postgres functions.
We've already started using this prefix on new functions and this commit
adds the prefix to to the old functions.
Use the postgres macro instead of manual subtraction everywhere to be more readable,
and gain the Assert.
Done with
`find ./src -type f -exec sed -i -e 's/Anum_\(.*\) - 1/AttrNumberGetAttrOffset(Anum_\1)/' {} \;`
Previously, the chunk dispatch node used arbiter index and other
information from the top-level Query node. This did not work with CTEs
(and probably subqueries) because the info for this insert was not at
the top-level. This commit changes the logic to use that same info from the
parent ModifyTable node (and ModifyTableState).
This commit also changes the logic to translate the arbiter indexes
from the hypertable index to the chunk index directly using our catalog
tables instead of re-inferring the index on the chunk.
This PR moves table, schema, and trigger drop handling into the event
trigger system. The event trigger system is a more reliable method of
intercepting object drops especially as they can CASCADE via other
object drops.
This PR also adds a test for DROP OWNED which was previously broken.
This Fixes at least two bugs:
1) A drop of a referenced table used to drop the associated
FK constraint but not the metadata associated with the constraint.
Fixes#43.
2) A drop of a column removed any indexes associated with the column
but not the metadata associated with the index.
Deletes on metadata in the TimescaleDB catalog has so far been a mix
of native deletes using the C-based catalog API and SQL-based DELETE
statements that CASCADEs.
This mixed environment is confusing, and SQL-based DELETEs do not
consistently clean up objects that are related to the deleted
metadata.
This change moves towards A C-based API for deletes that consistently
deletes also the dependent objects (such as indexes, tables and
constraints). Ideally, we should prohobit direct manipulation of
catalog tables using SQL statements to avoid ending up in a bad state.
Once all catalog manipulations happend via the native API, we can also
remove the cache invalidation triggers on the catalog tables.
Currently, chunk indexes are always created in the tablespace of the
index on the main table (which could be none/default one), even if the
chunks themselves are created in different tablespaces. This is
problematic in a multi-disk setting where each disk is a separate
tablespace where chunks are placed. The chunk indexes might exhaust
the space on the common (often default tablespace) which might not
have a lot of disk space. This also prohibits the database, including
index storage to grow by adding new tablespaces.
Instead, chunk indexes are now created in the "next" tablespace after
that of their chunks to both spread indexes across tablespaces and
avoid colocating indexes with their chunks (for I/O throughput
reasons). To optionally avoid this spreading, one can pin chunk
indexes to a specific tablespace by setting an explicit tablespace on
a main table index.
Source code indentation has been updated in PostgreSQL 10 to fix a
number of issues. This update applies this new indentation to the
entire code base.
The new indentation requires a new version of pg_bsd_indent, which can
be found here:
https://git.postgresql.org/git/pg_bsd_indent.git
This change is part of an effort to create a consistent way
of dealing with metadata catalog updates, which is currently
a mix of C API and INSERT/UPDATE/DELETE statements from SQL
code. This mix makes catalog handling unnecessarily complex as
there are multiple ways to update metadata, increasing the risk
of security issues with publically exposed SQL functions. It also
complicates things like cache invalidation, requiring different
mechanisms for C and SQL code. Catalog updates from SQL code
require triggers on metadata tables for cache invalidation that
do not work with native catalog updates.
The creation of chunks has been particularly messy in this regard,
making the code hard to follow. Especially the handling of a chunk's
constraints, where dimensional and other constraints were handled
differently. With this change, constraint handling is now consistent
across constraint types with a single API for updating metadata.
Reduce memory usage for out-of-order inserts
The chunk_result_relation_info should be put on the chunk memory
context. This will cause the rri constraint expr to also go onto
that context and be correctly freed when the chunk insert state
is destroyed.
Windows 64-bit binaries should now be buildable using the cmake
build system either from the command line or from Visual Studio.
Previous issues regarding unresolved symbols have been resolved
with compatibility header files to properly export symbols or
getting GUCs via normal APIs.
reindex allows you to reindex the indexes of only certain chunks,
filtering by time. This is a common use case because a user may
want to reindex chunks after they are no longer getting new data once.
reindex also has a recreate option which will not use REINDEX
but will rather CREATE INDEX a new index and then
DROP INDEX / RENAME new_index to old_name. This approach has advantages
in terms of blocking reads for a much shorter period of time. However,
it does more work and will use more disk space during the operation.
Clustering a table means reordering it according to an index.
This operation requires an exclusive lock and extensive
processing time. On large hypertables with many chunks, CLUSTER
risks blocking a lot of operations by holding locks for a long time.
This is alleviated by processing each chunk in a new transaction,
ensuring locks are only held on one chunk at a time.
The extension now works with PostgreSQL 10, while
retaining compatibility with version 9.6.
PostgreSQL 10 has numerous internal changes to functions and
APIs, which necessitates various glue code and compatibility
wrappers to seamlessly retain backwards compatiblity with older
versions.
Test output might also differ between versions. In particular,
the psql client generates version-specific output with `\d` and
EXPLAINs might differ due to new query optimizations. The test
suite has been modified as follows to handle these issues. First,
tests now use version-independent functions to query system
catalogs instead of using `\d`. Second, changes have been made to
the test suite to be able to verify some test outputs against
version-dependent reference files.
When creating an index on a chunk based on a index on a hypertable, it
is necessary to adjust the attribute numbers of referenced columns in
case the hypertable and the chunk have different number of attributes
(this can happen, e.g., due to removing columns on the hypertable that
aren't inherited by a newly created chunk). This adjustment was
handled for regular indexes, but not expression indexes, causing an
error to be raised. This change adds the proper handling of expression
indexes.
This change fixes two things that were overlooked in a prior
refactoring of chunk index handling.
First, column attribute numbers of a hypertable might not match a
chunk if, e.g., a column on the hypertable has been removed. In such
circumstances, indexes created on chunks based on a corresponding
hypertable index need to account for differences in column attribute
numbers. This change ensures that column attributes are always
translated to match the chunk an index is created on.
Second, ShareLock was acquired by mistake on each hypertable index
when recursing these indexes to chunks, potentially causing
deadlocks. ShareLock should only be taken on the heap relation that an
index is created on. This is now fixed. Further, locking during index
creation has been cleaned up so that it is easier to overview the
locks taken on various relations.
This change refactors the chunk index handling to make better use
of standard PostgreSQL catalog information, while removing the
hypertable_index metadata table and associated triggers, including
those on the chunk_index table. The chunk_index table itself is
also simplified.
A benefit of this refactoring is that indexes are no longer
created using string mangling to construct the CREATE INDEX command
for a chunk, based on the string definition of the hypertable
index. Instead, indexes are created in C using proper index-related
internal data structures.
Chunk indexes can now also be renamed and are added in the parent
index tablespace. Changing tablespace on a hypertable index also
recurses to chunks, as expected. Default indexes that are added when
creating a hypertable use the hypertable's tablespace.
Creating Hypertable indexes with the CONCURRENTLY modifier is
currently blocked, due to unclear semantics regarding concurrent
creation over many tables, including how to deal with snapshots.
