Previously, the executor code adjusted es_processed to set the
number of tuples that have been processed in the completion tag.
That was necessary since we had triggers that re-routed tuples to chunks
and cancelled the inserts on the base table (thus those tuples were not
counted in es_processed). We no longer use triggers and thus this logic
was no longer used. In fact, this was mostly dead code because
`additional_tuples` was always 0.
This PR adds the ability to have multiple different versions of the timescaledb
extension be used by different databases in the same PostgreSQL
instance (server).
This is accomplished by splitting this extension into two .so files.
1) timescaledb.so -- stuff under loader/. Really not a lot of code.
This code MUST be backwards compatible in the future.
2) timescaledb-version.so (most of our code). Need
not be backwards compatible.
Timescaledb.so becomes a small stub which is preloaded and whose main
reason for existing is to dynamically load the right
timescaledb-version.so when the time comes.
This change allows either of the above .so to be loaded in
shared_preload_libraries. But timescaledb.so allows for multiple
versions used on different databases in the same instance along
with smoother upgrades. Using timescaledb-version.so allows for
finer-grained control and lock-in and is appropriate in only a few
production environments.
This PR also adds version checking so that a clear failure message
will be displayed if the .so version does not match the SQL extension
version.
To support multi-version functionality we changed the way SQL update
scripts are generated. Previously, the system used a bunch of
intermediate upgrade scripts. So with 3 versions, you would have an
update script of 1--2, 2--3. But, this PR changes things so that we
produce direct "shortcut" update files: 1--3, 2--3.
This is done for 2 reasons:
1) Each of the update files should point to
$libdir/timescaledb-current_version. Since you cannot guarantee that
Previous .so for each intermediate version has been installed.
2) You don't want intermediate version updates installed without the
.so. For example, if you have versions 1,2,3
and you are installing version 3, you want the upgrade files 1--3,
2--3 but not 1--2 because if you have 1--2
then a user could do ALTER EXTENSION timescaledb UPDATE TO 2. But
the .so for version 2 may not be installed.
In order to test this functionality, we add a mock extension version .so
that we can test extension loading inside the regression framework.
When tables are created with storage parameters (reloptions),
these should be inherited by chunks so that one can set options
such as fillfactor and autovacuum settings on the hypertable.
This change makes chunks inherit the reloptions set on a hypertable
and allows altering options via the ALTER TABLE command.
This change is part of an effort to create a consistent way
of dealing with metadata catalog updates, which is currently
a mix of C API and INSERT/UPDATE/DELETE statements from SQL
code. This mix makes catalog handling unnecessarily complex as
there are multiple ways to update metadata, increasing the risk
of security issues with publically exposed SQL functions. It also
complicates things like cache invalidation, requiring different
mechanisms for C and SQL code. Catalog updates from SQL code
require triggers on metadata tables for cache invalidation that
do not work with native catalog updates.
The creation of chunks has been particularly messy in this regard,
making the code hard to follow. Especially the handling of a chunk's
constraints, where dimensional and other constraints were handled
differently. With this change, constraint handling is now consistent
across constraint types with a single API for updating metadata.
Reduce memory usage for out-of-order inserts
The chunk_result_relation_info should be put on the chunk memory
context. This will cause the rri constraint expr to also go onto
that context and be correctly freed when the chunk insert state
is destroyed.
Compatibility with pg_upgrade required 2 changes:
1) search_path on functions cannot be blank for pg_upgrade.
2) The timescaledb.restoring GUC had to apply to more code (now moved to
higher-level check)
`pg_upgrade` must be passed the following option: `-O "-c timescaledb.restoring='on'"`
Tablespaces can now be detached from hypertables using
`tablespace_detach()`. This function can either detach
a tablespace from all tables or only a specific table.
Having the ability to detach tablespace allows more
advanced storage management, for instance, one can detach
tablespaces that are running low on diskspace while attaching
new ones to replace the old ones.
Attaching tablespaces to hypertables is now handled
in native code, with improved permissions checking and
caching of tablespaces in the Hypertable data object.
Windows 64-bit binaries should now be buildable using the cmake
build system either from the command line or from Visual Studio.
Previous issues regarding unresolved symbols have been resolved
with compatibility header files to properly export symbols or
getting GUCs via normal APIs.
This change reduces the usage of SECURITY DEFINER on SQL
functions and fixes related permissions issues. It also
properly checks hypertable permissions relative the current_user
instead of the session_user, which otherwise breaks SET ROLE,
among other things.
Clustering a table means reordering it according to an index.
This operation requires an exclusive lock and extensive
processing time. On large hypertables with many chunks, CLUSTER
risks blocking a lot of operations by holding locks for a long time.
This is alleviated by processing each chunk in a new transaction,
ensuring locks are only held on one chunk at a time.
All partitioning functions now has the signature `int func(anyelement)`.
This cleans up some special handling that was necessary to support
the legacy partitioning function that expected text input.
The extension now works with PostgreSQL 10, while
retaining compatibility with version 9.6.
PostgreSQL 10 has numerous internal changes to functions and
APIs, which necessitates various glue code and compatibility
wrappers to seamlessly retain backwards compatiblity with older
versions.
Test output might also differ between versions. In particular,
the psql client generates version-specific output with `\d` and
EXPLAINs might differ due to new query optimizations. The test
suite has been modified as follows to handle these issues. First,
tests now use version-independent functions to query system
catalogs instead of using `\d`. Second, changes have been made to
the test suite to be able to verify some test outputs against
version-dependent reference files.
A recent change blocked changing types of space-partitioned hypertable
columns. However, this blocking should not apply to regular tables,
which otherwise causes a crash. This change fixes this issue by
properly checking that the the table is a hypertable.
Hash partitioning previously relied on coercing (casting) values to
strings before calculating a hash value, including creating CHECK
constraints with casts. This approach is fairly suboptimal from a
performance perspective and might have issues related to different
character encodings depending on system.
Hash partitioning now instead uses a partitioning function that takes
an anyelement type that calls type-dependent hash functions internal
to PostgreSQL. This should provide more efficient hashing both by
avoiding unnecessary string conversions and by using more optimal
type-specific hash functions.
Support for the previous hash partitioning function is preserved for
backwards compatibility. Hypertables created with the previous
function will continue to use to old hashing strategy, while new
tables will default to the updated hash partitioning.
For safety, this change also blocks changing types on hash-partitioned
columns, since it seems hard to guarantee the same hash result between
different types.
When recursing DDL operations to chunks, you often want
access to the hypertable in the chunk handler function. This
change gives the handler access to the hypertable data structure
rather then just the hypertable OID, obviating the need to do
a separate lookup that has likely already been done previously.
A previous commit mixed declarations and code, which causes
warnings on some compilers. This change quenches the warning
and fixes some indentation/formatting issues.
All regression tests will now use a non-superuser unless superuser is
necessary. This PR is a meant to prevent things like issue #226.
This PR also fixes some more permission bugs found during this testing.
When issuing rename or reindex commands that do not involve
relations, the relation rangevar in the utility statement
will be NULL.
This change adds extra checks to avoid calling, e.g.,
RangeVarGetRelid() when a relation's rangevar is NULL, as
that function does not accept NULL input values.
This change refactors the chunk index handling to make better use
of standard PostgreSQL catalog information, while removing the
hypertable_index metadata table and associated triggers, including
those on the chunk_index table. The chunk_index table itself is
also simplified.
A benefit of this refactoring is that indexes are no longer
created using string mangling to construct the CREATE INDEX command
for a chunk, based on the string definition of the hypertable
index. Instead, indexes are created in C using proper index-related
internal data structures.
Chunk indexes can now also be renamed and are added in the parent
index tablespace. Changing tablespace on a hypertable index also
recurses to chunks, as expected. Default indexes that are added when
creating a hypertable use the hypertable's tablespace.
Creating Hypertable indexes with the CONCURRENTLY modifier is
currently blocked, due to unclear semantics regarding concurrent
creation over many tables, including how to deal with snapshots.
Constraints on regular PostgreSQL tables need to be validatated so
that they don't have a foreign key constraint that references a
hypertable, as this is not yet supported. This change fixes a bug in
this validation caused by not checking for the proper node types when
parsing a CreateStmt node. The CreateStmt is now also parsed in the
DDL end hook, instead of the processUtility hook, which ensures that
the CreateStmt node has been run through parse analysis. The parse
analysis changes the contents of the node to be more consistent across
different CREATE statements. For instance, a CREATE LIKE statement
will look more similar to a regular CREATE statement after parse
analysis.
The ProcessUtility hook doesn't give any information on applied DDL
commands, which makes it hard to implement DDL processing that
requires the result of a DDL command on a hypertable (for instance,
adding a constraint or index without an explicit name).
This change splits the DDL processing over start and end hooks,
handling DDL commands before and after regular PostgreSQL processing,
respectively.
The start DDL hook is still based on the ProcessUtility hook, while
the end DDL hook is based on an event trigger that allows getting
information on the created/dropped/altered objects.
The current approach of handling alter table commands does not allow
getting the result of the command; for instance, the object address of
a created constraint on a hypertable. Thus there is no way to get the
auto-generated name of a constraint, which is needed when the
corresponding constraints are created on the hypertable's chunks.
Therefore, this change blocks the ability to create constraints
without an explicit name. When a better approach to handling alter
table is deviced, it is possible to remove this restriction.
During a database restore, the creation of triggers and constraints on
hypertables should not recurse to chunks since the chunks will be
restored with existing triggers and constraints. Failing to block
recursing during restore risks duplicating these objects on chunks.
Further, operations on chunks (e.g., ALTER TABLE) should be allowed
during the restore process since those operations are needed to
recreate the constraints and triggers.
To fix this, the "restoring" GUC is used to bypass custom handling of
events in the ProcessUtility hook.
- Move DDL upcalls to function macros for clarity and readability
- Rename some functions for consistency and remove unused parameters
- Refactor some code that did unecessary work
Adding a constraint using an existing index changes the existing
index's name as a side effect (PostgreSQL wants the index name to
match the constraint name). This creates a mismatch between the
index's name and our metadata in the chunk_index table. Further, since
the name change happens as a result of an internal command (and not
via DDL command that we can capture) there is currently no way to sync
up this information.
Applying triggers to chunks requires taking the definition
of a trigger on a hypertable and executing it on a chunk. Previously
this was done with string replacement in the trigger definition.
This was not especially safe, and thus we moved the logic to C
where we can do proper parsing/deparsing and replacement of the table
name. Another positive aspect is that we got rid of some DDL triggers.
Unfortunately, Postgres forbids relocating the "public api" schema
of extensions that use multiple schemas after initial installation.
This PR marks the extension as not relocatable and tests relocation
during initial install.
Streamline code and remove triggers from chunk and
chunk_constraint. Lots of additional cleanup. Also removes need to CASCADE
hypertable drops (fixes#88).
This PR add support for primary-key, foreign-key, unique, and exclusion constraints.
Previously supported are CHECK and NOT NULL constraints. Now, foreign key
constraints where a hypertable references a plain table is support
(while vice versa, with a plain table references a hypertable, is still not).
Previously, when issued on hypertable, database maintenance
commands, like VACUUM and REINDEX, only affected the main
table and did not recurse to chunks.
This change fixes that issue, allowing database maintainers
to issue single commands on hypertables that affect all
the data stored in the hypertable.
These commands (VACUUM, REINDEX) only work at the table level
for hypertables. If issued at other levels, e.g., schema, or
database, the behavior is the same as in standard PostgreSQL
as all tables are covered by default.
REINDEX commands that specify a hypertable index do not
recurse as that requires mapping the hypertable
index to the corresponding index on the chunk. This might
be fixed in a future update.
This fixes some issues that caused EXPLAIN and PREPAREd statements to
fail on INSERTs. The plan modification on INSERTs passed on a plan
node to the execution stage. However, that plan node pointer becomes
invalid when the plan is copied, as happens with prepared
statements. The fix required another modification to the plan by
wrapping also the ModifyTable plan node. This wrapping is necessary to
access the plan state node (ModifyTableState) on execution. Having
access to the ModifyTableState node is necessary to set the correct
arbiter index list for ON CONFLICT statements.
This change also fixes the output of EXPLAIN to properly show the plan
tree, and performs some cleanups.
With this change, hypertables no longer rely on an INSERT trigger to
dispatch tuples to chunks. While an INSERT trigger worked well for
both INSERTs and COPYs, it caused issues with supporting some regular
triggers on hypertables, and didn't support RETURNING statements and
upserts (ON CONFLICT DO UPDATE).
INSERTs are now handled by modifying the plan for INSERT statements. A
custom plan node is inserted as a subplan to a ModifyTable plan node,
taking care of dispatching tuples to chunks by setting the result
table for every tuple scanned.
COPYs are handled by modifying the regular copy code. Unfortunately,
this required copying a significant amount of regular PostgreSQL
source code since there are no hooks to add modifications. However,
since the modifications are small it should be fairly easy to keep the
code in sync with upstream changes.
SQL code is now split into setup, functions, and init files to
allow a subset to be run when the extension is updated. During
build, an update script is now also generated.
Make sure parameter names are in lowercase-underscore format. Also,
rename the query tree walker change_table_name_walker() to
hypertable_query_walker() to reflect that it is no long renaming
anything (only setting up state in case of a query on a hypertable).
Previously the count returned by insert and copy was wrong because
the count was reset on every execute. But, often there is an execute
in the middle of an insert (i.e. to create a chunk). This fixes the
logic to reset the count only at the start of the top-level statement.
Fixes#64
