We've decided to adopt the ts_ prefix on all exported C functions in
order to avoid having symbol conflicts with future postgres functions.
We've already started using this prefix on new functions and this commit
adds the prefix to to the old functions.
This PR causes errors to be generated when alter table commands
that are not explicitly listed/handled are encountered on hypertables.
Previously the behavior was to let postgres do it's usual actions,
This new behavior is safer and more appropriate for 1.0.
Previously setting tablespace was blocked for hypertables.
This commit enables them and adds proper testing.
Setting tablespaces of indexes was handled in a previous commit
here: 097db3d58928ab4784c679b8308ab0ebf3bf1fe0
We want to test row-level-security on hypertables. So,
we take the original rowsecurity test from core postgres
and systematically change it so that it operates on hypertables
instead just plain tables. This was done by looking for all CREATE
TABLE statements and adding a create_hypertable call for each
new tables (except when the table was a referenced "metadata" table).
This commit also allows enabling and disabling RLS rules on chunks.
This is more in-line with how Postgres itself handles RLS on inherited
children. The rowsecurity tests relating to partitioned tables were
converted to work on hypertables/chunks instead and the rules for
handling permissions on chunks were tested to be akin to the rules
for child partitions.
Previously the following commands would not throw
an error on hypertables and this PR makes sure that
they do in order to avoid corrupting hypertable
internals:
* ALTER TABLE hypertable ADD COLUMN new_device_id int UNIQUE;
* ALTER TABLE hypertable DROP COLUMN partitioning_column;
* ALTER TABLE hypertable DROP COLUMN time_open_dim;
* ALTER TABLE hypertable ALTER COLUMN time_open_dim TYPE TEXT;
Fixes#536.
Constraints with the NO INHERIT option does not make sense on a
hypertable's root table since these will not be enforced.
Previously, NO INHERIT constraints were blocked on chunks, and were
thus not enforced until chunk creation time, allowing creation of NO
INHERIT constraints on empty hypertables, but then causing failure at
chunk-creation time. Instead, NO INHERIT constraints are now properly
blocked at the hypertable level.
Add support for:
* ALTER TABLE ... CLUSTER ON
* ALTER TABLE ... SET WITHOUT CLUSTER
on both hypertables and chunks. Commands on hypertables get
passed down to chunks.
This PR adds better handling for the following commands:
* ALTER TABLE ... ALTER COLUMN ... SET (attribute_name = value)
* ALTER TABLE ... ALTER COLUMN ... RESET (attribute_name)
* ALTER TABLE ... ALTER COLUMN ... SET STATISTICS
* ALTER TABLE ... ALTER COLUMN ... SET STORAGE
For each of the above commands the associated settings are not properly
propagated to existing chunks and new chunks are created with the
same settings as the hypertable.
We also now allow these commands to be run on chunks.
Previously, when running a VACUUM ANALYZE or ANALYZE on
a hypertable the statics on the parent table (hypertable)
were not correctly updated. This fixes that problem
and adds tests.
Previously, chunks could be renamed and have their schema changed, but
the metadata in the TimescaleDB catalog was not updated in a
corresponding way. Further, renaming a chunk column was possible,
which could break functionality on the hypertable.
Then catalog metadata is now properly updated on name and schema
changes applied to chunks. Renaming chunk columns have been blocked
with an informational error message.
This PR moves table, schema, and trigger drop handling into the event
trigger system. The event trigger system is a more reliable method of
intercepting object drops especially as they can CASCADE via other
object drops.
This PR also adds a test for DROP OWNED which was previously broken.
This Fixes at least two bugs:
1) A drop of a referenced table used to drop the associated
FK constraint but not the metadata associated with the constraint.
Fixes#43.
2) A drop of a column removed any indexes associated with the column
but not the metadata associated with the index.
This change improves the handling of tablespaces as follows:
- Add if_not_attached / if_attached options to attach_tablespace() and
detach_tablespace(), respectively
- Block DROP tablespace if it is still attached to a table
- Block REVOKE if it means the table owner no longer has CREATE
permissions on an attached tablespace
- Make error messages follow the PostgreSQL style guide
Dropping a schema that a hypertable depends on should clean up
dependent metadata. There are two schemas that matter for hypertables:
the hypertable's schema and the associated schema where chunks are
stored.
This change deals with the above as follows:
- If the hypertable schema is dropped, the hypertable and all chunks
should be deleted as well, including metadata.
- If an associated schema is dropped, the hypertables that use that
associated schema will have their associated schemas reset to the
internal schema.
- Even if no hypertable currently uses the dropped schema as their
associated schema, there might be chunks that reside in the dropped
schema (e.g., if the associated schema was changed for their
hypertables), so those chunks should have the metadata deleted.
The compiler does not seem to like when I use the msvc enter/exit
guards for utils/guc.h, so the alternative is to grab the value
via GetConfigOptionByName.
This change refactors the handling of TRUNCATE so
that it is performed directly in process utility without
doing an upcall to PL/pgSQL.
It also adds handling for the ONLY modifier to TRUNCATE,
which shouldn't work on a hypertable. TRUNCATE now generates
an error if TRUNCATE ONLY is used on a hypertable.
Deletes on metadata in the TimescaleDB catalog has so far been a mix
of native deletes using the C-based catalog API and SQL-based DELETE
statements that CASCADEs.
This mixed environment is confusing, and SQL-based DELETEs do not
consistently clean up objects that are related to the deleted
metadata.
This change moves towards A C-based API for deletes that consistently
deletes also the dependent objects (such as indexes, tables and
constraints). Ideally, we should prohobit direct manipulation of
catalog tables using SQL statements to avoid ending up in a bad state.
Once all catalog manipulations happend via the native API, we can also
remove the cache invalidation triggers on the catalog tables.
The functions for adding and updating dimensions have been refactored
in C to:
- improve usage of proper error codes
- make messages that better conform with the PostgreSQL standard.
- improve security by avoiding that lots of code run under SECURITY DEFINER
A new if_not_exists option has also been added to add_dimension() and
a the number of partitions can now be set using the new
set_number_partitions() function.
A bug in the validation of smallint time intervals has been fixed. The
previous code didn't check for intervals > 0 and smallint intervals
accepted values up to UINT16_MAX instead of INT16_MAX.
Source code indentation has been updated in PostgreSQL 10 to fix a
number of issues. This update applies this new indentation to the
entire code base.
The new indentation requires a new version of pg_bsd_indent, which can
be found here:
https://git.postgresql.org/git/pg_bsd_indent.git
This commit adds logic for cache pinning to handle subtxn. It also makes
it easier to find cache pinning leaks. Finally, it fixes handling of
cross-commit operations like VACUUM and CLUSTER. Previously, such
operations incorrectly released that cache pin on the first commit
even though the object was used after that.
Renaming constraints on hypertables is now supported. Using the ONLY
option with RENAME CONSTRAINT is blocked for hypertables. Renaming
constraints on chunks is also blocked.
This commit also fixes a possible bug in chunk_constraint code.
Previously, `chunk_constraint_add_from_tuple` used GETSTRUCT on the
tuple to convert the tuple to `FormData_chunk_constraint`. This
was incorrect since some fields can be NULL. Using GETSTRUCT for
tables with NULLABLE fields is unsafe and indeed was incorrect·
in this case.
Previously, the executor code adjusted es_processed to set the
number of tuples that have been processed in the completion tag.
That was necessary since we had triggers that re-routed tuples to chunks
and cancelled the inserts on the base table (thus those tuples were not
counted in es_processed). We no longer use triggers and thus this logic
was no longer used. In fact, this was mostly dead code because
`additional_tuples` was always 0.
This PR adds the ability to have multiple different versions of the timescaledb
extension be used by different databases in the same PostgreSQL
instance (server).
This is accomplished by splitting this extension into two .so files.
1) timescaledb.so -- stuff under loader/. Really not a lot of code.
This code MUST be backwards compatible in the future.
2) timescaledb-version.so (most of our code). Need
not be backwards compatible.
Timescaledb.so becomes a small stub which is preloaded and whose main
reason for existing is to dynamically load the right
timescaledb-version.so when the time comes.
This change allows either of the above .so to be loaded in
shared_preload_libraries. But timescaledb.so allows for multiple
versions used on different databases in the same instance along
with smoother upgrades. Using timescaledb-version.so allows for
finer-grained control and lock-in and is appropriate in only a few
production environments.
This PR also adds version checking so that a clear failure message
will be displayed if the .so version does not match the SQL extension
version.
To support multi-version functionality we changed the way SQL update
scripts are generated. Previously, the system used a bunch of
intermediate upgrade scripts. So with 3 versions, you would have an
update script of 1--2, 2--3. But, this PR changes things so that we
produce direct "shortcut" update files: 1--3, 2--3.
This is done for 2 reasons:
1) Each of the update files should point to
$libdir/timescaledb-current_version. Since you cannot guarantee that
Previous .so for each intermediate version has been installed.
2) You don't want intermediate version updates installed without the
.so. For example, if you have versions 1,2,3
and you are installing version 3, you want the upgrade files 1--3,
2--3 but not 1--2 because if you have 1--2
then a user could do ALTER EXTENSION timescaledb UPDATE TO 2. But
the .so for version 2 may not be installed.
In order to test this functionality, we add a mock extension version .so
that we can test extension loading inside the regression framework.
When tables are created with storage parameters (reloptions),
these should be inherited by chunks so that one can set options
such as fillfactor and autovacuum settings on the hypertable.
This change makes chunks inherit the reloptions set on a hypertable
and allows altering options via the ALTER TABLE command.
This change is part of an effort to create a consistent way
of dealing with metadata catalog updates, which is currently
a mix of C API and INSERT/UPDATE/DELETE statements from SQL
code. This mix makes catalog handling unnecessarily complex as
there are multiple ways to update metadata, increasing the risk
of security issues with publically exposed SQL functions. It also
complicates things like cache invalidation, requiring different
mechanisms for C and SQL code. Catalog updates from SQL code
require triggers on metadata tables for cache invalidation that
do not work with native catalog updates.
The creation of chunks has been particularly messy in this regard,
making the code hard to follow. Especially the handling of a chunk's
constraints, where dimensional and other constraints were handled
differently. With this change, constraint handling is now consistent
across constraint types with a single API for updating metadata.
Reduce memory usage for out-of-order inserts
The chunk_result_relation_info should be put on the chunk memory
context. This will cause the rri constraint expr to also go onto
that context and be correctly freed when the chunk insert state
is destroyed.
Compatibility with pg_upgrade required 2 changes:
1) search_path on functions cannot be blank for pg_upgrade.
2) The timescaledb.restoring GUC had to apply to more code (now moved to
higher-level check)
`pg_upgrade` must be passed the following option: `-O "-c timescaledb.restoring='on'"`
Tablespaces can now be detached from hypertables using
`tablespace_detach()`. This function can either detach
a tablespace from all tables or only a specific table.
Having the ability to detach tablespace allows more
advanced storage management, for instance, one can detach
tablespaces that are running low on diskspace while attaching
new ones to replace the old ones.
Attaching tablespaces to hypertables is now handled
in native code, with improved permissions checking and
caching of tablespaces in the Hypertable data object.
Windows 64-bit binaries should now be buildable using the cmake
build system either from the command line or from Visual Studio.
Previous issues regarding unresolved symbols have been resolved
with compatibility header files to properly export symbols or
getting GUCs via normal APIs.
This change reduces the usage of SECURITY DEFINER on SQL
functions and fixes related permissions issues. It also
properly checks hypertable permissions relative the current_user
instead of the session_user, which otherwise breaks SET ROLE,
among other things.
Clustering a table means reordering it according to an index.
This operation requires an exclusive lock and extensive
processing time. On large hypertables with many chunks, CLUSTER
risks blocking a lot of operations by holding locks for a long time.
This is alleviated by processing each chunk in a new transaction,
ensuring locks are only held on one chunk at a time.
All partitioning functions now has the signature `int func(anyelement)`.
This cleans up some special handling that was necessary to support
the legacy partitioning function that expected text input.
The extension now works with PostgreSQL 10, while
retaining compatibility with version 9.6.
PostgreSQL 10 has numerous internal changes to functions and
APIs, which necessitates various glue code and compatibility
wrappers to seamlessly retain backwards compatiblity with older
versions.
Test output might also differ between versions. In particular,
the psql client generates version-specific output with `\d` and
EXPLAINs might differ due to new query optimizations. The test
suite has been modified as follows to handle these issues. First,
tests now use version-independent functions to query system
catalogs instead of using `\d`. Second, changes have been made to
the test suite to be able to verify some test outputs against
version-dependent reference files.
A recent change blocked changing types of space-partitioned hypertable
columns. However, this blocking should not apply to regular tables,
which otherwise causes a crash. This change fixes this issue by
properly checking that the the table is a hypertable.
Hash partitioning previously relied on coercing (casting) values to
strings before calculating a hash value, including creating CHECK
constraints with casts. This approach is fairly suboptimal from a
performance perspective and might have issues related to different
character encodings depending on system.
Hash partitioning now instead uses a partitioning function that takes
an anyelement type that calls type-dependent hash functions internal
to PostgreSQL. This should provide more efficient hashing both by
avoiding unnecessary string conversions and by using more optimal
type-specific hash functions.
Support for the previous hash partitioning function is preserved for
backwards compatibility. Hypertables created with the previous
function will continue to use to old hashing strategy, while new
tables will default to the updated hash partitioning.
For safety, this change also blocks changing types on hash-partitioned
columns, since it seems hard to guarantee the same hash result between
different types.
When recursing DDL operations to chunks, you often want
access to the hypertable in the chunk handler function. This
change gives the handler access to the hypertable data structure
rather then just the hypertable OID, obviating the need to do
a separate lookup that has likely already been done previously.
