This fixes the show_indexes test support function to properly show the
columns of the indexes instead of the table. The function now also
shows the expressions of expression indexes.
This adds a simple check to enforce that partitioning functions
are IMMUTABLE. This is a requirement since a partitioning function
must always return the same value given the same input.
Constraints with the NO INHERIT option does not make sense on a
hypertable's root table since these will not be enforced.
Previously, NO INHERIT constraints were blocked on chunks, and were
thus not enforced until chunk creation time, allowing creation of NO
INHERIT constraints on empty hypertables, but then causing failure at
chunk-creation time. Instead, NO INHERIT constraints are now properly
blocked at the hypertable level.
Add support for:
* ALTER TABLE ... CLUSTER ON
* ALTER TABLE ... SET WITHOUT CLUSTER
on both hypertables and chunks. Commands on hypertables get
passed down to chunks.
This PR adds better handling for the following commands:
* ALTER TABLE ... ALTER COLUMN ... SET (attribute_name = value)
* ALTER TABLE ... ALTER COLUMN ... RESET (attribute_name)
* ALTER TABLE ... ALTER COLUMN ... SET STATISTICS
* ALTER TABLE ... ALTER COLUMN ... SET STORAGE
For each of the above commands the associated settings are not properly
propagated to existing chunks and new chunks are created with the
same settings as the hypertable.
We also now allow these commands to be run on chunks.
Previously, when running a VACUUM ANALYZE or ANALYZE on
a hypertable the statics on the parent table (hypertable)
were not correctly updated. This fixes that problem
and adds tests.
A bug in the SQL for getting the size of chunks would use the
TOAST size of the main/dummy table as the toast size for the
chunks rather than each chunks' own toast size.
Previously, chunks could be renamed and have their schema changed, but
the metadata in the TimescaleDB catalog was not updated in a
corresponding way. Further, renaming a chunk column was possible,
which could break functionality on the hypertable.
Then catalog metadata is now properly updated on name and schema
changes applied to chunks. Renaming chunk columns have been blocked
with an informational error message.
Previously, upserts (ON CONFLICT) clauses did not work well on tables
where the hypertable attribute numbers did not match chunk attribute
numbers. This is common if the hypertable has dropped columns or
there were other alter commands run on the hypertable before
chunks were created.
This PR fixes the projection of the returning clause as well
as the update clauses. It also fixes the where clause for ON CONFLICT
UPDATE. These fixes are mostly about mapping the attribute numbers
from the hypertable attnos->chunk attnos. Some slot tupleDesc also
needed to be changed to the tupleDesc of the chunk.
Note that because of the limitations in PG 9.6 we had to copy over
some expressions from the ModifyTable plan node inside the chunk
dispatch. These original expressions are irrecoverable from the
ModifyTableState node or the ProjectionInfo structs in 9.6.
This optimization adds a HashAggregate plan to many group by queries.
In plain postgres, many time-series queries will not use the hash
aggregate because the planner will incorrectly assume that the number of
rows is much larger than it actually is and will use the less efficient
GroupAggregate instead of a HashAggregate to prevent running out of
memory.
The planner will assume a large number of rows because the statistics
planner for grouping assumes that the number of distinct items produced
by a function is the same as the number of distinct items going in. This
is not true for functions like time_bucket and date_trunc. This
optimization fixes the statistics and add the HashAggregate plan if
appropriate.
The statistics now rely on evaluating the spread of a variable and
dividing it by the interval in the time_bucket or date_trunc. This is
still an overestimate of the total number of groups but is better than
before. A further improvement on this will be to evaluate the quals
(WHERE clauses) on the query to try to derive a tighter spread on the
variable. This is left to a future optimization.
This planner optimization reduces planning times when a hypertable has many chunks.
It does this by expanding hypertable chunks manually, eliding the `expand_inherited_tables`
logic used by PG.
Slow planning time were previously seen because `expand_inherited_tables` expands all chunks of
a hypertable, without regard to constraints present in the query. Then, `get_relation_info` is
the called on all chunks before constraint exclusion. Getting the statistics an many chunks ends
up being expensive because RelationGetNumberOfBlocks has to open the file for each relation.
This gets even worse under high concurrency.
This logic solves this by expanding only the chunks needed to fulfil the query instead of all chunks.
In effect, it moves chunk exclusion up in the planning process. But, we actually don't use constraint
exclusion here, but rather a variant of range exclusion implemented
by HypertableRestrictInfo.
Getting an approximate row count for a hypertable involves getting
estimates for all of its chunks rather than just looking up a
single value in the catalog tables. This PR provides a convenience
function for doing the JOINs/summing.
Previously, the GUC to let the versioned-extension know that
the loader was present was incorrectly reset during DISCARD ALL.
This caused newly minted parallel workers to throw an error
about the extension not being preloaded since it did not
know that a loader was, in fact, preloaded. This PR
fixes this issue by changing the GUC to a rendezvous variable.
It also disables the loader in parallel workers since the loading
should have been handled in the parallel leader and the worker
start-up logic.
Previously, automatic chunk creation on INSERT failed due to lack of
permissions when the hypertable had triggers that needed to be
replicated to the new chunk. This happened because creating triggers
on tables requires TRIGGER permission, and the internal code used
CreateTrigger() that performs permissions checks. Thus, if the user
inserting tuples into the hypertable only had INSERT permissions, the
insert would fail whenever a new chunk was created.
This change fixes the issue by temporarily assuming the role of the
hypertable owner when executing CreateTrigger() on the new chunk.
Discovered a crash when doing an explain analyze on a cte containing an insert that
did not reference the cte in the select statement. This fixes by copying the eref
from the parent hypertable to the chunk so that columns can be described. It will
only copy the eref from the hypertable if there is a range table entry available.
Also modify copy so it actually sets a dummy variable as the RTE index rather than 1,
which is a valid RTE index, even if there is no RTE for the hypertable.
Previously, the chunk dispatch node used arbiter index and other
information from the top-level Query node. This did not work with CTEs
(and probably subqueries) because the info for this insert was not at
the top-level. This commit changes the logic to use that same info from the
parent ModifyTable node (and ModifyTableState).
This commit also changes the logic to translate the arbiter indexes
from the hypertable index to the chunk index directly using our catalog
tables instead of re-inferring the index on the chunk.
The explain output previously relied on the fact that the
estate->rtable was indexed the same way as the simple_rte_array
during planning. This is not always true when using CTEs. This
commit instead passes the oid of the hypertable to the EXPLAIN
directly.
Previously, all Subtxn aborts released all cache pins. This is
wrong because that can release pins that are still in use by
higher-level subtxns and top-level txns. Now, we release only
pins corresponding to the appropriate subtxn. That means that we
now track the subtxn a pin was created in.
This PR moves table, schema, and trigger drop handling into the event
trigger system. The event trigger system is a more reliable method of
intercepting object drops especially as they can CASCADE via other
object drops.
This PR also adds a test for DROP OWNED which was previously broken.
This commit adds a secondary FK to the update test. This
FK points to the same metadata table as the previous FK.
This case is added because it has caused problems in the
past.
This fixes an issue with the `if_not_exists` option to add_dimension()
that caused the command to fail on a non-empty table. When this option
is set, the command should only fail if the dimension does not exists
and the table is non-empty.
Prior to PostgreSQL 10.3, the output for a trigger function did
not include the implied 'public' schema in its output when
outputting the function. In 10.3 this was changed causing the
regression test to fail. To make this more robust, the table is
now put into an explicit schema so the test does not rely on
the inconsistent behavior across versions as to printing out
'public'.
Previously we just checked that we could upgrade from 0.1.0 until
latest but this missed some issues upgrading from intermediate
versions to latest. Now we check that all released versions can
successfully update.
Previously, when the FmgrInfo was initialized for partitioning
functions, the type information was cached in the fn_extra
field. However, the contract for fn_extra is that it is to be set by
the called function and not prior to its invokation. Setting fn_extra
prior to function invokation is an issue for custom partitioning
functions implemented in SQL, as the SQL call manager expects to use
fn_extra for its own cache.
This change avoids setting fn_extra prior to function invokation, and
instead information is cached in fn_extra within our provided
partitioning functions.
The native partitioning functions now also support nested functions,
i.e., the input to the function is the output of another function.
Tables can now hold existing data, which is optionally migrated from
the main table to chunks when create_hypertable() is called.
The data migration is similar to the COPY path, with the single
difference that the inserted/copied tuples come from an existing table
instead of being read from a file. After the data has been migrated,
the main table is truncated.
One potential downside of this approach is that all of this happens in
a single transaction, which means that the table is blocked while
migration is ongoing, preventing inserts by other transactions.
This Fixes at least two bugs:
1) A drop of a referenced table used to drop the associated
FK constraint but not the metadata associated with the constraint.
Fixes#43.
2) A drop of a column removed any indexes associated with the column
but not the metadata associated with the index.
This change improves the handling of tablespaces as follows:
- Add if_not_attached / if_attached options to attach_tablespace() and
detach_tablespace(), respectively
- Block DROP tablespace if it is still attached to a table
- Block REVOKE if it means the table owner no longer has CREATE
permissions on an attached tablespace
- Make error messages follow the PostgreSQL style guide
Dropping a schema that a hypertable depends on should clean up
dependent metadata. There are two schemas that matter for hypertables:
the hypertable's schema and the associated schema where chunks are
stored.
This change deals with the above as follows:
- If the hypertable schema is dropped, the hypertable and all chunks
should be deleted as well, including metadata.
- If an associated schema is dropped, the hypertables that use that
associated schema will have their associated schemas reset to the
internal schema.
- Even if no hypertable currently uses the dropped schema as their
associated schema, there might be chunks that reside in the dropped
schema (e.g., if the associated schema was changed for their
hypertables), so those chunks should have the metadata deleted.
This test adds a case where we use a trigger to automatically populate
a metadata table. Such uses are common in IOT where, for example,
you want to keep metadata associated with devices and you want
new devices to be auto-created.
This change refactors the handling of TRUNCATE so
that it is performed directly in process utility without
doing an upcall to PL/pgSQL.
It also adds handling for the ONLY modifier to TRUNCATE,
which shouldn't work on a hypertable. TRUNCATE now generates
an error if TRUNCATE ONLY is used on a hypertable.
When chunks are deleted, dimension slices can be orphaned, i.e., there
are no chunks or chunk constraints that reference such slices. This
change ensures that, when chunks are deleted, orphaned slices are also
deleted.
Deletes on metadata in the TimescaleDB catalog has so far been a mix
of native deletes using the C-based catalog API and SQL-based DELETE
statements that CASCADEs.
This mixed environment is confusing, and SQL-based DELETEs do not
consistently clean up objects that are related to the deleted
metadata.
This change moves towards A C-based API for deletes that consistently
deletes also the dependent objects (such as indexes, tables and
constraints). Ideally, we should prohobit direct manipulation of
catalog tables using SQL statements to avoid ending up in a bad state.
Once all catalog manipulations happend via the native API, we can also
remove the cache invalidation triggers on the catalog tables.
The functions for adding and updating dimensions have been refactored
in C to:
- improve usage of proper error codes
- make messages that better conform with the PostgreSQL standard.
- improve security by avoiding that lots of code run under SECURITY DEFINER
A new if_not_exists option has also been added to add_dimension() and
a the number of partitions can now be set using the new
set_number_partitions() function.
A bug in the validation of smallint time intervals has been fixed. The
previous code didn't check for intervals > 0 and smallint intervals
accepted values up to UINT16_MAX instead of INT16_MAX.
Issuing "CREATE EXTENSION IF NOT EXISTS timescaledb;" generates
an error if the extension is already loaded. This change makes
this command work as expected, with a NOTICE.
Error messages related to CREATE EXTENSION have also been
updated to better adhere to the official PostgreSQL style
guide and also returns the same or similar error codes
and messages as regular PostgreSQL.
Currently, chunk indexes are always created in the tablespace of the
index on the main table (which could be none/default one), even if the
chunks themselves are created in different tablespaces. This is
problematic in a multi-disk setting where each disk is a separate
tablespace where chunks are placed. The chunk indexes might exhaust
the space on the common (often default tablespace) which might not
have a lot of disk space. This also prohibits the database, including
index storage to grow by adding new tablespaces.
Instead, chunk indexes are now created in the "next" tablespace after
that of their chunks to both spread indexes across tablespaces and
avoid colocating indexes with their chunks (for I/O throughput
reasons). To optionally avoid this spreading, one can pin chunk
indexes to a specific tablespace by setting an explicit tablespace on
a main table index.
We add better accounting for number of items stored in a subspace
to allow better pruning. Instead of pruning based on the number of
dimension_slices in subsequent dimensions we now track number of total
items in the subspace store and prune based on that.
We add two GUC variables:
1) max_open_chunks_per_insert (default work_mem in bytes / 512. This
assumes an entry is 512 bytes)
2) max_cached_chunks_per_hypertable (default 100). Maximum cached chunks per
hypertable.
Renaming constraints on hypertables is now supported. Using the ONLY
option with RENAME CONSTRAINT is blocked for hypertables. Renaming
constraints on chunks is also blocked.
This commit also fixes a possible bug in chunk_constraint code.
Previously, `chunk_constraint_add_from_tuple` used GETSTRUCT on the
tuple to convert the tuple to `FormData_chunk_constraint`. This
was incorrect since some fields can be NULL. Using GETSTRUCT for
tables with NULLABLE fields is unsafe and indeed was incorrect·
in this case.
This PR adds the ability to have multiple different versions of the timescaledb
extension be used by different databases in the same PostgreSQL
instance (server).
This is accomplished by splitting this extension into two .so files.
1) timescaledb.so -- stuff under loader/. Really not a lot of code.
This code MUST be backwards compatible in the future.
2) timescaledb-version.so (most of our code). Need
not be backwards compatible.
Timescaledb.so becomes a small stub which is preloaded and whose main
reason for existing is to dynamically load the right
timescaledb-version.so when the time comes.
This change allows either of the above .so to be loaded in
shared_preload_libraries. But timescaledb.so allows for multiple
versions used on different databases in the same instance along
with smoother upgrades. Using timescaledb-version.so allows for
finer-grained control and lock-in and is appropriate in only a few
production environments.
This PR also adds version checking so that a clear failure message
will be displayed if the .so version does not match the SQL extension
version.
To support multi-version functionality we changed the way SQL update
scripts are generated. Previously, the system used a bunch of
intermediate upgrade scripts. So with 3 versions, you would have an
update script of 1--2, 2--3. But, this PR changes things so that we
produce direct "shortcut" update files: 1--3, 2--3.
This is done for 2 reasons:
1) Each of the update files should point to
$libdir/timescaledb-current_version. Since you cannot guarantee that
Previous .so for each intermediate version has been installed.
2) You don't want intermediate version updates installed without the
.so. For example, if you have versions 1,2,3
and you are installing version 3, you want the upgrade files 1--3,
2--3 but not 1--2 because if you have 1--2
then a user could do ALTER EXTENSION timescaledb UPDATE TO 2. But
the .so for version 2 may not be installed.
In order to test this functionality, we add a mock extension version .so
that we can test extension loading inside the regression framework.
When tables are created with storage parameters (reloptions),
these should be inherited by chunks so that one can set options
such as fillfactor and autovacuum settings on the hypertable.
This change makes chunks inherit the reloptions set on a hypertable
and allows altering options via the ALTER TABLE command.
