Previously, chunks could end up unaligned betweeen partitions if the chunk_time_interval
was updated between chunk creations. Now there is a check when a chunk is created whether there is
a chunk matching the time point in another partition and if there is, the new chunk is created
with the same interval.
Previously the count returned by insert and copy was wrong because
the count was reset on every execute. But, often there is an execute
in the middle of an insert (i.e. to create a chunk). This fixes the
logic to reset the count only at the start of the top-level statement.
Fixes#64
Clean up the table schema to get rid of legacy tables and functionality
that makes it more difficult to provide an upgrade path.
Notable changes:
* Get rid of legacy tables and code
* Simplify directory structure for SQL code
* Simplify table hierarchy: remove root table and make chunk tables
* inherit directly from main table
* Change chunk table suffix from _data to _chunk
* Simplify schema usage: _timescaledb_internal for internal functions.
* _timescaledb_catalog for metadata tables.
* Remove postgres_fdw dependency
* Improve code comments in sql code
The timestamp test broke with Postgres 9.6.3, because that version
contained an accuracy fix for the `to_timestamp()` function, giving
slightly different results for some timestamp values. This fixes the
timestamp test to expect the correct `to_timestamp()` output as of
version 9.6.3 of Postgres.
This fixes a bug that caused the wrong column to be referenced when
scanning the partition_epoch table. The bug did not usually manifest
itself because the referenced (hypertable) ID was typically
incremented in unison with the desired (partition epoch) ID.
Previously create_hypertable() would throw an error when called on
an already existing hypertable. This can now be skipped by setting
if_not_exists argument to true.
This PR removes the need to run setup_timescaledb. It also fixes
pg_dump and pg_restore. Previously, the database would restore in
a broken state because trigger functions were never attached to
meta tables (since setup_timescaledb() was not run). However, attaching
those triggers at extension creation also causes problems since the data
copy happens after extension creation but we don't want triggers fired
on the data restored (that could cause duplicate rows, for example).
The solution to this chicken-and-egg problem in this PR is to have
a special configuration (GUC) variable `timescaledb.restoring` that,
if 'on', would prevent the extension from attaching triggers at
extension creation. Then, after restoration, you'd call
restore_timescaledb() to attach the triggers and unset the GUC above.
This procedure is documented in the README as part of this PR.
Changed create_hypertable to add time DESC and partitioning_column, time DESC
indexes by default (the latter only if partitioning column not null).
Indexes only created if there is no index on time and
partitioning_column, time respectively. Index creation can be turned off
with the create_default_indexes parameter to create_hypertable (default true).
This PR disables query optimizations on regular tables by default.
The option timescaledb.optimize_plain_tables = 'on' enables them
again. timescaledb.disable_optimizations = 'on' disables all
optimizations (note the change from 'true' to 'on').
Previously, an ALTER TABLE ADD COLUMN IF NOT EXISTS on a hypertable
caused an error if the column already did exist. The reversed problem
with DROP COLUMN is also fixed. Resolves issue #42.
Previously a query like `SELECT "1dim" FROM "1dim"` on a hypertable would
fail since the change_table_walker did not properly change the types on
whole row vars. That is now fixed.
This PR adds more regression tests for index creation and tests for more
user-errors. Significantly, it checks for the presence of both the time
and spaced-partition columns in unique indexes. This is needed because
Timescale cannot guarantee uniqueness if colliding rows don't land in the
same chunk. Fixes#29.
remove all murmur3-related source code. Alter regression tests
to reflect new hash values for inputs, and a slightly different
set of input data to ensure that sufficient chunks and partitions
are tested. Some changes to .sh scripts in sql/setup that seem
to be used only to power the "unit tests", which I cannot
yet run successfully.
The dblink extension is blacklisted by some cloud-hosting providers and
is an unnecessary dependency for single-node operation. Since we don't plan
to use dblink to implement clustering this PR removes the dependency.
Handle case where there is an additional constraint on time.
This makes the pathkeys for the IndexPath incompatible with
upper level ordering. Since we know the appropriate transform
for those pathkeys already, apply it to those paths too.
Common time-series rollups are of the form SELECT date_trunc(const,
time) as t, aggregates... GROUP BY t ORDER BY t DESC LIMIT const2.
Previously, Postgres would not optimize such queries because it
could not match date_trunc(const, time) to any indexes on time, since
it has no way to know that a sort on time is always a valid sort on
date_trunc(const, time).
This commit implements this optimization for date_trunc but it could
apply to a wider range of functions, to be implemented later.
Previously, each test set their own (although mostly the same)
configuration for log output and error verbosity. This is now set
globally in the test runner so that tests only need to set these
configuration parameters if they need to override the defaults. The
log verbosity is also reduced so that errors aren't generated with the
line number of the source file that output the error. Line numbers in
the output can break tests when upgrading to a new PostgreSQL version
that outputs a different line number.
DROP EXTENSION didn't properly reset caches and other saved state
causing various errors related to bad state when the extension was
dropped and/or recreated later.
This patch adds functionality to track the state of the extension and
also signals DROP EXTENSION to other backends that might be running,
allowing them to reset their internal extension state.
If an error is generated in any of the insert triggers, the insert
state kept during a batch insert might be left in an undefined state,
breaking the next insert. This patch makes sure errors are captured in
the insert triggers so that the state can be cleaned up.
If a user attempts to setup a database while not connecting using
the network, port is NULL and thus fails constraint checks. Instead,
we now use the default Postgres port of 5432 when this happens.
Also, setup_db() is renamed to setup_timescaledb() for clarity in
the presence of other extensions.
These tests show that non-aggregated queries that have an ORDER BY time
DESC LIMIT x structure perform well. Postgres processes such queries by
using a time DESC index and then performs a MergeAppend. Some further
optimization to avoid touching unnecessary tables due to constraints
can be added but they are not a priority. Much worse is that these
optimizations do not work when grouping by time (either through
integer division or using date_trunc).
In order to use the non-aggregated query optimizations we had
to create an index with time as the leading field and no
WHERE clause on the index.
Previously, the planner used a direct query via the SPI interface to
retrieve metadata info needed for query planner functions like query
rewriting. This commit updates the planner to use our caching system.
This is a performance improvement for pretty much all operations,
both data modifications and queries.
For hypertables, this added a cache keyed by the main table OID and
added negative entries (because the planner often needs to know if a
table is /not/ a hypertable).
Previous to this commit non-superusers could not do anything inside
a database with the timescale extension loaded. Now, non-superuser
can create their own hypertables and work inside the db. There are
two big caveats:
1) All users have read/write permissions to the timescaledb
catalog.
2) Permission changes applied to the main tables are not
propagated to the associated tables.
This patch refactors the insert path to use insert triggers
instead of a temporary copy table. The copy table previously
served as an optimization for big batches where the cost of
inserting tuple-by-tuple into chunks was amortized by inserting
all tuples for a specific chunk in one insert. However, to avoid
deadlocks the tuples also had to inserted in a specific chunk
order, requiring adding an index to the copy table.
With trigger insertion, tuples are instead collected over a batch
into a sorting state, which is sorted in an "after" trigger. This
removes the overhead of the copy table and index. It also provides
a fast-path for single-tuple batches that avoids doing sorting
altogether.
Since create_hypertable() allows you to optionally specify a
partitioning column, it makes sense to default to one partition when
no column is specified and asking for the number of partitions when a
column is specified and the number of partitions is not (instead of
defaulting to one).
This patch also changes the order and type of partitioning-related
input arguments to create_hypertable() so that the number of
partitions can easily be specified alongside the partitioning column
and without type casting.
This patch fixes two deadlock cases.
The first case occurred as a result of taking partition and chunk
locks in inconsistent orders. When creating the first chunk C1
in a table, concurrent INSERT workers would race to create
that chunk. The result would be that the transactions queue up on
the partition lock P, effectively serializing these transactions.
This would lead to these concurrent transactions to insert
at very different offsets in time, one at a time. At some point
in the future, some n'th transaction Tn queued up on P would get
that lock as the preceeding inserters T1-(n-1) finish their inserts
and move on to their next batches. When Tn finally holds P, one of
the preceeding workers starts a new transaction that finds that it
needs to close C1, grabbing a lock on C1 and then on P. However,
it will block on P since Tn already holds P. Tn will also believe
it needs to close C1, thus trying to grab a lock on C1, but will
block, causing a deadlock.
The second case can occur on multi-partition hypertables. With
multiple partitions there are more than one open-ended chunk
at a time (one for each partition). This leads to a deadlock case
when two processes try to close (and thus lock) the chunks in
different order. For instance process P1 closes chunk C1 and then
C2, while process P2 locks in order C2 and C1.
The fix for the first case is to remove the partition lock
altogether. As it turns out, this lock is not needed.
Instead, transactions can race to create new chunks, thus causing
conflicts. A conflict in creating a new chunk can safely be
ignored and it also avoids taking unecessary locks. Removing the
partition lock also avoids the transaction serialization that
happens around this lock, which is especially bad for long-running
transactions (e.g., big INSERT batches).
The fix for the second multi-partition deadlock case is to always
close chunks in chunk ID order. This requires closing chunks at
the end of a transaction, once a transaction knows all the chunks
it needs to close. This also has the added benefit of reducing the
time a transaction holds exclusive locks on chunks, potentially
improving insert performance.
