This patch allows unique constraints on compressed chunks. When
trying to INSERT into compressed chunks with unique constraints
any potentially conflicting compressed batches will be decompressed
to let postgres do constraint checking on the INSERT.
With this patch only INSERT ON CONFLICT DO NOTHING will be supported.
For decompression only segment by information is considered to
determine conflicting batches. This will be enhanced in a follow-up
patch to also include orderby metadata to require decompressing
less batches.
Consider a compressed hypertable has many columns (like more than 600 columns).
In call to compress_chunk(), the compressed tuple size exceeds, 8K which causes
error as "row is too big: size 10856, maximum size 8160."
This patch estimates the tuple size of compressed hypertable and reports a
warning when compression is enabled on hypertable. Thus user gets aware of
this warning before calling compress_chunk().
Fixes#4398
When using a custom ENUM data type for compressed hypertable on the GROUP BY
clause raises an error.
Fixed it by generating scan paths for the query by checking if the SEGMENT BY
column is a custom ENUM type and then report a valid error message.
Fixes#3481
When enabling compression on a hypertable, the orderby option can be
omitted, which will set the default value of "time DESC".
However previously, when executing the same command twice not setting
orderby, the second time we would get an error that orderby was
previously set and must be specified.
For example when executing
`alter table set (timescaledb.compress, timescaledb.segmentby = '..')`
The reason for that error was that it's unclear if no orderby means
leave as is, or if it means set the default value.
But in the case where orderby is already set to the default value,
there is no ambiguity and both cases are equivalent, so the default
value can be reset without giving an error.
Fixes#4331
Stop throwing exception with message "column of relation already exists"
when running the command ALTER TABLE ... ADD COLUMN IF NOT EXISTS ...
on compressed hypertables.
Fix#4087
We cache the Chunk structs in RelOptInfo private data. They are later
used to estimate the chunk sizes, check which data nodes they belong
to, et cetera. Looking up the chunks is expensive, so this change
speeds up the planning.
This patch allows dropping columns from hypertables with compression
enabled when the dropped column is neither an orderby nor a
segmentby column.
Fixes#3683
When adding a new column with a default postgres doesnt materialize
the default expression but just stores it in catalog with a flag
signaling that it is missing from the physical row.
Any expressions used as default that do not require materialization
can be allowed on compressed hypertables as well.
The following statements will work on compressed hypertables with
this patch:
ALTER TABLE t ADD COLUMN c1 int DEFAULT 42;
ALTER TABLE t ADD COLUMN c2 text NOT NULL DEFAULT 'abc';
This PR removes the C code that executes the compression
policy. Instead we use a PL/pgSQL procedure to execute
the policy.
PG13.4 and PG12.8 introduced some changes
that require PortalContexts while executing transactions.
The compression policy procedure compresses chunks in
multiple transactions. We have seen some issues with snapshots
and portal management in the policy code (due to the
PG13.4 code changes). SPI API has transaction-portal management
code. However, the compression policy code does not use SPI
interfaces. But it is fairly easy to just convert this into
a PL/pgSQL procedure (which calls SPI) rather than replicating
portal managment code in C to manage multiple txns in the
compression policy.
This PR also disallows decompress_chunk, compress_chunk and
recompress_chunk in txn read only mode.
Fixes#3656
Errors and messages are overhauled to conform to the official
PostgreSQL style guide. In particular, the following things from the
guide has been given special attention:
* Correct capitalization of first letter: capitalize only for hints,
and detail messages.
* Correct handling of periods at the end of messages (should be elided
for primary message, but not detail and hint messages).
* The primary message should be short, factual, and avoid reference to
implementation details such as specific function names.
Some messages have also been reworded for clarity and to better
conform with the last bullet above (short primary message). In other
cases, messages have been updated to fix references to, e.g., function
parameters that used the wrong parameter name.
Closes#2364
This change replaces the add_drop_chunks_policy function with
add_retention_policy. This also renames the older_than parameter
of that function as retention_window. Likewise, the
remove_drop_chunks_policy is also being renamed
remove_retention_policy.
Fixes#2119
Drop Foreign Key constraints from uncompressed chunks during the
compression. This allows to cascade data deletion in FK-referenced
tables to compressed chunks. The foreign key constrains are restored
during decompression.
WITH OIDS option doesn't exist in PG12. This fix moves the test of
blocking compression for tables WITH OIDS into a separate file, which
is run only for PG version before 12.
The constraint check previously assumed that the col_meta
offset for a column was equal to that columns attribute
offset. This is incorrect in the presence of dropped columns.
Fixed to match on column names.
Fixes#1590
Since enabling compression creates limits on the hypertable
(e.g. types of constraints allowed) even if there are no
compressed chunks, we add the ability to turn off compression.
This is only possible if there are no compressed chunks.
This commit improves the API of compress_chunk and decompress_chunk:
- have it return the chunk regclass processed (or NULL in the
idempotent case);
- mark it as STRICT
- add if_not_compressed/if_compressed options for idempotency
Some small improvements:
- allow alter table with empty segment by if the original definition
had an empty segment by. Improve error msgs.
- block compression on tables with OIDs
- block compression on tables with RLS
Block most DDL commands on hypertables with compression enabled.
This restriction will be relaxed over time.
The only alter table commands currently allowed are:
Add/Remove index, Set storage options, clustering index,
set statistics and set tablespace.
We also disallow reseting compression options.
Primary and unqiue constraints are limited to segment_by and order_by
columns and foreign key constraints are limited to segment_by columns
when creating a compressed hypertable. There are no restrictions on
check constraints.
This simplifies the code and the access to the min/max
metadata. Before we used a custom type, but now the min/max
are just the same type as the underlying column and stored as two
columns.
This also removes the custom type that was used before.
- Fix declaration of functions wrt TSDLLEXPORT consistency
- Empty structs need to be created with '{ 0 }' syntax.
- Alignment sentinels have to use uint64 instead of a struct
with a 0-size member
- Add some more ORDER BY clauses in the tests to constrain
the order of results
- Add ANALYZE after running compression in
transparent-decompression test
Add drop policy can only be added to the public uncompressed hypertable.
This blocks a call when a policy is added to the internal
hypertable with compressed data.
This commit add handling for dropping of chunks and hypertables
in the presence of associated compressed objects. If the uncompressed
chunk/hypertable is dropped than drop the associated compressed object
using DROP_RESTRICT unless cascading is explicitly enabled.
Also add a compressed_chunk_id index on compressed tables for
figuring out whether a chunk is compressed or not.
Change a bunch of APIs to use DropBehavior instead of a cascade bool
to be more explicit.
Also test the drop chunks policy.
Add the column to the order by list if it's not already there.
This is never wrong and might improve performance. This
also guarantees that we have at least one ordering column
during compression and therefore can always use tuplesort
(o/w we'd need a non-tuplesort method of getting tuples).
Replace custom parsing of order by and segment by lists
with the postgres parser. The segment by list is now
parsed in the same way as the GROUP BY clause and the
order by list in the same way as the ORDER BY clause.
Also fix default for nulls first/last to follow the PG
convention: LAST for ASC, FIRST for DESC.
This is useful, if some or all compressed columns are NULL.
The count reflects the number of uncompressed rows that are
in the compressed row. Stored as a 32-bit integer.
Add support for compress_chunks function.
This also adds support for compress_orderby and compress_segmentby
parameters in ALTER TABLE. These parameteres are used by the
compress_chunks function.
The parsing code will most likely be changed to use PG raw_parser
function.
