1
0
mirror of https://github.com/timescale/timescaledb.git synced 2025-05-23 14:39:15 +08:00

3827 Commits

Author SHA1 Message Date
Zoltan Haindrich
9d3866a50e Accept all compression options on caggs
Enable to properly handle 'compress_segmentby' and 'compress_orderby'
compression options on continous aggregates.

ALTER MATERIALIZED VIEW test_table_cagg SET (
  timescaledb.compress = true,
  timescaledb.compress_segmentby = 'device_id'
);

Fixes 
2023-02-13 22:21:18 +01:00
Alexander Kuzmenkov
d00c1f3721 Fix some errors in processing of code coverage data
We were using mismatched compiler and gcov, which led to either
segfaults or errors like "GCOV_TAG_COUNTER_ARCS mismatch". Add some
cmake code that tries to find the gcov that matches the compiler.

This should hopefully fix some of the mysterious missing coverage
problems that we've been experiencing for a while.
2023-02-13 18:38:09 +04:00
Sven Klemm
9ec11d8af6 Bump PG version used in CI to 12.14, 13.10, 14.7 and 15.2 2023-02-12 14:23:12 +01:00
Sven Klemm
ef25fb9ec7 Add dist_ref_table_join generated test files to .gitignore 2023-02-11 11:31:26 +01:00
Alexander Kuzmenkov
484a4ea3fc Fix our codecov repository yaml
Codecov keeps complaining that it's invalid.
2023-02-11 00:12:18 +04:00
Rafia Sabih
ece15d66a4 Enable real time aggregation for caggs with joins 2023-02-10 22:12:29 +05:30
Konstantina Skovola
348796f9d9 Fix next_start calculation for fixed schedules
This patch fixes several issues with next_start calculation.

- Previously, the offset was added twice in some cases.
This is fixed by this patch.

- Additionally, schedule intervals with month components
were not handled correctly.
Internally, time_bucket with origin is used to calculate
the next start. However, in the case of month intervals, the
timestamp calculated for a bucket is always aligned on the first
day of the month, regardless of origin.
Therefore, previously the result was aligned with origin by adding
the difference between origin and its respective time bucket.
This difference was computed as a fixed length interval in terms
of days and time. That computation led to incorrect computation of
next start occasionally, for example when a job should be executed on
the last day of a month.
That is fixed by adding an appropriate interval of months to
initial_start and letting Postgres handle this computation properly.

Fixes 
2023-02-09 17:57:17 +02:00
Sven Klemm
756ef68d0a Fix compression_hypertable ordering reliance
The hypertable_compression test had on implicit reliance on the
ordering of tuples when querying the materialized results.
This patch makes the ordering explicit in this test.
2023-02-09 15:23:07 +01:00
Alexander Kuzmenkov
caf79e0f5e When backporting, fetch the main branch with large depth
Apparently in some cases we're getting a shallow repo in the Github
Actions.
2023-02-08 18:48:53 +04:00
Alexander Kuzmenkov
063a9dae29 Improve cost model for data node scans
1) Simplify the path generation for the parameterized data node scans.
1) Adjust the data node scan cost if it's an index scan, instead of always
   treating it as a sequential scan.
1) Hard-code the grouping estimation for distributed hypertable, instead
   of using the totally bogus per-column ndistinct value.
1) Add the GUC to disable parameterized
data node scan.
1) Add more tests.
2023-02-08 16:12:01 +04:00
Zoltan Haindrich
cad2440b58 Compression can't be enabled on caggs
The continuous aggregate creation failed in case segmentby/orderby
columns needed quotation.
2023-02-07 21:01:56 +01:00
Lakshmi Narayanan Sreethar
1eb8aa3f14 Post-release fixes for 2.9.3
Bumping the previous version and adding tests for 2.9.3.
2023-02-07 15:53:42 +05:30
Rafia Sabih
4cb76bc053 Cosmetic changes to create.c 2023-02-06 22:39:57 +05:30
Sven Klemm
8132908c97 Refactor chunk decompression functions
Restructure the code inside decompress_chunk slightly to make core
loop reusable by other functions.
2023-02-06 14:52:06 +01:00
Lakshmi Narayanan Sreethar
fb3ad7d6c6 Release 2.9.3
This release contains bug fixes since the 2.9.2 release.
This release is high priority for upgrade. We strongly recommend that you
upgrade as soon as possible.

**Bugfixes**
*  Skip bucketing when start or end of refresh job is null
*  Fix column ordering in compressed table index not following the order of a multi-column segment by definition
*  Don't enable clang-tidy by default
*  Fix year not being considered as a multiple of day/month in hierarchical continuous aggregates
*  Lock down search_path in SPI calls
2023-02-03 20:04:18 +05:30
Erik Nordström
206056ca12 Fix dist_hypertable test
A previous change accidentally broke the dist_hypertable test so that
it prematurely exited. This change restores the test so that it
executes properly.
2023-02-03 13:35:36 +01:00
Erik Nordström
b81033b835 Make data node command execution interruptible
The function to execute remote commands on data nodes used a blocking
libpq API that doesn't integrate with PostgreSQL interrupt handling,
making it impossible for a user or statement timeout to cancel a
remote command.

Refactor the remote command execution function to use a non-blocking
API and integrate with PostgreSQL signal handling via WaitEventSets.

Partial fix for .

Refactor remote command execution function
2023-02-03 13:15:28 +01:00
Alexander Kuzmenkov
c4d8f35307 Improve automated backports
A follow-up for the review comments in the previous PR.

1. Create one backport PR per one source PR, even with multiple commits.
1. Add a comment to the source PR if we fail to backport it for some
   reason.
2023-02-02 16:13:33 +04:00
Konstantina Skovola
6bc8980216 Fix year not multiple of day/month in nested CAgg
Previously all intervals were converted to seconds using "epoch"
with date_part. However, this treats a year as 365.25 days to
account for leap years, leading to the unexpected situation that
a year is not a multiple of a day or a month.

Fixed by treating month-only intervals as multiples of 30 days.

Fixes 
2023-02-02 12:14:37 +02:00
Alexander Kuzmenkov
9133319081 Fix pylint again
Apparently a new version is out and it complains about Exception being
too general.
2023-02-02 08:57:42 +04:00
Alexander Kuzmenkov
739fd00bb9 Add a workflow for automated backporting
1. Find the latest release branch
1. For each commit in main and not in release branch (compared by
   title), find the corresponding PR.
1. If the PR fixes an issue labeled "bug", and neither the PR nor the
   issue are labeled "no-backport", cherry-pick the commits from the PR
onto the release branch, and create a PR with these changes.
2023-02-01 16:25:02 +04:00
Sven Klemm
2a47462fbc Remove SECURITY DEFINER from get_telemetry_report
We should not broadly make functions security definer as that increases
the attack surface for our extension. Especially for our telemetry we
should strive to only run it with the minimum required privileges.
2023-02-01 13:10:23 +01:00
Alexander Kuzmenkov
44cd71a602 Fix the python code style check
For an unknown reason, pip started to install an older version of
prospector which is incompatible with the current pylint. Require the
new prospector version explicitly.
2023-02-01 15:00:00 +04:00
Sven Klemm
d8f19e57a0 Bump version of setup-wsl github action
The currently used version pulls in Node.js 12 which is deprecated
on github.

https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/
2023-02-01 11:53:48 +01:00
Sven Klemm
789bb26dfb Lock down search_path in SPI calls 2023-02-01 07:54:03 +01:00
Alexander Kuzmenkov
f75a51def7 Run yamllint in CI
Helps find errors in GitHub workflows.
2023-01-31 20:54:02 +04:00
Fabrízio de Royes Mello
c0f2ed1809 Mark cagg_watermark parallel safe
The `cagg_watermark` function perform just read-only operations so is
safe to make it parallel safe to take advantage of the Postgres
parallel query.

Since 2.7 when we introduced the new Continuous Aggregate format we
don't use partials anymore and those aggregate functions
`partialize_agg` and `finalize_agg` are not parallel safe, so make no
sense don't take advantage of Postgres parallel query for realtime
Continuous Aggregates.
2023-01-31 13:07:19 -03:00
Fabrízio de Royes Mello
e6173d1241 Remove unused function prototype 2023-01-31 09:36:59 -03:00
Erik Nordström
d489ed6f32 Fix use of prepared statement in async module
Broken code caused the async connection module to never send queries
using prepared statements. Instead, queries were sent using the
parameterized query statement instead.

Fix this so that prepared statements are used when created.
2023-01-31 12:01:03 +01:00
Lakshmi Narayanan Sreethar
1a3e7ad7d1 Run dist_move_chunk as a solo test in PG15
When run in a parallel group, the dist_move_chunk test can get into a
deadlock with another test running a 'DROP DATABASE' command. So, mark
it as a solo test to disallow it from running in a parallel group.

Closes 
2023-01-31 14:26:17 +05:30
Lakshmi Narayanan Sreethar
03b740cd70 Enable telemetry_stats testcase
The telemetry_stats testcase was accidentally disabled by PR .
2023-01-31 13:25:02 +05:30
Erik Nordström
5d12a3883d Make connection establishment interruptible
Refactor the data node connection establishment so that it is
interruptible, e.g., by ctrl-c or `statement_timeout`.

Previously, the connection establishment used blocking libpq calls. By
instead using asynchronous connection APIs and integrating with
PostgreSQL interrupt handling, the connection establishment can be
canceled by an interrupt caused by a statement timeout or a user.

Fixes 
2023-01-30 17:48:59 +01:00
Sven Klemm
01ea255f2f Rename variable in ts_chunk_dispatch_get_chunk_insert_state
The variable new_chunk was misleading since the chunk could be
either a new chunk or an existing chunk.
2023-01-30 16:47:16 +01:00
Sven Klemm
b229b3aefd Small decompress_chunk refactor
Refactor the decompression code to move the decompressor
initialization into a separate function.
2023-01-30 16:47:16 +01:00
Erik Nordström
cce0e18c36 Manage life-cycle of connections via memory contexts
Tie the life cycle of a data node connection to the memory context it
is created on. Previously, a data node connection was automatically
closed at the end of a transaction, although often a connection needs
to live beyond a single transaction. For example, a connection cache
is maintained for data node connections, and, for such cases, a flag
was set on a connection to avoid closing it automatically.

Instead of tying connections to transactions, they are now life-cycle
managed via memory contexts. This simplifies the handling of
connections and avoids having to create exceptions to closing
connections at transaction end.
2023-01-30 15:46:21 +01:00
Sven Klemm
872128438f Update checkout version in changelog-check
Update checkout to not pull in Node.js 12 which is deprecated on
GitHub.

https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/.
2023-01-30 15:22:08 +01:00
Mats Kindahl
5661ff1523 Add role-level security to job error log
Since the job error log can contain information from many different
sources and also from many different jobs it is important to ensure
that visibility of the job error log entries is restricted to job
owners.

This commit extend the view `timescaledb_information.job_errors` with
role-based checks so that a user can only see entries for jobs that she
has permission to view and restrict the permissions to
`_timescaledb_internal.job_errors` so that users only can view the job
error log through the view. A special case is added so that the
superuser and the database owner can see all log entries, even if there
is no associated job id with the log entry.

Closes 
2023-01-30 12:13:00 +01:00
Alexander Kuzmenkov
c89fb2550d Run sqlsmith on all commits in main
This would help to gather more statistics. We have some rare assertion
failures that happen with SQLSmith like 
2023-01-30 15:05:57 +04:00
Alexander Kuzmenkov
21a3f8206c Run python linter and formatter in CI
Helps find some errors and cosmetic problems.
2023-01-30 13:49:26 +04:00
Sven Klemm
334864127d Stop blocking RETURNING for compressed chunks
Recent refactorings in the INSERT into compressed chunk code
path allowed us to support this feature but the check to
prevent users from using this feature was not removed as part
of that patch. This patch removes the blocker code and adds a
minimal test case.
2023-01-30 09:49:52 +01:00
Rafia Sabih
043092a97f Fix timestamp out of range
When start or end for a refresh job is null, then it gives an
error while bucketing because start and end are already min and
max timestamp value before bucketing. Hence, skip bucketing for
this case.

Partly fixes 
2023-01-26 20:11:21 +05:30
Bharathy
684637a172 Post-release fixes for 2.9.2
Bumping the previous version and adding tests for 2.9.2
2023-01-25 17:54:54 +05:30
Alexander Kuzmenkov
552950d221 Save SQLSmith results to the CI DB
Mostly we are interested in stacktraces and failed queries here.
2023-01-24 20:33:50 +04:00
Rafia Sabih
a67b90e977 Allow joins in continuous aggregates
Enable the support of having join in the query used for creating
the continuous aggregates. It has follwoing restrictions-
1. Join can involve only one hypertable and one normal table
2. Join should be a inner join
3. Join condition can only be equality
2023-01-24 19:57:24 +05:30
Jan Nidzwetzki
9ae3ae33b7 Add scan plan logic for remote joins
This patch adds the missing functionality to create scan plans for
remote joins. Most of the code is a backport from PG Upstream.
2023-01-23 14:04:42 +01:00
Bharathy
f211294c61 Release 2.9.2
This release contains bug fixes since the 2.9.1 release.
We recommend that you upgrade at the next available opportunity.

**Bugfixes**
*  Fix issue with deleting data node and dropping the database on multi-node
*  Fix creating a CAgg on a CAgg where the time column is in a different order of the original hypertable
*  Fix adding column with NULL constraint to compressed hypertable
*  Fix CAgg on CAgg variable bucket size validation
*  Fix default data node availability status on multi-node
*  Fix ChunkAppend and ConstraintAwareAppend with TidRangeScan child subplan
*  Fix repartition behavior when attaching data node on multi-node
2023-01-23 15:55:10 +05:30
Alexander Kuzmenkov
d2254cb5c5 Don't enable clang-tidy by default
This is aimed at developers. If we enable it by default, it confuses our
users and slows down the build for them.
2023-01-20 16:56:08 +04:00
Konstantina Skovola
014b7b9664 Move .gitattributes to root and remove build-13
Commit effc8efe148c4ec0048bd7c1dfe0ca01df2afdc9
accidentally placed .gitattributes inside a build-13 directory
instead of the root of the project. This commit removes build-13 and
fixes the structure.
2023-01-20 10:21:11 +02:00
Jan Nidzwetzki
28dbeaa2ca Add cost estimation for remote joins
This patch adds the missing functionality to estimate the costs of
remote joins. Most of the code is a backport from PG Upstream.
2023-01-19 15:52:37 +01:00
Konstantina Skovola
effc8efe14 Add workflow to check for CHANGELOG update 2023-01-19 13:04:59 +02:00