3815 Commits

Author SHA1 Message Date
Rafia Sabih
4cb76bc053 Cosmetic changes to create.c 2023-02-06 22:39:57 +05:30
Sven Klemm
8132908c97 Refactor chunk decompression functions
Restructure the code inside decompress_chunk slightly to make core
loop reusable by other functions.
2023-02-06 14:52:06 +01:00
Lakshmi Narayanan Sreethar
fb3ad7d6c6 Release 2.9.3
This release contains bug fixes since the 2.9.2 release.
This release is high priority for upgrade. We strongly recommend that you
upgrade as soon as possible.

**Bugfixes**
* #4804 Skip bucketing when start or end of refresh job is null
* #5108 Fix column ordering in compressed table index not following the order of a multi-column segment by definition
* #5187 Don't enable clang-tidy by default
* #5255 Fix year not being considered as a multiple of day/month in hierarchical continuous aggregates
* #5259 Lock down search_path in SPI calls
2023-02-03 20:04:18 +05:30
Erik Nordström
206056ca12 Fix dist_hypertable test
A previous change accidentally broke the dist_hypertable test so that
it prematurely exited. This change restores the test so that it
executes properly.
2023-02-03 13:35:36 +01:00
Erik Nordström
b81033b835 Make data node command execution interruptible
The function to execute remote commands on data nodes used a blocking
libpq API that doesn't integrate with PostgreSQL interrupt handling,
making it impossible for a user or statement timeout to cancel a
remote command.

Refactor the remote command execution function to use a non-blocking
API and integrate with PostgreSQL signal handling via WaitEventSets.

Partial fix for #4958.

Refactor remote command execution function
2023-02-03 13:15:28 +01:00
Alexander Kuzmenkov
c4d8f35307 Improve automated backports
A follow-up for the review comments in the previous PR.

1. Create one backport PR per one source PR, even with multiple commits.
1. Add a comment to the source PR if we fail to backport it for some
   reason.
2023-02-02 16:13:33 +04:00
Konstantina Skovola
6bc8980216 Fix year not multiple of day/month in nested CAgg
Previously all intervals were converted to seconds using "epoch"
with date_part. However, this treats a year as 365.25 days to
account for leap years, leading to the unexpected situation that
a year is not a multiple of a day or a month.

Fixed by treating month-only intervals as multiples of 30 days.

Fixes #5231
2023-02-02 12:14:37 +02:00
Alexander Kuzmenkov
9133319081 Fix pylint again
Apparently a new version is out and it complains about Exception being
too general.
2023-02-02 08:57:42 +04:00
Alexander Kuzmenkov
739fd00bb9 Add a workflow for automated backporting
1. Find the latest release branch
1. For each commit in main and not in release branch (compared by
   title), find the corresponding PR.
1. If the PR fixes an issue labeled "bug", and neither the PR nor the
   issue are labeled "no-backport", cherry-pick the commits from the PR
onto the release branch, and create a PR with these changes.
2023-02-01 16:25:02 +04:00
Sven Klemm
2a47462fbc Remove SECURITY DEFINER from get_telemetry_report
We should not broadly make functions security definer as that increases
the attack surface for our extension. Especially for our telemetry we
should strive to only run it with the minimum required privileges.
2023-02-01 13:10:23 +01:00
Alexander Kuzmenkov
44cd71a602 Fix the python code style check
For an unknown reason, pip started to install an older version of
prospector which is incompatible with the current pylint. Require the
new prospector version explicitly.
2023-02-01 15:00:00 +04:00
Sven Klemm
d8f19e57a0 Bump version of setup-wsl github action
The currently used version pulls in Node.js 12 which is deprecated
on github.

https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/
2023-02-01 11:53:48 +01:00
Sven Klemm
789bb26dfb Lock down search_path in SPI calls 2023-02-01 07:54:03 +01:00
Alexander Kuzmenkov
f75a51def7 Run yamllint in CI
Helps find errors in GitHub workflows.
2023-01-31 20:54:02 +04:00
Fabrízio de Royes Mello
c0f2ed1809 Mark cagg_watermark parallel safe
The `cagg_watermark` function perform just read-only operations so is
safe to make it parallel safe to take advantage of the Postgres
parallel query.

Since 2.7 when we introduced the new Continuous Aggregate format we
don't use partials anymore and those aggregate functions
`partialize_agg` and `finalize_agg` are not parallel safe, so make no
sense don't take advantage of Postgres parallel query for realtime
Continuous Aggregates.
2023-01-31 13:07:19 -03:00
Fabrízio de Royes Mello
e6173d1241 Remove unused function prototype 2023-01-31 09:36:59 -03:00
Erik Nordström
d489ed6f32 Fix use of prepared statement in async module
Broken code caused the async connection module to never send queries
using prepared statements. Instead, queries were sent using the
parameterized query statement instead.

Fix this so that prepared statements are used when created.
2023-01-31 12:01:03 +01:00
Lakshmi Narayanan Sreethar
1a3e7ad7d1 Run dist_move_chunk as a solo test in PG15
When run in a parallel group, the dist_move_chunk test can get into a
deadlock with another test running a 'DROP DATABASE' command. So, mark
it as a solo test to disallow it from running in a parallel group.

Closes #4972
2023-01-31 14:26:17 +05:30
Lakshmi Narayanan Sreethar
03b740cd70 Enable telemetry_stats testcase
The telemetry_stats testcase was accidentally disabled by PR #5162.
2023-01-31 13:25:02 +05:30
Erik Nordström
5d12a3883d Make connection establishment interruptible
Refactor the data node connection establishment so that it is
interruptible, e.g., by ctrl-c or `statement_timeout`.

Previously, the connection establishment used blocking libpq calls. By
instead using asynchronous connection APIs and integrating with
PostgreSQL interrupt handling, the connection establishment can be
canceled by an interrupt caused by a statement timeout or a user.

Fixes #2757
2023-01-30 17:48:59 +01:00
Sven Klemm
01ea255f2f Rename variable in ts_chunk_dispatch_get_chunk_insert_state
The variable new_chunk was misleading since the chunk could be
either a new chunk or an existing chunk.
2023-01-30 16:47:16 +01:00
Sven Klemm
b229b3aefd Small decompress_chunk refactor
Refactor the decompression code to move the decompressor
initialization into a separate function.
2023-01-30 16:47:16 +01:00
Erik Nordström
cce0e18c36 Manage life-cycle of connections via memory contexts
Tie the life cycle of a data node connection to the memory context it
is created on. Previously, a data node connection was automatically
closed at the end of a transaction, although often a connection needs
to live beyond a single transaction. For example, a connection cache
is maintained for data node connections, and, for such cases, a flag
was set on a connection to avoid closing it automatically.

Instead of tying connections to transactions, they are now life-cycle
managed via memory contexts. This simplifies the handling of
connections and avoids having to create exceptions to closing
connections at transaction end.
2023-01-30 15:46:21 +01:00
Sven Klemm
872128438f Update checkout version in changelog-check
Update checkout to not pull in Node.js 12 which is deprecated on
GitHub.

https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/.
2023-01-30 15:22:08 +01:00
Mats Kindahl
5661ff1523 Add role-level security to job error log
Since the job error log can contain information from many different
sources and also from many different jobs it is important to ensure
that visibility of the job error log entries is restricted to job
owners.

This commit extend the view `timescaledb_information.job_errors` with
role-based checks so that a user can only see entries for jobs that she
has permission to view and restrict the permissions to
`_timescaledb_internal.job_errors` so that users only can view the job
error log through the view. A special case is added so that the
superuser and the database owner can see all log entries, even if there
is no associated job id with the log entry.

Closes #5217
2023-01-30 12:13:00 +01:00
Alexander Kuzmenkov
c89fb2550d Run sqlsmith on all commits in main
This would help to gather more statistics. We have some rare assertion
failures that happen with SQLSmith like #4185
2023-01-30 15:05:57 +04:00
Alexander Kuzmenkov
21a3f8206c Run python linter and formatter in CI
Helps find some errors and cosmetic problems.
2023-01-30 13:49:26 +04:00
Sven Klemm
334864127d Stop blocking RETURNING for compressed chunks
Recent refactorings in the INSERT into compressed chunk code
path allowed us to support this feature but the check to
prevent users from using this feature was not removed as part
of that patch. This patch removes the blocker code and adds a
minimal test case.
2023-01-30 09:49:52 +01:00
Rafia Sabih
043092a97f Fix timestamp out of range
When start or end for a refresh job is null, then it gives an
error while bucketing because start and end are already min and
max timestamp value before bucketing. Hence, skip bucketing for
this case.

Partly fixes #4117
2023-01-26 20:11:21 +05:30
Bharathy
684637a172 Post-release fixes for 2.9.2
Bumping the previous version and adding tests for 2.9.2
2023-01-25 17:54:54 +05:30
Alexander Kuzmenkov
552950d221 Save SQLSmith results to the CI DB
Mostly we are interested in stacktraces and failed queries here.
2023-01-24 20:33:50 +04:00
Rafia Sabih
a67b90e977 Allow joins in continuous aggregates
Enable the support of having join in the query used for creating
the continuous aggregates. It has follwoing restrictions-
1. Join can involve only one hypertable and one normal table
2. Join should be a inner join
3. Join condition can only be equality
2023-01-24 19:57:24 +05:30
Jan Nidzwetzki
9ae3ae33b7 Add scan plan logic for remote joins
This patch adds the missing functionality to create scan plans for
remote joins. Most of the code is a backport from PG Upstream.
2023-01-23 14:04:42 +01:00
Bharathy
f211294c61 Release 2.9.2
This release contains bug fixes since the 2.9.1 release.
We recommend that you upgrade at the next available opportunity.

**Bugfixes**
* #5114 Fix issue with deleting data node and dropping the database on multi-node
* #5133 Fix creating a CAgg on a CAgg where the time column is in a different order of the original hypertable
* #5152 Fix adding column with NULL constraint to compressed hypertable
* #5170 Fix CAgg on CAgg variable bucket size validation
* #5180 Fix default data node availability status on multi-node
* #5181 Fix ChunkAppend and ConstraintAwareAppend with TidRangeScan child subplan
* #5193 Fix repartition behavior when attaching data node on multi-node
2023-01-23 15:55:10 +05:30
Alexander Kuzmenkov
d2254cb5c5 Don't enable clang-tidy by default
This is aimed at developers. If we enable it by default, it confuses our
users and slows down the build for them.
2023-01-20 16:56:08 +04:00
Konstantina Skovola
014b7b9664 Move .gitattributes to root and remove build-13
Commit effc8efe148c4ec0048bd7c1dfe0ca01df2afdc9
accidentally placed .gitattributes inside a build-13 directory
instead of the root of the project. This commit removes build-13 and
fixes the structure.
2023-01-20 10:21:11 +02:00
Jan Nidzwetzki
28dbeaa2ca Add cost estimation for remote joins
This patch adds the missing functionality to estimate the costs of
remote joins. Most of the code is a backport from PG Upstream.
2023-01-19 15:52:37 +01:00
Konstantina Skovola
effc8efe14 Add workflow to check for CHANGELOG update 2023-01-19 13:04:59 +02:00
Erik Nordström
1676259840 Fix repartition behavior when attaching data node
When attaching a data node and specifying `repartition=>false`, the
current number of partitions should remain instead of recalculating
the partitioning based on the number of data nodes.

Fixes #5157
2023-01-18 16:20:49 +01:00
Jan Nidzwetzki
19065bbdf3 Introduce a FDW option to mark reference tables
With this patch, the ability to mark reference tables (tables that exist
on all data nodes of a multi-node installation) via an FDW option has
been added.
2023-01-18 15:11:02 +01:00
Bharathy
9a2cbe30a1 Fix ChunkAppend, ConstraintAwareAppend child subplan
When TidRangeScan is child of ChunkAppend or ConstraintAwareAppend node, an
error is reported as "invalid child of chunk append: Node (26)". This patch
fixes the issue by recognising TidRangeScan as a valid child.

Fixes: #4872
2023-01-18 18:06:30 +05:30
Mats Kindahl
8f4fa8e4cc Add build matrix to Windows and Linux builds
Build matrix is missing from the ignore workflows for the Windows and
Linux builds, so this commit adds them.
2023-01-18 09:41:11 +01:00
Alexander Kuzmenkov
6aa3d6e2fc Don't run the APT package test on PRs
We run them nightly on main.
2023-01-17 14:57:17 +04:00
Jan Nidzwetzki
1d3d81491b Update issue workflow actions and used token
This patch updates the used issue workflow actions and ensures that the
'secrets.ORG_AUTOMATION_TOKEN' is used by all actions.
2023-01-17 11:47:22 +01:00
Dmitry Simonenko
5c897ff75d Fix default data node availability status
Function alter_data_node() return uninitialized value for "available"
option when it is not presented in the option list.

Fix #5154
2023-01-16 16:33:10 +02:00
Sven Klemm
dbe89644b5 Remove no longer used compression code
The recent refactoring of INSERT into compression chunk made this
code obsolete but forgot to remove it in that patch.
2023-01-16 14:18:56 +01:00
Fabrízio de Royes Mello
73df496c75 Fix CAgg on CAgg variable bucket size validation
Previous attempt to fix it (PR #5130) was not entirely correct because
the bucket width calculation for interval width was wrong.

Fixed it by properly calculate the bucket width for intervals using the
Postgres internal function `interval_part` to extract the epoch of the
interval and execute the validations. For integer widths use the already
calculated bucket width.

Fixes #5158, #5168
2023-01-13 12:38:58 -03:00
Mats Kindahl
ca9d508ede Add missing ignore files
The workflow ignore files for 32-bit Linux builds and Windows was
missing from #5156 so these are added here.
2023-01-13 14:02:27 +01:00
Erik Nordström
1e7b9bc558 Fix issue with deleting data node and dropping database
When deleting a data node with the option `drop_database=>true`, the
database is deleted even if the command fails.

Fix this behavior by dropping the remote database at the end of the
delete data node operation so that other checks fail first.
2023-01-13 13:54:27 +01:00
Fabrízio de Royes Mello
4118a72575 Remove parallel safe from partialize_agg
Previous PR #4307 mark `partialize_agg` and `finalize_agg` as parallel
safe but this change is leading to incorrect results in some cases.

Those functions are supposed work in parallel but seems is not the case
and it is not evident yet the root cause and how to properly use it in
parallel queries so we decided to revert this change and provide correct
results to users.

Fixes #4922
2023-01-13 07:31:55 -03:00