This change introduces two ways of fetching data from data nodes: one
using cursors and another one using row-by-row mode. The major
benefit of row-by-row mode is that it enables running parallel plans
on data nodes. The default data fetcher uses row-by-row mode. A new
GUC `timescaledb.remote_data_fetcher` has been added to enable
switching between these two implementations (rowbyrow or cursor).
This change extends telemetry report and adds new 'distributed_db'
section which includes following keys: 'distributed_member',
'data_nodes_count' and 'distributed_hypertables_count'.
Planning of a data node rel during a distributed query should use the
accumulated stats from the individual chunks that the data node rel
represents. Since the data node rel is not a real base rel (i.e., it
doesn't correspond to a real relation) it doesn't have any statistics
in the `pg_catalog` that can be used for planning. Thus, some
functions, such as `set_baserel_size_estimates` will return strange
estimates for data node rels when the planner believes it has stats
(e.g., after an ANALYZE).
This change fixes this issue by not relying on the planner to compute
rel estimates for data node rels. Instead the accumulated estimates
based on the chunks queried by the data node rel are used. This also
obviates the need to compute these stats again.
Given the new size estimates that this change enables, some plan/test
outputs have changed and tests updated to deal with that.
The access node and the data node need to have the same encoding, the
same ctype, and the same collation, so the following changes where
made when bootstrapping the database on the data node:
- Explicity use `template0`
- Explicitly set the encoding of the database on the access node
- Explicitly set the `LC_CTYPE` of the database on the access node
- Explicitly set the `LC_COLLATE` to the collation of the database on
the access node
When not bootstrapping, it is checked that the encoding, `LC_CTYPE`,
and collation matches what the access node is using.
This change prepares the repo for developing the 2.0.0-beta3.
The CMake configuration required an update to handle both
a beta and dev suffix in the version string.
**For beta releases**, upgrading from an earlier version of the
extension (including previous beta releases) is not supported.
This release introduces *distributed hypertables*, a major new feature
that allows hypertables to scale out across multiple nodes for
increased performance and fault tolerance. Please review the
documentation to learn how to configure and use distributed
hypertables, including current limitations.
For some queries, with only one node "hit', push down can be forced by
matching the partitioning attributes with the group by expression.
However, push down didn't happen in some of these cases because of a
bug in `force_group_by_push_down` that didn't properly update the
number of attributes in the partitioning expression.
This change fixes the bug to always set the number of partitioning
attributes to match the group by clause.
Set max retries to 0 for the drop chunks background job in the
background worker reorder drop chunk test, so it will not do a retry,
which is not expected by the test, and thus the test passes on ARM.
The number of output rows estimated for "remote" upper rels could
sometimes erroneously be zero. This happened when computing the
estimate for upper rels with different pathkeys: in case of several
different path keys the estimation was not recalculated and instead
relied on cached values from the first calculation on the same
rel. However, the number of output rows was never cached so the second
pathkey estimated for the upper rel would always produce zero output
rows. This has now been fixed by storing the output rows in the upper
rel after the first estimation.
This fix affects some query plans so a number of tests are affected.
If `remote_txn_check_for_leaked_prepared_statements` do not have a
working connection, it will abort by crashing the server. Since there
are tests that kill the remote server in different phases of the 2PC,
bad timing might cause the server to crash rather than generate an
error.
This commit replace the assertion with both a status check and a check
that the correct number of rows and columns are returned and will
generate an status message with the error message, if any.
A new function, `get_chunk_relstats()`, allows fetching relstats
(basically `pg_class.{relpages,reltuples`) from remote chunks on data
nodes and writing it to the `pg_class` entry for the corresponding
local chunk. The function expects either a chunk or a hypertable as
input and returns the relstats for the given chunk or all chunks for
the given hypertable, respectively.
Importing relstats as described is useful as part of a distributed
ANALYZE/VACUUM that won't require fetching all data into the access
node for local sampling (like the current implemention does).
In a future change, this function will be called as part of a local
ANALYZE on the access node that runs ANALYZE on all data nodes
followed by importing of the resulting relstats for the analyzed
chunks.
Debug printouts are added at two locations:
* Inside `get_foreign_upper_paths` a notice with the existing paths is
send back together with the stage if the `show_upper` is set to show
the stage.
* Inside `set_rel_pathlist` a notice is printed with the existing paths
if the `show_rel` debug flag is set.
The debug printouts are sent back to the client as debug messages
(`DEBUG2`), allowing developers to quickly experiment and interactively
see what effect different statements has with the respect to the paths
generated.
In addition, the handling of `debug_optimizer_flag` was not correct and
is fixed here.
* If no `=` was provided to `show_upper`, it triggered a segmentation
fault.
* If the flag was reset, the internal data structure was not updated.
* If just one flag was updated in a `SET` command, the other flag was
kept intact.
This patch upgrades existing dist_cmd functions to support
executing commands which cannot be run inside an active transaction,
such as the VACUUM command.
This patch adds a way to check and print a notice message
to a user who wants to drop an access node database.
Since database drop can only be done using different database
connection, this functionality is implemented inside loader
extension.
Functionality of the security labels are used in order to mark
a distributed database and make this information accessible
by other databases in pg_shseclabel table.
This change allows certain mutable functions to be whitelisted so that
they can be safely pushed down to data nodes. Additionally, this change
will no longer prevent queries containing the `now` function from being
pushed down to data nodes, but will instead replace the function call
with the transaction start time (which is the same value which would be
used had the query been run solely on the access node).
This fixes a problem in the connection tests when converting ints to
datums on certain platforms (e.g., ARM). The appropriate datum
conversion macros weren't used when returning datum results, which
caused errors on ARM platforms.
When deleting a data node it currently clear the `dist_uuid` in the
database on the data node, which require it to be able to connect to
the data node and would also mean that it is possible to re-add the
data node to a new cluster without checking that it is in a consistent
state.
This commit remove the code that clear the `dist_uuid` and hence do not
need to connect to the data nodel. All tests are updated to reflect the
fact that no connection will be made to the data node and that the
`dist_uuid` is not cleared.
For the tests that are using the `remote_node_set_kill_event` function
to perform a hard termination and precise stages of the 2PC the
connection will close with different warning messages depending on the
timing of connection close and SSL close.
To avoid a flaky test, this commit set `client_min_messages` to `ERROR`
for the duration of the transaction. Since checks are done after the
transaction that the transaction is properly rolled back in the event
of a crash, the warning messages does not offer any additional
benefits.
Add a new `debug_optimizer_flags` option where you can request
optimization debug notices. With this commit, the flags `show_upper`
and `show_rel` are added.
The flag `show_upper` will permit sending back a message with the
resulting relation after creating upper paths. Since this is called at
different stages, the flag support setting specific stages where
printouts should be done using the format `show_upper=window,final`.
The flag `show_rel` will permit sending back the resulting relation
after executing `set_rel_pathlist` to add new paths for consideration.
The actual implementation to sent the notices will be in a separate
commit.
Process arguments of data node allow or block new chunks SQL API
functions separately, since the number of optional arguments is
different between allow and block functions. This fixes the bug with
memory access.
This change removes collation checks for foreign expressions. With
distributed hypertables, we assume that all participating nodes have
the same collation configuration so there should be no difference
(from a collation perspective) between executing an expression locally
and remotely.
We want to get more insights about what plans
are executed on data nodes. If a user runs
explain with verbose option we will connect to each data node,
run explain on data node and
print output together with existing explain output.
If something is wrong with a connection we would not be able to
start a transaction on the remote node. So when abort happens we
shoud not try to abort a transaction that hasn't started.
We use xact_depth to check if transaction actually started.
If we find one we remove it from our tx cache which should result in removing
the invalid connection as well.
In order to debug the optimizer it is necessary to provide the
`OPTIMIZER_DEBUG` preprocessor symbol, so added an option to enable
this for the code.
It still requires the PostgreSQL source code to be built with this
flag.
This change tweaks the `docker-build.sh` script to work against remote
Docker daemons. This is useful when testing on minikube and you want
to quickly build and deploy a new image with the current code without
having to push to a remote registry.
This change ensures that all dimension-releated DDL commands on
hypertables are distributed to its data nodes. Most importantly,
`add_dimension()` now applies across all data nodes of a hypertable
when issued on the access node. This ensures the dimension
configuration is the same on all nodes of a particular hypertable.
While the configuration of chunk time interval and number of
partitions do not matter that much for data nodes (since the access
node takes care of sizing chunks), functions like
`set_chunk_time_interval()` and `set_number_partitions()` are
distributed as well. This ensures that dimension-related configuration
changes apply across all nodes.
In some plans DataNodeScan can be buried under
Aggregate node. We now optimize for that as well.
We attempy async optimization only when distributed hypertable
is participating in the query.
When the access node executes `add_data_node`, bootstrapping the data
node is done by:
1. Optionally creating the database on the remote server.
2. Creating a schema for the TimescaleDB extension objects.
3. Creating the TimescaleDB extension in the database.
After bootstrapping, the `dist_uuid` of the data node and access node
is set to the `uuid` of the access node.
If `bootstrap` is `true`, bootstrapping of the data node is done.
If `boostrap` is `false`, bootstrapping is not done, but the procedure
attempts to connect to the database and verify that the TimescaleDB
extension is loaded and that the `dist_uuid` is clear. If it is not
possible to connect to the database, or if `dist_uuid` is set,
`add_data_node` will fail.
Certain functions invoked on an access node need to be passed on to
data nodes to ensure any mutations happen also on those
nodes. Examples of such functions are `drop_chunks`, `add_dimension`,
`set_chunk_time_interval`, etc. So far, the approach has been to
deparse these "manually" on a case-by-case basis.
This change implements a generalized deparsing function that deparses
the function based on the function call info (`FunctionCallInfo`) that
holds the information about any invoked function that can be used to
deparse the function call.
The `drop_chunks` function has been updated to use this generalized
deparsing functionality when it is invoking remote nodes.
Compare remote connection extension version with the one installed
on the access node. Show a warning message if it differs.
Check happens during add_data_node() call and after creating new
data node connection.
This commit changes so that the same user is used both on the access
node and the data nodes when executing a `add_data_node`, which means
that the `bootstrap_user` parameter is removed.
Since most tests assume that you can pass a separate user with
superuser privileges to `add_data_node`, this affected a lot of tests.
In some cases a cursor might have pending data fetch requests while
we attempt to close it.We need to make sure to throw away any pending
cursor requests before proceeding with sending CLOSE request or cursor
rewind.
This commit also prevents `ANALYZE disttable` from failing (due to
fetching random tuples and not using next_tuple marker)
int64 is defined differently depending on environment variables.
Implementation of deparse, which is used to generate commands to
create hypertable on data nodes, has assumed only one possible
definition. This is fix to use correct formatting.
The general idea with AsyncAppend is to do more things in parallel
when executing a query on a distributed hypertable. With AsyncAppend,
we use TimescaleDB's Async API to asynchronously create cursors and
fetch data from data nodes. We modify existing planner code to inject
an AsyncAppend node which will take care of sending/processing async
requests. An AsyncAppend node is being injected as a parent of Append
or MergeAppend and uses their child nodes (DataNodeScan) to perform
async calls.
Change `add_data_node` so that host parameter is required. If the host
parameter is not provided, or is `NULL`, an error will be printed.
Also change logic for how the default value for `port` is picked. Now
it will by default use the port given in the configuration file.
The commit update all the result files, add the `host` parameter to all
calls of `add_data_node` and add a few tests to check that an error is
given when `host` is not provided.
Replaces assert of expected result after completing COPY command with
reporting error on unexpected state, since asserts are removed on
release builds.
Checks that a data node connection is created with the TimescaleDB
foreign data wrapper. Refactors existing code to remove duplicated
code related to TimescaleDB foreign data wrapper.
