mirror of
https://github.com/facebook/rocksdb.git
synced 2025-05-15 09:23:03 +08:00
cc487ba367
Summary: ## Background Compaction statistics are collected at various levels across different classes and structs. * `InternalStats::CompactionStats`: Per-level Compaction Stats within a job (can be at subcompaction level which later get aggregated to the compaction level) * `InternalStats::CompactionStatsFull`: Contains two per-level compaction stats - `output_level_stats` for primary output level stats and `proximal_level_stats` for proximal level stats. Proximal level statistics are only relevant when using Tiered Storage with the per-key placement feature enabled. * `InternalStats::CompactionOutputsStats`: Simplified version of `InternalStats::CompactionStats`. Only has a subset of fields from `InternalStats::CompactionStats` * `CompactionJobStats`: Job-level Compaction Stats. (can be at subcompaction level which later get aggregated to the compaction level) Please note that some fields in Job-level stats are not in Per-level stats and they don't map 1-to-1 today. ## Issues * In non-remote compactions, proximal level compaction statistics were not being aggregated into job-level statistics. Job level statistics were missing stats for proximal level for tiered storage compactions with per-key-replacement feature enabled. * During remote compactions, proximal level compaction statistics were pre-aggregated into job-level statistics on the remote side. However, per-level compaction statistics were not part of the serialized compaction result, so that primary host lost that information and weren't able to populate `per_key_placement_comp_stats_` and `internal_stats_.proximal_level_stats` properly during the installation. * `TieredCompactionTest` was only checking if (expected stats > 0 && actual stats > 0) instead actual value comparison ## Fixes * Renamed `compaction_stats_` to `internal_stats_` for `InternalStats::CompactionStatsFull` in `CompactionJob` for better readability * Removed the usage of `InternalStats::CompactionOutputsStats` and consolidated them to `InternalStats::CompactionStats`. * Remote Compactions now include the internal stats in the serialized `CompactionServiceResult`. `output_level_stats` and `proximal_level_stats` get later propagated in sub_compact output stats accordingly. * `CompactionJob::UpdateCompactionJobStats()` now takes `CompactionStatsFull` and aggregates the `proximal_level_stats` as well * `TieredCompactionTest` is now doing the actual value comparisons for input/output file counts and record counts. Follow up is needed to do the same for the bytes read / written. Pull Request resolved: https://github.com/facebook/rocksdb/pull/13464 Test Plan: Unit Tests updated to verify stats ``` ./compaction_service_test ``` ``` ./tiered_compaction_test ``` Reviewed By: pdillinger Differential Revision: D71220393 Pulled By: jaykorean fbshipit-source-id: ad70bffd9614ced683f90c7570a17def9b5c8f3f
43 lines
1.3 KiB
C++
43 lines
1.3 KiB
C++
// Copyright (c) Meta Platforms, Inc. and affiliates.
|
|
//
|
|
// This source code is licensed under both the GPLv2 (found in the
|
|
// COPYING file in the root directory) and Apache 2.0 License
|
|
// (found in the LICENSE.Apache file in the root directory).
|
|
//
|
|
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
|
|
// Use of this source code is governed by a BSD-style license that can be
|
|
// found in the LICENSE file. See the AUTHORS file for names of contributors.
|
|
|
|
#pragma once
|
|
|
|
#include "db/compaction/compaction.h"
|
|
#include "db/compaction/subcompaction_state.h"
|
|
#include "db/internal_stats.h"
|
|
|
|
// Data structures used for compaction_job and compaction_service_job which has
|
|
// the list of sub_compact_states and the aggregated information for the
|
|
// compaction.
|
|
namespace ROCKSDB_NAMESPACE {
|
|
|
|
// Maintains state for the entire compaction
|
|
class CompactionState {
|
|
public:
|
|
Compaction* const compaction;
|
|
|
|
// REQUIRED: subcompaction states are stored in order of increasing key-range
|
|
std::vector<SubcompactionState> sub_compact_states;
|
|
Status status;
|
|
|
|
void AggregateCompactionStats(
|
|
InternalStats::CompactionStatsFull& internal_stats,
|
|
CompactionJobStats& job_stats);
|
|
|
|
explicit CompactionState(Compaction* c) : compaction(c) {}
|
|
|
|
Slice SmallestUserKey();
|
|
|
|
Slice LargestUserKey();
|
|
};
|
|
|
|
} // namespace ROCKSDB_NAMESPACE
|