117 Commits

Author SHA1 Message Date
sfc-gh-tclinkenbeard
156a617ed8 Replace xxhash64 with xxhash3
The goal here is to improve performance
2020-11-21 18:25:35 -08:00
sfc-gh-tclinkenbeard
bcda617b80 Benchmark hashing algorithms in flowbench 2020-11-17 23:14:55 -08:00
sfc-gh-tclinkenbeard
e9bceda8ca Add code coverage macros to PageChecksumCodec::checksum 2020-11-16 19:13:18 -08:00
sfc-gh-tclinkenbeard
287ab51502 Fix some bugs in PageChecksumCodec::checksum 2020-11-15 20:35:05 -08:00
sfc-gh-tclinkenbeard
145cddbb99 Clear the first 8 bits of xxHash64 checksum of SQLite pages
When upgrading a cluster to 7.0 which has stale data from 6.2, we don't
want to calculate an xxHash64 checksum for every page that's read. By
clearing the first 8 bits of all checksums written using xxHash64, the
server can skip this checksum in 255/256 cases when reading pages
written from version <=6.2.
2020-11-15 17:48:44 -08:00
sfc-gh-tclinkenbeard
c5689c4a72 Use xxhash64 for checksumming SQLite pages 2020-11-15 17:48:44 -08:00
sfc-gh-tclinkenbeard
4669f837fa Add uses of makeReference 2020-11-07 22:10:18 -08:00
Xin Dong
410d418c3e Revert "Merge pull request #3770 from dongxinEric/feature/ss-commit-pipelining"
This reverts commit e258dffc65a2cf3e7768f2c9f833f9f553a52d74, reversing
changes made to 0e96233015c25608df620a8c1b81f9945bbd6a33.

SS commit seems to be causing write performance degredation.
2020-10-19 09:28:38 -07:00
sfc-gh-tclinkenbeard
dc4612712d Merge remote-tracking branch 'origin/master' into use-override-more 2020-10-09 07:12:41 -07:00
Steve Atherton
e258dffc65
Merge pull request #3770 from dongxinEric/feature/ss-commit-pipelining
Pipeline the commit process in SS.
2020-10-07 20:34:57 -07:00
sfc-gh-tclinkenbeard
8571dcfe28 Use override where applicable in fdbserver 2020-10-07 18:41:19 -07:00
Xin Dong
34d1d04904 Commit pipelining attempt #2 2020-09-23 20:03:04 -07:00
sfc-gh-tclinkenbeard
0814841827 Replace NULL with nullptr in fdbserver 2020-09-20 11:31:49 -07:00
sfc-gh-tclinkenbeard
77e3b314f6 Mark IKeyValueStore::getStorageBytes const 2020-06-17 18:20:43 -07:00
sfc-gh-tclinkenbeard
2a393633b6 Improved const-correctness
Using the recently added IndexedSet::const_iterator
(https://github.com/apple/foundationdb/pull/3185), we can improve the
const-correctness of many functions. In this PR const is added where
applicable. Also, wherever I came across the following while adding
const, I made the following changes:

- virtual function overrides are marked as override
- NULL is replaced with nullptr
- git clang-format is applied
2020-06-17 18:20:43 -07:00
A.J. Beamon
1c6aef76b5 When one of the sqlite reader or writer thread pools fail, fail the other with the same error. 2020-02-24 12:39:04 -08:00
Evan Tschannen
96258b9809 Merge branch 'release-6.2'
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	fdbcli/fdbcli.actor.cpp
#	fdbclient/ManagementAPI.actor.cpp
#	fdbrpc/FlowTransport.actor.cpp
#	fdbserver/ClusterController.actor.cpp
#	fdbserver/DataDistribution.actor.cpp
#	fdbserver/DataDistribution.actor.h
#	fdbserver/DataDistributionQueue.actor.cpp
#	fdbserver/KeyValueStoreMemory.actor.cpp
#	fdbserver/MasterProxyServer.actor.cpp
#	fdbserver/QuietDatabase.actor.cpp
#	fdbserver/SkipList.cpp
#	fdbserver/StorageMetrics.actor.h
#	fdbserver/TLogServer.actor.cpp
#	fdbserver/fdbserver.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/KVStoreTest.actor.cpp
#	flow/CMakeLists.txt
#	flow/Knobs.cpp
#	flow/Knobs.h
#	flow/genericactors.actor.cpp
#	flow/serialize.h
2020-02-21 19:09:16 -08:00
A.J. Beamon
df2b0452b4 Step 3 of fixing storage server range reads: change return type of readRange from VectorRef<KeyValueRef> to RangeResultRef. 2020-02-06 13:19:24 -08:00
mpilman
d09e07f1f5 Merge remote-tracking branch 'upstream/master' into features/icc 2020-02-04 10:26:18 -08:00
Alex Miller
da73164eda Move crc32c from fdbrpc to flow
So that we can use it from a piece of flow code without breaking module
boundaries.

Also rename generated-constants to crc32c-generated-constants so that
it's more apparent that they're related files.
2020-01-13 18:19:30 -08:00
Andrew Noyes
41583aa576 Guard unlikely indexCode calculation per Steve 2019-12-03 21:48:30 -08:00
Andrew Noyes
36e9f40fc2 Fix negative shift exponent
fdbserver/KeyValueStoreSQLite.actor.cpp:438:11: runtime error: shift exponent -1 is negative
2019-12-02 12:51:33 -08:00
Alex Miller
7a500cd37f A giant translation of TaskFooPriority -> TaskPriority::Foo
This is so that APIs that take priorities don't take ints, which are
common and easy to accidentally pass the wrong thing.
2019-06-25 02:47:35 -07:00
mpilman
844dd60202 FDB compiling with intel compiler 2019-06-20 09:29:01 -07:00
Steve Atherton
ba52623637
Merge pull request #1582 from tclinken/features/sqlite-crc32c
Use crc32 for sqlite page checksums
2019-06-17 14:20:41 -07:00
A.J. Beamon
603721e125 Merge branch 'master' into thread-safe-random-number-generation
# Conflicts:
#	fdbclient/ManagementAPI.actor.cpp
#	fdbrpc/AsyncFileCached.actor.h
#	fdbrpc/genericactors.actor.cpp
#	fdbrpc/sim2.actor.cpp
#	fdbserver/DiskQueue.actor.cpp
#	fdbserver/workloads/BulkSetup.actor.h
#	flow/ActorCollection.actor.cpp
#	flow/Net2.actor.cpp
#	flow/Trace.cpp
#	flow/flow.cpp
2019-05-23 08:35:47 -07:00
Evan Tschannen
f4fbaac6b0 Merge branch 'release-6.1'
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	versions.target
2019-05-19 10:27:59 -07:00
Trevor Clinkenbeard
12ff747e6a Avoid tracing in PageChecksumCodec::checksum if silent flag is set 2019-05-17 10:49:53 -07:00
Trevor Clinkenbeard
3fac380b90 Avoid tracing in PageChecksumCodec::checksum if silent flag is set 2019-05-17 10:43:28 -07:00
Trevor Clinkenbeard
20e93c67ea Allow sqlite pages to be checked for CRC32 checksum
Future versions of FDB will write sqlite pages with CRC32 checksums. In
order to roll back to this version from a version that writes CRC32
checksums, this version must be able to verify those checksums.
2019-05-17 01:05:06 -07:00
Trevor Clinkenbeard
d7bcbe1210 Refactored PageChecksumCodec::checksum 2019-05-16 16:07:35 -07:00
Trevor Clinkenbeard
90d886df95 Trace both hashlittle2 and crc32 checksums for SQLitePageChecksumFailure 2019-05-16 15:51:21 -07:00
Trevor Clinkenbeard
04a72bdad6 Eliminate duplicate code in PageChecksumCodec::checksum 2019-05-16 11:09:37 -07:00
Trevor Clinkenbeard
aca90cd4e2 Don't use memcpy in PageChecksumCodec::checksum 2019-05-16 07:25:58 -07:00
Trevor Clinkenbeard
601c38ad82 Use crc32 for sqlite page checksums 2019-05-14 13:43:55 -07:00
A.J. Beamon
5f55f3f613 Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used. 2019-05-10 14:01:52 -07:00
Evan Tschannen
2d5043c665 Merge branch 'release-6.1'
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	versions.target
2019-04-30 18:27:04 -07:00
Evan Tschannen
9ff8aca1da Increased the SQLITE_CHUNK_SIZE to 100MB (left at 4MB for simulation) 2019-04-26 13:53:56 -07:00
A.J. Beamon
253d2400ef Merge branch 'release-6.1' into speed-up-and-parameterize-spring-cleaning
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
2019-04-23 14:38:52 -07:00
A.J. Beamon
ea7abff9df Clean up from review 2019-04-23 14:16:52 -07:00
A.J. Beamon
4ad0496b39 Increase the frequency that lazy deletes are run. Add more parameters for better control over the spring cleaning process. 2019-04-23 14:01:51 -07:00
Stephen Atherton
83db547306 Implemented the chunk size and db size hint fileControl options in our SQLite VFS implementation. KeyValueStoreSQLite now sets file chunk size based on a new knob, SQLITE_CHUNK_SIZE_PAGES. 2019-04-23 04:50:58 -07:00
Evan Tschannen
6220a5ce0f
Merge pull request #1370 from jzhou77/fix-unreferenced
Remove unused functions
2019-04-09 11:49:45 -07:00
mpilman
1c16f87a4e Remove trace-calls to printable (in non-workloads) 2019-04-05 13:12:19 -07:00
mpilman
bb82f8560a process all volatile ints correctly in traces 2019-04-05 13:12:19 -07:00
mpilman
c008e16c81 Defer formatting in traces to make them cheaper
This is the first part of making `TraceEvent` cheaper. The main idea is
to defer calls to any code that formats string. These are the main
changes:

- TraceEvent::detail now takes a c-string instead of std::string for
  literals. This prevents unnecessary allocations if the trace is not
  going to be printed in the first place (for example for SevDebug).
  Before that `detail` expected a `std::string` as key, which mean that
  any string literal would be copied on each call.
- Templates Traceable and SpecialTraceMetricType. These templates can be
  specialized for any type that needs to be printed. The actual
  formatting will be deferred to after the `enabled` check. This
  provides two benefits: (1) if a TraceEvent is disabled, we don't pay
  for the formatting and (2) TraceEvent can trace types that it doesn't
  know about.
- TraceEvent::enabled will be set in the constructor if the Severity is
  passed. This will make sure that `TraceEvent::init` is not called.
- `TraceEvent::detail` will be inlined. So for disabled TraceEvent
  calls, a call to detail will only introduce a if-branch which is much
  cheaper than a function call.
2019-04-05 13:12:19 -07:00
Jingyu Zhou
f7f8ddd894 Fix warnings on unused variables
Found by -Wunused-variable flag.
2019-04-01 14:00:20 -07:00
Andrew Noyes
067a445e06 Replace unused _ variables with wait(success(...)) 2019-02-12 17:30:30 -08:00
Robert Escriva
268093a96d Adjust all includes to be relative to the root.
Remove the use of relative paths.  A header at foo/bar.h could be included by
files under foo/ with "bar.h", but would be included everywhere else as
"foo/bar.h".  Adjust so that every include references such a header with the
latter form.

Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-19 17:35:33 +00:00
Evan Tschannen
3922e477a5 Merge branch 'release-6.0'
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	fdbclient/ManagementAPI.actor.cpp
#	fdbserver/ClusterController.actor.cpp
#	fdbserver/DataDistribution.actor.cpp
#	fdbserver/LogSystemDiskQueueAdapter.actor.cpp
#	fdbserver/SimulatedCluster.actor.cpp
#	fdbserver/TLogServer.actor.cpp
2018-10-03 16:57:18 -07:00