apple/foundationdb
1
0
Fork 0
mirror of https://github.com/apple/foundationdb.git synced 2025-05-16 02:42:23 +08:00
Code Issues Packages Projects Releases Wiki Activity
21,551 Commits 35 Branches 318 Tags
Commit Graph

145 Commits

Author SHA1 Message Date
Steve Atherton
4982787a80 Another attempt to fix SQLite lifetime issues. KeyValueStoreSQLite::doClose() can now access the Writer instance and call disableRateControl() on its SQLiteDB instance prior to stopping the reader and writer thread pools. This should prevent rate limiting from interfering with SQLite shutdown. 2021-01-30 18:18:36 -08:00
Steve Atherton
b0fd6bfbcf More bug fixes in SQLite file handle lifetimes after storage server restart (including simulated process reboot) when a write rate limit is used. Added several debug trace events, renamed some VFSAsync* events to be more consistent. 2021-01-30 18:18:16 -08:00
Steve Atherton
52c2695f59 Bug fixes involving use of AFCached rate control in simulation. 2021-01-30 18:13:44 -08:00
Steve Atherton
e56fe02a25 Merge branch 'add-throttling-on-AsyncfileCached' of github.com:sfc-gh-clin/foundationdb into afcache-write-limit 2021-01-30 18:11:54 -08:00
Steve Atherton
14af5857ed Another refactor of SQLite injected error handling to address failures of the previous attempt. Some code cleanup. Added documentation of injected error handling works and why. 2021-01-30 18:06:05 -08:00
Steve Atherton
e72d486215 Refactored how injected fault state is managed to make debugging injected fault handling easier. Moved injected error tracking for open() into an INetwork global since it should be unique per simulated process. Fixed bug where injected error in WAL file was not recognized as injected because it occurred before SQLiteDB's vfsWAL pointer was initialized. Simplified logic in SQLiteDB. Added comments. 2021-01-30 18:06:04 -08:00
Steve Atherton
1a5c40fcef Bug fixes: Check for null pager and wal pointers, and initialize vfsWAL after SQLite has opened the WAL. 2021-01-30 18:06:04 -08:00
Steve Atherton
e1f8023577 Rewrote how injected faults are handled in SQLite to be more reliable and work with an upcoming write throttling feature in AsyncFileCached. 2021-01-30 18:06:04 -08:00
Chaoguang Lin
4c4d1ec6b0 Use VFSAsyncFile::checkInjectedError() to replace old injected fault detection, see pr#4212 2021-01-29 15:28:08 -08:00
Steve Atherton
14e39a8747 Another bug fix in SQLite file instance lifetimes. Writer can be destroyed before doClose() or stopOnError() have a chance to react, so Writer can't be used to get access to file handles to remove the rate limiter and trigger pending requests to complete. New logic is for Writer initialization to place opened file handles in KeyValueStoreSQLite instance and move disableRateControl() there, to be called from doClose(). 2021-01-28 00:59:05 -08:00
Steve Atherton
e3a6531477 Another attempt to fix SQLite lifetime issues. KeyValueStoreSQLite::doClose() can now access the Writer instance and call disableRateControl() on its SQLiteDB instance prior to stopping the reader and writer thread pools. This should prevent rate limiting from interfering with SQLite shutdown. 2021-01-27 05:31:02 -08:00
Steve Atherton
989fee908a More bug fixes in SQLite file handle lifetimes after storage server restart (including simulated process reboot) when a write rate limit is used. Added several debug trace events, renamed some VFSAsync* events to be more consistent. 2021-01-27 00:01:33 -08:00
Andrew Noyes
564c70acb4
Merge branch 'master' into anoyes/modernize-use-override 2021-01-25 18:26:18 -08:00
Andrew Noyes
4ee97c0784 Use clang-tidy to automatically fix missing overrides 
Use `clang-tidy -p . $file -checks='-*,modernize-use-override' -header-filter='.*' -fix`
to fix missing overrides, and then use git clang-format to reformat just
those changes. This went pretty well for most files.

Formatting the following files went off the rails, so I'm going to
follow up with a commit that's just clang-tidy and no clang-format.

- fdbclient/DatabaseBackupAgent.actor.cpp
- fdbclient/FileBackupAgent.actor.cpp
- fdbserver/OldTLogServer_4_6.actor.cpp
- fdbmonitor/SimpleIni.h
- fdbserver/workloads/ClientTransactionProfileCorrectness.actor.cpp
 2021-01-26 02:04:12 +00:00
Andrew Noyes
adde95bea5 Merge branch 'release-6.3' into anoyes/merge-6.3-to-master 
Commit merge conflicts for now
 2021-01-25 18:55:24 +00:00
Steve Atherton
0d32bd0c44 Bug fixes involving use of AFCached rate control in simulation. 2021-01-24 01:50:09 -08:00
Steve Atherton
523a8c08db Merge branch 'add-throttling-on-AsyncfileCached' of github.com:sfc-gh-clin/foundationdb into afcache-write-limit 
# Conflicts:
#	flow/Knobs.h
 2021-01-17 05:30:55 -08:00
Steve Atherton
11927f6fff Another refactor of SQLite injected error handling to address failures of the previous attempt. Some code cleanup. Added documentation of injected error handling works and why. 2021-01-17 04:17:13 -08:00
Steve Atherton
f1adafaf8c Refactored how injected fault state is managed to make debugging injected fault handling easier. Moved injected error tracking for open() into an INetwork global since it should be unique per simulated process. Fixed bug where injected error in WAL file was not recognized as injected because it occurred before SQLiteDB's vfsWAL pointer was initialized. Simplified logic in SQLiteDB. Added comments. 2021-01-16 05:04:30 -08:00
Steve Atherton
8df6874417 Bug fixes: Check for null pager and wal pointers, and initialize vfsWAL after SQLite has opened the WAL. 2021-01-15 21:13:31 -08:00
Steve Atherton
6da78b1b19 Rewrote how injected faults are handled in SQLite to be more reliable and work with an upcoming write throttling feature in AsyncFileCached. 2021-01-15 19:29:14 -08:00
Chaoguang Lin
0613a149a6 Add rate control on both db file and log file 2020-12-11 00:22:26 -08:00
Chaoguang Lin
9156d38216 Add knob SQLITE_READER_THREADS to control the number of read threads 2020-12-09 12:09:07 -08:00
Chaoguang Lin
5b55216252 Add rate control to AsyncFileCached 2020-12-09 12:03:32 -08:00
Andrew Noyes
7fbc4d7391 Resolve conflicts 2020-12-04 23:58:42 +00:00
Andrew Noyes
877997632d Merge branch 'release-6.3' into anoyes/merge-release-6.3-master 
Include conflict markers for review purposes
 2020-12-04 01:38:07 +00:00
sfc-gh-tclinkenbeard
aa07df6a91 Backport to 6.3 the ability to read xxhash3 checksum for sqlite pages 2020-11-24 00:28:49 -08:00
Trevor Clinkenbeard
4029217c2f
Update fdbserver/KeyValueStoreSQLite.actor.cpp 
Co-authored-by: Andrew Noyes <63815641+sfc-gh-anoyes@users.noreply.github.com>
 2020-11-23 10:07:08 -08:00
sfc-gh-tclinkenbeard
156a617ed8 Replace xxhash64 with xxhash3 
The goal here is to improve performance
 2020-11-21 18:25:35 -08:00
sfc-gh-tclinkenbeard
bcda617b80 Benchmark hashing algorithms in flowbench 2020-11-17 23:14:55 -08:00
sfc-gh-tclinkenbeard
e9bceda8ca Add code coverage macros to PageChecksumCodec::checksum 2020-11-16 19:13:18 -08:00
sfc-gh-tclinkenbeard
287ab51502 Fix some bugs in PageChecksumCodec::checksum 2020-11-15 20:35:05 -08:00
sfc-gh-tclinkenbeard
145cddbb99 Clear the first 8 bits of xxHash64 checksum of SQLite pages 
When upgrading a cluster to 7.0 which has stale data from 6.2, we don't
want to calculate an xxHash64 checksum for every page that's read. By
clearing the first 8 bits of all checksums written using xxHash64, the
server can skip this checksum in 255/256 cases when reading pages
written from version <=6.2.
 2020-11-15 17:48:44 -08:00
sfc-gh-tclinkenbeard
c5689c4a72 Use xxhash64 for checksumming SQLite pages 2020-11-15 17:48:44 -08:00
sfc-gh-tclinkenbeard
4669f837fa Add uses of makeReference 2020-11-07 22:10:18 -08:00
Xin Dong
410d418c3e Revert "Merge pull request #3770 from dongxinEric/feature/ss-commit-pipelining" 
This reverts commit e258dffc65a2cf3e7768f2c9f833f9f553a52d74, reversing
changes made to 0e96233015c25608df620a8c1b81f9945bbd6a33.

SS commit seems to be causing write performance degredation.
 2020-10-19 09:28:38 -07:00
sfc-gh-tclinkenbeard
dc4612712d Merge remote-tracking branch 'origin/master' into use-override-more 2020-10-09 07:12:41 -07:00
Steve Atherton
e258dffc65
Merge pull request #3770 from dongxinEric/feature/ss-commit-pipelining 
Pipeline the commit process in SS.
 2020-10-07 20:34:57 -07:00
sfc-gh-tclinkenbeard
8571dcfe28 Use override where applicable in fdbserver 2020-10-07 18:41:19 -07:00
Xin Dong
34d1d04904 Commit pipelining attempt #2 2020-09-23 20:03:04 -07:00
sfc-gh-tclinkenbeard
0814841827 Replace NULL with nullptr in fdbserver 2020-09-20 11:31:49 -07:00
sfc-gh-tclinkenbeard
77e3b314f6 Mark IKeyValueStore::getStorageBytes const 2020-06-17 18:20:43 -07:00
sfc-gh-tclinkenbeard
2a393633b6 Improved const-correctness 
Using the recently added IndexedSet::const_iterator
(https://github.com/apple/foundationdb/pull/3185), we can improve the
const-correctness of many functions. In this PR const is added where
applicable. Also, wherever I came across the following while adding
const, I made the following changes:

- virtual function overrides are marked as override
- NULL is replaced with nullptr
- git clang-format is applied
 2020-06-17 18:20:43 -07:00
A.J. Beamon
1c6aef76b5 When one of the sqlite reader or writer thread pools fail, fail the other with the same error. 2020-02-24 12:39:04 -08:00
Evan Tschannen
96258b9809 Merge branch 'release-6.2' 
# Conflicts:
#	documentation/sphinx/source/release-notes.rst
#	fdbcli/fdbcli.actor.cpp
#	fdbclient/ManagementAPI.actor.cpp
#	fdbrpc/FlowTransport.actor.cpp
#	fdbserver/ClusterController.actor.cpp
#	fdbserver/DataDistribution.actor.cpp
#	fdbserver/DataDistribution.actor.h
#	fdbserver/DataDistributionQueue.actor.cpp
#	fdbserver/KeyValueStoreMemory.actor.cpp
#	fdbserver/MasterProxyServer.actor.cpp
#	fdbserver/QuietDatabase.actor.cpp
#	fdbserver/SkipList.cpp
#	fdbserver/StorageMetrics.actor.h
#	fdbserver/TLogServer.actor.cpp
#	fdbserver/fdbserver.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/KVStoreTest.actor.cpp
#	flow/CMakeLists.txt
#	flow/Knobs.cpp
#	flow/Knobs.h
#	flow/genericactors.actor.cpp
#	flow/serialize.h
 2020-02-21 19:09:16 -08:00
A.J. Beamon
df2b0452b4 Step 3 of fixing storage server range reads: change return type of readRange from VectorRef<KeyValueRef> to RangeResultRef. 2020-02-06 13:19:24 -08:00
mpilman
d09e07f1f5 Merge remote-tracking branch 'upstream/master' into features/icc 2020-02-04 10:26:18 -08:00
Alex Miller
da73164eda Move crc32c from fdbrpc to flow 
So that we can use it from a piece of flow code without breaking module
boundaries.

Also rename generated-constants to crc32c-generated-constants so that
it's more apparent that they're related files.
 2020-01-13 18:19:30 -08:00
Andrew Noyes
41583aa576 Guard unlikely indexCode calculation per Steve 2019-12-03 21:48:30 -08:00
Andrew Noyes
36e9f40fc2 Fix negative shift exponent 
fdbserver/KeyValueStoreSQLite.actor.cpp:438:11: runtime error: shift exponent -1 is negative
 2019-12-02 12:51:33 -08:00