8309c2ce26
filter mutations in change feeds to the requested range
2021-09-03 12:06:13 -07:00
afd8df4bb9
Merge commit '7f2b6c2c7ec2efcd7339e3c6531e4906322477c3' into network-thread-busyness-burstiness-score
2021-09-03 09:48:11 -07:00
4bbae59bb0
Fixed a few bugs and added a backup popping mechanism
2021-09-02 21:11:44 -07:00
27151b0831
support reads from range feeds that span multiple storage servers
2021-09-01 14:35:51 -07:00
46adada5ff
Cleaned up debugging and fixed a couple bugs
2021-08-31 12:30:43 -05:00
2fa97f60fe
Update fdbclient/NativeAPI.actor.cpp
...
Co-authored-by: Trevor Clinkenbeard <trevor.clinkenbeard@snowflake.com>
2021-08-31 08:53:37 -07:00
2c788c233d
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-08-27 17:07:47 -07:00
2d6fb93316
Compute a burstiness score based on the max busyness interval in a measurement period.
2021-08-27 16:29:15 -07:00
8d49c98a41
Added simulation workload for blob granules and fixed some bugs
2021-08-26 13:48:05 -05:00
714aca4f3c
Refactored blob granule reads to actually be in NativeAPI
2021-08-25 15:01:38 -05:00
b9628bdb50
Merge branch 'master' of github.com:apple/foundationdb into add-flow-db-interface
2021-08-17 20:24:41 +00:00
53e83cf653
Remove change to Database
2021-08-17 20:10:43 +00:00
179c313e08
Use DatabaseContext instead of Database in template functions
2021-08-17 20:02:25 +00:00
925c4a935d
Log size of locationCache in client
...
We observed that multi-threaded client has more memory usage,
and we would like to understand it better.
One suspect is that this shard-to-SS cache is playing a role,
logging the size of it to get better vision.
2021-08-16 10:31:14 -07:00
a278d2977a
renamed range feeds to change feeds, data distribution support almost complete
2021-08-13 14:27:15 -07:00
e444d3781c
Various TSS improvements from snowblower testing
2021-08-13 10:24:15 -05:00
28f3ffd2ec
Fix a crash in debug mode
...
For unknown reasons, The previous implementation crashed while
(according to valgrind and ubsan) trying to interpret uninitialized memory as a
shared_ptr. This only happened when CMAKE_BUILD_TYPE=Debug. This change
works for debug mode and also still addresses this TSAN issue the
original change addressed.
2021-08-12 16:19:20 -07:00
a605fb3852
Merge pull request #5026 from sfc-gh-ljoswiak/fixes/alp6
...
Actor sampling
2021-08-11 13:44:17 -07:00
94239098a5
added additional rangefeed commands to fdbcli
2021-08-10 23:59:05 -07:00
7891477a7c
added support for streaming range feeds
2021-08-10 21:28:25 -07:00
618c72f7e2
Draft to use template to rewrite ThrottleApi
2021-08-11 01:07:36 +00:00
e5e8a56b66
Merge branch 'master' of https://github.com/apple/foundationdb into debugging-cleanup
...
# Conflicts:
# fdbbackup/FileDecoder.actor.cpp
2021-08-09 23:36:57 -07:00
ed28aecde0
Merge branch 'master' into feature-range-feed
2021-08-09 20:40:55 -07:00
42ae870c84
added support for querying specific range feed versions
2021-08-09 20:39:28 -07:00
ecc078a42f
Fix TSAN warnings
...
Avoid doing thread-unsafe things in the ReadYourWritesTransaction
default constructor since we now will be calling it in a foreign thread.
Here's the warning we're fixing:
Write of size 8 at 0x7b6800000398 by thread T1:
#0 N2::Net2::updateNow() /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/flow/Net2.actor.cpp:192:33 (libfdb_c.so+0xe74568)
#1 N2::Net2::run() /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/flow/Net2.actor.cpp:1459:3 (libfdb_c.so+0xe74568)
#2 runNetwork() /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/fdbclient/NativeAPI.actor.cpp:2036:13 (libfdb_c.so+0x575004)
#3 ThreadSafeApi::runNetwork() /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../fdbclient/ThreadSafeTransaction.cpp:445:3 (libfdb_c.so+0xcda9cc)
#4 MultiVersionApi::runNetwork() /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/fdbclient/MultiVersionTransaction.actor.cpp:1741:20 (libfdb_c.so+0x4eab8d)
#5 fdb_run_network /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../bindings/c/fdb_c.cpp:130:2 (libfdb_c.so+0x4bb425)
#6 decltype(std::__1::forward<int (*)()>(fp)()) std::__1::__invoke<int (*)()>(int (*&&)()) /usr/local/bin/../include/c++/v1/type_traits:3899:1 (fdb_c_unit_tests+0x389816)
#7 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, int (*)()>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, int (*)()>&, std::__1::__tuple_indices<>) /usr/local/bin/../include/c++/v1/thread:280:5 (fdb_c_unit_tests+0x389816)
#8 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, int (*)()> >(void*) /usr/local/bin/../include/c++/v1/thread:291:5 (fdb_c_unit_tests+0x389816)
Previous read of size 8 at 0x7b6800000398 by main thread:
#0 N2::Net2::now() const /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/flow/Net2.actor.cpp:160:39 (libfdb_c.so+0xe7ea5d)
#1 Span::Span(UID, Location, std::initializer_list<UID> const&) /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../flow/Tracing.h:38:41 (libfdb_c.so+0x5837b4)
#2 Transaction::Transaction() /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/fdbclient/NativeAPI.actor.cpp:4092:64 (libfdb_c.so+0x5837b4)
#3 ReadYourWritesTransaction::ReadYourWritesTransaction() /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../fdbclient/ReadYourWrites.h:134:2 (libfdb_c.so+0x7cb4a8)
#4 ISingleThreadTransaction::allocateOnForeignThread(ISingleThreadTransaction::Type) /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../fdbclient/ISingleThreadTransaction.cpp:29:17 (libfdb_c.so+0xcf785f)
#5 ThreadSafeTransaction::ThreadSafeTransaction(DatabaseContext*, ISingleThreadTransaction::Type) /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../fdbclient/ThreadSafeTransaction.cpp:149:23 (libfdb_c.so+0xcc9540)
#6 ThreadSafeDatabase::createTransaction() /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../fdbclient/ThreadSafeTransaction.cpp:50:37 (libfdb_c.so+0xcc4b1c)
#7 fdb_database_create_transaction /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../bindings/c/fdb_c.cpp:332:2 (libfdb_c.so+0x4bce1d)
#8 fdb::Transaction::Transaction(FDB_database*) /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../bindings/c/test/unit/fdb_api.cpp:111:24 (fdb_c_unit_tests+0x3925d0)
#9 _DOCTEST_ANON_FUNC_10() /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../bindings/c/test/unit/unit_tests.cpp:250:19 (fdb_c_unit_tests+0x32a7a8)
#10 doctest::Context::run() /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/doctest/src/doctest/doctest/doctest.h:6112:21 (fdb_c_unit_tests+0x3262e4)
#11 main /home/jenkins/fdb/extra/long/path/to/work/around/strange/cpack/debug/rpm/behavior/_build/../bindings/c/test/unit/unit_tests.cpp:2286:20 (fdb_c_unit_tests+0x36303d)
2021-08-09 11:16:52 -07:00
bc9a0e1315
first attempt to add data distribution support for range feeds
2021-08-09 10:05:56 -07:00
2bf7b9b7a9
Remove many calls to .toString(), .printable(), and printable(...) from TraceEvent .detail() values to avoid double-escaping characters such that the same values look different in different events and can't be easily searched for. Also fixed a few warnings found by clang.
2021-08-08 00:04:54 -07:00
39eff8c569
Add trace_partial_file_suffix network option
2021-08-03 09:43:42 -07:00
5dc9a97230
Merge branch 'master' into fixes/alp6
2021-08-01 20:42:52 -07:00
0989c28a6b
made range feeds durable on the storage server
2021-07-30 15:23:42 -07:00
e9409b02fe
fixed the build
2021-07-28 14:02:12 -07:00
c74047c665
Merge remote-tracking branch 'origin/master' into fix-more-clang-warnings
2021-07-28 11:51:02 -07:00
9f571255bd
Merge pull request #5271 from sfc-gh-ajbeamon/do-not-reset-committed-transaction
...
Do not partially reset a transaction when it is committed or fails to commit with an error
2021-07-28 09:02:54 -07:00
406562b282
Merge branch 'master' of https://github.com/apple/foundationdb into feature-range-feed
2021-07-27 17:04:58 -07:00
3eed4084e2
Merge branch 'master' into fixes/alp6
2021-07-27 11:26:53 -07:00
59d535149e
Merge branch 'master' into fixes/alp6
2021-07-27 10:07:18 -07:00
e9a1679467
Disable sampling everywhere except fdbserver
2021-07-27 09:53:23 -07:00
507c1f11e3
Add .log() to bare TraceEvent() invocations without any .detail()s to avoid clang-tidy warning about immediate destruction of object without use.
2021-07-26 19:55:10 -07:00
34f82e7a15
Do not partially reset a transaction when it is committed or fails to commit with an error.
2021-07-26 09:51:44 -07:00
64dc1dc185
Fix -Wreorder-ctor warnings in NativeAPI.actor.cpp and several other files
2021-07-24 00:23:06 -07:00
6f81155784
Merge remote-tracking branch 'origin/master' into const-serverdbinfo
2021-07-20 10:18:40 -07:00
f596a81073
Rename ::TRUE and ::FALSE in BooleanParams to ::True and ::False so as to not conflict with the TRUE and FALSE macros provided by the Windows and MacOS SDKs.
2021-07-17 00:11:40 -07:00
4f853b19a6
More const-correctness improvements for Reference<AsyncVar<T>> objects
2021-07-12 21:28:38 -07:00
15fe00927a
Merge branch 'master' of https://github.com/apple/foundationdb into feature-range-feed
...
# Conflicts:
# fdbcli/fdbcli.actor.cpp
# fdbclient/NativeAPI.actor.cpp
# fdbclient/StorageServerInterface.h
# fdbclient/SystemData.cpp
# fdbclient/SystemData.h
# fdbserver/storageserver.actor.cpp
# flow/ProtocolVersion.h
2021-07-12 19:10:14 -07:00
4059215180
Value should not be present for ENABLE_RUN_LOOP_PROFILING
...
This fixes an unintentional change from
79ff07a071
2021-07-12 16:07:56 -07:00
77cd95473d
Merge remote-tracking branch 'origin/master' into const-serverdbinfo
2021-07-12 11:00:22 -07:00
02f243c821
Merge pull request #5110 from sfc-gh-jslocum/tss_streaming_master
...
Added Streaming Read comparison for TSS
2021-07-12 10:44:15 -07:00
6c1d913ab8
Prevent masterServer from modifying db
2021-07-11 21:11:21 -07:00
ccffd6f47f
Fix option validation for DISABLE_CLIENT_STATISTICS_LOGGING
2021-07-04 19:06:39 -07:00
8cc40e3a2b
Expand use of BOOLEAN_PARAM
2021-07-02 21:41:50 -07:00
79ff07a071
Added *BOOLEAN_PARAM macros to enforce documentation of boolean parameters
2021-07-02 15:04:42 -07:00
9f1afbb709
Added Streaming Read comparison for TSS
2021-07-01 19:00:05 -05:00
ab0ab765c8
Merge pull request #4965 from sfc-gh-jslocum/tss_mapping_changes
...
TSS Mismatch Changes
2021-06-29 08:24:25 -07:00
73752f441b
exclude locality:clang-format, ranged loops, documentation, tracking addStoragesever for exclusion.
2021-06-23 18:03:27 -07:00
62355571d0
exclude servers based on locality match
2021-06-23 18:03:27 -07:00
244e8f3527
Merge branch 'master' of https://github.com/apple/foundationdb into feature-streaming-reply-master
...
# Conflicts:
# fdbclient/CMakeLists.txt
# fdbclient/Knobs.h
# fdbserver/Knobs.h
2021-06-21 16:35:21 -07:00
1cec223434
cherry pick streaming replies to master
2021-06-19 09:47:13 -07:00
2d248a5926
Collect actors waiting on disk and network
2021-06-17 16:50:34 -07:00
826e70916c
Merge remote-tracking branch 'origin/master' into config-db
2021-06-17 09:47:41 -07:00
805c4200ce
Add collector to get running actor name
2021-06-16 18:08:51 -07:00
fe936207a9
Replace lower priority txn request when limit is hit.
2021-06-15 14:00:06 -07:00
0ff041222a
Add removals back
2021-06-13 18:23:59 -07:00
112be7a763
Remove most work
2021-06-13 16:19:02 -07:00
56dadaa428
TSS Mismatch Changes
2021-06-11 23:13:16 +00:00
41c790b299
Merge remote-tracking branch 'origin/master' into config-db
2021-06-10 22:31:23 -07:00
c272304e60
Manage global flow knobs with global knob collection
2021-06-09 22:33:00 -07:00
c6d96c4124
Added TSS Quarantine and fixed TSS mapping bug in CommitProxy
2021-06-09 14:42:50 +00:00
838d847d4e
Merge pull request #4860 from sfc-gh-xwang/ppwtest
...
implement perpetual storage wiggling feature
2021-06-04 16:18:39 -07:00
486a04659f
Lazy inititialization
2021-06-04 15:01:18 -07:00
9b36f69b8d
Merge pull request #4892 from sfc-gh-jslocum/tss_mappingv2
...
TSS Mapping Change
2021-06-04 14:57:51 -07:00
153de33f57
Revert "Merge pull request #4802 from sfc-gh-ljoswiak/revert/actor-lineage"
...
This reverts commit 6499fa178e1512631957
2021-06-04 13:31:55 -07:00
ba25b95c6a
Fix global config not updating on server processes
2021-06-03 14:27:24 -07:00
365b0bc4d8
better api
2021-06-03 19:57:44 +00:00
89dadf1475
Create IKnobCollection interface
2021-06-02 23:40:52 -07:00
351325b3af
comment modification; wait perpetual wiggling close
2021-06-03 05:13:20 +00:00
b3e4f182ef
TSS Mapping Change
2021-06-02 17:30:09 +00:00
4c059670a0
Parse configuration knob values on the client
2021-06-02 10:04:46 -07:00
a775f92fca
Merge remote-tracking branch 'origin/master' into config-db
2021-06-01 15:39:34 -07:00
8b9c8b33fc
manually merge with master
2021-06-01 17:51:42 +00:00
d67184163b
Merge pull request #4556 from sfc-gh-jslocum/tss
...
Testing Storage Server
2021-06-01 09:11:10 -07:00
750901dd1d
Reduce the frequency that buggified reads are failed so that transactions with a lot of reads aren't doomed to almost always fail.
2021-05-27 21:54:59 -07:00
24c0c3361a
change quietdatabase
2021-05-25 21:20:24 +00:00
4257ac2b4d
More TSS Changes/Fixes
2021-05-25 20:37:48 +00:00
ce82c9653e
Testing Storage Server implementation
2021-05-25 20:28:50 +00:00
6165af81fa
Merge branch 'apple:master' into feature-range-feed
2021-05-21 15:21:34 -07:00
bb61a20851
Improve const-correctness of ISingleThreadTransaction
2021-05-15 00:23:21 -07:00
4ea760b2a9
Revert "Merge pull request #4136 from sfc-gh-mpilman/features/actor-lineage"
...
This reverts commit da41534618e6300905d6
2021-05-10 20:26:12 -07:00
331dd2e377
Replace config_transaction option with use_config_database
2021-05-10 13:18:31 -07:00
cdf98f987d
Revert "Fix global config not triggering changes on server processes"
2021-05-10 12:18:28 -07:00
71ba6f4501
Switch std::function to AsyncTrigger
2021-05-07 16:05:03 -07:00
0fc1552683
Fix global config updates not triggering on fdbserver processes
2021-05-06 15:46:34 -07:00
5c2d7b6080
Create RangeResult type alias
2021-05-03 13:14:16 -07:00
8dcd779fc4
Merge branch 'master' into features/actor-lineage
2021-05-02 14:11:42 -07:00
2621c7153a
clang format PR
2021-04-30 10:41:35 -07:00
d652287bae
Merge branch 'master' of https://github.com/apple/foundationdb into feature-range-feed
...
# Conflicts:
# fdbclient/DatabaseContext.h
# fdbclient/NativeAPI.actor.cpp
# fdbclient/StorageServerInterface.h
# fdbserver/storageserver.actor.cpp
2021-04-30 10:29:42 -07:00
f513543305
fix special keyspace register
2021-04-28 16:35:09 -06:00
904a39e473
Merge pull request #4667 from sfc-gh-ajbeamon/feature-mvc-monitor-protocol-version
...
Use fewer connections in the multi-version client
2021-04-28 14:13:17 -07:00
f5b0008346
Merge branch 'features/actor-lineage' into features/actor-lineage-fluentd
2021-04-28 09:31:20 -06:00
54919d4f3b
Merge remote-tracking branch 'sfc/features/actor-lineage' into features/actor-lineage
2021-04-28 09:22:14 -06:00
4fab2ecd30
Merge remote-tracking branch 'origin/master' into features/actor-lineage
2021-04-28 09:20:54 -06:00
Evan Tschannen