* Ingest sst files rather than their keyvalue content.
* fdbclient/ServerKnobs.cpp
* fdbclient/include/fdbclient/ServerKnobs.h
Add BULK_LOAD_USE_SST_INGEST knob.
* fdbclient/include/fdbclient/IKeyValueStore.actor.h
Add ingestSSTFiles and supportSStIngestion.
* fdbserver/KeyValueStoreRocksDB.actor.cpp
* fdbserver/KeyValueStoreShardedRocksDB.actor.cpp
Implement ingestSSTFiles and supportSStIngestion
* fdbserver/storageserver.actor.cpp
If BULK_LOAD_USE_SST_INGEST and BulkLoadType::SST, ingest sst file
rather than read keyvalues.
* fdbclient/tests/fdb_cluster_fixture.sh
Use rocksdb instead of sqllite in tests.
* Revert "Ingest sst files rather than their keyvalue content."
This reverts commit 6d50bf15d3566de17fa6febf2afba0f220846ca6.
* ctests pass
* Restore ingest minus metric
* Add back ingestDuration metric
* Formatting
---------
Co-authored-by: michael stack <stack@duboce.com>
The original `ThreadReturnPromiseStream` had a fundamental flaw: it didn't handle reference counting
in a thread-safe way. This led to subtle and hard-to-debug race conditions, especially since Future
and Promise often live on different threads in this use case.
This patch introduces a new implementation of ThreadReturnPromiseStream that uses an atomics backed
spinlock to safely manage reference counts. As a result, the Flow compiler also needs to support a
new type: `ThreadFutureStream`, in addition to the existing `FutureStream`.
The underlying `NotifiedQueue` now has a parallel implementation called `ThreadNotifiedQueue` to
make this possible. The higher-level thread-safe promise and promise counterpart are built on top of
that.
The new `ThreadFutureStream` is compatible with both Flow and C++ coroutines. The original
non-threaded Flow primitives remain untouched and free from the overhead of atomics.
Joshua:
20250417-000356-vishesh-16ce18efeab40158 compressed=True data_size=41124943 duration=5436274 ended=99998 fail=1 fail_fast=10 max_runs=100000 pass=99997 priority=100 remaining=0:00:00 runtime=1:00:47 sanity=False started=100000 submitted=20250417-000356 timeout=5400 username=vishesh
clang-format
`CreateColumnFamilyWithImport()` expects that the value inside
handle is `nullptr`. This patch fixed a codepath where we pass
a stale handle left by destroyed column family.
Jingyu Zhou