Default to nb instead of nc for sstable formats

patch by Berenguer Blasi; reviewed by Francisco Guerrero, Jacek Lewandowski, Michael Semb Wever for CASSANDRA-19010
This commit is contained in:
Bereng 2023-11-07 07:24:57 +01:00
parent 928037206e
commit 0e4c2f4bef
14 changed files with 52 additions and 49 deletions

View File

@ -4,7 +4,7 @@ CASSANDRA-14092: MAXIMUM TTL EXPIRATION DATE
The maximum expiration timestamp that can be represented by the storage engine has been raised to
2106-02-07T06:28:13+00:00 from the previous 2038-01-19T03:14:06+00:00, which means that INSERTS using
TTL that would expire after this date are not currently supported. Notice that on previous versions <5.0
or when 5.0 is ran in compatibility mode (<="nc" sstable formats) the limit stays at at 2038-01-19T03:14:06+00:00.
or when 5.0 is ran in compatibility mode (<="oa" sstable formats) the limit stays at at 2038-01-19T03:14:06+00:00.
There is a new yaml property storage_compatibility_mode that determines the
Cassandra major version we want to stay compatible with. Its default is CASSANDRA_4, which means that

View File

@ -4,7 +4,7 @@ PLEASE READ: MAXIMUM TTL EXPIRATION DATE NOTICE (CASSANDRA-14092 & CASSANDRA-142
The maximum expiration timestamp that can be represented by the storage engine has been
raised to 2106-02-07T06:28:13+00:00 (2038-01-19T03:14:06+00:00 if in compatibility mode
with Cassandra <5.0, sstable version <=nc), which means that inserts with TTL that expire after
with Cassandra <5.0, sstable version <=oa), which means that inserts with TTL that expire after
this date are not currently supported. By default, INSERTS with TTL exceeding the
maximum supported date are rejected, but it's possible to choose a different
expiration overflow policy. See CASSANDRA-14092.txt for more details.
@ -165,7 +165,7 @@ New features
- On virtual tables, it is not strictly necessary to specify `ALLOW FILTERING` for select statements which would
normally require it, except `system_views.system_logs`.
- More accurate skipping of sstables in read path due to better handling of min/max clustering and lower bound;
SSTable format has been bumped to 'nc' because there are new fields in stats metadata\
SSTable format has been bumped to 'oa' because there are new fields in stats metadata\
- Added MaxSSTableSize and MaxSSTableDuration metrics to TableMetrics. The former returns the size of the biggest
SSTable of a table or 0 when there is not any SSTable. The latter returns the maximum duration, computed as
`maxTimestamp - minTimestamp`, effectively non-zero for SSTables produced by TimeWindowCompactionStrategy.

View File

@ -2089,7 +2089,7 @@ drop_compact_storage_enabled: false
# The chosen storage compatiblity mode will determine the versions of the written sstables, commitlogs, hints,
# etc. Those storage elements will use the higher minor versions of the major version that corresponds to the
# Cassandra version we want to stay compatible with. For example, if we want to stay compatible with Cassandra 4.0
# or 4.1, the value of this property should be 4, and that will make us use 'nc' sstables.
# or 4.1, the value of this property should be 4, and that will make us use 'oa' sstables.
#
# This will also determine if certain features depending on newer formats are available. For example, extended TTLs
# up to 2106 depend on the sstable, commitlog, hints and messaging versions that were introduced by Cassandra 5.0,

View File

@ -193,9 +193,9 @@
# Defines how to handle INSERT requests with TTL exceeding the maximum supported expiration date:
# * REJECT: this is the default policy and will reject any requests with expiration date timestamp
# after 2106-02-07T06:28:13+00:00 or 2038-01-19T03:14:06+00:00 depending on compatibility mode (<="nc" sstable formats)
# after 2106-02-07T06:28:13+00:00 or 2038-01-19T03:14:06+00:00 depending on compatibility mode (<="oa" sstable formats)
# * CAP: any insert with TTL expiring after 2106-02-07T06:28:13+00:00 or 2038-01-19T03:14:06+00:00,
# depending on compatibility mode (<="nc" sstable formats), will expire on 2106-02-07T06:28:13+00:00
# depending on compatibility mode (<="oa" sstable formats), will expire on 2106-02-07T06:28:13+00:00
# or 2038-01-19T03:14:06+00:00 and the client will receive a warning.
# * CAP_NOWARN: same as previous, except that the client warning will not be emitted.
#

View File

@ -271,7 +271,7 @@ It is possible to specify `--scan` option to get additional statistics about a t
Example
....
sstablemetadata -s /var/lib/cassandra/data/data/ks/tb-dbead8501c9511ee805bb3deafc4499b/nc-1-big-Data.db
sstablemetadata -s /var/lib/cassandra/data/data/ks/tb-dbead8501c9511ee805bb3deafc4499b/oa-1-big-Data.db
....
This gives you statistics such as:

View File

@ -46,9 +46,9 @@ Use the path to the SSTable file as the only argument.
Example:
....
sstablepartitions data/data/k/t-d7be5e90e90111ed8b54efe3c39cb0bb/nc-8-big-Data.db
sstablepartitions data/data/k/t-d7be5e90e90111ed8b54efe3c39cb0bb/oa-8-big-Data.db
Processing k.t-d7be5e90e90111ed8b54efe3c39cb0bb #8 (big-nc) (1.368 GiB uncompressed, 534.979 MiB on disk)
Processing k.t-d7be5e90e90111ed8b54efe3c39cb0bb #8 (big-oa) (1.368 GiB uncompressed, 534.979 MiB on disk)
Partition size Row count Cell count Tombstone count
~p50 767.519 KiB 770 1916 0
~p75 2.238 MiB 2299 5722 0
@ -70,7 +70,7 @@ Example:
....
sstablepartitions data/data/k/t-d7be5e90e90111ed8b54efe3c39cb0bb
Processing k.t-d7be5e90e90111ed8b54efe3c39cb0bb #8 (big-nc) (1.368 GiB uncompressed, 534.979 MiB on disk)
Processing k.t-d7be5e90e90111ed8b54efe3c39cb0bb #8 (big-oa) (1.368 GiB uncompressed, 534.979 MiB on disk)
Partition size Row count Cell count Tombstone count
~p50 767.519 KiB 770 1916 0
~p75 2.238 MiB 2299 5722 0
@ -82,7 +82,7 @@ Processing k.t-d7be5e90e90111ed8b54efe3c39cb0bb #8 (big-nc) (1.368 GiB uncompres
max 356.067 MiB 310706 932118 2450
count 210
Processing k.t-d7be5e90e90111ed8b54efe3c39cb0bb #9 (big-nc) (457.540 MiB uncompressed, 174.880 MiB on disk)
Processing k.t-d7be5e90e90111ed8b54efe3c39cb0bb #9 (big-oa) (457.540 MiB uncompressed, 174.880 MiB on disk)
Partition size Row count Cell count Tombstone count
~p50 1.865 MiB 1597 4768 0
~p75 13.858 MiB 14237 42510 0
@ -102,15 +102,15 @@ Use the `--min-size` option to specify the minimum size a partition must have to
Example:
....
sstablepartitions data/data/k/t-d7be5e90e90111ed8b54efe3c39cb0bb/nc-8-big-Data.db --min-size 100MiB
sstablepartitions data/data/k/t-d7be5e90e90111ed8b54efe3c39cb0bb/oa-8-big-Data.db --min-size 100MiB
Processing k.t-d7be5e90e90111ed8b54efe3c39cb0bb #8 (big-nc) (1.368 GiB uncompressed, 534.979 MiB on disk)
Processing k.t-d7be5e90e90111ed8b54efe3c39cb0bb #8 (big-oa) (1.368 GiB uncompressed, 534.979 MiB on disk)
Partition: '13' (0000000d) live, size: 105.056 MiB, rows: 91490, cells: 274470, tombstones: 50 (row:50, range:0, complex:0, cell:0, row-TTLd:0, cell-TTLd:0)
Partition: '1' (00000001) live, size: 127.241 MiB, rows: 111065, cells: 333195, tombstones: 50 (row:50, range:0, complex:0, cell:0, row-TTLd:0, cell-TTLd:0)
Partition: '8' (00000008) live, size: 356.067 MiB, rows: 310706, cells: 932118, tombstones: 0 (row:0, range:0, complex:0, cell:0, row-TTLd:0, cell-TTLd:0)
Partition: '2' (00000002) live, size: 213.341 MiB, rows: 186582, cells: 559125, tombstones: 978 (row:978, range:0, complex:0, cell:0, row-TTLd:0, cell-TTLd:0)
Summary of k.t-d7be5e90e90111ed8b54efe3c39cb0bb #8 (big-nc):
File: /Users/adelapena/src/cassandra/trunk/data/data/k/t-d7be5e90e90111ed8b54efe3c39cb0bb/nc-8-big-Data.db
Summary of k.t-d7be5e90e90111ed8b54efe3c39cb0bb #8 (big-oa):
File: /Users/adelapena/src/cassandra/trunk/data/data/k/t-d7be5e90e90111ed8b54efe3c39cb0bb/oa-8-big-Data.db
4 partitions match
Keys: 13 1 8 2
Partition size Row count Cell count Tombstone count
@ -132,15 +132,15 @@ Use the `--min-tombstones` option to specify the minimum number of tombstones a
Example:
....
sstablepartitions data/data/k/t-d7be5e90e90111ed8b54efe3c39cb0bb/nc-8-big-Data.db --min-tombstones 1000
sstablepartitions data/data/k/t-d7be5e90e90111ed8b54efe3c39cb0bb/oa-8-big-Data.db --min-tombstones 1000
Processing k.t-d7be5e90e90111ed8b54efe3c39cb0bb #8 (big-nc) (1.368 GiB uncompressed, 534.979 MiB on disk)
Processing k.t-d7be5e90e90111ed8b54efe3c39cb0bb #8 (big-oa) (1.368 GiB uncompressed, 534.979 MiB on disk)
Partition: '55' (00000037) live, size: 1.290 MiB, rows: 2317, cells: 3474, tombstones: 1159 (row:1159, range:0, complex:0, cell:0, row-TTLd:0, cell-TTLd:0)
Partition: '28' (0000001c) live, size: 1.198 MiB, rows: 2099, cells: 3147, tombstones: 1050 (row:1050, range:0, complex:0, cell:0, row-TTLd:0, cell-TTLd:0)
Partition: '89' (00000059) live, size: 1.346 MiB, rows: 2226, cells: 3339, tombstones: 1113 (row:1113, range:0, complex:0, cell:0, row-TTLd:0, cell-TTLd:0)
Partition: '21' (00000015) live, size: 3.853 MiB, rows: 4900, cells: 9927, tombstones: 2450 (row:2450, range:0, complex:0, cell:0, row-TTLd:0, cell-TTLd:0)
Summary of k.t-d7be5e90e90111ed8b54efe3c39cb0bb #8 (big-nc):
File: /Users/adelapena/src/cassandra/trunk/data/data/k/t-d7be5e90e90111ed8b54efe3c39cb0bb/nc-8-big-Data.db
Summary of k.t-d7be5e90e90111ed8b54efe3c39cb0bb #8 (big-oa):
File: /Users/adelapena/src/cassandra/trunk/data/data/k/t-d7be5e90e90111ed8b54efe3c39cb0bb/oa-8-big-Data.db
4 partitions match
Keys: 55 28 89 21
Partition size Row count Cell count Tombstone count
@ -162,10 +162,10 @@ Use the `--csv` option to output a CSV machine-readable output, combined with an
Example:
....
sstablepartitions data/data/k/t-d7be5e90e90111ed8b54efe3c39cb0bb/nc-8-big-Data.db --min-size 100MiB --csv
sstablepartitions data/data/k/t-d7be5e90e90111ed8b54efe3c39cb0bb/oa-8-big-Data.db --min-size 100MiB --csv
key,keyBinary,live,offset,size,rowCount,cellCount,tombstoneCount,rowTombstoneCount,rangeTombstoneCount,complexTombstoneCount,cellTombstoneCount,rowTtlExpired,cellTtlExpired,directory,keyspace,table,index,snapshot,backup,generation,format,version
"13",0000000d,true,186403543,110158965,91490,274470,50,50,0,0,0,0,0,/Users/adelapena/src/cassandra/trunk/data/data/k/t-d7be5e90e90111ed8b54efe3c39cb0bb/nc-8-big-Data.db,k,t,,,,8,big,nc
"1",00000001,true,325141542,133422183,111065,333195,50,50,0,0,0,0,0,/Users/adelapena/src/cassandra/trunk/data/data/k/t-d7be5e90e90111ed8b54efe3c39cb0bb/nc-8-big-Data.db,k,t,,,,8,big,nc
"8",00000008,true,477133752,373362819,310706,932118,0,0,0,0,0,0,0,/Users/adelapena/src/cassandra/trunk/data/data/k/t-d7be5e90e90111ed8b54efe3c39cb0bb/nc-8-big-Data.db,k,t,,,,8,big,nc
"2",00000002,true,851841363,223704192,186582,559125,978,978,0,0,0,0,0,/Users/adelapena/src/cassandra/trunk/data/data/k/t-d7be5e90e90111ed8b54efe3c39cb0bb/nc-8-big-Data.db,k,t,,,,8,big,nc
"13",0000000d,true,186403543,110158965,91490,274470,50,50,0,0,0,0,0,/Users/adelapena/src/cassandra/trunk/data/data/k/t-d7be5e90e90111ed8b54efe3c39cb0bb/oa-8-big-Data.db,k,t,,,,8,big,oa
"1",00000001,true,325141542,133422183,111065,333195,50,50,0,0,0,0,0,/Users/adelapena/src/cassandra/trunk/data/data/k/t-d7be5e90e90111ed8b54efe3c39cb0bb/oa-8-big-Data.db,k,t,,,,8,big,oa
"8",00000008,true,477133752,373362819,310706,932118,0,0,0,0,0,0,0,/Users/adelapena/src/cassandra/trunk/data/data/k/t-d7be5e90e90111ed8b54efe3c39cb0bb/oa-8-big-Data.db,k,t,,,,8,big,oa
"2",00000002,true,851841363,223704192,186582,559125,978,978,0,0,0,0,0,/Users/adelapena/src/cassandra/trunk/data/data/k/t-d7be5e90e90111ed8b54efe3c39cb0bb/oa-8-big-Data.db,k,t,,,,8,big,oa
....

View File

@ -458,7 +458,7 @@ public abstract class SSTableReader extends SSTable implements UnfilteredSource,
this.openReason = builder.getOpenReason();
this.first = builder.getFirst();
this.last = builder.getLast();
this.bounds = AbstractBounds.strictlyWrapsAround(first.getToken(), last.getToken())
this.bounds = first == null || last == null || AbstractBounds.strictlyWrapsAround(first.getToken(), last.getToken())
? null // this will cause the validation to fail, but the reader is opened with no validation,
// e.g. for scrubbing, we should accept screwed bounds
: AbstractBounds.bounds(first.getToken(), true, last.getToken(), true);

View File

@ -74,14 +74,14 @@ public abstract class Version
public abstract boolean hasOldBfFormat();
/**
* @deprecated it is replaced by {@link #hasImprovedMinMax()} since 'nc' and to be completetly removed since 'oa'
* @deprecated it is replaced by {@link #hasImprovedMinMax()} since 'oa' and to be completetly removed after 'oa'
*/
/** @deprecated See CASSANDRA-18134 */
@Deprecated(since = "5.0")
public abstract boolean hasAccurateMinMax();
/**
* @deprecated it is replaced by {@link #hasImprovedMinMax()} since 'nc' and to be completetly removed since 'oa'
* @deprecated it is replaced by {@link #hasImprovedMinMax()} since 'oa' and to be completetly removed after 'oa'
*/
/** @deprecated See CASSANDRA-18134 */
@Deprecated(since = "5.0")

View File

@ -337,7 +337,7 @@ public class BigFormat extends AbstractSSTableFormat<BigTableReader, BigTableWri
static class BigVersion extends Version
{
public static final String current_version = DatabaseDescriptor.getStorageCompatibilityMode().isBefore(5) ? "nc" : "oa";
public static final String current_version = DatabaseDescriptor.getStorageCompatibilityMode().isBefore(5) ? "nb" : "oa";
public static final String earliest_supported_version = "ma";
// ma (3.0.0): swap bf hash order
@ -349,8 +349,8 @@ public class BigFormat extends AbstractSSTableFormat<BigTableReader, BigTableWri
// na (4.0-rc1): uncompressed chunks, pending repair session, isTransient, checksummed sstable metadata file, new Bloomfilter format
// nb (4.0.0): originating host id
// nc (5.0): improved min/max, partition level deletion presence marker, key range (CASSANDRA-18134)
// oa (5.0): Long deletionTime to prevent TTL overflow
// oa (5.0): improved min/max, partition level deletion presence marker, key range (CASSANDRA-18134)
// Long deletionTime to prevent TTL overflow
// token space coverage
//
// NOTE: When adding a new version:
@ -389,8 +389,8 @@ public class BigFormat extends AbstractSSTableFormat<BigTableReader, BigTableWri
hasCommitLogLowerBound = version.compareTo("mb") >= 0;
hasCommitLogIntervals = version.compareTo("mc") >= 0;
hasAccurateMinMax = version.matches("(m[d-z])|(n[a-z])"); // deprecated in 'nc' and to be removed in 'oa'
hasLegacyMinMax = version.matches("(m[a-z])|(n[a-z])"); // deprecated in 'nc' and to be removed in 'oa'
hasAccurateMinMax = version.matches("(m[d-z])|(n[a-z])"); // deprecated in 'oa' and to be removed after 'oa'
hasLegacyMinMax = version.matches("(m[a-z])|(n[a-z])"); // deprecated in 'oa' and to be removed after 'oa'
// When adding a new version you might need to add it here
hasOriginatingHostId = version.compareTo("nb") >= 0 || version.matches("(m[e-z])");
hasMaxCompressedLength = version.compareTo("na") >= 0;
@ -398,9 +398,9 @@ public class BigFormat extends AbstractSSTableFormat<BigTableReader, BigTableWri
hasIsTransient = version.compareTo("na") >= 0;
hasMetadataChecksum = version.compareTo("na") >= 0;
hasOldBfFormat = version.compareTo("na") < 0;
hasImprovedMinMax = version.compareTo("nc") >= 0;
hasPartitionLevelDeletionPresenceMarker = version.compareTo("nc") >= 0;
hasKeyRange = version.compareTo("nc") >= 0;
hasImprovedMinMax = version.compareTo("oa") >= 0;
hasPartitionLevelDeletionPresenceMarker = version.compareTo("oa") >= 0;
hasKeyRange = version.compareTo("oa") >= 0;
hasUintDeletionTime = version.compareTo("oa") >= 0;
hasTokenSpaceCoverage = version.compareTo("oa") >= 0;
}

View File

@ -33,10 +33,10 @@ import static org.apache.cassandra.utils.StorageCompatibilityMode.NONE;
import static org.apache.cassandra.utils.StorageCompatibilityMode.UPGRADING;
/**
* Tests TTL the overflow policy triggers at the correct limit: year 2038 <=nc or 2186 >nc
* Tests TTL the overflow policy triggers at the correct limit: year 2038 <=nb or 2186 >=oa
* <p>
* <=nc overflow policy triggers at year 2038. That could be <=4.1 or 5.0 with 4.x storage compatibility
* >nc overflow policy triggers at year 2106. That is >=5.0 using >=5.x storage compatibility
* <=oa overflow policy triggers at year 2038. That could be <=4.1 or 5.0 with 4.x storage compatibility
* >oa overflow policy triggers at year 2106. That is >=5.0 using >=5.x storage compatibility
*
* @see StorageCompatibilityMode
*/
@ -94,13 +94,13 @@ public class MixedModeTTLOverflowUpgradeTest extends UpgradeTestBase
assertPolicyTriggersAt2038(cluster.coordinator(1));
assertPolicyTriggersAt2038(cluster.coordinator(2));
// We restart one node on 5.0 >nc hence 2038 should still be the limit as the other node is 5.0 <=nc
// We're on compatibility mode where nc and oa nodes are a possibility
// We restart one node on 5.0 >oa hence 2038 should still be the limit as the other node is 5.0 <=oa
// We're on compatibility mode where oa and oa nodes are a possibility
restartNodeWithCompatibilityMode(cluster, 1, UPGRADING);
assertPolicyTriggersAt2038(cluster.coordinator(1));
assertPolicyTriggersAt2038(cluster.coordinator(2));
// We restart the other node so they're all on 5.0 >nc hence 2106 should be the limit
// We restart the other node so they're all on 5.0 >oa hence 2106 should be the limit
restartNodeWithCompatibilityMode(cluster, 2, UPGRADING);
assertPolicyTriggersAt2106(cluster.coordinator(1));
assertPolicyTriggersAt2106(cluster.coordinator(2));

View File

@ -67,9 +67,9 @@ public class DeletionTimeDeSerBench
final static ArrayList<DeletionTime> TEST_DTS_30PC_LIVE = generateDTs(30);
// Parameters
final static String ncSstableParam = "NC";
final static String nbSstableParam = "NB";
final static String oaSstableParam = "OA";
@Param({ ncSstableParam, oaSstableParam })
@Param({ nbSstableParam, oaSstableParam })
String sstableParam;
final static String live70PcParam = "70PcLive";
@ -341,7 +341,7 @@ public class DeletionTimeDeSerBench
{
switch (value)
{
case ncSstableParam:
case nbSstableParam:
return new DeletionTime.LegacySerializer();
case oaSstableParam:

View File

@ -101,7 +101,7 @@ public class LegacySSTableTest
// Get all versions up to the current one. Useful for testing in compatibility mode C18301
private static String[] getValidLegacyVersions()
{
String[] versions = {"oa", "da", "nc", "nb", "na", "me", "md", "mc", "mb", "ma"};
String[] versions = {"oa", "da", "nb", "na", "me", "md", "mc", "mb", "ma"};
return Arrays.stream(versions).filter((v) -> v.compareTo(BigFormat.getInstance().getLatestVersion().toString()) <= 0).toArray(String[]::new);
}

View File

@ -40,7 +40,7 @@ public class VersionSupportedFeaturesTest extends AbstractTestVersionSupportedFe
@Override
protected Stream<String> getPartitionLevelDeletionPresenceMarkerSupportedVersions()
{
return range("nc", "zz");
return range("oa", "zz");
}
@Override
@ -52,13 +52,13 @@ public class VersionSupportedFeaturesTest extends AbstractTestVersionSupportedFe
@Override
protected Stream<String> getImprovedMinMaxSupportedVersions()
{
return range("nc", "zz");
return range("oa", "zz");
}
@Override
protected Stream<String> getKeyRangeSupportedVersions()
{
return range("nc", "zz");
return range("oa", "zz");
}
@Override

View File

@ -76,6 +76,7 @@ public class MetadataSerializerTest
Map<MetadataType, MetadataComponent> originalMetadata = constructMetadata(false);
MetadataSerializer serializer = new MetadataSerializer();
Version latestVersion = DatabaseDescriptor.getSelectedSSTableFormat().getLatestVersion();
File statsFile = serialize(originalMetadata, serializer, DatabaseDescriptor.getSelectedSSTableFormat().getLatestVersion());
Descriptor desc = new Descriptor(statsFile.parent(), "", "", new SequenceBasedSSTableId(0), DatabaseDescriptor.getSelectedSSTableFormat());
@ -85,7 +86,9 @@ public class MetadataSerializerTest
for (MetadataType type : MetadataType.values())
{
assertEquals(originalMetadata.get(type), deserialized.get(type));
if ((type != MetadataType.STATS) || latestVersion.hasImprovedMinMax())
assertEquals(originalMetadata.get(type), deserialized.get(type));
}
}
}