natas_hw
/
cassandra
mirror of https://github.com/apache/cassandra
0
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Storage engine refactor, a.k.a CASSANDRA-8099 

Initial patch, see ticket for details
This commit is contained in:
Sylvain Lebresne 2014-09-01 18:54:46 +02:00
parent 5c31a86334
commit a991b64811
645 changed files with 49618 additions and 42464 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

56
bin/sstablekeys
View File

@ -1,56 +0,0 @@
#!/bin/sh
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
if [ "x$CASSANDRA_INCLUDE" = "x" ]; then
for include in "`dirname "$0"`/cassandra.in.sh" \
"$HOME/.cassandra.in.sh" \
/usr/share/cassandra/cassandra.in.sh \
/usr/local/share/cassandra/cassandra.in.sh \
/opt/cassandra/cassandra.in.sh; do
if [ -r "$include" ]; then
. "$include"
break
fi
done
elif [ -r "$CASSANDRA_INCLUDE" ]; then
. "$CASSANDRA_INCLUDE"
fi
# Use JAVA_HOME if set, otherwise look for java in PATH
if [ -x "$JAVA_HOME/bin/java" ]; then
JAVA="$JAVA_HOME/bin/java"
else
JAVA="`which java`"
fi
if [ -z "$CLASSPATH" ]; then
echo "You must set the CLASSPATH var" >&2
exit 1
fi
if [ $# -eq "0" ]; then
echo "Usage: `basename "$0"` <sstable>"
exit 2
fi
"$JAVA" $JAVA_AGENT -cp "$CLASSPATH" $JVM_OPTS -Dstorage-config="$CASSANDRA_CONF" \
-Dcassandra.storagedir="$cassandra_storagedir" \
-Dlogback.configurationFile=logback-tools.xml \
org.apache.cassandra.tools.SSTableExport "$@" -e
# vi:ai sw=4 ts=4 tw=0 et

41
bin/sstablekeys.bat
View File

@ -1,41 +0,0 @@
@REM
@REM Licensed to the Apache Software Foundation (ASF) under one or more
@REM contributor license agreements. See the NOTICE file distributed with
@REM this work for additional information regarding copyright ownership.
@REM The ASF licenses this file to You under the Apache License, Version 2.0
@REM (the "License"); you may not use this file except in compliance with
@REM the License. You may obtain a copy of the License at
@REM
@REM http://www.apache.org/licenses/LICENSE-2.0
@REM
@REM Unless required by applicable law or agreed to in writing, software
@REM distributed under the License is distributed on an "AS IS" BASIS,
@REM WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
@REM See the License for the specific language governing permissions and
@REM limitations under the License.
@echo off
if "%OS%" == "Windows_NT" setlocal
pushd "%~dp0"
call cassandra.in.bat
if NOT DEFINED CASSANDRA_MAIN set CASSANDRA_MAIN=org.apache.cassandra.tools.SSTableExport
if NOT DEFINED JAVA_HOME goto :err
REM ***** JAVA options *****
set JAVA_OPTS=^
-Dlogback.configurationFile=logback-tools.xml
set TOOLS_PARAMS=
"%JAVA_HOME%\bin\java" %JAVA_OPTS% %CASSANDRA_PARAMS% -cp %CASSANDRA_CLASSPATH% "%CASSANDRA_MAIN%" %1 -e
goto finally
:err
echo JAVA_HOME environment variable must be set!
pause
:finally
ENDLOCAL

2
build.xml
View File

@ -94,7 +94,7 @@
<property name="maven-repository-url" value="https://repository.apache.org/content/repositories/snapshots"/>
<property name="maven-repository-id" value="apache.snapshots.https"/>
<property name="test.timeout" value="60000" />
<property name="test.timeout" value="1200000" />
<property name="test.long.timeout" value="600000" />
<property name="test.burn.timeout" value="600000" />

376
guide_8099.md Normal file
View File

@ -0,0 +1,376 @@
Overview of CASSANDRA-8099 changes
==================================
The goal of this document is to provide an overview of the main changes done in
CASSANDRA-8099 so it's easier to dive into the patch. This assumes knowledge of
the pre-existing code.
CASSANDRA-8099 refactors the abstractions used by the storage engine and as
such impact most of the code of said engine. The changes can be though of as
following two main guidelines:
1. the new abstractions are much more iterator-based (i.e. it tries harder to
avoid materializing everything in memory),
2. and they are closer to the CQL representation of things (i.e. the storage
engine is aware of more structure, making it able to optimize accordingly).
Note that while those changes have heavy impact on the actual code, the basic
mechanisms of the read and write paths are largely unchanged.
In the following, I'll start by describe the new abstractions introduced by the
patch. I'll then provide a quick reference of existing class to what it becomes
in the patch, after which I'll discuss how the refactor handles a number of
more specific points. Lastly, the patch introduces some change to the on-wire
and on-disk format so I'll discuss those quickly.
Main new abstractions
---------------------
### Atom: Row and RangeTombstoneMarker
Where the existing storage engine is mainly handling cells, the new engine
groups cells into rows, and rows becomes the more central building block. A
`Row` is identified by the value of it's clustering columns which are stored in
a `Clustering` object (see below), and it associate a number of cells to each
of its non-PK non-static columns (we'll discuss static columns more
specifically later).
The patch distinguishes 2 kind of columns: simple and complex ones. The
_simple_ columns can have only 1 cell associated to them (or none), while the
_complex_ ones will have an arbitrary number of cells associated. Currently,
the complex columns are only the non frozen collections (but we'll have
non-frozen udt at some point and who knows what in the future).
Like before, we also have to deal with range tombstones. However, instead of
dealing with full range tombstones, we generally deal with
`RangeTombstoneMarker` which is just one of the bound of the range tombstone
(so that a range tombstone is composed of 2 "marker" in practice, its start and
its end). I'll discuss the reasoning for this a bit more later. A
`RangeTombstoneMarker` is identified by a `Slice.Bound` (which is to RT markers
what the `Clustering` is to `Row`) and simply store its deletion information.
The engine thus mainly work with rows and range tombstone markers, and they are
both grouped under the common `Atom` interface. An "unfiltered" is thus just that:
either a row or a range tombstone marker.
> Side Note: the "Atom" naming is pretty bad. I've reused it mainly because it
> plays a similar role to the existing OnDiskAtom, but it's arguably crappy now
> because a row is definitively not "indivisible". Anyway, renaming suggestions
> are more than welcome. The only alternative I've come up so far are "Block"
> or "Element" but I'm not entirely convinced by either.
### ClusteringPrefix, Clustering, Slice.Bound and ClusteringComparator
Atoms are sorted (within a partition). They are ordered by their
`ClusteringPrefix`, which is mainly a common interface for the `Clustering` of
`Row`, and the `Slice.Bound` of `RangeTombstoneMarker`. More generally, a
`ClusteringPrefix` is a prefix of the clustering values for the clustering
columns of the table involved, with a `Clustering` being the special case where
all values are provided. A `Slice.Bound` can be a true prefix however, having
only some of the clustering values. Further, a `Slice.Bound` can be either a
start or end bound, and it can be inclusive or exclusive. Sorting make sure that
a start bound is before anything it "selects" (and conversely for the end). A
`Slice` is then just 2 `Slice.Bound`: a start and a end, and selects anything
that sorts between those.
`ClusteringPrefix` are compared through the table `ClusteringComparator`, which
is like our existing table comparator except that it only include comparators
for the clustering column values. In particular, it includes neither the
comparator for the column names themselves, nor the post-column-name comparator
for collections (the latter being handled through the `CellPath`, see below).
There is a also a `Clusterable` interface that `Atom` implements and that
simply marks object that can be compared by a `ClusteringComparator`, i.e.
objects that have a `ClusteringPrefix`.
### Cell
A cell holds the informations on a single value. It corresponds to a (CQL)
column, has a value, a timestamp and optional ttl and local deletion time.
Further, as said above, complex columns (collections) will have multiple
associated cells. Those cells are distinguished by their `CellPath`, which are
compared through a comparator that depends on the column type. The cells of
simple columns just have a `null` cell path.
### AtomIterator and PartitionIterator
As often as possible, atoms are manipulated through iterators. And this through
`AtomIterator`, which is an iterator over the atoms of a single partition, and
`PartitionIterator`, which is an iterator of `AtomIterator`, i.e. an iterator
over multiple partitions. In other words a single partition query fundamentally
returns an `AtomIterator`, while a range query returns a `PartitionIterator`.
Those iterators are closeable and the code has to be make sure to always close
them as they will often have resources to clean (like an OpOrder.Group to close
or files to release).
The read path mainly consists in getting unfiltered and partition iterators from
sstables and memtable and merging, filtering and transforming them. There is a
number of functions to do just that (merging, filtering and transforming) in
`AtomIterators` and `PartitionIterators`, but there is also a number of classes
(`RowFilteringAtomIterator`, `CountingAtomIterator`, ...) that wraps one of
those iterator type to apply some filtering/transformation.
`AtomIterator` and `PartitionIterator` also have their doppelgänger
`RowIterator` and `DataIterator` which exists for the sake of making it easier
for the upper layers (StorageProxy and above) to deal with deletion. We'll
discuss those later.
### Partition: PartitionUpdate, AtomicBTreePartition and CachedPartition
While we avoid materializing partitions in memory as much as possible (favoring
the iterative approach), there is cases where we need/want to hold some subpart
of a partition in memory and we have a generic `Partition` interface for those.
A `Partition` basically corresponds to materializing an `AtomIterator` and
is thus somewhat equivalent to the existing
`ColumnFamily` (but again, many existing usage of `ColumnFamily` simply use
`AtomIterator`). `Partition` is mainly used through the following
implementations:
* `PartitionUpdate`: this is what a `Mutation` holds and is used to gather
update and apply them to memtables.
* `AtomicBTreePartition`: this is the direct counterpart of AtomicBTreeColumns.
The difference being that the BTree holds rows instead of cells. On updates, we
merge the rows together to create a new one.
* `CachedPartition`: this is used by the row cache.
### Read commands
The `ReadCommand` class still exists, but instead of being just for single
partition reads, it's now a common abstract class for both single partition and
range reads. It then has 2 subclass: `SinglePartitionReadCommand` and
`PartitionRangeReadCommand`, the former of which has 2 subclasses itself:
`SinglePartitionSliceCommand` and `SinglePartitionNamesCommand`. All `ReadCommand`,
have a `ColumnFilter` and a `DataLimits` (see below). `PartitionRangeReadCommand`
additionally has a `DataRange`, which is mostly just a range of partition key
with a `PartitionFilter`, while `SinglePartitionReadCommand` has a partition
key and a `PartitionFilter` (see below too).
The code to execute those queries locally, which used to be in `ColumnFamilyStore`
and `CollationController` is now in those `ReadCommand` classes. For instance, the
`CollationController` code for names queries is in `SinglePartitionNamesCommand`,
and the code to decide if we use a 2ndary index or not is directly in
`ReadCommand.executeLocally()`.
Note that because they share a common class, all `ReadCommand` actually
return a `PartitionIterator` (an iterator over partitions), even single partition
ones (that iterator will just return one or zero result). It actually allows to
generalize (and simplify) the "response resolver". Instead of having separate resolver
for range and single partition queries, we only have `DataResolver` and
`DigestResolver` that work for any read command. This does mean that the patch
fixes CASSANDRA-2986, and that we could use digest queries for range if we
wanted to (not necessarily saying it's a good idea).
### ColumnFilter
`ColumnFilter` is the new `List<IndexExpression>`. It holds those column restrictions that
can't be directly fulfilled by the `PartitionFilter`, i.e. those that require either a
2ndary index, or filtering.
### PartitionFilter
`PartitionFilter` is the new `IDiskAtomFilter`/`QueryFilter`. There is still 2 variants:
`SlicePartitionFilter` and `NamesPartitionFilter`. Both variant includes the actual columns
that are queried (as we don't return full CQL rows anymore), and both can be
reversed. A names filter queries a bunch of rows by names, i.e. has a set of
`Clustering`. A slice filter queries one or more slice of rows. A slice filter
does not however include a limit since that is dealt with by `DataLimits` which
is in `ReadCommand` directly.
### DataLimits
`DataLimits` implement the limits on a query. This is meant to abstract the differences between
how we count for thrift and for CQL. Further, for CQL, this allow to have a limit per partition,
which clean up how DISTINCT queries are handled and allow for CASSANDRA-7017 (the patch doesn't
add support for the PER PARTITION LIMIT syntax of that ticket, but handle it
internally otherwise).
### SliceableAtomIterator
The code also use the `SliceableAtomIterator` abstraction. A
`SliceableAtomIterator` is an `AtomIterator` for which we basically know how to seek
into efficiently. In particular, we have a `SSTableIterator` which is a
`SliceableAtomIterator`. That `SSTableIterator` replaces both the existing
`SSTableNamesIterator` and `SSTableSliceIterator`, and the respective
`PartitionFilter` uses that `SliceableAtomIterator` interface to query what
they want exactly.
What did that become again?
---------------------------
For quick reference, here's the rough correspondence of old classes to new classes:
* `ColumnFamily`: for writes, this is handled by `PartitionUpdate` and
`AtomicBTreePartition`. For reads, this is replaced by `AtomIterator` (or
`RowIterator`, see the parts on tombstones below). For the row cache, there is
a specific `CachedPartition` (which is actually an interface, the implementing
class being `ArrayBackedPartition`)
* `Cell`: there is still a `Cell` class, which is roughly the same thing than
the old one, except that instead of having a cell name, cells are now in a
row and correspond to a column.
* `QueryFilter`: doesn't really exists anymore. What it was holding is now in
`SinglePartitionReadCommand`.
* `AbstractRangeCommand` is now `PartitionRangeReadCommand`.
* `IDiskAtomFilter` is now `PartitionFilter`.
* `List<IndexExpression>` is now `ColumnFilter`.
* `RowDataResolver`, `RowDigestResolver` and `RangeSliceResponseResolver` are
now `DataResolver` and `DigestResolver`.
* `Row` is now `AtomIterator`.
* `List<Row>` is now `PartitionIterator` (or `DataIterator`, see the part about
tombstones below).
* `AbstractCompactedRow` and`LazilyCompactedRow` are not really needed anymore.
Their corresponding code is in `CompactionIterable`.
Noteworthy points
-----------------
### Dealing with tombstones and shadowed cells
There is a few aspects worth noting regarding the handling of deleted and
shadowed data:
1. it's part of the contract of an `AtomIterator` that it must not shadow it's
own data. In other words, it should not return a cell that is deleted by one
of its own range tombstone, or by its partition level deletion. In practice
this means that we get rid of shadowed data quickly (a good thing) and that
there is a limited amount of places that have to deal with shadowing
(merging being one).
2. Upper layer of the code (anything above StorageProxy) don't care about
deleted data (and deletion informations in general). Basically, as soon as
we've merge results for the replica, we don't need tombstones anymore.
So, instead of requiring all those upper layer to filter tombstones
themselves (which is error prone), we get rid of them as soon as we can,
i.e. as soon as we've resolved replica responses (so in the
`ResponseResolver`). To do that and to make it clear when tombstones have
been filtered (and make the code cleaner), we transform an `AtomIterator`
into a `RowIterator`. Both being essentially the same thing, except that a
`RowIterator` only return live stuffs. Which mean in particular that it's an
iterator of `Row` (since an unfiltered is either a row or a range tombstone and
we've filtered tombstones). Similarly, a `PartitionIterator` becomes a
`DataIterator`, which is just an iterator of `RowIterator`.
3. In the existing code a CQL row deletion involves a range tombstone. But as
row deletions are pretty frequent and range tombstone have inherent
inefficiencies, the patch adds the notion of row deletion, which is just
some optional deletion information on the `Row`. This can be though as just
an optimization of range tombstones that span only a single row.
4. As mentioned at the beginning of this document, the code splits range
tombstones into 2 range tombstone marker, one for each bound. The problem
with storing full range tombstone as we currently do is that it makes it
harder to merge them efficiently, and in particular to "normalize"
overlapping ones. In practice, a given `AtomIterator` guarantees that there
is only one range tombstone to worry about at any given time. In other
words, at any point of the iterator, either there is a single open range
tombstone or there is none, which makes things easier and more efficient.
It's worth noting that we still have the `RangeTombstone` class because for
in-memory structures (`PartitionUpdate`, `AtomicBTreePartition`, ...) we
currently use the existing `RangeTombstoneList` out of convenience. This is
kind of an implementation detail that could change in the future.
### statics
Static columns are handled separately from the rest of the rows. In practice,
each `Partition`/`AtomIterator` has a separate "static row" that holds the
value for all the static columns (that row can of course be empty). That row
doesn't really correspond to any CQL row, it's content is simply "merged" to
the result set in `SelectStatement` (very much like we already do).
### Row liveness
In CQL, a row can exists even if only its PK columns have values. In other
words, a `Row` can be live even if it doesn't have any cells. Currently, we
handle this through the "row marker" (i.e. through a specific cell). The patch
makes this slightly less hacky by adding a timestamp (and potentially a ttl) to
each row, which can be though as the timestamp (and ttl) of the PK columns.
Further, when we query only some specific columns in a row, we need to add a
row (with nulls) in the result set if a row is live (it has live cells or the
timestamp we described above) even if it has no actual values for the queried
columns. We currently deal with that by querying entire row all the time, but
the patch change that. This does mean that even when we query only some
columns, we still need to have the information on whether the row itself is
live or not. And because we'll merge results from different sources (which can
include deletions), a boolean is not enough, so we include in a `Row` object
the maximum live timestamp known for this row. Which currently mean that in
practice, we do scan the full row on disk but filter cells we're not interested
by right away (and in fact, we don't even deserialize the value of such cells).
This might be optimizable later on, but expiring data makes that harder (we
typically can't just pre-compute that max live timestamp when writing the
sstable since it can depend on the time of the query).
### Flyweight pattern
The patch makes relatively heavy use of a "flyweight"-like pattern. Typically,
`Row` and `Cell` data are stored in arrays, and a given `AtomIterator` will
only use a single `Row` object (and `Cell` object) that points in those arrays
(and thus change at each call to `hasNext()/next()`). This does mean that the
objects returned that way shouldn't be aliased (taken a reference of) without
care. The patch uses an `Aliasable` interface to mark object that may use this
pattern and should thus potentially be copied in the rare cases where a
reference should be kept on them.
### Reversed queries
The patch slightly change the way reversed queries are handled. First, while
a reverse query should currently reverse its slices in `SliceQueryFilter`, this
is not the case anymore: `SlicePartitionFilter` always keep its slices in
clustering order and simply handles those from the end to the beginning on
reversed queries. Further, if a query is reversed, the results are returned in
this reverse order all the way through. Which differs from the current code
where `ColumnFamily` actually always holds cells in forward order (making it a
mess for paging and forcing us to re-reverse everything in the result set).
### Compaction
As the storage engine works iteratively, compaction simply has to get iterators
on the sstables, merge them and write the result back, along with a simple
filter that skip purgeable tombstones in the process. So there isn't really a
need for `AbstractCompactedRow`/`LazilyCompactedRow` anymore and the
compaction/purging code is now in `CompactionIterable` directly.
### Short reads
The current way to handle short reads would require us to consume the whole
result before deciding on a retry (and we currently retry the whole command),
which doesn't work too well in an iterative world. So the patch moves read
protection in `DataResolver` (where it kind of belong anyway) and we don't
retry the full command anymore. Instead, if we realize that a given node has a
short read while its result is consumed, we simply query this node (and this
node only) for a continuation of the result. On top of avoiding the retry of
the whole read (and limiting the number of node queried on the retry), this
also make it trivial to solve CASSANDRA-8933.
Storage format (on-disk and on-wire)
------------------------------------
Given that the main abstractions are changed, the existing on-wire
`ColumnFamily` format is not appropriate anymore and the patch switches to a
new format. The same can be told of the on-disk format, and while it is not an
objective of CASSANDRA-8099 to get fancy on the on-disk format, using the
on-wire format as on-disk format was actually relatively simple (until more
substantial changes land with CASSANDRA-7447) and the patch does that too.
For a given partition, the format simply serialize rows one after another
(atoms in practice). For the on-disk format, this means that it is now rows
that are indexed, not cells. The format uses a header that is written at the
beginning of each partition for the on-wire format (in
`AtomIteratorSerializer`) and is kept as a new sstable `Component` for
sstables. The details of the format are described in the javadoc of
`AtomIteratorSerializer` (only used by the on-wire format) and of
`AtomSerializer`, so let me just point the following differences compared to
the current format:
* Clustering values are only serialized once per row (we even skip serializing the
number of elements since that is fixed for a given table).
* Column names are not written for every row. Instead they are written once in
the header. For a given row, we support two small variant: dense and sparse.
When dense, cells come in the order the columns have in the header, meaning
that if a row doesn't have a particular column, this column will still use
a byte. When sparse, we don't have anything if the column doesn't have a cell,
but each cell has an additional 2 bytes which points into the header (so with
vint it should rarely take more than 1 byte in practice). The variant used
is automatically decided based on stats on how many columns set a row has on
average for the source we serialize.
* Values for fixed-width cell values are serialized without a size.
* If a cell has the same timestamp than its row, that timestamp is not repeated
for the cell. Same for the ttl (if applicable).
* Timestamps, ttls and local deletion times are delta encoded so that they are
ripe for vint encoding. The current version of the patch does not yet
activate vint encoding however (neither for on-wire or on-disk).

2
src/java/org/apache/cassandra/cache/AutoSavingCache.java vendored
View File

@ -207,7 +207,7 @@ public class AutoSavingCache<K extends CacheKey, V> extends InstrumentingCache<K
else
type = OperationType.UNKNOWN;
info = new CompactionInfo(CFMetaData.denseCFMetaData(SystemKeyspace.NAME, cacheType.toString(), BytesType.instance),
info = new CompactionInfo(CFMetaData.createFake(SystemKeyspace.NAME, cacheType.toString()),
type,
0,
keysEstimate,

29
src/java/org/apache/cassandra/cache/CounterCacheKey.java vendored
View File

@ -21,29 +21,44 @@ import java.nio.ByteBuffer;
import java.util.Arrays;
import java.util.UUID;
import org.apache.cassandra.db.composites.CellName;
import org.apache.cassandra.db.composites.CellNames;
import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.db.*;
import org.apache.cassandra.db.rows.CellPath;
import org.apache.cassandra.db.marshal.CompositeType;
import org.apache.cassandra.utils.*;
public class CounterCacheKey implements CacheKey
{
private static final long EMPTY_SIZE = ObjectSizes.measure(new CounterCacheKey(null, ByteBufferUtil.EMPTY_BYTE_BUFFER, CellNames.simpleDense(ByteBuffer.allocate(1))))
private static final long EMPTY_SIZE = ObjectSizes.measure(new CounterCacheKey(null, ByteBufferUtil.EMPTY_BYTE_BUFFER, ByteBuffer.allocate(1)))
+ ObjectSizes.measure(new UUID(0, 0));
public final UUID cfId;
public final byte[] partitionKey;
public final byte[] cellName;
private CounterCacheKey(UUID cfId, ByteBuffer partitionKey, CellName cellName)
public CounterCacheKey(UUID cfId, ByteBuffer partitionKey, ByteBuffer cellName)
{
this.cfId = cfId;
this.partitionKey = ByteBufferUtil.getArray(partitionKey);
this.cellName = ByteBufferUtil.getArray(cellName.toByteBuffer());
this.cellName = ByteBufferUtil.getArray(cellName);
}
public static CounterCacheKey create(UUID cfId, ByteBuffer partitionKey, CellName cellName)
public static CounterCacheKey create(UUID cfId, ByteBuffer partitionKey, Clustering clustering, ColumnDefinition c, CellPath path)
{
return new CounterCacheKey(cfId, partitionKey, cellName);
return new CounterCacheKey(cfId, partitionKey, makeCellName(clustering, c, path));
}
private static ByteBuffer makeCellName(Clustering clustering, ColumnDefinition c, CellPath path)
{
int cs = clustering.size();
ByteBuffer[] values = new ByteBuffer[cs + 1 + (path == null ? 0 : path.size())];
for (int i = 0; i < cs; i++)
values[i] = clustering.get(i);
values[cs] = c.name.bytes;
if (path != null)
for (int i = 0; i < path.size(); i++)
values[cs + 1 + i] = path.get(i);
return CompositeType.build(values);
}
public UUID getCFId()

8
src/java/org/apache/cassandra/cache/OHCProvider.java vendored
View File

@ -28,8 +28,8 @@ import java.util.UUID;
import com.google.common.base.Function;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.db.ColumnFamily;
import org.apache.cassandra.db.TypeSizes;
import org.apache.cassandra.db.partitions.CachedPartition;
import org.apache.cassandra.io.util.DataOutputPlus;
import org.apache.cassandra.io.util.Memory;
import org.apache.cassandra.net.MessagingService;
@ -159,7 +159,7 @@ public class OHCProvider implements CacheProvider<RowCacheKey, IRowCacheEntry>
if (isSentinel)
out.writeLong(((RowCacheSentinel) entry).sentinelId);
else
ColumnFamily.serializer.serialize((ColumnFamily) entry, new DataOutputPlusAdapter(out), MessagingService.current_version);
CachedPartition.cacheSerializer.serialize((CachedPartition)entry, new DataOutputPlusAdapter(out));
}
public IRowCacheEntry deserialize(DataInput in) throws IOException
@ -167,7 +167,7 @@ public class OHCProvider implements CacheProvider<RowCacheKey, IRowCacheEntry>
boolean isSentinel = in.readBoolean();
if (isSentinel)
return new RowCacheSentinel(in.readLong());
return ColumnFamily.serializer.deserialize(in, MessagingService.current_version);
return CachedPartition.cacheSerializer.deserialize(in);
}
public int serializedSize(IRowCacheEntry entry)
@ -177,7 +177,7 @@ public class OHCProvider implements CacheProvider<RowCacheKey, IRowCacheEntry>
if (entry instanceof RowCacheSentinel)
size += typeSizes.sizeof(((RowCacheSentinel) entry).sentinelId);
else
size += ColumnFamily.serializer.serializedSize((ColumnFamily) entry, typeSizes, MessagingService.current_version);
size += CachedPartition.cacheSerializer.serializedSize((CachedPartition) entry, typeSizes);
return size;
}
}

9
src/java/org/apache/cassandra/cache/SerializingCacheProvider.java vendored
View File

@ -21,8 +21,8 @@ import java.io.DataInput;
import java.io.IOException;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.db.ColumnFamily;
import org.apache.cassandra.db.TypeSizes;
import org.apache.cassandra.db.partitions.CachedPartition;
import org.apache.cassandra.io.ISerializer;
import org.apache.cassandra.io.util.DataOutputPlus;
import org.apache.cassandra.net.MessagingService;
@ -45,7 +45,7 @@ public class SerializingCacheProvider implements CacheProvider<RowCacheKey, IRow
if (isSentinel)
out.writeLong(((RowCacheSentinel) entry).sentinelId);
else
ColumnFamily.serializer.serialize((ColumnFamily) entry, out, MessagingService.current_version);
CachedPartition.cacheSerializer.serialize((CachedPartition)entry, out);
}
public IRowCacheEntry deserialize(DataInput in) throws IOException
@ -53,7 +53,8 @@ public class SerializingCacheProvider implements CacheProvider<RowCacheKey, IRow
boolean isSentinel = in.readBoolean();
if (isSentinel)
return new RowCacheSentinel(in.readLong());
return ColumnFamily.serializer.deserialize(in, MessagingService.current_version);
return CachedPartition.cacheSerializer.deserialize(in);
}
public long serializedSize(IRowCacheEntry entry, TypeSizes typeSizes)
@ -62,7 +63,7 @@ public class SerializingCacheProvider implements CacheProvider<RowCacheKey, IRow
if (entry instanceof RowCacheSentinel)
size += typeSizes.sizeof(((RowCacheSentinel) entry).sentinelId);
else
size += ColumnFamily.serializer.serializedSize((ColumnFamily) entry, typeSizes, MessagingService.current_version);
size += CachedPartition.cacheSerializer.serializedSize((CachedPartition) entry, typeSizes);
return size;
}
}

1058
src/java/org/apache/cassandra/config/CFMetaData.java
View File

File diff suppressed because it is too large Load Diff

203
src/java/org/apache/cassandra/config/ColumnDefinition.java
View File

@ -26,18 +26,19 @@ import com.google.common.base.Objects;
import com.google.common.collect.Lists;
import org.apache.cassandra.cql3.*;
import org.apache.cassandra.db.rows.*;
import org.apache.cassandra.db.marshal.*;
import org.apache.cassandra.serializers.MarshalException;
import org.apache.cassandra.exceptions.*;
import org.apache.cassandra.utils.ByteBufferUtil;
public class ColumnDefinition extends ColumnSpecification
public class ColumnDefinition extends ColumnSpecification implements Comparable<ColumnDefinition>
{
/*
* The type of CQL3 column this definition represents.
* There is 3 main type of CQL3 columns: those parts of the partition key,
* those parts of the clustering key and the other, regular ones.
* But when COMPACT STORAGE is used, there is by design only one regular
* column, whose name is not stored in the data contrarily to the column of
* type REGULAR. Hence the COMPACT_VALUE type to distinguish it below.
* There is 4 main type of CQL3 columns: those parts of the partition key,
* those parts of the clustering key and amongst the others, regular and
* static ones.
*
* Note that thrift only knows about definitions of type REGULAR (and
* the ones whose componentIndex == null).
@ -47,8 +48,12 @@ public class ColumnDefinition extends ColumnSpecification
PARTITION_KEY,
CLUSTERING_COLUMN,
REGULAR,
STATIC,
COMPACT_VALUE
STATIC;
public boolean isPrimaryKeyKind()
{
return this == PARTITION_KEY || this == CLUSTERING_COLUMN;
}
}
public final Kind kind;
@ -64,14 +69,17 @@ public class ColumnDefinition extends ColumnSpecification
*/
private final Integer componentIndex;
private final Comparator<CellPath> cellPathComparator;
private final Comparator<Cell> cellComparator;
public static ColumnDefinition partitionKeyDef(CFMetaData cfm, ByteBuffer name, AbstractType<?> validator, Integer componentIndex)
{
return new ColumnDefinition(cfm, name, validator, componentIndex, Kind.PARTITION_KEY);
}
public static ColumnDefinition partitionKeyDef(String ksName, String cfName, ByteBuffer name, AbstractType<?> validator, Integer componentIndex)
public static ColumnDefinition partitionKeyDef(String ksName, String cfName, String name, AbstractType<?> validator, Integer componentIndex)
{
return new ColumnDefinition(ksName, cfName, new ColumnIdentifier(name, UTF8Type.instance), validator, null, null, null, componentIndex, Kind.PARTITION_KEY);
return new ColumnDefinition(ksName, cfName, ColumnIdentifier.getInterned(name, true), validator, null, null, null, componentIndex, Kind.PARTITION_KEY);
}
public static ColumnDefinition clusteringKeyDef(CFMetaData cfm, ByteBuffer name, AbstractType<?> validator, Integer componentIndex)
@ -79,26 +87,31 @@ public class ColumnDefinition extends ColumnSpecification
return new ColumnDefinition(cfm, name, validator, componentIndex, Kind.CLUSTERING_COLUMN);
}
public static ColumnDefinition clusteringKeyDef(String ksName, String cfName, String name, AbstractType<?> validator, Integer componentIndex)
{
return new ColumnDefinition(ksName, cfName, ColumnIdentifier.getInterned(name, true), validator, null, null, null, componentIndex, Kind.CLUSTERING_COLUMN);
}
public static ColumnDefinition regularDef(CFMetaData cfm, ByteBuffer name, AbstractType<?> validator, Integer componentIndex)
{
return new ColumnDefinition(cfm, name, validator, componentIndex, Kind.REGULAR);
}
public static ColumnDefinition regularDef(String ksName, String cfName, String name, AbstractType<?> validator, Integer componentIndex)
{
return new ColumnDefinition(ksName, cfName, ColumnIdentifier.getInterned(name, true), validator, null, null, null, componentIndex, Kind.REGULAR);
}
public static ColumnDefinition staticDef(CFMetaData cfm, ByteBuffer name, AbstractType<?> validator, Integer componentIndex)
{
return new ColumnDefinition(cfm, name, validator, componentIndex, Kind.STATIC);
}
public static ColumnDefinition compactValueDef(CFMetaData cfm, ByteBuffer name, AbstractType<?> validator)
{
return new ColumnDefinition(cfm, name, validator, null, Kind.COMPACT_VALUE);
}
public ColumnDefinition(CFMetaData cfm, ByteBuffer name, AbstractType<?> validator, Integer componentIndex, Kind kind)
{
this(cfm.ksName,
cfm.cfName,
new ColumnIdentifier(name, cfm.getComponentComparator(componentIndex, kind)),
ColumnIdentifier.getInterned(name, cfm.getColumnDefinitionNameComparator(kind)),
validator,
null,
null,
@ -107,6 +120,11 @@ public class ColumnDefinition extends ColumnSpecification
kind);
}
public ColumnDefinition(String ksName, String cfName, ColumnIdentifier name, AbstractType<?> type, Integer componentIndex, Kind kind)
{
this(ksName, cfName, name, type, null, null, null, componentIndex, kind);
}
@VisibleForTesting
public ColumnDefinition(String ksName,
String cfName,
@ -119,11 +137,55 @@ public class ColumnDefinition extends ColumnSpecification
Kind kind)
{
super(ksName, cfName, name, validator);
assert name != null && validator != null;
assert name != null && validator != null && kind != null;
assert name.isInterned();
this.kind = kind;
this.indexName = indexName;
this.componentIndex = componentIndex;
this.setIndexType(indexType, indexOptions);
this.cellPathComparator = makeCellPathComparator(kind, validator);
this.cellComparator = makeCellComparator(cellPathComparator);
}
private static Comparator<CellPath> makeCellPathComparator(Kind kind, AbstractType<?> validator)
{
if (kind.isPrimaryKeyKind() || !validator.isCollection() || !validator.isMultiCell())
return null;
final CollectionType type = (CollectionType)validator;
return new Comparator<CellPath>()
{
public int compare(CellPath path1, CellPath path2)
{
if (path1.size() == 0 || path2.size() == 0)
{
if (path1 == CellPath.BOTTOM)
return path2 == CellPath.BOTTOM ? 0 : -1;
if (path1 == CellPath.TOP)
return path2 == CellPath.TOP ? 0 : 1;
return path2 == CellPath.BOTTOM ? 1 : -1;
}
// This will get more complicated once we have non-frozen UDT and nested collections
assert path1.size() == 1 && path2.size() == 1;
return type.nameComparator().compare(path1.get(0), path2.get(0));
}
};
}
private static Comparator<Cell> makeCellComparator(final Comparator<CellPath> cellPathComparator)
{
return new Comparator<Cell>()
{
public int compare(Cell c1, Cell c2)
{
int cmp = c1.column().compareTo(c2.column());
if (cmp != 0 || cellPathComparator == null)
return cmp;
return cellPathComparator.compare(c1.path(), c2.path());
}
};
}
public ColumnDefinition copy()
@ -166,11 +228,6 @@ public class ColumnDefinition extends ColumnSpecification
return kind == Kind.REGULAR;
}
public boolean isCompactValue()
{
return kind == Kind.COMPACT_VALUE;
}
// The componentIndex. This never return null however for convenience sake:
// if componentIndex == null, this return 0. So caller should first check
// isOnAllComponents() to distinguish if that's a possibility.
@ -220,23 +277,26 @@ public class ColumnDefinition extends ColumnSpecification
.toString();
}
public boolean isThriftCompatible()
{
return kind == ColumnDefinition.Kind.REGULAR && componentIndex == null;
}
public boolean isPrimaryKeyColumn()
{
return kind == Kind.PARTITION_KEY || kind == Kind.CLUSTERING_COLUMN;
return kind.isPrimaryKeyKind();
}
/**
* Whether the name of this definition is serialized in the cell nane, i.e. whether
* it's not just a non-stored CQL metadata.
*/
public boolean isPartOfCellName()
public boolean isPartOfCellName(boolean isCQL3Table, boolean isSuper)
{
return kind == Kind.REGULAR || kind == Kind.STATIC;
// When converting CQL3 tables to thrift, any regular or static column ends up in the cell name.
// When it's a compact table however, the REGULAR definition is the name for the cell value of "dynamic"
// column (so it's not part of the cell name) and it's static columns that ends up in the cell name.
if (isCQL3Table)
return kind == Kind.REGULAR || kind == Kind.STATIC;
else if (isSuper)
return kind == Kind.REGULAR;
else
return kind == Kind.STATIC;
}
public ColumnDefinition apply(ColumnDefinition def) throws ConfigurationException
@ -333,4 +393,87 @@ public class ColumnDefinition extends ColumnSpecification
}
});
}
public int compareTo(ColumnDefinition other)
{
if (this == other)
return 0;
if (kind != other.kind)
return kind.ordinal() < other.kind.ordinal() ? -1 : 1;
if (position() != other.position())
return position() < other.position() ? -1 : 1;
if (isStatic() != other.isStatic())
return isStatic() ? -1 : 1;
if (isComplex() != other.isComplex())
return isComplex() ? 1 : -1;
return ByteBufferUtil.compareUnsigned(name.bytes, other.name.bytes);
}
public Comparator<CellPath> cellPathComparator()
{
return cellPathComparator;
}
public Comparator<Cell> cellComparator()
{
return cellComparator;
}
public boolean isComplex()
{
return cellPathComparator != null;
}
public CellPath.Serializer cellPathSerializer()
{
// Collections are our only complex so far, so keep it simple
return CollectionType.cellPathSerializer;
}
public void validateCellValue(ByteBuffer value)
{
type.validateCellValue(value);
}
public void validateCellPath(CellPath path)
{
if (!isComplex())
throw new MarshalException("Only complex cells should have a cell path");
assert type instanceof CollectionType;
((CollectionType)type).nameComparator().validate(path.get(0));
}
public static String toCQLString(Iterable<ColumnDefinition> defs)
{
return toCQLString(defs.iterator());
}
public static String toCQLString(Iterator<ColumnDefinition> defs)
{
if (!defs.hasNext())
return "";
StringBuilder sb = new StringBuilder();
sb.append(defs.next().name);
while (defs.hasNext())
sb.append(", ").append(defs.next().name);
return sb.toString();
}
/**
* The type of the cell values for cell belonging to this column.
*
* This is the same than the column type, except for collections where it's the 'valueComparator'
* of the collection.
*/
public AbstractType<?> cellValueType()
{
return type instanceof CollectionType
? ((CollectionType)type).valueComparator()
: type;
}
}

8
src/java/org/apache/cassandra/config/DatabaseDescriptor.java
View File

@ -1341,9 +1341,15 @@ public class DatabaseDescriptor
}
@VisibleForTesting
public static void setAutoSnapshot(boolean autoSnapshot) {
public static void setAutoSnapshot(boolean autoSnapshot)
{
conf.auto_snapshot = autoSnapshot;
}
@VisibleForTesting
public static boolean getAutoSnapshot()
{
return conf.auto_snapshot;
}
public static boolean isAutoBootstrap()
{

7
src/java/org/apache/cassandra/config/YamlConfigurationLoader.java
View File

@ -70,8 +70,11 @@ public class YamlConfigurationLoader implements ConfigurationLoader
{
String required = "file:" + File.separator + File.separator;
if (!configUrl.startsWith(required))
throw new ConfigurationException("Expecting URI in variable: [cassandra.config]. Please prefix the file with " + required + File.separator +
" for local files or " + required + "<server>" + File.separator + " for remote files. Aborting. If you are executing this from an external tool, it needs to set Config.setClientMode(true) to avoid loading configuration.");
throw new ConfigurationException(String.format(
"Expecting URI in variable: [cassandra.config]. Found[%s]. Please prefix the file with [%s%s] for local " +
"files and [%s<server>%s] for remote files. If you are executing this from an external tool, it needs " +
"to set Config.setClientMode(true) to avoid loading configuration.",
configUrl, required, File.separator, required, File.separator));
throw new ConfigurationException("Cannot locate " + configUrl + ". If this is a local file, please confirm you've provided " + required + File.separator + " as a URI prefix.");
}
}

7
src/java/org/apache/cassandra/cql3/Attributes.java
View File

@ -23,7 +23,6 @@ import java.util.Collections;
import com.google.common.collect.Iterables;
import org.apache.cassandra.cql3.functions.Function;
import org.apache.cassandra.db.ExpiringCell;
import org.apache.cassandra.db.marshal.Int32Type;
import org.apache.cassandra.db.marshal.LongType;
import org.apache.cassandra.exceptions.InvalidRequestException;
@ -36,6 +35,8 @@ import org.apache.cassandra.utils.ByteBufferUtil;
*/
public class Attributes
{
public static final int MAX_TTL = 20 * 365 * 24 * 60 * 60; // 20 years in seconds
private final Term timestamp;
private final Term timeToLive;
@ -121,8 +122,8 @@ public class Attributes
if (ttl < 0)
throw new InvalidRequestException("A TTL must be greater or equal to 0, but was " + ttl);
if (ttl > ExpiringCell.MAX_TTL)
throw new InvalidRequestException(String.format("ttl is too large. requested (%d) maximum (%d)", ttl, ExpiringCell.MAX_TTL));
if (ttl > MAX_TTL)
throw new InvalidRequestException(String.format("ttl is too large. requested (%d) maximum (%d)", ttl, MAX_TTL));
return ttl;
}

156
src/java/org/apache/cassandra/cql3/ColumnCondition.java
View File

@ -20,18 +20,12 @@ package org.apache.cassandra.cql3;
import java.nio.ByteBuffer;
import java.util.*;
import com.google.common.base.Predicate;
import com.google.common.collect.Iterables;
import com.google.common.collect.Iterators;
import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.cql3.functions.Function;
import org.apache.cassandra.db.Cell;
import org.apache.cassandra.db.ColumnFamily;
import org.apache.cassandra.db.composites.CellName;
import org.apache.cassandra.db.composites.CellNameType;
import org.apache.cassandra.db.composites.Composite;
import org.apache.cassandra.db.filter.ColumnSlice;
import org.apache.cassandra.db.rows.*;
import org.apache.cassandra.db.marshal.*;
import org.apache.cassandra.exceptions.InvalidRequestException;
import org.apache.cassandra.transport.Server;
@ -160,20 +154,19 @@ public class ColumnCondition
/**
* Validates whether this condition applies to {@code current}.
*/
public abstract boolean appliesTo(Composite rowPrefix, ColumnFamily current, long now) throws InvalidRequestException;
public abstract boolean appliesTo(Row row) throws InvalidRequestException;
public ByteBuffer getCollectionElementValue()
{
return null;
}
protected boolean isSatisfiedByValue(ByteBuffer value, Cell c, AbstractType<?> type, Operator operator, long now) throws InvalidRequestException
protected boolean isSatisfiedByValue(ByteBuffer value, Cell c, AbstractType<?> type, Operator operator) throws InvalidRequestException
{
ByteBuffer columnValue = (c == null || !c.isLive(now)) ? null : c.value();
return compareWithOperator(operator, type, value, columnValue);
return compareWithOperator(operator, type, value, c == null ? null : c.value());
}
/** Returns true if the operator is satisfied (i.e. "value operator otherValue == true"), false otherwise. */
/** Returns true if the operator is satisfied (i.e. "otherValue operator value == true"), false otherwise. */
protected boolean compareWithOperator(Operator operator, AbstractType<?> type, ByteBuffer value, ByteBuffer otherValue) throws InvalidRequestException
{
if (value == ByteBufferUtil.UNSET_BYTE_BUFFER)
@ -195,41 +188,29 @@ public class ColumnCondition
// the condition value is not null, so only NEQ can return true
return operator == Operator.NEQ;
}
int comparison = type.compare(otherValue, value);
switch (operator)
{
case EQ:
return comparison == 0;
case LT:
return comparison < 0;
case LTE:
return comparison <= 0;
case GT:
return comparison > 0;
case GTE:
return comparison >= 0;
case NEQ:
return comparison != 0;
default:
// we shouldn't get IN, CONTAINS, or CONTAINS KEY here
throw new AssertionError();
}
return operator.isSatisfiedBy(type, otherValue, value);
}
}
protected Iterator<Cell> collectionColumns(CellName collection, ColumnFamily cf, final long now)
{
// We are testing for collection equality, so we need to have the expected values *and* only those.
ColumnSlice[] collectionSlice = new ColumnSlice[]{ collection.slice() };
// Filter live columns, this makes things simpler afterwards
return Iterators.filter(cf.iterator(collectionSlice), new Predicate<Cell>()
{
public boolean apply(Cell c)
{
// we only care about live columns
return c.isLive(now);
}
});
}
private static Cell getCell(Row row, ColumnDefinition column)
{
// If we're asking for a given cell, and we didn't got any row from our read, it's
// the same as not having said cell.
return row == null ? null : row.getCell(column);
}
private static Cell getCell(Row row, ColumnDefinition column, CellPath path)
{
// If we're asking for a given cell, and we didn't got any row from our read, it's
// the same as not having said cell.
return row == null ? null : row.getCell(column, path);
}
private static Iterator<Cell> getCells(Row row, ColumnDefinition column)
{
// If we're asking for a complex cells, and we didn't got any row from our read, it's
// the same as not having any cells for that column.
return row == null ? Collections.<Cell>emptyIterator() : row.getCells(column);
}
/**
@ -247,10 +228,9 @@ public class ColumnCondition
this.value = condition.value.bindAndGet(options);
}
public boolean appliesTo(Composite rowPrefix, ColumnFamily current, long now) throws InvalidRequestException
public boolean appliesTo(Row row) throws InvalidRequestException
{
CellName name = current.metadata().comparator.create(rowPrefix, column);
return isSatisfiedByValue(value, current.getColumn(name), column.type, operator, now);
return isSatisfiedByValue(value, getCell(row, column), column.type, operator);
}
}
@ -276,12 +256,12 @@ public class ColumnCondition
}
}
public boolean appliesTo(Composite rowPrefix, ColumnFamily current, long now) throws InvalidRequestException
public boolean appliesTo(Row row) throws InvalidRequestException
{
CellName name = current.metadata().comparator.create(rowPrefix, column);
Cell c = getCell(row, column);
for (ByteBuffer value : inValues)
{
if (isSatisfiedByValue(value, current.getColumn(name), column.type, Operator.EQ, now))
if (isSatisfiedByValue(value, c, column.type, Operator.EQ))
return true;
}
return false;
@ -303,7 +283,7 @@ public class ColumnCondition
this.value = condition.value.bindAndGet(options);
}
public boolean appliesTo(Composite rowPrefix, ColumnFamily current, final long now) throws InvalidRequestException
public boolean appliesTo(Row row) throws InvalidRequestException
{
if (collectionElement == null)
throw new InvalidRequestException("Invalid null value for " + (column.type instanceof MapType ? "map" : "list") + " element access");
@ -313,14 +293,13 @@ public class ColumnCondition
MapType mapType = (MapType) column.type;
if (column.type.isMultiCell())
{
Cell cell = current.getColumn(current.metadata().comparator.create(rowPrefix, column, collectionElement));
return isSatisfiedByValue(value, cell, mapType.getValuesType(), operator, now);
Cell cell = getCell(row, column, CellPath.create(collectionElement));
return isSatisfiedByValue(value, cell, ((MapType) column.type).getValuesType(), operator);
}
else
{
Cell cell = current.getColumn(current.metadata().comparator.create(rowPrefix, column));
ByteBuffer mapElementValue = cell.isLive(now) ? mapType.getSerializer().getSerializedValue(cell.value(), collectionElement, mapType.getKeysType())
: null;
Cell cell = getCell(row, column);
ByteBuffer mapElementValue = mapType.getSerializer().getSerializedValue(cell.value(), collectionElement, mapType.getKeysType());
return compareWithOperator(operator, mapType.getValuesType(), value, mapElementValue);
}
}
@ -329,16 +308,13 @@ public class ColumnCondition
ListType listType = (ListType) column.type;
if (column.type.isMultiCell())
{
ByteBuffer columnValue = getListItem(
collectionColumns(current.metadata().comparator.create(rowPrefix, column), current, now),
getListIndex(collectionElement));
return compareWithOperator(operator, listType.getElementsType(), value, columnValue);
ByteBuffer columnValue = getListItem(getCells(row, column), getListIndex(collectionElement));
return compareWithOperator(operator, ((ListType)column.type).getElementsType(), value, columnValue);
}
else
{
Cell cell = current.getColumn(current.metadata().comparator.create(rowPrefix, column));
ByteBuffer listElementValue = cell.isLive(now) ? listType.getSerializer().getElement(cell.value(), getListIndex(collectionElement))
: null;
Cell cell = getCell(row, column);
ByteBuffer listElementValue = listType.getSerializer().getElement(cell.value(), getListIndex(collectionElement));
return compareWithOperator(operator, listType.getElementsType(), value, listElementValue);
}
}
@ -387,33 +363,31 @@ public class ColumnCondition
}
}
public boolean appliesTo(Composite rowPrefix, ColumnFamily current, final long now) throws InvalidRequestException
public boolean appliesTo(Row row) throws InvalidRequestException
{
if (collectionElement == null)
throw new InvalidRequestException("Invalid null value for " + (column.type instanceof MapType ? "map" : "list") + " element access");
CellNameType nameType = current.metadata().comparator;
if (column.type instanceof MapType)
{
MapType mapType = (MapType) column.type;
AbstractType<?> valueType = mapType.getValuesType();
if (column.type.isMultiCell())
{
CellName name = nameType.create(rowPrefix, column, collectionElement);
Cell item = current.getColumn(name);
Cell item = getCell(row, column, CellPath.create(collectionElement));
for (ByteBuffer value : inValues)
{
if (isSatisfiedByValue(value, item, valueType, Operator.EQ, now))
if (isSatisfiedByValue(value, item, valueType, Operator.EQ))
return true;
}
return false;
}
else
{
Cell cell = current.getColumn(nameType.create(rowPrefix, column));
ByteBuffer mapElementValue = null;
if (cell != null && cell.isLive(now))
mapElementValue = mapType.getSerializer().getSerializedValue(cell.value(), collectionElement, mapType.getKeysType());
Cell cell = getCell(row, column);
ByteBuffer mapElementValue = cell == null
? null
: mapType.getSerializer().getSerializedValue(cell.value(), collectionElement, mapType.getKeysType());
for (ByteBuffer value : inValues)
{
if (value == null)
@ -433,9 +407,7 @@ public class ColumnCondition
AbstractType<?> elementsType = listType.getElementsType();
if (column.type.isMultiCell())
{
ByteBuffer columnValue = ElementAccessBound.getListItem(
collectionColumns(nameType.create(rowPrefix, column), current, now),
ElementAccessBound.getListIndex(collectionElement));
ByteBuffer columnValue = ElementAccessBound.getListItem(getCells(row, column), ElementAccessBound.getListIndex(collectionElement));
for (ByteBuffer value : inValues)
{
@ -445,10 +417,10 @@ public class ColumnCondition
}
else
{
Cell cell = current.getColumn(nameType.create(rowPrefix, column));
ByteBuffer listElementValue = null;
if (cell != null && cell.isLive(now))
listElementValue = listType.getSerializer().getElement(cell.value(), ElementAccessBound.getListIndex(collectionElement));
Cell cell = getCell(row, column);
ByteBuffer listElementValue = cell == null
? null
: listType.getSerializer().getElement(cell.value(), ElementAccessBound.getListIndex(collectionElement));
for (ByteBuffer value : inValues)
{
@ -479,13 +451,13 @@ public class ColumnCondition
this.value = condition.value.bind(options);
}
public boolean appliesTo(Composite rowPrefix, ColumnFamily current, final long now) throws InvalidRequestException
public boolean appliesTo(Row row) throws InvalidRequestException
{
CollectionType type = (CollectionType)column.type;
if (type.isMultiCell())
{
Iterator<Cell> iter = collectionColumns(current.metadata().comparator.create(rowPrefix, column), current, now);
Iterator<Cell> iter = getCells(row, column);
if (value == null)
{
if (operator == Operator.EQ)
@ -500,13 +472,13 @@ public class ColumnCondition
}
// frozen collections
Cell cell = current.getColumn(current.metadata().comparator.create(rowPrefix, column));
Cell cell = getCell(row, column);
if (value == null)
{
if (operator == Operator.EQ)
return cell == null || !cell.isLive(now);
return cell == null;
else if (operator == Operator.NEQ)
return cell != null && cell.isLive(now);
return cell != null;
else
throw new InvalidRequestException(String.format("Invalid comparison with null for operator \"%s\"", operator));
}
@ -551,7 +523,7 @@ public class ColumnCondition
return (operator == Operator.GT) || (operator == Operator.GTE) || (operator == Operator.NEQ);
// for lists we use the cell value; for sets we use the cell name
ByteBuffer cellValue = isSet? iter.next().name().collectionElement() : iter.next().value();
ByteBuffer cellValue = isSet ? iter.next().path().get(0) : iter.next().value();
int comparison = type.compare(cellValue, conditionIter.next());
if (comparison != 0)
return evaluateComparisonWithOperator(comparison, operator);
@ -609,7 +581,7 @@ public class ColumnCondition
Cell c = iter.next();
// compare the keys
int comparison = type.getKeysType().compare(c.name().collectionElement(), conditionEntry.getKey());
int comparison = type.getKeysType().compare(c.path().get(0), conditionEntry.getKey());
if (comparison != 0)
return evaluateComparisonWithOperator(comparison, operator);
@ -683,29 +655,27 @@ public class ColumnCondition
}
}
public boolean appliesTo(Composite rowPrefix, ColumnFamily current, final long now) throws InvalidRequestException
public boolean appliesTo(Row row) throws InvalidRequestException
{
CollectionType type = (CollectionType)column.type;
CellName name = current.metadata().comparator.create(rowPrefix, column);
if (type.isMultiCell())
{
// copy iterator contents so that we can properly reuse them for each comparison with an IN value
List<Cell> cells = newArrayList(collectionColumns(name, current, now));
for (Term.Terminal value : inValues)
{
if (CollectionBound.valueAppliesTo(type, cells.iterator(), value, Operator.EQ))
if (CollectionBound.valueAppliesTo(type, getCells(row, column), value, Operator.EQ))
return true;
}
return false;
}
else
{
Cell cell = current.getColumn(name);
Cell cell = getCell(row, column);
for (Term.Terminal value : inValues)
{
if (value == null)
{
if (cell == null || !cell.isLive(now))
if (cell == null)
return true;
}
else if (type.compare(value.get(Server.VERSION_3), cell.value()) == 0)

70
src/java/org/apache/cassandra/cql3/ColumnIdentifier.java
View File

@ -20,6 +20,10 @@ package org.apache.cassandra.cql3;
import java.nio.ByteBuffer;
import java.util.List;
import java.util.Locale;
import java.nio.ByteBuffer;
import java.util.concurrent.ConcurrentMap;
import com.google.common.collect.MapMaker;
import org.apache.cassandra.cache.IMeasurableMemory;
import org.apache.cassandra.config.CFMetaData;
@ -28,7 +32,7 @@ import org.apache.cassandra.cql3.selection.Selectable;
import org.apache.cassandra.cql3.selection.Selector;
import org.apache.cassandra.cql3.selection.SimpleSelector;
import org.apache.cassandra.db.marshal.AbstractType;
import org.apache.cassandra.db.marshal.CompositeType;
import org.apache.cassandra.exceptions.InvalidRequestException;
import org.apache.cassandra.db.marshal.UTF8Type;
import org.apache.cassandra.exceptions.InvalidRequestException;
import org.apache.cassandra.utils.ByteBufferUtil;
@ -43,25 +47,57 @@ public class ColumnIdentifier extends org.apache.cassandra.cql3.selection.Select
{
public final ByteBuffer bytes;
private final String text;
private final boolean interned;
private static final long EMPTY_SIZE = ObjectSizes.measure(new ColumnIdentifier("", true));
private static final long EMPTY_SIZE = ObjectSizes.measure(new ColumnIdentifier(ByteBufferUtil.EMPTY_BYTE_BUFFER, "", false));
private static final ConcurrentMap<ByteBuffer, ColumnIdentifier> internedInstances = new MapMaker().weakValues().makeMap();
public ColumnIdentifier(String rawText, boolean keepCase)
{
this.text = keepCase ? rawText : rawText.toLowerCase(Locale.US);
this.bytes = ByteBufferUtil.bytes(this.text);
this.interned = false;
}
public ColumnIdentifier(ByteBuffer bytes, AbstractType<?> type)
{
this.bytes = bytes;
this.text = type.getString(bytes);
this(bytes, type.getString(bytes), false);
}
public ColumnIdentifier(ByteBuffer bytes, String text)
private ColumnIdentifier(ByteBuffer bytes, String text, boolean interned)
{
this.bytes = bytes;
this.text = text;
this.interned = interned;
}
public static ColumnIdentifier getInterned(ByteBuffer bytes, AbstractType<?> type)
{
return getInterned(bytes, type.getString(bytes));
}
public static ColumnIdentifier getInterned(String rawText, boolean keepCase)
{
String text = keepCase ? rawText : rawText.toLowerCase(Locale.US);
ByteBuffer bytes = ByteBufferUtil.bytes(text);
return getInterned(bytes, text);
}
public static ColumnIdentifier getInterned(ByteBuffer bytes, String text)
{
ColumnIdentifier id = internedInstances.get(bytes);
if (id != null)
return id;
ColumnIdentifier created = new ColumnIdentifier(bytes, text, true);
ColumnIdentifier previous = internedInstances.putIfAbsent(bytes, created);
return previous == null ? created : previous;
}
public boolean isInterned()
{
return interned;
}
@Override
@ -73,8 +109,6 @@ public class ColumnIdentifier extends org.apache.cassandra.cql3.selection.Select
@Override
public final boolean equals(Object o)
{
// Note: it's worth checking for reference equality since we intern those
// in SparseCellNameType
if (this == o)
return true;
@ -106,7 +140,7 @@ public class ColumnIdentifier extends org.apache.cassandra.cql3.selection.Select
public ColumnIdentifier clone(AbstractAllocator allocator)
{
return new ColumnIdentifier(allocator.clone(bytes), text);
return interned ? this : new ColumnIdentifier(allocator.clone(bytes), text, false);
}
public Selector.Factory newSelectorFactory(CFMetaData cfm, List<ColumnDefinition> defs) throws InvalidRequestException
@ -137,20 +171,22 @@ public class ColumnIdentifier extends org.apache.cassandra.cql3.selection.Select
public ColumnIdentifier prepare(CFMetaData cfm)
{
AbstractType<?> comparator = cfm.comparator.asAbstractType();
if (cfm.getIsDense() || comparator instanceof CompositeType || comparator instanceof UTF8Type)
return new ColumnIdentifier(text, true);
if (!cfm.isStaticCompactTable())
return getInterned(text, true);
// We have a Thrift-created table with a non-text comparator. We need to parse column names with the comparator
// to get the correct ByteBuffer representation. However, this doesn't apply to key aliases, so we need to
// make a special check for those and treat them normally. See CASSANDRA-8178.
AbstractType<?> thriftColumnNameType = cfm.thriftColumnNameType();
if (thriftColumnNameType instanceof UTF8Type)
return getInterned(text, true);
// We have a Thrift-created table with a non-text comparator. Check if we have a match column, otherwise assume we should use
// thriftColumnNameType
ByteBuffer bufferName = ByteBufferUtil.bytes(text);
for (ColumnDefinition def : cfm.partitionKeyColumns())
for (ColumnDefinition def : cfm.allColumns())
{
if (def.name.bytes.equals(bufferName))
return new ColumnIdentifier(text, true);
return def.name;
}
return new ColumnIdentifier(comparator.fromString(rawText), text);
return getInterned(thriftColumnNameType.fromString(rawText), text);
}
public boolean processesSelection()

31
src/java/org/apache/cassandra/cql3/Constants.java
View File

@ -23,8 +23,7 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.db.*;
import org.apache.cassandra.db.composites.CellName;
import org.apache.cassandra.db.composites.Composite;
import org.apache.cassandra.db.rows.*;
import org.apache.cassandra.db.marshal.AbstractType;
import org.apache.cassandra.db.marshal.BytesType;
import org.apache.cassandra.db.marshal.CounterColumnType;
@ -319,14 +318,13 @@ public abstract class Constants
super(column, t);
}
public void execute(ByteBuffer rowKey, ColumnFamily cf, Composite prefix, UpdateParameters params) throws InvalidRequestException
public void execute(DecoratedKey partitionKey, Clustering clustering, Row.Writer writer, UpdateParameters params) throws InvalidRequestException
{
ByteBuffer value = t.bindAndGet(params.options);
if (value != ByteBufferUtil.UNSET_BYTE_BUFFER) // use reference equality and not object equality
{
CellName cname = cf.getComparator().create(prefix, column);
cf.addColumn(value == null ? params.makeTombstone(cname) : params.makeColumn(cname, value));
}
if (value == null)
params.addTombstone(column, writer);
else if (value != ByteBufferUtil.UNSET_BYTE_BUFFER) // use reference equality and not object equality
params.addCell(clustering, column, writer, value);
}
}
@ -337,7 +335,7 @@ public abstract class Constants
super(column, t);
}
public void execute(ByteBuffer rowKey, ColumnFamily cf, Composite prefix, UpdateParameters params) throws InvalidRequestException
public void execute(DecoratedKey partitionKey, Clustering clustering, Row.Writer writer, UpdateParameters params) throws InvalidRequestException
{
ByteBuffer bytes = t.bindAndGet(params.options);
if (bytes == null)
@ -346,8 +344,7 @@ public abstract class Constants
return;
long increment = ByteBufferUtil.toLong(bytes);
CellName cname = cf.getComparator().create(prefix, column);
cf.addColumn(params.makeCounter(cname, increment));
params.addCounter(column, writer, increment);
}
}
@ -358,7 +355,7 @@ public abstract class Constants
super(column, t);
}
public void execute(ByteBuffer rowKey, ColumnFamily cf, Composite prefix, UpdateParameters params) throws InvalidRequestException
public void execute(DecoratedKey partitionKey, Clustering clustering, Row.Writer writer, UpdateParameters params) throws InvalidRequestException
{
ByteBuffer bytes = t.bindAndGet(params.options);
if (bytes == null)
@ -370,8 +367,7 @@ public abstract class Constants
if (increment == Long.MIN_VALUE)
throw new InvalidRequestException("The negation of " + increment + " overflows supported counter precision (signed 8 bytes integer)");
CellName cname = cf.getComparator().create(prefix, column);
cf.addColumn(params.makeCounter(cname, -increment));
params.addCounter(column, writer, -increment);
}
}
@ -384,13 +380,12 @@ public abstract class Constants
super(column, null);
}
public void execute(ByteBuffer rowKey, ColumnFamily cf, Composite prefix, UpdateParameters params) throws InvalidRequestException
public void execute(DecoratedKey partitionKey, Clustering clustering, Row.Writer writer, UpdateParameters params) throws InvalidRequestException
{
CellName cname = cf.getComparator().create(prefix, column);
if (column.type.isMultiCell())
cf.addAtom(params.makeRangeTombstone(cname.slice()));
params.setComplexDeletionTime(column, writer);
else
cf.addColumn(params.makeTombstone(cname));
params.addTombstone(column, writer);
}
};
}

47
src/java/org/apache/cassandra/cql3/Cql.g
View File

@ -318,7 +318,7 @@ selectClause returns [List<RawSelector> expr]
selector returns [RawSelector s]
@init{ ColumnIdentifier alias = null; }
: us=unaliasedSelector (K_AS c=ident { alias = c; })? { $s = new RawSelector(us, alias); }
: us=unaliasedSelector (K_AS c=noncol_ident { alias = c; })? { $s = new RawSelector(us, alias); }
;
unaliasedSelector returns [Selectable.Raw s]
@ -339,7 +339,7 @@ selectionFunctionArgs returns [List<Selectable.Raw> a]
selectCountClause returns [List<RawSelector> expr]
@init{ ColumnIdentifier alias = new ColumnIdentifier("count", false); }
: K_COUNT '(' countArgument ')' (K_AS c=ident { alias = c; })? { $expr = new ArrayList<RawSelector>(); $expr.add( new RawSelector(new Selectable.WithFunction.Raw(FunctionName.nativeFunction("countRows"), Collections.<Selectable.Raw>emptyList()), alias));}
: K_COUNT '(' countArgument ')' (K_AS c=noncol_ident { alias = c; })? { $expr = new ArrayList<RawSelector>(); $expr.add( new RawSelector(new Selectable.WithFunction.Raw(FunctionName.nativeFunction("countRows"), Collections.<Selectable.Raw>emptyList()), alias));}
;
countArgument
@ -404,7 +404,7 @@ jsonInsertStatement [CFName cf] returns [UpdateStatement.ParsedInsertJson expr]
jsonValue returns [Json.Raw value]
:
| s=STRING_LITERAL { $value = new Json.Literal($s.text); }
| ':' id=ident { $value = newJsonBindVariables(id); }
| ':' id=noncol_ident { $value = newJsonBindVariables(id); }
| QMARK { $value = newJsonBindVariables(null); }
;
@ -604,8 +604,8 @@ createFunctionStatement returns [CreateFunctionStatement expr]
fn=functionName
'('
(
k=ident v=comparatorType { argsNames.add(k); argsTypes.add(v); }
( ',' k=ident v=comparatorType { argsNames.add(k); argsTypes.add(v); } )*
k=noncol_ident v=comparatorType { argsNames.add(k); argsTypes.add(v); }
( ',' k=noncol_ident v=comparatorType { argsNames.add(k); argsTypes.add(v); } )*
)?
')'
( (K_RETURNS K_NULL) | (K_CALLED { calledOnNullInput=true; })) K_ON K_NULL K_INPUT
@ -706,7 +706,7 @@ createTypeStatement returns [CreateTypeStatement expr]
;
typeColumns[CreateTypeStatement expr]
: k=ident v=comparatorType { $expr.addDefinition(k, v); }
: k=noncol_ident v=comparatorType { $expr.addDefinition(k, v); }
;
@ -801,12 +801,12 @@ alterTableStatement returns [AlterTableStatement expr]
*/
alterTypeStatement returns [AlterTypeStatement expr]
: K_ALTER K_TYPE name=userTypeName
( K_ALTER f=ident K_TYPE v=comparatorType { $expr = AlterTypeStatement.alter(name, f, v); }
| K_ADD f=ident v=comparatorType { $expr = AlterTypeStatement.addition(name, f, v); }
( K_ALTER f=noncol_ident K_TYPE v=comparatorType { $expr = AlterTypeStatement.alter(name, f, v); }
| K_ADD f=noncol_ident v=comparatorType { $expr = AlterTypeStatement.addition(name, f, v); }
| K_RENAME
{ Map<ColumnIdentifier, ColumnIdentifier> renames = new HashMap<ColumnIdentifier, ColumnIdentifier>(); }
id1=ident K_TO toId1=ident { renames.put(id1, toId1); }
( K_AND idn=ident K_TO toIdn=ident { renames.put(idn, toIdn); } )*
id1=noncol_ident K_TO toId1=noncol_ident { renames.put(id1, toId1); }
( K_AND idn=noncol_ident K_TO toIdn=noncol_ident { renames.put(idn, toIdn); } )*
{ $expr = AlterTypeStatement.renames(name, renames); }
)
;
@ -1112,8 +1112,15 @@ cident returns [ColumnIdentifier.Raw id]
| k=unreserved_keyword { $id = new ColumnIdentifier.Raw(k, false); }
;
// Identifiers that do not refer to columns or where the comparator is known to be text
// Column identifiers where the comparator is known to be text
ident returns [ColumnIdentifier id]
: t=IDENT { $id = ColumnIdentifier.getInterned($t.text, false); }
| t=QUOTED_NAME { $id = ColumnIdentifier.getInterned($t.text, true); }
| k=unreserved_keyword { $id = ColumnIdentifier.getInterned(k, false); }
;
// Identifiers that do not refer to columns
noncol_ident returns [ColumnIdentifier id]
: t=IDENT { $id = new ColumnIdentifier($t.text, false); }
| t=QUOTED_NAME { $id = new ColumnIdentifier($t.text, true); }
| k=unreserved_keyword { $id = new ColumnIdentifier(k, false); }
@ -1136,7 +1143,7 @@ columnFamilyName returns [CFName name]
;
userTypeName returns [UTName name]
: (ks=ident '.')? ut=non_type_ident { return new UTName(ks, ut); }
: (ks=noncol_ident '.')? ut=non_type_ident { return new UTName(ks, ut); }
;
userOrRoleName returns [RoleName name]
@ -1211,7 +1218,7 @@ usertypeLiteral returns [UserTypes.Literal ut]
@init{ Map<ColumnIdentifier, Term.Raw> m = new HashMap<ColumnIdentifier, Term.Raw>(); }
@after{ $ut = new UserTypes.Literal(m); }
// We don't allow empty literals because that conflicts with sets/maps and is currently useless since we don't allow empty user types
: '{' k1=ident ':' v1=term { m.put(k1, v1); } ( ',' kn=ident ':' vn=term { m.put(kn, vn); } )* '}'
: '{' k1=noncol_ident ':' v1=term { m.put(k1, v1); } ( ',' kn=noncol_ident ':' vn=term { m.put(kn, vn); } )* '}'
;
tupleLiteral returns [Tuples.Literal tt]
@ -1226,14 +1233,14 @@ value returns [Term.Raw value]
| u=usertypeLiteral { $value = u; }
| t=tupleLiteral { $value = t; }
| K_NULL { $value = Constants.NULL_LITERAL; }
| ':' id=ident { $value = newBindVariables(id); }
| ':' id=noncol_ident { $value = newBindVariables(id); }
| QMARK { $value = newBindVariables(null); }
;
intValue returns [Term.Raw value]
:
| t=INTEGER { $value = Constants.Literal.integer($t.text); }
| ':' id=ident { $value = newBindVariables(id); }
| ':' id=noncol_ident { $value = newBindVariables(id); }
| QMARK { $value = newBindVariables(null); }
;
@ -1334,8 +1341,8 @@ properties[PropertyDefinitions props]
;
property[PropertyDefinitions props]
: k=ident '=' simple=propertyValue { try { $props.addProperty(k.toString(), simple); } catch (SyntaxException e) { addRecognitionError(e.getMessage()); } }
| k=ident '=' map=mapLiteral { try { $props.addProperty(k.toString(), convertPropertyMap(map)); } catch (SyntaxException e) { addRecognitionError(e.getMessage()); } }
: k=noncol_ident '=' simple=propertyValue { try { $props.addProperty(k.toString(), simple); } catch (SyntaxException e) { addRecognitionError(e.getMessage()); } }
| k=noncol_ident '=' map=mapLiteral { try { $props.addProperty(k.toString(), convertPropertyMap(map)); } catch (SyntaxException e) { addRecognitionError(e.getMessage()); } }
;
propertyValue returns [String str]
@ -1388,7 +1395,7 @@ relation[List<Relation> clauses]
inMarker returns [AbstractMarker.INRaw marker]
: QMARK { $marker = newINBindVariables(null); }
| ':' name=ident { $marker = newINBindVariables(name); }
| ':' name=noncol_ident { $marker = newINBindVariables(name); }
;
tupleOfIdentifiers returns [List<ColumnIdentifier.Raw> ids]
@ -1408,7 +1415,7 @@ tupleOfTupleLiterals returns [List<Tuples.Literal> literals]
markerForTuple returns [Tuples.Raw marker]
: QMARK { $marker = newTupleBindVariables(null); }
| ':' name=ident { $marker = newTupleBindVariables(name); }
| ':' name=noncol_ident { $marker = newTupleBindVariables(name); }
;
tupleOfMarkersForTuples returns [List<Tuples.Raw> markers]
@ -1418,7 +1425,7 @@ tupleOfMarkersForTuples returns [List<Tuples.Raw> markers]
inMarkerForTuple returns [Tuples.INRaw marker]
: QMARK { $marker = newTupleINBindVariables(null); }
| ':' name=ident { $marker = newTupleINBindVariables(name); }
| ':' name=noncol_ident { $marker = newTupleINBindVariables(name); }
;
comparatorType returns [CQL3Type.Raw t]

113
src/java/org/apache/cassandra/cql3/Lists.java
View File

@ -21,15 +21,16 @@ import static org.apache.cassandra.cql3.Constants.UNSET_VALUE;
import java.nio.ByteBuffer;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import java.util.concurrent.atomic.AtomicReference;
import com.google.common.collect.Iterators;
import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.cql3.functions.Function;
import org.apache.cassandra.db.Cell;
import org.apache.cassandra.db.ColumnFamily;
import org.apache.cassandra.db.composites.CellName;
import org.apache.cassandra.db.composites.Composite;
import org.apache.cassandra.db.*;
import org.apache.cassandra.db.rows.*;
import org.apache.cassandra.db.marshal.Int32Type;
import org.apache.cassandra.db.marshal.ListType;
import org.apache.cassandra.exceptions.InvalidRequestException;
@ -299,20 +300,37 @@ public abstract class Lists
super(column, t);
}
public void execute(ByteBuffer rowKey, ColumnFamily cf, Composite prefix, UpdateParameters params) throws InvalidRequestException
public void execute(DecoratedKey partitionKey, Clustering clustering, Row.Writer writer, UpdateParameters params) throws InvalidRequestException
{
Term.Terminal value = t.bind(params.options);
if (column.type.isMultiCell() && value != UNSET_VALUE)
{
// delete + append
CellName name = cf.getComparator().create(prefix, column);
cf.addAtom(params.makeTombstoneForOverwrite(name.slice()));
}
if (value != UNSET_VALUE)
Appender.doAppend(cf, prefix, column, params, value);
if (value == UNSET_VALUE)
return;
// delete + append
if (column.type.isMultiCell())
params.setComplexDeletionTimeForOverwrite(column, writer);
Appender.doAppend(value, clustering, writer, column, params);
}
}
private static int existingSize(Row row, ColumnDefinition column)
{
if (row == null)
return 0;
Iterator<Cell> cells = row.getCells(column);
return cells == null ? 0 : Iterators.size(cells);
}
private static Cell existingElement(Row row, ColumnDefinition column, int idx)
{
assert row != null;
Iterator<Cell> cells = row.getCells(column);
assert cells != null;
return Iterators.get(cells, idx);
}
public static class SetterByIndex extends Operation
{
private final Term idx;
@ -336,7 +354,7 @@ public abstract class Lists
idx.collectMarkerSpecification(boundNames);
}
public void execute(ByteBuffer rowKey, ColumnFamily cf, Composite prefix, UpdateParameters params) throws InvalidRequestException
public void execute(DecoratedKey partitionKey, Clustering clustering, Row.Writer writer, UpdateParameters params) throws InvalidRequestException
{
// we should not get here for frozen lists
assert column.type.isMultiCell() : "Attempted to set an individual element on a frozen list";
@ -349,17 +367,18 @@ public abstract class Lists
if (index == ByteBufferUtil.UNSET_BYTE_BUFFER)
throw new InvalidRequestException("Invalid unset value for list index");
List<Cell> existingList = params.getPrefetchedList(rowKey, column.name);
Row existingRow = params.getPrefetchedRow(partitionKey, clustering);
int existingSize = existingSize(existingRow, column);
int idx = ByteBufferUtil.toInt(index);
if (existingList == null || existingList.size() == 0)
if (existingSize == 0)
throw new InvalidRequestException("Attempted to set an element on a list which is null");
if (idx < 0 || idx >= existingList.size())
throw new InvalidRequestException(String.format("List index %d out of bound, list has size %d", idx, existingList.size()));
if (idx < 0 || idx >= existingSize)
throw new InvalidRequestException(String.format("List index %d out of bound, list has size %d", idx, existingSize));
CellName elementName = existingList.get(idx).name();
CellPath elementPath = existingElement(existingRow, column, idx).path();
if (value == null)
{
cf.addColumn(params.makeTombstone(elementName));
params.addTombstone(column, writer);
}
else if (value != ByteBufferUtil.UNSET_BYTE_BUFFER)
{
@ -369,7 +388,7 @@ public abstract class Lists
FBUtilities.MAX_UNSIGNED_SHORT,
value.remaining()));
cf.addColumn(params.makeColumn(elementName, value));
params.addCell(clustering, column, writer, elementPath, value);
}
}
}
@ -381,15 +400,14 @@ public abstract class Lists
super(column, t);
}
public void execute(ByteBuffer rowKey, ColumnFamily cf, Composite prefix, UpdateParameters params) throws InvalidRequestException
public void execute(DecoratedKey partitionKey, Clustering clustering, Row.Writer writer, UpdateParameters params) throws InvalidRequestException
{
assert column.type.isMultiCell() : "Attempted to append to a frozen list";
Term.Terminal value = t.bind(params.options);
if (value != UNSET_VALUE)
doAppend(cf, prefix, column, params, value);
doAppend(value, clustering, writer, column, params);
}
static void doAppend(ColumnFamily cf, Composite prefix, ColumnDefinition column, UpdateParameters params, Term.Terminal value) throws InvalidRequestException
static void doAppend(Term.Terminal value, Clustering clustering, Row.Writer writer, ColumnDefinition column, UpdateParameters params) throws InvalidRequestException
{
if (column.type.isMultiCell())
{
@ -401,17 +419,16 @@ public abstract class Lists
for (ByteBuffer buffer : ((Value) value).elements)
{
ByteBuffer uuid = ByteBuffer.wrap(UUIDGen.getTimeUUIDBytes());
cf.addColumn(params.makeColumn(cf.getComparator().create(prefix, column, uuid), buffer));
params.addCell(clustering, column, writer, CellPath.create(uuid), buffer);
}
}
else
{
// for frozen lists, we're overwriting the whole cell value
CellName name = cf.getComparator().create(prefix, column);
if (value == null)
cf.addAtom(params.makeTombstone(name));
params.addTombstone(column, writer);
else
cf.addColumn(params.makeColumn(name, value.get(Server.CURRENT_VERSION)));
params.addCell(clustering, column, writer, value.get(Server.CURRENT_VERSION));
}
}
}
@ -423,7 +440,7 @@ public abstract class Lists
super(column, t);
}
public void execute(ByteBuffer rowKey, ColumnFamily cf, Composite prefix, UpdateParameters params) throws InvalidRequestException
public void execute(DecoratedKey partitionKey, Clustering clustering, Row.Writer writer, UpdateParameters params) throws InvalidRequestException
{
assert column.type.isMultiCell() : "Attempted to prepend to a frozen list";
Term.Terminal value = t.bind(params.options);
@ -437,7 +454,7 @@ public abstract class Lists
{
PrecisionTime pt = PrecisionTime.getNext(time);
ByteBuffer uuid = ByteBuffer.wrap(UUIDGen.getTimeUUIDBytes(pt.millis, pt.nanos));
cf.addColumn(params.makeColumn(cf.getComparator().create(prefix, column, uuid), toAdd.get(i)));
params.addCell(clustering, column, writer, CellPath.create(uuid), toAdd.get(i));
}
}
}
@ -455,30 +472,28 @@ public abstract class Lists
return true;
}
public void execute(ByteBuffer rowKey, ColumnFamily cf, Composite prefix, UpdateParameters params) throws InvalidRequestException
public void execute(DecoratedKey partitionKey, Clustering clustering, Row.Writer writer, UpdateParameters params) throws InvalidRequestException
{
assert column.type.isMultiCell() : "Attempted to delete from a frozen list";
List<Cell> existingList = params.getPrefetchedList(rowKey, column.name);
// We want to call bind before possibly returning to reject queries where the value provided is not a list.
Term.Terminal value = t.bind(params.options);
if (existingList == null)
throw new InvalidRequestException("Attempted to delete an element from a list which is null");
if (existingList.isEmpty())
return;
if (value == null || value == UNSET_VALUE)
Row existingRow = params.getPrefetchedRow(partitionKey, clustering);
Iterator<Cell> cells = existingRow == null ? null : existingRow.getCells(column);
if (value == null || value == UNSET_VALUE || cells == null)
return;
// Note: below, we will call 'contains' on this toDiscard list for each element of existingList.
// Meaning that if toDiscard is big, converting it to a HashSet might be more efficient. However,
// the read-before-write this operation requires limits its usefulness on big lists, so in practice
// toDiscard will be small and keeping a list will be more efficient.
List<ByteBuffer> toDiscard = ((Value) value).elements;
for (Cell cell : existingList)
List<ByteBuffer> toDiscard = ((Value)value).elements;
while (cells.hasNext())
{
Cell cell = cells.next();
if (toDiscard.contains(cell.value()))
cf.addColumn(params.makeTombstone(cell.name()));
params.addTombstone(column, writer, cell.path());
}
}
}
@ -496,7 +511,7 @@ public abstract class Lists
return true;
}
public void execute(ByteBuffer rowKey, ColumnFamily cf, Composite prefix, UpdateParameters params) throws InvalidRequestException
public void execute(DecoratedKey partitionKey, Clustering clustering, Row.Writer writer, UpdateParameters params) throws InvalidRequestException
{
assert column.type.isMultiCell() : "Attempted to delete an item by index from a frozen list";
Term.Terminal index = t.bind(params.options);
@ -505,15 +520,15 @@ public abstract class Lists
if (index == Constants.UNSET_VALUE)
return;
List<Cell> existingList = params.getPrefetchedList(rowKey, column.name);
Row existingRow = params.getPrefetchedRow(partitionKey, clustering);
int existingSize = existingSize(existingRow, column);
int idx = ByteBufferUtil.toInt(index.get(params.options.getProtocolVersion()));
if (existingList == null || existingList.size() == 0)
if (existingSize == 0)
throw new InvalidRequestException("Attempted to delete an element from a list which is null");
if (idx < 0 || idx >= existingList.size())
throw new InvalidRequestException(String.format("List index %d out of bound, list has size %d", idx, existingList.size()));
if (idx < 0 || idx >= existingSize)
throw new InvalidRequestException(String.format("List index %d out of bound, list has size %d", idx, existingSize));
CellName elementName = existingList.get(idx).name();
cf.addColumn(params.makeTombstone(elementName));
params.addTombstone(column, writer, existingElement(existingRow, column, idx).path());
}
}
}

52
src/java/org/apache/cassandra/cql3/Maps.java
View File

@ -26,9 +26,9 @@ import com.google.common.collect.Iterables;
import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.cql3.functions.Function;
import org.apache.cassandra.db.ColumnFamily;
import org.apache.cassandra.db.composites.CellName;
import org.apache.cassandra.db.composites.Composite;
import org.apache.cassandra.db.Clustering;
import org.apache.cassandra.db.DecoratedKey;
import org.apache.cassandra.db.rows.*;
import org.apache.cassandra.db.marshal.MapType;
import org.apache.cassandra.exceptions.InvalidRequestException;
import org.apache.cassandra.serializers.CollectionSerializer;
@ -290,17 +290,16 @@ public abstract class Maps
super(column, t);
}
public void execute(ByteBuffer rowKey, ColumnFamily cf, Composite prefix, UpdateParameters params) throws InvalidRequestException
public void execute(DecoratedKey partitionKey, Clustering clustering, Row.Writer writer, UpdateParameters params) throws InvalidRequestException
{
Term.Terminal value = t.bind(params.options);
if (column.type.isMultiCell() && value != UNSET_VALUE)
{
// delete + put
CellName name = cf.getComparator().create(prefix, column);
cf.addAtom(params.makeTombstoneForOverwrite(name.slice()));
}
if (value != UNSET_VALUE)
Putter.doPut(cf, prefix, column, params, value);
if (value == UNSET_VALUE)
return;
// delete + put
if (column.type.isMultiCell())
params.setComplexDeletionTimeForOverwrite(column, writer);
Putter.doPut(value, clustering, writer, column, params);
}
}
@ -321,7 +320,7 @@ public abstract class Maps
k.collectMarkerSpecification(boundNames);
}
public void execute(ByteBuffer rowKey, ColumnFamily cf, Composite prefix, UpdateParameters params) throws InvalidRequestException
public void execute(DecoratedKey partitionKey, Clustering clustering, Row.Writer writer, UpdateParameters params) throws InvalidRequestException
{
assert column.type.isMultiCell() : "Attempted to set a value for a single key on a frozen map";
ByteBuffer key = k.bindAndGet(params.options);
@ -331,11 +330,11 @@ public abstract class Maps
if (key == ByteBufferUtil.UNSET_BYTE_BUFFER)
throw new InvalidRequestException("Invalid unset map key");
CellName cellName = cf.getComparator().create(prefix, column, key);
CellPath path = CellPath.create(key);
if (value == null)
{
cf.addColumn(params.makeTombstone(cellName));
params.addTombstone(column, writer, path);
}
else if (value != ByteBufferUtil.UNSET_BYTE_BUFFER)
{
@ -345,7 +344,7 @@ public abstract class Maps
FBUtilities.MAX_UNSIGNED_SHORT,
value.remaining()));
cf.addColumn(params.makeColumn(cellName, value));
params.addCell(clustering, column, writer, path, value);
}
}
}
@ -357,15 +356,15 @@ public abstract class Maps
super(column, t);
}
public void execute(ByteBuffer rowKey, ColumnFamily cf, Composite prefix, UpdateParameters params) throws InvalidRequestException
public void execute(DecoratedKey partitionKey, Clustering clustering, Row.Writer writer, UpdateParameters params) throws InvalidRequestException
{
assert column.type.isMultiCell() : "Attempted to add items to a frozen map";
Term.Terminal value = t.bind(params.options);
if (value != UNSET_VALUE)
doPut(cf, prefix, column, params, value);
doPut(value, clustering, writer, column, params);
}
static void doPut(ColumnFamily cf, Composite prefix, ColumnDefinition column, UpdateParameters params, Term.Terminal value) throws InvalidRequestException
static void doPut(Term.Terminal value, Clustering clustering, Row.Writer writer, ColumnDefinition column, UpdateParameters params) throws InvalidRequestException
{
if (column.type.isMultiCell())
{
@ -374,19 +373,15 @@ public abstract class Maps
Map<ByteBuffer, ByteBuffer> elements = ((Value) value).map;
for (Map.Entry<ByteBuffer, ByteBuffer> entry : elements.entrySet())
{
CellName cellName = cf.getComparator().create(prefix, column, entry.getKey());
cf.addColumn(params.makeColumn(cellName, entry.getValue()));
}
params.addCell(clustering, column, writer, CellPath.create(entry.getKey()), entry.getValue());
}
else
{
// for frozen maps, we're overwriting the whole cell
CellName cellName = cf.getComparator().create(prefix, column);
if (value == null)
cf.addAtom(params.makeTombstone(cellName));
params.addTombstone(column, writer);
else
cf.addColumn(params.makeColumn(cellName, value.get(Server.CURRENT_VERSION)));
params.addCell(clustering, column, writer, value.get(Server.CURRENT_VERSION));
}
}
}
@ -398,7 +393,7 @@ public abstract class Maps
super(column, k);
}
public void execute(ByteBuffer rowKey, ColumnFamily cf, Composite prefix, UpdateParameters params) throws InvalidRequestException
public void execute(DecoratedKey partitionKey, Clustering clustering, Row.Writer writer, UpdateParameters params) throws InvalidRequestException
{
assert column.type.isMultiCell() : "Attempted to delete a single key in a frozen map";
Term.Terminal key = t.bind(params.options);
@ -407,8 +402,7 @@ public abstract class Maps
if (key == Constants.UNSET_VALUE)
throw new InvalidRequestException("Invalid unset map key");
CellName cellName = cf.getComparator().create(prefix, column, key.get(params.options.getProtocolVersion()));
cf.addColumn(params.makeTombstone(cellName));
params.addTombstone(column, writer, CellPath.create(key.get(params.options.getProtocolVersion())));
}
}
}

10
src/java/org/apache/cassandra/cql3/MultiColumnRelation.java
View File

@ -131,7 +131,7 @@ public class MultiColumnRelation extends Relation
{
List<ColumnDefinition> receivers = receivers(cfm);
Term term = toTerm(receivers, getValue(), cfm.ksName, boundNames);
return new MultiColumnRestriction.EQ(receivers, term);
return new MultiColumnRestriction.EQRestriction(receivers, term);
}
@Override
@ -143,9 +143,9 @@ public class MultiColumnRelation extends Relation
if (terms == null)
{
Term term = toTerm(receivers, getValue(), cfm.ksName, boundNames);
return new MultiColumnRestriction.InWithMarker(receivers, (AbstractMarker) term);
return new MultiColumnRestriction.InRestrictionWithMarker(receivers, (AbstractMarker) term);
}
return new MultiColumnRestriction.InWithValues(receivers, terms);
return new MultiColumnRestriction.InRestrictionWithValues(receivers, terms);
}
@Override
@ -156,7 +156,7 @@ public class MultiColumnRelation extends Relation
{
List<ColumnDefinition> receivers = receivers(cfm);
Term term = toTerm(receivers(cfm), getValue(), cfm.ksName, boundNames);
return new MultiColumnRestriction.Slice(receivers, bound, inclusive, term);
return new MultiColumnRestriction.SliceRestriction(receivers, bound, inclusive, term);
}
@Override
@ -214,4 +214,4 @@ public class MultiColumnRelation extends Relation
.append(valuesOrMarker)
.toString();
}
}
}

14
src/java/org/apache/cassandra/cql3/Operation.java
View File

@ -17,13 +17,13 @@
*/
package org.apache.cassandra.cql3;
import java.nio.ByteBuffer;
import java.util.Collections;
import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.cql3.functions.Function;
import org.apache.cassandra.db.ColumnFamily;
import org.apache.cassandra.db.composites.Composite;
import org.apache.cassandra.db.Clustering;
import org.apache.cassandra.db.DecoratedKey;
import org.apache.cassandra.db.rows.Row;
import org.apache.cassandra.db.marshal.*;
import org.apache.cassandra.exceptions.InvalidRequestException;
@ -86,12 +86,12 @@ public abstract class Operation
/**
* Execute the operation.
*
* @param rowKey row key for the update.
* @param cf the column family to which to add the updates generated by this operation.
* @param prefix the prefix that identify the CQL3 row this operation applies to.
* @param partitionKey partition key for the update.
* @param clustering the clustering for the row on which the operation applies
* @param writer the row update to which to add the updates generated by this operation.
* @param params parameters of the update.
*/
public abstract void execute(ByteBuffer rowKey, ColumnFamily cf, Composite prefix, UpdateParameters params) throws InvalidRequestException;
public abstract void execute(DecoratedKey partitionKey, Clustering clustering, Row.Writer writer, UpdateParameters params) throws InvalidRequestException;
/**
* A parsed raw UPDATE operation.

70
src/java/org/apache/cassandra/cql3/Operator.java
View File

@ -20,6 +20,9 @@ package org.apache.cassandra.cql3;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import java.nio.ByteBuffer;
import org.apache.cassandra.db.marshal.AbstractType;
public enum Operator
{
@ -38,12 +41,6 @@ public enum Operator
{
return "<";
}
@Override
public Operator reverse()
{
return GT;
}
},
LTE(3)
{
@ -52,12 +49,6 @@ public enum Operator
{
return "<=";
}
@Override
public Operator reverse()
{
return GTE;
}
},
GTE(1)
{
@ -66,12 +57,6 @@ public enum Operator
{
return ">=";
}
@Override
public Operator reverse()
{
return LTE;
}
},
GT(2)
{
@ -80,12 +65,6 @@ public enum Operator
{
return ">";
}
@Override
public Operator reverse()
{
return LT;
}
},
IN(7)
{
@ -152,19 +131,42 @@ public enum Operator
throw new IOException(String.format("Cannot resolve Relation.Type from binary representation: %s", b));
}
/**
* Whether 2 values satisfy this operator (given the type they should be compared with).
*
* @throws AssertionError for IN, CONTAINS and CONTAINS_KEY as this doesn't make sense for this function.
*/
public boolean isSatisfiedBy(AbstractType<?> type, ByteBuffer leftOperand, ByteBuffer rightOperand)
{
int comparison = type.compareForCQL(leftOperand, rightOperand);
switch (this)
{
case EQ:
return comparison == 0;
case LT:
return comparison < 0;
case LTE:
return comparison <= 0;
case GT:
return comparison > 0;
case GTE:
return comparison >= 0;
case NEQ:
return comparison != 0;
default:
// we shouldn't get IN, CONTAINS, or CONTAINS KEY here
throw new AssertionError();
}
}
public int serializedSize()
{
return 4;
}
@Override
public String toString()
{
return this.name();
}
/**
* Returns the reverse operator if this one.
*
* @return the reverse operator of this one.
*/
public Operator reverse()
{
return this;
}
}

69
src/java/org/apache/cassandra/cql3/QueryProcessor.java
View File

@ -41,13 +41,14 @@ import org.apache.cassandra.cql3.functions.FunctionName;
import org.apache.cassandra.cql3.functions.Functions;
import org.apache.cassandra.cql3.statements.*;
import org.apache.cassandra.db.*;
import org.apache.cassandra.db.composites.*;
import org.apache.cassandra.db.rows.RowIterator;
import org.apache.cassandra.db.partitions.PartitionIterator;
import org.apache.cassandra.db.partitions.PartitionIterators;
import org.apache.cassandra.db.marshal.AbstractType;
import org.apache.cassandra.exceptions.*;
import org.apache.cassandra.metrics.CQLMetrics;
import org.apache.cassandra.service.*;
import org.apache.cassandra.service.pager.QueryPager;
import org.apache.cassandra.service.pager.QueryPagers;
import org.apache.cassandra.thrift.ThriftClientState;
import org.apache.cassandra.tracing.Tracing;
import org.apache.cassandra.transport.messages.ResultMessage;
@ -192,28 +193,6 @@ public class QueryProcessor implements QueryHandler
}
}
public static void validateCellNames(Iterable<CellName> cellNames, CellNameType type) throws InvalidRequestException
{
for (CellName name : cellNames)
validateCellName(name, type);
}
public static void validateCellName(CellName name, CellNameType type) throws InvalidRequestException
{
validateComposite(name, type);
if (name.isEmpty())
throw new InvalidRequestException("Invalid empty value for clustering column of COMPACT TABLE");
}
public static void validateComposite(Composite name, CType type) throws InvalidRequestException
{
long serializedSize = type.serializer().serializedSize(name, TypeSizes.NATIVE);
if (serializedSize > Cell.MAX_NAME_LENGTH)
throw new InvalidRequestException(String.format("The sum of all clustering columns is too long (%s > %s)",
serializedSize,
Cell.MAX_NAME_LENGTH));
}
public ResultMessage processStatement(CQLStatement statement, QueryState queryState, QueryOptions options)
throws RequestExecutionException, RequestValidationException
{
@ -276,6 +255,11 @@ public class QueryProcessor implements QueryHandler
}
private static QueryOptions makeInternalOptions(ParsedStatement.Prepared prepared, Object[] values)
{
return makeInternalOptions(prepared, values, ConsistencyLevel.ONE);
}
private static QueryOptions makeInternalOptions(ParsedStatement.Prepared prepared, Object[] values, ConsistencyLevel cl)
{
if (prepared.boundNames.size() != values.length)
throw new IllegalArgumentException(String.format("Invalid number of values. Expecting %d but got %d", prepared.boundNames.size(), values.length));
@ -287,7 +271,7 @@ public class QueryProcessor implements QueryHandler
AbstractType type = prepared.boundNames.get(i).type;
boundValues.add(value instanceof ByteBuffer || value == null ? (ByteBuffer)value : type.decompose(value));
}
return QueryOptions.forInternalCalls(boundValues);
return QueryOptions.forInternalCalls(cl, boundValues);
}
private static ParsedStatement.Prepared prepareInternal(String query) throws RequestValidationException
@ -313,6 +297,24 @@ public class QueryProcessor implements QueryHandler
return null;
}
public static UntypedResultSet execute(String query, ConsistencyLevel cl, QueryState state, Object... values)
throws RequestExecutionException
{
try
{
ParsedStatement.Prepared prepared = prepareInternal(query);
ResultMessage result = prepared.statement.execute(state, makeInternalOptions(prepared, values));
if (result instanceof ResultMessage.Rows)
return UntypedResultSet.create(((ResultMessage.Rows)result).result);
else
return null;
}
catch (RequestValidationException e)
{
throw new RuntimeException("Error validating " + query, e);
}
}
public static UntypedResultSet executeInternalWithPaging(String query, int pageSize, Object... values)
{
ParsedStatement.Prepared prepared = prepareInternal(query);
@ -320,7 +322,7 @@ public class QueryProcessor implements QueryHandler
throw new IllegalArgumentException("Only SELECTs can be paged");
SelectStatement select = (SelectStatement)prepared.statement;
QueryPager pager = QueryPagers.localPager(select.getPageableCommand(makeInternalOptions(prepared, values)));
QueryPager pager = select.getQuery(makeInternalOptions(prepared, values), FBUtilities.nowInSeconds()).getPager(null);
return UntypedResultSet.create(select, pager, pageSize);
}
@ -339,16 +341,19 @@ public class QueryProcessor implements QueryHandler
return null;
}
public static UntypedResultSet resultify(String query, Row row)
public static UntypedResultSet resultify(String query, RowIterator partition)
{
return resultify(query, Collections.singletonList(row));
return resultify(query, PartitionIterators.singletonIterator(partition));
}
public static UntypedResultSet resultify(String query, List<Row> rows)
public static UntypedResultSet resultify(String query, PartitionIterator partitions)
{
SelectStatement ss = (SelectStatement) getStatement(query, null).statement;
ResultSet cqlRows = ss.process(rows);
return UntypedResultSet.create(cqlRows);
try (PartitionIterator iter = partitions)
{
SelectStatement ss = (SelectStatement) getStatement(query, null).statement;
ResultSet cqlRows = ss.process(iter, FBUtilities.nowInSeconds());
return UntypedResultSet.create(cqlRows);
}
}
public ResultMessage.Prepared prepare(String query,

51
src/java/org/apache/cassandra/cql3/Sets.java
View File

@ -26,12 +26,10 @@ import com.google.common.base.Joiner;
import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.cql3.functions.Function;
import org.apache.cassandra.db.ColumnFamily;
import org.apache.cassandra.db.composites.CellName;
import org.apache.cassandra.db.composites.Composite;
import org.apache.cassandra.db.marshal.AbstractType;
import org.apache.cassandra.db.marshal.MapType;
import org.apache.cassandra.db.marshal.SetType;
import org.apache.cassandra.db.Clustering;
import org.apache.cassandra.db.DecoratedKey;
import org.apache.cassandra.db.rows.*;
import org.apache.cassandra.db.marshal.*;
import org.apache.cassandra.exceptions.InvalidRequestException;
import org.apache.cassandra.serializers.CollectionSerializer;
import org.apache.cassandra.serializers.MarshalException;
@ -259,17 +257,16 @@ public abstract class Sets
super(column, t);
}
public void execute(ByteBuffer rowKey, ColumnFamily cf, Composite prefix, UpdateParameters params) throws InvalidRequestException
public void execute(DecoratedKey partitionKey, Clustering clustering, Row.Writer writer, UpdateParameters params) throws InvalidRequestException
{
Term.Terminal value = t.bind(params.options);
if (column.type.isMultiCell() && value != UNSET_VALUE)
{
// delete + add
CellName name = cf.getComparator().create(prefix, column);
cf.addAtom(params.makeTombstoneForOverwrite(name.slice()));
}
if (value != UNSET_VALUE)
Adder.doAdd(cf, prefix, column, params, value);
if (value == UNSET_VALUE)
return;
// delete + add
if (column.type.isMultiCell())
params.setComplexDeletionTimeForOverwrite(column, writer);
Adder.doAdd(value, clustering, writer, column, params);
}
}
@ -280,15 +277,15 @@ public abstract class Sets
super(column, t);
}
public void execute(ByteBuffer rowKey, ColumnFamily cf, Composite prefix, UpdateParameters params) throws InvalidRequestException
public void execute(DecoratedKey partitionKey, Clustering clustering, Row.Writer writer, UpdateParameters params) throws InvalidRequestException
{
assert column.type.isMultiCell() : "Attempted to add items to a frozen set";
Term.Terminal value = t.bind(params.options);
if (value != UNSET_VALUE)
doAdd(cf, prefix, column, params, value);
doAdd(value, clustering, writer, column, params);
}
static void doAdd(ColumnFamily cf, Composite prefix, ColumnDefinition column, UpdateParameters params, Term.Terminal value) throws InvalidRequestException
static void doAdd(Term.Terminal value, Clustering clustering, Row.Writer writer, ColumnDefinition column, UpdateParameters params) throws InvalidRequestException
{
if (column.type.isMultiCell())
{
@ -299,18 +296,17 @@ public abstract class Sets
{
if (bb == ByteBufferUtil.UNSET_BYTE_BUFFER)
continue;
CellName cellName = cf.getComparator().create(prefix, column, bb);
cf.addColumn(params.makeColumn(cellName, ByteBufferUtil.EMPTY_BYTE_BUFFER));
params.addCell(clustering, column, writer, CellPath.create(bb), ByteBufferUtil.EMPTY_BYTE_BUFFER);
}
}
else
{
// for frozen sets, we're overwriting the whole cell
CellName cellName = cf.getComparator().create(prefix, column);
if (value == null)
cf.addAtom(params.makeTombstone(cellName));
params.addTombstone(column, writer);
else
cf.addColumn(params.makeColumn(cellName, value.get(Server.CURRENT_VERSION)));
params.addCell(clustering, column, writer, value.get(Server.CURRENT_VERSION));
}
}
}
@ -323,7 +319,7 @@ public abstract class Sets
super(column, t);
}
public void execute(ByteBuffer rowKey, ColumnFamily cf, Composite prefix, UpdateParameters params) throws InvalidRequestException
public void execute(DecoratedKey partitionKey, Clustering clustering, Row.Writer writer, UpdateParameters params) throws InvalidRequestException
{
assert column.type.isMultiCell() : "Attempted to remove items from a frozen set";
@ -337,7 +333,7 @@ public abstract class Sets
: Collections.singleton(value.get(params.options.getProtocolVersion()));
for (ByteBuffer bb : toDiscard)
cf.addColumn(params.makeTombstone(cf.getComparator().create(prefix, column, bb)));
params.addTombstone(column, writer, CellPath.create(bb));
}
}
@ -348,15 +344,14 @@ public abstract class Sets
super(column, k);
}
public void execute(ByteBuffer rowKey, ColumnFamily cf, Composite prefix, UpdateParameters params) throws InvalidRequestException
public void execute(DecoratedKey partitionKey, Clustering clustering, Row.Writer writer, UpdateParameters params) throws InvalidRequestException
{
assert column.type.isMultiCell() : "Attempted to delete a single element in a frozen set";
Term.Terminal elt = t.bind(params.options);
if (elt == null)
throw new InvalidRequestException("Invalid null set element");
CellName cellName = cf.getComparator().create(prefix, column, elt.get(params.options.getProtocolVersion()));
cf.addColumn(params.makeTombstone(cellName));
params.addTombstone(column, writer, CellPath.create(elt.get(params.options.getProtocolVersion())));
}
}
}

27
src/java/org/apache/cassandra/cql3/SingleColumnRelation.java
View File

@ -140,13 +140,13 @@ public final class SingleColumnRelation extends Relation
ColumnDefinition columnDef = toColumnDefinition(cfm, entity);
if (mapKey == null)
{
Term term = toTerm(toReceivers(columnDef), value, cfm.ksName, boundNames);
return new SingleColumnRestriction.EQ(columnDef, term);
Term term = toTerm(toReceivers(columnDef, cfm.isDense()), value, cfm.ksName, boundNames);
return new SingleColumnRestriction.EQRestriction(columnDef, term);
}
List<? extends ColumnSpecification> receivers = toReceivers(columnDef);
List<? extends ColumnSpecification> receivers = toReceivers(columnDef, cfm.isDense());
Term entryKey = toTerm(Collections.singletonList(receivers.get(0)), mapKey, cfm.ksName, boundNames);
Term entryValue = toTerm(Collections.singletonList(receivers.get(1)), value, cfm.ksName, boundNames);
return new SingleColumnRestriction.Contains(columnDef, entryKey, entryValue);
return new SingleColumnRestriction.ContainsRestriction(columnDef, entryKey, entryValue);
}
@Override
@ -154,14 +154,14 @@ public final class SingleColumnRelation extends Relation
VariableSpecifications boundNames) throws InvalidRequestException
{
ColumnDefinition columnDef = cfm.getColumnDefinition(getEntity().prepare(cfm));
List<? extends ColumnSpecification> receivers = toReceivers(columnDef);
List<? extends ColumnSpecification> receivers = toReceivers(columnDef, cfm.isDense());
List<Term> terms = toTerms(receivers, inValues, cfm.ksName, boundNames);
if (terms == null)
{
Term term = toTerm(receivers, value, cfm.ksName, boundNames);
return new SingleColumnRestriction.InWithMarker(columnDef, (Lists.Marker) term);
return new SingleColumnRestriction.InRestrictionWithMarker(columnDef, (Lists.Marker) term);
}
return new SingleColumnRestriction.InWithValues(columnDef, terms);
return new SingleColumnRestriction.InRestrictionWithValues(columnDef, terms);
}
@Override
@ -171,8 +171,8 @@ public final class SingleColumnRelation extends Relation
boolean inclusive) throws InvalidRequestException
{
ColumnDefinition columnDef = toColumnDefinition(cfm, entity);
Term term = toTerm(toReceivers(columnDef), value, cfm.ksName, boundNames);
return new SingleColumnRestriction.Slice(columnDef, bound, inclusive, term);
Term term = toTerm(toReceivers(columnDef, cfm.isDense()), value, cfm.ksName, boundNames);
return new SingleColumnRestriction.SliceRestriction(columnDef, bound, inclusive, term);
}
@Override
@ -181,22 +181,23 @@ public final class SingleColumnRelation extends Relation
boolean isKey) throws InvalidRequestException
{
ColumnDefinition columnDef = toColumnDefinition(cfm, entity);
Term term = toTerm(toReceivers(columnDef), value, cfm.ksName, boundNames);
return new SingleColumnRestriction.Contains(columnDef, term, isKey);
Term term = toTerm(toReceivers(columnDef, cfm.isDense()), value, cfm.ksName, boundNames);
return new SingleColumnRestriction.ContainsRestriction(columnDef, term, isKey);
}
/**
* Returns the receivers for this relation.
* @param columnDef the column definition
* @param isDense whether the table is a dense one
*
* @return the receivers for the specified relation.
* @throws InvalidRequestException if the relation is invalid
*/
private List<? extends ColumnSpecification> toReceivers(ColumnDefinition columnDef) throws InvalidRequestException
private List<? extends ColumnSpecification> toReceivers(ColumnDefinition columnDef, boolean isDense) throws InvalidRequestException
{
ColumnSpecification receiver = columnDef;
checkFalse(columnDef.isCompactValue(),
checkFalse(!columnDef.isPrimaryKeyColumn() && isDense,
"Predicates on the non-primary-key column (%s) of a COMPACT table are not yet supported",
columnDef.name);

4
src/java/org/apache/cassandra/cql3/TokenRelation.java
View File

@ -69,7 +69,7 @@ public final class TokenRelation extends Relation
{
List<ColumnDefinition> columnDefs = getColumnDefinitions(cfm);
Term term = toTerm(toReceivers(cfm, columnDefs), value, cfm.ksName, boundNames);
return new TokenRestriction.EQ(cfm.getKeyValidatorAsCType(), columnDefs, term);
return new TokenRestriction.EQRestriction(cfm.getKeyValidatorAsClusteringComparator(), columnDefs, term);
}
@Override
@ -86,7 +86,7 @@ public final class TokenRelation extends Relation
{
List<ColumnDefinition> columnDefs = getColumnDefinitions(cfm);
Term term = toTerm(toReceivers(cfm, columnDefs), value, cfm.ksName, boundNames);
return new TokenRestriction.Slice(cfm.getKeyValidatorAsCType(), columnDefs, bound, inclusive, term);
return new TokenRestriction.SliceRestriction(cfm.getKeyValidatorAsClusteringComparator(), columnDefs, bound, inclusive, term);
}
@Override

45
src/java/org/apache/cassandra/cql3/UntypedResultSet.java
View File

@ -24,9 +24,16 @@ import java.util.*;
import com.google.common.collect.AbstractIterator;
import org.apache.cassandra.config.CFMetaData;
import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.cql3.statements.SelectStatement;
import org.apache.cassandra.db.*;
import org.apache.cassandra.db.rows.*;
import org.apache.cassandra.db.partitions.PartitionIterator;
import org.apache.cassandra.db.marshal.*;
import org.apache.cassandra.service.pager.QueryPager;
import org.apache.cassandra.transport.Server;
import org.apache.cassandra.utils.FBUtilities;
/** a utility for doing internal cql-based queries */
public abstract class UntypedResultSet implements Iterable<UntypedResultSet.Row>
@ -174,11 +181,16 @@ public abstract class UntypedResultSet implements Iterable<UntypedResultSet.Row>
protected Row computeNext()
{
int nowInSec = FBUtilities.nowInSeconds();
while (currentPage == null || !currentPage.hasNext())
{
if (pager.isExhausted())
return endOfData();
currentPage = select.process(pager.fetchPage(pageSize)).rows.iterator();
try (ReadOrderGroup orderGroup = pager.startOrderGroup(); PartitionIterator iter = pager.fetchPageInternal(pageSize, orderGroup))
{
currentPage = select.process(iter, nowInSec).rows.iterator();
}
}
return new Row(metadata, currentPage.next());
}
@ -208,6 +220,37 @@ public abstract class UntypedResultSet implements Iterable<UntypedResultSet.Row>
data.put(names.get(i).name.toString(), columns.get(i));
}
public static Row fromInternalRow(CFMetaData metadata, DecoratedKey key, org.apache.cassandra.db.rows.Row row)
{
Map<String, ByteBuffer> data = new HashMap<>();
ByteBuffer[] keyComponents = SelectStatement.getComponents(metadata, key);
for (ColumnDefinition def : metadata.partitionKeyColumns())
data.put(def.name.toString(), keyComponents[def.position()]);
Clustering clustering = row.clustering();
for (ColumnDefinition def : metadata.clusteringColumns())
data.put(def.name.toString(), clustering.get(def.position()));
for (ColumnDefinition def : metadata.partitionColumns())
{
if (def.isComplex())
{
Iterator<Cell> cells = row.getCells(def);
if (cells != null)
data.put(def.name.toString(), ((CollectionType)def.type).serializeForNativeProtocol(def, cells, Server.VERSION_3));
}
else
{
Cell cell = row.getCell(def);
if (cell != null)
data.put(def.name.toString(), cell.value());
}
}
return new Row(data);
}
public boolean has(String column)
{
// Note that containsKey won't work because we may have null values

160
src/java/org/apache/cassandra/cql3/UpdateParameters.java
View File

@ -18,15 +18,18 @@
package org.apache.cassandra.cql3;
import java.nio.ByteBuffer;
import java.util.Collections;
import java.util.List;
import java.util.Map;
import org.apache.cassandra.config.CFMetaData;
import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.db.*;
import org.apache.cassandra.db.composites.CellName;
import org.apache.cassandra.db.filter.ColumnSlice;
import org.apache.cassandra.db.filter.ColumnFilter;
import org.apache.cassandra.db.rows.*;
import org.apache.cassandra.db.context.CounterContext;
import org.apache.cassandra.db.index.SecondaryIndexManager;
import org.apache.cassandra.db.partitions.*;
import org.apache.cassandra.exceptions.InvalidRequestException;
import org.apache.cassandra.utils.ByteBufferUtil;
import org.apache.cassandra.utils.FBUtilities;
/**
@ -36,22 +39,39 @@ public class UpdateParameters
{
public final CFMetaData metadata;
public final QueryOptions options;
public final long timestamp;
private final int ttl;
public final int localDeletionTime;
private final LivenessInfo defaultLiveness;
private final LivenessInfo deletionLiveness;
private final DeletionTime deletionTime;
private final SecondaryIndexManager indexManager;
// For lists operation that require a read-before-write. Will be null otherwise.
private final Map<ByteBuffer, CQL3Row> prefetchedLists;
private final Map<DecoratedKey, Partition> prefetchedRows;
public UpdateParameters(CFMetaData metadata, QueryOptions options, long timestamp, int ttl, Map<ByteBuffer, CQL3Row> prefetchedLists)
public UpdateParameters(CFMetaData metadata, QueryOptions options, long timestamp, int ttl, Map<DecoratedKey, Partition> prefetchedRows, boolean validateIndexedColumns)
throws InvalidRequestException
{
this.metadata = metadata;
this.options = options;
this.timestamp = timestamp;
this.ttl = ttl;
this.localDeletionTime = (int)(System.currentTimeMillis() / 1000);
this.prefetchedLists = prefetchedLists;
int nowInSec = FBUtilities.nowInSeconds();
this.defaultLiveness = SimpleLivenessInfo.forUpdate(timestamp, ttl, nowInSec, metadata);
this.deletionLiveness = SimpleLivenessInfo.forDeletion(timestamp, nowInSec);
this.deletionTime = new SimpleDeletionTime(timestamp, nowInSec);
this.prefetchedRows = prefetchedRows;
// Index column validation triggers a call to Keyspace.open() which we want to be able to avoid in some case (e.g. when using CQLSSTableWriter)
if (validateIndexedColumns)
{
SecondaryIndexManager manager = Keyspace.openAndGetStore(metadata).indexManager;
indexManager = manager.hasIndexes() ? manager : null;
}
else
{
indexManager = null;
}
// We use MIN_VALUE internally to mean the absence of of timestamp (in Selection, in sstable stats, ...), so exclude
// it to avoid potential confusion.
@ -59,48 +79,106 @@ public class UpdateParameters
throw new InvalidRequestException(String.format("Out of bound timestamp, must be in [%d, %d]", Long.MIN_VALUE + 1, Long.MAX_VALUE));
}
public Cell makeColumn(CellName name, ByteBuffer value) throws InvalidRequestException
public void newPartition(DecoratedKey partitionKey) throws InvalidRequestException
{
QueryProcessor.validateCellName(name, metadata.comparator);
return AbstractCell.create(name, value, timestamp, ttl, metadata);
if (indexManager != null)
indexManager.validate(partitionKey);
}
public Cell makeCounter(CellName name, long delta) throws InvalidRequestException
{
QueryProcessor.validateCellName(name, metadata.comparator);
return new BufferCounterUpdateCell(name, delta, FBUtilities.timestampMicros());
}
public Cell makeTombstone(CellName name) throws InvalidRequestException
public void writeClustering(Clustering clustering, Row.Writer writer) throws InvalidRequestException
{
QueryProcessor.validateCellName(name, metadata.comparator);
return new BufferDeletedCell(name, localDeletionTime, timestamp);
if (indexManager != null)
indexManager.validate(clustering);
if (metadata.isDense() && !metadata.isCompound())
{
// If it's a COMPACT STORAGE table with a single clustering column, the clustering value is
// translated in Thrift to the full Thrift column name, and for backward compatibility we
// don't want to allow that to be empty (even though this would be fine for the storage engine).
assert clustering.size() == 1;
ByteBuffer value = clustering.get(0);
if (value == null || !value.hasRemaining())
throw new InvalidRequestException("Invalid empty or null value for column " + metadata.clusteringColumns().get(0).name);
}
Rows.writeClustering(clustering, writer);
}
public RangeTombstone makeRangeTombstone(ColumnSlice slice) throws InvalidRequestException
public void writePartitionKeyLivenessInfo(Row.Writer writer)
{
QueryProcessor.validateComposite(slice.start, metadata.comparator);
QueryProcessor.validateComposite(slice.finish, metadata.comparator);
return new RangeTombstone(slice.start, slice.finish, timestamp, localDeletionTime);
writer.writePartitionKeyLivenessInfo(defaultLiveness);
}
public RangeTombstone makeTombstoneForOverwrite(ColumnSlice slice) throws InvalidRequestException
public void writeRowDeletion(Row.Writer writer)
{
QueryProcessor.validateComposite(slice.start, metadata.comparator);
QueryProcessor.validateComposite(slice.finish, metadata.comparator);
return new RangeTombstone(slice.start, slice.finish, timestamp - 1, localDeletionTime);
writer.writeRowDeletion(deletionTime);
}
public List<Cell> getPrefetchedList(ByteBuffer rowKey, ColumnIdentifier cql3ColumnName)
public void addTombstone(ColumnDefinition column, Row.Writer writer) throws InvalidRequestException
{
if (prefetchedLists == null)
return Collections.emptyList();
addTombstone(column, writer, null);
}
CQL3Row row = prefetchedLists.get(rowKey);
if (row == null)
return Collections.<Cell>emptyList();
public void addTombstone(ColumnDefinition column, Row.Writer writer, CellPath path) throws InvalidRequestException
{
writer.writeCell(column, false, ByteBufferUtil.EMPTY_BYTE_BUFFER, deletionLiveness, path);
}
List<Cell> cql3List = row.getMultiCellColumn(cql3ColumnName);
return (cql3List == null) ? Collections.<Cell>emptyList() : cql3List;
public void addCell(Clustering clustering, ColumnDefinition column, Row.Writer writer, ByteBuffer value) throws InvalidRequestException
{
addCell(clustering, column, writer, null, value);
}
public void addCell(Clustering clustering, ColumnDefinition column, Row.Writer writer, CellPath path, ByteBuffer value) throws InvalidRequestException
{
if (indexManager != null)
indexManager.validate(column, value, path);
writer.writeCell(column, false, value, defaultLiveness, path);
}
public void addCounter(ColumnDefinition column, Row.Writer writer, long increment) throws InvalidRequestException
{
assert defaultLiveness.ttl() == LivenessInfo.NO_TTL;
// In practice, the actual CounterId (and clock really) that we use doesn't matter, because we will
// actually ignore it in CounterMutation when we do the read-before-write to create the actual value
// that is applied. In other words, this is not the actual value that will be written to the memtable
// because this will be replaced in CounterMutation.updateWithCurrentValue().
// As an aside, since we don't care about the CounterId/clock, we used to only send the incremement,
// but that makes things a bit more complex as this means we need to be able to distinguish inside
// PartitionUpdate between counter updates that has been processed by CounterMutation and those that
// haven't.
ByteBuffer value = CounterContext.instance().createLocal(increment);
writer.writeCell(column, true, value, defaultLiveness, null);
}
public void setComplexDeletionTime(ColumnDefinition column, Row.Writer writer)
{
writer.writeComplexDeletion(column, deletionTime);
}
public void setComplexDeletionTimeForOverwrite(ColumnDefinition column, Row.Writer writer)
{
writer.writeComplexDeletion(column, new SimpleDeletionTime(deletionTime.markedForDeleteAt() - 1, deletionTime.localDeletionTime()));
}
public DeletionTime deletionTime()
{
return deletionTime;
}
public RangeTombstone makeRangeTombstone(CBuilder cbuilder)
{
return new RangeTombstone(cbuilder.buildSlice(), deletionTime);
}
public Row getPrefetchedRow(DecoratedKey key, Clustering clustering)
{
if (prefetchedRows == null)
return null;
Partition partition = prefetchedRows.get(key);
return partition == null ? null : partition.searchIterator(ColumnFilter.selection(partition.columns()), false).next(clustering);
}
}

7
src/java/org/apache/cassandra/cql3/functions/TokenFct.java
View File

@ -22,7 +22,8 @@ import java.util.List;
import org.apache.cassandra.config.CFMetaData;
import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.db.composites.CBuilder;
import org.apache.cassandra.cql3.statements.SelectStatement;
import org.apache.cassandra.db.CBuilder;
import org.apache.cassandra.db.marshal.AbstractType;
import org.apache.cassandra.dht.IPartitioner;
import org.apache.cassandra.exceptions.InvalidRequestException;
@ -52,7 +53,7 @@ public class TokenFct extends NativeScalarFunction
public ByteBuffer execute(int protocolVersion, List<ByteBuffer> parameters) throws InvalidRequestException
{
CBuilder builder = cfm.getKeyValidatorAsCType().builder();
CBuilder builder = CBuilder.create(cfm.getKeyValidatorAsClusteringComparator());
for (int i = 0; i < parameters.size(); i++)
{
ByteBuffer bb = parameters.get(i);
@ -60,6 +61,6 @@ public class TokenFct extends NativeScalarFunction
return null;
builder.add(bb);
}
return partitioner.getTokenFactory().toByteArray(partitioner.getToken(builder.build().toByteBuffer()));
return partitioner.getTokenFactory().toByteArray(partitioner.getToken(CFMetaData.serializePartitionKey(builder.build())));
}
}

11
src/java/org/apache/cassandra/cql3/restrictions/AbstractPrimaryKeyRestrictions.java
View File

@ -18,11 +18,12 @@
package org.apache.cassandra.cql3.restrictions;
import java.nio.ByteBuffer;
import java.util.List;
import java.util.*;
import org.apache.cassandra.cql3.QueryOptions;
import org.apache.cassandra.cql3.statements.Bound;
import org.apache.cassandra.db.composites.CType;
import org.apache.cassandra.db.ClusteringPrefix;
import org.apache.cassandra.db.ClusteringComparator;
import org.apache.cassandra.exceptions.InvalidRequestException;
/**
@ -33,11 +34,11 @@ abstract class AbstractPrimaryKeyRestrictions extends AbstractRestriction implem
/**
* The composite type.
*/
protected final CType ctype;
protected final ClusteringComparator comparator;
public AbstractPrimaryKeyRestrictions(CType ctype)
public AbstractPrimaryKeyRestrictions(ClusteringComparator comparator)
{
this.ctype = ctype;
this.comparator = comparator;
}
@Override

14
src/java/org/apache/cassandra/cql3/restrictions/AbstractRestriction.java
View File

@ -22,7 +22,7 @@ import java.nio.ByteBuffer;
import org.apache.cassandra.cql3.ColumnSpecification;
import org.apache.cassandra.cql3.QueryOptions;
import org.apache.cassandra.cql3.statements.Bound;
import org.apache.cassandra.db.composites.CompositesBuilder;
import org.apache.cassandra.db.MultiCBuilder;
import org.apache.cassandra.exceptions.InvalidRequestException;
import static org.apache.cassandra.cql3.statements.RequestValidations.checkBindValueSet;
@ -77,7 +77,7 @@ abstract class AbstractRestriction implements Restriction
}
@Override
public CompositesBuilder appendBoundTo(CompositesBuilder builder, Bound bound, QueryOptions options)
public MultiCBuilder appendBoundTo(MultiCBuilder builder, Bound bound, QueryOptions options)
{
return appendTo(builder, options);
}
@ -87,14 +87,4 @@ abstract class AbstractRestriction implements Restriction
{
return true;
}
protected static ByteBuffer validateIndexedValue(ColumnSpecification columnSpec,
ByteBuffer value)
throws InvalidRequestException
{
checkNotNull(value, "Unsupported null value for indexed column %s", columnSpec.name);
checkBindValueSet(value, "Unsupported unset value for indexed column %s", columnSpec.name);
checkFalse(value.remaining() > 0xFFFF, "Index expression values may not be larger than 64K");
return value;
}
}

27
src/java/org/apache/cassandra/cql3/restrictions/ForwardingPrimaryKeyRestrictions.java
View File

@ -18,16 +18,15 @@
package org.apache.cassandra.cql3.restrictions;
import java.nio.ByteBuffer;
import java.util.Collection;
import java.util.List;
import java.util.*;
import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.cql3.QueryOptions;
import org.apache.cassandra.cql3.functions.Function;
import org.apache.cassandra.cql3.statements.Bound;
import org.apache.cassandra.db.IndexExpression;
import org.apache.cassandra.db.composites.Composite;
import org.apache.cassandra.db.composites.CompositesBuilder;
import org.apache.cassandra.db.*;
import org.apache.cassandra.db.Slice;
import org.apache.cassandra.db.filter.RowFilter;
import org.apache.cassandra.db.index.SecondaryIndexManager;
import org.apache.cassandra.exceptions.InvalidRequestException;
@ -87,15 +86,15 @@ abstract class ForwardingPrimaryKeyRestrictions implements PrimaryKeyRestriction
}
@Override
public CompositesBuilder appendTo(CompositesBuilder builder, QueryOptions options)
public MultiCBuilder appendTo(MultiCBuilder builder, QueryOptions options)
{
return getDelegate().appendTo(builder, options);
}
@Override
public List<Composite> valuesAsComposites(QueryOptions options) throws InvalidRequestException
public NavigableSet<Clustering> valuesAsClustering(QueryOptions options) throws InvalidRequestException
{
return getDelegate().valuesAsComposites(options);
return getDelegate().valuesAsClustering(options);
}
@Override
@ -105,13 +104,13 @@ abstract class ForwardingPrimaryKeyRestrictions implements PrimaryKeyRestriction
}
@Override
public List<Composite> boundsAsComposites(Bound bound, QueryOptions options) throws InvalidRequestException
public SortedSet<Slice.Bound> boundsAsClustering(Bound bound, QueryOptions options) throws InvalidRequestException
{
return getDelegate().boundsAsComposites(bound, options);
return getDelegate().boundsAsClustering(bound, options);
}
@Override
public CompositesBuilder appendBoundTo(CompositesBuilder builder, Bound bound, QueryOptions options)
public MultiCBuilder appendBoundTo(MultiCBuilder builder, Bound bound, QueryOptions options)
{
return getDelegate().appendBoundTo(builder, bound, options);
}
@ -177,10 +176,8 @@ abstract class ForwardingPrimaryKeyRestrictions implements PrimaryKeyRestriction
}
@Override
public void addIndexExpressionTo(List<IndexExpression> expressions,
SecondaryIndexManager indexManager,
QueryOptions options) throws InvalidRequestException
public void addRowFilterTo(RowFilter filter, SecondaryIndexManager indexManager, QueryOptions options) throws InvalidRequestException
{
getDelegate().addIndexExpressionTo(expressions, indexManager, options);
getDelegate().addRowFilterTo(filter, indexManager, options);
}
}

70
src/java/org/apache/cassandra/cql3/restrictions/MultiColumnRestriction.java
View File

@ -24,8 +24,8 @@ import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.cql3.*;
import org.apache.cassandra.cql3.functions.Function;
import org.apache.cassandra.cql3.statements.Bound;
import org.apache.cassandra.db.IndexExpression;
import org.apache.cassandra.db.composites.CompositesBuilder;
import org.apache.cassandra.db.*;
import org.apache.cassandra.db.filter.RowFilter;
import org.apache.cassandra.db.index.SecondaryIndex;
import org.apache.cassandra.db.index.SecondaryIndexManager;
import org.apache.cassandra.exceptions.InvalidRequestException;
@ -108,7 +108,7 @@ public abstract class MultiColumnRestriction extends AbstractRestriction
{
for (ColumnDefinition columnDef : columnDefs)
{
SecondaryIndex index = indexManager.getIndexForColumn(columnDef.name.bytes);
SecondaryIndex index = indexManager.getIndexForColumn(columnDef);
if (index != null && isSupportedBy(index))
return true;
}
@ -124,11 +124,11 @@ public abstract class MultiColumnRestriction extends AbstractRestriction
*/
protected abstract boolean isSupportedBy(SecondaryIndex index);
public static class EQ extends MultiColumnRestriction
public static class EQRestriction extends MultiColumnRestriction
{
protected final Term value;
public EQ(List<ColumnDefinition> columnDefs, Term value)
public EQRestriction(List<ColumnDefinition> columnDefs, Term value)
{
super(columnDefs);
this.value = value;
@ -160,7 +160,7 @@ public abstract class MultiColumnRestriction extends AbstractRestriction
}
@Override
public CompositesBuilder appendTo(CompositesBuilder builder, QueryOptions options)
public MultiCBuilder appendTo(MultiCBuilder builder, QueryOptions options)
{
Tuples.Value t = ((Tuples.Value) value.bind(options));
List<ByteBuffer> values = t.getElements();
@ -173,9 +173,7 @@ public abstract class MultiColumnRestriction extends AbstractRestriction
}
@Override
public final void addIndexExpressionTo(List<IndexExpression> expressions,
SecondaryIndexManager indexManager,
QueryOptions options) throws InvalidRequestException
public final void addRowFilterTo(RowFilter filter, SecondaryIndexManager indexMananger, QueryOptions options) throws InvalidRequestException
{
Tuples.Value t = ((Tuples.Value) value.bind(options));
List<ByteBuffer> values = t.getElements();
@ -183,19 +181,23 @@ public abstract class MultiColumnRestriction extends AbstractRestriction
for (int i = 0, m = columnDefs.size(); i < m; i++)
{
ColumnDefinition columnDef = columnDefs.get(i);
ByteBuffer component = validateIndexedValue(columnDef, values.get(i));
expressions.add(new IndexExpression(columnDef.name.bytes, Operator.EQ, component));
filter.add(columnDef, Operator.EQ, values.get(i));
}
}
}
public abstract static class IN extends MultiColumnRestriction
public abstract static class INRestriction extends MultiColumnRestriction
{
public INRestriction(List<ColumnDefinition> columnDefs)
{
super(columnDefs);
}
/**
* {@inheritDoc}
*/
@Override
public CompositesBuilder appendTo(CompositesBuilder builder, QueryOptions options)
public MultiCBuilder appendTo(MultiCBuilder builder, QueryOptions options)
{
List<List<ByteBuffer>> splitInValues = splitValues(options);
builder.addAllElementsToAll(splitInValues);
@ -205,11 +207,6 @@ public abstract class MultiColumnRestriction extends AbstractRestriction
return builder;
}
public IN(List<ColumnDefinition> columnDefs)
{
super(columnDefs);
}
@Override
public boolean isIN()
{
@ -230,9 +227,9 @@ public abstract class MultiColumnRestriction extends AbstractRestriction
}
@Override
public final void addIndexExpressionTo(List<IndexExpression> expressions,
SecondaryIndexManager indexManager,
QueryOptions options) throws InvalidRequestException
public final void addRowFilterTo(RowFilter filter,
SecondaryIndexManager indexManager,
QueryOptions options) throws InvalidRequestException
{
List<List<ByteBuffer>> splitInValues = splitValues(options);
checkTrue(splitInValues.size() == 1, "IN restrictions are not supported on indexed columns");
@ -241,8 +238,7 @@ public abstract class MultiColumnRestriction extends AbstractRestriction
for (int i = 0, m = columnDefs.size(); i < m; i++)
{
ColumnDefinition columnDef = columnDefs.get(i);
ByteBuffer component = validateIndexedValue(columnDef, values.get(i));
expressions.add(new IndexExpression(columnDef.name.bytes, Operator.EQ, component));
filter.add(columnDef, Operator.EQ, values.get(i));
}
}
@ -253,11 +249,11 @@ public abstract class MultiColumnRestriction extends AbstractRestriction
* An IN restriction that has a set of terms for in values.
* For example: "SELECT ... WHERE (a, b, c) IN ((1, 2, 3), (4, 5, 6))" or "WHERE (a, b, c) IN (?, ?)"
*/
public static class InWithValues extends MultiColumnRestriction.IN
public static class InRestrictionWithValues extends INRestriction
{
protected final List<Term> values;
public InWithValues(List<ColumnDefinition> columnDefs, List<Term> values)
public InRestrictionWithValues(List<ColumnDefinition> columnDefs, List<Term> values)
{
super(columnDefs);
this.values = values;
@ -292,11 +288,11 @@ public abstract class MultiColumnRestriction extends AbstractRestriction
* An IN restriction that uses a single marker for a set of IN values that are tuples.
* For example: "SELECT ... WHERE (a, b, c) IN ?"
*/
public static class InWithMarker extends MultiColumnRestriction.IN
public static class InRestrictionWithMarker extends INRestriction
{
protected final AbstractMarker marker;
public InWithMarker(List<ColumnDefinition> columnDefs, AbstractMarker marker)
public InRestrictionWithMarker(List<ColumnDefinition> columnDefs, AbstractMarker marker)
{
super(columnDefs);
this.marker = marker;
@ -324,16 +320,16 @@ public abstract class MultiColumnRestriction extends AbstractRestriction
}
}
public static class Slice extends MultiColumnRestriction
public static class SliceRestriction extends MultiColumnRestriction
{
private final TermSlice slice;
public Slice(List<ColumnDefinition> columnDefs, Bound bound, boolean inclusive, Term term)
public SliceRestriction(List<ColumnDefinition> columnDefs, Bound bound, boolean inclusive, Term term)
{
this(columnDefs, TermSlice.newInstance(bound, inclusive, term));
}
private Slice(List<ColumnDefinition> columnDefs, TermSlice slice)
private SliceRestriction(List<ColumnDefinition> columnDefs, TermSlice slice)
{
super(columnDefs);
this.slice = slice;
@ -346,13 +342,13 @@ public abstract class MultiColumnRestriction extends AbstractRestriction
}
@Override
public CompositesBuilder appendTo(CompositesBuilder builder, QueryOptions options)
public MultiCBuilder appendTo(MultiCBuilder builder, QueryOptions options)
{
throw new UnsupportedOperationException();
}
@Override
public CompositesBuilder appendBoundTo(CompositesBuilder builder, Bound bound, QueryOptions options)
public MultiCBuilder appendBoundTo(MultiCBuilder builder, Bound bound, QueryOptions options)
{
List<ByteBuffer> vals = componentBounds(bound, options);
@ -395,7 +391,7 @@ public abstract class MultiColumnRestriction extends AbstractRestriction
"Column \"%s\" cannot be restricted by both an equality and an inequality relation",
getColumnsInCommons(otherRestriction));
Slice otherSlice = (Slice) otherRestriction;
SliceRestriction otherSlice = (SliceRestriction) otherRestriction;
if (!getFirstColumn().equals(otherRestriction.getFirstColumn()))
{
@ -414,13 +410,13 @@ public abstract class MultiColumnRestriction extends AbstractRestriction
getColumnsInCommons(otherRestriction));
List<ColumnDefinition> newColumnDefs = columnDefs.size() >= otherSlice.columnDefs.size() ? columnDefs : otherSlice.columnDefs;
return new Slice(newColumnDefs, slice.merge(otherSlice.slice));
return new SliceRestriction(newColumnDefs, slice.merge(otherSlice.slice));
}
@Override
public final void addIndexExpressionTo(List<IndexExpression> expressions,
SecondaryIndexManager indexManager,
QueryOptions options) throws InvalidRequestException
public final void addRowFilterTo(RowFilter filter,
SecondaryIndexManager indexManager,
QueryOptions options) throws InvalidRequestException
{
throw invalidRequest("Slice restrictions are not supported on indexed columns");
}

104
src/java/org/apache/cassandra/cql3/restrictions/PrimaryKeyRestrictionSet.java
View File

@ -18,19 +18,18 @@
package org.apache.cassandra.cql3.restrictions;
import java.nio.ByteBuffer;
import java.util.Collection;
import java.util.Collections;
import java.util.List;
import java.util.*;
import org.apache.cassandra.config.CFMetaData;
import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.cql3.QueryOptions;
import org.apache.cassandra.cql3.functions.Function;
import org.apache.cassandra.cql3.statements.Bound;
import org.apache.cassandra.db.IndexExpression;
import org.apache.cassandra.db.composites.*;
import org.apache.cassandra.db.composites.Composite.EOC;
import org.apache.cassandra.db.*;
import org.apache.cassandra.db.filter.RowFilter;
import org.apache.cassandra.db.index.SecondaryIndexManager;
import org.apache.cassandra.exceptions.InvalidRequestException;
import org.apache.cassandra.utils.FBUtilities;
import static org.apache.cassandra.cql3.statements.RequestValidations.checkFalse;
import static org.apache.cassandra.cql3.statements.RequestValidations.invalidRequest;
@ -65,18 +64,25 @@ final class PrimaryKeyRestrictionSet extends AbstractPrimaryKeyRestrictions
*/
private boolean contains;
public PrimaryKeyRestrictionSet(CType ctype)
/**
* <code>true</code> if the restrictions corresponding to a partition key, <code>false</code> if it's clustering columns.
*/
private boolean isPartitionKey;
public PrimaryKeyRestrictionSet(ClusteringComparator comparator, boolean isPartitionKey)
{
super(ctype);
super(comparator);
this.restrictions = new RestrictionSet();
this.eq = true;
this.isPartitionKey = isPartitionKey;
}
private PrimaryKeyRestrictionSet(PrimaryKeyRestrictionSet primaryKeyRestrictions,
Restriction restriction) throws InvalidRequestException
{
super(primaryKeyRestrictions.ctype);
super(primaryKeyRestrictions.comparator);
this.restrictions = primaryKeyRestrictions.restrictions.addRestriction(restriction);
this.isPartitionKey = primaryKeyRestrictions.isPartitionKey;
if (!primaryKeyRestrictions.isEmpty())
{
@ -104,6 +110,15 @@ final class PrimaryKeyRestrictionSet extends AbstractPrimaryKeyRestrictions
this.eq = true;
}
private List<ByteBuffer> toByteBuffers(SortedSet<? extends ClusteringPrefix> clusterings)
{
// It's currently a tad hard to follow that this is only called for partition key so we should fix that
List<ByteBuffer> l = new ArrayList<>(clusterings.size());
for (ClusteringPrefix clustering : clusterings)
l.add(CFMetaData.serializePartitionKey(clustering));
return l;
}
@Override
public boolean isSlice()
{
@ -161,13 +176,13 @@ final class PrimaryKeyRestrictionSet extends AbstractPrimaryKeyRestrictions
}
@Override
public List<Composite> valuesAsComposites(QueryOptions options) throws InvalidRequestException
public NavigableSet<Clustering> valuesAsClustering(QueryOptions options) throws InvalidRequestException
{
return appendTo(new CompositesBuilder(ctype), options).build();
return appendTo(MultiCBuilder.create(comparator), options).build();
}
@Override
public CompositesBuilder appendTo(CompositesBuilder builder, QueryOptions options)
public MultiCBuilder appendTo(MultiCBuilder builder, QueryOptions options)
{
for (Restriction r : restrictions)
{
@ -179,27 +194,22 @@ final class PrimaryKeyRestrictionSet extends AbstractPrimaryKeyRestrictions
}
@Override
public CompositesBuilder appendBoundTo(CompositesBuilder builder, Bound bound, QueryOptions options)
public MultiCBuilder appendBoundTo(MultiCBuilder builder, Bound bound, QueryOptions options)
{
throw new UnsupportedOperationException();
}
@Override
public List<Composite> boundsAsComposites(Bound bound, QueryOptions options) throws InvalidRequestException
public SortedSet<Slice.Bound> boundsAsClustering(Bound bound, QueryOptions options) throws InvalidRequestException
{
CompositesBuilder builder = new CompositesBuilder(ctype);
// The end-of-component of composite doesn't depend on whether the
// component type is reversed or not (i.e. the ReversedType is applied
// to the component comparator but not to the end-of-component itself),
// it only depends on whether the slice is reversed
MultiCBuilder builder = MultiCBuilder.create(comparator);
int keyPosition = 0;
for (Restriction r : restrictions)
{
ColumnDefinition def = r.getFirstColumn();
// In a restriction, we always have Bound.START < Bound.END for the "base" comparator.
// So if we're doing a reverse slice, we must inverse the bounds when giving them as start and end of the slice filter.
// But if the actual comparator itself is reversed, we must inversed the bounds too.
// The bound of this method is refering to the clustering order. So if said clustering order
// is reversed for this column, we should reverse the restriction we use.
Bound b = !def.isReversedType() ? bound : bound.reverse();
if (keyPosition != def.position() || r.isContains())
break;
@ -211,49 +221,41 @@ final class PrimaryKeyRestrictionSet extends AbstractPrimaryKeyRestrictions
// There wasn't any non EQ relation on that key, we select all records having the preceding component as prefix.
// For composites, if there was preceding component and we're computing the end, we must change the last component
// End-Of-Component, otherwise we would be selecting only one record.
return builder.buildWithEOC(bound.isEnd() ? EOC.END : EOC.START);
return builder.buildBound(bound.isStart(), true);
}
r.appendBoundTo(builder, b, options);
Composite.EOC eoc = eocFor(r, bound, b);
return builder.buildWithEOC(eoc);
return builder.buildBound(bound.isStart(), r.isInclusive(b));
}
r.appendBoundTo(builder, b, options);
if (builder.hasMissingElements())
return Collections.emptyList();
return FBUtilities.<Slice.Bound>emptySortedSet(comparator);
keyPosition = r.getLastColumn().position() + 1;
}
// Means no relation at all or everything was an equal
// Note: if the builder is "full", there is no need to use the end-of-component bit. For columns selection,
// it would be harmless to do it. However, we use this method got the partition key too. And when a query
// with 2ndary index is done, and with the the partition provided with an EQ, we'll end up here, and in that
// case using the eoc would be bad, since for the random partitioner we have no guarantee that
// prefix.end() will sort after prefix (see #5240).
EOC eoc = !builder.hasRemaining() ? EOC.NONE : (bound.isEnd() ? EOC.END : EOC.START);
return builder.buildWithEOC(eoc);
// Everything was an equal (or there was nothing)
return builder.buildBound(bound.isStart(), true);
}
@Override
public List<ByteBuffer> values(QueryOptions options) throws InvalidRequestException
{
return Composites.toByteBuffers(valuesAsComposites(options));
if (!isPartitionKey)
throw new UnsupportedOperationException();
return toByteBuffers(valuesAsClustering(options));
}
@Override
public List<ByteBuffer> bounds(Bound b, QueryOptions options) throws InvalidRequestException
{
return Composites.toByteBuffers(boundsAsComposites(b, options));
}
if (!isPartitionKey)
throw new UnsupportedOperationException();
private static Composite.EOC eocFor(Restriction r, Bound eocBound, Bound inclusiveBound)
{
if (eocBound.isStart())
return r.isInclusive(inclusiveBound) ? Composite.EOC.NONE : Composite.EOC.END;
return r.isInclusive(inclusiveBound) ? Composite.EOC.END : Composite.EOC.START;
return toByteBuffers(boundsAsClustering(b, options));
}
@Override
@ -279,30 +281,24 @@ final class PrimaryKeyRestrictionSet extends AbstractPrimaryKeyRestrictions
}
@Override
public void addIndexExpressionTo(List<IndexExpression> expressions,
SecondaryIndexManager indexManager,
QueryOptions options) throws InvalidRequestException
public void addRowFilterTo(RowFilter filter,
SecondaryIndexManager indexManager,
QueryOptions options) throws InvalidRequestException
{
Boolean clusteringColumns = null;
int position = 0;
for (Restriction restriction : restrictions)
{
ColumnDefinition columnDef = restriction.getFirstColumn();
// PrimaryKeyRestrictionSet contains only one kind of column, either partition key or clustering columns.
// Therefore we only need to check the column kind once. All the other columns will be of the same kind.
if (clusteringColumns == null)
clusteringColumns = columnDef.isClusteringColumn() ? Boolean.TRUE : Boolean.FALSE;
// We ignore all the clustering columns that can be handled by slices.
if (clusteringColumns && !restriction.isContains()&& position == columnDef.position())
if (!isPartitionKey && !restriction.isContains()&& position == columnDef.position())
{
position = restriction.getLastColumn().position() + 1;
if (!restriction.hasSupportingIndex(indexManager))
continue;
}
restriction.addIndexExpressionTo(expressions, indexManager, options);
restriction.addRowFilterTo(filter, indexManager, options);
}
}

9
src/java/org/apache/cassandra/cql3/restrictions/PrimaryKeyRestrictions.java
View File

@ -19,10 +19,13 @@ package org.apache.cassandra.cql3.restrictions;
import java.nio.ByteBuffer;
import java.util.List;
import java.util.NavigableSet;
import java.util.SortedSet;
import org.apache.cassandra.cql3.QueryOptions;
import org.apache.cassandra.cql3.statements.Bound;
import org.apache.cassandra.db.composites.Composite;
import org.apache.cassandra.db.Clustering;
import org.apache.cassandra.db.Slice;
import org.apache.cassandra.exceptions.InvalidRequestException;
/**
@ -36,9 +39,9 @@ interface PrimaryKeyRestrictions extends Restriction, Restrictions
public List<ByteBuffer> values(QueryOptions options) throws InvalidRequestException;
public List<Composite> valuesAsComposites(QueryOptions options) throws InvalidRequestException;
public NavigableSet<Clustering> valuesAsClustering(QueryOptions options) throws InvalidRequestException;
public List<ByteBuffer> bounds(Bound b, QueryOptions options) throws InvalidRequestException;
public List<Composite> boundsAsComposites(Bound bound, QueryOptions options) throws InvalidRequestException;
public SortedSet<Slice.Bound> boundsAsClustering(Bound bound, QueryOptions options) throws InvalidRequestException;
}

32
src/java/org/apache/cassandra/cql3/restrictions/Restriction.java
View File

@ -18,14 +18,13 @@
package org.apache.cassandra.cql3.restrictions;
import java.util.Collection;
import java.util.List;
import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.cql3.QueryOptions;
import org.apache.cassandra.cql3.functions.Function;
import org.apache.cassandra.cql3.statements.Bound;
import org.apache.cassandra.db.IndexExpression;
import org.apache.cassandra.db.composites.CompositesBuilder;
import org.apache.cassandra.db.MultiCBuilder;
import org.apache.cassandra.db.filter.RowFilter;
import org.apache.cassandra.db.index.SecondaryIndexManager;
import org.apache.cassandra.exceptions.InvalidRequestException;
@ -105,35 +104,34 @@ public interface Restriction
public boolean hasSupportingIndex(SecondaryIndexManager indexManager);
/**
* Adds to the specified list the <code>IndexExpression</code>s corresponding to this <code>Restriction</code>.
* Adds to the specified row filter the expressions corresponding to this <code>Restriction</code>.
*
* @param expressions the list to add the <code>IndexExpression</code>s to
* @param filter the row filter to add expressions to
* @param indexManager the secondary index manager
* @param options the query options
* @throws InvalidRequestException if this <code>Restriction</code> cannot be converted into
* <code>IndexExpression</code>s
* @throws InvalidRequestException if this <code>Restriction</code> cannot be converted into a row filter
*/
public void addIndexExpressionTo(List<IndexExpression> expressions,
SecondaryIndexManager indexManager,
QueryOptions options)
throws InvalidRequestException;
public void addRowFilterTo(RowFilter filter,
SecondaryIndexManager indexManager,
QueryOptions options)
throws InvalidRequestException;
/**
* Appends the values of this <code>Restriction</code> to the specified builder.
*
* @param builder the <code>CompositesBuilder</code> to append to.
* @param builder the <code>MultiCBuilder</code> to append to.
* @param options the query options
* @return the <code>CompositesBuilder</code>
* @return the <code>MultiCBuilder</code>
*/
public CompositesBuilder appendTo(CompositesBuilder builder, QueryOptions options);
public MultiCBuilder appendTo(MultiCBuilder builder, QueryOptions options);
/**
* Appends the values of the <code>Restriction</code> for the specified bound to the specified builder.
*
* @param builder the <code>CompositesBuilder</code> to append to.
* @param builder the <code>MultiCBuilder</code> to append to.
* @param bound the bound
* @param options the query options
* @return the <code>CompositesBuilder</code>
* @return the <code>MultiCBuilder</code>
*/
public CompositesBuilder appendBoundTo(CompositesBuilder builder, Bound bound, QueryOptions options);
public MultiCBuilder appendBoundTo(MultiCBuilder builder, Bound bound, QueryOptions options);
}

12
src/java/org/apache/cassandra/cql3/restrictions/RestrictionSet.java
View File

@ -24,8 +24,8 @@ import com.google.common.collect.Iterables;
import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.cql3.QueryOptions;
import org.apache.cassandra.cql3.functions.Function;
import org.apache.cassandra.cql3.restrictions.SingleColumnRestriction.Contains;
import org.apache.cassandra.db.IndexExpression;
import org.apache.cassandra.cql3.restrictions.SingleColumnRestriction.ContainsRestriction;
import org.apache.cassandra.db.filter.RowFilter;
import org.apache.cassandra.db.index.SecondaryIndexManager;
import org.apache.cassandra.exceptions.InvalidRequestException;
@ -66,12 +66,10 @@ final class RestrictionSet implements Restrictions, Iterable<Restriction>
}
@Override
public final void addIndexExpressionTo(List<IndexExpression> expressions,
SecondaryIndexManager indexManager,
QueryOptions options) throws InvalidRequestException
public final void addRowFilterTo(RowFilter filter, SecondaryIndexManager indexManager, QueryOptions options) throws InvalidRequestException
{
for (Restriction restriction : restrictions.values())
restriction.addIndexExpressionTo(expressions, indexManager, options);
restriction.addRowFilterTo(filter, indexManager, options);
}
@Override
@ -245,7 +243,7 @@ final class RestrictionSet implements Restrictions, Iterable<Restriction>
{
if (restriction.isContains())
{
Contains contains = (Contains) restriction;
ContainsRestriction contains = (ContainsRestriction) restriction;
numberOfContains += (contains.numberOfValues() + contains.numberOfKeys() + contains.numberOfEntries());
}
}

15
src/java/org/apache/cassandra/cql3/restrictions/Restrictions.java
View File

@ -18,12 +18,11 @@
package org.apache.cassandra.cql3.restrictions;
import java.util.Collection;
import java.util.List;
import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.cql3.QueryOptions;
import org.apache.cassandra.cql3.functions.Function;
import org.apache.cassandra.db.IndexExpression;
import org.apache.cassandra.db.filter.RowFilter;
import org.apache.cassandra.db.index.SecondaryIndexManager;
import org.apache.cassandra.exceptions.InvalidRequestException;
@ -54,18 +53,14 @@ interface Restrictions
public boolean hasSupportingIndex(SecondaryIndexManager indexManager);
/**
* Adds to the specified list the <code>IndexExpression</code>s corresponding to this <code>Restriction</code>.
* Adds to the specified row filter the expressions corresponding to this <code>Restrictions</code>.
*
* @param expressions the list to add the <code>IndexExpression</code>s to
* @param filter the row filter to add expressions to
* @param indexManager the secondary index manager
* @param options the query options
* @throws InvalidRequestException if this <code>Restriction</code> cannot be converted into
* <code>IndexExpression</code>s
* @throws InvalidRequestException if this <code>Restrictions</code> cannot be converted into a row filter
*/
public void addIndexExpressionTo(List<IndexExpression> expressions,
SecondaryIndexManager indexManager,
QueryOptions options)
throws InvalidRequestException;
public void addRowFilterTo(RowFilter filter, SecondaryIndexManager indexManager, QueryOptions options) throws InvalidRequestException;
/**
* Checks if this <code>PrimaryKeyRestrictionSet</code> is empty or not.

77
src/java/org/apache/cassandra/cql3/restrictions/ReversedPrimaryKeyRestrictions.java
View File

@ -1,77 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.cql3.restrictions;
import java.nio.ByteBuffer;
import java.util.Collections;
import java.util.List;
import org.apache.cassandra.cql3.QueryOptions;
import org.apache.cassandra.cql3.statements.Bound;
import org.apache.cassandra.db.composites.Composite;
import org.apache.cassandra.exceptions.InvalidRequestException;
/**
* <code>PrimaryKeyRestrictions</code> decorator that reverse the slices.
*/
final class ReversedPrimaryKeyRestrictions extends ForwardingPrimaryKeyRestrictions
{
/**
* The decorated restrictions.
*/
private PrimaryKeyRestrictions restrictions;
public ReversedPrimaryKeyRestrictions(PrimaryKeyRestrictions restrictions)
{
this.restrictions = restrictions;
}
@Override
public PrimaryKeyRestrictions mergeWith(Restriction restriction) throws InvalidRequestException
{
return new ReversedPrimaryKeyRestrictions(this.restrictions.mergeWith(restriction));
}
@Override
public List<ByteBuffer> bounds(Bound bound, QueryOptions options) throws InvalidRequestException
{
List<ByteBuffer> buffers = restrictions.bounds(bound.reverse(), options);
Collections.reverse(buffers);
return buffers;
}
@Override
public List<Composite> boundsAsComposites(Bound bound, QueryOptions options) throws InvalidRequestException
{
List<Composite> composites = restrictions.boundsAsComposites(bound.reverse(), options);
Collections.reverse(composites);
return composites;
}
@Override
public boolean isInclusive(Bound bound)
{
return this.restrictions.isInclusive(bound.reverse());
}
@Override
protected PrimaryKeyRestrictions getDelegate()
{
return this.restrictions;
}
}

133
src/java/org/apache/cassandra/cql3/restrictions/SingleColumnRestriction.java
View File

@ -27,11 +27,10 @@ import org.apache.cassandra.cql3.*;
import org.apache.cassandra.cql3.Term.Terminal;
import org.apache.cassandra.cql3.functions.Function;
import org.apache.cassandra.cql3.statements.Bound;
import org.apache.cassandra.db.IndexExpression;
import org.apache.cassandra.db.composites.CompositesBuilder;
import org.apache.cassandra.db.MultiCBuilder;
import org.apache.cassandra.db.filter.RowFilter;
import org.apache.cassandra.db.index.SecondaryIndex;
import org.apache.cassandra.db.index.SecondaryIndexManager;
import org.apache.cassandra.db.marshal.CompositeType;
import org.apache.cassandra.exceptions.InvalidRequestException;
import static org.apache.cassandra.cql3.statements.RequestValidations.checkBindValueSet;
@ -73,7 +72,7 @@ public abstract class SingleColumnRestriction extends AbstractRestriction
@Override
public boolean hasSupportingIndex(SecondaryIndexManager indexManager)
{
SecondaryIndex index = indexManager.getIndexForColumn(columnDef.name.bytes);
SecondaryIndex index = indexManager.getIndexForColumn(columnDef);
return index != null && isSupportedBy(index);
}
@ -97,11 +96,11 @@ public abstract class SingleColumnRestriction extends AbstractRestriction
*/
protected abstract boolean isSupportedBy(SecondaryIndex index);
public static final class EQ extends SingleColumnRestriction
public static final class EQRestriction extends SingleColumnRestriction
{
private final Term value;
public EQ(ColumnDefinition columnDef, Term value)
public EQRestriction(ColumnDefinition columnDef, Term value)
{
super(columnDef);
this.value = value;
@ -120,16 +119,15 @@ public abstract class SingleColumnRestriction extends AbstractRestriction
}
@Override
public void addIndexExpressionTo(List<IndexExpression> expressions,
SecondaryIndexManager indexManager,
QueryOptions options) throws InvalidRequestException
public void addRowFilterTo(RowFilter filter,
SecondaryIndexManager indexManager,
QueryOptions options) throws InvalidRequestException
{
ByteBuffer buffer = validateIndexedValue(columnDef, value.bindAndGet(options));
expressions.add(new IndexExpression(columnDef.name.bytes, Operator.EQ, buffer));
filter.add(columnDef, Operator.EQ, value.bindAndGet(options));
}
@Override
public CompositesBuilder appendTo(CompositesBuilder builder, QueryOptions options)
public MultiCBuilder appendTo(MultiCBuilder builder, QueryOptions options)
{
builder.addElementToAll(value.bindAndGet(options));
checkFalse(builder.containsNull(), "Invalid null value in condition for column %s", columnDef.name);
@ -156,9 +154,9 @@ public abstract class SingleColumnRestriction extends AbstractRestriction
}
}
public static abstract class IN extends SingleColumnRestriction
public static abstract class INRestriction extends SingleColumnRestriction
{
public IN(ColumnDefinition columnDef)
public INRestriction(ColumnDefinition columnDef)
{
super(columnDef);
}
@ -176,7 +174,7 @@ public abstract class SingleColumnRestriction extends AbstractRestriction
}
@Override
public CompositesBuilder appendTo(CompositesBuilder builder, QueryOptions options)
public MultiCBuilder appendTo(MultiCBuilder builder, QueryOptions options)
{
builder.addEachElementToAll(getValues(options));
checkFalse(builder.containsNull(), "Invalid null value in condition for column %s", columnDef.name);
@ -185,15 +183,14 @@ public abstract class SingleColumnRestriction extends AbstractRestriction
}
@Override
public void addIndexExpressionTo(List<IndexExpression> expressions,
SecondaryIndexManager indexManager,
QueryOptions options) throws InvalidRequestException
public void addRowFilterTo(RowFilter filter,
SecondaryIndexManager indexManager,
QueryOptions options) throws InvalidRequestException
{
List<ByteBuffer> values = getValues(options);
checkTrue(values.size() == 1, "IN restrictions are not supported on indexed columns");
ByteBuffer value = validateIndexedValue(columnDef, values.get(0));
expressions.add(new IndexExpression(columnDef.name.bytes, Operator.EQ, value));
filter.add(columnDef, Operator.EQ, values.get(0));
}
@Override
@ -205,11 +202,11 @@ public abstract class SingleColumnRestriction extends AbstractRestriction
protected abstract List<ByteBuffer> getValues(QueryOptions options) throws InvalidRequestException;
}
public static class InWithValues extends IN
public static class InRestrictionWithValues extends INRestriction
{
protected final List<Term> values;
public InWithValues(ColumnDefinition columnDef, List<Term> values)
public InRestrictionWithValues(ColumnDefinition columnDef, List<Term> values)
{
super(columnDef);
this.values = values;
@ -237,11 +234,11 @@ public abstract class SingleColumnRestriction extends AbstractRestriction
}
}
public static class InWithMarker extends IN
public static class InRestrictionWithMarker extends INRestriction
{
protected final AbstractMarker marker;
public InWithMarker(ColumnDefinition columnDef, AbstractMarker marker)
public InRestrictionWithMarker(ColumnDefinition columnDef, AbstractMarker marker)
{
super(columnDef);
this.marker = marker;
@ -270,11 +267,11 @@ public abstract class SingleColumnRestriction extends AbstractRestriction
}
}
public static class Slice extends SingleColumnRestriction
public static class SliceRestriction extends SingleColumnRestriction
{
private final TermSlice slice;
public Slice(ColumnDefinition columnDef, Bound bound, boolean inclusive, Term term)
public SliceRestriction(ColumnDefinition columnDef, Bound bound, boolean inclusive, Term term)
{
super(columnDef);
slice = TermSlice.newInstance(bound, inclusive, term);
@ -293,7 +290,7 @@ public abstract class SingleColumnRestriction extends AbstractRestriction
}
@Override
public CompositesBuilder appendTo(CompositesBuilder builder, QueryOptions options)
public MultiCBuilder appendTo(MultiCBuilder builder, QueryOptions options)
{
throw new UnsupportedOperationException();
}
@ -305,7 +302,7 @@ public abstract class SingleColumnRestriction extends AbstractRestriction
}
@Override
public CompositesBuilder appendBoundTo(CompositesBuilder builder, Bound bound, QueryOptions options)
public MultiCBuilder appendBoundTo(MultiCBuilder builder, Bound bound, QueryOptions options)
{
ByteBuffer value = slice.bound(bound).bindAndGet(options);
checkBindValueSet(value, "Invalid unset value for column %s", columnDef.name);
@ -326,7 +323,7 @@ public abstract class SingleColumnRestriction extends AbstractRestriction
"Column \"%s\" cannot be restricted by both an equality and an inequality relation",
columnDef.name);
SingleColumnRestriction.Slice otherSlice = (SingleColumnRestriction.Slice) otherRestriction;
SingleColumnRestriction.SliceRestriction otherSlice = (SingleColumnRestriction.SliceRestriction) otherRestriction;
checkFalse(hasBound(Bound.START) && otherSlice.hasBound(Bound.START),
"More than one restriction was found for the start bound on %s", columnDef.name);
@ -334,27 +331,15 @@ public abstract class SingleColumnRestriction extends AbstractRestriction
checkFalse(hasBound(Bound.END) && otherSlice.hasBound(Bound.END),
"More than one restriction was found for the end bound on %s", columnDef.name);
return new Slice(columnDef, slice.merge(otherSlice.slice));
return new SliceRestriction(columnDef, slice.merge(otherSlice.slice));
}
@Override
public void addIndexExpressionTo(List<IndexExpression> expressions,
SecondaryIndexManager indexManager,
QueryOptions options) throws InvalidRequestException
public void addRowFilterTo(RowFilter filter, SecondaryIndexManager indexManager, QueryOptions options) throws InvalidRequestException
{
for (Bound b : Bound.values())
{
if (hasBound(b))
{
ByteBuffer value = validateIndexedValue(columnDef, slice.bound(b).bindAndGet(options));
Operator op = slice.getIndexOperator(b);
// If the underlying comparator for name is reversed, we need to reverse the IndexOperator: user operation
// always refer to the "forward" sorting even if the clustering order is reversed, but the 2ndary code does
// use the underlying comparator as is.
op = columnDef.isReversedType() ? op.reverse() : op;
expressions.add(new IndexExpression(columnDef.name.bytes, op, value));
}
}
filter.add(columnDef, slice.getIndexOperator(b), slice.bound(b).bindAndGet(options));
}
@Override
@ -369,7 +354,7 @@ public abstract class SingleColumnRestriction extends AbstractRestriction
return String.format("SLICE%s", slice);
}
private Slice(ColumnDefinition columnDef, TermSlice slice)
private SliceRestriction(ColumnDefinition columnDef, TermSlice slice)
{
super(columnDef);
this.slice = slice;
@ -377,14 +362,14 @@ public abstract class SingleColumnRestriction extends AbstractRestriction
}
// This holds CONTAINS, CONTAINS_KEY, and map[key] = value restrictions because we might want to have any combination of them.
public static final class Contains extends SingleColumnRestriction
public static final class ContainsRestriction extends SingleColumnRestriction
{
private List<Term> values = new ArrayList<>(); // for CONTAINS
private List<Term> keys = new ArrayList<>(); // for CONTAINS_KEY
private List<Term> entryKeys = new ArrayList<>(); // for map[key] = value
private List<Term> entryValues = new ArrayList<>(); // for map[key] = value
public Contains(ColumnDefinition columnDef, Term t, boolean isKey)
public ContainsRestriction(ColumnDefinition columnDef, Term t, boolean isKey)
{
super(columnDef);
if (isKey)
@ -393,7 +378,7 @@ public abstract class SingleColumnRestriction extends AbstractRestriction
values.add(t);
}
public Contains(ColumnDefinition columnDef, Term mapKey, Term mapValue)
public ContainsRestriction(ColumnDefinition columnDef, Term mapKey, Term mapValue)
{
super(columnDef);
entryKeys.add(mapKey);
@ -401,7 +386,7 @@ public abstract class SingleColumnRestriction extends AbstractRestriction
}
@Override
public CompositesBuilder appendTo(CompositesBuilder builder, QueryOptions options)
public MultiCBuilder appendTo(MultiCBuilder builder, QueryOptions options)
{
throw new UnsupportedOperationException();
}
@ -419,33 +404,27 @@ public abstract class SingleColumnRestriction extends AbstractRestriction
"Collection column %s can only be restricted by CONTAINS, CONTAINS KEY, or map-entry equality",
columnDef.name);
SingleColumnRestriction.Contains newContains = new Contains(columnDef);
SingleColumnRestriction.ContainsRestriction newContains = new ContainsRestriction(columnDef);
copyKeysAndValues(this, newContains);
copyKeysAndValues((Contains) otherRestriction, newContains);
copyKeysAndValues((ContainsRestriction) otherRestriction, newContains);
return newContains;
}
@Override
public void addIndexExpressionTo(List<IndexExpression> expressions,
SecondaryIndexManager indexManager,
QueryOptions options)
throws InvalidRequestException
public void addRowFilterTo(RowFilter filter, SecondaryIndexManager indexManager, QueryOptions options) throws InvalidRequestException
{
addExpressionsFor(expressions, bindAndGet(values, options), Operator.CONTAINS);
addExpressionsFor(expressions, bindAndGet(keys, options), Operator.CONTAINS_KEY);
addExpressionsFor(expressions, entries(options), Operator.EQ);
}
for (ByteBuffer value : bindAndGet(values, options))
filter.add(columnDef, Operator.CONTAINS, value);
for (ByteBuffer key : bindAndGet(keys, options))
filter.add(columnDef, Operator.CONTAINS_KEY, key);
private void addExpressionsFor(List<IndexExpression> target, List<ByteBuffer> values,
Operator op) throws InvalidRequestException
{
for (ByteBuffer value : values)
{
validateIndexedValue(columnDef, value);
target.add(new IndexExpression(columnDef.name.bytes, op, value));
}
List<ByteBuffer> eks = bindAndGet(entryKeys, options);
List<ByteBuffer> evs = bindAndGet(entryValues, options);
assert eks.size() == evs.size();
for (int i = 0; i < eks.size(); i++)
filter.addMapEquality(columnDef, eks.get(i), Operator.EQ, evs.get(i));
}
@Override
@ -502,7 +481,7 @@ public abstract class SingleColumnRestriction extends AbstractRestriction
}
@Override
public CompositesBuilder appendBoundTo(CompositesBuilder builder, Bound bound, QueryOptions options)
public MultiCBuilder appendBoundTo(MultiCBuilder builder, Bound bound, QueryOptions options)
{
throw new UnsupportedOperationException();
}
@ -513,20 +492,6 @@ public abstract class SingleColumnRestriction extends AbstractRestriction
throw new UnsupportedOperationException();
}
private List<ByteBuffer> entries(QueryOptions options) throws InvalidRequestException
{
List<ByteBuffer> entryBuffers = new ArrayList<>(entryKeys.size());
List<ByteBuffer> keyBuffers = bindAndGet(entryKeys, options);
List<ByteBuffer> valueBuffers = bindAndGet(entryValues, options);
for (int i = 0; i < entryKeys.size(); i++)
{
if (valueBuffers.get(i) == null)
throw new InvalidRequestException("Unsupported null value for map-entry equality");
entryBuffers.add(CompositeType.build(keyBuffers.get(i), valueBuffers.get(i)));
}
return entryBuffers;
}
/**
* Binds the query options to the specified terms and returns the resulting values.
*
@ -549,7 +514,7 @@ public abstract class SingleColumnRestriction extends AbstractRestriction
* @param from the <code>Contains</code> to copy from
* @param to the <code>Contains</code> to copy to
*/
private static void copyKeysAndValues(Contains from, Contains to)
private static void copyKeysAndValues(ContainsRestriction from, ContainsRestriction to)
{
to.values.addAll(from.values);
to.keys.addAll(from.keys);
@ -557,7 +522,7 @@ public abstract class SingleColumnRestriction extends AbstractRestriction
to.entryValues.addAll(from.entryValues);
}
private Contains(ColumnDefinition columnDef)
private ContainsRestriction(ColumnDefinition columnDef)
{
super(columnDef);
}

146
src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java
View File

@ -29,7 +29,7 @@ import org.apache.cassandra.cql3.*;
import org.apache.cassandra.cql3.functions.Function;
import org.apache.cassandra.cql3.statements.Bound;
import org.apache.cassandra.db.*;
import org.apache.cassandra.db.composites.Composite;
import org.apache.cassandra.db.filter.RowFilter;
import org.apache.cassandra.db.index.SecondaryIndexManager;
import org.apache.cassandra.dht.*;
import org.apache.cassandra.exceptions.InvalidRequestException;
@ -67,7 +67,7 @@ public final class StatementRestrictions
private RestrictionSet nonPrimaryKeyRestrictions;
/**
* The restrictions used to build the index expressions
* The restrictions used to build the row filter
*/
private final List<Restrictions> indexRestrictions = new ArrayList<>();
@ -95,28 +95,28 @@ public final class StatementRestrictions
private StatementRestrictions(CFMetaData cfm)
{
this.cfm = cfm;
this.partitionKeyRestrictions = new PrimaryKeyRestrictionSet(cfm.getKeyValidatorAsCType());
this.clusteringColumnsRestrictions = new PrimaryKeyRestrictionSet(cfm.comparator);
this.partitionKeyRestrictions = new PrimaryKeyRestrictionSet(cfm.getKeyValidatorAsClusteringComparator(), true);
this.clusteringColumnsRestrictions = new PrimaryKeyRestrictionSet(cfm.comparator, false);
this.nonPrimaryKeyRestrictions = new RestrictionSet();
}
public StatementRestrictions(CFMetaData cfm,
List<Relation> whereClause,
VariableSpecifications boundNames,
boolean selectsOnlyStaticColumns,
boolean selectACollection) throws InvalidRequestException
List<Relation> whereClause,
VariableSpecifications boundNames,
boolean selectsOnlyStaticColumns,
boolean selectACollection,
boolean useFiltering) throws InvalidRequestException
{
this.cfm = cfm;
this.partitionKeyRestrictions = new PrimaryKeyRestrictionSet(cfm.getKeyValidatorAsCType());
this.clusteringColumnsRestrictions = new PrimaryKeyRestrictionSet(cfm.comparator);
this.nonPrimaryKeyRestrictions = new RestrictionSet();
this(cfm);
/*
* WHERE clause. For a given entity, rules are: - EQ relation conflicts with anything else (including a 2nd EQ)
* - Can't have more than one LT(E) relation (resp. GT(E) relation) - IN relation are restricted to row keys
* (for now) and conflicts with anything else (we could allow two IN for the same entity but that doesn't seem
* very useful) - The value_alias cannot be restricted in any way (we don't support wide rows with indexed value
* in CQL so far)
* WHERE clause. For a given entity, rules are:
* - EQ relation conflicts with anything else (including a 2nd EQ)
* - Can't have more than one LT(E) relation (resp. GT(E) relation)
* - IN relation are restricted to row keys (for now) and conflicts with anything else (we could
* allow two IN for the same entity but that doesn't seem very useful)
* - The value_alias cannot be restricted in any way (we don't support wide rows with indexed value
* in CQL so far)
*/
for (Relation relation : whereClause)
addRestriction(relation.toRestriction(cfm, boundNames));
@ -133,7 +133,7 @@ public final class StatementRestrictions
processPartitionKeyRestrictions(hasQueriableIndex);
// Some but not all of the partition key columns have been specified;
// hence we need turn these restrictions into index expressions.
// hence we need turn these restrictions into a row filter.
if (usesSecondaryIndexing)
indexRestrictions.add(partitionKeyRestrictions);
@ -155,7 +155,11 @@ public final class StatementRestrictions
// there is restrictions not covered by the PK.
if (!nonPrimaryKeyRestrictions.isEmpty())
{
usesSecondaryIndexing = true;
if (hasQueriableIndex)
usesSecondaryIndexing = true;
else if (!useFiltering)
throw new InvalidRequestException("No supported secondary index found for the non primary key columns restrictions");
indexRestrictions.add(nonPrimaryKeyRestrictions);
}
@ -191,6 +195,19 @@ public final class StatementRestrictions
nonPrimaryKeyRestrictions = nonPrimaryKeyRestrictions.addRestriction(restriction);
}
/**
* Returns the non-PK column that are restricted.
*/
public Set<ColumnDefinition> nonPKRestrictedColumns()
{
Set<ColumnDefinition> columns = new HashSet<>();
for (Restrictions r : indexRestrictions)
for (ColumnDefinition def : r.getColumnDefs())
if (!def.isPrimaryKeyColumn())
columns.add(def);
return columns;
}
/**
* Checks if the restrictions on the partition key is an IN restriction.
*
@ -309,17 +326,16 @@ public final class StatementRestrictions
usesSecondaryIndexing = true;
}
public List<IndexExpression> getIndexExpressions(SecondaryIndexManager indexManager,
QueryOptions options) throws InvalidRequestException
public RowFilter getRowFilter(SecondaryIndexManager indexManager, QueryOptions options) throws InvalidRequestException
{
if (!usesSecondaryIndexing || indexRestrictions.isEmpty())
return Collections.emptyList();
if (indexRestrictions.isEmpty())
return RowFilter.NONE;
List<IndexExpression> expressions = new ArrayList<>();
RowFilter filter = RowFilter.create();
for (Restrictions restrictions : indexRestrictions)
restrictions.addIndexExpressionTo(expressions, indexManager, options);
restrictions.addRowFilterTo(filter, indexManager, options);
return expressions;
return filter;
}
/**
@ -345,8 +361,7 @@ public final class StatementRestrictions
private ByteBuffer getPartitionKeyBound(Bound b, QueryOptions options) throws InvalidRequestException
{
// Deal with unrestricted partition key components (special-casing is required to deal with 2i queries on the
// first
// component of a composite partition key).
// first component of a composite partition key).
if (hasPartitionKeyUnrestrictedComponents())
return ByteBufferUtil.EMPTY_BYTE_BUFFER;
@ -361,7 +376,7 @@ public final class StatementRestrictions
* @return the partition key bounds
* @throws InvalidRequestException if the query is invalid
*/
public AbstractBounds<RowPosition> getPartitionKeyBounds(QueryOptions options) throws InvalidRequestException
public AbstractBounds<PartitionPosition> getPartitionKeyBounds(QueryOptions options) throws InvalidRequestException
{
IPartitioner p = StorageService.getPartitioner();
@ -373,14 +388,14 @@ public final class StatementRestrictions
return getPartitionKeyBounds(p, options);
}
private AbstractBounds<RowPosition> getPartitionKeyBounds(IPartitioner p,
private AbstractBounds<PartitionPosition> getPartitionKeyBounds(IPartitioner p,
QueryOptions options) throws InvalidRequestException
{
ByteBuffer startKeyBytes = getPartitionKeyBound(Bound.START, options);
ByteBuffer finishKeyBytes = getPartitionKeyBound(Bound.END, options);
RowPosition startKey = RowPosition.ForKey.get(startKeyBytes, p);
RowPosition finishKey = RowPosition.ForKey.get(finishKeyBytes, p);
PartitionPosition startKey = PartitionPosition.ForKey.get(startKeyBytes, p);
PartitionPosition finishKey = PartitionPosition.ForKey.get(finishKeyBytes, p);
if (startKey.compareTo(finishKey) > 0 && !finishKey.isMinimum())
return null;
@ -397,7 +412,7 @@ public final class StatementRestrictions
: new ExcludingBounds<>(startKey, finishKey);
}
private AbstractBounds<RowPosition> getPartitionKeyBoundsForTokenRestrictions(IPartitioner p,
private AbstractBounds<PartitionPosition> getPartitionKeyBoundsForTokenRestrictions(IPartitioner p,
QueryOptions options)
throws InvalidRequestException
{
@ -422,8 +437,8 @@ public final class StatementRestrictions
&& (cmp > 0 || (cmp == 0 && (!includeStart || !includeEnd))))
return null;
RowPosition start = includeStart ? startToken.minKeyBound() : startToken.maxKeyBound();
RowPosition end = includeEnd ? endToken.maxKeyBound() : endToken.minKeyBound();
PartitionPosition start = includeStart ? startToken.minKeyBound() : startToken.maxKeyBound();
PartitionPosition end = includeEnd ? endToken.maxKeyBound() : endToken.minKeyBound();
return new Range<>(start, end);
}
@ -438,17 +453,6 @@ public final class StatementRestrictions
return p.getTokenFactory().fromByteArray(value);
}
/**
* Checks if the query does not contains any restriction on the clustering columns.
*
* @return <code>true</code> if the query does not contains any restriction on the clustering columns,
* <code>false</code> otherwise.
*/
public boolean hasNoClusteringColumnsRestriction()
{
return clusteringColumnsRestrictions.isEmpty();
}
/**
* Checks if the query has some restrictions on the clustering columns.
*
@ -460,39 +464,16 @@ public final class StatementRestrictions
return !clusteringColumnsRestrictions.isEmpty();
}
// For non-composite slices, we don't support internally the difference between exclusive and
// inclusive bounds, so we deal with it manually.
public boolean isNonCompositeSliceWithExclusiveBounds()
{
return !cfm.comparator.isCompound()
&& clusteringColumnsRestrictions.isSlice()
&& (!clusteringColumnsRestrictions.isInclusive(Bound.START) || !clusteringColumnsRestrictions.isInclusive(Bound.END));
}
/**
* Returns the requested clustering columns as <code>Composite</code>s.
* Returns the requested clustering columns.
*
* @param options the query options
* @return the requested clustering columns as <code>Composite</code>s
* @return the requested clustering columns
* @throws InvalidRequestException if the query is not valid
*/
public List<Composite> getClusteringColumnsAsComposites(QueryOptions options) throws InvalidRequestException
public NavigableSet<Clustering> getClusteringColumns(QueryOptions options) throws InvalidRequestException
{
return clusteringColumnsRestrictions.valuesAsComposites(options);
}
/**
* Returns the bounds (start or end) of the clustering columns as <code>Composites</code>.
*
* @param b the bound type
* @param options the query options
* @return the bounds (start or end) of the clustering columns as <code>Composites</code>
* @throws InvalidRequestException if the request is not valid
*/
public List<Composite> getClusteringColumnsBoundsAsComposites(Bound b,
QueryOptions options) throws InvalidRequestException
{
return clusteringColumnsRestrictions.boundsAsComposites(b, options);
return clusteringColumnsRestrictions.valuesAsClustering(options);
}
/**
@ -503,9 +484,9 @@ public final class StatementRestrictions
* @return the bounds (start or end) of the clustering columns
* @throws InvalidRequestException if the request is not valid
*/
public List<ByteBuffer> getClusteringColumnsBounds(Bound b, QueryOptions options) throws InvalidRequestException
public SortedSet<Slice.Bound> getClusteringColumnsBounds(Bound b, QueryOptions options) throws InvalidRequestException
{
return clusteringColumnsRestrictions.bounds(b, options);
return clusteringColumnsRestrictions.boundsAsClustering(b, options);
}
/**
@ -527,15 +508,9 @@ public final class StatementRestrictions
*/
public boolean isColumnRange()
{
// Due to CASSANDRA-5762, we always do a slice for CQL3 tables (not dense, composite).
// Static CF (non dense but non composite) never entails a column slice however
if (!cfm.comparator.isDense())
return cfm.comparator.isCompound();
// Otherwise (i.e. for compact table where we don't have a row marker anyway and thus don't care about
// CASSANDRA-5762),
// it is a range query if it has at least one the column alias for which no relation is defined or is not EQ.
return clusteringColumnsRestrictions.size() < cfm.clusteringColumns().size() || clusteringColumnsRestrictions.isSlice();
return clusteringColumnsRestrictions.size() < cfm.clusteringColumns().size()
|| (!clusteringColumnsRestrictions.isEQ() && !clusteringColumnsRestrictions.isIN());
}
/**
@ -564,9 +539,4 @@ public final class StatementRestrictions
// so far, 2i means that you've restricted a non static column, so the query is somewhat non-sensical.
checkFalse(selectsOnlyStaticColumns, "Queries using 2ndary indexes don't support selecting only static columns");
}
public void reverse()
{
clusteringColumnsRestrictions = new ReversedPrimaryKeyRestrictions(clusteringColumnsRestrictions);
}
}

11
src/java/org/apache/cassandra/cql3/restrictions/TokenFilter.java
View File

@ -18,8 +18,7 @@
package org.apache.cassandra.cql3.restrictions;
import java.nio.ByteBuffer;
import java.util.ArrayList;
import java.util.List;
import java.util.*;
import com.google.common.collect.BoundType;
import com.google.common.collect.ImmutableRangeSet;
@ -28,7 +27,7 @@ import com.google.common.collect.RangeSet;
import org.apache.cassandra.cql3.QueryOptions;
import org.apache.cassandra.cql3.statements.Bound;
import org.apache.cassandra.db.composites.Composite;
import org.apache.cassandra.db.*;
import org.apache.cassandra.dht.IPartitioner;
import org.apache.cassandra.dht.Token;
import org.apache.cassandra.exceptions.InvalidRequestException;
@ -84,7 +83,7 @@ final class TokenFilter extends ForwardingPrimaryKeyRestrictions
}
@Override
public List<Composite> valuesAsComposites(QueryOptions options) throws InvalidRequestException
public NavigableSet<Clustering> valuesAsClustering(QueryOptions options) throws InvalidRequestException
{
throw new UnsupportedOperationException();
}
@ -117,9 +116,9 @@ final class TokenFilter extends ForwardingPrimaryKeyRestrictions
}
@Override
public List<Composite> boundsAsComposites(Bound bound, QueryOptions options) throws InvalidRequestException
public SortedSet<Slice.Bound> boundsAsClustering(Bound bound, QueryOptions options) throws InvalidRequestException
{
return tokenRestriction.boundsAsComposites(bound, options);
return tokenRestriction.boundsAsClustering(bound, options);
}
/**

49
src/java/org/apache/cassandra/cql3/restrictions/TokenRestriction.java
View File

@ -18,9 +18,7 @@
package org.apache.cassandra.cql3.restrictions;
import java.nio.ByteBuffer;
import java.util.Collection;
import java.util.Collections;
import java.util.List;
import java.util.*;
import com.google.common.base.Joiner;
@ -29,10 +27,8 @@ import org.apache.cassandra.cql3.QueryOptions;
import org.apache.cassandra.cql3.Term;
import org.apache.cassandra.cql3.functions.Function;
import org.apache.cassandra.cql3.statements.Bound;
import org.apache.cassandra.db.IndexExpression;
import org.apache.cassandra.db.composites.CType;
import org.apache.cassandra.db.composites.Composite;
import org.apache.cassandra.db.composites.CompositesBuilder;
import org.apache.cassandra.db.*;
import org.apache.cassandra.db.filter.RowFilter;
import org.apache.cassandra.db.index.SecondaryIndexManager;
import org.apache.cassandra.exceptions.InvalidRequestException;
@ -51,12 +47,12 @@ public abstract class TokenRestriction extends AbstractPrimaryKeyRestrictions
/**
* Creates a new <code>TokenRestriction</code> that apply to the specified columns.
*
* @param ctype the composite type
* @param comparator the clustering comparator
* @param columnDefs the definition of the columns to which apply the token restriction
*/
public TokenRestriction(CType ctype, List<ColumnDefinition> columnDefs)
public TokenRestriction(ClusteringComparator comparator, List<ColumnDefinition> columnDefs)
{
super(ctype);
super(comparator);
this.columnDefs = columnDefs;
}
@ -91,27 +87,25 @@ public abstract class TokenRestriction extends AbstractPrimaryKeyRestrictions
}
@Override
public final void addIndexExpressionTo(List<IndexExpression> expressions,
SecondaryIndexManager indexManager,
QueryOptions options)
public void addRowFilterTo(RowFilter filter, SecondaryIndexManager indexManager, QueryOptions options)
{
throw new UnsupportedOperationException("Index expression cannot be created for token restriction");
}
@Override
public CompositesBuilder appendTo(CompositesBuilder builder, QueryOptions options)
public MultiCBuilder appendTo(MultiCBuilder builder, QueryOptions options)
{
throw new UnsupportedOperationException();
}
@Override
public List<Composite> valuesAsComposites(QueryOptions options) throws InvalidRequestException
public NavigableSet<Clustering> valuesAsClustering(QueryOptions options) throws InvalidRequestException
{
throw new UnsupportedOperationException();
}
@Override
public List<Composite> boundsAsComposites(Bound bound, QueryOptions options) throws InvalidRequestException
public SortedSet<Slice.Bound> boundsAsClustering(Bound bound, QueryOptions options) throws InvalidRequestException
{
throw new UnsupportedOperationException();
}
@ -153,16 +147,16 @@ public abstract class TokenRestriction extends AbstractPrimaryKeyRestrictions
if (restriction instanceof PrimaryKeyRestrictions)
return (PrimaryKeyRestrictions) restriction;
return new PrimaryKeyRestrictionSet(ctype).mergeWith(restriction);
return new PrimaryKeyRestrictionSet(comparator, true).mergeWith(restriction);
}
public static final class EQ extends TokenRestriction
public static final class EQRestriction extends TokenRestriction
{
private final Term value;
public EQ(CType ctype, List<ColumnDefinition> columnDefs, Term value)
public EQRestriction(ClusteringComparator comparator, List<ColumnDefinition> columnDefs, Term value)
{
super(ctype, columnDefs);
super(comparator, columnDefs);
this.value = value;
}
@ -192,13 +186,13 @@ public abstract class TokenRestriction extends AbstractPrimaryKeyRestrictions
}
}
public static class Slice extends TokenRestriction
public static class SliceRestriction extends TokenRestriction
{
private final TermSlice slice;
public Slice(CType ctype, List<ColumnDefinition> columnDefs, Bound bound, boolean inclusive, Term term)
public SliceRestriction(ClusteringComparator comparator, List<ColumnDefinition> columnDefs, Bound bound, boolean inclusive, Term term)
{
super(ctype, columnDefs);
super(comparator, columnDefs);
slice = TermSlice.newInstance(bound, inclusive, term);
}
@ -246,7 +240,7 @@ public abstract class TokenRestriction extends AbstractPrimaryKeyRestrictions
throw invalidRequest("Columns \"%s\" cannot be restricted by both an equality and an inequality relation",
getColumnNamesAsString());
TokenRestriction.Slice otherSlice = (TokenRestriction.Slice) otherRestriction;
TokenRestriction.SliceRestriction otherSlice = (TokenRestriction.SliceRestriction) otherRestriction;
if (hasBound(Bound.START) && otherSlice.hasBound(Bound.START))
throw invalidRequest("More than one restriction was found for the start bound on %s",
@ -256,7 +250,7 @@ public abstract class TokenRestriction extends AbstractPrimaryKeyRestrictions
throw invalidRequest("More than one restriction was found for the end bound on %s",
getColumnNamesAsString());
return new Slice(ctype, columnDefs, slice.merge(otherSlice.slice));
return new SliceRestriction(comparator, columnDefs, slice.merge(otherSlice.slice));
}
@Override
@ -264,10 +258,9 @@ public abstract class TokenRestriction extends AbstractPrimaryKeyRestrictions
{
return String.format("SLICE%s", slice);
}
private Slice(CType ctype, List<ColumnDefinition> columnDefs, TermSlice slice)
private SliceRestriction(ClusteringComparator comparator, List<ColumnDefinition> columnDefs, TermSlice slice)
{
super(ctype, columnDefs);
super(comparator, columnDefs);
this.slice = slice;
}
}

57
src/java/org/apache/cassandra/cql3/selection/Selection.java
View File

@ -29,9 +29,7 @@ import org.apache.cassandra.config.CFMetaData;
import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.cql3.*;
import org.apache.cassandra.cql3.functions.Function;
import org.apache.cassandra.db.Cell;
import org.apache.cassandra.db.CounterCell;
import org.apache.cassandra.db.ExpiringCell;
import org.apache.cassandra.db.rows.Cell;
import org.apache.cassandra.db.context.CounterContext;
import org.apache.cassandra.db.marshal.UTF8Type;
import org.apache.cassandra.exceptions.InvalidRequestException;
@ -120,9 +118,6 @@ public abstract class Selection
*/
public boolean containsACollection()
{
if (!cfm.comparator.hasCollections())
return false;
for (ColumnDefinition def : getColumns())
if (def.type.isCollection() && def.type.isMultiCell())
return true;
@ -237,9 +232,9 @@ public abstract class Selection
return columnMapping;
}
public ResultSetBuilder resultSetBuilder(long now, boolean isJson) throws InvalidRequestException
public ResultSetBuilder resultSetBuilder(boolean isJons) throws InvalidRequestException
{
return new ResultSetBuilder(now, isJson);
return new ResultSetBuilder(isJons);
}
public abstract boolean isAggregate();
@ -277,18 +272,22 @@ public abstract class Selection
List<ByteBuffer> current;
final long[] timestamps;
final int[] ttls;
final long now;
private final boolean isJson;
private ResultSetBuilder(long now, boolean isJson) throws InvalidRequestException
private ResultSetBuilder(boolean isJson) throws InvalidRequestException
{
this.resultSet = new ResultSet(getResultMetadata(isJson).copy(), new ArrayList<List<ByteBuffer>>());
this.selectors = newSelectors();
this.timestamps = collectTimestamps ? new long[columns.size()] : null;
this.ttls = collectTTLs ? new int[columns.size()] : null;
this.now = now;
this.isJson = isJson;
// We use MIN_VALUE to indicate no timestamp and -1 for no ttl
if (timestamps != null)
Arrays.fill(timestamps, Long.MIN_VALUE);
if (ttls != null)
Arrays.fill(ttls, -1);
}
public void add(ByteBuffer v)
@ -296,25 +295,28 @@ public abstract class Selection
current.add(v);
}
public void add(Cell c)
public void add(Cell c, int nowInSec)
{
current.add(isDead(c) ? null : value(c));
if (c == null)
{
current.add(null);
return;
}
current.add(value(c));
if (timestamps != null)
{
timestamps[current.size() - 1] = isDead(c) ? Long.MIN_VALUE : c.timestamp();
}
timestamps[current.size() - 1] = c.livenessInfo().timestamp();
if (ttls != null)
{
int ttl = -1;
if (!isDead(c) && c instanceof ExpiringCell)
ttl = c.getLocalDeletionTime() - (int) (now / 1000);
ttls[current.size() - 1] = ttl;
}
ttls[current.size() - 1] = c.livenessInfo().remainingTTL(nowInSec);
}
private boolean isDead(Cell c)
private ByteBuffer value(Cell c)
{
return c == null || !c.isLive(now);
return c.isCounterCell()
? ByteBufferUtil.bytes(CounterContext.instance().total(c.value()))
: c.value();
}
public void newRow(int protocolVersion) throws InvalidRequestException
@ -378,13 +380,6 @@ public abstract class Selection
sb.append("}");
return Collections.singletonList(UTF8Type.instance.getSerializer().serialize(sb.toString()));
}
private ByteBuffer value(Cell c)
{
return (c instanceof CounterCell)
? ByteBufferUtil.bytes(CounterContext.instance().total(c.value()))
: c.value();
}
}
private static interface Selectors

2
src/java/org/apache/cassandra/cql3/selection/Selector.java
View File

@ -59,7 +59,7 @@ public abstract class Selector implements AssignmentTestable
{
return new ColumnSpecification(cfm.ksName,
cfm.cfName,
new ColumnIdentifier(getColumnName(), true),
ColumnIdentifier.getInterned(getColumnName(), true),
getReturnType());
}

80
src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java
View File

@ -92,12 +92,12 @@ public class AlterTableStatement extends SchemaAlteringStatement
{
case ADD:
assert columnName != null;
if (cfm.comparator.isDense())
if (cfm.isDense())
throw new InvalidRequestException("Cannot add new column to a COMPACT STORAGE table");
if (isStatic)
{
if (!cfm.comparator.isCompound())
if (!cfm.isCompound())
throw new InvalidRequestException("Static columns are not allowed in COMPACT STORAGE tables");
if (cfm.clusteringColumns().isEmpty())
throw new InvalidRequestException("Static columns are only useful (and thus allowed) if the table has at least one clustering column");
@ -122,26 +122,23 @@ public class AlterTableStatement extends SchemaAlteringStatement
AbstractType<?> type = validator.getType();
if (type.isCollection() && type.isMultiCell())
{
if (!cfm.comparator.supportCollections())
throw new InvalidRequestException("Cannot use non-frozen collections with a non-composite PRIMARY KEY");
if (!cfm.isCompound())
throw new InvalidRequestException("Cannot use non-frozen collections in COMPACT STORAGE tables");
if (cfm.isSuper())
throw new InvalidRequestException("Cannot use non-frozen collections with super column families");
// If there used to be a collection column with the same name (that has been dropped), it will
// still be appear in the ColumnToCollectionType because or reasons explained on #6276. The same
// reason mean that we can't allow adding a new collection with that name (see the ticket for details).
if (cfm.comparator.hasCollections())
{
CollectionType previous = cfm.comparator.collectionType() == null ? null : cfm.comparator.collectionType().defined.get(columnName.bytes);
if (previous != null && !type.isCompatibleWith(previous))
throw new InvalidRequestException(String.format("Cannot add a collection with the name %s " +
"because a collection with the same name and a different type has already been used in the past", columnName));
}
cfm.comparator = cfm.comparator.addOrUpdateCollection(columnName, (CollectionType)type);
// If there used to be a collection column with the same name (that has been dropped), we could still have
// some data using the old type, and so we can't allow adding a collection with the same name unless
// the types are compatible (see #6276).
CFMetaData.DroppedColumn dropped = cfm.getDroppedColumns().get(columnName);
// We could have type == null for old dropped columns, in which case we play it safe and refuse
if (dropped != null && (dropped.type == null || (dropped.type instanceof CollectionType && !type.isCompatibleWith(dropped.type))))
throw new InvalidRequestException(String.format("Cannot add a collection with the name %s " +
"because a collection with the same name and a different type%s has already been used in the past",
columnName, dropped.type == null ? "" : " (" + dropped.type.asCQL3Type() + ")"));
}
Integer componentIndex = cfm.comparator.isCompound() ? cfm.comparator.clusteringPrefixSize() : null;
Integer componentIndex = cfm.isCompound() ? cfm.comparator.size() : null;
cfm.addColumnDefinition(isStatic
? ColumnDefinition.staticDef(cfm, columnName.bytes, type, componentIndex)
: ColumnDefinition.regularDef(cfm, columnName.bytes, type, componentIndex));
@ -158,28 +155,13 @@ public class AlterTableStatement extends SchemaAlteringStatement
case PARTITION_KEY:
if (validatorType instanceof CounterColumnType)
throw new InvalidRequestException(String.format("counter type is not supported for PRIMARY KEY part %s", columnName));
if (cfm.getKeyValidator() instanceof CompositeType)
{
List<AbstractType<?>> oldTypes = ((CompositeType) cfm.getKeyValidator()).types;
if (!validatorType.isValueCompatibleWith(oldTypes.get(def.position())))
throw new ConfigurationException(String.format("Cannot change %s from type %s to type %s: types are incompatible.",
columnName,
oldTypes.get(def.position()).asCQL3Type(),
validator));
List<AbstractType<?>> newTypes = new ArrayList<AbstractType<?>>(oldTypes);
newTypes.set(def.position(), validatorType);
cfm.keyValidator(CompositeType.getInstance(newTypes));
}
else
{
if (!validatorType.isValueCompatibleWith(cfm.getKeyValidator()))
throw new ConfigurationException(String.format("Cannot change %s from type %s to type %s: types are incompatible.",
columnName,
cfm.getKeyValidator().asCQL3Type(),
validator));
cfm.keyValidator(validatorType);
}
AbstractType<?> currentType = cfm.getKeyValidatorAsClusteringComparator().subtype(def.position());
if (!validatorType.isValueCompatibleWith(currentType))
throw new ConfigurationException(String.format("Cannot change %s from type %s to type %s: types are incompatible.",
columnName,
currentType.asCQL3Type(),
validator));
break;
case CLUSTERING_COLUMN:
AbstractType<?> oldType = cfm.comparator.subtype(def.position());
@ -192,16 +174,6 @@ public class AlterTableStatement extends SchemaAlteringStatement
oldType.asCQL3Type(),
validator));
cfm.comparator = cfm.comparator.setSubtype(def.position(), validatorType);
break;
case COMPACT_VALUE:
// See below
if (!validatorType.isValueCompatibleWith(cfm.getDefaultValidator()))
throw new ConfigurationException(String.format("Cannot change %s from type %s to type %s: types are incompatible.",
columnName,
cfm.getDefaultValidator().asCQL3Type(),
validator));
cfm.defaultValidator(validatorType);
break;
case REGULAR:
case STATIC:
@ -215,14 +187,6 @@ public class AlterTableStatement extends SchemaAlteringStatement
columnName,
def.type.asCQL3Type(),
validator));
// For collections, if we alter the type, we need to update the comparator too since it includes
// the type too (note that isValueCompatibleWith above has validated that the new type doesn't
// change the underlying sorting order, but we still don't want to have a discrepancy between the type
// in the comparator and the one in the ColumnDefinition as that would be dodgy).
if (validatorType.isCollection() && validatorType.isMultiCell())
cfm.comparator = cfm.comparator.addOrUpdateCollection(def.name, (CollectionType)validatorType);
break;
}
// In any case, we update the column definition
@ -231,7 +195,7 @@ public class AlterTableStatement extends SchemaAlteringStatement
case DROP:
assert columnName != null;
if (!cfm.isCQL3Table())
if (!cfm.isCQLTable())
throw new InvalidRequestException("Cannot drop columns from a non-CQL3 table");
if (def == null)
throw new InvalidRequestException(String.format("Column %s was not found in table %s", columnName, columnFamily()));
@ -244,7 +208,7 @@ public class AlterTableStatement extends SchemaAlteringStatement
case REGULAR:
case STATIC:
ColumnDefinition toDelete = null;
for (ColumnDefinition columnDef : cfm.regularAndStaticColumns())
for (ColumnDefinition columnDef : cfm.partitionColumns())
{
if (columnDef.name.equals(columnName))
{

41
src/java/org/apache/cassandra/cql3/statements/AlterTypeStatement.java
View File

@ -23,7 +23,7 @@ import java.util.*;
import org.apache.cassandra.auth.Permission;
import org.apache.cassandra.config.*;
import org.apache.cassandra.cql3.*;
import org.apache.cassandra.db.composites.CellNames;
import org.apache.cassandra.db.ClusteringComparator;
import org.apache.cassandra.db.marshal.*;
import org.apache.cassandra.exceptions.*;
import org.apache.cassandra.service.ClientState;
@ -150,28 +150,6 @@ public abstract class AlterTypeStatement extends SchemaAlteringStatement
// We need to update this validator ...
cfm.addOrReplaceColumnDefinition(def.withNewType(t));
// ... but if it's part of the comparator or key validator, we need to go update those too.
switch (def.kind)
{
case PARTITION_KEY:
cfm.keyValidator(updateWith(cfm.getKeyValidator(), keyspace, toReplace, updated));
break;
case CLUSTERING_COLUMN:
cfm.comparator = CellNames.fromAbstractType(updateWith(cfm.comparator.asAbstractType(), keyspace, toReplace, updated), cfm.comparator.isDense());
break;
default:
// If it's a collection, we still want to modify the comparator because the collection is aliased in it
if (def.type instanceof CollectionType && def.type.isMultiCell())
{
t = updateWith(cfm.comparator.asAbstractType(), keyspace, toReplace, updated);
// If t == null, all relevant comparators were updated via updateWith, which reaches into types and
// collections
if (t != null)
cfm.comparator = CellNames.fromAbstractType(t, cfm.comparator.isDense());
}
break;
}
return true;
}
@ -203,23 +181,6 @@ public abstract class AlterTypeStatement extends SchemaAlteringStatement
List<AbstractType<?>> updatedTypes = updateTypes(ct.types, keyspace, toReplace, updated);
return updatedTypes == null ? null : CompositeType.getInstance(updatedTypes);
}
else if (type instanceof ColumnToCollectionType)
{
ColumnToCollectionType ctct = (ColumnToCollectionType)type;
Map<ByteBuffer, CollectionType> updatedTypes = null;
for (Map.Entry<ByteBuffer, CollectionType> entry : ctct.defined.entrySet())
{
AbstractType<?> t = updateWith(entry.getValue(), keyspace, toReplace, updated);
if (t == null)
continue;
if (updatedTypes == null)
updatedTypes = new HashMap<>(ctct.defined);
updatedTypes.put(entry.getKey(), (CollectionType)t);
}
return updatedTypes == null ? null : ColumnToCollectionType.getInstance(updatedTypes);
}
else if (type instanceof CollectionType)
{
if (type instanceof ListType)

173
src/java/org/apache/cassandra/cql3/statements/BatchStatement.java
View File

@ -32,7 +32,8 @@ import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.cql3.*;
import org.apache.cassandra.db.*;
import org.apache.cassandra.db.composites.Composite;
import org.apache.cassandra.db.rows.RowIterator;
import org.apache.cassandra.db.partitions.*;
import org.apache.cassandra.exceptions.*;
import org.apache.cassandra.service.ClientState;
import org.apache.cassandra.service.ClientWarn;
@ -40,6 +41,7 @@ import org.apache.cassandra.service.QueryState;
import org.apache.cassandra.service.StorageProxy;
import org.apache.cassandra.thrift.Column;
import org.apache.cassandra.tracing.Tracing;
import org.apache.cassandra.service.StorageService;
import org.apache.cassandra.transport.messages.ResultMessage;
import org.apache.cassandra.utils.NoSpamLogger;
import org.apache.cassandra.utils.Pair;
@ -57,6 +59,10 @@ public class BatchStatement implements CQLStatement
private final int boundTerms;
public final Type type;
private final List<ModificationStatement> statements;
private final PartitionColumns updatedColumns;
private final PartitionColumns conditionColumns;
private final boolean updatesRegularRows;
private final boolean updatesStaticRow;
private final Attributes attrs;
private final boolean hasConditions;
private static final Logger logger = LoggerFactory.getLogger(BatchStatement.class);
@ -72,14 +78,33 @@ public class BatchStatement implements CQLStatement
*/
public BatchStatement(int boundTerms, Type type, List<ModificationStatement> statements, Attributes attrs)
{
boolean hasConditions = false;
for (ModificationStatement statement : statements)
hasConditions |= statement.hasConditions();
this.boundTerms = boundTerms;
this.type = type;
this.statements = statements;
this.attrs = attrs;
boolean hasConditions = false;
PartitionColumns.Builder regularBuilder = PartitionColumns.builder();
PartitionColumns.Builder conditionBuilder = PartitionColumns.builder();
boolean updateRegular = false;
boolean updateStatic = false;
for (ModificationStatement stmt : statements)
{
regularBuilder.addAll(stmt.updatedColumns());
updateRegular |= stmt.updatesRegularRows();
if (stmt.hasConditions())
{
hasConditions = true;
conditionBuilder.addAll(stmt.conditionColumns());
updateStatic |= stmt.updatesStaticRow();
}
}
this.updatedColumns = regularBuilder.build();
this.conditionColumns = conditionBuilder.build();
this.updatesRegularRows = updateRegular;
this.updatesStaticRow = updateStatic;
this.hasConditions = hasConditions;
}
@ -199,6 +224,18 @@ public class BatchStatement implements CQLStatement
return ms;
}
private PartitionColumns updatedColumns()
{
return updatedColumns;
}
private int updatedRows()
{
// Note: it's possible for 2 statements to actually apply to the same row, but that's just an estimation
// for sizing our PartitionUpdate backing array, so it's good enough.
return statements.size();
}
private void addStatementMutations(ModificationStatement statement,
QueryOptions options,
boolean local,
@ -218,25 +255,33 @@ public class BatchStatement implements CQLStatement
// we don't want to recreate mutations every time as this is particularly inefficient when applying
// multiple batch to the same partition (see #6737).
List<ByteBuffer> keys = statement.buildPartitionKeyNames(options);
Composite clusteringPrefix = statement.createClusteringPrefix(options);
UpdateParameters params = statement.makeUpdateParameters(keys, clusteringPrefix, options, local, now);
CBuilder clustering = statement.createClustering(options);
UpdateParameters params = statement.makeUpdateParameters(keys, clustering, options, local, now);
for (ByteBuffer key : keys)
{
IMutation mutation = ksMap.get(key);
DecoratedKey dk = StorageService.getPartitioner().decorateKey(key);
IMutation mutation = ksMap.get(dk.getKey());
Mutation mut;
if (mutation == null)
{
mut = new Mutation(ksName, key);
mut = new Mutation(ksName, dk);
mutation = statement.cfm.isCounter() ? new CounterMutation(mut, options.getConsistency()) : mut;
ksMap.put(key, mutation);
ksMap.put(dk.getKey(), mutation);
}
else
{
mut = statement.cfm.isCounter() ? ((CounterMutation) mutation).getMutation() : (Mutation) mutation;
}
statement.addUpdateForKey(mut.addOrGet(statement.cfm), key, clusteringPrefix, params);
PartitionUpdate upd = mut.get(statement.cfm);
if (upd == null)
{
upd = new PartitionUpdate(statement.cfm, dk, updatedColumns(), updatedRows());
mut.add(upd);
}
statement.addUpdateForKey(upd, clustering, params);
}
}
@ -245,56 +290,55 @@ public class BatchStatement implements CQLStatement
*
* @param cfs ColumnFamilies that will store the batch's mutations.
*/
public static void verifyBatchSize(Iterable<ColumnFamily> cfs) throws InvalidRequestException
public static void verifyBatchSize(Iterable<PartitionUpdate> updates) throws InvalidRequestException
{
long size = 0;
long warnThreshold = DatabaseDescriptor.getBatchSizeWarnThreshold();
long failThreshold = DatabaseDescriptor.getBatchSizeFailThreshold();
for (ColumnFamily cf : cfs)
size += cf.dataSize();
for (PartitionUpdate update : updates)
size += update.dataSize();
if (size > warnThreshold)
{
Set<String> ksCfPairs = new HashSet<>();
for (ColumnFamily cf : cfs)
ksCfPairs.add(String.format("%s.%s", cf.metadata().ksName, cf.metadata().cfName));
Set<String> tableNames = new HashSet<>();
for (PartitionUpdate update : updates)
tableNames.add(String.format("%s.%s", update.metadata().ksName, update.metadata().cfName));
String format = "Batch of prepared statements for {} is of size {}, exceeding specified threshold of {} by {}.{}";
if (size > failThreshold)
{
Tracing.trace(format, ksCfPairs, size, failThreshold, size - failThreshold, " (see batch_size_fail_threshold_in_kb)");
logger.error(format, ksCfPairs, size, failThreshold, size - failThreshold, " (see batch_size_fail_threshold_in_kb)");
Tracing.trace(format, tableNames, size, failThreshold, size - failThreshold, " (see batch_size_fail_threshold_in_kb)");
logger.error(format, tableNames, size, failThreshold, size - failThreshold, " (see batch_size_fail_threshold_in_kb)");
throw new InvalidRequestException("Batch too large");
}
else if (logger.isWarnEnabled())
{
logger.warn(format, ksCfPairs, size, warnThreshold, size - warnThreshold, "");
logger.warn(format, tableNames, size, warnThreshold, size - warnThreshold, "");
}
ClientWarn.warn(MessageFormatter.arrayFormat(format, new Object[] {ksCfPairs, size, warnThreshold, size - warnThreshold, ""}).getMessage());
ClientWarn.warn(MessageFormatter.arrayFormat(format, new Object[] {tableNames, size, warnThreshold, size - warnThreshold, ""}).getMessage());
}
}
private void verifyBatchType(Collection<? extends IMutation> mutations)
private void verifyBatchType(Iterable<PartitionUpdate> updates)
{
if (type != Type.LOGGED && mutations.size() > 1)
if (type != Type.LOGGED && Iterables.size(updates) > 1)
{
Set<String> ksCfPairs = new HashSet<>();
Set<ByteBuffer> keySet = new HashSet<>();
Set<DecoratedKey> keySet = new HashSet<>();
Set<String> tableNames = new HashSet<>();
for (IMutation im : mutations)
for (PartitionUpdate update : updates)
{
keySet.add(im.key());
for (ColumnFamily cf : im.getColumnFamilies())
ksCfPairs.add(String.format("%s.%s", cf.metadata().ksName, cf.metadata().cfName));
keySet.add(update.partitionKey());
tableNames.add(String.format("%s.%s", update.metadata().ksName, update.metadata().cfName));
}
NoSpamLogger.log(logger, NoSpamLogger.Level.WARN, 1, TimeUnit.MINUTES, unloggedBatchWarning,
keySet.size(), keySet.size() == 1 ? "" : "s",
ksCfPairs.size() == 1 ? "" : "s", ksCfPairs);
tableNames.size() == 1 ? "" : "s", tableNames);
ClientWarn.warn(MessageFormatter.arrayFormat(unloggedBatchWarning, new Object[]{keySet.size(), keySet.size() == 1 ? "" : "s",
ksCfPairs.size() == 1 ? "" : "s", ksCfPairs}).getMessage());
tableNames.size() == 1 ? "" : "s", tableNames}).getMessage());
}
}
@ -326,17 +370,17 @@ public class BatchStatement implements CQLStatement
private void executeWithoutConditions(Collection<? extends IMutation> mutations, ConsistencyLevel cl) throws RequestExecutionException, RequestValidationException
{
// Extract each collection of cfs from it's IMutation and then lazily concatenate all of them into a single Iterable.
Iterable<ColumnFamily> cfs = Iterables.concat(Iterables.transform(mutations, new Function<IMutation, Collection<ColumnFamily>>()
// Extract each collection of updates from it's IMutation and then lazily concatenate all of them into a single Iterable.
Iterable<PartitionUpdate> updates = Iterables.concat(Iterables.transform(mutations, new Function<IMutation, Collection<PartitionUpdate>>()
{
public Collection<ColumnFamily> apply(IMutation im)
public Collection<PartitionUpdate> apply(IMutation im)
{
return im.getColumnFamilies();
return im.getPartitionUpdates();
}
}));
verifyBatchSize(cfs);
verifyBatchType(mutations);
verifyBatchSize(updates);
verifyBatchType(updates);
boolean mutateAtomic = (type == Type.LOGGED && mutations.size() > 1);
StorageProxy.mutateWithTriggers(mutations, cl, mutateAtomic);
@ -349,27 +393,26 @@ public class BatchStatement implements CQLStatement
CQL3CasRequest casRequest = p.left;
Set<ColumnDefinition> columnsWithConditions = p.right;
ColumnFamily result = StorageProxy.cas(casRequest.cfm.ksName,
casRequest.cfm.cfName,
casRequest.key,
casRequest,
options.getSerialConsistency(),
options.getConsistency(),
state.getClientState());
String ksName = casRequest.cfm.ksName;
String tableName = casRequest.cfm.cfName;
return new ResultMessage.Rows(ModificationStatement.buildCasResultSet(casRequest.cfm.ksName,
casRequest.key,
casRequest.cfm.cfName,
result,
columnsWithConditions,
true,
options.forStatement(0)));
try (RowIterator result = StorageProxy.cas(ksName,
tableName,
casRequest.key,
casRequest,
options.getSerialConsistency(),
options.getConsistency(),
state.getClientState()))
{
return new ResultMessage.Rows(ModificationStatement.buildCasResultSet(ksName, tableName, result, columnsWithConditions, true, options.forStatement(0)));
}
}
private Pair<CQL3CasRequest,Set<ColumnDefinition>> makeCasRequest(BatchQueryOptions options, QueryState state)
{
long now = state.getTimestamp();
ByteBuffer key = null;
DecoratedKey key = null;
CQL3CasRequest casRequest = null;
Set<ColumnDefinition> columnsWithConditions = new LinkedHashSet<>();
@ -383,25 +426,25 @@ public class BatchStatement implements CQLStatement
throw new IllegalArgumentException("Batch with conditions cannot span multiple partitions (you cannot use IN on the partition key)");
if (key == null)
{
key = pks.get(0);
casRequest = new CQL3CasRequest(statement.cfm, key, true);
key = StorageService.getPartitioner().decorateKey(pks.get(0));
casRequest = new CQL3CasRequest(statement.cfm, key, true, conditionColumns, updatesRegularRows, updatesStaticRow);
}
else if (!key.equals(pks.get(0)))
else if (!key.getKey().equals(pks.get(0)))
{
throw new InvalidRequestException("Batch with conditions cannot span multiple partitions");
}
Composite clusteringPrefix = statement.createClusteringPrefix(statementOptions);
CBuilder cbuilder = statement.createClustering(statementOptions);
if (statement.hasConditions())
{
statement.addConditions(clusteringPrefix, casRequest, statementOptions);
statement.addConditions(cbuilder.build(), casRequest, statementOptions);
// As soon as we have a ifNotExists, we set columnsWithConditions to null so that everything is in the resultSet
if (statement.hasIfNotExistCondition() || statement.hasIfExistCondition())
columnsWithConditions = null;
else if (columnsWithConditions != null)
Iterables.addAll(columnsWithConditions, statement.getColumnsWithConditions());
}
casRequest.addRowUpdate(clusteringPrefix, statement, statementOptions, timestamp);
casRequest.addRowUpdate(cbuilder, statement, statementOptions, timestamp);
}
return Pair.create(casRequest, columnsWithConditions);
@ -436,15 +479,13 @@ public class BatchStatement implements CQLStatement
CQL3CasRequest request = p.left;
Set<ColumnDefinition> columnsWithConditions = p.right;
ColumnFamily result = ModificationStatement.casInternal(request, state);
String ksName = request.cfm.ksName;
String tableName = request.cfm.cfName;
return new ResultMessage.Rows(ModificationStatement.buildCasResultSet(request.cfm.ksName,
request.key,
request.cfm.cfName,
result,
columnsWithConditions,
true,
options.forStatement(0)));
try (RowIterator result = ModificationStatement.casInternal(request, state))
{
return new ResultMessage.Rows(ModificationStatement.buildCasResultSet(ksName, tableName, result, columnsWithConditions, true, options.forStatement(0)));
}
}
public interface BatchVariables

184
src/java/org/apache/cassandra/cql3/statements/CQL3CasRequest.java
View File

@ -25,8 +25,8 @@ import com.google.common.collect.Multimap;
import org.apache.cassandra.cql3.*;
import org.apache.cassandra.config.CFMetaData;
import org.apache.cassandra.db.*;
import org.apache.cassandra.db.composites.Composite;
import org.apache.cassandra.db.filter.*;
import org.apache.cassandra.db.partitions.*;
import org.apache.cassandra.exceptions.InvalidRequestException;
import org.apache.cassandra.service.CASRequest;
import org.apache.cassandra.utils.Pair;
@ -36,37 +36,45 @@ import org.apache.cassandra.utils.Pair;
*/
public class CQL3CasRequest implements CASRequest
{
final CFMetaData cfm;
final ByteBuffer key;
final long now;
final boolean isBatch;
public final CFMetaData cfm;
public final DecoratedKey key;
public final boolean isBatch;
private final PartitionColumns conditionColumns;
private final boolean updatesRegularRows;
private final boolean updatesStaticRow;
private boolean hasExists; // whether we have an exist or if not exist condition
// We index RowCondition by the prefix of the row they applied to for 2 reasons:
// We index RowCondition by the clustering of the row they applied to for 2 reasons:
// 1) this allows to keep things sorted to build the ColumnSlice array below
// 2) this allows to detect when contradictory conditions are set (not exists with some other conditions on the same row)
private final SortedMap<Composite, RowCondition> conditions;
private final SortedMap<Clustering, RowCondition> conditions;
private final List<RowUpdate> updates = new ArrayList<>();
public CQL3CasRequest(CFMetaData cfm, ByteBuffer key, boolean isBatch)
public CQL3CasRequest(CFMetaData cfm,
DecoratedKey key,
boolean isBatch,
PartitionColumns conditionColumns,
boolean updatesRegularRows,
boolean updatesStaticRow)
{
this.cfm = cfm;
// When checking if conditions apply, we want to use a fixed reference time for a whole request to check
// for expired cells. Note that this is unrelated to the cell timestamp.
this.now = System.currentTimeMillis();
this.key = key;
this.conditions = new TreeMap<>(cfm.comparator);
this.isBatch = isBatch;
this.conditionColumns = conditionColumns;
this.updatesRegularRows = updatesRegularRows;
this.updatesStaticRow = updatesStaticRow;
}
public void addRowUpdate(Composite prefix, ModificationStatement stmt, QueryOptions options, long timestamp)
public void addRowUpdate(CBuilder cbuilder, ModificationStatement stmt, QueryOptions options, long timestamp)
{
updates.add(new RowUpdate(prefix, stmt, options, timestamp));
updates.add(new RowUpdate(cbuilder, stmt, options, timestamp));
}
public void addNotExist(Composite prefix) throws InvalidRequestException
public void addNotExist(Clustering clustering) throws InvalidRequestException
{
RowCondition previous = conditions.put(prefix, new NotExistCondition(prefix, now));
RowCondition previous = conditions.put(clustering, new NotExistCondition(clustering));
if (previous != null && !(previous instanceof NotExistCondition))
{
// these should be prevented by the parser, but it doesn't hurt to check
@ -75,23 +83,25 @@ public class CQL3CasRequest implements CASRequest
else
throw new InvalidRequestException("Cannot mix IF conditions and IF NOT EXISTS for the same row");
}
hasExists = true;
}
public void addExist(Composite prefix) throws InvalidRequestException
public void addExist(Clustering clustering) throws InvalidRequestException
{
RowCondition previous = conditions.put(prefix, new ExistCondition(prefix, now));
RowCondition previous = conditions.put(clustering, new ExistCondition(clustering));
// this should be prevented by the parser, but it doesn't hurt to check
if (previous instanceof NotExistCondition)
throw new InvalidRequestException("Cannot mix IF EXISTS and IF NOT EXISTS conditions for the same row");
hasExists = true;
}
public void addConditions(Composite prefix, Collection<ColumnCondition> conds, QueryOptions options) throws InvalidRequestException
public void addConditions(Clustering clustering, Collection<ColumnCondition> conds, QueryOptions options) throws InvalidRequestException
{
RowCondition condition = conditions.get(prefix);
RowCondition condition = conditions.get(clustering);
if (condition == null)
{
condition = new ColumnsConditions(prefix, now);
conditions.put(prefix, condition);
condition = new ColumnsConditions(clustering);
conditions.put(clustering, condition);
}
else if (!(condition instanceof ColumnsConditions))
{
@ -100,24 +110,43 @@ public class CQL3CasRequest implements CASRequest
((ColumnsConditions)condition).addConditions(conds, options);
}
public IDiskAtomFilter readFilter()
private PartitionColumns columnsToRead()
{
// If all our conditions are columns conditions (IF x = ?), then it's enough to query
// the columns from the conditions. If we have a IF EXISTS or IF NOT EXISTS however,
// we need to query all columns for the row since if the condition fails, we want to
// return everything to the user. Static columns make this a bit more complex, in that
// if an insert only static columns, then the existence condition applies only to the
// static columns themselves, and so we don't want to include regular columns in that
// case.
if (hasExists)
{
PartitionColumns allColumns = cfm.partitionColumns();
Columns statics = updatesStaticRow ? allColumns.statics : Columns.NONE;
Columns regulars = updatesRegularRows ? allColumns.regulars : Columns.NONE;
return new PartitionColumns(statics, regulars);
}
return conditionColumns;
}
public SinglePartitionReadCommand readCommand(int nowInSec)
{
assert !conditions.isEmpty();
ColumnSlice[] slices = new ColumnSlice[conditions.size()];
int i = 0;
Slices.Builder builder = new Slices.Builder(cfm.comparator, conditions.size());
// We always read CQL rows entirely as on CAS failure we want to be able to distinguish between "row exists
// but all values for which there were conditions are null" and "row doesn't exists", and we can't rely on the
// row marker for that (see #6623)
for (Composite prefix : conditions.keySet())
slices[i++] = prefix.slice();
for (Clustering clustering : conditions.keySet())
{
if (clustering != Clustering.STATIC_CLUSTERING)
builder.add(Slice.make(clustering));
}
int toGroup = cfm.comparator.isDense() ? -1 : cfm.clusteringColumns().size();
slices = ColumnSlice.deoverlapSlices(slices, cfm.comparator);
assert ColumnSlice.validateSlices(slices, cfm.comparator, false);
return new SliceQueryFilter(slices, false, slices.length, toGroup);
ClusteringIndexSliceFilter filter = new ClusteringIndexSliceFilter(builder.build(), false);
return SinglePartitionReadCommand.create(cfm, nowInSec, key, ColumnFilter.selection(columnsToRead()), filter);
}
public boolean appliesTo(ColumnFamily current) throws InvalidRequestException
public boolean appliesTo(FilteredPartition current) throws InvalidRequestException
{
for (RowCondition condition : conditions.values())
{
@ -127,16 +156,24 @@ public class CQL3CasRequest implements CASRequest
return true;
}
public ColumnFamily makeUpdates(ColumnFamily current) throws InvalidRequestException
private PartitionColumns updatedColumns()
{
ColumnFamily cf = ArrayBackedSortedColumns.factory.create(cfm);
PartitionColumns.Builder builder = PartitionColumns.builder();
for (RowUpdate upd : updates)
upd.applyUpdates(current, cf);
builder.addAll(upd.stmt.updatedColumns());
return builder.build();
}
public PartitionUpdate makeUpdates(FilteredPartition current) throws InvalidRequestException
{
PartitionUpdate update = new PartitionUpdate(cfm, key, updatedColumns(), conditions.size());
for (RowUpdate upd : updates)
upd.applyUpdates(current, update);
if (isBatch)
BatchStatement.verifyBatchSize(Collections.singleton(cf));
BatchStatement.verifyBatchSize(Collections.singleton(update));
return cf;
return update;
}
/**
@ -147,89 +184,62 @@ public class CQL3CasRequest implements CASRequest
*/
private class RowUpdate
{
private final Composite rowPrefix;
private final CBuilder cbuilder;
private final ModificationStatement stmt;
private final QueryOptions options;
private final long timestamp;
private RowUpdate(Composite rowPrefix, ModificationStatement stmt, QueryOptions options, long timestamp)
private RowUpdate(CBuilder cbuilder, ModificationStatement stmt, QueryOptions options, long timestamp)
{
this.rowPrefix = rowPrefix;
this.cbuilder = cbuilder;
this.stmt = stmt;
this.options = options;
this.timestamp = timestamp;
}
public void applyUpdates(ColumnFamily current, ColumnFamily updates) throws InvalidRequestException
public void applyUpdates(FilteredPartition current, PartitionUpdate updates) throws InvalidRequestException
{
Map<ByteBuffer, CQL3Row> map = null;
if (stmt.requiresRead())
{
// Uses the "current" values read by Paxos for lists operation that requires a read
Iterator<CQL3Row> iter = cfm.comparator.CQL3RowBuilder(cfm, now).group(current.iterator(new ColumnSlice[]{ rowPrefix.slice() }));
if (iter.hasNext())
{
map = Collections.singletonMap(key, iter.next());
assert !iter.hasNext() : "We shoudn't be updating more than one CQL row per-ModificationStatement";
}
}
UpdateParameters params = new UpdateParameters(cfm, options, timestamp, stmt.getTimeToLive(options), map);
stmt.addUpdateForKey(updates, key, rowPrefix, params);
Map<DecoratedKey, Partition> map = stmt.requiresRead() ? Collections.<DecoratedKey, Partition>singletonMap(key, current) : null;
UpdateParameters params = new UpdateParameters(cfm, options, timestamp, stmt.getTimeToLive(options), map, true);
stmt.addUpdateForKey(updates, cbuilder, params);
}
}
private static abstract class RowCondition
{
public final Composite rowPrefix;
protected final long now;
public final Clustering clustering;
protected RowCondition(Composite rowPrefix, long now)
protected RowCondition(Clustering clustering)
{
this.rowPrefix = rowPrefix;
this.now = now;
this.clustering = clustering;
}
public abstract boolean appliesTo(ColumnFamily current) throws InvalidRequestException;
public abstract boolean appliesTo(FilteredPartition current) throws InvalidRequestException;
}
private static class NotExistCondition extends RowCondition
{
private NotExistCondition(Composite rowPrefix, long now)
private NotExistCondition(Clustering clustering)
{
super(rowPrefix, now);
super(clustering);
}
public boolean appliesTo(ColumnFamily current)
public boolean appliesTo(FilteredPartition current)
{
if (current == null)
return true;
Iterator<Cell> iter = current.iterator(new ColumnSlice[]{ rowPrefix.slice() });
while (iter.hasNext())
if (iter.next().isLive(now))
return false;
return true;
return current == null || current.getRow(clustering) == null;
}
}
private static class ExistCondition extends RowCondition
{
private ExistCondition(Composite rowPrefix, long now)
private ExistCondition(Clustering clustering)
{
super (rowPrefix, now);
super(clustering);
}
public boolean appliesTo(ColumnFamily current)
public boolean appliesTo(FilteredPartition current)
{
if (current == null)
return false;
Iterator<Cell> iter = current.iterator(new ColumnSlice[]{ rowPrefix.slice() });
while (iter.hasNext())
if (iter.next().isLive(now))
return true;
return false;
return current != null && current.getRow(clustering) != null;
}
}
@ -237,9 +247,9 @@ public class CQL3CasRequest implements CASRequest
{
private final Multimap<Pair<ColumnIdentifier, ByteBuffer>, ColumnCondition.Bound> conditions = HashMultimap.create();
private ColumnsConditions(Composite rowPrefix, long now)
private ColumnsConditions(Clustering clustering)
{
super(rowPrefix, now);
super(clustering);
}
public void addConditions(Collection<ColumnCondition> conds, QueryOptions options) throws InvalidRequestException
@ -251,14 +261,16 @@ public class CQL3CasRequest implements CASRequest
}
}
public boolean appliesTo(ColumnFamily current) throws InvalidRequestException
public boolean appliesTo(FilteredPartition current) throws InvalidRequestException
{
if (current == null)
return conditions.isEmpty();
for (ColumnCondition.Bound condition : conditions.values())
if (!condition.appliesTo(rowPrefix, current, now))
{
if (!condition.appliesTo(current.getRow(clustering)))
return false;
}
return true;
}
}

18
src/java/org/apache/cassandra/cql3/statements/CreateIndexStatement.java
View File

@ -111,12 +111,14 @@ public class CreateIndexStatement extends SchemaAlteringStatement
properties.validate();
// TODO: we could lift that limitation
if ((cfm.comparator.isDense() || !cfm.comparator.isCompound()) && cd.isPrimaryKeyColumn())
throw new InvalidRequestException("Secondary indexes are not supported on PRIMARY KEY columns in COMPACT STORAGE tables");
if (cd.kind == ColumnDefinition.Kind.COMPACT_VALUE)
throw new InvalidRequestException("Secondary indexes are not supported on COMPACT STORAGE tables that have clustering columns");
if (cfm.isCompactTable())
{
if (!cfm.isStaticCompactTable())
throw new InvalidRequestException("Secondary indexes are not supported on COMPACT STORAGE tables that have clustering columns");
else if (cd.isPrimaryKeyColumn())
// TODO: we could lift that limitation
throw new InvalidRequestException("Secondary indexes are not supported on PRIMARY KEY columns in COMPACT STORAGE tables");
}
// It would be possible to support 2ndary index on static columns (but not without modifications of at least ExtendedFilter and
// CompositesIndex) and maybe we should, but that means a query like:
@ -124,7 +126,7 @@ public class CreateIndexStatement extends SchemaAlteringStatement
// would pull the full partition every time the static column of partition is 'bar', which sounds like offering a
// fair potential for foot-shooting, so I prefer leaving that to a follow up ticket once we have identified cases where
// such indexing is actually useful.
if (cd.isStatic())
if (!cfm.isCompactTable() && cd.isStatic())
throw new InvalidRequestException("Secondary indexes are not allowed on static columns");
if (cd.kind == ColumnDefinition.Kind.PARTITION_KEY && cd.isOnAllComponents())
@ -174,7 +176,7 @@ public class CreateIndexStatement extends SchemaAlteringStatement
{
cd.setIndexType(IndexType.CUSTOM, properties.getOptions());
}
else if (cfm.comparator.isCompound())
else if (cfm.isCompound())
{
Map<String, String> options = Collections.emptyMap();
// For now, we only allow indexing values for collections, but we could later allow

300
src/java/org/apache/cassandra/cql3/statements/CreateTableStatement.java
View File

@ -26,11 +26,8 @@ import org.apache.commons.lang3.StringUtils;
import org.apache.cassandra.auth.*;
import org.apache.cassandra.config.*;
import org.apache.cassandra.cql3.CFName;
import org.apache.cassandra.cql3.CQL3Type;
import org.apache.cassandra.cql3.ColumnIdentifier;
import org.apache.cassandra.db.ColumnFamilyType;
import org.apache.cassandra.db.composites.*;
import org.apache.cassandra.cql3.*;
import org.apache.cassandra.db.*;
import org.apache.cassandra.db.marshal.*;
import org.apache.cassandra.exceptions.*;
import org.apache.cassandra.io.compress.CompressionParameters;
@ -43,15 +40,18 @@ import org.apache.cassandra.utils.ByteBufferUtil;
/** A <code>CREATE TABLE</code> parsed from a CQL query statement. */
public class CreateTableStatement extends SchemaAlteringStatement
{
public CellNameType comparator;
private AbstractType<?> defaultValidator;
private AbstractType<?> keyValidator;
private List<AbstractType<?>> keyTypes;
private List<AbstractType<?>> clusteringTypes;
private final List<ByteBuffer> keyAliases = new ArrayList<ByteBuffer>();
private final List<ByteBuffer> columnAliases = new ArrayList<ByteBuffer>();
private Map<ByteBuffer, CollectionType> collections = new HashMap<>();
private final List<ColumnIdentifier> keyAliases = new ArrayList<>();
private final List<ColumnIdentifier> columnAliases = new ArrayList<>();
private ByteBuffer valueAlias;
private boolean isDense;
private boolean isCompound;
private boolean hasCounters;
// use a TreeMap to preserve ordering across JDK versions (see CASSANDRA-9492)
private final Map<ColumnIdentifier, AbstractType> columns = new TreeMap<>(new Comparator<ColumnIdentifier>()
@ -90,22 +90,6 @@ public class CreateTableStatement extends SchemaAlteringStatement
// validated in announceMigration()
}
// Column definitions
private List<ColumnDefinition> getColumns(CFMetaData cfm)
{
List<ColumnDefinition> columnDefs = new ArrayList<>(columns.size());
Integer componentIndex = comparator.isCompound() ? comparator.clusteringPrefixSize() : null;
for (Map.Entry<ColumnIdentifier, AbstractType> col : columns.entrySet())
{
ColumnIdentifier id = col.getKey();
columnDefs.add(staticColumns.contains(id)
? ColumnDefinition.staticDef(cfm, col.getKey().bytes, col.getValue(), componentIndex)
: ColumnDefinition.regularDef(cfm, col.getKey().bytes, col.getValue(), componentIndex));
}
return columnDefs;
}
public boolean announceMigration(boolean isLocalOnly) throws RequestValidationException
{
try
@ -142,6 +126,46 @@ public class CreateTableStatement extends SchemaAlteringStatement
}
}
public CFMetaData.Builder metadataBuilder()
{
CFMetaData.Builder builder = CFMetaData.Builder.create(keyspace(), columnFamily(), isDense, isCompound, hasCounters);
for (int i = 0; i < keyAliases.size(); i++)
builder.addPartitionKey(keyAliases.get(i), keyTypes.get(i));
for (int i = 0; i < columnAliases.size(); i++)
builder.addClusteringColumn(columnAliases.get(i), clusteringTypes.get(i));
boolean isStaticCompact = !isDense && !isCompound;
for (Map.Entry<ColumnIdentifier, AbstractType> entry : columns.entrySet())
{
ColumnIdentifier name = entry.getKey();
// Note that for "static" no-clustering compact storage we use static for the defined columns
if (staticColumns.contains(name) || isStaticCompact)
builder.addStaticColumn(name, entry.getValue());
else
builder.addRegularColumn(name, entry.getValue());
}
boolean isCompactTable = isDense || !isCompound;
if (isCompactTable)
{
CompactTables.DefaultNames names = CompactTables.defaultNameGenerator(builder.usedColumnNames());
// Compact tables always have a clustering and a single regular value.
if (isStaticCompact)
{
builder.addClusteringColumn(names.defaultClusteringName(), UTF8Type.instance);
builder.addRegularColumn(names.defaultCompactValueName(), hasCounters ? CounterColumnType.instance : BytesType.instance);
}
else if (isDense && !builder.hasRegulars())
{
// Even for dense, we might not have our regular column if it wasn't part of the declaration. If
// that's the case, add it but with a specific EmptyType so we can recognize that case later
builder.addRegularColumn(names.defaultCompactValueName(), EmptyType.instance);
}
}
return builder;
}
/**
* Returns a CFMetaData instance based on the parameters parsed from this
* <code>CREATE</code> statement, or defaults where applicable.
@ -151,48 +175,16 @@ public class CreateTableStatement extends SchemaAlteringStatement
*/
public CFMetaData getCFMetaData() throws RequestValidationException
{
CFMetaData newCFMD;
newCFMD = new CFMetaData(keyspace(),
columnFamily(),
ColumnFamilyType.Standard,
comparator);
CFMetaData newCFMD = metadataBuilder().build();
applyPropertiesTo(newCFMD);
return newCFMD;
}
public void applyPropertiesTo(CFMetaData cfmd) throws RequestValidationException
{
cfmd.defaultValidator(defaultValidator)
.keyValidator(keyValidator)
.addAllColumnDefinitions(getColumns(cfmd))
.isDense(isDense);
addColumnMetadataFromAliases(cfmd, keyAliases, keyValidator, ColumnDefinition.Kind.PARTITION_KEY);
addColumnMetadataFromAliases(cfmd, columnAliases, comparator.asAbstractType(), ColumnDefinition.Kind.CLUSTERING_COLUMN);
if (valueAlias != null)
addColumnMetadataFromAliases(cfmd, Collections.singletonList(valueAlias), defaultValidator, ColumnDefinition.Kind.COMPACT_VALUE);
properties.applyToCFMetadata(cfmd);
}
private void addColumnMetadataFromAliases(CFMetaData cfm, List<ByteBuffer> aliases, AbstractType<?> comparator, ColumnDefinition.Kind kind)
{
if (comparator instanceof CompositeType)
{
CompositeType ct = (CompositeType)comparator;
for (int i = 0; i < aliases.size(); ++i)
if (aliases.get(i) != null)
cfm.addOrReplaceColumnDefinition(new ColumnDefinition(cfm, aliases.get(i), ct.types.get(i), i, kind));
}
else
{
assert aliases.size() <= 1;
if (!aliases.isEmpty() && aliases.get(0) != null)
cfm.addOrReplaceColumnDefinition(new ColumnDefinition(cfm, aliases.get(0), comparator, null, kind));
}
}
public static class RawStatement extends CFStatement
{
private final Map<ColumnIdentifier, CQL3Type.Raw> definitions = new HashMap<>();
@ -233,169 +225,107 @@ public class CreateTableStatement extends SchemaAlteringStatement
CreateTableStatement stmt = new CreateTableStatement(cfName, properties, ifNotExists, staticColumns);
boolean hasCounters = false;
Map<ByteBuffer, CollectionType> definedMultiCellCollections = null;
for (Map.Entry<ColumnIdentifier, CQL3Type.Raw> entry : definitions.entrySet())
{
ColumnIdentifier id = entry.getKey();
CQL3Type pt = entry.getValue().prepare(keyspace());
if (pt.isCollection() && ((CollectionType) pt.getType()).isMultiCell())
{
if (definedMultiCellCollections == null)
definedMultiCellCollections = new HashMap<>();
definedMultiCellCollections.put(id.bytes, (CollectionType) pt.getType());
}
else if (entry.getValue().isCounter())
hasCounters = true;
if (pt.isCollection() && ((CollectionType)pt.getType()).isMultiCell())
stmt.collections.put(id.bytes, (CollectionType)pt.getType());
if (entry.getValue().isCounter())
stmt.hasCounters = true;
stmt.columns.put(id, pt.getType()); // we'll remove what is not a column below
}
if (keyAliases.isEmpty())
throw new InvalidRequestException("No PRIMARY KEY specifed (exactly one required)");
else if (keyAliases.size() > 1)
if (keyAliases.size() > 1)
throw new InvalidRequestException("Multiple PRIMARY KEYs specifed (exactly one required)");
else if (hasCounters && properties.getDefaultTimeToLive() > 0)
if (stmt.hasCounters && properties.getDefaultTimeToLive() > 0)
throw new InvalidRequestException("Cannot set default_time_to_live on a table with counters");
List<ColumnIdentifier> kAliases = keyAliases.get(0);
List<AbstractType<?>> keyTypes = new ArrayList<AbstractType<?>>(kAliases.size());
stmt.keyTypes = new ArrayList<AbstractType<?>>(kAliases.size());
for (ColumnIdentifier alias : kAliases)
{
stmt.keyAliases.add(alias.bytes);
stmt.keyAliases.add(alias);
AbstractType<?> t = getTypeAndRemove(stmt.columns, alias);
if (t instanceof CounterColumnType)
throw new InvalidRequestException(String.format("counter type is not supported for PRIMARY KEY part %s", alias));
if (staticColumns.contains(alias))
throw new InvalidRequestException(String.format("Static column %s cannot be part of the PRIMARY KEY", alias));
keyTypes.add(t);
stmt.keyTypes.add(t);
}
stmt.keyValidator = keyTypes.size() == 1 ? keyTypes.get(0) : CompositeType.getInstance(keyTypes);
// Dense means that no part of the comparator stores a CQL column name. This means
// COMPACT STORAGE with at least one columnAliases (otherwise it's a thrift "static" CF).
stmt.isDense = useCompactStorage && !columnAliases.isEmpty();
stmt.clusteringTypes = new ArrayList<>(columnAliases.size());
// Handle column aliases
if (columnAliases.isEmpty())
for (ColumnIdentifier t : columnAliases)
{
if (useCompactStorage)
stmt.columnAliases.add(t);
AbstractType<?> type = getTypeAndRemove(stmt.columns, t);
if (type instanceof CounterColumnType)
throw new InvalidRequestException(String.format("counter type is not supported for PRIMARY KEY part %s", t));
if (staticColumns.contains(t))
throw new InvalidRequestException(String.format("Static column %s cannot be part of the PRIMARY KEY", t));
stmt.clusteringTypes.add(type);
}
// We've handled anything that is not a rpimary key so stmt.columns only contains NON-PK columns. So
// if it's a counter table, make sure we don't have non-counter types
if (stmt.hasCounters)
{
for (AbstractType<?> type : stmt.columns.values())
if (!type.isCounter())
throw new InvalidRequestException("Cannot mix counter and non counter columns in the same table");
}
// Dense means that on the thrift side, no part of the "thrift column name" stores a "CQL/metadata column name".
// This means COMPACT STORAGE with at least one clustering type (otherwise it's a thrift "static" CF).
stmt.isDense = useCompactStorage && !stmt.clusteringTypes.isEmpty();
// Compound means that on the thrift side, the "thrift column name" is a composite one. It's the case unless
// we use compact storage COMPACT STORAGE and we have either no clustering columns (thrift "static" CF) or
// only one of them (if more than one, it's a "dense composite").
stmt.isCompound = !(useCompactStorage && stmt.clusteringTypes.size() <= 1);
// For COMPACT STORAGE, we reject any "feature" that we wouldn't be able to translate back to thrift.
if (useCompactStorage)
{
if (!stmt.collections.isEmpty())
throw new InvalidRequestException("Non-frozen collection types are not supported with COMPACT STORAGE");
if (!staticColumns.isEmpty())
throw new InvalidRequestException("Static columns are not supported in COMPACT STORAGE tables");
if (stmt.clusteringTypes.isEmpty())
{
// There should remain some column definition since it is a non-composite "static" CF
// It's a thrift "static CF" so there should be some columns definition
if (stmt.columns.isEmpty())
throw new InvalidRequestException("No definition found that is not part of the PRIMARY KEY");
if (definedMultiCellCollections != null)
throw new InvalidRequestException("Non-frozen collection types are not supported with COMPACT STORAGE");
stmt.comparator = new SimpleSparseCellNameType(UTF8Type.instance);
}
else
{
stmt.comparator = definedMultiCellCollections == null
? new CompoundSparseCellNameType(Collections.<AbstractType<?>>emptyList())
: new CompoundSparseCellNameType.WithCollection(Collections.<AbstractType<?>>emptyList(), ColumnToCollectionType.getInstance(definedMultiCellCollections));
}
}
else
{
// If we use compact storage and have only one alias, it is a
// standard "dynamic" CF, otherwise it's a composite
if (useCompactStorage && columnAliases.size() == 1)
{
if (definedMultiCellCollections != null)
throw new InvalidRequestException("Collection types are not supported with COMPACT STORAGE");
ColumnIdentifier alias = columnAliases.get(0);
if (staticColumns.contains(alias))
throw new InvalidRequestException(String.format("Static column %s cannot be part of the PRIMARY KEY", alias));
stmt.columnAliases.add(alias.bytes);
AbstractType<?> at = getTypeAndRemove(stmt.columns, alias);
if (at instanceof CounterColumnType)
throw new InvalidRequestException(String.format("counter type is not supported for PRIMARY KEY part %s", stmt.columnAliases.get(0)));
stmt.comparator = new SimpleDenseCellNameType(at);
}
else
{
List<AbstractType<?>> types = new ArrayList<AbstractType<?>>(columnAliases.size() + 1);
for (ColumnIdentifier t : columnAliases)
{
stmt.columnAliases.add(t.bytes);
AbstractType<?> type = getTypeAndRemove(stmt.columns, t);
if (type instanceof CounterColumnType)
throw new InvalidRequestException(String.format("counter type is not supported for PRIMARY KEY part %s", t));
if (staticColumns.contains(t))
throw new InvalidRequestException(String.format("Static column %s cannot be part of the PRIMARY KEY", t));
types.add(type);
}
if (useCompactStorage)
{
if (definedMultiCellCollections != null)
throw new InvalidRequestException("Collection types are not supported with COMPACT STORAGE");
stmt.comparator = new CompoundDenseCellNameType(types);
}
else
{
stmt.comparator = definedMultiCellCollections == null
? new CompoundSparseCellNameType(types)
: new CompoundSparseCellNameType.WithCollection(types, ColumnToCollectionType.getInstance(definedMultiCellCollections));
}
}
}
if (!staticColumns.isEmpty())
{
// Only CQL3 tables can have static columns
if (useCompactStorage)
throw new InvalidRequestException("Static columns are not supported in COMPACT STORAGE tables");
// Static columns only make sense if we have at least one clustering column. Otherwise everything is static anyway
if (columnAliases.isEmpty())
throw new InvalidRequestException("Static columns are only useful (and thus allowed) if the table has at least one clustering column");
}
if (useCompactStorage && !stmt.columnAliases.isEmpty())
{
if (stmt.columns.isEmpty())
{
// The only value we'll insert will be the empty one, so the default validator don't matter
stmt.defaultValidator = BytesType.instance;
// We need to distinguish between
// * I'm upgrading from thrift so the valueAlias is null
// * I've defined my table with only a PK (and the column value will be empty)
// So, we use an empty valueAlias (rather than null) for the second case
stmt.valueAlias = ByteBufferUtil.EMPTY_BYTE_BUFFER;
}
else
if (stmt.isDense)
{
// We can have no columns (only the PK), but we can't have more than one.
if (stmt.columns.size() > 1)
throw new InvalidRequestException(String.format("COMPACT STORAGE with composite PRIMARY KEY allows no more than one column not part of the PRIMARY KEY (got: %s)", StringUtils.join(stmt.columns.keySet(), ", ")));
Map.Entry<ColumnIdentifier, AbstractType> lastEntry = stmt.columns.entrySet().iterator().next();
stmt.defaultValidator = lastEntry.getValue();
stmt.valueAlias = lastEntry.getKey().bytes;
stmt.columns.remove(lastEntry.getKey());
}
else
{
// we are in the "static" case, so we need at least one column defined. For non-compact however, having
// just the PK is fine.
if (stmt.columns.isEmpty())
throw new InvalidRequestException("COMPACT STORAGE with non-composite PRIMARY KEY require one column not part of the PRIMARY KEY, none given");
}
}
else
{
// For compact, we are in the "static" case, so we need at least one column defined. For non-compact however, having
// just the PK is fine since we have CQL3 row marker.
if (useCompactStorage && stmt.columns.isEmpty())
throw new InvalidRequestException("COMPACT STORAGE with non-composite PRIMARY KEY require one column not part of the PRIMARY KEY, none given");
// There is no way to insert/access a column that is not defined for non-compact storage, so
// the actual validator don't matter much (except that we want to recognize counter CF as limitation apply to them).
stmt.defaultValidator = !stmt.columns.isEmpty() && (stmt.columns.values().iterator().next() instanceof CounterColumnType)
? CounterColumnType.instance
: BytesType.instance;
if (stmt.clusteringTypes.isEmpty() && !staticColumns.isEmpty())
{
// Static columns only make sense if we have at least one clustering column. Otherwise everything is static anyway
if (columnAliases.isEmpty())
throw new InvalidRequestException("Static columns are only useful (and thus allowed) if the table has at least one clustering column");
}
}
// If we give a clustering order, we must explicitly do so for all aliases and in the order of the PK
if (!definedOrdering.isEmpty())
{

65
src/java/org/apache/cassandra/cql3/statements/DeleteStatement.java
View File

@ -17,7 +17,6 @@
*/
package org.apache.cassandra.cql3.statements;
import java.nio.ByteBuffer;
import java.util.*;
import com.google.common.collect.Iterators;
@ -27,7 +26,8 @@ import org.apache.cassandra.cql3.restrictions.Restriction;
import org.apache.cassandra.config.CFMetaData;
import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.db.*;
import org.apache.cassandra.db.composites.Composite;
import org.apache.cassandra.db.rows.Row;
import org.apache.cassandra.db.partitions.*;
import org.apache.cassandra.exceptions.*;
import org.apache.cassandra.utils.Pair;
@ -46,44 +46,57 @@ public class DeleteStatement extends ModificationStatement
return false;
}
public void addUpdateForKey(ColumnFamily cf, ByteBuffer key, Composite prefix, UpdateParameters params)
public void addUpdateForKey(PartitionUpdate update, CBuilder cbuilder, UpdateParameters params)
throws InvalidRequestException
{
List<Operation> deletions = getOperations();
List<Operation> regularDeletions = getRegularOperations();
List<Operation> staticDeletions = getStaticOperations();
if (prefix.size() < cfm.clusteringColumns().size() && !deletions.isEmpty())
if (regularDeletions.isEmpty() && staticDeletions.isEmpty())
{
// In general, we can't delete specific columns if not all clustering columns have been specified.
// However, if we delete only static colums, it's fine since we won't really use the prefix anyway.
for (Operation deletion : deletions)
if (!deletion.column.isStatic())
throw new InvalidRequestException(String.format("Primary key column '%s' must be specified in order to delete column '%s'", getFirstEmptyKey().name, deletion.column.name));
}
if (deletions.isEmpty())
{
// We delete the slice selected by the prefix.
// However, for performance reasons, we distinguish 2 cases:
// - It's a full internal row delete
// - It's a full cell name (i.e it's a dense layout and the prefix is full)
if (prefix.isEmpty())
// We're not deleting any specific columns so it's either a full partition deletion ....
if (cbuilder.count() == 0)
{
// No columns specified, delete the row
cf.delete(new DeletionInfo(params.timestamp, params.localDeletionTime));
update.addPartitionDeletion(params.deletionTime());
}
else if (cfm.comparator.isDense() && prefix.size() == cfm.clusteringColumns().size())
// ... or a row deletion ...
else if (cbuilder.remainingCount() == 0)
{
cf.addAtom(params.makeTombstone(cfm.comparator.create(prefix, null)));
Clustering clustering = cbuilder.build();
Row.Writer writer = update.writer();
params.writeClustering(clustering, writer);
params.writeRowDeletion(writer);
writer.endOfRow();
}
// ... or a range of rows deletion.
else
{
cf.addAtom(params.makeRangeTombstone(prefix.slice()));
update.addRangeTombstone(params.makeRangeTombstone(cbuilder));
}
}
else
{
for (Operation op : deletions)
op.execute(key, cf, prefix, params);
if (!regularDeletions.isEmpty())
{
// We can't delete specific (regular) columns if not all clustering columns have been specified.
if (cbuilder.remainingCount() > 0)
throw new InvalidRequestException(String.format("Primary key column '%s' must be specified in order to delete column '%s'", getFirstEmptyKey().name, regularDeletions.get(0).column.name));
Clustering clustering = cbuilder.build();
Row.Writer writer = update.writer();
params.writeClustering(clustering, writer);
for (Operation op : regularDeletions)
op.execute(update.partitionKey(), clustering, writer, params);
writer.endOfRow();
}
if (!staticDeletions.isEmpty())
{
Row.Writer writer = update.staticWriter();
for (Operation op : staticDeletions)
op.execute(update.partitionKey(), Clustering.STATIC_CLUSTERING, writer, params);
writer.endOfRow();
}
}
}

6
src/java/org/apache/cassandra/cql3/statements/DropTypeStatement.java
View File

@ -119,12 +119,6 @@ public class DropTypeStatement extends SchemaAlteringStatement
if (isUsedBy(subtype))
return true;
}
else if (toCheck instanceof ColumnToCollectionType)
{
for (CollectionType collection : ((ColumnToCollectionType)toCheck).defined.values())
if (isUsedBy(collection))
return true;
}
else if (toCheck instanceof CollectionType)
{
if (toCheck instanceof ListType)

340
src/java/org/apache/cassandra/cql3/statements/ModificationStatement.java
View File

@ -21,7 +21,8 @@ import java.nio.ByteBuffer;
import java.util.*;
import com.google.common.collect.Iterables;
import com.google.common.collect.Lists;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.cassandra.auth.Permission;
import org.apache.cassandra.config.CFMetaData;
@ -33,19 +34,20 @@ import org.apache.cassandra.cql3.restrictions.Restriction;
import org.apache.cassandra.cql3.restrictions.SingleColumnRestriction;
import org.apache.cassandra.cql3.selection.Selection;
import org.apache.cassandra.db.*;
import org.apache.cassandra.db.composites.Composite;
import org.apache.cassandra.db.composites.CompositesBuilder;
import org.apache.cassandra.db.filter.ColumnSlice;
import org.apache.cassandra.db.filter.SliceQueryFilter;
import org.apache.cassandra.db.rows.*;
import org.apache.cassandra.db.partitions.*;
import org.apache.cassandra.db.filter.*;
import org.apache.cassandra.db.marshal.BooleanType;
import org.apache.cassandra.exceptions.*;
import org.apache.cassandra.service.ClientState;
import org.apache.cassandra.service.QueryState;
import org.apache.cassandra.service.StorageProxy;
import org.apache.cassandra.service.StorageService;
import org.apache.cassandra.service.paxos.Commit;
import org.apache.cassandra.thrift.ThriftValidation;
import org.apache.cassandra.transport.messages.ResultMessage;
import org.apache.cassandra.triggers.TriggerExecutor;
import org.apache.cassandra.utils.FBUtilities;
import org.apache.cassandra.utils.Pair;
import org.apache.cassandra.utils.UUIDGen;
@ -58,6 +60,8 @@ import static org.apache.cassandra.cql3.statements.RequestValidations.invalidReq
*/
public abstract class ModificationStatement implements CQLStatement
{
protected static final Logger logger = LoggerFactory.getLogger(ModificationStatement.class);
private static final ColumnIdentifier CAS_RESULT_COLUMN = new ColumnIdentifier("[applied]", false);
public static enum StatementType { INSERT, UPDATE, DELETE }
@ -68,7 +72,15 @@ public abstract class ModificationStatement implements CQLStatement
public final Attributes attrs;
protected final Map<ColumnIdentifier, Restriction> processedKeys = new HashMap<>();
private final List<Operation> columnOperations = new ArrayList<Operation>();
private final List<Operation> regularOperations = new ArrayList<>();
private final List<Operation> staticOperations = new ArrayList<>();
// TODO: If we had a builder for this statement, we could have updatedColumns/conditionColumns final and only have
// updatedColumnsBuilder/conditionColumnsBuilder in the builder ...
private PartitionColumns updatedColumns;
private PartitionColumns.Builder updatedColumnsBuilder = PartitionColumns.builder();
private PartitionColumns conditionColumns;
private PartitionColumns.Builder conditionColumnsBuilder = PartitionColumns.builder();
// Separating normal and static conditions makes things somewhat easier
private List<ColumnCondition> columnConditions;
@ -103,25 +115,24 @@ public abstract class ModificationStatement implements CQLStatement
Iterable<Function> functions = attrs.getFunctions();
for (Restriction restriction : processedKeys.values())
functions = Iterables.concat(functions, restriction.getFunctions());
functions = Iterables.concat(functions, restriction.getFunctions());
if (columnOperations != null)
for (Operation operation : columnOperations)
functions = Iterables.concat(functions, operation.getFunctions());
for (Operation operation : allOperations())
functions = Iterables.concat(functions, operation.getFunctions());
if (columnConditions != null)
for (ColumnCondition condition : columnConditions)
functions = Iterables.concat(functions, condition.getFunctions());
if (staticConditions != null)
for (ColumnCondition condition : staticConditions)
functions = Iterables.concat(functions, condition.getFunctions());
for (ColumnCondition condition : allConditions())
functions = Iterables.concat(functions, condition.getFunctions());
return functions;
}
public boolean hasNoClusteringColumns()
{
return hasNoClusteringColumns;
}
public abstract boolean requireFullClusteringKey();
public abstract void addUpdateForKey(ColumnFamily updates, ByteBuffer key, Composite prefix, UpdateParameters params) throws InvalidRequestException;
public abstract void addUpdateForKey(PartitionUpdate update, CBuilder clusteringBuilder, UpdateParameters params) throws InvalidRequestException;
public int getBoundTerms()
{
@ -184,16 +195,75 @@ public abstract class ModificationStatement implements CQLStatement
public void addOperation(Operation op)
{
updatedColumnsBuilder.add(op.column);
// If the operation requires a read-before-write and we're doing a conditional read, we want to read
// the affected column as part of the read-for-conditions paxos phase (see #7499).
if (op.requiresRead())
conditionColumnsBuilder.add(op.column);
if (op.column.isStatic())
{
setsStaticColumns = true;
staticOperations.add(op);
}
else
{
setsRegularColumns = true;
columnOperations.add(op);
regularOperations.add(op);
}
}
public List<Operation> getOperations()
public PartitionColumns updatedColumns()
{
return columnOperations;
return updatedColumns;
}
public PartitionColumns conditionColumns()
{
return conditionColumns;
}
public boolean updatesRegularRows()
{
// We're updating regular rows if all the clustering columns are provided.
// Note that the only case where we're allowed not to provide clustering
// columns is if we set some static columns, and in that case no clustering
// columns should be given. So in practice, it's enough to check if we have
// either the table has no clustering or if it has at least one of them set.
return cfm.clusteringColumns().isEmpty() || !hasNoClusteringColumns;
}
public boolean updatesStaticRow()
{
return !staticOperations.isEmpty();
}
private void finishPreparation()
{
updatedColumns = updatedColumnsBuilder.build();
// Compact tables have not row marker. So if we don't actually update any particular column,
// this means that we're only updating the PK, which we allow if only those were declared in
// the definition. In that case however, we do went to write the compactValueColumn (since again
// we can't use a "row marker") so add it automatically.
if (cfm.isCompactTable() && updatedColumns.isEmpty() && updatesRegularRows())
updatedColumns = cfm.partitionColumns();
conditionColumns = conditionColumnsBuilder.build();
}
public List<Operation> getRegularOperations()
{
return regularOperations;
}
public List<Operation> getStaticOperations()
{
return staticOperations;
}
public Iterable<Operation> allOperations()
{
return Iterables.concat(staticOperations, regularOperations);
}
public Iterable<ColumnDefinition> getColumnsWithConditions()
@ -205,8 +275,19 @@ public abstract class ModificationStatement implements CQLStatement
staticConditions == null ? Collections.<ColumnDefinition>emptyList() : Iterables.transform(staticConditions, getColumnForCondition));
}
public Iterable<ColumnCondition> allConditions()
{
if (staticConditions == null)
return columnConditions == null ? Collections.<ColumnCondition>emptySet(): columnConditions;
if (columnConditions == null)
return staticConditions;
return Iterables.concat(staticConditions, columnConditions);
}
public void addCondition(ColumnCondition cond)
{
conditionColumnsBuilder.add(cond.column);
List<ColumnCondition> conds = null;
if (cond.column.isStatic())
{
@ -255,7 +336,7 @@ public abstract class ModificationStatement implements CQLStatement
public void addKeyValue(ColumnDefinition def, Term value) throws InvalidRequestException
{
addKeyValues(def, new SingleColumnRestriction.EQ(def, value));
addKeyValues(def, new SingleColumnRestriction.EQRestriction(def, value));
}
public void processWhereClause(List<Relation> whereClause, VariableSpecifications names) throws InvalidRequestException
@ -303,26 +384,25 @@ public abstract class ModificationStatement implements CQLStatement
public List<ByteBuffer> buildPartitionKeyNames(QueryOptions options)
throws InvalidRequestException
{
CompositesBuilder keyBuilder = new CompositesBuilder(cfm.getKeyValidatorAsCType());
MultiCBuilder keyBuilder = MultiCBuilder.create(cfm.getKeyValidatorAsClusteringComparator());
for (ColumnDefinition def : cfm.partitionKeyColumns())
{
Restriction r = checkNotNull(processedKeys.get(def.name), "Missing mandatory PRIMARY KEY part %s", def.name);
r.appendTo(keyBuilder, options);
}
return Lists.transform(keyBuilder.build(), new com.google.common.base.Function<Composite, ByteBuffer>()
NavigableSet<Clustering> clusterings = keyBuilder.build();
List<ByteBuffer> keys = new ArrayList<ByteBuffer>(clusterings.size());
for (Clustering clustering : clusterings)
{
@Override
public ByteBuffer apply(Composite composite)
{
ByteBuffer byteBuffer = composite.toByteBuffer();
ThriftValidation.validateKey(cfm, byteBuffer);
return byteBuffer;
}
});
ByteBuffer key = CFMetaData.serializePartitionKey(clustering);
ThriftValidation.validateKey(cfm, key);
keys.add(key);
}
return keys;
}
public Composite createClusteringPrefix(QueryOptions options)
public CBuilder createClustering(QueryOptions options)
throws InvalidRequestException
{
// If the only updated/deleted columns are static, then we don't need clustering columns.
@ -339,7 +419,7 @@ public abstract class ModificationStatement implements CQLStatement
{
// If we set no non-static columns, then it's fine not to have clustering columns
if (hasNoClusteringColumns)
return cfm.comparator.staticPrefix();
return CBuilder.STATIC_BUILDER;
// If we do have clustering columns however, then either it's an INSERT and the query is valid
// but we still need to build a proper prefix, or it's not an INSERT, and then we want to reject
@ -354,13 +434,15 @@ public abstract class ModificationStatement implements CQLStatement
}
}
return createClusteringPrefixBuilderInternal(options);
return createClusteringInternal(options);
}
private Composite createClusteringPrefixBuilderInternal(QueryOptions options)
private CBuilder createClusteringInternal(QueryOptions options)
throws InvalidRequestException
{
CompositesBuilder builder = new CompositesBuilder(cfm.comparator);
CBuilder builder = CBuilder.create(cfm.comparator);
MultiCBuilder multiBuilder = MultiCBuilder.wrap(builder);
ColumnDefinition firstEmptyKey = null;
for (ColumnDefinition def : cfm.clusteringColumns())
{
@ -368,7 +450,7 @@ public abstract class ModificationStatement implements CQLStatement
if (r == null)
{
firstEmptyKey = def;
checkFalse(requireFullClusteringKey() && !cfm.comparator.isDense() && cfm.comparator.isCompound(),
checkFalse(requireFullClusteringKey() && !cfm.isDense() && cfm.isCompound(),
"Missing mandatory PRIMARY KEY part %s", def.name);
}
else if (firstEmptyKey != null)
@ -377,10 +459,10 @@ public abstract class ModificationStatement implements CQLStatement
}
else
{
r.appendTo(builder, options);
r.appendTo(multiBuilder, options);
}
}
return builder.build().get(0); // We only allow IN for row keys so far
return builder;
}
protected ColumnDefinition getFirstEmptyKey()
@ -396,14 +478,14 @@ public abstract class ModificationStatement implements CQLStatement
public boolean requiresRead()
{
// Lists SET operation incurs a read.
for (Operation op : columnOperations)
for (Operation op : allOperations())
if (op.requiresRead())
return true;
return false;
}
protected Map<ByteBuffer, CQL3Row> readRequiredRows(Collection<ByteBuffer> partitionKeys, Composite clusteringPrefix, boolean local, ConsistencyLevel cl)
protected Map<DecoratedKey, Partition> readRequiredLists(Collection<ByteBuffer> partitionKeys, CBuilder cbuilder, boolean local, ConsistencyLevel cl)
throws RequestExecutionException, RequestValidationException
{
if (!requiresRead())
@ -418,32 +500,54 @@ public abstract class ModificationStatement implements CQLStatement
throw new InvalidRequestException(String.format("Write operation require a read but consistency %s is not supported on reads", cl));
}
ColumnSlice[] slices = new ColumnSlice[]{ clusteringPrefix.slice() };
List<ReadCommand> commands = new ArrayList<ReadCommand>(partitionKeys.size());
long now = System.currentTimeMillis();
// TODO: no point in recomputing that every time. Should move to preparation phase.
PartitionColumns.Builder builder = PartitionColumns.builder();
for (Operation op : allOperations())
if (op.requiresRead())
builder.add(op.column);
PartitionColumns toRead = builder.build();
NavigableSet<Clustering> clusterings = FBUtilities.singleton(cbuilder.build(), cfm.comparator);
List<SinglePartitionReadCommand<?>> commands = new ArrayList<>(partitionKeys.size());
int nowInSec = FBUtilities.nowInSeconds();
for (ByteBuffer key : partitionKeys)
commands.add(new SliceFromReadCommand(keyspace(),
key,
columnFamily(),
now,
new SliceQueryFilter(slices, false, Integer.MAX_VALUE)));
commands.add(new SinglePartitionNamesCommand(cfm,
nowInSec,
ColumnFilter.selection(toRead),
RowFilter.NONE,
DataLimits.NONE,
StorageService.getPartitioner().decorateKey(key),
new ClusteringIndexNamesFilter(clusterings, false)));
List<Row> rows = local
? SelectStatement.readLocally(keyspace(), commands)
: StorageProxy.read(commands, cl);
Map<DecoratedKey, Partition> map = new HashMap();
Map<ByteBuffer, CQL3Row> map = new HashMap<ByteBuffer, CQL3Row>();
for (Row row : rows)
SinglePartitionReadCommand.Group group = new SinglePartitionReadCommand.Group(commands, DataLimits.NONE);
if (local)
{
if (row.cf == null || row.cf.isEmpty())
continue;
Iterator<CQL3Row> iter = cfm.comparator.CQL3RowBuilder(cfm, now).group(row.cf.getSortedColumns().iterator());
if (iter.hasNext())
try (ReadOrderGroup orderGroup = group.startOrderGroup(); PartitionIterator iter = group.executeInternal(orderGroup))
{
map.put(row.key.getKey(), iter.next());
// We can only update one CQ3Row per partition key at a time (we don't allow IN for clustering key)
assert !iter.hasNext();
return asMaterializedMap(iter);
}
}
else
{
try (PartitionIterator iter = group.execute(cl, null))
{
return asMaterializedMap(iter);
}
}
}
private Map<DecoratedKey, Partition> asMaterializedMap(PartitionIterator iterator)
{
Map<DecoratedKey, Partition> map = new HashMap();
while (iterator.hasNext())
{
try (RowIterator partition = iterator.next())
{
map.put(partition.partitionKey(), FilteredPartition.create(partition));
}
}
return map;
@ -492,14 +596,16 @@ public abstract class ModificationStatement implements CQLStatement
{
CQL3CasRequest request = makeCasRequest(queryState, options);
ColumnFamily result = StorageProxy.cas(keyspace(),
columnFamily(),
request.key,
request,
options.getSerialConsistency(),
options.getConsistency(),
queryState.getClientState());
return new ResultMessage.Rows(buildCasResultSet(request.key, result, options));
try (RowIterator result = StorageProxy.cas(keyspace(),
columnFamily(),
request.key,
request,
options.getSerialConsistency(),
options.getConsistency(),
queryState.getClientState()))
{
return new ResultMessage.Rows(buildCasResultSet(result, options));
}
}
private CQL3CasRequest makeCasRequest(QueryState queryState, QueryOptions options)
@ -509,54 +615,54 @@ public abstract class ModificationStatement implements CQLStatement
if (keys.size() > 1)
throw new InvalidRequestException("IN on the partition key is not supported with conditional updates");
ByteBuffer key = keys.get(0);
DecoratedKey key = StorageService.getPartitioner().decorateKey(keys.get(0));
long now = options.getTimestamp(queryState);
Composite prefix = createClusteringPrefix(options);
CBuilder cbuilder = createClustering(options);
CQL3CasRequest request = new CQL3CasRequest(cfm, key, false);
addConditions(prefix, request, options);
request.addRowUpdate(prefix, this, options, now);
CQL3CasRequest request = new CQL3CasRequest(cfm, key, false, conditionColumns(), updatesRegularRows(), updatesStaticRow());
addConditions(cbuilder.build(), request, options);
request.addRowUpdate(cbuilder, this, options, now);
return request;
}
public void addConditions(Composite clusteringPrefix, CQL3CasRequest request, QueryOptions options) throws InvalidRequestException
public void addConditions(Clustering clustering, CQL3CasRequest request, QueryOptions options) throws InvalidRequestException
{
if (ifNotExists)
{
// If we use ifNotExists, if the statement applies to any non static columns, then the condition is on the row of the non-static
// columns and the prefix should be the clusteringPrefix. But if only static columns are set, then the ifNotExists apply to the existence
// columns and the prefix should be the clustering. But if only static columns are set, then the ifNotExists apply to the existence
// of any static columns and we should use the prefix for the "static part" of the partition.
request.addNotExist(clusteringPrefix);
request.addNotExist(clustering);
}
else if (ifExists)
{
request.addExist(clusteringPrefix);
request.addExist(clustering);
}
else
{
if (columnConditions != null)
request.addConditions(clusteringPrefix, columnConditions, options);
request.addConditions(clustering, columnConditions, options);
if (staticConditions != null)
request.addConditions(cfm.comparator.staticPrefix(), staticConditions, options);
request.addConditions(Clustering.STATIC_CLUSTERING, staticConditions, options);
}
}
private ResultSet buildCasResultSet(ByteBuffer key, ColumnFamily cf, QueryOptions options) throws InvalidRequestException
private ResultSet buildCasResultSet(RowIterator partition, QueryOptions options) throws InvalidRequestException
{
return buildCasResultSet(keyspace(), key, columnFamily(), cf, getColumnsWithConditions(), false, options);
return buildCasResultSet(keyspace(), columnFamily(), partition, getColumnsWithConditions(), false, options);
}
public static ResultSet buildCasResultSet(String ksName, ByteBuffer key, String cfName, ColumnFamily cf, Iterable<ColumnDefinition> columnsWithConditions, boolean isBatch, QueryOptions options)
public static ResultSet buildCasResultSet(String ksName, String tableName, RowIterator partition, Iterable<ColumnDefinition> columnsWithConditions, boolean isBatch, QueryOptions options)
throws InvalidRequestException
{
boolean success = cf == null;
boolean success = partition == null;
ColumnSpecification spec = new ColumnSpecification(ksName, cfName, CAS_RESULT_COLUMN, BooleanType.instance);
ColumnSpecification spec = new ColumnSpecification(ksName, tableName, CAS_RESULT_COLUMN, BooleanType.instance);
ResultSet.ResultMetadata metadata = new ResultSet.ResultMetadata(Collections.singletonList(spec));
List<List<ByteBuffer>> rows = Collections.singletonList(Collections.singletonList(BooleanType.instance.decompose(success)));
ResultSet rs = new ResultSet(metadata, rows);
return success ? rs : merge(rs, buildCasFailureResultSet(key, cf, columnsWithConditions, isBatch, options));
return success ? rs : merge(rs, buildCasFailureResultSet(partition, columnsWithConditions, isBatch, options));
}
private static ResultSet merge(ResultSet left, ResultSet right)
@ -582,10 +688,10 @@ public abstract class ModificationStatement implements CQLStatement
return new ResultSet(new ResultSet.ResultMetadata(specs), rows);
}
private static ResultSet buildCasFailureResultSet(ByteBuffer key, ColumnFamily cf, Iterable<ColumnDefinition> columnsWithConditions, boolean isBatch, QueryOptions options)
private static ResultSet buildCasFailureResultSet(RowIterator partition, Iterable<ColumnDefinition> columnsWithConditions, boolean isBatch, QueryOptions options)
throws InvalidRequestException
{
CFMetaData cfm = cf.metadata();
CFMetaData cfm = partition.metadata();
Selection selection;
if (columnsWithConditions == null)
{
@ -609,9 +715,8 @@ public abstract class ModificationStatement implements CQLStatement
}
long now = System.currentTimeMillis();
Selection.ResultSetBuilder builder = selection.resultSetBuilder(now, false);
SelectStatement.forSelection(cfm, selection).processColumnFamily(key, cf, options, now, builder);
Selection.ResultSetBuilder builder = selection.resultSetBuilder(false);
SelectStatement.forSelection(cfm, selection).processPartition(partition, options, builder, FBUtilities.nowInSeconds());
return builder.build(options.getProtocolVersion());
}
@ -640,31 +745,31 @@ public abstract class ModificationStatement implements CQLStatement
public ResultMessage executeInternalWithCondition(QueryState state, QueryOptions options) throws RequestValidationException, RequestExecutionException
{
CQL3CasRequest request = makeCasRequest(state, options);
ColumnFamily result = casInternal(request, state);
return new ResultMessage.Rows(buildCasResultSet(request.key, result, options));
try (RowIterator result = casInternal(request, state))
{
return new ResultMessage.Rows(buildCasResultSet(result, options));
}
}
static ColumnFamily casInternal(CQL3CasRequest request, QueryState state)
static RowIterator casInternal(CQL3CasRequest request, QueryState state)
{
UUID ballot = UUIDGen.getTimeUUIDFromMicros(state.getTimestamp());
CFMetaData metadata = Schema.instance.getCFMetaData(request.cfm.ksName, request.cfm.cfName);
ReadCommand readCommand = ReadCommand.create(request.cfm.ksName, request.key, request.cfm.cfName, request.now, request.readFilter());
Keyspace keyspace = Keyspace.open(request.cfm.ksName);
Row row = readCommand.getRow(keyspace);
ColumnFamily current = row.cf;
if (!request.appliesTo(current))
SinglePartitionReadCommand readCommand = request.readCommand(FBUtilities.nowInSeconds());
FilteredPartition current;
try (ReadOrderGroup orderGroup = readCommand.startOrderGroup(); PartitionIterator iter = readCommand.executeInternal(orderGroup))
{
if (current == null)
current = ArrayBackedSortedColumns.factory.create(metadata);
return current;
current = FilteredPartition.create(PartitionIterators.getOnlyElement(iter, readCommand));
}
ColumnFamily updates = request.makeUpdates(current);
updates = TriggerExecutor.instance.execute(request.key, updates);
if (!request.appliesTo(current))
return current.rowIterator();
Commit proposal = Commit.newProposal(request.key, ballot, updates);
PartitionUpdate updates = request.makeUpdates(current);
updates = TriggerExecutor.instance.execute(updates);
Commit proposal = Commit.newProposal(ballot, updates);
proposal.makeMutation().apply();
return null;
}
@ -683,17 +788,18 @@ public abstract class ModificationStatement implements CQLStatement
throws RequestExecutionException, RequestValidationException
{
List<ByteBuffer> keys = buildPartitionKeyNames(options);
Composite clusteringPrefix = createClusteringPrefix(options);
CBuilder clustering = createClustering(options);
UpdateParameters params = makeUpdateParameters(keys, clusteringPrefix, options, local, now);
UpdateParameters params = makeUpdateParameters(keys, clustering, options, local, now);
Collection<IMutation> mutations = new ArrayList<IMutation>(keys.size());
for (ByteBuffer key: keys)
{
ThriftValidation.validateKey(cfm, key);
ColumnFamily cf = ArrayBackedSortedColumns.factory.create(cfm);
addUpdateForKey(cf, key, clusteringPrefix, params);
Mutation mut = new Mutation(cfm.ksName, key, cf);
DecoratedKey dk = StorageService.getPartitioner().decorateKey(key);
PartitionUpdate upd = new PartitionUpdate(cfm, dk, updatedColumns(), 1);
addUpdateForKey(upd, clustering, params);
Mutation mut = new Mutation(upd);
mutations.add(isCounter() ? new CounterMutation(mut, options.getConsistency()) : mut);
}
@ -701,15 +807,15 @@ public abstract class ModificationStatement implements CQLStatement
}
public UpdateParameters makeUpdateParameters(Collection<ByteBuffer> keys,
Composite prefix,
CBuilder clustering,
QueryOptions options,
boolean local,
long now)
throws RequestExecutionException, RequestValidationException
{
// Some lists operation requires reading
Map<ByteBuffer, CQL3Row> rows = readRequiredRows(keys, prefix, local, options.getConsistency());
return new UpdateParameters(cfm, options, getTimestamp(now, options), getTimeToLive(options), rows);
Map<DecoratedKey, Partition> lists = readRequiredLists(keys, clustering, local, options.getConsistency());
return new UpdateParameters(cfm, options, getTimestamp(now, options), getTimeToLive(options), lists, true);
}
/**
@ -803,6 +909,8 @@ public abstract class ModificationStatement implements CQLStatement
stmt.validateWhereClauseForConditions();
}
stmt.finishPreparation();
return stmt;
}

719
src/java/org/apache/cassandra/cql3/statements/SelectStatement.java
View File

@ -23,7 +23,8 @@ import java.util.*;
import com.google.common.base.Objects;
import com.google.common.base.Predicate;
import com.google.common.collect.Iterables;
import com.google.common.collect.Iterators;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.cassandra.auth.Permission;
import org.apache.cassandra.config.CFMetaData;
@ -34,7 +35,8 @@ import org.apache.cassandra.cql3.restrictions.StatementRestrictions;
import org.apache.cassandra.cql3.selection.RawSelector;
import org.apache.cassandra.cql3.selection.Selection;
import org.apache.cassandra.db.*;
import org.apache.cassandra.db.composites.*;
import org.apache.cassandra.db.rows.*;
import org.apache.cassandra.db.partitions.*;
import org.apache.cassandra.db.filter.*;
import org.apache.cassandra.db.index.SecondaryIndexManager;
import org.apache.cassandra.db.marshal.CollectionType;
@ -43,12 +45,9 @@ import org.apache.cassandra.db.marshal.Int32Type;
import org.apache.cassandra.dht.AbstractBounds;
import org.apache.cassandra.exceptions.*;
import org.apache.cassandra.serializers.MarshalException;
import org.apache.cassandra.service.ClientState;
import org.apache.cassandra.service.QueryState;
import org.apache.cassandra.service.StorageProxy;
import org.apache.cassandra.service.pager.Pageable;
import org.apache.cassandra.service.*;
import org.apache.cassandra.service.pager.QueryPager;
import org.apache.cassandra.service.pager.QueryPagers;
import org.apache.cassandra.service.pager.PagingState;
import org.apache.cassandra.thrift.ThriftValidation;
import org.apache.cassandra.transport.messages.ResultMessage;
import org.apache.cassandra.utils.ByteBufferUtil;
@ -71,6 +70,8 @@ import static org.apache.cassandra.utils.ByteBufferUtil.UNSET_BYTE_BUFFER;
*/
public class SelectStatement implements CQLStatement
{
private static final Logger logger = LoggerFactory.getLogger(SelectStatement.class);
private static final int DEFAULT_COUNT_PAGE_SIZE = 10000;
private final int boundTerms;
@ -88,6 +89,8 @@ public class SelectStatement implements CQLStatement
*/
private final Comparator<List<ByteBuffer>> orderingComparator;
private final ColumnFilter queriedColumns;
// Used by forSelection below
private static final Parameters defaultParameters = new Parameters(Collections.<ColumnIdentifier.Raw, Boolean>emptyMap(), false, false, false);
@ -108,6 +111,7 @@ public class SelectStatement implements CQLStatement
this.orderingComparator = orderingComparator;
this.parameters = parameters;
this.limit = limit;
this.queriedColumns = gatherQueriedColumns();
}
public Iterable<Function> getFunctions()
@ -117,6 +121,23 @@ public class SelectStatement implements CQLStatement
limit != null ? limit.getFunctions() : Collections.<Function>emptySet());
}
// Note that the queried columns internally is different from the one selected by the
// user as it also include any column for which we have a restriction on.
private ColumnFilter gatherQueriedColumns()
{
if (selection.isWildcard())
return ColumnFilter.all(cfm);
ColumnFilter.Builder builder = ColumnFilter.allColumnsBuilder(cfm);
// Adds all selected columns
for (ColumnDefinition def : selection.getColumns())
if (!def.isPrimaryKeyColumn())
builder.add(def);
// as well as any restricted column (so we can actually apply the restriction)
builder.addAll(restrictions.nonPKRestrictedColumns());
return builder.build();
}
// Creates a simple select based on the given selection.
// Note that the results select statement should not be used for actual queries, but only for processing already
// queried data through processColumnFamily.
@ -161,31 +182,16 @@ public class SelectStatement implements CQLStatement
cl.validateForRead(keyspace());
int limit = getLimit(options);
long now = System.currentTimeMillis();
Pageable command = getPageableCommand(options, limit, now);
int nowInSec = FBUtilities.nowInSeconds();
ReadQuery query = getQuery(options, nowInSec);
int pageSize = getPageSize(options);
if (pageSize <= 0 || command == null || !QueryPagers.mayNeedPaging(command, pageSize))
return execute(command, options, limit, now, state);
if (pageSize <= 0 || query.limits().count() <= pageSize)
return execute(query, options, state, nowInSec);
QueryPager pager = QueryPagers.pager(command, cl, state.getClientState(), options.getPagingState());
return execute(pager, options, limit, now, pageSize);
}
private Pageable getPageableCommand(QueryOptions options, int limit, long now) throws RequestValidationException
{
int limitForQuery = updateLimitForQuery(limit);
if (restrictions.isKeyRange() || restrictions.usesSecondaryIndexing())
return getRangeCommand(options, limitForQuery, now);
List<ReadCommand> commands = getSliceCommands(options, limitForQuery, now);
return commands == null ? null : new Pageable.ReadCommands(commands, limitForQuery);
}
public Pageable getPageableCommand(QueryOptions options) throws RequestValidationException
{
return getPageableCommand(options, getLimit(options), System.currentTimeMillis());
QueryPager pager = query.getPager(options.getPagingState());
return execute(Pager.forDistributedQuery(pager, cl, state.getClientState()), options, pageSize, nowInSec);
}
private int getPageSize(QueryOptions options)
@ -201,103 +207,162 @@ public class SelectStatement implements CQLStatement
return pageSize;
}
private ResultMessage.Rows execute(Pageable command, QueryOptions options, int limit, long now, QueryState state)
throws RequestValidationException, RequestExecutionException
public ReadQuery getQuery(QueryOptions options, int nowInSec) throws RequestValidationException
{
List<Row> rows;
if (command == null)
{
rows = Collections.<Row>emptyList();
}
else
{
rows = command instanceof Pageable.ReadCommands
? StorageProxy.read(((Pageable.ReadCommands)command).commands, options.getConsistency(), state.getClientState())
: StorageProxy.getRangeSlice((RangeSliceCommand)command, options.getConsistency());
}
DataLimits limit = getLimit(options);
if (restrictions.isKeyRange() || restrictions.usesSecondaryIndexing())
return getRangeCommand(options, limit, nowInSec);
return processResults(rows, options, limit, now);
return getSliceCommands(options, limit, nowInSec);
}
private ResultMessage.Rows execute(QueryPager pager, QueryOptions options, int limit, long now, int pageSize)
private ResultMessage.Rows execute(ReadQuery query, QueryOptions options, QueryState state, int nowInSec) throws RequestValidationException, RequestExecutionException
{
try (PartitionIterator data = query.execute(options.getConsistency(), state.getClientState()))
{
return processResults(data, options, nowInSec);
}
}
// Simple wrapper class to avoid some code duplication
private static abstract class Pager
{
protected QueryPager pager;
protected Pager(QueryPager pager)
{
this.pager = pager;
}
public static Pager forInternalQuery(QueryPager pager, ReadOrderGroup orderGroup)
{
return new InternalPager(pager, orderGroup);
}
public static Pager forDistributedQuery(QueryPager pager, ConsistencyLevel consistency, ClientState clientState)
{
return new NormalPager(pager, consistency, clientState);
}
public boolean isExhausted()
{
return pager.isExhausted();
}
public PagingState state()
{
return pager.state();
}
public abstract PartitionIterator fetchPage(int pageSize);
public static class NormalPager extends Pager
{
private final ConsistencyLevel consistency;
private final ClientState clientState;
private NormalPager(QueryPager pager, ConsistencyLevel consistency, ClientState clientState)
{
super(pager);
this.consistency = consistency;
this.clientState = clientState;
}
public PartitionIterator fetchPage(int pageSize)
{
return pager.fetchPage(pageSize, consistency, clientState);
}
}
public static class InternalPager extends Pager
{
private final ReadOrderGroup orderGroup;
private InternalPager(QueryPager pager, ReadOrderGroup orderGroup)
{
super(pager);
this.orderGroup = orderGroup;
}
public PartitionIterator fetchPage(int pageSize)
{
return pager.fetchPageInternal(pageSize, orderGroup);
}
}
}
private ResultMessage.Rows execute(Pager pager, QueryOptions options, int pageSize, int nowInSec)
throws RequestValidationException, RequestExecutionException
{
if (selection.isAggregate())
return pageAggregateQuery(pager, options, pageSize, now);
return pageAggregateQuery(pager, options, pageSize, nowInSec);
// We can't properly do post-query ordering if we page (see #6722)
checkFalse(needsPostQueryOrdering(),
"Cannot page queries with both ORDER BY and a IN restriction on the partition key;"
+ " you must either remove the ORDER BY or the IN and sort client side, or disable paging for this query");
"Cannot page queries with both ORDER BY and a IN restriction on the partition key;"
+ " you must either remove the ORDER BY or the IN and sort client side, or disable paging for this query");
List<Row> page = pager.fetchPage(pageSize);
ResultMessage.Rows msg = processResults(page, options, limit, now);
ResultMessage.Rows msg;
try (PartitionIterator page = pager.fetchPage(pageSize))
{
msg = processResults(page, options, nowInSec);
}
// Please note that the isExhausted state of the pager only gets updated when we've closed the page, so this
// shouldn't be moved inside the 'try' above.
if (!pager.isExhausted())
msg.result.metadata.setHasMorePages(pager.state());
return msg;
}
private ResultMessage.Rows pageAggregateQuery(QueryPager pager, QueryOptions options, int pageSize, long now)
throws RequestValidationException, RequestExecutionException
private ResultMessage.Rows pageAggregateQuery(Pager pager, QueryOptions options, int pageSize, int nowInSec)
throws RequestValidationException, RequestExecutionException
{
Selection.ResultSetBuilder result = selection.resultSetBuilder(now, parameters.isJson);
Selection.ResultSetBuilder result = selection.resultSetBuilder(parameters.isJson);
while (!pager.isExhausted())
{
for (org.apache.cassandra.db.Row row : pager.fetchPage(pageSize))
try (PartitionIterator iter = pager.fetchPage(pageSize))
{
// Not columns match the query, skip
if (row.cf == null)
continue;
processColumnFamily(row.key.getKey(), row.cf, options, now, result);
while (iter.hasNext())
processPartition(iter.next(), options, result, nowInSec);
}
}
return new ResultMessage.Rows(result.build(options.getProtocolVersion()));
}
public ResultMessage.Rows processResults(List<Row> rows, QueryOptions options, int limit, long now) throws RequestValidationException
private ResultMessage.Rows processResults(PartitionIterator partitions, QueryOptions options, int nowInSec) throws RequestValidationException
{
ResultSet rset = process(rows, options, limit, now);
ResultSet rset = process(partitions, options, nowInSec);
return new ResultMessage.Rows(rset);
}
static List<Row> readLocally(String keyspaceName, List<ReadCommand> cmds)
{
Keyspace keyspace = Keyspace.open(keyspaceName);
List<Row> rows = new ArrayList<Row>(cmds.size());
for (ReadCommand cmd : cmds)
rows.add(cmd.getRow(keyspace));
return rows;
}
public ResultMessage.Rows executeInternal(QueryState state, QueryOptions options) throws RequestExecutionException, RequestValidationException
{
int limit = getLimit(options);
long now = System.currentTimeMillis();
Pageable command = getPageableCommand(options, limit, now);
int nowInSec = FBUtilities.nowInSeconds();
ReadQuery query = getQuery(options, nowInSec);
int pageSize = getPageSize(options);
if (pageSize <= 0 || command == null || !QueryPagers.mayNeedPaging(command, pageSize))
try (ReadOrderGroup orderGroup = query.startOrderGroup())
{
List<Row> rows = command == null
? Collections.<Row>emptyList()
: (command instanceof Pageable.ReadCommands
? readLocally(keyspace(), ((Pageable.ReadCommands)command).commands)
: ((RangeSliceCommand)command).executeLocally());
return processResults(rows, options, limit, now);
if (pageSize <= 0 || query.limits().count() <= pageSize)
{
try (PartitionIterator data = query.executeInternal(orderGroup))
{
return processResults(data, options, nowInSec);
}
}
else
{
QueryPager pager = query.getPager(options.getPagingState());
return execute(Pager.forInternalQuery(pager, orderGroup), options, pageSize, nowInSec);
}
}
QueryPager pager = QueryPagers.localPager(command);
return execute(pager, options, limit, now, pageSize);
}
public ResultSet process(List<Row> rows) throws InvalidRequestException
public ResultSet process(PartitionIterator partitions, int nowInSec) throws InvalidRequestException
{
QueryOptions options = QueryOptions.DEFAULT;
return process(rows, options, getLimit(options), System.currentTimeMillis());
return process(partitions, QueryOptions.DEFAULT, nowInSec);
}
public String keyspace()
@ -326,372 +391,239 @@ public class SelectStatement implements CQLStatement
return restrictions;
}
private List<ReadCommand> getSliceCommands(QueryOptions options, int limit, long now) throws RequestValidationException
private ReadQuery getSliceCommands(QueryOptions options, DataLimits limit, int nowInSec) throws RequestValidationException
{
Collection<ByteBuffer> keys = restrictions.getPartitionKeys(options);
if (keys.isEmpty())
return ReadQuery.EMPTY;
List<ReadCommand> commands = new ArrayList<>(keys.size());
IDiskAtomFilter filter = makeFilter(options, limit);
ClusteringIndexFilter filter = makeClusteringIndexFilter(options);
if (filter == null)
return null;
return ReadQuery.EMPTY;
RowFilter rowFilter = getRowFilter(options);
// Note that we use the total limit for every key, which is potentially inefficient.
// However, IN + LIMIT is not a very sensible choice.
List<SinglePartitionReadCommand<?>> commands = new ArrayList<>(keys.size());
for (ByteBuffer key : keys)
{
QueryProcessor.validateKey(key);
// We should not share the slice filter amongst the commands (hence the cloneShallow), due to
// SliceQueryFilter not being immutable due to its columnCounter used by the lastCounted() method
// (this is fairly ugly and we should change that but that's probably not a tiny refactor to do that cleanly)
commands.add(ReadCommand.create(keyspace(), ByteBufferUtil.clone(key), columnFamily(), now, filter.cloneShallow()));
DecoratedKey dk = StorageService.getPartitioner().decorateKey(ByteBufferUtil.clone(key));
commands.add(SinglePartitionReadCommand.create(cfm, nowInSec, queriedColumns, rowFilter, limit, dk, filter));
}
return commands;
return new SinglePartitionReadCommand.Group(commands, limit);
}
private RangeSliceCommand getRangeCommand(QueryOptions options, int limit, long now) throws RequestValidationException
private ReadQuery getRangeCommand(QueryOptions options, DataLimits limit, int nowInSec) throws RequestValidationException
{
IDiskAtomFilter filter = makeFilter(options, limit);
if (filter == null)
return null;
ClusteringIndexFilter clusteringIndexFilter = makeClusteringIndexFilter(options);
if (clusteringIndexFilter == null)
return ReadQuery.EMPTY;
List<IndexExpression> expressions = getValidatedIndexExpressions(options);
RowFilter rowFilter = getRowFilter(options);
// The LIMIT provided by the user is the number of CQL row he wants returned.
// We want to have getRangeSlice to count the number of columns, not the number of keys.
AbstractBounds<RowPosition> keyBounds = restrictions.getPartitionKeyBounds(options);
AbstractBounds<PartitionPosition> keyBounds = restrictions.getPartitionKeyBounds(options);
return keyBounds == null
? null
: new RangeSliceCommand(keyspace(), columnFamily(), now, filter, keyBounds, expressions, limit, !parameters.isDistinct, false);
? ReadQuery.EMPTY
: new PartitionRangeReadCommand(cfm, nowInSec, queriedColumns, rowFilter, limit, new DataRange(keyBounds, clusteringIndexFilter));
}
private ColumnSlice makeStaticSlice()
{
// Note: we could use staticPrefix.start() for the start bound, but EMPTY gives us the
// same effect while saving a few CPU cycles.
return isReversed
? new ColumnSlice(cfm.comparator.staticPrefix().end(), Composites.EMPTY)
: new ColumnSlice(Composites.EMPTY, cfm.comparator.staticPrefix().end());
}
private IDiskAtomFilter makeFilter(QueryOptions options, int limit)
private ClusteringIndexFilter makeClusteringIndexFilter(QueryOptions options)
throws InvalidRequestException
{
int toGroup = cfm.comparator.isDense() ? -1 : cfm.clusteringColumns().size();
if (parameters.isDistinct)
{
// For distinct, we only care about fetching the beginning of each partition. If we don't have
// static columns, we in fact only care about the first cell, so we query only that (we don't "group").
// If we do have static columns, we do need to fetch the first full group (to have the static columns values).
// See the comments on IGNORE_TOMBSTONED_PARTITIONS and CASSANDRA-8490 for why we use a special value for
// DISTINCT queries on the partition key only.
toGroup = selection.containsStaticColumns() ? toGroup : SliceQueryFilter.IGNORE_TOMBSTONED_PARTITIONS;
return new SliceQueryFilter(ColumnSlice.ALL_COLUMNS_ARRAY, false, 1, toGroup);
// We need to be able to distinguish between partition having live rows and those that don't. But
// doing so is not trivial since "having a live row" depends potentially on
// 1) when the query is performed, due to TTLs
// 2) how thing reconcile together between different nodes
// so that it's hard to really optimize properly internally. So to keep it simple, we simply query
// for the first row of the partition and hence uses Slices.ALL. We'll limit it to the first live
// row however in getLimit().
return new ClusteringIndexSliceFilter(Slices.ALL, false);
}
else if (restrictions.isColumnRange())
if (restrictions.isColumnRange())
{
List<Composite> startBounds = restrictions.getClusteringColumnsBoundsAsComposites(Bound.START, options);
List<Composite> endBounds = restrictions.getClusteringColumnsBoundsAsComposites(Bound.END, options);
assert startBounds.size() == endBounds.size();
Slices slices = makeSlices(options);
if (slices == Slices.NONE && !selection.containsStaticColumns())
return null;
// Handles fetching static columns. Note that for 2i, the filter is just used to restrict
// the part of the index to query so adding the static slice would be useless and confusing.
// For 2i, static columns are retrieve in CompositesSearcher with each index hit.
ColumnSlice staticSlice = selection.containsStaticColumns() && !restrictions.usesSecondaryIndexing()
? makeStaticSlice()
: null;
// The case where startBounds == 1 is common enough that it's worth optimizing
if (startBounds.size() == 1)
{
ColumnSlice slice = new ColumnSlice(startBounds.get(0), endBounds.get(0));
if (slice.isAlwaysEmpty(cfm.comparator, isReversed))
return staticSlice == null ? null : sliceFilter(staticSlice, limit, toGroup);
if (staticSlice == null)
return sliceFilter(slice, limit, toGroup);
if (isReversed)
return slice.includes(cfm.comparator.reverseComparator(), staticSlice.start)
? sliceFilter(new ColumnSlice(slice.start, staticSlice.finish), limit, toGroup)
: sliceFilter(new ColumnSlice[]{ slice, staticSlice }, limit, toGroup);
else
return slice.includes(cfm.comparator, staticSlice.finish)
? sliceFilter(new ColumnSlice(staticSlice.start, slice.finish), limit, toGroup)
: sliceFilter(new ColumnSlice[]{ staticSlice, slice }, limit, toGroup);
}
List<ColumnSlice> l = new ArrayList<ColumnSlice>(startBounds.size());
for (int i = 0; i < startBounds.size(); i++)
{
ColumnSlice slice = new ColumnSlice(startBounds.get(i), endBounds.get(i));
if (!slice.isAlwaysEmpty(cfm.comparator, isReversed))
l.add(slice);
}
if (l.isEmpty())
return staticSlice == null ? null : sliceFilter(staticSlice, limit, toGroup);
if (staticSlice == null)
return sliceFilter(l.toArray(new ColumnSlice[l.size()]), limit, toGroup);
// The slices should not overlap. We know the slices built from startBounds/endBounds don't, but if there is
// a static slice, it could overlap with the 2nd slice. Check for it and correct if that's the case
ColumnSlice[] slices;
if (isReversed)
{
if (l.get(l.size() - 1).includes(cfm.comparator.reverseComparator(), staticSlice.start))
{
slices = l.toArray(new ColumnSlice[l.size()]);
slices[slices.length-1] = new ColumnSlice(slices[slices.length-1].start, Composites.EMPTY);
}
else
{
slices = l.toArray(new ColumnSlice[l.size()+1]);
slices[slices.length-1] = staticSlice;
}
}
else
{
if (l.get(0).includes(cfm.comparator, staticSlice.finish))
{
slices = new ColumnSlice[l.size()];
slices[0] = new ColumnSlice(Composites.EMPTY, l.get(0).finish);
for (int i = 1; i < l.size(); i++)
slices[i] = l.get(i);
}
else
{
slices = new ColumnSlice[l.size()+1];
slices[0] = staticSlice;
for (int i = 0; i < l.size(); i++)
slices[i+1] = l.get(i);
}
}
return sliceFilter(slices, limit, toGroup);
return new ClusteringIndexSliceFilter(slices, isReversed);
}
else
{
SortedSet<CellName> cellNames = getRequestedColumns(options);
if (cellNames == null) // in case of IN () for the last column of the key
NavigableSet<Clustering> clusterings = getRequestedRows(options);
if (clusterings.isEmpty() && !selection.containsStaticColumns()) // in case of IN () for the last column of the key
return null;
QueryProcessor.validateCellNames(cellNames, cfm.comparator);
return new NamesQueryFilter(cellNames, true);
return new ClusteringIndexNamesFilter(clusterings, isReversed);
}
}
private SliceQueryFilter sliceFilter(ColumnSlice slice, int limit, int toGroup)
private Slices makeSlices(QueryOptions options)
throws InvalidRequestException
{
return sliceFilter(new ColumnSlice[]{ slice }, limit, toGroup);
}
SortedSet<Slice.Bound> startBounds = restrictions.getClusteringColumnsBounds(Bound.START, options);
SortedSet<Slice.Bound> endBounds = restrictions.getClusteringColumnsBounds(Bound.END, options);
assert startBounds.size() == endBounds.size();
private SliceQueryFilter sliceFilter(ColumnSlice[] slices, int limit, int toGroup)
{
assert ColumnSlice.validateSlices(slices, cfm.comparator, isReversed) : String.format("Invalid slices: " + Arrays.toString(slices) + (isReversed ? " (reversed)" : ""));
return new SliceQueryFilter(slices, isReversed, limit, toGroup);
// The case where startBounds == 1 is common enough that it's worth optimizing
if (startBounds.size() == 1)
{
Slice.Bound start = startBounds.first();
Slice.Bound end = endBounds.first();
return cfm.comparator.compare(start, end) > 0
? Slices.NONE
: Slices.with(cfm.comparator, Slice.make(start, end));
}
Slices.Builder builder = new Slices.Builder(cfm.comparator, startBounds.size());
Iterator<Slice.Bound> startIter = startBounds.iterator();
Iterator<Slice.Bound> endIter = endBounds.iterator();
while (startIter.hasNext() && endIter.hasNext())
{
Slice.Bound start = startIter.next();
Slice.Bound end = endIter.next();
// Ignore slices that are nonsensical
if (cfm.comparator.compare(start, end) > 0)
continue;
builder.add(start, end);
}
return builder.build();
}
/**
* May be used by custom QueryHandler implementations
*/
public int getLimit(QueryOptions options) throws InvalidRequestException
public DataLimits getLimit(QueryOptions options) throws InvalidRequestException
{
if (limit != null)
int userLimit = -1;
// If we aggregate, the limit really apply to the number of rows returned to the user, not to what is queried, and
// since in practice we currently only aggregate at top level (we have no GROUP BY support yet), we'll only ever
// return 1 result and can therefore basically ignore the user LIMIT in this case.
// Whenever we support GROUP BY, we'll have to add a new DataLimits kind that knows how things are grouped and is thus
// able to apply the user limit properly.
if (limit != null && !selection.isAggregate())
{
ByteBuffer b = checkNotNull(limit.bindAndGet(options), "Invalid null value of limit");
// treat UNSET limit value as 'unlimited'
if (b == UNSET_BYTE_BUFFER)
return Integer.MAX_VALUE;
try
if (b != UNSET_BYTE_BUFFER)
{
Int32Type.instance.validate(b);
int l = Int32Type.instance.compose(b);
checkTrue(l > 0, "LIMIT must be strictly positive");
return l;
}
catch (MarshalException e)
{
throw new InvalidRequestException("Invalid limit value");
try
{
Int32Type.instance.validate(b);
userLimit = Int32Type.instance.compose(b);
checkTrue(userLimit > 0, "LIMIT must be strictly positive");
}
catch (MarshalException e)
{
throw new InvalidRequestException("Invalid limit value");
}
}
}
return Integer.MAX_VALUE;
if (parameters.isDistinct)
return userLimit < 0 ? DataLimits.DISTINCT_NONE : DataLimits.distinctLimits(userLimit);
return userLimit < 0 ? DataLimits.NONE : DataLimits.cqlLimits(userLimit);
}
private int updateLimitForQuery(int limit)
{
// Internally, we don't support exclusive bounds for slices. Instead, we query one more element if necessary
// and exclude it later (in processColumnFamily)
return restrictions.isNonCompositeSliceWithExclusiveBounds() && limit != Integer.MAX_VALUE
? limit + 1
: limit;
}
private SortedSet<CellName> getRequestedColumns(QueryOptions options) throws InvalidRequestException
private NavigableSet<Clustering> getRequestedRows(QueryOptions options) throws InvalidRequestException
{
// Note: getRequestedColumns don't handle static columns, but due to CASSANDRA-5762
// we always do a slice for CQL3 tables, so it's ok to ignore them here
assert !restrictions.isColumnRange();
SortedSet<CellName> columns = new TreeSet<CellName>(cfm.comparator);
for (Composite composite : restrictions.getClusteringColumnsAsComposites(options))
columns.addAll(addSelectedColumns(composite));
return columns;
}
private SortedSet<CellName> addSelectedColumns(Composite prefix)
{
if (cfm.comparator.isDense())
{
return FBUtilities.singleton(cfm.comparator.create(prefix, null), cfm.comparator);
}
else
{
SortedSet<CellName> columns = new TreeSet<CellName>(cfm.comparator);
// We need to query the selected column as well as the marker
// column (for the case where the row exists but has no columns outside the PK)
// Two exceptions are "static CF" (non-composite non-compact CF) and "super CF"
// that don't have marker and for which we must query all columns instead
if (cfm.comparator.isCompound() && !cfm.isSuper())
{
// marker
columns.add(cfm.comparator.rowMarker(prefix));
// selected columns
for (ColumnDefinition def : selection.getColumns())
if (def.isRegular() || def.isStatic())
columns.add(cfm.comparator.create(prefix, def));
}
else
{
// We now that we're not composite so we can ignore static columns
for (ColumnDefinition def : cfm.regularColumns())
columns.add(cfm.comparator.create(prefix, def));
}
return columns;
}
return restrictions.getClusteringColumns(options);
}
/**
* May be used by custom QueryHandler implementations
*/
public List<IndexExpression> getValidatedIndexExpressions(QueryOptions options) throws InvalidRequestException
public RowFilter getRowFilter(QueryOptions options) throws InvalidRequestException
{
if (!restrictions.usesSecondaryIndexing())
return Collections.emptyList();
ColumnFamilyStore cfs = Keyspace.open(keyspace()).getColumnFamilyStore(columnFamily());
SecondaryIndexManager secondaryIndexManager = cfs.indexManager;
List<IndexExpression> expressions = restrictions.getIndexExpressions(secondaryIndexManager, options);
secondaryIndexManager.validateIndexSearchersForQuery(expressions);
return expressions;
RowFilter filter = restrictions.getRowFilter(secondaryIndexManager, options);
secondaryIndexManager.validateFilter(filter);
return filter;
}
private CellName makeExclusiveSliceBound(Bound bound, CellNameType type, QueryOptions options) throws InvalidRequestException
private ResultSet process(PartitionIterator partitions, QueryOptions options, int nowInSec) throws InvalidRequestException
{
if (restrictions.areRequestedBoundsInclusive(bound))
return null;
return type.makeCellName(restrictions.getClusteringColumnsBounds(bound, options).get(0));
}
private Iterator<Cell> applySliceRestriction(final Iterator<Cell> cells, final QueryOptions options) throws InvalidRequestException
{
final CellNameType type = cfm.comparator;
final CellName excludedStart = makeExclusiveSliceBound(Bound.START, type, options);
final CellName excludedEnd = makeExclusiveSliceBound(Bound.END, type, options);
return Iterators.filter(cells, new Predicate<Cell>()
Selection.ResultSetBuilder result = selection.resultSetBuilder(parameters.isJson);
while (partitions.hasNext())
{
public boolean apply(Cell c)
try (RowIterator partition = partitions.next())
{
// For dynamic CF, the column could be out of the requested bounds (because we don't support strict bounds internally (unless
// the comparator is composite that is)), filter here
return !((excludedStart != null && type.compare(c.name(), excludedStart) == 0)
|| (excludedEnd != null && type.compare(c.name(), excludedEnd) == 0));
processPartition(partition, options, result, nowInSec);
}
});
}
private ResultSet process(List<Row> rows, QueryOptions options, int limit, long now) throws InvalidRequestException
{
Selection.ResultSetBuilder result = selection.resultSetBuilder(now, parameters.isJson);
for (org.apache.cassandra.db.Row row : rows)
{
// Not columns match the query, skip
if (row.cf == null)
continue;
processColumnFamily(row.key.getKey(), row.cf, options, now, result);
}
ResultSet cqlRows = result.build(options.getProtocolVersion());
orderResults(cqlRows);
// Internal calls always return columns in the comparator order, even when reverse was set
if (isReversed)
cqlRows.reverse();
// Trim result if needed to respect the user limit
cqlRows.trim(limit);
return cqlRows;
}
// Used by ModificationStatement for CAS operations
void processColumnFamily(ByteBuffer key, ColumnFamily cf, QueryOptions options, long now, Selection.ResultSetBuilder result)
throws InvalidRequestException
public static ByteBuffer[] getComponents(CFMetaData cfm, DecoratedKey dk)
{
CFMetaData cfm = cf.metadata();
ByteBuffer[] keyComponents = null;
ByteBuffer key = dk.getKey();
if (cfm.getKeyValidator() instanceof CompositeType)
{
keyComponents = ((CompositeType)cfm.getKeyValidator()).split(key);
return ((CompositeType)cfm.getKeyValidator()).split(key);
}
else
{
keyComponents = new ByteBuffer[]{ key };
return new ByteBuffer[]{ key };
}
}
Iterator<Cell> cells = cf.getSortedColumns().iterator();
if (restrictions.isNonCompositeSliceWithExclusiveBounds())
cells = applySliceRestriction(cells, options);
// Used by ModificationStatement for CAS operations
void processPartition(RowIterator partition, QueryOptions options, Selection.ResultSetBuilder result, int nowInSec)
throws InvalidRequestException
{
int protocolVersion = options.getProtocolVersion();
CQL3Row.RowIterator iter = cfm.comparator.CQL3RowBuilder(cfm, now).group(cells);
// If there is static columns but there is no non-static row, then provided the select was a full
// partition selection (i.e. not a 2ndary index search and there was no condition on clustering columns)
// then we want to include the static columns in the result set (and we're done).
CQL3Row staticRow = iter.getStaticRow();
if (staticRow != null && !iter.hasNext() && !restrictions.usesSecondaryIndexing() && restrictions.hasNoClusteringColumnsRestriction())
ByteBuffer[] keyComponents = getComponents(cfm, partition.partitionKey());
Row staticRow = partition.staticRow().takeAlias();
// If there is no rows, then provided the select was a full partition selection
// (i.e. not a 2ndary index search and there was no condition on clustering columns),
// we want to include static columns and we're done.
if (!partition.hasNext())
{
result.newRow(protocolVersion);
for (ColumnDefinition def : selection.getColumns())
if (!staticRow.isEmpty() && (!restrictions.usesSecondaryIndexing() || cfm.isStaticCompactTable()) && !restrictions.hasClusteringColumnsRestriction())
{
switch (def.kind)
result.newRow(protocolVersion);
for (ColumnDefinition def : selection.getColumns())
{
case PARTITION_KEY:
result.add(keyComponents[def.position()]);
break;
case STATIC:
addValue(result, def, staticRow, options);
break;
default:
result.add((ByteBuffer)null);
switch (def.kind)
{
case PARTITION_KEY:
result.add(keyComponents[def.position()]);
break;
case STATIC:
addValue(result, def, staticRow, nowInSec, protocolVersion);
break;
default:
result.add((ByteBuffer)null);
}
}
}
return;
}
while (iter.hasNext())
while (partition.hasNext())
{
CQL3Row cql3Row = iter.next();
// Respect requested order
Row row = partition.next();
result.newRow(protocolVersion);
// Respect selection order
for (ColumnDefinition def : selection.getColumns())
@ -702,41 +634,35 @@ public class SelectStatement implements CQLStatement
result.add(keyComponents[def.position()]);
break;
case CLUSTERING_COLUMN:
result.add(cql3Row.getClusteringColumn(def.position()));
break;
case COMPACT_VALUE:
result.add(cql3Row.getColumn(null));
result.add(row.clustering().get(def.position()));
break;
case REGULAR:
addValue(result, def, cql3Row, options);
addValue(result, def, row, nowInSec, protocolVersion);
break;
case STATIC:
addValue(result, def, staticRow, options);
addValue(result, def, staticRow, nowInSec, protocolVersion);
break;
}
}
}
}
private static void addValue(Selection.ResultSetBuilder result, ColumnDefinition def, CQL3Row row, QueryOptions options)
private static void addValue(Selection.ResultSetBuilder result, ColumnDefinition def, Row row, int nowInSec, int protocolVersion)
{
if (row == null)
if (def.isComplex())
{
result.add((ByteBuffer)null);
return;
// Collections are the only complex types we have so far
assert def.type.isCollection() && def.type.isMultiCell();
Iterator<Cell> cells = row.getCells(def);
if (cells == null)
result.add((ByteBuffer)null);
else
result.add(((CollectionType)def.type).serializeForNativeProtocol(def, cells, protocolVersion));
}
if (def.type.isMultiCell())
else
{
List<Cell> cells = row.getMultiCellColumn(def.name);
ByteBuffer buffer = cells == null
? null
: ((CollectionType)def.type).serializeForNativeProtocol(def, cells, options.getProtocolVersion());
result.add(buffer);
return;
result.add(row.getCell(def), nowInSec);
}
result.add(row.getColumn(def.name));
}
private boolean needsPostQueryOrdering()
@ -796,9 +722,6 @@ public class SelectStatement implements CQLStatement
isReversed = isReversed(cfm);
}
if (isReversed)
restrictions.reverse();
checkNeedsFiltering(restrictions);
SelectStatement stmt = new SelectStatement(cfm,
@ -832,7 +755,8 @@ public class SelectStatement implements CQLStatement
whereClause,
boundNames,
selection.containsOnlyStaticColumns(),
selection.containsACollection());
selection.containsACollection(),
parameters.allowFiltering);
}
catch (UnrecognizedEntityException e)
{
@ -980,47 +904,12 @@ public class SelectStatement implements CQLStatement
{
// We will potentially filter data if either:
// - Have more than one IndexExpression
// - Have no index expression and the column filter is not the identity
// - Have no index expression and the row filter is not the identity
checkFalse(restrictions.needFiltering(),
"Cannot execute this query as it might involve data filtering and " +
"thus may have unpredictable performance. If you want to execute " +
"this query despite the performance unpredictability, use ALLOW FILTERING");
}
// We don't internally support exclusive slice bounds on non-composite tables. To deal with it we do an
// inclusive slice and remove post-query the value that shouldn't be returned. One problem however is that
// if there is a user limit, that limit may make the query return before the end of the slice is reached,
// in which case, once we'll have removed bound post-query, we might end up with less results than
// requested which would be incorrect. For single-partition query, this is not a problem, we just ask for
// one more result (see updateLimitForQuery()) since that's enough to compensate for that problem. For key
// range however, each returned row may include one result that will have to be trimmed, so we would have
// to bump the query limit by N where N is the number of rows we will return, but we don't know that in
// advance. So, since we currently don't have a good way to handle such query, we refuse it (#7059) rather
// than answering with something that is wrong.
if (restrictions.isNonCompositeSliceWithExclusiveBounds() && restrictions.isKeyRange() && limit != null)
{
SingleColumnRelation rel = findInclusiveClusteringRelationForCompact(restrictions.cfm);
throw invalidRequest("The query requests a restriction of rows with a strict bound (%s) over a range of partitions. "
+ "This is not supported by the underlying storage engine for COMPACT tables if a LIMIT is provided. "
+ "Please either make the condition non strict (%s) or remove the user LIMIT", rel, rel.withNonStrictOperator());
}
}
private SingleColumnRelation findInclusiveClusteringRelationForCompact(CFMetaData cfm)
{
for (Relation r : whereClause)
{
// We only call this when sliceRestriction != null, i.e. for compact table with non composite comparator,
// so it can't be a MultiColumnRelation.
SingleColumnRelation rel = (SingleColumnRelation)r;
if (cfm.getColumnDefinition(rel.getEntity().prepare(cfm)).isClusteringColumn()
&& (rel.operator() == Operator.GT || rel.operator() == Operator.LT))
return rel;
}
// We're not supposed to call this method unless we know this can't happen
throw new AssertionError();
}
private boolean containsAlias(final ColumnIdentifier name)

126
src/java/org/apache/cassandra/cql3/statements/UpdateStatement.java
View File

@ -17,22 +17,18 @@
*/
package org.apache.cassandra.cql3.statements;
import java.nio.ByteBuffer;
import java.util.*;
import org.apache.cassandra.cql3.*;
import org.apache.cassandra.config.CFMetaData;
import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.db.*;
import org.apache.cassandra.db.composites.Composite;
import org.apache.cassandra.db.index.SecondaryIndex;
import org.apache.cassandra.db.index.SecondaryIndexManager;
import org.apache.cassandra.db.rows.*;
import org.apache.cassandra.db.partitions.PartitionUpdate;
import org.apache.cassandra.exceptions.*;
import org.apache.cassandra.utils.ByteBufferUtil;
import org.apache.cassandra.utils.Pair;
import static org.apache.cassandra.cql3.statements.RequestValidations.invalidRequest;
/**
* An <code>UPDATE</code> statement parsed from a CQL query statement.
*
@ -51,98 +47,52 @@ public class UpdateStatement extends ModificationStatement
return true;
}
public void addUpdateForKey(ColumnFamily cf,
ByteBuffer key,
Composite prefix,
UpdateParameters params) throws InvalidRequestException
public void addUpdateForKey(PartitionUpdate update, CBuilder cbuilder, UpdateParameters params)
throws InvalidRequestException
{
addUpdateForKey(cf, key, prefix, params, true);
}
params.newPartition(update.partitionKey());
public void addUpdateForKey(ColumnFamily cf,
ByteBuffer key,
Composite prefix,
UpdateParameters params,
boolean validateIndexedColumns) throws InvalidRequestException
{
// Inserting the CQL row marker (see #4361)
// We always need to insert a marker for INSERT, because of the following situation:
// CREATE TABLE t ( k int PRIMARY KEY, c text );
// INSERT INTO t(k, c) VALUES (1, 1)
// DELETE c FROM t WHERE k = 1;
// SELECT * FROM t;
// The last query should return one row (but with c == null). Adding the marker with the insert make sure
// the semantic is correct (while making sure a 'DELETE FROM t WHERE k = 1' does remove the row entirely)
//
// We do not insert the marker for UPDATE however, as this amount to updating the columns in the WHERE
// clause which is inintuitive (#6782)
//
// We never insert markers for Super CF as this would confuse the thrift side.
if (type == StatementType.INSERT && cfm.isCQL3Table() && !prefix.isStatic())
cf.addColumn(params.makeColumn(cfm.comparator.rowMarker(prefix), ByteBufferUtil.EMPTY_BYTE_BUFFER));
List<Operation> updates = getOperations();
if (cfm.comparator.isDense())
if (updatesRegularRows())
{
if (prefix.isEmpty())
throw new InvalidRequestException(String.format("Missing PRIMARY KEY part %s", cfm.clusteringColumns().get(0)));
Clustering clustering = cbuilder.build();
Row.Writer writer = update.writer();
params.writeClustering(clustering, writer);
// An empty name for the compact value is what we use to recognize the case where there is not column
// outside the PK, see CreateStatement.
if (!cfm.compactValueColumn().name.bytes.hasRemaining())
// We update the row timestamp (ex-row marker) only on INSERT (#6782)
// Further, COMPACT tables semantic differs from "CQL3" ones in that a row exists only if it has
// a non-null column, so we don't want to set the row timestamp for them.
if (type == StatementType.INSERT && cfm.isCQLTable())
params.writePartitionKeyLivenessInfo(writer);
List<Operation> updates = getRegularOperations();
// For compact tablw, when we translate it to thrift, we don't have a row marker. So we don't accept an insert/update
// that only sets the PK unless the is no declared non-PK columns (in the latter we just set the value empty).
// For a dense layout, when we translate it to thrift, we don't have a row marker. So we don't accept an insert/update
// that only sets the PK unless the is no declared non-PK columns (which we recognize because in that case the compact
// value is of type "EmptyType").
if (cfm.isCompactTable() && updates.isEmpty())
{
// There is no column outside the PK. So no operation could have passed through validation
assert updates.isEmpty();
new Constants.Setter(cfm.compactValueColumn(), EMPTY).execute(key, cf, prefix, params);
}
else
{
// dense means we don't have a row marker, so don't accept to set only the PK. See CASSANDRA-5648.
if (updates.isEmpty())
if (CompactTables.hasEmptyCompactValue(cfm))
updates = Collections.<Operation>singletonList(new Constants.Setter(cfm.compactValueColumn(), EMPTY));
else
throw new InvalidRequestException(String.format("Column %s is mandatory for this COMPACT STORAGE table", cfm.compactValueColumn().name));
for (Operation update : updates)
update.execute(key, cf, prefix, params);
}
}
else
{
for (Operation update : updates)
update.execute(key, cf, prefix, params);
for (Operation op : updates)
op.execute(update.partitionKey(), clustering, writer, params);
writer.endOfRow();
}
// validateIndexedColumns trigger a call to Keyspace.open() which we want to be able to avoid in some case
//(e.g. when using CQLSSTableWriter)
if (validateIndexedColumns)
validateIndexedColumns(key, cf);
}
/**
* Checks if the values of the indexed columns are valid.
*
* @param key row key for the column family
* @param cf the column family
* @throws InvalidRequestException if one of the values of the indexed columns is not valid
*/
private void validateIndexedColumns(ByteBuffer key, ColumnFamily cf)
{
SecondaryIndexManager indexManager = Keyspace.open(cfm.ksName).getColumnFamilyStore(cfm.cfId).indexManager;
if (indexManager.hasIndexes())
if (updatesStaticRow())
{
for (Cell cell : cf)
{
// Indexed values must be validated by any applicable index. See CASSANDRA-3057/4240/8081 for more details
SecondaryIndex failedIndex = indexManager.validate(key, cell);
if (failedIndex != null)
{
throw invalidRequest(String.format("Can't index column value of size %d for index %s on %s.%s",
cell.value().remaining(),
failedIndex.getIndexName(),
cfm.ksName,
cfm.cfName));
}
}
Row.Writer writer = update.staticWriter();
for (Operation op : getStaticOperations())
op.execute(update.partitionKey(), Clustering.STATIC_CLUSTERING, writer, params);
writer.endOfRow();
}
}

236
src/java/org/apache/cassandra/db/AbstractCell.java
View File

@ -1,236 +0,0 @@
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.db;
import java.io.DataInput;
import java.io.IOError;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.security.MessageDigest;
import java.util.Iterator;
import com.google.common.collect.AbstractIterator;
import org.apache.cassandra.config.CFMetaData;
import org.apache.cassandra.db.composites.CellName;
import org.apache.cassandra.db.composites.CellNameType;
import org.apache.cassandra.db.context.CounterContext;
import org.apache.cassandra.db.marshal.AbstractType;
import org.apache.cassandra.io.sstable.format.Version;
import org.apache.cassandra.serializers.MarshalException;
import org.apache.cassandra.utils.FBUtilities;
public abstract class AbstractCell implements Cell
{
public static Iterator<OnDiskAtom> onDiskIterator(final DataInput in,
final ColumnSerializer.Flag flag,
final int expireBefore,
final Version version,
final CellNameType type)
{
return new AbstractIterator<OnDiskAtom>()
{
protected OnDiskAtom computeNext()
{
OnDiskAtom atom;
try
{
atom = type.onDiskAtomSerializer().deserializeFromSSTable(in, flag, expireBefore, version);
}
catch (IOException e)
{
throw new IOError(e);
}
if (atom == null)
return endOfData();
return atom;
}
};
}
public boolean isLive()
{
return true;
}
public boolean isLive(long now)
{
return true;
}
public int cellDataSize()
{
return name().dataSize() + value().remaining() + TypeSizes.NATIVE.sizeof(timestamp());
}
public int serializedSize(CellNameType type, TypeSizes typeSizes)
{
/*
* Size of a column is =
* size of a name (short + length of the string)
* + 1 byte to indicate if the column has been deleted
* + 8 bytes for timestamp
* + 4 bytes which basically indicates the size of the byte array
* + entire byte array.
*/
int valueSize = value().remaining();
return ((int)type.cellSerializer().serializedSize(name(), typeSizes)) + 1 + typeSizes.sizeof(timestamp()) + typeSizes.sizeof(valueSize) + valueSize;
}
public int serializationFlags()
{
return 0;
}
public Cell diff(Cell cell)
{
if (timestamp() < cell.timestamp())
return cell;
return null;
}
public void updateDigest(MessageDigest digest)
{
digest.update(name().toByteBuffer().duplicate());
digest.update(value().duplicate());
FBUtilities.updateWithLong(digest, timestamp());
FBUtilities.updateWithByte(digest, serializationFlags());
}
public int getLocalDeletionTime()
{
return Integer.MAX_VALUE;
}
public Cell reconcile(Cell cell)
{
long ts1 = timestamp(), ts2 = cell.timestamp();
if (ts1 != ts2)
return ts1 < ts2 ? cell : this;
if (isLive() != cell.isLive())
return isLive() ? cell : this;
return value().compareTo(cell.value()) < 0 ? cell : this;
}
@Override
public boolean equals(Object o)
{
return this == o || (o instanceof Cell && equals((Cell) o));
}
public boolean equals(Cell cell)
{
return timestamp() == cell.timestamp() && name().equals(cell.name()) && value().equals(cell.value())
&& serializationFlags() == cell.serializationFlags();
}
public int hashCode()
{
throw new UnsupportedOperationException();
}
public String getString(CellNameType comparator)
{
return String.format("%s:%b:%d@%d",
comparator.getString(name()),
!isLive(),
value().remaining(),
timestamp());
}
public void validateName(CFMetaData metadata) throws MarshalException
{
metadata.comparator.validate(name());
}
public void validateFields(CFMetaData metadata) throws MarshalException
{
validateName(metadata);
AbstractType<?> valueValidator = metadata.getValueValidator(name());
if (valueValidator != null)
valueValidator.validateCellValue(value());
}
public static Cell create(CellName name, ByteBuffer value, long timestamp, int ttl, CFMetaData metadata)
{
if (ttl <= 0)
ttl = metadata.getDefaultTimeToLive();
return ttl > 0
? new BufferExpiringCell(name, value, timestamp, ttl)
: new BufferCell(name, value, timestamp);
}
public Cell diffCounter(Cell cell)
{
assert this instanceof CounterCell : "Wrong class type: " + getClass();
if (timestamp() < cell.timestamp())
return cell;
// Note that if at that point, cell can't be a tombstone. Indeed,
// cell is the result of merging us with other nodes results, and
// merging a CounterCell with a tombstone never return a tombstone
// unless that tombstone timestamp is greater that the CounterCell
// one.
assert cell instanceof CounterCell : "Wrong class type: " + cell.getClass();
if (((CounterCell) this).timestampOfLastDelete() < ((CounterCell) cell).timestampOfLastDelete())
return cell;
CounterContext.Relationship rel = CounterCell.contextManager.diff(cell.value(), value());
return (rel == CounterContext.Relationship.GREATER_THAN || rel == CounterContext.Relationship.DISJOINT) ? cell : null;
}
/** This is temporary until we start creating Cells of the different type (buffer vs. native) */
public Cell reconcileCounter(Cell cell)
{
assert this instanceof CounterCell : "Wrong class type: " + getClass();
// No matter what the counter cell's timestamp is, a tombstone always takes precedence. See CASSANDRA-7346.
if (cell instanceof DeletedCell)
return cell;
assert (cell instanceof CounterCell) : "Wrong class type: " + cell.getClass();
// live < live last delete
if (timestamp() < ((CounterCell) cell).timestampOfLastDelete())
return cell;
long timestampOfLastDelete = ((CounterCell) this).timestampOfLastDelete();
// live last delete > live
if (timestampOfLastDelete > cell.timestamp())
return this;
// live + live. return one of the cells if its context is a superset of the other's, or merge them otherwise
ByteBuffer context = CounterCell.contextManager.merge(value(), cell.value());
if (context == value() && timestamp() >= cell.timestamp() && timestampOfLastDelete >= ((CounterCell) cell).timestampOfLastDelete())
return this;
else if (context == cell.value() && cell.timestamp() >= timestamp() && ((CounterCell) cell).timestampOfLastDelete() >= timestampOfLastDelete)
return cell;
else // merge clocks and timestamps.
return new BufferCounterCell(name(),
context,
Math.max(timestamp(), cell.timestamp()),
Math.max(timestampOfLastDelete, ((CounterCell) cell).timestampOfLastDelete()));
}
}

93
src/java/org/apache/cassandra/db/AbstractClusteringPrefix.java Normal file
View File

@ -0,0 +1,93 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.db;
import java.nio.ByteBuffer;
import java.security.MessageDigest;
import java.util.Objects;
import org.apache.cassandra.utils.FBUtilities;
public abstract class AbstractClusteringPrefix implements ClusteringPrefix
{
public ClusteringPrefix clustering()
{
return this;
}
public int dataSize()
{
int size = 0;
for (int i = 0; i < size(); i++)
{
ByteBuffer bb = get(i);
size += bb == null ? 0 : bb.remaining();
}
return size;
}
public void digest(MessageDigest digest)
{
for (int i = 0; i < size(); i++)
{
ByteBuffer bb = get(i);
if (bb != null)
digest.update(bb.duplicate());
}
FBUtilities.updateWithByte(digest, kind().ordinal());
}
public void writeTo(Writer writer)
{
for (int i = 0; i < size(); i++)
writer.writeClusteringValue(get(i));
}
public long unsharedHeapSize()
{
// unsharedHeapSize is used inside the cache and in memtables. Implementations that are
// safe to use there (SimpleClustering, Slice.Bound.SimpleBound and MemtableRow.* classes) overwrite this.
throw new UnsupportedOperationException();
}
@Override
public final int hashCode()
{
int result = 31;
for (int i = 0; i < size(); i++)
result += 31 * Objects.hashCode(get(i));
return 31 * result + Objects.hashCode(kind());
}
@Override
public final boolean equals(Object o)
{
if(!(o instanceof ClusteringPrefix))
return false;
ClusteringPrefix that = (ClusteringPrefix)o;
if (this.kind() != that.kind() || this.size() != that.size())
return false;
for (int i = 0; i < size(); i++)
if (!Objects.equals(this.get(i), that.get(i)))
return false;
return true;
}
}

164
src/java/org/apache/cassandra/db/AbstractLivenessInfo.java Normal file
View File

@ -0,0 +1,164 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.db;
import java.util.Objects;
import java.security.MessageDigest;
import org.apache.cassandra.serializers.MarshalException;
import org.apache.cassandra.utils.FBUtilities;
/**
* Base abstract class for {@code LivenessInfo} implementations.
*
* All {@code LivenessInfo} should extends this class unless it has a very
* good reason not to.
*/
public abstract class AbstractLivenessInfo implements LivenessInfo
{
public boolean hasTimestamp()
{
return timestamp() != NO_TIMESTAMP;
}
public boolean hasTTL()
{
return ttl() != NO_TTL;
}
public boolean hasLocalDeletionTime()
{
return localDeletionTime() != NO_DELETION_TIME;
}
public int remainingTTL(int nowInSec)
{
if (!hasTTL())
return -1;
int remaining = localDeletionTime() - nowInSec;
return remaining >= 0 ? remaining : -1;
}
public boolean isLive(int nowInSec)
{
// Note that we don't rely on localDeletionTime() only because if we were to, we
// could potentially consider a tombstone as a live cell (due to time skew). So
// if a cell has a local deletion time and no ttl, it's a tombstone and consider
// dead no matter what it's actual local deletion value is.
return hasTimestamp() && (!hasLocalDeletionTime() || (hasTTL() && nowInSec < localDeletionTime()));
}
public void digest(MessageDigest digest)
{
FBUtilities.updateWithLong(digest, timestamp());
FBUtilities.updateWithInt(digest, localDeletionTime());
FBUtilities.updateWithInt(digest, ttl());
}
public void validate()
{
if (ttl() < 0)
throw new MarshalException("A TTL should not be negative");
if (localDeletionTime() < 0)
throw new MarshalException("A local deletion time should not be negative");
if (hasTTL() && !hasLocalDeletionTime())
throw new MarshalException("Shoud not have a TTL without an associated local deletion time");
}
public int dataSize()
{
int size = 0;
if (hasTimestamp())
size += TypeSizes.NATIVE.sizeof(timestamp());
if (hasTTL())
size += TypeSizes.NATIVE.sizeof(ttl());
if (hasLocalDeletionTime())
size += TypeSizes.NATIVE.sizeof(localDeletionTime());
return size;
}
public boolean supersedes(LivenessInfo other)
{
return timestamp() > other.timestamp();
}
public LivenessInfo mergeWith(LivenessInfo other)
{
return supersedes(other) ? this : other;
}
public LivenessInfo takeAlias()
{
return new SimpleLivenessInfo(timestamp(), ttl(), localDeletionTime());
};
public LivenessInfo withUpdatedTimestamp(long newTimestamp)
{
if (!hasTimestamp())
return this;
return new SimpleLivenessInfo(newTimestamp, ttl(), localDeletionTime());
}
public boolean isPurgeable(long maxPurgeableTimestamp, int gcBefore)
{
return timestamp() < maxPurgeableTimestamp && localDeletionTime() < gcBefore;
}
@Override
public String toString()
{
StringBuilder sb = new StringBuilder();
sb.append('[');
boolean needSpace = false;
if (hasTimestamp())
{
sb.append("ts=").append(timestamp());
needSpace = true;
}
if (hasTTL())
{
sb.append(needSpace ? ' ' : "").append("ttl=").append(ttl());
needSpace = true;
}
if (hasLocalDeletionTime())
sb.append(needSpace ? ' ' : "").append("ldt=").append(localDeletionTime());
sb.append(']');
return sb.toString();
}
@Override
public boolean equals(Object other)
{
if(!(other instanceof LivenessInfo))
return false;
LivenessInfo that = (LivenessInfo)other;
return this.timestamp() == that.timestamp()
&& this.ttl() == that.ttl()
&& this.localDeletionTime() == that.localDeletionTime();
}
@Override
public int hashCode()
{
return Objects.hash(timestamp(), ttl(), localDeletionTime());
}
}

710
src/java/org/apache/cassandra/db/AbstractNativeCell.java
View File

@ -1,710 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.db;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.security.MessageDigest;
import net.nicoulaj.compilecommand.annotations.Inline;
import org.apache.cassandra.config.CFMetaData;
import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.cql3.ColumnIdentifier;
import org.apache.cassandra.db.composites.*;
import org.apache.cassandra.db.filter.ColumnSlice;
import org.apache.cassandra.db.marshal.AbstractType;
import org.apache.cassandra.db.marshal.CompositeType;
import org.apache.cassandra.utils.ByteBufferUtil;
import org.apache.cassandra.utils.FBUtilities;
import org.apache.cassandra.utils.FastByteOperations;
import org.apache.cassandra.utils.concurrent.OpOrder;
import org.apache.cassandra.utils.memory.*;
/**
* <pre>
* {@code
* Packs a CellName AND a Cell into one off-heap representation.
* Layout is:
*
* Note we store the ColumnIdentifier in full as bytes. This seems an okay tradeoff for now, as we just
* look it back up again when we need to, and in the near future we hope to switch to ints, longs or
* UUIDs representing column identifiers on disk, at which point we can switch that here as well.
*
* [timestamp][value offset][name size]][name extra][name offset deltas][cell names][value][Descendants]
* [ 8b ][ 4b ][ 2b ][ 1b ][ each 2b ][ arb < 64k][ arb ][ arbitrary ]
*
* descendants: any overriding classes will put their state here
* name offsets are deltas from their base offset, and don't include the first offset, or the end position of the final entry,
* i.e. there will be size - 1 entries, and each is a delta that is added to the offset of the position of the first name
* (which is always CELL_NAME_OFFSETS_OFFSET + (2 * (size - 1))). The length of the final name fills up any remaining
* space upto the value offset
* name extra: lowest 2 bits indicate the clustering size delta (i.e. how many name items are NOT part of the clustering key)
* the next 2 bits indicate the CellNameType
* the next bit indicates if the column is a static or clustered/dynamic column
* }
* </pre>
*/
public abstract class AbstractNativeCell extends AbstractCell implements CellName
{
static final int TIMESTAMP_OFFSET = 4;
private static final int VALUE_OFFSET_OFFSET = 12;
private static final int CELL_NAME_SIZE_OFFSET = 16;
private static final int CELL_NAME_EXTRA_OFFSET = 18;
private static final int CELL_NAME_OFFSETS_OFFSET = 19;
private static final int CELL_NAME_SIZE_DELTA_MASK = 3;
private static final int CELL_NAME_TYPE_SHIFT = 2;
private static final int CELL_NAME_TYPE_MASK = 7;
private static enum NameType
{
COMPOUND_DENSE(0 << 2), COMPOUND_SPARSE(1 << 2), COMPOUND_SPARSE_STATIC(2 << 2), SIMPLE_DENSE(3 << 2), SIMPLE_SPARSE(4 << 2);
static final NameType[] TYPES = NameType.values();
final int bits;
NameType(int bits)
{
this.bits = bits;
}
static NameType typeOf(CellName name)
{
if (name instanceof CompoundDenseCellName)
{
assert !name.isStatic();
return COMPOUND_DENSE;
}
if (name instanceof CompoundSparseCellName)
return name.isStatic() ? COMPOUND_SPARSE_STATIC : COMPOUND_SPARSE;
if (name instanceof SimpleDenseCellName)
{
assert !name.isStatic();
return SIMPLE_DENSE;
}
if (name instanceof SimpleSparseCellName)
{
assert !name.isStatic();
return SIMPLE_SPARSE;
}
if (name instanceof NativeCell)
return ((NativeCell) name).nametype();
throw new AssertionError();
}
}
private final long peer; // peer is assigned by peer updater in setPeer method
AbstractNativeCell()
{
peer = -1;
}
public AbstractNativeCell(NativeAllocator allocator, OpOrder.Group writeOp, Cell copyOf)
{
int size = sizeOf(copyOf);
peer = allocator.allocate(size, writeOp);
MemoryUtil.setInt(peer, size);
construct(copyOf);
}
protected int sizeOf(Cell cell)
{
int size = CELL_NAME_OFFSETS_OFFSET + Math.max(0, cell.name().size() - 1) * 2 + cell.value().remaining();
CellName name = cell.name();
for (int i = 0; i < name.size(); i++)
size += name.get(i).remaining();
return size;
}
protected void construct(Cell from)
{
setLong(TIMESTAMP_OFFSET, from.timestamp());
CellName name = from.name();
int nameSize = name.size();
int offset = CELL_NAME_SIZE_OFFSET;
setShort(offset, (short) nameSize);
assert nameSize - name.clusteringSize() <= 2;
byte cellNameExtraBits = (byte) ((nameSize - name.clusteringSize()) | NameType.typeOf(name).bits);
setByte(offset += 2, cellNameExtraBits);
offset += 1;
short cellNameDelta = 0;
for (int i = 1; i < nameSize; i++)
{
cellNameDelta += name.get(i - 1).remaining();
setShort(offset, cellNameDelta);
offset += 2;
}
for (int i = 0; i < nameSize; i++)
{
ByteBuffer bb = name.get(i);
setBytes(offset, bb);
offset += bb.remaining();
}
setInt(VALUE_OFFSET_OFFSET, offset);
setBytes(offset, from.value());
}
// the offset at which to read the short that gives the names
private int nameDeltaOffset(int i)
{
return CELL_NAME_OFFSETS_OFFSET + ((i - 1) * 2);
}
int valueStartOffset()
{
return getInt(VALUE_OFFSET_OFFSET);
}
private int valueEndOffset()
{
return (int) (internalSize() - postfixSize());
}
protected int postfixSize()
{
return 0;
}
@Override
public ByteBuffer value()
{
long offset = valueStartOffset();
return getByteBuffer(offset, (int) (internalSize() - (postfixSize() + offset))).order(ByteOrder.BIG_ENDIAN);
}
private int clusteringSizeDelta()
{
return getByte(CELL_NAME_EXTRA_OFFSET) & CELL_NAME_SIZE_DELTA_MASK;
}
public boolean isStatic()
{
return nametype() == NameType.COMPOUND_SPARSE_STATIC;
}
NameType nametype()
{
return NameType.TYPES[(((int) this.getByte(CELL_NAME_EXTRA_OFFSET)) >> CELL_NAME_TYPE_SHIFT) & CELL_NAME_TYPE_MASK];
}
public long minTimestamp()
{
return timestamp();
}
public long maxTimestamp()
{
return timestamp();
}
public int clusteringSize()
{
return size() - clusteringSizeDelta();
}
@Override
public ColumnIdentifier cql3ColumnName(CFMetaData metadata)
{
switch (nametype())
{
case SIMPLE_SPARSE:
return getIdentifier(metadata, get(clusteringSize()));
case COMPOUND_SPARSE_STATIC:
case COMPOUND_SPARSE:
ByteBuffer buffer = get(clusteringSize());
if (buffer.remaining() == 0)
return CompoundSparseCellNameType.rowMarkerId;
return getIdentifier(metadata, buffer);
case SIMPLE_DENSE:
case COMPOUND_DENSE:
return null;
default:
throw new AssertionError();
}
}
public ByteBuffer collectionElement()
{
return isCollectionCell() ? get(size() - 1) : null;
}
// we always have a collection element if our clustering size is 2 less than our total size,
// and we never have one otherwiss
public boolean isCollectionCell()
{
return clusteringSizeDelta() == 2;
}
public boolean isSameCQL3RowAs(CellNameType type, CellName other)
{
switch (nametype())
{
case SIMPLE_DENSE:
case COMPOUND_DENSE:
return type.compare(this, other) == 0;
case COMPOUND_SPARSE_STATIC:
case COMPOUND_SPARSE:
int clusteringSize = clusteringSize();
if (clusteringSize != other.clusteringSize() || other.isStatic() != isStatic())
return false;
for (int i = 0; i < clusteringSize; i++)
if (type.subtype(i).compare(get(i), other.get(i)) != 0)
return false;
return true;
case SIMPLE_SPARSE:
return true;
default:
throw new AssertionError();
}
}
public int size()
{
return getShort(CELL_NAME_SIZE_OFFSET);
}
public boolean isEmpty()
{
return size() == 0;
}
public ByteBuffer get(int i)
{
return get(i, null);
}
private ByteBuffer get(int i, AbstractAllocator copy)
{
// remember to take dense/sparse into account, and only return EOC when not dense
int size = size();
assert i >= 0 && i < size();
int cellNamesOffset = nameDeltaOffset(size);
int startDelta = i == 0 ? 0 : getShort(nameDeltaOffset(i));
int endDelta = i < size - 1 ? getShort(nameDeltaOffset(i + 1)) : valueStartOffset() - cellNamesOffset;
int length = endDelta - startDelta;
if (copy == null)
return getByteBuffer(cellNamesOffset + startDelta, length).order(ByteOrder.BIG_ENDIAN);
ByteBuffer result = copy.allocate(length);
FastByteOperations.UnsafeOperations.copy(null, peer + cellNamesOffset + startDelta, result, 0, length);
return result;
}
private static final ThreadLocal<byte[]> BUFFER = new ThreadLocal<byte[]>()
{
protected byte[] initialValue()
{
return new byte[256];
}
};
protected void writeComponentTo(MessageDigest digest, int i, boolean includeSize)
{
// remember to take dense/sparse into account, and only return EOC when not dense
int size = size();
assert i >= 0 && i < size();
int cellNamesOffset = nameDeltaOffset(size);
int startDelta = i == 0 ? 0 : getShort(nameDeltaOffset(i));
int endDelta = i < size - 1 ? getShort(nameDeltaOffset(i + 1)) : valueStartOffset() - cellNamesOffset;
int componentStart = cellNamesOffset + startDelta;
int count = endDelta - startDelta;
if (includeSize)
FBUtilities.updateWithShort(digest, count);
writeMemoryTo(digest, componentStart, count);
}
protected void writeMemoryTo(MessageDigest digest, int from, int count)
{
// only batch if we have more than 16 bytes remaining to transfer, otherwise fall-back to single-byte updates
int i = 0, batchEnd = count - 16;
if (i < batchEnd)
{
byte[] buffer = BUFFER.get();
while (i < batchEnd)
{
int transfer = Math.min(count - i, 256);
getBytes(from + i, buffer, 0, transfer);
digest.update(buffer, 0, transfer);
i += transfer;
}
}
while (i < count)
digest.update(getByte(from + i++));
}
public EOC eoc()
{
return EOC.NONE;
}
public Composite withEOC(EOC eoc)
{
throw new UnsupportedOperationException();
}
public Composite start()
{
throw new UnsupportedOperationException();
}
public Composite end()
{
throw new UnsupportedOperationException();
}
public ColumnSlice slice()
{
throw new UnsupportedOperationException();
}
public boolean isPrefixOf(CType type, Composite c)
{
if (size() > c.size() || isStatic() != c.isStatic())
return false;
for (int i = 0; i < size(); i++)
{
if (type.subtype(i).compare(get(i), c.get(i)) != 0)
return false;
}
return true;
}
public ByteBuffer toByteBuffer()
{
// for simple sparse we just return our one name buffer
switch (nametype())
{
case SIMPLE_DENSE:
case SIMPLE_SPARSE:
return get(0);
case COMPOUND_DENSE:
case COMPOUND_SPARSE_STATIC:
case COMPOUND_SPARSE:
// This is the legacy format of composites.
// See org.apache.cassandra.db.marshal.CompositeType for details.
ByteBuffer result = ByteBuffer.allocate(cellDataSize());
if (isStatic())
ByteBufferUtil.writeShortLength(result, CompositeType.STATIC_MARKER);
for (int i = 0; i < size(); i++)
{
ByteBuffer bb = get(i);
ByteBufferUtil.writeShortLength(result, bb.remaining());
result.put(bb);
result.put((byte) 0);
}
result.flip();
return result;
default:
throw new AssertionError();
}
}
protected void updateWithName(MessageDigest digest)
{
// for simple sparse we just return our one name buffer
switch (nametype())
{
case SIMPLE_DENSE:
case SIMPLE_SPARSE:
writeComponentTo(digest, 0, false);
break;
case COMPOUND_DENSE:
case COMPOUND_SPARSE_STATIC:
case COMPOUND_SPARSE:
// This is the legacy format of composites.
// See org.apache.cassandra.db.marshal.CompositeType for details.
if (isStatic())
FBUtilities.updateWithShort(digest, CompositeType.STATIC_MARKER);
for (int i = 0; i < size(); i++)
{
writeComponentTo(digest, i, true);
digest.update((byte) 0);
}
break;
default:
throw new AssertionError();
}
}
protected void updateWithValue(MessageDigest digest)
{
int offset = valueStartOffset();
int length = valueEndOffset() - offset;
writeMemoryTo(digest, offset, length);
}
@Override // this is the NAME dataSize, only!
public int dataSize()
{
switch (nametype())
{
case SIMPLE_DENSE:
case SIMPLE_SPARSE:
return valueStartOffset() - nameDeltaOffset(size());
case COMPOUND_DENSE:
case COMPOUND_SPARSE_STATIC:
case COMPOUND_SPARSE:
int size = size();
return valueStartOffset() - nameDeltaOffset(size) + 3 * size + (isStatic() ? 2 : 0);
default:
throw new AssertionError();
}
}
public boolean equals(Object obj)
{
if (obj == this)
return true;
if (obj instanceof CellName)
return equals((CellName) obj);
if (obj instanceof Cell)
return equals((Cell) obj);
return false;
}
public boolean equals(CellName that)
{
int size = this.size();
if (size != that.size())
return false;
for (int i = 0 ; i < size ; i++)
if (!get(i).equals(that.get(i)))
return false;
return true;
}
private static final ByteBuffer[] EMPTY = new ByteBuffer[0];
@Override
public CellName copy(CFMetaData cfm, AbstractAllocator allocator)
{
ByteBuffer[] r;
switch (nametype())
{
case SIMPLE_DENSE:
return CellNames.simpleDense(get(0, allocator));
case COMPOUND_DENSE:
r = new ByteBuffer[size()];
for (int i = 0; i < r.length; i++)
r[i] = get(i, allocator);
return CellNames.compositeDense(r);
case COMPOUND_SPARSE_STATIC:
case COMPOUND_SPARSE:
int clusteringSize = clusteringSize();
r = clusteringSize == 0 ? EMPTY : new ByteBuffer[clusteringSize()];
for (int i = 0; i < clusteringSize; i++)
r[i] = get(i, allocator);
ByteBuffer nameBuffer = get(r.length);
ColumnIdentifier name;
if (nameBuffer.remaining() == 0)
{
name = CompoundSparseCellNameType.rowMarkerId;
}
else
{
name = getIdentifier(cfm, nameBuffer);
}
if (clusteringSizeDelta() == 2)
{
ByteBuffer element = allocator.clone(get(size() - 1));
return CellNames.compositeSparseWithCollection(r, element, name, isStatic());
}
return CellNames.compositeSparse(r, name, isStatic());
case SIMPLE_SPARSE:
return CellNames.simpleSparse(getIdentifier(cfm, get(0)));
}
throw new IllegalStateException();
}
private static ColumnIdentifier getIdentifier(CFMetaData cfMetaData, ByteBuffer name)
{
ColumnDefinition def = cfMetaData.getColumnDefinition(name);
if (def != null)
{
return def.name;
}
else
{
// it's safe to simply grab based on clusteringPrefixSize() as we are only called if not a dense type
AbstractType<?> type = cfMetaData.comparator.subtype(cfMetaData.comparator.clusteringPrefixSize());
return new ColumnIdentifier(HeapAllocator.instance.clone(name), type);
}
}
@Override
public Cell withUpdatedName(CellName newName)
{
throw new UnsupportedOperationException();
}
@Override
public Cell withUpdatedTimestamp(long newTimestamp)
{
throw new UnsupportedOperationException();
}
protected long internalSize()
{
return MemoryUtil.getInt(peer);
}
private void checkPosition(long offset, long size)
{
assert size >= 0;
assert peer > 0 : "Memory was freed";
assert offset >= 0 && offset + size <= internalSize() : String.format("Illegal range: [%d..%d), size: %s", offset, offset + size, internalSize());
}
protected final void setByte(long offset, byte b)
{
checkPosition(offset, 1);
MemoryUtil.setByte(peer + offset, b);
}
protected final void setShort(long offset, short s)
{
checkPosition(offset, 1);
MemoryUtil.setShort(peer + offset, s);
}
protected final void setInt(long offset, int l)
{
checkPosition(offset, 4);
MemoryUtil.setInt(peer + offset, l);
}
protected final void setLong(long offset, long l)
{
checkPosition(offset, 8);
MemoryUtil.setLong(peer + offset, l);
}
protected final void setBytes(long offset, ByteBuffer buffer)
{
int start = buffer.position();
int count = buffer.limit() - start;
if (count == 0)
return;
checkPosition(offset, count);
MemoryUtil.setBytes(peer + offset, buffer);
}
protected final byte getByte(long offset)
{
checkPosition(offset, 1);
return MemoryUtil.getByte(peer + offset);
}
protected final void getBytes(long offset, byte[] trg, int trgOffset, int count)
{
checkPosition(offset, count);
MemoryUtil.getBytes(peer + offset, trg, trgOffset, count);
}
protected final int getShort(long offset)
{
checkPosition(offset, 2);
return MemoryUtil.getShort(peer + offset);
}
protected final int getInt(long offset)
{
checkPosition(offset, 4);
return MemoryUtil.getInt(peer + offset);
}
protected final long getLong(long offset)
{
checkPosition(offset, 8);
return MemoryUtil.getLong(peer + offset);
}
protected final ByteBuffer getByteBuffer(long offset, int length)
{
checkPosition(offset, length);
return MemoryUtil.getByteBuffer(peer + offset, length);
}
// requires isByteOrderComparable to be true. Compares the name components only; ; may need to compare EOC etc still
@Inline
public final int compareTo(final Composite that)
{
if (isStatic() != that.isStatic())
{
// Static sorts before non-static no matter what, except for empty which
// always sort first
if (isEmpty())
return that.isEmpty() ? 0 : -1;
if (that.isEmpty())
return 1;
return isStatic() ? -1 : 1;
}
int size = size();
int size2 = that.size();
int minSize = Math.min(size, size2);
int startDelta = 0;
int cellNamesOffset = nameDeltaOffset(size);
for (int i = 0 ; i < minSize ; i++)
{
int endDelta = i < size - 1 ? getShort(nameDeltaOffset(i + 1)) : valueStartOffset() - cellNamesOffset;
long offset = peer + cellNamesOffset + startDelta;
int length = endDelta - startDelta;
int cmp = FastByteOperations.UnsafeOperations.compareTo(null, offset, length, that.get(i));
if (cmp != 0)
return cmp;
startDelta = endDelta;
}
EOC eoc = that.eoc();
if (size == size2)
return this.eoc().compareTo(eoc);
return size < size2 ? this.eoc().prefixComparisonResult : -eoc.prefixComparisonResult;
}
public final int compareToSimple(final Composite that)
{
assert size() == 1 && that.size() == 1;
int length = valueStartOffset() - nameDeltaOffset(1);
long offset = peer + nameDeltaOffset(1);
return FastByteOperations.UnsafeOperations.compareTo(null, offset, length, that.get(0));
}
}

101
src/java/org/apache/cassandra/db/AbstractRangeCommand.java
View File

@ -1,101 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.db;
import java.util.List;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.db.filter.*;
import org.apache.cassandra.db.index.*;
import org.apache.cassandra.dht.AbstractBounds;
import org.apache.cassandra.net.MessageOut;
import org.apache.cassandra.service.IReadCommand;
public abstract class AbstractRangeCommand implements IReadCommand
{
public final String keyspace;
public final String columnFamily;
public final long timestamp;
public final AbstractBounds<RowPosition> keyRange;
public final IDiskAtomFilter predicate;
public final List<IndexExpression> rowFilter;
public final SecondaryIndexSearcher searcher;
public AbstractRangeCommand(String keyspace, String columnFamily, long timestamp, AbstractBounds<RowPosition> keyRange, IDiskAtomFilter predicate, List<IndexExpression> rowFilter)
{
this.keyspace = keyspace;
this.columnFamily = columnFamily;
this.timestamp = timestamp;
this.keyRange = keyRange;
this.predicate = predicate;
this.rowFilter = rowFilter;
SecondaryIndexManager indexManager = Keyspace.open(keyspace).getColumnFamilyStore(columnFamily).indexManager;
this.searcher = indexManager.getHighestSelectivityIndexSearcher(rowFilter);
}
public boolean requiresScanningAllRanges()
{
return searcher != null && searcher.requiresScanningAllRanges(rowFilter);
}
public List<Row> postReconciliationProcessing(List<Row> rows)
{
return searcher == null ? trim(rows) : trim(searcher.postReconciliationProcessing(rowFilter, rows));
}
private List<Row> trim(List<Row> rows)
{
if (countCQL3Rows() || ignoredTombstonedPartitions())
return rows;
else
return rows.size() > limit() ? rows.subList(0, limit()) : rows;
}
public String getKeyspace()
{
return keyspace;
}
public abstract MessageOut<? extends AbstractRangeCommand> createMessage();
public abstract AbstractRangeCommand forSubRange(AbstractBounds<RowPosition> range);
public abstract AbstractRangeCommand withUpdatedLimit(int newLimit);
public abstract int limit();
public abstract boolean countCQL3Rows();
/**
* Returns true if tombstoned partitions should not be included in results or count towards the limit.
* See CASSANDRA-8490 for more details on why this is needed (and done this way).
* */
public boolean ignoredTombstonedPartitions()
{
if (!(predicate instanceof SliceQueryFilter))
return false;
return ((SliceQueryFilter) predicate).compositesToGroup == SliceQueryFilter.IGNORE_TOMBSTONED_PARTITIONS;
}
public abstract List<Row> executeLocally();
public long getTimeout()
{
return DatabaseDescriptor.getRangeRpcTimeout();
}
}

62
src/java/org/apache/cassandra/db/Aliasable.java Normal file
View File

@ -0,0 +1,62 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.db;
/**
* This interface marks objects that are only valid in a restricted scope and
* shouldn't be simply aliased outside of this scope (in other words, you should
* not keep a reference to the object that escaped said scope as the object will
* likely become invalid).
*
* For instance, most {@link RowIterator} implementation reuse the same {@link
* Row} object during iteration. This means that the following code would be
* incorrect.
* <pre>
* RowIterator iter = ...;
* Row someRow = null;
* while (iter.hasNext())
* {
* Row row = iter.next();
* if (someCondition(row))
* someRow = row; // This isn't safe
* doSomethingElse();
* }
* useRow(someRow);
* </pre>
* The problem being that, because the row iterator reuse the same object,
* {@code someRow} will not point to the row that had met {@code someCondition}
* at the end of the iteration ({@code someRow} will point to the last iterated
* row in practice).
*
* When code do need to alias such {@code Aliasable} object, it should call the
* {@code takeAlias} method that will make a copy of the object if necessary.
*
* Of course, the {@code takeAlias} should not be abused, as it defeat the purpose
* of sharing objects in the first place.
*
* Also note that some implementation of an {@code Aliasable} object may be
* safe to alias, in which case its {@code takeAlias} method will be a no-op.
*/
public interface Aliasable<T>
{
/**
* Returns either this object (if it's safe to alias) or a copy of it
* (it it isn't safe to alias).
*/
public T takeAlias();
}

774
src/java/org/apache/cassandra/db/ArrayBackedSortedColumns.java
View File

@ -1,774 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.db;
import java.util.*;
import com.google.common.base.Function;
import com.google.common.collect.AbstractIterator;
import com.google.common.collect.Iterables;
import org.apache.cassandra.config.CFMetaData;
import org.apache.cassandra.db.composites.CellName;
import org.apache.cassandra.db.composites.Composite;
import org.apache.cassandra.db.filter.ColumnSlice;
import org.apache.cassandra.utils.BatchRemoveIterator;
import org.apache.cassandra.utils.memory.AbstractAllocator;
import org.apache.cassandra.utils.SearchIterator;
/**
* A ColumnFamily backed by an array.
* This implementation is not synchronized and should only be used when
* thread-safety is not required. This implementation makes sense when the
* main operations performed are iterating over the cells and adding cells
* (especially if insertion is in sorted order).
*/
public class ArrayBackedSortedColumns extends ColumnFamily
{
private static final Cell[] EMPTY_ARRAY = new Cell[0];
private static final int MINIMAL_CAPACITY = 10;
private final boolean reversed;
private DeletionInfo deletionInfo;
private Cell[] cells;
private int size;
private int sortedSize;
private volatile boolean isSorted;
public static final ColumnFamily.Factory<ArrayBackedSortedColumns> factory = new Factory<ArrayBackedSortedColumns>()
{
public ArrayBackedSortedColumns create(CFMetaData metadata, boolean insertReversed, int initialCapacity)
{
return new ArrayBackedSortedColumns(metadata, insertReversed, initialCapacity == 0 ? EMPTY_ARRAY : new Cell[initialCapacity], 0, 0);
}
};
private ArrayBackedSortedColumns(CFMetaData metadata, boolean reversed, Cell[] cells, int size, int sortedSize)
{
super(metadata);
this.reversed = reversed;
this.deletionInfo = DeletionInfo.live();
this.cells = cells;
this.size = size;
this.sortedSize = sortedSize;
this.isSorted = size == sortedSize;
}
protected ArrayBackedSortedColumns(CFMetaData metadata, boolean reversed)
{
this(metadata, reversed, EMPTY_ARRAY, 0, 0);
}
private ArrayBackedSortedColumns(ArrayBackedSortedColumns original)
{
super(original.metadata);
this.reversed = original.reversed;
this.deletionInfo = DeletionInfo.live(); // this is INTENTIONALLY not set to original.deletionInfo.
this.cells = Arrays.copyOf(original.cells, original.size);
this.size = original.size;
this.sortedSize = original.sortedSize;
this.isSorted = original.isSorted;
}
public static ArrayBackedSortedColumns localCopy(ColumnFamily original, AbstractAllocator allocator)
{
ArrayBackedSortedColumns copy = new ArrayBackedSortedColumns(original.metadata, false, new Cell[original.getColumnCount()], 0, 0);
for (Cell cell : original)
copy.internalAdd(cell.localCopy(original.metadata, allocator));
copy.sortedSize = copy.size; // internalAdd doesn't update sortedSize.
copy.delete(original);
return copy;
}
public ColumnFamily.Factory getFactory()
{
return factory;
}
public ColumnFamily cloneMe()
{
return new ArrayBackedSortedColumns(this);
}
public boolean isInsertReversed()
{
return reversed;
}
public BatchRemoveIterator<Cell> batchRemoveIterator()
{
maybeSortCells();
return new BatchRemoveIterator<Cell>()
{
private final Iterator<Cell> iter = iterator();
private BitSet removedIndexes = new BitSet(size);
private int idx = -1;
private boolean shouldCallNext = false;
private boolean isCommitted = false;
private boolean removedAnything = false;
public void commit()
{
if (isCommitted)
throw new IllegalStateException();
isCommitted = true;
if (!removedAnything)
return;
int retainedCount = 0;
int clearIdx, setIdx = -1;
// shift all [clearIdx, setIdx) segments to the left, skipping any removed columns
while (true)
{
clearIdx = removedIndexes.nextClearBit(setIdx + 1);
if (clearIdx >= size)
break; // nothing left to retain
setIdx = removedIndexes.nextSetBit(clearIdx + 1);
if (setIdx < 0)
setIdx = size; // no removals past retainIdx - copy all remaining cells
if (retainedCount != clearIdx)
System.arraycopy(cells, clearIdx, cells, retainedCount, setIdx - clearIdx);
retainedCount += (setIdx - clearIdx);
}
for (int i = retainedCount; i < size; i++)
cells[i] = null;
size = sortedSize = retainedCount;
}
public boolean hasNext()
{
return iter.hasNext();
}
public Cell next()
{
idx++;
shouldCallNext = false;
return iter.next();
}
public void remove()
{
if (shouldCallNext)
throw new IllegalStateException();
removedIndexes.set(reversed ? size - idx - 1 : idx);
removedAnything = true;
shouldCallNext = true;
}
};
}
private Comparator<Composite> internalComparator()
{
return reversed ? getComparator().reverseComparator() : getComparator();
}
private void maybeSortCells()
{
if (!isSorted)
sortCells();
}
/**
* synchronized so that concurrent (read-only) accessors don't mess the internal state.
*/
private synchronized void sortCells()
{
if (isSorted)
return; // Just sorted by a previous call
Comparator<Cell> comparator = reversed
? getComparator().columnReverseComparator()
: getComparator().columnComparator(false);
// Sort the unsorted segment - will still potentially contain duplicate (non-reconciled) cells
Arrays.sort(cells, sortedSize, size, comparator);
// Determine the merge start position for that segment
int pos = binarySearch(0, sortedSize, cells[sortedSize].name(), internalComparator());
if (pos < 0)
pos = -pos - 1;
// Copy [pos, lastSortedCellIndex] cells into a separate array
Cell[] leftCopy = pos == sortedSize
? EMPTY_ARRAY
: Arrays.copyOfRange(cells, pos, sortedSize);
// Store the beginning (inclusive) and the end (exclusive) indexes of the right segment
int rightStart = sortedSize;
int rightEnd = size;
// 'Trim' the sizes to what's left without the leftCopy
size = sortedSize = pos;
// Merge the cells from both segments. When adding from the left segment we can rely on it not having any
// duplicate cells, and thus omit the comparison with the previously entered cell - we'll never need to reconcile.
int l = 0, r = rightStart;
while (l < leftCopy.length && r < rightEnd)
{
int cmp = comparator.compare(leftCopy[l], cells[r]);
if (cmp < 0)
append(leftCopy[l++]);
else if (cmp == 0)
append(leftCopy[l++].reconcile(cells[r++]));
else
appendOrReconcile(cells[r++]);
}
while (l < leftCopy.length)
append(leftCopy[l++]);
while (r < rightEnd)
appendOrReconcile(cells[r++]);
// Nullify the remainder of the array (in case we had duplicate cells that got reconciled)
for (int i = size; i < rightEnd; i++)
cells[i] = null;
// Fully sorted at this point
isSorted = true;
}
private void appendOrReconcile(Cell cell)
{
if (size > 0 && cells[size - 1].name().equals(cell.name()))
reconcileWith(size - 1, cell);
else
append(cell);
}
private void append(Cell cell)
{
cells[size] = cell;
size++;
sortedSize++;
}
public Cell getColumn(CellName name)
{
maybeSortCells();
int pos = binarySearch(name);
return pos >= 0 ? cells[pos] : null;
}
/**
* Adds a cell, assuming that:
* - it's non-gc-able (if a tombstone) or not a tombstone
* - it has a more recent timestamp than any partition/range tombstone shadowing it
* - it sorts *strictly after* the current-last cell in the array.
*/
public void maybeAppendColumn(Cell cell, DeletionInfo.InOrderTester tester, int gcBefore)
{
if (cell.getLocalDeletionTime() >= gcBefore && !tester.isDeleted(cell))
appendColumn(cell);
}
/**
* Adds a cell, assuming that it sorts *strictly after* the current-last cell in the array.
*/
public void appendColumn(Cell cell)
{
internalAdd(cell);
sortedSize++;
}
public void addColumn(Cell cell)
{
if (size == 0)
{
internalAdd(cell);
sortedSize++;
return;
}
if (!isSorted)
{
internalAdd(cell);
return;
}
int c = internalComparator().compare(cells[size - 1].name(), cell.name());
if (c < 0)
{
// Append to the end
internalAdd(cell);
sortedSize++;
}
else if (c == 0)
{
// Resolve against the last cell
reconcileWith(size - 1, cell);
}
else
{
int pos = binarySearch(cell.name());
if (pos >= 0) // Reconcile with an existing cell
{
reconcileWith(pos, cell);
}
else
{
internalAdd(cell); // Append to the end, making cells unsorted from now on
isSorted = false;
}
}
}
public void addAll(ColumnFamily other)
{
delete(other.deletionInfo());
if (!other.hasColumns())
return;
// In reality, with ABSC being the only remaining container (aside from ABTC), other will aways be ABSC.
if (size == 0 && other instanceof ArrayBackedSortedColumns)
{
fastAddAll((ArrayBackedSortedColumns) other);
}
else
{
Iterator<Cell> iterator = reversed ? other.reverseIterator() : other.iterator();
while (iterator.hasNext())
addColumn(iterator.next());
}
}
// Fast path, when this ABSC is empty.
private void fastAddAll(ArrayBackedSortedColumns other)
{
if (other.isInsertReversed() == isInsertReversed())
{
cells = Arrays.copyOf(other.cells, other.cells.length);
size = other.size;
sortedSize = other.sortedSize;
isSorted = other.isSorted;
}
else
{
if (cells.length < other.getColumnCount())
cells = new Cell[Math.max(MINIMAL_CAPACITY, other.getColumnCount())];
Iterator<Cell> iterator = reversed ? other.reverseIterator() : other.iterator();
while (iterator.hasNext())
cells[size++] = iterator.next();
sortedSize = size;
isSorted = true;
}
}
/**
* Add a cell to the array, 'resizing' it first if necessary (if it doesn't fit).
*/
private void internalAdd(Cell cell)
{
if (cells.length == size)
cells = Arrays.copyOf(cells, Math.max(MINIMAL_CAPACITY, size * 3 / 2 + 1));
cells[size++] = cell;
}
/**
* Remove the cell at a given index, shifting the rest of the array to the left if needed.
* Please note that we mostly remove from the end, so the shifting should be rare.
*/
private void internalRemove(int index)
{
int moving = size - index - 1;
if (moving > 0)
System.arraycopy(cells, index + 1, cells, index, moving);
cells[--size] = null;
}
/**
* Reconcile with a cell at position i.
* Assume that i is a valid position.
*/
private void reconcileWith(int i, Cell cell)
{
cells[i] = cell.reconcile(cells[i]);
}
private int binarySearch(CellName name)
{
return binarySearch(0, size, name, internalComparator());
}
/**
* Simple binary search for a given cell name.
* The return value has the exact same meaning that the one of Collections.binarySearch().
* (We don't use Collections.binarySearch() directly because it would require us to create
* a fake Cell (as well as an Cell comparator) to do the search, which is ugly.
*/
private int binarySearch(int fromIndex, int toIndex, Composite name, Comparator<Composite> comparator)
{
int low = fromIndex;
int mid = toIndex;
int high = mid - 1;
int result = -1;
while (low <= high)
{
mid = (low + high) >> 1;
if ((result = comparator.compare(name, cells[mid].name())) > 0)
low = mid + 1;
else if (result == 0)
return mid;
else
high = mid - 1;
}
return -mid - (result < 0 ? 1 : 2);
}
public Collection<Cell> getSortedColumns()
{
return new CellCollection(reversed);
}
public Collection<Cell> getReverseSortedColumns()
{
return new CellCollection(!reversed);
}
public int getColumnCount()
{
maybeSortCells();
return size;
}
public boolean hasColumns()
{
return size > 0;
}
public void clear()
{
setDeletionInfo(DeletionInfo.live());
for (int i = 0; i < size; i++)
cells[i] = null;
size = sortedSize = 0;
isSorted = true;
}
public DeletionInfo deletionInfo()
{
return deletionInfo;
}
public void delete(DeletionTime delTime)
{
deletionInfo.add(delTime);
}
public void delete(DeletionInfo newInfo)
{
deletionInfo.add(newInfo);
}
protected void delete(RangeTombstone tombstone)
{
deletionInfo.add(tombstone, getComparator());
}
public void setDeletionInfo(DeletionInfo newInfo)
{
deletionInfo = newInfo;
}
/**
* Purges any tombstones with a local deletion time before gcBefore.
* @param gcBefore a timestamp (in seconds) before which tombstones should be purged
*/
public void purgeTombstones(int gcBefore)
{
deletionInfo.purge(gcBefore);
}
public Iterable<CellName> getColumnNames()
{
return Iterables.transform(new CellCollection(false), new Function<Cell, CellName>()
{
public CellName apply(Cell cell)
{
return cell.name();
}
});
}
public Iterator<Cell> iterator(ColumnSlice[] slices)
{
maybeSortCells();
return slices.length == 1
? slice(slices[0], reversed, null)
: new SlicesIterator(slices, reversed);
}
public Iterator<Cell> reverseIterator(ColumnSlice[] slices)
{
maybeSortCells();
return slices.length == 1
? slice(slices[0], !reversed, null)
: new SlicesIterator(slices, !reversed);
}
public SearchIterator<CellName, Cell> searchIterator()
{
maybeSortCells();
return new SearchIterator<CellName, Cell>()
{
// the first index that we could find the next key at, i.e. one larger
// than the last key's location
private int i = 0;
// We assume a uniform distribution of keys,
// so we keep track of how many keys were skipped to satisfy last lookup, and only look at twice that
// many keys for next lookup initially, extending to whole range only if we couldn't find it in that subrange
private int range = size / 2;
public boolean hasNext()
{
return i < size;
}
public Cell next(CellName name)
{
if (!isSorted || !hasNext())
throw new IllegalStateException();
// optimize for runs of sequential matches, as in CollationController
// checking to see if we've found the desired cells yet (CASSANDRA-6933)
int c = metadata.comparator.compare(name, cells[i].name());
if (c <= 0)
return c < 0 ? null : cells[i++];
// use range to manually force a better bsearch "pivot" by breaking it into two calls:
// first for i..i+range, then i+range..size if necessary.
// https://issues.apache.org/jira/browse/CASSANDRA-6933?focusedCommentId=13958264&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13958264
int limit = Math.min(size, i + range);
int i2 = binarySearch(i + 1, limit, name, internalComparator());
if (-1 - i2 == limit)
i2 = binarySearch(limit, size, name, internalComparator());
// i2 can't be zero since we already checked cells[i] above
if (i2 > 0)
{
range = i2 - i;
i = i2 + 1;
return cells[i2];
}
i2 = -1 - i2;
range = i2 - i;
i = i2;
return null;
}
};
}
private class SlicesIterator extends AbstractIterator<Cell>
{
private final ColumnSlice[] slices;
private final boolean invert;
private int idx = 0;
private int previousSliceEnd;
private Iterator<Cell> currentSlice;
public SlicesIterator(ColumnSlice[] slices, boolean invert)
{
this.slices = slices;
this.invert = invert;
previousSliceEnd = invert ? size : 0;
}
protected Cell computeNext()
{
if (currentSlice == null)
{
if (idx >= slices.length)
return endOfData();
currentSlice = slice(slices[idx++], invert, this);
}
if (currentSlice.hasNext())
return currentSlice.next();
currentSlice = null;
return computeNext();
}
}
/**
* @return a sub-range of our cells as an Iterator, between the provided composites (inclusive)
*
* @param slice The slice with the inclusive start and finish bounds
* @param invert If the sort order of our collection is opposite to the desired sort order of the result;
* this results in swapping the start/finish (since they are provided based on the desired
* sort order, not our sort order), to normalise to our sort order, and a backwards iterator is returned
* @param iter If this slice is part of a multi-slice, the iterator will be updated to ensure cells are visited only once
*/
private Iterator<Cell> slice(ColumnSlice slice, boolean invert, SlicesIterator iter)
{
Composite start = invert ? slice.finish : slice.start;
Composite finish = invert ? slice.start : slice.finish;
int lowerBound = 0, upperBound = size;
if (iter != null)
{
if (invert)
upperBound = iter.previousSliceEnd;
else
lowerBound = iter.previousSliceEnd;
}
if (!start.isEmpty())
{
lowerBound = binarySearch(lowerBound, upperBound, start, internalComparator());
if (lowerBound < 0)
lowerBound = -lowerBound - 1;
}
if (!finish.isEmpty())
{
upperBound = binarySearch(lowerBound, upperBound, finish, internalComparator());
upperBound = upperBound < 0
? -upperBound - 1
: upperBound + 1; // upperBound is exclusive for the iterators
}
// If we're going backwards (wrt our sort order) we store the startIdx and use it as our upper bound next round
if (iter != null)
iter.previousSliceEnd = invert ? lowerBound : upperBound;
return invert
? new BackwardsCellIterator(lowerBound, upperBound)
: new ForwardsCellIterator(lowerBound, upperBound);
}
private final class BackwardsCellIterator implements Iterator<Cell>
{
private int idx, end;
private boolean shouldCallNext = true;
// lowerBound inclusive, upperBound exclusive
private BackwardsCellIterator(int lowerBound, int upperBound)
{
idx = upperBound - 1;
end = lowerBound - 1;
}
public boolean hasNext()
{
return idx > end;
}
public Cell next()
{
try
{
shouldCallNext = false;
return cells[idx--];
}
catch (ArrayIndexOutOfBoundsException e)
{
NoSuchElementException ne = new NoSuchElementException(e.getMessage());
ne.initCause(e);
throw ne;
}
}
public void remove()
{
if (shouldCallNext)
throw new IllegalStateException();
shouldCallNext = true;
internalRemove(idx + 1);
sortedSize--;
}
}
private final class ForwardsCellIterator implements Iterator<Cell>
{
private int idx, end;
private boolean shouldCallNext = true;
// lowerBound inclusive, upperBound exclusive
private ForwardsCellIterator(int lowerBound, int upperBound)
{
idx = lowerBound;
end = upperBound;
}
public boolean hasNext()
{
return idx < end;
}
public Cell next()
{
try
{
shouldCallNext = false;
return cells[idx++];
}
catch (ArrayIndexOutOfBoundsException e)
{
NoSuchElementException ne = new NoSuchElementException(e.getMessage());
ne.initCause(e);
throw ne;
}
}
public void remove()
{
if (shouldCallNext)
throw new IllegalStateException();
shouldCallNext = true;
internalRemove(--idx);
sortedSize--;
end--;
}
}
private final class CellCollection extends AbstractCollection<Cell>
{
private final boolean invert;
private CellCollection(boolean invert)
{
this.invert = invert;
}
public int size()
{
return getColumnCount();
}
public Iterator<Cell> iterator()
{
maybeSortCells();
return invert
? new BackwardsCellIterator(0, size)
: new ForwardsCellIterator(0, size);
}
}
}

128
src/java/org/apache/cassandra/db/AtomDeserializer.java
View File

@ -1,128 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.db;
import java.io.DataInput;
import java.io.IOException;
import org.apache.cassandra.db.composites.CellName;
import org.apache.cassandra.db.composites.CellNameType;
import org.apache.cassandra.db.composites.Composite;
import org.apache.cassandra.io.sstable.format.Version;
/**
* Helper class to deserialize OnDiskAtom efficiently.
*
* More precisely, this class is used by the low-level readers
* (IndexedSliceReader and SSTableNamesIterator) to ensure we don't
* do more work than necessary (i.e. we don't allocate/deserialize
* objects for things we don't care about).
*/
public class AtomDeserializer
{
private final CellNameType type;
private final CellNameType.Deserializer nameDeserializer;
private final DataInput in;
private final ColumnSerializer.Flag flag;
private final int expireBefore;
private final Version version;
// The "flag" for the next name (which correspond to the "masks" in ColumnSerializer) if it has been
// read already, Integer.MIN_VALUE otherwise;
private int nextFlags = Integer.MIN_VALUE;
public AtomDeserializer(CellNameType type, DataInput in, ColumnSerializer.Flag flag, int expireBefore, Version version)
{
this.type = type;
this.nameDeserializer = type.newDeserializer(in);
this.in = in;
this.flag = flag;
this.expireBefore = expireBefore;
this.version = version;
}
/**
* Whether or not there is more atom to read.
*/
public boolean hasNext() throws IOException
{
return nameDeserializer.hasNext();
}
/**
* Whether or not some atom has been read but not processed (neither readNext() nor
* skipNext() has been called for that atom) yet.
*/
public boolean hasUnprocessed() throws IOException
{
return nameDeserializer.hasUnprocessed();
}
/**
* Compare the provided composite to the next atom to read on disk.
*
* This will not read/deserialize the whole atom but only what is necessary for the
* comparison. Whenever we know what to do with this atom (read it or skip it),
* readNext or skipNext should be called.
*/
public int compareNextTo(Composite composite) throws IOException
{
return nameDeserializer.compareNextTo(composite);
}
/**
* Returns whether the next atom is a range tombstone or not.
*
* Please note that this should only be called after compareNextTo() has been called.
*/
public boolean nextIsRangeTombstone() throws IOException
{
nextFlags = in.readUnsignedByte();
return (nextFlags & ColumnSerializer.RANGE_TOMBSTONE_MASK) != 0;
}
/**
* Returns the next atom.
*/
public OnDiskAtom readNext() throws IOException
{
Composite name = nameDeserializer.readNext();
assert !name.isEmpty(); // This would imply hasNext() hasn't been called
nextFlags = nextFlags == Integer.MIN_VALUE ? in.readUnsignedByte() : nextFlags;
OnDiskAtom atom = (nextFlags & ColumnSerializer.RANGE_TOMBSTONE_MASK) != 0
? type.rangeTombstoneSerializer().deserializeBody(in, name, version)
: type.columnSerializer().deserializeColumnBody(in, (CellName)name, nextFlags, flag, expireBefore);
nextFlags = Integer.MIN_VALUE;
return atom;
}
/**
* Skips the next atom.
*/
public void skipNext() throws IOException
{
nameDeserializer.skipNext();
nextFlags = nextFlags == Integer.MIN_VALUE ? in.readUnsignedByte() : nextFlags;
if ((nextFlags & ColumnSerializer.RANGE_TOMBSTONE_MASK) != 0)
type.rangeTombstoneSerializer().skipBody(in, version);
else
type.columnSerializer().skipColumnBody(in, nextFlags);
nextFlags = Integer.MIN_VALUE;
}
}

578
src/java/org/apache/cassandra/db/AtomicBTreeColumns.java
View File

@ -1,578 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.db;
import java.util.AbstractCollection;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Comparator;
import java.util.Iterator;
import java.util.List;
import java.util.concurrent.atomic.AtomicIntegerFieldUpdater;
import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
import com.google.common.base.Function;
import com.google.common.base.Functions;
import com.google.common.collect.AbstractIterator;
import com.google.common.collect.Iterators;
import org.apache.cassandra.config.CFMetaData;
import org.apache.cassandra.db.composites.CellName;
import org.apache.cassandra.db.composites.Composite;
import org.apache.cassandra.db.filter.ColumnSlice;
import org.apache.cassandra.db.marshal.BytesType;
import org.apache.cassandra.utils.*;
import org.apache.cassandra.utils.SearchIterator;
import org.apache.cassandra.utils.btree.BTree;
import org.apache.cassandra.utils.btree.BTreeSearchIterator;
import org.apache.cassandra.utils.btree.UpdateFunction;
import org.apache.cassandra.utils.concurrent.Locks;
import org.apache.cassandra.utils.concurrent.OpOrder;
import org.apache.cassandra.utils.memory.HeapAllocator;
import org.apache.cassandra.utils.memory.MemtableAllocator;
import org.apache.cassandra.utils.memory.NativePool;
import static org.apache.cassandra.db.index.SecondaryIndexManager.Updater;
/**
* A thread-safe and atomic ISortedColumns implementation.
* Operations (in particular addAll) on this implemenation are atomic and
* isolated (in the sense of ACID). Typically a addAll is guaranteed that no
* other thread can see the state where only parts but not all columns have
* been added.
* <p>
* WARNING: removing element through getSortedColumns().iterator() is *not* supported
* </p>
*/
public class AtomicBTreeColumns extends ColumnFamily
{
static final long EMPTY_SIZE = ObjectSizes.measure(new AtomicBTreeColumns(CFMetaData.denseCFMetaData("keyspace", "table", BytesType.instance), null))
+ ObjectSizes.measure(new Holder(null, null));
// Reserved values for wasteTracker field. These values must not be consecutive (see avoidReservedValues)
private static final int TRACKER_NEVER_WASTED = 0;
private static final int TRACKER_PESSIMISTIC_LOCKING = Integer.MAX_VALUE;
// The granularity with which we track wasted allocation/work; we round up
private static final int ALLOCATION_GRANULARITY_BYTES = 1024;
// The number of bytes we have to waste in excess of our acceptable realtime rate of waste (defined below)
private static final long EXCESS_WASTE_BYTES = 10 * 1024 * 1024L;
private static final int EXCESS_WASTE_OFFSET = (int) (EXCESS_WASTE_BYTES / ALLOCATION_GRANULARITY_BYTES);
// Note this is a shift, because dividing a long time and then picking the low 32 bits doesn't give correct rollover behavior
private static final int CLOCK_SHIFT = 17;
// CLOCK_GRANULARITY = 1^9ns >> CLOCK_SHIFT == 132us == (1/7.63)ms
/**
* (clock + allocation) granularity are combined to give us an acceptable (waste) allocation rate that is defined by
* the passage of real time of ALLOCATION_GRANULARITY_BYTES/CLOCK_GRANULARITY, or in this case 7.63Kb/ms, or 7.45Mb/s
*
* in wasteTracker we maintain within EXCESS_WASTE_OFFSET before the current time; whenever we waste bytes
* we increment the current value if it is within this window, and set it to the min of the window plus our waste
* otherwise.
*/
private volatile int wasteTracker = TRACKER_NEVER_WASTED;
private static final AtomicIntegerFieldUpdater<AtomicBTreeColumns> wasteTrackerUpdater = AtomicIntegerFieldUpdater.newUpdater(AtomicBTreeColumns.class, "wasteTracker");
private static final Function<Cell, CellName> NAME = new Function<Cell, CellName>()
{
public CellName apply(Cell column)
{
return column.name();
}
};
public static final Factory<AtomicBTreeColumns> factory = new Factory<AtomicBTreeColumns>()
{
public AtomicBTreeColumns create(CFMetaData metadata, boolean insertReversed, int initialCapacity)
{
if (insertReversed)
throw new IllegalArgumentException();
return new AtomicBTreeColumns(metadata);
}
};
private static final DeletionInfo LIVE = DeletionInfo.live();
// This is a small optimization: DeletionInfo is mutable, but we know that we will always copy it in that class,
// so we can safely alias one DeletionInfo.live() reference and avoid some allocations.
private static final Holder EMPTY = new Holder(BTree.empty(), LIVE);
private volatile Holder ref;
private static final AtomicReferenceFieldUpdater<AtomicBTreeColumns, Holder> refUpdater = AtomicReferenceFieldUpdater.newUpdater(AtomicBTreeColumns.class, Holder.class, "ref");
private AtomicBTreeColumns(CFMetaData metadata)
{
this(metadata, EMPTY);
}
private AtomicBTreeColumns(CFMetaData metadata, Holder holder)
{
super(metadata);
this.ref = holder;
}
public Factory getFactory()
{
return factory;
}
public ColumnFamily cloneMe()
{
return new AtomicBTreeColumns(metadata, ref);
}
public DeletionInfo deletionInfo()
{
return ref.deletionInfo;
}
public void delete(DeletionTime delTime)
{
delete(new DeletionInfo(delTime));
}
protected void delete(RangeTombstone tombstone)
{
delete(new DeletionInfo(tombstone, getComparator()));
}
public SearchIterator<CellName, Cell> searchIterator()
{
return new BTreeSearchIterator<>(ref.tree, asymmetricComparator());
}
public void delete(DeletionInfo info)
{
if (info.isLive())
return;
// Keeping deletion info for max markedForDeleteAt value
while (true)
{
Holder current = ref;
DeletionInfo curDelInfo = current.deletionInfo;
DeletionInfo newDelInfo = info.mayModify(curDelInfo) ? curDelInfo.copy().add(info) : curDelInfo;
if (refUpdater.compareAndSet(this, current, current.with(newDelInfo)))
break;
}
}
public void setDeletionInfo(DeletionInfo newInfo)
{
ref = ref.with(newInfo);
}
public void purgeTombstones(int gcBefore)
{
while (true)
{
Holder current = ref;
if (!current.deletionInfo.hasPurgeableTombstones(gcBefore))
break;
DeletionInfo purgedInfo = current.deletionInfo.copy();
purgedInfo.purge(gcBefore);
if (refUpdater.compareAndSet(this, current, current.with(purgedInfo)))
break;
}
}
/**
* This is only called by Memtable.resolve, so only AtomicBTreeColumns needs to implement it.
*
* @return the difference in size seen after merging the given columns
*/
public Pair<Long, Long> addAllWithSizeDelta(final ColumnFamily cm, MemtableAllocator allocator, OpOrder.Group writeOp, Updater indexer)
{
ColumnUpdater updater = new ColumnUpdater(this, cm.metadata, allocator, writeOp, indexer);
DeletionInfo inputDeletionInfoCopy = null;
boolean monitorOwned = false;
try
{
if (usePessimisticLocking())
{
Locks.monitorEnterUnsafe(this);
monitorOwned = true;
}
while (true)
{
Holder current = ref;
updater.ref = current;
updater.reset();
DeletionInfo deletionInfo;
if (cm.deletionInfo().mayModify(current.deletionInfo))
{
if (inputDeletionInfoCopy == null)
inputDeletionInfoCopy = cm.deletionInfo().copy(HeapAllocator.instance);
deletionInfo = current.deletionInfo.copy().add(inputDeletionInfoCopy);
updater.allocated(deletionInfo.unsharedHeapSize() - current.deletionInfo.unsharedHeapSize());
}
else
{
deletionInfo = current.deletionInfo;
}
Object[] tree = BTree.update(current.tree, metadata.comparator.columnComparator(Memtable.MEMORY_POOL instanceof NativePool), cm, cm.getColumnCount(), true, updater);
if (tree != null && refUpdater.compareAndSet(this, current, new Holder(tree, deletionInfo)))
{
indexer.updateRowLevelIndexes();
updater.finish();
return Pair.create(updater.dataSize, updater.colUpdateTimeDelta);
}
else if (!monitorOwned)
{
boolean shouldLock = usePessimisticLocking();
if (!shouldLock)
{
shouldLock = updateWastedAllocationTracker(updater.heapSize);
}
if (shouldLock)
{
Locks.monitorEnterUnsafe(this);
monitorOwned = true;
}
}
}
}
finally
{
if (monitorOwned)
Locks.monitorExitUnsafe(this);
}
}
boolean usePessimisticLocking()
{
return wasteTracker == TRACKER_PESSIMISTIC_LOCKING;
}
/**
* Update the wasted allocation tracker state based on newly wasted allocation information
*
* @param wastedBytes the number of bytes wasted by this thread
* @return true if the caller should now proceed with pessimistic locking because the waste limit has been reached
*/
private boolean updateWastedAllocationTracker(long wastedBytes) {
// Early check for huge allocation that exceeds the limit
if (wastedBytes < EXCESS_WASTE_BYTES)
{
// We round up to ensure work < granularity are still accounted for
int wastedAllocation = ((int) (wastedBytes + ALLOCATION_GRANULARITY_BYTES - 1)) / ALLOCATION_GRANULARITY_BYTES;
int oldTrackerValue;
while (TRACKER_PESSIMISTIC_LOCKING != (oldTrackerValue = wasteTracker))
{
// Note this time value has an arbitrary offset, but is a constant rate 32 bit counter (that may wrap)
int time = (int) (System.nanoTime() >>> CLOCK_SHIFT);
int delta = oldTrackerValue - time;
if (oldTrackerValue == TRACKER_NEVER_WASTED || delta >= 0 || delta < -EXCESS_WASTE_OFFSET)
delta = -EXCESS_WASTE_OFFSET;
delta += wastedAllocation;
if (delta >= 0)
break;
if (wasteTrackerUpdater.compareAndSet(this, oldTrackerValue, avoidReservedValues(time + delta)))
return false;
}
}
// We have definitely reached our waste limit so set the state if it isn't already
wasteTrackerUpdater.set(this, TRACKER_PESSIMISTIC_LOCKING);
// And tell the caller to proceed with pessimistic locking
return true;
}
private static int avoidReservedValues(int wasteTracker)
{
if (wasteTracker == TRACKER_NEVER_WASTED || wasteTracker == TRACKER_PESSIMISTIC_LOCKING)
return wasteTracker + 1;
return wasteTracker;
}
// no particular reason not to implement these next methods, we just haven't needed them yet
public void addColumn(Cell column)
{
throw new UnsupportedOperationException();
}
public void maybeAppendColumn(Cell cell, DeletionInfo.InOrderTester tester, int gcBefore)
{
throw new UnsupportedOperationException();
}
public void appendColumn(Cell cell)
{
throw new UnsupportedOperationException();
}
public void addAll(ColumnFamily cf)
{
throw new UnsupportedOperationException();
}
public void clear()
{
throw new UnsupportedOperationException();
}
public Cell getColumn(CellName name)
{
return (Cell) BTree.find(ref.tree, asymmetricComparator(), name);
}
private Comparator<Object> asymmetricComparator()
{
return metadata.comparator.asymmetricColumnComparator(Memtable.MEMORY_POOL instanceof NativePool);
}
public Iterable<CellName> getColumnNames()
{
return collection(false, NAME);
}
public Collection<Cell> getSortedColumns()
{
return collection(true, Functions.<Cell>identity());
}
public Collection<Cell> getReverseSortedColumns()
{
return collection(false, Functions.<Cell>identity());
}
private <V> Collection<V> collection(final boolean forwards, final Function<Cell, V> f)
{
final Holder ref = this.ref;
return new AbstractCollection<V>()
{
public Iterator<V> iterator()
{
return Iterators.transform(BTree.<Cell>slice(ref.tree, forwards), f);
}
public int size()
{
return BTree.slice(ref.tree, true).count();
}
};
}
public int getColumnCount()
{
return BTree.slice(ref.tree, true).count();
}
public boolean hasColumns()
{
return !BTree.isEmpty(ref.tree);
}
public Iterator<Cell> iterator(ColumnSlice[] slices)
{
return slices.length == 1
? slice(ref.tree, asymmetricComparator(), slices[0].start, slices[0].finish, true)
: new SliceIterator(ref.tree, asymmetricComparator(), true, slices);
}
public Iterator<Cell> reverseIterator(ColumnSlice[] slices)
{
return slices.length == 1
? slice(ref.tree, asymmetricComparator(), slices[0].finish, slices[0].start, false)
: new SliceIterator(ref.tree, asymmetricComparator(), false, slices);
}
public boolean isInsertReversed()
{
return false;
}
public BatchRemoveIterator<Cell> batchRemoveIterator()
{
throw new UnsupportedOperationException();
}
private static final class Holder
{
final DeletionInfo deletionInfo;
// the btree of columns
final Object[] tree;
Holder(Object[] tree, DeletionInfo deletionInfo)
{
this.tree = tree;
this.deletionInfo = deletionInfo;
}
Holder with(DeletionInfo info)
{
return new Holder(this.tree, info);
}
}
// the function we provide to the btree utilities to perform any column replacements
private static final class ColumnUpdater implements UpdateFunction<Cell>
{
final AtomicBTreeColumns updating;
final CFMetaData metadata;
final MemtableAllocator allocator;
final OpOrder.Group writeOp;
final Updater indexer;
Holder ref;
long dataSize;
long heapSize;
long colUpdateTimeDelta = Long.MAX_VALUE;
final MemtableAllocator.DataReclaimer reclaimer;
List<Cell> inserted; // TODO: replace with walk of aborted BTree
private ColumnUpdater(AtomicBTreeColumns updating, CFMetaData metadata, MemtableAllocator allocator, OpOrder.Group writeOp, Updater indexer)
{
this.updating = updating;
this.allocator = allocator;
this.writeOp = writeOp;
this.indexer = indexer;
this.metadata = metadata;
this.reclaimer = allocator.reclaimer();
}
public Cell apply(Cell insert)
{
indexer.insert(insert);
insert = insert.localCopy(metadata, allocator, writeOp);
this.dataSize += insert.cellDataSize();
this.heapSize += insert.unsharedHeapSizeExcludingData();
if (inserted == null)
inserted = new ArrayList<>();
inserted.add(insert);
return insert;
}
public Cell apply(Cell existing, Cell update)
{
Cell reconciled = existing.reconcile(update);
indexer.update(existing, reconciled);
if (existing != reconciled)
{
reconciled = reconciled.localCopy(metadata, allocator, writeOp);
dataSize += reconciled.cellDataSize() - existing.cellDataSize();
heapSize += reconciled.unsharedHeapSizeExcludingData() - existing.unsharedHeapSizeExcludingData();
if (inserted == null)
inserted = new ArrayList<>();
inserted.add(reconciled);
discard(existing);
//Getting the minimum delta for an update containing multiple columns
colUpdateTimeDelta = Math.min(Math.abs(existing.timestamp() - update.timestamp()), colUpdateTimeDelta);
}
return reconciled;
}
protected void reset()
{
this.dataSize = 0;
this.heapSize = 0;
if (inserted != null)
{
for (Cell cell : inserted)
abort(cell);
inserted.clear();
}
reclaimer.cancel();
}
protected void abort(Cell abort)
{
reclaimer.reclaimImmediately(abort);
}
protected void discard(Cell discard)
{
reclaimer.reclaim(discard);
}
public boolean abortEarly()
{
return updating.ref != ref;
}
public void allocated(long heapSize)
{
this.heapSize += heapSize;
}
protected void finish()
{
allocator.onHeap().allocate(heapSize, writeOp);
reclaimer.commit();
}
}
private static class SliceIterator extends AbstractIterator<Cell>
{
private final Object[] btree;
private final boolean forwards;
private final Comparator<Object> comparator;
private final ColumnSlice[] slices;
private int idx = 0;
private Iterator<Cell> currentSlice;
SliceIterator(Object[] btree, Comparator<Object> comparator, boolean forwards, ColumnSlice[] slices)
{
this.btree = btree;
this.comparator = comparator;
this.slices = slices;
this.forwards = forwards;
}
protected Cell computeNext()
{
while (currentSlice != null || idx < slices.length)
{
if (currentSlice == null)
{
ColumnSlice slice = slices[idx++];
if (forwards)
currentSlice = slice(btree, comparator, slice.start, slice.finish, true);
else
currentSlice = slice(btree, comparator, slice.finish, slice.start, false);
}
if (currentSlice.hasNext())
return currentSlice.next();
currentSlice = null;
}
return endOfData();
}
}
private static Iterator<Cell> slice(Object[] btree, Comparator<Object> comparator, Composite start, Composite finish, boolean forwards)
{
return BTree.slice(btree,
comparator,
start.isEmpty() ? null : start,
true,
finish.isEmpty() ? null : finish,
true,
forwards);
}
}

21
src/java/org/apache/cassandra/db/BatchlogManager.java
View File

@ -38,6 +38,7 @@ import org.slf4j.LoggerFactory;
import org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.cql3.UntypedResultSet;
import org.apache.cassandra.db.partitions.PartitionUpdate;
import org.apache.cassandra.db.compaction.CompactionManager;
import org.apache.cassandra.db.marshal.UUIDType;
import org.apache.cassandra.dht.Token;
@ -138,12 +139,12 @@ public class BatchlogManager implements BatchlogManagerMBean
@VisibleForTesting
static Mutation getBatchlogMutationFor(Collection<Mutation> mutations, UUID uuid, int version, long now)
{
ColumnFamily cf = ArrayBackedSortedColumns.factory.create(SystemKeyspace.Batchlog);
CFRowAdder adder = new CFRowAdder(cf, SystemKeyspace.Batchlog.comparator.builder().build(), now);
adder.add("data", serializeMutations(mutations, version))
.add("written_at", new Date(now / 1000))
.add("version", version);
return new Mutation(SystemKeyspace.NAME, UUIDType.instance.decompose(uuid), cf);
return new RowUpdateBuilder(SystemKeyspace.Batchlog, now, uuid)
.clustering()
.add("data", serializeMutations(mutations, version))
.add("written_at", new Date(now / 1000))
.add("version", version)
.build();
}
private static ByteBuffer serializeMutations(Collection<Mutation> mutations, int version)
@ -186,7 +187,7 @@ public class BatchlogManager implements BatchlogManagerMBean
SystemKeyspace.NAME,
SystemKeyspace.BATCHLOG,
PAGE_SIZE),
id);
id);
}
cleanup();
@ -196,8 +197,8 @@ public class BatchlogManager implements BatchlogManagerMBean
private void deleteBatch(UUID id)
{
Mutation mutation = new Mutation(SystemKeyspace.NAME, UUIDType.instance.decompose(id));
mutation.delete(SystemKeyspace.BATCHLOG, FBUtilities.timestampMicros());
Mutation mutation = new Mutation(SystemKeyspace.NAME, StorageService.getPartitioner().decorateKey(UUIDType.instance.decompose(id)));
mutation.add(PartitionUpdate.fullPartitionDelete(SystemKeyspace.Batchlog, mutation.key(), FBUtilities.timestampMicros(), FBUtilities.nowInSeconds()));
mutation.apply();
}
@ -382,7 +383,7 @@ public class BatchlogManager implements BatchlogManagerMBean
{
Set<InetAddress> liveEndpoints = new HashSet<>();
String ks = mutation.getKeyspaceName();
Token tk = StorageService.getPartitioner().getToken(mutation.key());
Token tk = mutation.key().getToken();
for (InetAddress endpoint : Iterables.concat(StorageService.instance.getNaturalEndpoints(ks, tk),
StorageService.instance.getTokenMetadata().pendingEndpointsFor(tk, ks)))

103
src/java/org/apache/cassandra/db/BufferCell.java
View File

@ -1,103 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.db;
import java.nio.ByteBuffer;
import org.apache.cassandra.config.CFMetaData;
import org.apache.cassandra.db.composites.CellName;
import org.apache.cassandra.db.composites.CellNames;
import org.apache.cassandra.utils.ByteBufferUtil;
import org.apache.cassandra.utils.ObjectSizes;
import org.apache.cassandra.utils.concurrent.OpOrder;
import org.apache.cassandra.utils.memory.AbstractAllocator;
import org.apache.cassandra.utils.memory.MemtableAllocator;
public class BufferCell extends AbstractCell
{
private static final long EMPTY_SIZE = ObjectSizes.measure(new BufferCell(CellNames.simpleDense(ByteBuffer.allocate(1))));
protected final CellName name;
protected final ByteBuffer value;
protected final long timestamp;
BufferCell(CellName name)
{
this(name, ByteBufferUtil.EMPTY_BYTE_BUFFER);
}
public BufferCell(CellName name, ByteBuffer value)
{
this(name, value, 0);
}
public BufferCell(CellName name, ByteBuffer value, long timestamp)
{
assert name != null;
assert value != null;
this.name = name;
this.value = value;
this.timestamp = timestamp;
}
@Override
public Cell withUpdatedName(CellName newName)
{
return new BufferCell(newName, value, timestamp);
}
@Override
public Cell withUpdatedTimestamp(long newTimestamp)
{
return new BufferCell(name, value, newTimestamp);
}
@Override
public CellName name() {
return name;
}
@Override
public ByteBuffer value() {
return value;
}
@Override
public long timestamp() {
return timestamp;
}
@Override
public long unsharedHeapSizeExcludingData()
{
return EMPTY_SIZE + name.unsharedHeapSizeExcludingData() + ObjectSizes.sizeOnHeapExcludingData(value);
}
@Override
public Cell localCopy(CFMetaData metadata, AbstractAllocator allocator)
{
return new BufferCell(name.copy(metadata, allocator), allocator.clone(value), timestamp);
}
@Override
public Cell localCopy(CFMetaData metadata, MemtableAllocator allocator, OpOrder.Group opGroup)
{
return allocator.clone(this, metadata, opGroup);
}
}

176
src/java/org/apache/cassandra/db/BufferCounterCell.java
View File

@ -1,176 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.db;
import java.nio.ByteBuffer;
import java.security.MessageDigest;
import org.apache.cassandra.config.CFMetaData;
import org.apache.cassandra.db.composites.CellName;
import org.apache.cassandra.db.composites.CellNameType;
import org.apache.cassandra.serializers.MarshalException;
import org.apache.cassandra.utils.FBUtilities;
import org.apache.cassandra.utils.concurrent.OpOrder;
import org.apache.cassandra.utils.memory.AbstractAllocator;
import org.apache.cassandra.utils.memory.MemtableAllocator;
public class BufferCounterCell extends BufferCell implements CounterCell
{
private final long timestampOfLastDelete;
public BufferCounterCell(CellName name, ByteBuffer value, long timestamp)
{
this(name, value, timestamp, Long.MIN_VALUE);
}
public BufferCounterCell(CellName name, ByteBuffer value, long timestamp, long timestampOfLastDelete)
{
super(name, value, timestamp);
this.timestampOfLastDelete = timestampOfLastDelete;
}
public static CounterCell create(CellName name, ByteBuffer value, long timestamp, long timestampOfLastDelete, ColumnSerializer.Flag flag)
{
if (flag == ColumnSerializer.Flag.FROM_REMOTE || (flag == ColumnSerializer.Flag.LOCAL && contextManager.shouldClearLocal(value)))
value = contextManager.clearAllLocal(value);
return new BufferCounterCell(name, value, timestamp, timestampOfLastDelete);
}
// For use by tests of compatibility with pre-2.1 counter only.
public static CounterCell createLocal(CellName name, long value, long timestamp, long timestampOfLastDelete)
{
return new BufferCounterCell(name, contextManager.createLocal(value), timestamp, timestampOfLastDelete);
}
@Override
public Cell withUpdatedName(CellName newName)
{
return new BufferCounterCell(newName, value, timestamp, timestampOfLastDelete);
}
@Override
public long timestampOfLastDelete()
{
return timestampOfLastDelete;
}
@Override
public long total()
{
return contextManager.total(value);
}
@Override
public int cellDataSize()
{
// A counter column adds 8 bytes for timestampOfLastDelete to Cell.
return super.cellDataSize() + TypeSizes.NATIVE.sizeof(timestampOfLastDelete);
}
@Override
public int serializedSize(CellNameType type, TypeSizes typeSizes)
{
return super.serializedSize(type, typeSizes) + typeSizes.sizeof(timestampOfLastDelete);
}
@Override
public Cell diff(Cell cell)
{
return diffCounter(cell);
}
/*
* We have to special case digest creation for counter column because
* we don't want to include the information about which shard of the
* context is a delta or not, since this information differs from node to
* node.
*/
@Override
public void updateDigest(MessageDigest digest)
{
digest.update(name().toByteBuffer().duplicate());
// We don't take the deltas into account in a digest
contextManager.updateDigest(digest, value());
FBUtilities.updateWithLong(digest, timestamp);
FBUtilities.updateWithByte(digest, serializationFlags());
FBUtilities.updateWithLong(digest, timestampOfLastDelete);
}
@Override
public Cell reconcile(Cell cell)
{
return reconcileCounter(cell);
}
@Override
public boolean hasLegacyShards()
{
return contextManager.hasLegacyShards(value);
}
@Override
public CounterCell localCopy(CFMetaData metadata, AbstractAllocator allocator)
{
return new BufferCounterCell(name.copy(metadata, allocator), allocator.clone(value), timestamp, timestampOfLastDelete);
}
@Override
public CounterCell localCopy(CFMetaData metadata, MemtableAllocator allocator, OpOrder.Group opGroup)
{
return allocator.clone(this, metadata, opGroup);
}
@Override
public String getString(CellNameType comparator)
{
return String.format("%s:false:%s@%d!%d",
comparator.getString(name()),
contextManager.toString(value()),
timestamp(),
timestampOfLastDelete);
}
@Override
public int serializationFlags()
{
return ColumnSerializer.COUNTER_MASK;
}
@Override
public void validateFields(CFMetaData metadata) throws MarshalException
{
validateName(metadata);
// We cannot use the value validator as for other columns as the CounterColumnType validate a long,
// which is not the internal representation of counters
contextManager.validateContext(value());
}
@Override
public Cell markLocalToBeCleared()
{
ByteBuffer marked = contextManager.markLocalToBeCleared(value());
return marked == value() ? this : new BufferCounterCell(name(), marked, timestamp(), timestampOfLastDelete);
}
@Override
public boolean equals(Cell cell)
{
return super.equals(cell) && timestampOfLastDelete == ((CounterCell) cell).timestampOfLastDelete();
}
}

96
src/java/org/apache/cassandra/db/BufferCounterUpdateCell.java
View File

@ -1,96 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.db;
import java.nio.ByteBuffer;
import org.apache.cassandra.config.CFMetaData;
import org.apache.cassandra.db.composites.CellName;
import org.apache.cassandra.db.composites.CellNameType;
import org.apache.cassandra.utils.ByteBufferUtil;
import org.apache.cassandra.utils.concurrent.OpOrder;
import org.apache.cassandra.utils.memory.AbstractAllocator;
import org.apache.cassandra.utils.memory.MemtableAllocator;
public class BufferCounterUpdateCell extends BufferCell implements CounterUpdateCell
{
public BufferCounterUpdateCell(CellName name, long value, long timestamp)
{
this(name, ByteBufferUtil.bytes(value), timestamp);
}
public BufferCounterUpdateCell(CellName name, ByteBuffer value, long timestamp)
{
super(name, value, timestamp);
}
@Override
public Cell withUpdatedName(CellName newName)
{
return new BufferCounterUpdateCell(newName, value, timestamp);
}
public long delta()
{
return value().getLong(value.position());
}
@Override
public Cell diff(Cell cell)
{
// Diff is used during reads, but we should never read those columns
throw new UnsupportedOperationException("This operation is unsupported on CounterUpdateCell.");
}
@Override
public Cell reconcile(Cell cell)
{
// No matter what the counter cell's timestamp is, a tombstone always takes precedence. See CASSANDRA-7346.
if (cell instanceof DeletedCell)
return cell;
assert cell instanceof CounterUpdateCell : "Wrong class type.";
// The only time this could happen is if a batch ships two increments for the same cell. Hence we simply sum the deltas.
return new BufferCounterUpdateCell(name, delta() + ((CounterUpdateCell) cell).delta(), Math.max(timestamp, cell.timestamp()));
}
@Override
public int serializationFlags()
{
return ColumnSerializer.COUNTER_UPDATE_MASK;
}
@Override
public Cell localCopy(CFMetaData metadata, AbstractAllocator allocator)
{
throw new UnsupportedOperationException();
}
@Override
public Cell localCopy(CFMetaData metadata, MemtableAllocator allocator, OpOrder.Group opGroup)
{
throw new UnsupportedOperationException();
}
@Override
public String getString(CellNameType comparator)
{
return String.format("%s:%s@%d", comparator.getString(name()), ByteBufferUtil.toLong(value), timestamp());
}
}

118
src/java/org/apache/cassandra/db/BufferDeletedCell.java
View File

@ -1,118 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.db;
import java.nio.ByteBuffer;
import java.security.MessageDigest;
import org.apache.cassandra.config.CFMetaData;
import org.apache.cassandra.db.composites.CellName;
import org.apache.cassandra.serializers.MarshalException;
import org.apache.cassandra.utils.FBUtilities;
import org.apache.cassandra.utils.concurrent.OpOrder;
import org.apache.cassandra.utils.memory.AbstractAllocator;
import org.apache.cassandra.utils.ByteBufferUtil;
import org.apache.cassandra.utils.memory.MemtableAllocator;
public class BufferDeletedCell extends BufferCell implements DeletedCell
{
public BufferDeletedCell(CellName name, int localDeletionTime, long timestamp)
{
this(name, ByteBufferUtil.bytes(localDeletionTime), timestamp);
}
public BufferDeletedCell(CellName name, ByteBuffer value, long timestamp)
{
super(name, value, timestamp);
}
@Override
public Cell withUpdatedName(CellName newName)
{
return new BufferDeletedCell(newName, value, timestamp);
}
@Override
public Cell withUpdatedTimestamp(long newTimestamp)
{
return new BufferDeletedCell(name, value, newTimestamp);
}
@Override
public boolean isLive()
{
return false;
}
@Override
public boolean isLive(long now)
{
return false;
}
@Override
public int getLocalDeletionTime()
{
return value().getInt(value.position());
}
@Override
public Cell reconcile(Cell cell)
{
if (cell instanceof DeletedCell)
return super.reconcile(cell);
return cell.reconcile(this);
}
@Override
public DeletedCell localCopy(CFMetaData metadata, AbstractAllocator allocator)
{
return new BufferDeletedCell(name.copy(metadata, allocator), allocator.clone(value), timestamp);
}
@Override
public DeletedCell localCopy(CFMetaData metadata, MemtableAllocator allocator, OpOrder.Group opGroup)
{
return allocator.clone(this, metadata, opGroup);
}
@Override
public int serializationFlags()
{
return ColumnSerializer.DELETION_MASK;
}
@Override
public void validateFields(CFMetaData metadata) throws MarshalException
{
validateName(metadata);
if (value().remaining() != 4)
throw new MarshalException("A tombstone value should be 4 bytes long");
if (getLocalDeletionTime() < 0)
throw new MarshalException("The local deletion time should not be negative");
}
@Override
public void updateDigest(MessageDigest digest)
{
digest.update(name().toByteBuffer().duplicate());
FBUtilities.updateWithLong(digest, timestamp());
FBUtilities.updateWithByte(digest, serializationFlags());
}
}

187
src/java/org/apache/cassandra/db/BufferExpiringCell.java
View File

@ -1,187 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.db;
import java.nio.ByteBuffer;
import java.security.MessageDigest;
import org.apache.cassandra.config.CFMetaData;
import org.apache.cassandra.db.composites.CellName;
import org.apache.cassandra.db.composites.CellNameType;
import org.apache.cassandra.serializers.MarshalException;
import org.apache.cassandra.utils.FBUtilities;
import org.apache.cassandra.utils.concurrent.OpOrder;
import org.apache.cassandra.utils.memory.AbstractAllocator;
import org.apache.cassandra.utils.memory.MemtableAllocator;
public class BufferExpiringCell extends BufferCell implements ExpiringCell
{
private final int localExpirationTime;
private final int timeToLive;
public BufferExpiringCell(CellName name, ByteBuffer value, long timestamp, int timeToLive)
{
this(name, value, timestamp, timeToLive, (int) (System.currentTimeMillis() / 1000) + timeToLive);
}
public BufferExpiringCell(CellName name, ByteBuffer value, long timestamp, int timeToLive, int localExpirationTime)
{
super(name, value, timestamp);
assert timeToLive > 0 : timeToLive;
assert localExpirationTime > 0 : localExpirationTime;
this.timeToLive = timeToLive;
this.localExpirationTime = localExpirationTime;
}
public int getTimeToLive()
{
return timeToLive;
}
@Override
public Cell withUpdatedName(CellName newName)
{
return new BufferExpiringCell(newName, value(), timestamp(), timeToLive, localExpirationTime);
}
@Override
public Cell withUpdatedTimestamp(long newTimestamp)
{
return new BufferExpiringCell(name(), value(), newTimestamp, timeToLive, localExpirationTime);
}
@Override
public int cellDataSize()
{
return super.cellDataSize() + TypeSizes.NATIVE.sizeof(localExpirationTime) + TypeSizes.NATIVE.sizeof(timeToLive);
}
@Override
public int serializedSize(CellNameType type, TypeSizes typeSizes)
{
/*
* An expired column adds to a Cell :
* 4 bytes for the localExpirationTime
* + 4 bytes for the timeToLive
*/
return super.serializedSize(type, typeSizes) + typeSizes.sizeof(localExpirationTime) + typeSizes.sizeof(timeToLive);
}
@Override
public void updateDigest(MessageDigest digest)
{
super.updateDigest(digest);
FBUtilities.updateWithInt(digest, timeToLive);
}
@Override
public int getLocalDeletionTime()
{
return localExpirationTime;
}
@Override
public ExpiringCell localCopy(CFMetaData metadata, AbstractAllocator allocator)
{
return new BufferExpiringCell(name.copy(metadata, allocator), allocator.clone(value), timestamp, timeToLive, localExpirationTime);
}
@Override
public ExpiringCell localCopy(CFMetaData metadata, MemtableAllocator allocator, OpOrder.Group opGroup)
{
return allocator.clone(this, metadata, opGroup);
}
@Override
public String getString(CellNameType comparator)
{
return String.format("%s!%d", super.getString(comparator), timeToLive);
}
@Override
public boolean isLive()
{
return isLive(System.currentTimeMillis());
}
@Override
public boolean isLive(long now)
{
return (int) (now / 1000) < getLocalDeletionTime();
}
@Override
public int serializationFlags()
{
return ColumnSerializer.EXPIRATION_MASK;
}
@Override
public void validateFields(CFMetaData metadata) throws MarshalException
{
super.validateFields(metadata);
if (timeToLive <= 0)
throw new MarshalException("A column TTL should be > 0, but was " + timeToLive);
if (localExpirationTime < 0)
throw new MarshalException("The local expiration time should not be negative but was " + localExpirationTime);
}
@Override
public Cell reconcile(Cell cell)
{
long ts1 = timestamp(), ts2 = cell.timestamp();
if (ts1 != ts2)
return ts1 < ts2 ? cell : this;
// we should prefer tombstones
if (cell instanceof DeletedCell)
return cell;
int c = value().compareTo(cell.value());
if (c != 0)
return c < 0 ? cell : this;
// If we have same timestamp and value, prefer the longest ttl
if (cell instanceof ExpiringCell)
{
int let1 = localExpirationTime, let2 = cell.getLocalDeletionTime();
if (let1 < let2)
return cell;
}
return this;
}
@Override
public boolean equals(Cell cell)
{
if (!super.equals(cell))
return false;
ExpiringCell that = (ExpiringCell) cell;
return getLocalDeletionTime() == that.getLocalDeletionTime() && getTimeToLive() == that.getTimeToLive();
}
/** @return Either a DeletedCell, or an ExpiringCell. */
public static Cell create(CellName name, ByteBuffer value, long timestamp, int timeToLive, int localExpirationTime, int expireBefore, ColumnSerializer.Flag flag)
{
if (localExpirationTime >= expireBefore || flag == ColumnSerializer.Flag.PRESERVE_SIZE)
return new BufferExpiringCell(name, value, timestamp, timeToLive, localExpirationTime);
// The column is now expired, we can safely return a simple tombstone. Note that
// as long as the expiring column and the tombstone put together live longer than GC grace seconds,
// we'll fulfil our responsibility to repair. See discussion at
// http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/repair-compaction-and-tombstone-rows-td7583481.html
return new BufferDeletedCell(name, localExpirationTime - timeToLive, timestamp);
}
}

231
src/java/org/apache/cassandra/db/CBuilder.java Normal file
View File

@ -0,0 +1,231 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.db;
import java.nio.ByteBuffer;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import org.apache.cassandra.db.marshal.AbstractType;
/**
* Allows to build ClusteringPrefixes, either Clustering or Slice.Bound.
*/
public abstract class CBuilder
{
public static CBuilder STATIC_BUILDER = new CBuilder()
{
public int count()
{
return 0;
}
public int remainingCount()
{
return 0;
}
public ClusteringComparator comparator()
{
throw new UnsupportedOperationException();
}
public CBuilder add(ByteBuffer value)
{
throw new UnsupportedOperationException();
}
public CBuilder add(Object value)
{
throw new UnsupportedOperationException();
}
public Clustering build()
{
return Clustering.STATIC_CLUSTERING;
}
public Slice.Bound buildBound(boolean isStart, boolean isInclusive)
{
throw new UnsupportedOperationException();
}
public Slice buildSlice()
{
throw new UnsupportedOperationException();
}
public Clustering buildWith(ByteBuffer value)
{
throw new UnsupportedOperationException();
}
public Clustering buildWith(List<ByteBuffer> newValues)
{
throw new UnsupportedOperationException();
}
public Slice.Bound buildBoundWith(ByteBuffer value, boolean isStart, boolean isInclusive)
{
throw new UnsupportedOperationException();
}
public Slice.Bound buildBoundWith(List<ByteBuffer> newValues, boolean isStart, boolean isInclusive)
{
throw new UnsupportedOperationException();
}
};
public static CBuilder create(ClusteringComparator comparator)
{
return new ArrayBackedBuilder(comparator);
}
public abstract int count();
public abstract int remainingCount();
public abstract ClusteringComparator comparator();
public abstract CBuilder add(ByteBuffer value);
public abstract CBuilder add(Object value);
public abstract Clustering build();
public abstract Slice.Bound buildBound(boolean isStart, boolean isInclusive);
public abstract Slice buildSlice();
public abstract Clustering buildWith(ByteBuffer value);
public abstract Clustering buildWith(List<ByteBuffer> newValues);
public abstract Slice.Bound buildBoundWith(ByteBuffer value, boolean isStart, boolean isInclusive);
public abstract Slice.Bound buildBoundWith(List<ByteBuffer> newValues, boolean isStart, boolean isInclusive);
private static class ArrayBackedBuilder extends CBuilder
{
private final ClusteringComparator type;
private final ByteBuffer[] values;
private int size;
private boolean built;
public ArrayBackedBuilder(ClusteringComparator type)
{
this.type = type;
this.values = new ByteBuffer[type.size()];
}
public int count()
{
return size;
}
public int remainingCount()
{
return values.length - size;
}
public ClusteringComparator comparator()
{
return type;
}
public CBuilder add(ByteBuffer value)
{
if (isDone())
throw new IllegalStateException();
values[size++] = value;
return this;
}
public CBuilder add(Object value)
{
return add(((AbstractType)type.subtype(size)).decompose(value));
}
private boolean isDone()
{
return remainingCount() == 0 || built;
}
public Clustering build()
{
// We don't allow to add more element to a builder that has been built so
// that we don't have to copy values.
built = true;
// Currently, only dense table can leave some clustering column out (see #7990)
return size == 0 ? Clustering.EMPTY : new SimpleClustering(values);
}
public Slice.Bound buildBound(boolean isStart, boolean isInclusive)
{
// We don't allow to add more element to a builder that has been built so
// that we don't have to copy values (even though we have to do it in most cases).
built = true;
if (size == 0)
return isStart ? Slice.Bound.BOTTOM : Slice.Bound.TOP;
return Slice.Bound.create(Slice.Bound.boundKind(isStart, isInclusive),
size == values.length ? values : Arrays.copyOfRange(values, 0, size));
}
public Slice buildSlice()
{
// We don't allow to add more element to a builder that has been built so
// that we don't have to copy values.
built = true;
if (size == 0)
return Slice.ALL;
return Slice.make(buildBound(true, true), buildBound(false, true));
}
public Clustering buildWith(ByteBuffer value)
{
assert size+1 == type.size();
ByteBuffer[] newValues = Arrays.copyOf(values, size+1);
newValues[size] = value;
return new SimpleClustering(newValues);
}
public Clustering buildWith(List<ByteBuffer> newValues)
{
assert size + newValues.size() == type.size();
ByteBuffer[] buffers = Arrays.copyOf(values, size + newValues.size());
int newSize = size;
for (ByteBuffer value : newValues)
buffers[newSize++] = value;
return new SimpleClustering(buffers);
}
public Slice.Bound buildBoundWith(ByteBuffer value, boolean isStart, boolean isInclusive)
{
ByteBuffer[] newValues = Arrays.copyOf(values, size+1);
newValues[size] = value;
return Slice.Bound.create(Slice.Bound.boundKind(isStart, isInclusive), newValues);
}
public Slice.Bound buildBoundWith(List<ByteBuffer> newValues, boolean isStart, boolean isInclusive)
{
ByteBuffer[] buffers = Arrays.copyOf(values, size + newValues.size());
int newSize = size;
for (ByteBuffer value : newValues)
buffers[newSize++] = value;
return Slice.Bound.create(Slice.Bound.boundKind(isStart, isInclusive), buffers);
}
}
}

121
src/java/org/apache/cassandra/db/CFRowAdder.java
View File

@ -1,121 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.db;
import java.nio.ByteBuffer;
import org.apache.cassandra.cql3.ColumnIdentifier;
import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.db.composites.CellName;
import org.apache.cassandra.db.composites.Composite;
import org.apache.cassandra.db.marshal.AbstractType;
import org.apache.cassandra.db.marshal.CollectionType;
import org.apache.cassandra.db.marshal.ListType;
import org.apache.cassandra.db.marshal.MapType;
import org.apache.cassandra.utils.ByteBufferUtil;
import org.apache.cassandra.utils.UUIDGen;
/**
* Convenience object to populate a given CQL3 row in a ColumnFamily object.
*
* This is meant for when performance is not of the utmost importance. When
* performance matters, it might be worth allocating such builder.
*/
public class CFRowAdder
{
public final ColumnFamily cf;
public final Composite prefix;
public final long timestamp;
public final int ttl;
private final int ldt;
public CFRowAdder(ColumnFamily cf, Composite prefix, long timestamp)
{
this(cf, prefix, timestamp, 0);
}
public CFRowAdder(ColumnFamily cf, Composite prefix, long timestamp, int ttl)
{
this.cf = cf;
this.prefix = prefix;
this.timestamp = timestamp;
this.ttl = ttl;
this.ldt = (int) (System.currentTimeMillis() / 1000);
// If a CQL3 table, add the row marker
if (cf.metadata().isCQL3Table() && !prefix.isStatic())
cf.addColumn(new BufferCell(cf.getComparator().rowMarker(prefix), ByteBufferUtil.EMPTY_BYTE_BUFFER, timestamp));
}
public CFRowAdder add(String cql3ColumnName, Object value)
{
ColumnDefinition def = getDefinition(cql3ColumnName);
return add(cf.getComparator().create(prefix, def), def, value);
}
public CFRowAdder resetCollection(String cql3ColumnName)
{
ColumnDefinition def = getDefinition(cql3ColumnName);
assert def.type.isCollection() && def.type.isMultiCell();
Composite name = cf.getComparator().create(prefix, def);
cf.addAtom(new RangeTombstone(name.start(), name.end(), timestamp - 1, ldt));
return this;
}
public CFRowAdder addMapEntry(String cql3ColumnName, Object key, Object value)
{
ColumnDefinition def = getDefinition(cql3ColumnName);
assert def.type instanceof MapType;
MapType mt = (MapType)def.type;
CellName name = cf.getComparator().create(prefix, def, mt.getKeysType().decompose(key));
return add(name, def, value);
}
public CFRowAdder addListEntry(String cql3ColumnName, Object value)
{
ColumnDefinition def = getDefinition(cql3ColumnName);
assert def.type instanceof ListType;
CellName name = cf.getComparator().create(prefix, def, ByteBuffer.wrap(UUIDGen.getTimeUUIDBytes()));
return add(name, def, value);
}
private ColumnDefinition getDefinition(String name)
{
return cf.metadata().getColumnDefinition(new ColumnIdentifier(name, false));
}
private CFRowAdder add(CellName name, ColumnDefinition def, Object value)
{
if (value == null)
{
cf.addColumn(new BufferDeletedCell(name, ldt, timestamp));
}
else
{
AbstractType valueType = def.type.isCollection()
? ((CollectionType) def.type).valueComparator()
: def.type;
ByteBuffer valueBytes = value instanceof ByteBuffer ? (ByteBuffer)value : valueType.decompose(value);
if (ttl == 0)
cf.addColumn(new BufferCell(name, valueBytes, timestamp));
else
cf.addColumn(new BufferExpiringCell(name, valueBytes, timestamp, ttl));
}
return this;
}
}

69
src/java/org/apache/cassandra/db/Cell.java
View File

@ -1,69 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.db;
import java.nio.ByteBuffer;
import org.apache.cassandra.config.CFMetaData;
import org.apache.cassandra.db.composites.CellName;
import org.apache.cassandra.db.composites.CellNameType;
import org.apache.cassandra.utils.concurrent.OpOrder;
import org.apache.cassandra.utils.memory.AbstractAllocator;
import org.apache.cassandra.utils.FBUtilities;
import org.apache.cassandra.utils.memory.MemtableAllocator;
/**
* Cell is immutable, which prevents all kinds of confusion in a multithreaded environment.
*/
public interface Cell extends OnDiskAtom
{
public static final int MAX_NAME_LENGTH = FBUtilities.MAX_UNSIGNED_SHORT;
public Cell withUpdatedName(CellName newName);
public Cell withUpdatedTimestamp(long newTimestamp);
@Override
public CellName name();
public ByteBuffer value();
public boolean isLive();
public boolean isLive(long now);
public int cellDataSize();
// returns the size of the Cell and all references on the heap, excluding any costs associated with byte arrays
// that would be allocated by a localCopy, as these will be accounted for by the allocator
public long unsharedHeapSizeExcludingData();
public int serializedSize(CellNameType type, TypeSizes typeSizes);
public int serializationFlags();
public Cell diff(Cell cell);
public Cell reconcile(Cell cell);
public Cell localCopy(CFMetaData metadata, AbstractAllocator allocator);
public Cell localCopy(CFMetaData metaData, MemtableAllocator allocator, OpOrder.Group opGroup);
public String getString(CellNameType comparator);
}

11
src/java/org/apache/cassandra/service/IReadCommand.java → src/java/org/apache/cassandra/db/Clusterable.java
View File

@ -15,10 +15,13 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.service;
package org.apache.cassandra.db;
public interface IReadCommand
/**
* Common class for objects that are identified by a clustering prefix, and can be thus sorted by a
* {@link ClusteringComparator}.
*/
public interface Clusterable
{
public String getKeyspace();
public long getTimeout();
public ClusteringPrefix clustering();
}

171
src/java/org/apache/cassandra/db/Clustering.java Normal file
View File

@ -0,0 +1,171 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.db;
import java.io.DataInput;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.util.*;
import org.apache.cassandra.config.CFMetaData;
import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.db.marshal.AbstractType;
import org.apache.cassandra.io.util.DataOutputPlus;
/**
* The clustering column values for a row.
* <p>
* A {@code Clustering} is a {@code ClusteringPrefix} that must always be "complete", i.e. have
* as many values as there is clustering columns in the table it is part of. It is the clustering
* prefix used by rows.
* <p>
* Note however that while it's size must be equal to the table clustering size, a clustering can have
* {@code null} values, and this mostly for thrift backward compatibility (in practice, if a value is null,
* all of the following ones will be too because that's what thrift allows, but it's never assumed by the
* code so we could start generally allowing nulls for clustering columns if we wanted to).
*/
public abstract class Clustering extends AbstractClusteringPrefix
{
public static final Serializer serializer = new Serializer();
/**
* The special cased clustering used by all static rows. It is a special case in the
* sense that it's always empty, no matter how many clustering columns the table has.
*/
public static final Clustering STATIC_CLUSTERING = new EmptyClustering()
{
@Override
public Kind kind()
{
return Kind.STATIC_CLUSTERING;
}
@Override
public String toString(CFMetaData metadata)
{
return "STATIC";
}
};
/** Empty clustering for tables having no clustering columns. */
public static final Clustering EMPTY = new EmptyClustering();
public Kind kind()
{
return Kind.CLUSTERING;
}
public Clustering takeAlias()
{
ByteBuffer[] values = new ByteBuffer[size()];
for (int i = 0; i < size(); i++)
values[i] = get(i);
return new SimpleClustering(values);
}
public String toString(CFMetaData metadata)
{
StringBuilder sb = new StringBuilder();
for (int i = 0; i < size(); i++)
{
ColumnDefinition c = metadata.clusteringColumns().get(i);
sb.append(i == 0 ? "" : ", ").append(c.name).append("=").append(get(i) == null ? "null" : c.type.getString(get(i)));
}
return sb.toString();
}
public String toCQLString(CFMetaData metadata)
{
StringBuilder sb = new StringBuilder();
for (int i = 0; i < size(); i++)
{
ColumnDefinition c = metadata.clusteringColumns().get(i);
sb.append(i == 0 ? "" : ", ").append(c.type.getString(get(i)));
}
return sb.toString();
}
private static class EmptyClustering extends Clustering
{
private static final ByteBuffer[] EMPTY_VALUES_ARRAY = new ByteBuffer[0];
public int size()
{
return 0;
}
public ByteBuffer get(int i)
{
throw new UnsupportedOperationException();
}
public ByteBuffer[] getRawValues()
{
return EMPTY_VALUES_ARRAY;
}
@Override
public Clustering takeAlias()
{
return this;
}
@Override
public long unsharedHeapSize()
{
return 0;
}
@Override
public String toString(CFMetaData metadata)
{
return "EMPTY";
}
}
/**
* Serializer for Clustering object.
* <p>
* Because every clustering in a given table must have the same size (ant that size cannot actually change once the table
* has been defined), we don't record that size.
*/
public static class Serializer
{
public void serialize(Clustering clustering, DataOutputPlus out, int version, List<AbstractType<?>> types) throws IOException
{
ClusteringPrefix.serializer.serializeValuesWithoutSize(clustering, out, version, types);
}
public long serializedSize(Clustering clustering, int version, List<AbstractType<?>> types, TypeSizes sizes)
{
return ClusteringPrefix.serializer.valuesWithoutSizeSerializedSize(clustering, version, types, sizes);
}
public void deserialize(DataInput in, int version, List<AbstractType<?>> types, Writer writer) throws IOException
{
ClusteringPrefix.serializer.deserializeValuesWithoutSize(in, types.size(), version, types, writer);
}
public Clustering deserialize(DataInput in, int version, List<AbstractType<?>> types) throws IOException
{
SimpleClustering.Builder builder = SimpleClustering.builder(types.size());
deserialize(in, version, types, builder);
return builder.build();
}
}
}

291
src/java/org/apache/cassandra/db/ClusteringComparator.java Normal file
View File

@ -0,0 +1,291 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.db;
import java.nio.ByteBuffer;
import java.util.Arrays;
import java.util.Comparator;
import java.util.List;
import java.util.Objects;
import com.google.common.base.Joiner;
import org.apache.cassandra.db.marshal.AbstractType;
import org.apache.cassandra.utils.ByteBufferUtil;
import static org.apache.cassandra.io.sstable.IndexHelper.IndexInfo;
/**
* A comparator of clustering prefixes (or more generally of {@link Clusterable}}.
* <p>
* This is essentially just a composite comparator that the clustering values of the provided
* clustering prefixes in lexicographical order, with each component being compared based on
* the type of the clustering column this is a value of.
*/
public class ClusteringComparator implements Comparator<Clusterable>
{
private final List<AbstractType<?>> clusteringTypes;
private final boolean isByteOrderComparable;
private final Comparator<IndexInfo> indexComparator;
private final Comparator<IndexInfo> indexReverseComparator;
private final Comparator<Clusterable> reverseComparator;
public ClusteringComparator(AbstractType<?>... clusteringTypes)
{
this(Arrays.<AbstractType<?>>asList(clusteringTypes));
}
public ClusteringComparator(List<AbstractType<?>> clusteringTypes)
{
this.clusteringTypes = clusteringTypes;
this.isByteOrderComparable = isByteOrderComparable(clusteringTypes);
this.indexComparator = new Comparator<IndexInfo>()
{
public int compare(IndexInfo o1, IndexInfo o2)
{
return ClusteringComparator.this.compare(o1.lastName, o2.lastName);
}
};
this.indexReverseComparator = new Comparator<IndexInfo>()
{
public int compare(IndexInfo o1, IndexInfo o2)
{
return ClusteringComparator.this.compare(o1.firstName, o2.firstName);
}
};
this.reverseComparator = new Comparator<Clusterable>()
{
public int compare(Clusterable c1, Clusterable c2)
{
return ClusteringComparator.this.compare(c2, c1);
}
};
}
private static boolean isByteOrderComparable(Iterable<AbstractType<?>> types)
{
boolean isByteOrderComparable = true;
for (AbstractType<?> type : types)
isByteOrderComparable &= type.isByteOrderComparable();
return isByteOrderComparable;
}
/**
* The number of clustering columns for the table this is the comparator of.
*/
public int size()
{
return clusteringTypes.size();
}
/**
* The "subtypes" of this clustering comparator, that is the types of the clustering
* columns for the table this is a comparator of.
*/
public List<AbstractType<?>> subtypes()
{
return clusteringTypes;
}
/**
* Returns the type of the ith clustering column of the table.
*/
public AbstractType<?> subtype(int i)
{
return clusteringTypes.get(i);
}
/**
* Creates a row clustering based on the clustering values.
* <p>
* Every argument can either be a {@code ByteBuffer}, in which case it is used as-is, or a object
* corresponding to the type of the corresponding clustering column, in which case it will be
* converted to a byte buffer using the column type.
*
* @param values the values to use for the created clustering. There should be exactly {@code size()}
* values which must be either byte buffers or of the type the column expect.
*
* @return the newly created clustering.
*/
public Clustering make(Object... values)
{
if (values.length != size())
throw new IllegalArgumentException(String.format("Invalid number of components, expecting %d but got %d", size(), values.length));
CBuilder builder = CBuilder.create(this);
for (int i = 0; i < values.length; i++)
{
Object val = values[i];
if (val instanceof ByteBuffer)
builder.add((ByteBuffer)val);
else
builder.add(val);
}
return builder.build();
}
public int compare(Clusterable c1, Clusterable c2)
{
return compare(c1.clustering(), c2.clustering());
}
public int compare(ClusteringPrefix c1, ClusteringPrefix c2)
{
int s1 = c1.size();
int s2 = c2.size();
int minSize = Math.min(s1, s2);
for (int i = 0; i < minSize; i++)
{
int cmp = compareComponent(i, c1.get(i), c2.get(i));
if (cmp != 0)
return cmp;
}
if (s1 == s2)
return ClusteringPrefix.Kind.compare(c1.kind(), c2.kind());
return s1 < s2 ? c1.kind().prefixComparisonResult : -c2.kind().prefixComparisonResult;
}
public int compare(Clustering c1, Clustering c2)
{
for (int i = 0; i < size(); i++)
{
int cmp = compareComponent(i, c1.get(i), c2.get(i));
if (cmp != 0)
return cmp;
}
return 0;
}
public int compareComponent(int i, ByteBuffer v1, ByteBuffer v2)
{
if (v1 == null)
return v1 == null ? 0 : -1;
if (v2 == null)
return 1;
return isByteOrderComparable
? ByteBufferUtil.compareUnsigned(v1, v2)
: clusteringTypes.get(i).compare(v1, v2);
}
/**
* Returns whether this clustering comparator is compatible with the provided one,
* that is if the provided one can be safely replaced by this new one.
*
* @param previous the previous comparator that we want to replace and test
* compatibility with.
*
* @return whether {@code previous} can be safely replaced by this comparator.
*/
public boolean isCompatibleWith(ClusteringComparator previous)
{
if (this == previous)
return true;
// Extending with new components is fine, shrinking is not
if (size() < previous.size())
return false;
for (int i = 0; i < previous.size(); i++)
{
AbstractType<?> tprev = previous.subtype(i);
AbstractType<?> tnew = subtype(i);
if (!tnew.isCompatibleWith(tprev))
return false;
}
return true;
}
/**
* Validates the provided prefix for corrupted data.
*
* @param clustering the clustering prefix to validate.
*
* @throws MarshalException if {@code clustering} contains some invalid data.
*/
public void validate(ClusteringPrefix clustering)
{
for (int i = 0; i < clustering.size(); i++)
{
ByteBuffer value = clustering.get(i);
if (value != null)
subtype(i).validate(value);
}
}
public Comparator<IndexInfo> indexComparator(boolean reversed)
{
return reversed ? indexReverseComparator : indexComparator;
}
public Comparator<Clusterable> reversed()
{
return reverseComparator;
}
/**
* Whether the two provided clustering prefix are on the same clustering values.
*
* @param c1 the first prefix.
* @param c2 the second prefix.
* @return whether {@code c1} and {@code c2} have the same clustering values (but not necessarily
* the same "kind") or not.
*/
public boolean isOnSameClustering(ClusteringPrefix c1, ClusteringPrefix c2)
{
if (c1.size() != c2.size())
return false;
for (int i = 0; i < c1.size(); i++)
{
if (compareComponent(i, c1.get(i), c2.get(i)) != 0)
return false;
}
return true;
}
@Override
public String toString()
{
return String.format("comparator(%s)", Joiner.on(", ").join(clusteringTypes));
}
@Override
public boolean equals(Object o)
{
if (this == o)
return true;
if (!(o instanceof ClusteringComparator))
return false;
ClusteringComparator that = (ClusteringComparator)o;
return this.clusteringTypes.equals(that.clusteringTypes);
}
@Override
public int hashCode()
{
return Objects.hashCode(clusteringTypes);
}
}

513
src/java/org/apache/cassandra/db/ClusteringPrefix.java Normal file
View File

@ -0,0 +1,513 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.db;
import java.io.DataInput;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.security.MessageDigest;
import java.util.*;
import org.apache.cassandra.cache.IMeasurableMemory;
import org.apache.cassandra.config.CFMetaData;
import org.apache.cassandra.db.rows.*;
import org.apache.cassandra.db.marshal.AbstractType;
import org.apache.cassandra.io.util.DataOutputPlus;
import org.apache.cassandra.utils.ByteBufferUtil;
/**
* A clustering prefix is basically the unit of what a {@link ClusteringComparator} can compare.
* <p>
* It holds values for the clustering columns of a table (potentially only a prefix of all of them) and it has
* a "kind" that allows us to implement slices with inclusive and exclusive bounds.
* <p>
* In practice, {@code ClusteringPrefix} is just the common parts to its 2 main subtype: {@link Clustering} and
* {@link Slice.Bound}, where:
* 1) {@code Clustering} represents the clustering values for a row, i.e. the values for it's clustering columns.
* 2) {@code Slice.Bound} represents a bound (start or end) of a slice (of rows).
* See those classes for more details.
*/
public interface ClusteringPrefix extends Aliasable<ClusteringPrefix>, IMeasurableMemory, Clusterable
{
public static final Serializer serializer = new Serializer();
/**
* The kind of clustering prefix this actually is.
*
* The kind {@code STATIC_CLUSTERING} is only implemented by {@link Clustering.STATIC_CLUSTERING} and {@code CLUSTERING} is
* implemented by the {@link Clustering} class. The rest is used by {@link Slice.Bound} and {@link RangeTombstone.Bound}.
*/
public enum Kind
{
// WARNING: the ordering of that enum matters because we use ordinal() in the serialization
EXCL_END_BOUND(0, -1),
INCL_START_BOUND(1, -1),
EXCL_END_INCL_START_BOUNDARY(1, -1),
STATIC_CLUSTERING(2, -1),
CLUSTERING(3, 0),
INCL_END_EXCL_START_BOUNDARY(4, -1),
INCL_END_BOUND(4, 1),
EXCL_START_BOUND(5, 1);
private final int comparison;
// If clusterable c1 has this Kind and is a strict prefix of clusterable c2, then this
// is the result of compare(c1, c2). Basically, this is the same as comparing the kind of c1 to
// CLUSTERING.
public final int prefixComparisonResult;
private Kind(int comparison, int prefixComparisonResult)
{
this.comparison = comparison;
this.prefixComparisonResult = prefixComparisonResult;
}
/**
* Compares the 2 provided kind.
* <p>
* Note: this should be used instead of {@link #compareTo} when comparing clustering prefixes. We do
* not override that latter method because it is final for an enum.
*/
public static int compare(Kind k1, Kind k2)
{
return Integer.compare(k1.comparison, k2.comparison);
}
/**
* Returns the inverse of the current kind.
* <p>
* This invert both start into end (and vice-versa) and inclusive into exclusive (and vice-versa).
*
* @return the invert of this kind. For instance, if this kind is an exlusive start, this return
* an inclusive end.
*/
public Kind invert()
{
switch (this)
{
case EXCL_START_BOUND: return INCL_END_BOUND;
case INCL_START_BOUND: return EXCL_END_BOUND;
case EXCL_END_BOUND: return INCL_START_BOUND;
case INCL_END_BOUND: return EXCL_START_BOUND;
case EXCL_END_INCL_START_BOUNDARY: return INCL_END_EXCL_START_BOUNDARY;
case INCL_END_EXCL_START_BOUNDARY: return EXCL_END_INCL_START_BOUNDARY;
default: return this;
}
}
public boolean isBound()
{
switch (this)
{
case INCL_START_BOUND:
case INCL_END_BOUND:
case EXCL_START_BOUND:
case EXCL_END_BOUND:
return true;
}
return false;
}
public boolean isBoundary()
{
switch (this)
{
case INCL_END_EXCL_START_BOUNDARY:
case EXCL_END_INCL_START_BOUNDARY:
return true;
}
return false;
}
public boolean isStart()
{
switch (this)
{
case INCL_START_BOUND:
case EXCL_END_INCL_START_BOUNDARY:
case INCL_END_EXCL_START_BOUNDARY:
case EXCL_START_BOUND:
return true;
default:
return false;
}
}
public boolean isEnd()
{
switch (this)
{
case INCL_END_BOUND:
case EXCL_END_INCL_START_BOUNDARY:
case INCL_END_EXCL_START_BOUNDARY:
case EXCL_END_BOUND:
return true;
default:
return false;
}
}
public boolean isOpen(boolean reversed)
{
return reversed ? isEnd() : isStart();
}
public boolean isClose(boolean reversed)
{
return reversed ? isStart() : isEnd();
}
public Kind closeBoundOfBoundary(boolean reversed)
{
assert isBoundary();
return reversed
? (this == INCL_END_EXCL_START_BOUNDARY ? EXCL_START_BOUND : INCL_START_BOUND)
: (this == INCL_END_EXCL_START_BOUNDARY ? INCL_END_BOUND : EXCL_END_BOUND);
}
public Kind openBoundOfBoundary(boolean reversed)
{
assert isBoundary();
return reversed
? (this == INCL_END_EXCL_START_BOUNDARY ? INCL_END_BOUND : EXCL_END_BOUND)
: (this == INCL_END_EXCL_START_BOUNDARY ? EXCL_START_BOUND : INCL_START_BOUND);
}
}
public Kind kind();
/**
* The number of values in this prefix.
*
* There can't be more values that the this is a prefix of has of clustering columns.
*
* @return the number of values in this prefix.
*/
public int size();
/**
* Retrieves the ith value of this prefix.
*
* @param i the index of the value to retrieve. Must be such that {@code 0 <= i < size()}.
*
* @return the ith value of this prefix. Note that a value can be {@code null}.
*/
public ByteBuffer get(int i);
public void digest(MessageDigest digest);
// Used to verify if batches goes over a given size
public int dataSize();
public String toString(CFMetaData metadata);
public void writeTo(Writer writer);
/**
* The values of this prefix as an array.
* <p>
* Please note that this may or may not require an array creation. So 1) you should *not*
* modify the returned array and 2) it's more efficient to use {@link #size()} and
* {@link #get} unless you actually need an array.
*
* @return the values for this prefix as an array.
*/
public ByteBuffer[] getRawValues();
/**
* Interface for writing a clustering prefix.
* <p>
* Each value for the prefix should simply be written in order.
*/
public interface Writer
{
/**
* Write the next value to the writer.
*
* @param value the value to write.
*/
public void writeClusteringValue(ByteBuffer value);
}
public static class Serializer
{
public void serialize(ClusteringPrefix clustering, DataOutputPlus out, int version, List<AbstractType<?>> types) throws IOException
{
// We shouldn't serialize static clusterings
assert clustering.kind() != Kind.STATIC_CLUSTERING;
if (clustering.kind() == Kind.CLUSTERING)
{
out.writeByte(clustering.kind().ordinal());
Clustering.serializer.serialize((Clustering)clustering, out, version, types);
}
else
{
Slice.Bound.serializer.serialize((Slice.Bound)clustering, out, version, types);
}
}
public ClusteringPrefix deserialize(DataInput in, int version, List<AbstractType<?>> types) throws IOException
{
Kind kind = Kind.values()[in.readByte()];
// We shouldn't serialize static clusterings
assert kind != Kind.STATIC_CLUSTERING;
if (kind == Kind.CLUSTERING)
return Clustering.serializer.deserialize(in, version, types);
else
return Slice.Bound.serializer.deserializeValues(in, kind, version, types);
}
public long serializedSize(ClusteringPrefix clustering, int version, List<AbstractType<?>> types, TypeSizes sizes)
{
// We shouldn't serialize static clusterings
assert clustering.kind() != Kind.STATIC_CLUSTERING;
if (clustering.kind() == Kind.CLUSTERING)
return 1 + Clustering.serializer.serializedSize((Clustering)clustering, version, types, sizes);
else
return Slice.Bound.serializer.serializedSize((Slice.Bound)clustering, version, types, sizes);
}
void serializeValuesWithoutSize(ClusteringPrefix clustering, DataOutputPlus out, int version, List<AbstractType<?>> types) throws IOException
{
if (clustering.size() == 0)
return;
writeHeader(clustering, out);
for (int i = 0; i < clustering.size(); i++)
{
ByteBuffer v = clustering.get(i);
if (v == null || !v.hasRemaining())
continue; // handled in the header
types.get(i).writeValue(v, out);
}
}
long valuesWithoutSizeSerializedSize(ClusteringPrefix clustering, int version, List<AbstractType<?>> types, TypeSizes sizes)
{
if (clustering.size() == 0)
return 0;
long size = headerBytesCount(clustering.size());
for (int i = 0; i < clustering.size(); i++)
{
ByteBuffer v = clustering.get(i);
if (v == null || !v.hasRemaining())
continue; // handled in the header
size += types.get(i).writtenLength(v, sizes);
}
return size;
}
void deserializeValuesWithoutSize(DataInput in, int size, int version, List<AbstractType<?>> types, ClusteringPrefix.Writer writer) throws IOException
{
if (size == 0)
return;
int[] header = readHeader(size, in);
for (int i = 0; i < size; i++)
{
if (isNull(header, i))
writer.writeClusteringValue(null);
else if (isEmpty(header, i))
writer.writeClusteringValue(ByteBufferUtil.EMPTY_BYTE_BUFFER);
else
writer.writeClusteringValue(types.get(i).readValue(in));
}
}
private int headerBytesCount(int size)
{
// For each component, we store 2 bit to know if the component is empty or null (or neither).
// We thus handle 4 component per byte
return size / 4 + (size % 4 == 0 ? 0 : 1);
}
/**
* Whatever the type of a given clustering column is, its value can always be either empty or null. So we at least need to distinguish those
* 2 values, and because we want to be able to store fixed width values without appending their (fixed) size first, we need a way to encode
* empty values too. So for that, every clustering prefix includes a "header" that contains 2 bits per element in the prefix. For each element,
* those 2 bits encode whether the element is null, empty, or none of those.
*/
private void writeHeader(ClusteringPrefix clustering, DataOutputPlus out) throws IOException
{
int nbBytes = headerBytesCount(clustering.size());
for (int i = 0; i < nbBytes; i++)
{
int b = 0;
for (int j = 0; j < 4; j++)
{
int c = i * 4 + j;
if (c >= clustering.size())
break;
ByteBuffer v = clustering.get(c);
if (v == null)
b |= (1 << (j * 2) + 1);
else if (!v.hasRemaining())
b |= (1 << (j * 2));
}
out.writeByte((byte)b);
}
}
private int[] readHeader(int size, DataInput in) throws IOException
{
int nbBytes = headerBytesCount(size);
int[] header = new int[nbBytes];
for (int i = 0; i < nbBytes; i++)
header[i] = in.readUnsignedByte();
return header;
}
private boolean isNull(int[] header, int i)
{
int b = header[i / 4];
int mask = 1 << ((i % 4) * 2) + 1;
return (b & mask) != 0;
}
private boolean isEmpty(int[] header, int i)
{
int b = header[i / 4];
int mask = 1 << ((i % 4) * 2);
return (b & mask) != 0;
}
}
/**
* Helper class that makes the deserialization of clustering prefixes faster.
* <p>
* The main reason for this is that when we deserialize rows from sstables, there is many cases where we have
* a bunch of rows to skip at the beginning of an index block because those rows are before the requested slice.
* This class make sure we can answer the question "is the next row on disk before the requested slice" with as
* little work as possible. It does that by providing a comparison method that deserialize only what is needed
* to decide of the comparison.
*/
public static class Deserializer
{
private final ClusteringComparator comparator;
private final DataInput in;
private final SerializationHeader serializationHeader;
private boolean nextIsRow;
private int[] nextHeader;
private int nextSize;
private ClusteringPrefix.Kind nextKind;
private int deserializedSize;
private final ByteBuffer[] nextValues;
public Deserializer(ClusteringComparator comparator, DataInput in, SerializationHeader header)
{
this.comparator = comparator;
this.in = in;
this.serializationHeader = header;
this.nextValues = new ByteBuffer[comparator.size()];
}
public void prepare(int flags) throws IOException
{
assert !UnfilteredSerializer.isStatic(flags) : "Flags = " + flags;
this.nextIsRow = UnfilteredSerializer.kind(flags) == Unfiltered.Kind.ROW;
this.nextKind = nextIsRow ? Kind.CLUSTERING : ClusteringPrefix.Kind.values()[in.readByte()];
this.nextSize = nextIsRow ? comparator.size() : in.readUnsignedShort();
this.nextHeader = serializer.readHeader(nextSize, in);
this.deserializedSize = 0;
}
public int compareNextTo(Slice.Bound bound) throws IOException
{
if (bound == Slice.Bound.TOP)
return -1;
for (int i = 0; i < bound.size(); i++)
{
if (!hasComponent(i))
return nextKind.prefixComparisonResult;
int cmp = comparator.compareComponent(i, nextValues[i], bound.get(i));
if (cmp != 0)
return cmp;
}
if (bound.size() == nextSize)
return nextKind.compareTo(bound.kind());
// We know that we'll have exited already if nextSize < bound.size
return -bound.kind().prefixComparisonResult;
}
private boolean hasComponent(int i) throws IOException
{
if (i >= nextSize)
return false;
while (deserializedSize <= i)
deserializeOne();
return true;
}
private boolean deserializeOne() throws IOException
{
if (deserializedSize == nextSize)
return false;
int i = deserializedSize++;
nextValues[i] = serializer.isNull(nextHeader, i)
? null
: (serializer.isEmpty(nextHeader, i) ? ByteBufferUtil.EMPTY_BYTE_BUFFER : serializationHeader.clusteringTypes().get(i).readValue(in));
return true;
}
private void deserializeAll() throws IOException
{
while (deserializeOne())
continue;
}
public RangeTombstone.Bound.Kind deserializeNextBound(RangeTombstone.Bound.Writer writer) throws IOException
{
assert !nextIsRow;
deserializeAll();
for (int i = 0; i < nextSize; i++)
writer.writeClusteringValue(nextValues[i]);
writer.writeBoundKind(nextKind);
return nextKind;
}
public void deserializeNextClustering(Clustering.Writer writer) throws IOException
{
assert nextIsRow && nextSize == nextValues.length;
deserializeAll();
for (int i = 0; i < nextSize; i++)
writer.writeClusteringValue(nextValues[i]);
}
public ClusteringPrefix.Kind skipNext() throws IOException
{
for (int i = deserializedSize; i < nextSize; i++)
if (!serializer.isNull(nextHeader, i) && !serializer.isEmpty(nextHeader, i))
serializationHeader.clusteringTypes().get(i).skipValue(in);
return nextKind;
}
}
}

334
src/java/org/apache/cassandra/db/CollationController.java
View File

@ -1,334 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.db;
import java.io.Closeable;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Iterator;
import java.util.List;
import java.util.TreeSet;
import com.google.common.base.Function;
import com.google.common.collect.Iterables;
import com.google.common.collect.Iterators;
import org.apache.cassandra.concurrent.Stage;
import org.apache.cassandra.concurrent.StageManager;
import org.apache.cassandra.db.columniterator.OnDiskAtomIterator;
import org.apache.cassandra.db.composites.CellName;
import org.apache.cassandra.db.filter.NamesQueryFilter;
import org.apache.cassandra.db.filter.QueryFilter;
import org.apache.cassandra.db.marshal.CounterColumnType;
import org.apache.cassandra.io.sstable.format.SSTableReader;
import org.apache.cassandra.io.util.FileUtils;
import org.apache.cassandra.tracing.Tracing;
import org.apache.cassandra.utils.SearchIterator;
import org.apache.cassandra.utils.memory.HeapAllocator;
public class CollationController
{
private final ColumnFamilyStore cfs;
private final QueryFilter filter;
private final int gcBefore;
private int sstablesIterated = 0;
public CollationController(ColumnFamilyStore cfs, QueryFilter filter, int gcBefore)
{
this.cfs = cfs;
this.filter = filter;
this.gcBefore = gcBefore;
}
public ColumnFamily getTopLevelColumns(boolean copyOnHeap)
{
return filter.filter instanceof NamesQueryFilter
&& cfs.metadata.getDefaultValidator() != CounterColumnType.instance
? collectTimeOrderedData(copyOnHeap)
: collectAllData(copyOnHeap);
}
/**
* Collects data in order of recency, using the sstable maxtimestamp data.
* Once we have data for all requests columns that is newer than the newest remaining maxtimestamp,
* we stop.
*/
private ColumnFamily collectTimeOrderedData(boolean copyOnHeap)
{
final ColumnFamily container = ArrayBackedSortedColumns.factory.create(cfs.metadata, filter.filter.isReversed());
List<OnDiskAtomIterator> iterators = new ArrayList<>();
boolean isEmpty = true;
Tracing.trace("Acquiring sstable references");
ColumnFamilyStore.ViewFragment view = cfs.select(cfs.viewFilter(filter.key));
DeletionInfo returnDeletionInfo = container.deletionInfo();
try
{
Tracing.trace("Merging memtable contents");
for (Memtable memtable : view.memtables)
{
ColumnFamily cf = memtable.getColumnFamily(filter.key);
if (cf != null)
{
filter.delete(container.deletionInfo(), cf);
isEmpty = false;
Iterator<Cell> iter = filter.getIterator(cf);
while (iter.hasNext())
{
Cell cell = iter.next();
if (copyOnHeap)
cell = cell.localCopy(cfs.metadata, HeapAllocator.instance);
container.addColumn(cell);
}
}
}
// avoid changing the filter columns of the original filter
// (reduceNameFilter removes columns that are known to be irrelevant)
NamesQueryFilter namesFilter = (NamesQueryFilter) filter.filter;
TreeSet<CellName> filterColumns = new TreeSet<>(namesFilter.columns);
QueryFilter reducedFilter = new QueryFilter(filter.key, filter.cfName, namesFilter.withUpdatedColumns(filterColumns), filter.timestamp);
/* add the SSTables on disk */
Collections.sort(view.sstables, SSTableReader.maxTimestampComparator);
// read sorted sstables
for (SSTableReader sstable : view.sstables)
{
// if we've already seen a row tombstone with a timestamp greater
// than the most recent update to this sstable, we're done, since the rest of the sstables
// will also be older
if (sstable.getMaxTimestamp() < returnDeletionInfo.getTopLevelDeletion().markedForDeleteAt)
break;
long currentMaxTs = sstable.getMaxTimestamp();
reduceNameFilter(reducedFilter, container, currentMaxTs);
if (((NamesQueryFilter) reducedFilter.filter).columns.isEmpty())
break;
Tracing.trace("Merging data from sstable {}", sstable.descriptor.generation);
sstable.incrementReadCount();
OnDiskAtomIterator iter = reducedFilter.getSSTableColumnIterator(sstable);
iterators.add(iter);
isEmpty = false;
if (iter.getColumnFamily() != null)
{
container.delete(iter.getColumnFamily());
sstablesIterated++;
while (iter.hasNext())
container.addAtom(iter.next());
}
}
// we need to distinguish between "there is no data at all for this row" (BF will let us rebuild that efficiently)
// and "there used to be data, but it's gone now" (we should cache the empty CF so we don't need to rebuild that slower)
if (isEmpty)
return null;
// do a final collate. toCollate is boilerplate required to provide a CloseableIterator
ColumnFamily returnCF = container.cloneMeShallow();
Tracing.trace("Collating all results");
filter.collateOnDiskAtom(returnCF, container.iterator(), gcBefore);
// "hoist up" the requested data into a more recent sstable
if (sstablesIterated > cfs.getMinimumCompactionThreshold()
&& !cfs.isAutoCompactionDisabled()
&& cfs.getCompactionStrategyManager().shouldDefragment())
{
// !!WARNING!! if we stop copying our data to a heap-managed object,
// we will need to track the lifetime of this mutation as well
Tracing.trace("Defragmenting requested data");
final Mutation mutation = new Mutation(cfs.keyspace.getName(), filter.key.getKey(), returnCF.cloneMe());
StageManager.getStage(Stage.MUTATION).execute(new Runnable()
{
public void run()
{
// skipping commitlog and index updates is fine since we're just de-fragmenting existing data
Keyspace.open(mutation.getKeyspaceName()).apply(mutation, false, false);
}
});
}
// Caller is responsible for final removeDeletedCF. This is important for cacheRow to work correctly:
return returnCF;
}
finally
{
for (OnDiskAtomIterator iter : iterators)
FileUtils.closeQuietly(iter);
}
}
/**
* remove columns from @param filter where we already have data in @param container newer than @param sstableTimestamp
*/
private void reduceNameFilter(QueryFilter filter, ColumnFamily container, long sstableTimestamp)
{
if (container == null)
return;
SearchIterator<CellName, Cell> searchIter = container.searchIterator();
for (Iterator<CellName> iterator = ((NamesQueryFilter) filter.filter).columns.iterator(); iterator.hasNext() && searchIter.hasNext(); )
{
CellName filterColumn = iterator.next();
Cell cell = searchIter.next(filterColumn);
if (cell != null && cell.timestamp() > sstableTimestamp)
iterator.remove();
}
}
/**
* Collects data the brute-force way: gets an iterator for the filter in question
* from every memtable and sstable, then merges them together.
*/
private ColumnFamily collectAllData(boolean copyOnHeap)
{
Tracing.trace("Acquiring sstable references");
ColumnFamilyStore.ViewFragment view = cfs.select(cfs.viewFilter(filter.key));
List<Iterator<? extends OnDiskAtom>> iterators = new ArrayList<>(Iterables.size(view.memtables) + view.sstables.size());
ColumnFamily returnCF = ArrayBackedSortedColumns.factory.create(cfs.metadata, filter.filter.isReversed());
DeletionInfo returnDeletionInfo = returnCF.deletionInfo();
try
{
Tracing.trace("Merging memtable tombstones");
for (Memtable memtable : view.memtables)
{
final ColumnFamily cf = memtable.getColumnFamily(filter.key);
if (cf != null)
{
filter.delete(returnDeletionInfo, cf);
Iterator<Cell> iter = filter.getIterator(cf);
if (copyOnHeap)
{
iter = Iterators.transform(iter, new Function<Cell, Cell>()
{
public Cell apply(Cell cell)
{
return cell.localCopy(cf.metadata, HeapAllocator.instance);
}
});
}
iterators.add(iter);
}
}
/*
* We can't eliminate full sstables based on the timestamp of what we've already read like
* in collectTimeOrderedData, but we still want to eliminate sstable whose maxTimestamp < mostRecentTombstone
* we've read. We still rely on the sstable ordering by maxTimestamp since if
* maxTimestamp_s1 > maxTimestamp_s0,
* we're guaranteed that s1 cannot have a row tombstone such that
* timestamp(tombstone) > maxTimestamp_s0
* since we necessarily have
* timestamp(tombstone) <= maxTimestamp_s1
* In other words, iterating in maxTimestamp order allow to do our mostRecentTombstone elimination
* in one pass, and minimize the number of sstables for which we read a rowTombstone.
*/
Collections.sort(view.sstables, SSTableReader.maxTimestampComparator);
List<SSTableReader> skippedSSTables = null;
long minTimestamp = Long.MAX_VALUE;
int nonIntersectingSSTables = 0;
for (SSTableReader sstable : view.sstables)
{
minTimestamp = Math.min(minTimestamp, sstable.getMinTimestamp());
// if we've already seen a row tombstone with a timestamp greater
// than the most recent update to this sstable, we can skip it
if (sstable.getMaxTimestamp() < returnDeletionInfo.getTopLevelDeletion().markedForDeleteAt)
break;
if (!filter.shouldInclude(sstable))
{
nonIntersectingSSTables++;
// sstable contains no tombstone if maxLocalDeletionTime == Integer.MAX_VALUE, so we can safely skip those entirely
if (sstable.getSSTableMetadata().maxLocalDeletionTime != Integer.MAX_VALUE)
{
if (skippedSSTables == null)
skippedSSTables = new ArrayList<>();
skippedSSTables.add(sstable);
}
continue;
}
sstable.incrementReadCount();
OnDiskAtomIterator iter = filter.getSSTableColumnIterator(sstable);
iterators.add(iter);
if (iter.getColumnFamily() != null)
{
ColumnFamily cf = iter.getColumnFamily();
returnCF.delete(cf);
sstablesIterated++;
}
}
int includedDueToTombstones = 0;
// Check for row tombstone in the skipped sstables
if (skippedSSTables != null)
{
for (SSTableReader sstable : skippedSSTables)
{
if (sstable.getMaxTimestamp() <= minTimestamp)
continue;
sstable.incrementReadCount();
OnDiskAtomIterator iter = filter.getSSTableColumnIterator(sstable);
ColumnFamily cf = iter.getColumnFamily();
// we are only interested in row-level tombstones here, and only if markedForDeleteAt is larger than minTimestamp
if (cf != null && cf.deletionInfo().getTopLevelDeletion().markedForDeleteAt > minTimestamp)
{
includedDueToTombstones++;
iterators.add(iter);
returnCF.delete(cf.deletionInfo().getTopLevelDeletion());
sstablesIterated++;
}
else
{
FileUtils.closeQuietly(iter);
}
}
}
if (Tracing.isTracing())
Tracing.trace("Skipped {}/{} non-slice-intersecting sstables, included {} due to tombstones",
nonIntersectingSSTables, view.sstables.size(), includedDueToTombstones);
// we need to distinguish between "there is no data at all for this row" (BF will let us rebuild that efficiently)
// and "there used to be data, but it's gone now" (we should cache the empty CF so we don't need to rebuild that slower)
if (iterators.isEmpty())
return null;
Tracing.trace("Merging data from memtables and {} sstables", sstablesIterated);
filter.collateOnDiskAtom(returnCF, iterators, gcBefore);
// Caller is responsible for final removeDeletedCF. This is important for cacheRow to work correctly:
return returnCF;
}
finally
{
for (Object iter : iterators)
if (iter instanceof Closeable)
FileUtils.closeQuietly((Closeable) iter);
}
}
public int getSstablesIterated()
{
return sstablesIterated;
}
}

565
src/java/org/apache/cassandra/db/ColumnFamily.java
View File

@ -1,565 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.db;
import java.io.DataInputStream;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.security.MessageDigest;
import java.util.Collection;
import java.util.Collections;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.UUID;
import com.google.common.collect.ImmutableMap;
import org.apache.commons.lang3.builder.HashCodeBuilder;
import org.apache.cassandra.cache.IRowCacheEntry;
import org.apache.cassandra.config.CFMetaData;
import org.apache.cassandra.config.Schema;
import org.apache.cassandra.db.composites.CellName;
import org.apache.cassandra.db.composites.CellNameType;
import org.apache.cassandra.db.composites.CellNames;
import org.apache.cassandra.db.filter.ColumnCounter;
import org.apache.cassandra.db.filter.ColumnSlice;
import org.apache.cassandra.io.sstable.ColumnNameHelper;
import org.apache.cassandra.io.sstable.ColumnStats;
import org.apache.cassandra.io.sstable.SSTable;
import org.apache.cassandra.io.util.DataOutputBuffer;
import org.apache.cassandra.net.MessagingService;
import org.apache.cassandra.utils.*;
/**
* A sorted map of columns.
* This represents the backing map of a colum family.
*
* Whether the implementation is thread safe or not is left to the
* implementing classes.
*/
public abstract class ColumnFamily implements Iterable<Cell>, IRowCacheEntry
{
/* The column serializer for this Column Family. Create based on config. */
public static final ColumnFamilySerializer serializer = new ColumnFamilySerializer();
protected final CFMetaData metadata;
protected ColumnFamily(CFMetaData metadata)
{
assert metadata != null;
this.metadata = metadata;
}
public <T extends ColumnFamily> T cloneMeShallow(ColumnFamily.Factory<T> factory, boolean reversedInsertOrder)
{
T cf = factory.create(metadata, reversedInsertOrder);
cf.delete(this);
return cf;
}
public ColumnFamily cloneMeShallow()
{
return cloneMeShallow(false);
}
public ColumnFamily cloneMeShallow(boolean reversed)
{
return cloneMeShallow(getFactory(), reversed);
}
public ColumnFamilyType getType()
{
return metadata.cfType;
}
public int liveCQL3RowCount(long now)
{
ColumnCounter counter = getComparator().isDense()
? new ColumnCounter(now)
: new ColumnCounter.GroupByPrefix(now, getComparator(), metadata.clusteringColumns().size());
return counter.countAll(this).live();
}
/**
* Clones the column map.
*/
public abstract ColumnFamily cloneMe();
public UUID id()
{
return metadata.cfId;
}
/**
* @return The CFMetaData for this row
*/
public CFMetaData metadata()
{
return metadata;
}
public void addColumn(CellName name, ByteBuffer value, long timestamp)
{
addColumn(name, value, timestamp, 0);
}
public void addColumn(CellName name, ByteBuffer value, long timestamp, int timeToLive)
{
assert !metadata().isCounter();
Cell cell = AbstractCell.create(name, value, timestamp, timeToLive, metadata());
addColumn(cell);
}
public void addCounter(CellName name, long value)
{
addColumn(new BufferCounterUpdateCell(name, value, FBUtilities.timestampMicros()));
}
public void addTombstone(CellName name, ByteBuffer localDeletionTime, long timestamp)
{
addColumn(new BufferDeletedCell(name, localDeletionTime, timestamp));
}
public void addTombstone(CellName name, int localDeletionTime, long timestamp)
{
addColumn(new BufferDeletedCell(name, localDeletionTime, timestamp));
}
public void addAtom(OnDiskAtom atom)
{
if (atom instanceof Cell)
{
addColumn((Cell)atom);
}
else
{
assert atom instanceof RangeTombstone;
delete((RangeTombstone)atom);
}
}
/**
* Clear this column family, removing all columns and deletion info.
*/
public abstract void clear();
/**
* Returns a {@link DeletionInfo.InOrderTester} for the deletionInfo() of
* this column family. Please note that for ThreadSafe implementation of ColumnFamily,
* this tester will remain valid even if new tombstones are added to this ColumnFamily
* *as long as said addition is done in comparator order*. For AtomicSortedColumns,
* the tester will correspond to the state of when this method is called.
*/
public DeletionInfo.InOrderTester inOrderDeletionTester()
{
return deletionInfo().inOrderTester();
}
/**
* Returns the factory used for this ISortedColumns implementation.
*/
public abstract Factory getFactory();
public abstract DeletionInfo deletionInfo();
public abstract void setDeletionInfo(DeletionInfo info);
public abstract void delete(DeletionInfo info);
public abstract void delete(DeletionTime deletionTime);
protected abstract void delete(RangeTombstone tombstone);
public abstract SearchIterator<CellName, Cell> searchIterator();
/**
* Purges top-level and range tombstones whose localDeletionTime is older than gcBefore.
* @param gcBefore a timestamp (in seconds) before which tombstones should be purged
*/
public abstract void purgeTombstones(int gcBefore);
/**
* Adds a cell to this cell map.
* If a cell with the same name is already present in the map, it will
* be replaced by the newly added cell.
*/
public abstract void addColumn(Cell cell);
/**
* Adds a cell if it's non-gc-able and isn't shadowed by a partition/range tombstone with a higher timestamp.
* Requires that the cell to add is sorted strictly after the last cell in the container.
*/
public abstract void maybeAppendColumn(Cell cell, DeletionInfo.InOrderTester tester, int gcBefore);
/**
* Appends a cell. Requires that the cell to add is sorted strictly after the last cell in the container.
*/
public abstract void appendColumn(Cell cell);
/**
* Adds all the columns of a given column map to this column map.
* This is equivalent to:
* <code>
* for (Cell c : cm)
* addColumn(c, ...);
* </code>
* but is potentially faster.
*/
public abstract void addAll(ColumnFamily cm);
/**
* Get a column given its name, returning null if the column is not
* present.
*/
public abstract Cell getColumn(CellName name);
/**
* Returns an iterable with the names of columns in this column map in the same order
* as the underlying columns themselves.
*/
public abstract Iterable<CellName> getColumnNames();
/**
* Returns the columns of this column map as a collection.
* The columns in the returned collection should be sorted as the columns
* in this map.
*/
public abstract Collection<Cell> getSortedColumns();
/**
* Returns the columns of this column map as a collection.
* The columns in the returned collection should be sorted in reverse
* order of the columns in this map.
*/
public abstract Collection<Cell> getReverseSortedColumns();
/**
* Returns the number of columns in this map.
*/
public abstract int getColumnCount();
/**
* Returns whether or not there are any columns present.
*/
public abstract boolean hasColumns();
/**
* Returns true if this contains no columns or deletion info
*/
public boolean isEmpty()
{
return deletionInfo().isLive() && !hasColumns();
}
/**
* Returns an iterator over the columns of this map that returns only the matching @param slices.
* The provided slices must be in order and must be non-overlapping.
*/
public abstract Iterator<Cell> iterator(ColumnSlice[] slices);
/**
* Returns a reversed iterator over the columns of this map that returns only the matching @param slices.
* The provided slices must be in reversed order and must be non-overlapping.
*/
public abstract Iterator<Cell> reverseIterator(ColumnSlice[] slices);
/**
* Returns if this map only support inserts in reverse order.
*/
public abstract boolean isInsertReversed();
/**
* If `columns` has any tombstones (top-level or range tombstones), they will be applied to this set of columns.
*/
public void delete(ColumnFamily columns)
{
delete(columns.deletionInfo());
}
/*
* This function will calculate the difference between 2 column families.
* The external input is assumed to be a superset of internal.
*/
public ColumnFamily diff(ColumnFamily cfComposite)
{
assert cfComposite.id().equals(id());
ColumnFamily cfDiff = ArrayBackedSortedColumns.factory.create(metadata);
cfDiff.delete(cfComposite.deletionInfo());
// (don't need to worry about cfNew containing Columns that are shadowed by
// the delete tombstone, since cfNew was generated by CF.resolve, which
// takes care of those for us.)
for (Cell cellExternal : cfComposite)
{
CellName cName = cellExternal.name();
Cell cellInternal = getColumn(cName);
if (cellInternal == null)
{
cfDiff.addColumn(cellExternal);
}
else
{
Cell cellDiff = cellInternal.diff(cellExternal);
if (cellDiff != null)
{
cfDiff.addColumn(cellDiff);
}
}
}
cfDiff.setDeletionInfo(deletionInfo().diff(cfComposite.deletionInfo()));
if (!cfDiff.isEmpty())
return cfDiff;
return null;
}
public long dataSize()
{
long size = 0;
for (Cell cell : this)
size += cell.cellDataSize();
return size;
}
public long maxTimestamp()
{
long maxTimestamp = deletionInfo().maxTimestamp();
for (Cell cell : this)
maxTimestamp = Math.max(maxTimestamp, cell.timestamp());
return maxTimestamp;
}
@Override
public int hashCode()
{
HashCodeBuilder builder = new HashCodeBuilder(373, 75437)
.append(metadata)
.append(deletionInfo());
for (Cell cell : this)
builder.append(cell);
return builder.toHashCode();
}
@Override
public boolean equals(Object o)
{
if (this == o)
return true;
if (o == null || !(o instanceof ColumnFamily))
return false;
ColumnFamily comparison = (ColumnFamily) o;
return metadata.equals(comparison.metadata)
&& deletionInfo().equals(comparison.deletionInfo())
&& ByteBufferUtil.compareUnsigned(digest(this), digest(comparison)) == 0;
}
@Override
public String toString()
{
StringBuilder sb = new StringBuilder("ColumnFamily(");
sb.append(metadata.cfName);
if (isMarkedForDelete())
sb.append(" -").append(deletionInfo()).append("-");
sb.append(" [").append(CellNames.getColumnsString(getComparator(), this)).append("])");
return sb.toString();
}
public static ByteBuffer digest(ColumnFamily cf)
{
MessageDigest digest = FBUtilities.threadLocalMD5Digest();
if (cf != null)
cf.updateDigest(digest);
return ByteBuffer.wrap(digest.digest());
}
public void updateDigest(MessageDigest digest)
{
for (Cell cell : this)
cell.updateDigest(digest);
deletionInfo().updateDigest(digest);
}
public static ColumnFamily diff(ColumnFamily cf1, ColumnFamily cf2)
{
if (cf1 == null)
return cf2;
return cf1.diff(cf2);
}
public ColumnStats getColumnStats()
{
// note that we default to MIN_VALUE/MAX_VALUE here to be able to override them later in this method
// we are checking row/range tombstones and actual cells - there should always be data that overrides
// these with actual values
ColumnStats.MinLongTracker minTimestampTracker = new ColumnStats.MinLongTracker(Long.MIN_VALUE);
ColumnStats.MaxLongTracker maxTimestampTracker = new ColumnStats.MaxLongTracker(Long.MAX_VALUE);
StreamingHistogram tombstones = new StreamingHistogram(SSTable.TOMBSTONE_HISTOGRAM_BIN_SIZE);
ColumnStats.MaxIntTracker maxDeletionTimeTracker = new ColumnStats.MaxIntTracker(Integer.MAX_VALUE);
List<ByteBuffer> minColumnNamesSeen = Collections.emptyList();
List<ByteBuffer> maxColumnNamesSeen = Collections.emptyList();
boolean hasLegacyCounterShards = false;
if (deletionInfo().getTopLevelDeletion().localDeletionTime < Integer.MAX_VALUE)
{
tombstones.update(deletionInfo().getTopLevelDeletion().localDeletionTime);
maxDeletionTimeTracker.update(deletionInfo().getTopLevelDeletion().localDeletionTime);
minTimestampTracker.update(deletionInfo().getTopLevelDeletion().markedForDeleteAt);
maxTimestampTracker.update(deletionInfo().getTopLevelDeletion().markedForDeleteAt);
}
Iterator<RangeTombstone> it = deletionInfo().rangeIterator();
while (it.hasNext())
{
RangeTombstone rangeTombstone = it.next();
tombstones.update(rangeTombstone.getLocalDeletionTime());
minTimestampTracker.update(rangeTombstone.timestamp());
maxTimestampTracker.update(rangeTombstone.timestamp());
maxDeletionTimeTracker.update(rangeTombstone.getLocalDeletionTime());
minColumnNamesSeen = ColumnNameHelper.minComponents(minColumnNamesSeen, rangeTombstone.min, metadata.comparator);
maxColumnNamesSeen = ColumnNameHelper.maxComponents(maxColumnNamesSeen, rangeTombstone.max, metadata.comparator);
}
for (Cell cell : this)
{
minTimestampTracker.update(cell.timestamp());
maxTimestampTracker.update(cell.timestamp());
maxDeletionTimeTracker.update(cell.getLocalDeletionTime());
int deletionTime = cell.getLocalDeletionTime();
if (deletionTime < Integer.MAX_VALUE)
tombstones.update(deletionTime);
minColumnNamesSeen = ColumnNameHelper.minComponents(minColumnNamesSeen, cell.name(), metadata.comparator);
maxColumnNamesSeen = ColumnNameHelper.maxComponents(maxColumnNamesSeen, cell.name(), metadata.comparator);
if (cell instanceof CounterCell)
hasLegacyCounterShards = hasLegacyCounterShards || ((CounterCell) cell).hasLegacyShards();
}
return new ColumnStats(getColumnCount(),
minTimestampTracker.get(),
maxTimestampTracker.get(),
maxDeletionTimeTracker.get(),
tombstones,
minColumnNamesSeen,
maxColumnNamesSeen,
hasLegacyCounterShards);
}
public boolean isMarkedForDelete()
{
return !deletionInfo().isLive();
}
/**
* @return the comparator whose sorting order the contained columns conform to
*/
public CellNameType getComparator()
{
return metadata.comparator;
}
public boolean hasOnlyTombstones(long now)
{
for (Cell cell : this)
if (cell.isLive(now))
return false;
return true;
}
public Iterator<Cell> iterator()
{
return getSortedColumns().iterator();
}
public Iterator<Cell> reverseIterator()
{
return getReverseSortedColumns().iterator();
}
public Map<CellName, ByteBuffer> asMap()
{
ImmutableMap.Builder<CellName, ByteBuffer> builder = ImmutableMap.builder();
for (Cell cell : this)
builder.put(cell.name(), cell.value());
return builder.build();
}
public static ColumnFamily fromBytes(ByteBuffer bytes)
{
if (bytes == null)
return null;
try
{
return serializer.deserialize(new DataInputStream(ByteBufferUtil.inputStream(bytes)),
ArrayBackedSortedColumns.factory,
ColumnSerializer.Flag.LOCAL,
MessagingService.current_version);
}
catch (IOException e)
{
throw new RuntimeException(e);
}
}
public ByteBuffer toBytes()
{
try (DataOutputBuffer out = new DataOutputBuffer())
{
serializer.serialize(this, out, MessagingService.current_version);
return ByteBuffer.wrap(out.getData(), 0, out.getLength());
}
}
/**
* @return an iterator where the removes are carried out once everything has been iterated
*/
public abstract BatchRemoveIterator<Cell> batchRemoveIterator();
public abstract static class Factory <T extends ColumnFamily>
{
/**
* Returns a (initially empty) column map whose columns are sorted
* according to the provided comparator.
* The {@code insertReversed} flag is an hint on how we expect insertion to be perfomed,
* either in sorted or reverse sorted order. This is used by ArrayBackedSortedColumns to
* allow optimizing for both forward and reversed slices. This does not matter for ThreadSafeSortedColumns.
* Note that this is only an hint on how we expect to do insertion, this does not change the map sorting.
*/
public abstract T create(CFMetaData metadata, boolean insertReversed, int initialCapacity);
public T create(CFMetaData metadata, boolean insertReversed)
{
return create(metadata, insertReversed, 0);
}
public T create(CFMetaData metadata)
{
return create(metadata, false);
}
public T create(String keyspace, String cfName)
{
return create(Schema.instance.getCFMetaData(keyspace, cfName));
}
}
}

172
src/java/org/apache/cassandra/db/ColumnFamilySerializer.java
View File

@ -1,172 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.db;
import java.io.DataInput;
import java.io.IOException;
import java.util.UUID;
import org.apache.cassandra.config.Schema;
import org.apache.cassandra.io.ISSTableSerializer;
import org.apache.cassandra.io.IVersionedSerializer;
import org.apache.cassandra.io.sstable.format.Version;
import org.apache.cassandra.io.util.DataOutputPlus;
import org.apache.cassandra.net.MessagingService;
import org.apache.cassandra.utils.UUIDSerializer;
public class ColumnFamilySerializer implements IVersionedSerializer<ColumnFamily>, ISSTableSerializer<ColumnFamily>
{
/*
* Serialized ColumnFamily format:
*
* [serialized for intra-node writes only, e.g. returning a query result]
* <cf nullability boolean: false if the cf is null>
* <cf id>
*
* [in sstable only]
* <column bloom filter>
* <sparse column index, start/finish columns every ColumnIndexSizeInKB of data>
*
* [always present]
* <local deletion time>
* <client-provided deletion time>
* <column count>
* <columns, serialized individually>
*/
public void serialize(ColumnFamily cf, DataOutputPlus out, int version)
{
try
{
if (cf == null)
{
out.writeBoolean(false);
return;
}
out.writeBoolean(true);
serializeCfId(cf.id(), out, version);
cf.getComparator().deletionInfoSerializer().serialize(cf.deletionInfo(), out, version);
ColumnSerializer columnSerializer = cf.getComparator().columnSerializer();
int count = cf.getColumnCount();
out.writeInt(count);
int written = 0;
for (Cell cell : cf)
{
columnSerializer.serialize(cell, out);
written++;
}
assert count == written: "Table had " + count + " columns, but " + written + " written";
}
catch (IOException e)
{
throw new RuntimeException(e);
}
}
public ColumnFamily deserialize(DataInput in, int version) throws IOException
{
return deserialize(in, ColumnSerializer.Flag.LOCAL, version);
}
public ColumnFamily deserialize(DataInput in, ColumnSerializer.Flag flag, int version) throws IOException
{
return deserialize(in, ArrayBackedSortedColumns.factory, flag, version);
}
public ColumnFamily deserialize(DataInput in, ColumnFamily.Factory factory, ColumnSerializer.Flag flag, int version) throws IOException
{
if (!in.readBoolean())
return null;
ColumnFamily cf = factory.create(Schema.instance.getCFMetaData(deserializeCfId(in, version)));
if (cf.metadata().isSuper() && version < MessagingService.VERSION_20)
{
SuperColumns.deserializerSuperColumnFamily(in, cf, flag, version);
}
else
{
cf.delete(cf.getComparator().deletionInfoSerializer().deserialize(in, version));
ColumnSerializer columnSerializer = cf.getComparator().columnSerializer();
int size = in.readInt();
for (int i = 0; i < size; ++i)
cf.addColumn(columnSerializer.deserialize(in, flag));
}
return cf;
}
public long contentSerializedSize(ColumnFamily cf, TypeSizes typeSizes, int version)
{
long size = cf.getComparator().deletionInfoSerializer().serializedSize(cf.deletionInfo(), typeSizes, version);
size += typeSizes.sizeof(cf.getColumnCount());
ColumnSerializer columnSerializer = cf.getComparator().columnSerializer();
for (Cell cell : cf)
size += columnSerializer.serializedSize(cell, typeSizes);
return size;
}
public long serializedSize(ColumnFamily cf, TypeSizes typeSizes, int version)
{
if (cf == null)
{
return typeSizes.sizeof(false);
}
else
{
return typeSizes.sizeof(true) /* nullness bool */
+ cfIdSerializedSize(cf.id(), typeSizes, version) /* id */
+ contentSerializedSize(cf, typeSizes, version);
}
}
public long serializedSize(ColumnFamily cf, int version)
{
return serializedSize(cf, TypeSizes.NATIVE, version);
}
public void serializeForSSTable(ColumnFamily cf, DataOutputPlus out)
{
// Column families shouldn't be written directly to disk, use ColumnIndex.Builder instead
throw new UnsupportedOperationException();
}
public ColumnFamily deserializeFromSSTable(DataInput in, Version version)
{
throw new UnsupportedOperationException();
}
public void serializeCfId(UUID cfId, DataOutputPlus out, int version) throws IOException
{
UUIDSerializer.serializer.serialize(cfId, out, version);
}
public UUID deserializeCfId(DataInput in, int version) throws IOException
{
UUID cfId = UUIDSerializer.serializer.deserialize(in, version);
if (Schema.instance.getCF(cfId) == null)
throw new UnknownColumnFamilyException("Couldn't find cfId=" + cfId, cfId);
return cfId;
}
public int cfIdSerializedSize(UUID cfId, TypeSizes typeSizes, int version)
{
return typeSizes.sizeof(cfId);
}
}

661
src/java/org/apache/cassandra/db/ColumnFamilyStore.java
View File

@ -47,19 +47,14 @@ import org.apache.cassandra.cache.*;
import org.apache.cassandra.concurrent.*;
import org.apache.cassandra.config.*;
import org.apache.cassandra.config.CFMetaData.SpeculativeRetry;
import org.apache.cassandra.db.rows.*;
import org.apache.cassandra.db.commitlog.CommitLog;
import org.apache.cassandra.db.commitlog.ReplayPosition;
import org.apache.cassandra.db.compaction.*;
import org.apache.cassandra.db.composites.CellName;
import org.apache.cassandra.db.composites.CellNameType;
import org.apache.cassandra.db.composites.Composite;
import org.apache.cassandra.db.filter.ColumnSlice;
import org.apache.cassandra.db.filter.ExtendedFilter;
import org.apache.cassandra.db.filter.IDiskAtomFilter;
import org.apache.cassandra.db.filter.QueryFilter;
import org.apache.cassandra.db.filter.SliceQueryFilter;
import org.apache.cassandra.db.filter.*;
import org.apache.cassandra.db.index.SecondaryIndex;
import org.apache.cassandra.db.index.SecondaryIndexManager;
import org.apache.cassandra.db.partitions.*;
import org.apache.cassandra.dht.*;
import org.apache.cassandra.dht.Range;
import org.apache.cassandra.exceptions.ConfigurationException;
@ -76,7 +71,6 @@ import org.apache.cassandra.metrics.ColumnFamilyMetrics.Sampler;
import org.apache.cassandra.service.CacheService;
import org.apache.cassandra.service.StorageService;
import org.apache.cassandra.streaming.StreamLockfile;
import org.apache.cassandra.tracing.Tracing;
import org.apache.cassandra.utils.*;
import org.apache.cassandra.utils.concurrent.*;
import org.apache.cassandra.utils.TopKSampler.SamplerResult;
@ -592,12 +586,9 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean
{
if (def.isIndexed())
{
CellNameType indexComparator = SecondaryIndex.getIndexComparator(metadata, def);
if (indexComparator != null)
{
CFMetaData indexMetadata = CFMetaData.newIndexMetadata(metadata, def, indexComparator);
CFMetaData indexMetadata = SecondaryIndex.newIndexMetadata(metadata, def);
if (indexMetadata != null)
scrubDataDirectories(indexMetadata);
}
}
}
}
@ -1220,7 +1211,7 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean
return;
RowCacheKey cacheKey = new RowCacheKey(metadata.cfId, key);
invalidateCachedRow(cacheKey);
invalidateCachedPartition(cacheKey);
}
/**
@ -1230,11 +1221,12 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean
* param @ key - key for update/insert
* param @ columnFamily - columnFamily changes
*/
public void apply(DecoratedKey key, ColumnFamily columnFamily, SecondaryIndexManager.Updater indexer, OpOrder.Group opGroup, ReplayPosition replayPosition)
public void apply(PartitionUpdate update, SecondaryIndexManager.Updater indexer, OpOrder.Group opGroup, ReplayPosition replayPosition)
{
long start = System.nanoTime();
Memtable mt = data.getMemtableFor(opGroup, replayPosition);
final long timeDelta = mt.put(key, columnFamily, indexer, opGroup);
long timeDelta = mt.put(update, indexer, opGroup);
DecoratedKey key = update.partitionKey();
maybeUpdateRowCache(key);
metric.samplers.get(Sampler.WRITES).addSample(key.getKey(), key.hashCode(), 1);
metric.writeLatency.addNano(System.nanoTime() - start);
@ -1242,93 +1234,6 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean
metric.colUpdateTimeDeltaHistogram.update(timeDelta);
}
/**
* Purges gc-able top-level and range tombstones, returning `cf` if there are any columns or tombstones left,
* null otherwise.
* @param gcBefore a timestamp (in seconds); tombstones with a localDeletionTime before this will be purged
*/
public static ColumnFamily removeDeletedCF(ColumnFamily cf, int gcBefore)
{
// purge old top-level and range tombstones
cf.purgeTombstones(gcBefore);
// if there are no columns or tombstones left, return null
return !cf.hasColumns() && !cf.isMarkedForDelete() ? null : cf;
}
/**
* Removes deleted columns and purges gc-able tombstones.
* @return an updated `cf` if any columns or tombstones remain, null otherwise
*/
public static ColumnFamily removeDeleted(ColumnFamily cf, int gcBefore)
{
return removeDeleted(cf, gcBefore, SecondaryIndexManager.nullUpdater);
}
/*
This is complicated because we need to preserve deleted columns and columnfamilies
until they have been deleted for at least GC_GRACE_IN_SECONDS. But, we do not need to preserve
their contents; just the object itself as a "tombstone" that can be used to repair other
replicas that do not know about the deletion.
*/
public static ColumnFamily removeDeleted(ColumnFamily cf, int gcBefore, SecondaryIndexManager.Updater indexer)
{
if (cf == null)
{
return null;
}
return removeDeletedCF(removeDeletedColumnsOnly(cf, gcBefore, indexer), gcBefore);
}
/**
* Removes only per-cell tombstones, cells that are shadowed by a row-level or range tombstone, or
* columns that have been dropped from the schema (for CQL3 tables only).
* @return the updated ColumnFamily
*/
public static ColumnFamily removeDeletedColumnsOnly(ColumnFamily cf, int gcBefore, SecondaryIndexManager.Updater indexer)
{
BatchRemoveIterator<Cell> iter = cf.batchRemoveIterator();
DeletionInfo.InOrderTester tester = cf.inOrderDeletionTester();
boolean hasDroppedColumns = !cf.metadata.getDroppedColumns().isEmpty();
while (iter.hasNext())
{
Cell c = iter.next();
// remove columns if
// (a) the column itself is gcable or
// (b) the column is shadowed by a CF tombstone
// (c) the column has been dropped from the CF schema (CQL3 tables only)
if (c.getLocalDeletionTime() < gcBefore || tester.isDeleted(c) || (hasDroppedColumns && isDroppedColumn(c, cf.metadata())))
{
iter.remove();
indexer.remove(c);
}
}
iter.commit();
return cf;
}
// returns true if
// 1. this column has been dropped from schema and
// 2. if it has been re-added since then, this particular column was inserted before the last drop
private static boolean isDroppedColumn(Cell c, CFMetaData meta)
{
Long droppedAt = meta.getDroppedColumns().get(c.name().cql3ColumnName(meta));
return droppedAt != null && c.timestamp() <= droppedAt;
}
private void removeDroppedColumns(ColumnFamily cf)
{
if (cf == null || cf.metadata.getDroppedColumns().isEmpty())
return;
BatchRemoveIterator<Cell> iter = cf.batchRemoveIterator();
while (iter.hasNext())
if (isDroppedColumn(iter.next(), metadata))
iter.remove();
iter.commit();
}
/**
* @param sstables
* @return sstables whose key range overlaps with that of the given sstables, not including itself.
@ -1348,7 +1253,7 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean
Set<SSTableReader> results = null;
for (SSTableReader sstable : sstables)
{
Set<SSTableReader> overlaps = ImmutableSet.copyOf(tree.search(Interval.<RowPosition, SSTableReader>create(sstable.first, sstable.last)));
Set<SSTableReader> overlaps = ImmutableSet.copyOf(tree.search(Interval.<PartitionPosition, SSTableReader>create(sstable.first, sstable.last)));
results = results == null ? overlaps : Sets.union(results, overlaps).immutableCopy();
}
results = Sets.difference(results, ImmutableSet.copyOf(sstables));
@ -1532,9 +1437,6 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean
return valid;
}
/**
* Package protected for access from the CompactionManager.
*/
@ -1553,249 +1455,30 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean
return data.getUncompacting();
}
public ColumnFamily getColumnFamily(DecoratedKey key,
Composite start,
Composite finish,
boolean reversed,
int limit,
long timestamp)
{
return getColumnFamily(QueryFilter.getSliceFilter(key, name, start, finish, reversed, limit, timestamp));
}
/**
* Fetch the row and columns given by filter.key if it is in the cache; if not, read it from disk and cache it
*
* If row is cached, and the filter given is within its bounds, we return from cache, otherwise from disk
*
* If row is not cached, we figure out what filter is "biggest", read that from disk, then
* filter the result and either cache that or return it.
*
* @param cfId the column family to read the row from
* @param filter the columns being queried.
* @return the requested data for the filter provided
*/
private ColumnFamily getThroughCache(UUID cfId, QueryFilter filter)
{
assert isRowCacheEnabled()
: String.format("Row cache is not enabled on table [" + name + "]");
RowCacheKey key = new RowCacheKey(cfId, filter.key);
// attempt a sentinel-read-cache sequence. if a write invalidates our sentinel, we'll return our
// (now potentially obsolete) data, but won't cache it. see CASSANDRA-3862
// TODO: don't evict entire rows on writes (#2864)
IRowCacheEntry cached = CacheService.instance.rowCache.get(key);
if (cached != null)
{
if (cached instanceof RowCacheSentinel)
{
// Some other read is trying to cache the value, just do a normal non-caching read
Tracing.trace("Row cache miss (race)");
metric.rowCacheMiss.inc();
return getTopLevelColumns(filter, Integer.MIN_VALUE);
}
ColumnFamily cachedCf = (ColumnFamily)cached;
if (isFilterFullyCoveredBy(filter.filter, cachedCf, filter.timestamp))
{
metric.rowCacheHit.inc();
Tracing.trace("Row cache hit");
return filterColumnFamily(cachedCf, filter);
}
metric.rowCacheHitOutOfRange.inc();
Tracing.trace("Ignoring row cache as cached value could not satisfy query");
return getTopLevelColumns(filter, Integer.MIN_VALUE);
}
metric.rowCacheMiss.inc();
Tracing.trace("Row cache miss");
RowCacheSentinel sentinel = new RowCacheSentinel();
boolean sentinelSuccess = CacheService.instance.rowCache.putIfAbsent(key, sentinel);
ColumnFamily data = null;
ColumnFamily toCache = null;
try
{
// If we are explicitely asked to fill the cache with full partitions, we go ahead and query the whole thing
if (metadata.getCaching().rowCache.cacheFullPartitions())
{
data = getTopLevelColumns(QueryFilter.getIdentityFilter(filter.key, name, filter.timestamp), Integer.MIN_VALUE);
toCache = data;
Tracing.trace("Populating row cache with the whole partition");
if (sentinelSuccess && toCache != null)
CacheService.instance.rowCache.replace(key, sentinel, toCache);
return filterColumnFamily(data, filter);
}
// Otherwise, if we want to cache the result of the query we're about to do, we must make sure this query
// covers what needs to be cached. And if the user filter does not satisfy that, we sometimes extend said
// filter so we can populate the cache but only if:
// 1) we can guarantee it is a strict extension, i.e. that we will still fetch the data asked by the user.
// 2) the extension does not make us query more than getRowsPerPartitionToCache() (as a mean to limit the
// amount of extra work we'll do on a user query for the purpose of populating the cache).
//
// In practice, we can only guarantee those 2 points if the filter is one that queries the head of the
// partition (and if that filter actually counts CQL3 rows since that's what we cache and it would be
// bogus to compare the filter count to the 'rows to cache' otherwise).
if (filter.filter.isHeadFilter() && filter.filter.countCQL3Rows(metadata.comparator))
{
SliceQueryFilter sliceFilter = (SliceQueryFilter)filter.filter;
int rowsToCache = metadata.getCaching().rowCache.rowsToCache;
SliceQueryFilter cacheSlice = readFilterForCache();
QueryFilter cacheFilter = new QueryFilter(filter.key, name, cacheSlice, filter.timestamp);
// If the filter count is less than the number of rows cached, we simply extend it to make sure we do cover the
// number of rows to cache, and if that count is greater than the number of rows to cache, we simply filter what
// needs to be cached afterwards.
if (sliceFilter.count < rowsToCache)
{
toCache = getTopLevelColumns(cacheFilter, Integer.MIN_VALUE);
if (toCache != null)
{
Tracing.trace("Populating row cache ({} rows cached)", cacheSlice.lastCounted());
data = filterColumnFamily(toCache, filter);
}
}
else
{
data = getTopLevelColumns(filter, Integer.MIN_VALUE);
if (data != null)
{
// The filter limit was greater than the number of rows to cache. But, if the filter had a non-empty
// finish bound, we may have gotten less than what needs to be cached, in which case we shouldn't cache it
// (otherwise a cache hit would assume the whole partition is cached which is not the case).
if (sliceFilter.finish().isEmpty() || sliceFilter.lastCounted() >= rowsToCache)
{
toCache = filterColumnFamily(data, cacheFilter);
Tracing.trace("Caching {} rows (out of {} requested)", cacheSlice.lastCounted(), sliceFilter.count);
}
else
{
Tracing.trace("Not populating row cache, not enough rows fetched ({} fetched but {} required for the cache)", sliceFilter.lastCounted(), rowsToCache);
}
}
}
if (sentinelSuccess && toCache != null)
CacheService.instance.rowCache.replace(key, sentinel, toCache);
return data;
}
else
{
Tracing.trace("Fetching data but not populating cache as query does not query from the start of the partition");
return getTopLevelColumns(filter, Integer.MIN_VALUE);
}
}
finally
{
if (sentinelSuccess && toCache == null)
invalidateCachedRow(key);
}
}
public SliceQueryFilter readFilterForCache()
{
// We create a new filter everytime before for now SliceQueryFilter is unfortunatly mutable.
return new SliceQueryFilter(ColumnSlice.ALL_COLUMNS_ARRAY, false, metadata.getCaching().rowCache.rowsToCache, metadata.clusteringColumns().size());
}
public boolean isFilterFullyCoveredBy(IDiskAtomFilter filter, ColumnFamily cachedCf, long now)
public boolean isFilterFullyCoveredBy(ClusteringIndexFilter filter, DataLimits limits, CachedPartition cached, int nowInSec)
{
// We can use the cached value only if we know that no data it doesn't contain could be covered
// by the query filter, that is if:
// 1) either the whole partition is cached
// 2) or we can ensure than any data the filter selects are in the cached partition
// 2) or we can ensure than any data the filter selects is in the cached partition
// When counting rows to decide if the whole row is cached, we should be careful with expiring
// columns: if we use a timestamp newer than the one that was used when populating the cache, we might
// end up deciding the whole partition is cached when it's really not (just some rows expired since the
// cf was cached). This is the reason for Integer.MIN_VALUE below.
boolean wholePartitionCached = cachedCf.liveCQL3RowCount(Integer.MIN_VALUE) < metadata.getCaching().rowCache.rowsToCache;
// We can guarantee that a partition is fully cached if the number of rows it contains is less than
// what we're caching. Wen doing that, we should be careful about expiring cells: we should count
// something expired that wasn't when the partition was cached, or we could decide that the whole
// partition is cached when it's not. This is why we use CachedPartition#cachedLiveRows.
if (cached.cachedLiveRows() < metadata.getCaching().rowCache.rowsToCache)
return true;
// Contrarily to the "wholePartitionCached" check above, we do want isFullyCoveredBy to take the
// timestamp of the query into account when dealing with expired columns. Otherwise, we could think
// the cached partition has enough live rows to satisfy the filter when it doesn't because some
// are now expired.
return wholePartitionCached || filter.isFullyCoveredBy(cachedCf, now);
// If the whole partition isn't cached, then we must guarantee that the filter cannot select data that
// is not in the cache. We can guarantee that if either the filter is a "head filter" and the cached
// partition has more live rows that queried (where live rows refers to the rows that are live now),
// or if we can prove that everything the filter selects is in the cached partition based on its content.
return (filter.isHeadFilter() && limits.hasEnoughLiveData(cached, nowInSec)) || filter.isFullyCoveredBy(cached);
}
public int gcBefore(long now)
public int gcBefore(int nowInSec)
{
return (int) (now / 1000) - metadata.getGcGraceSeconds();
}
/**
* get a list of columns starting from a given column, in a specified order.
* only the latest version of a column is returned.
* @return null if there is no data and no tombstones; otherwise a ColumnFamily
*/
public ColumnFamily getColumnFamily(QueryFilter filter)
{
assert name.equals(filter.getColumnFamilyName()) : filter.getColumnFamilyName();
ColumnFamily result = null;
long start = System.nanoTime();
try
{
int gcBefore = gcBefore(filter.timestamp);
if (isRowCacheEnabled())
{
assert !isIndex(); // CASSANDRA-5732
UUID cfId = metadata.cfId;
ColumnFamily cached = getThroughCache(cfId, filter);
if (cached == null)
{
logger.trace("cached row is empty");
return null;
}
result = cached;
}
else
{
ColumnFamily cf = getTopLevelColumns(filter, gcBefore);
if (cf == null)
return null;
result = removeDeletedCF(cf, gcBefore);
}
removeDroppedColumns(result);
if (filter.filter instanceof SliceQueryFilter)
{
// Log the number of tombstones scanned on single key queries
metric.tombstoneScannedHistogram.update(((SliceQueryFilter) filter.filter).lastTombstones());
metric.liveScannedHistogram.update(((SliceQueryFilter) filter.filter).lastLive());
}
}
finally
{
metric.readLatency.addNano(System.nanoTime() - start);
}
return result;
}
/**
* Filter a cached row, which will not be modified by the filter, but may be modified by throwing out
* tombstones that are no longer relevant.
* The returned column family won't be thread safe.
*/
ColumnFamily filterColumnFamily(ColumnFamily cached, QueryFilter filter)
{
if (cached == null)
return null;
ColumnFamily cf = cached.cloneMeShallow(ArrayBackedSortedColumns.factory, filter.filter.isReversed());
int gcBefore = gcBefore(filter.timestamp);
filter.collateOnDiskAtom(cf, filter.getIterator(cached), gcBefore);
return removeDeletedCF(cf, gcBefore);
return nowInSec - metadata.getGcGraceSeconds();
}
public Set<SSTableReader> getUnrepairedSSTables()
@ -1881,7 +1564,7 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean
* @return a ViewFragment containing the sstables and memtables that may need to be merged
* for rows within @param rowBounds, inclusive, according to the interval tree.
*/
public Function<View, List<SSTableReader>> viewFilter(final AbstractBounds<RowPosition> rowBounds)
public Function<View, List<SSTableReader>> viewFilter(final AbstractBounds<PartitionPosition> rowBounds)
{
return new Function<View, List<SSTableReader>>()
{
@ -1896,14 +1579,14 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean
* @return a ViewFragment containing the sstables and memtables that may need to be merged
* for rows for all of @param rowBoundsCollection, inclusive, according to the interval tree.
*/
public Function<View, List<SSTableReader>> viewFilter(final Collection<AbstractBounds<RowPosition>> rowBoundsCollection, final boolean includeRepaired)
public Function<View, List<SSTableReader>> viewFilter(final Collection<AbstractBounds<PartitionPosition>> rowBoundsCollection, final boolean includeRepaired)
{
return new Function<View, List<SSTableReader>>()
{
public List<SSTableReader> apply(View view)
{
Set<SSTableReader> sstables = Sets.newHashSet();
for (AbstractBounds<RowPosition> rowBounds : rowBoundsCollection)
for (AbstractBounds<PartitionPosition> rowBounds : rowBoundsCollection)
{
for (SSTableReader sstable : view.sstablesInBounds(rowBounds))
{
@ -1934,20 +1617,6 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean
}
}
public ColumnFamily getTopLevelColumns(QueryFilter filter, int gcBefore)
{
Tracing.trace("Executing single-partition query on {}", name);
CollationController controller = new CollationController(this, filter, gcBefore);
ColumnFamily columns;
try (OpOrder.Group op = readOrdering.start())
{
columns = controller.getTopLevelColumns(Memtable.MEMORY_POOL.needToCopyOnHeap());
}
if (columns != null)
metric.samplers.get(Sampler.READS).addSample(filter.key.getKey(), filter.key.hashCode(), 1);
metric.updateSSTableIterated(controller.getSstablesIterated());
return columns;
}
public void beginLocalSampling(String sampler, int capacity)
{
@ -1982,7 +1651,7 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean
RowCacheKey key = keyIter.next();
DecoratedKey dk = partitioner.decorateKey(ByteBuffer.wrap(key.key));
if (key.cfId.equals(metadata.cfId) && !Range.isInRanges(dk.getToken(), ranges))
invalidateCachedRow(dk);
invalidateCachedPartition(dk);
}
if (metadata.isCounter())
@ -1998,247 +1667,7 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean
}
}
public static abstract class AbstractScanIterator extends AbstractIterator<Row> implements CloseableIterator<Row>
{
public boolean needsFiltering()
{
return true;
}
}
/**
* Iterate over a range of rows and columns from memtables/sstables.
*
* @param range The range of keys and columns within those keys to fetch
*/
@SuppressWarnings("resource")
private AbstractScanIterator getSequentialIterator(final DataRange range, long now)
{
assert !(range.keyRange() instanceof Range) || !((Range<?>)range.keyRange()).isWrapAround() || range.keyRange().right.isMinimum() : range.keyRange();
final ViewFragment view = select(viewFilter(range.keyRange()));
Tracing.trace("Executing seq scan across {} sstables for {}", view.sstables.size(), range.keyRange().getString(metadata.getKeyValidator()));
final CloseableIterator<Row> iterator = RowIteratorFactory.getIterator(view.memtables, view.sstables, range, this, now);
// todo this could be pushed into SSTableScanner
return new AbstractScanIterator()
{
protected Row computeNext()
{
while (true)
{
// pull a row out of the iterator
if (!iterator.hasNext())
return endOfData();
Row current = iterator.next();
DecoratedKey key = current.key;
if (!range.stopKey().isMinimum() && range.stopKey().compareTo(key) < 0)
return endOfData();
// skipping outside of assigned range
if (!range.contains(key))
continue;
if (logger.isTraceEnabled())
logger.trace("scanned {}", metadata.getKeyValidator().getString(key.getKey()));
return current;
}
}
public void close() throws IOException
{
iterator.close();
}
};
}
@VisibleForTesting
public List<Row> getRangeSlice(final AbstractBounds<RowPosition> range,
List<IndexExpression> rowFilter,
IDiskAtomFilter columnFilter,
int maxResults)
{
return getRangeSlice(range, rowFilter, columnFilter, maxResults, System.currentTimeMillis());
}
public List<Row> getRangeSlice(final AbstractBounds<RowPosition> range,
List<IndexExpression> rowFilter,
IDiskAtomFilter columnFilter,
int maxResults,
long now)
{
return getRangeSlice(makeExtendedFilter(range, columnFilter, rowFilter, maxResults, false, false, now));
}
/**
* Allows generic range paging with the slice column filter.
* Typically, suppose we have rows A, B, C ... Z having each some columns in [1, 100].
* And suppose we want to page through the query that for all rows returns the columns
* within [25, 75]. For that, we need to be able to do a range slice starting at (row r, column c)
* and ending at (row Z, column 75), *but* that only return columns in [25, 75].
* That is what this method allows. The columnRange is the "window" of columns we are interested
* in each row, and columnStart (resp. columnEnd) is the start (resp. end) for the first
* (resp. last) requested row.
*/
public ExtendedFilter makeExtendedFilter(AbstractBounds<RowPosition> keyRange,
SliceQueryFilter columnRange,
Composite columnStart,
Composite columnStop,
List<IndexExpression> rowFilter,
int maxResults,
boolean countCQL3Rows,
long now)
{
DataRange dataRange = new DataRange.Paging(keyRange, columnRange, columnStart, columnStop, metadata);
return ExtendedFilter.create(this, dataRange, rowFilter, maxResults, countCQL3Rows, now);
}
public List<Row> getRangeSlice(AbstractBounds<RowPosition> range,
List<IndexExpression> rowFilter,
IDiskAtomFilter columnFilter,
int maxResults,
long now,
boolean countCQL3Rows,
boolean isPaging)
{
return getRangeSlice(makeExtendedFilter(range, columnFilter, rowFilter, maxResults, countCQL3Rows, isPaging, now));
}
public ExtendedFilter makeExtendedFilter(AbstractBounds<RowPosition> range,
IDiskAtomFilter columnFilter,
List<IndexExpression> rowFilter,
int maxResults,
boolean countCQL3Rows,
boolean isPaging,
long timestamp)
{
DataRange dataRange;
if (isPaging)
{
assert columnFilter instanceof SliceQueryFilter;
SliceQueryFilter sfilter = (SliceQueryFilter)columnFilter;
assert sfilter.slices.length == 1;
// create a new SliceQueryFilter that selects all cells, but pass the original slice start and finish
// through to DataRange.Paging to be used on the first and last partitions
SliceQueryFilter newFilter = new SliceQueryFilter(ColumnSlice.ALL_COLUMNS_ARRAY, sfilter.isReversed(), sfilter.count);
dataRange = new DataRange.Paging(range, newFilter, sfilter.start(), sfilter.finish(), metadata);
}
else
{
dataRange = new DataRange(range, columnFilter);
}
return ExtendedFilter.create(this, dataRange, rowFilter, maxResults, countCQL3Rows, timestamp);
}
public List<Row> getRangeSlice(ExtendedFilter filter)
{
long start = System.nanoTime();
try (OpOrder.Group op = readOrdering.start())
{
return filter(getSequentialIterator(filter.dataRange, filter.timestamp), filter);
}
finally
{
metric.rangeLatency.addNano(System.nanoTime() - start);
}
}
@VisibleForTesting
public List<Row> search(AbstractBounds<RowPosition> range,
List<IndexExpression> clause,
IDiskAtomFilter dataFilter,
int maxResults)
{
return search(range, clause, dataFilter, maxResults, System.currentTimeMillis());
}
public List<Row> search(AbstractBounds<RowPosition> range,
List<IndexExpression> clause,
IDiskAtomFilter dataFilter,
int maxResults,
long now)
{
return search(makeExtendedFilter(range, dataFilter, clause, maxResults, false, false, now));
}
public List<Row> search(ExtendedFilter filter)
{
Tracing.trace("Executing indexed scan for {}", filter.dataRange.keyRange().getString(metadata.getKeyValidator()));
return indexManager.search(filter);
}
public List<Row> filter(AbstractScanIterator rowIterator, ExtendedFilter filter)
{
logger.trace("Filtering {} for rows matching {}", rowIterator, filter);
List<Row> rows = new ArrayList<Row>();
int columnsCount = 0;
int total = 0, matched = 0;
boolean ignoreTombstonedPartitions = filter.ignoreTombstonedPartitions();
try
{
while (rowIterator.hasNext() && matched < filter.maxRows() && columnsCount < filter.maxColumns())
{
// get the raw columns requested, and additional columns for the expressions if necessary
Row rawRow = rowIterator.next();
total++;
ColumnFamily data = rawRow.cf;
if (rowIterator.needsFiltering())
{
IDiskAtomFilter extraFilter = filter.getExtraFilter(rawRow.key, data);
if (extraFilter != null)
{
ColumnFamily cf = filter.cfs.getColumnFamily(new QueryFilter(rawRow.key, name, extraFilter, filter.timestamp));
if (cf != null)
data.addAll(cf);
}
removeDroppedColumns(data);
if (!filter.isSatisfiedBy(rawRow.key, data, null, null))
continue;
logger.trace("{} satisfies all filter expressions", data);
// cut the resultset back to what was requested, if necessary
data = filter.prune(rawRow.key, data);
}
else
{
removeDroppedColumns(data);
}
rows.add(new Row(rawRow.key, data));
if (!ignoreTombstonedPartitions || !data.hasOnlyTombstones(filter.timestamp))
matched++;
if (data != null)
columnsCount += filter.lastCounted(data);
// Update the underlying filter to avoid querying more columns per slice than necessary and to handle paging
filter.updateFilter(columnsCount);
}
return rows;
}
finally
{
try
{
rowIterator.close();
Tracing.trace("Scanned {} rows and matched {}", total, matched);
}
catch (IOException e)
{
throw new RuntimeException(e);
}
}
}
public CellNameType getComparator()
public ClusteringComparator getComparator()
{
return metadata.comparator;
}
@ -2388,20 +1817,20 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean
}
/**
* @return the cached row for @param key if it is already present in the cache.
* That is, unlike getThroughCache, it will not readAndCache the row if it is not present, nor
* @return the cached partition for @param key if it is already present in the cache.
* Not that this will not readAndCache the parition if it is not present, nor
* are these calls counted in cache statistics.
*
* Note that this WILL cause deserialization of a SerializingCache row, so if all you
* need to know is whether a row is present or not, use containsCachedRow instead.
* Note that this WILL cause deserialization of a SerializingCache partition, so if all you
* need to know is whether a partition is present or not, use containsCachedParition instead.
*/
public ColumnFamily getRawCachedRow(DecoratedKey key)
public CachedPartition getRawCachedPartition(DecoratedKey key)
{
if (!isRowCacheEnabled())
return null;
IRowCacheEntry cached = CacheService.instance.rowCache.getInternal(new RowCacheKey(metadata.cfId, key));
return cached == null || cached instanceof RowCacheSentinel ? null : (ColumnFamily)cached;
return cached == null || cached instanceof RowCacheSentinel ? null : (CachedPartition)cached;
}
private void invalidateCaches()
@ -2415,37 +1844,37 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean
/**
* @return true if @param key is contained in the row cache
*/
public boolean containsCachedRow(DecoratedKey key)
public boolean containsCachedParition(DecoratedKey key)
{
return CacheService.instance.rowCache.getCapacity() != 0 && CacheService.instance.rowCache.containsKey(new RowCacheKey(metadata.cfId, key));
}
public void invalidateCachedRow(RowCacheKey key)
public void invalidateCachedPartition(RowCacheKey key)
{
CacheService.instance.rowCache.remove(key);
}
public void invalidateCachedRow(DecoratedKey key)
public void invalidateCachedPartition(DecoratedKey key)
{
UUID cfId = Schema.instance.getId(keyspace.getName(), this.name);
if (cfId == null)
return; // secondary index
invalidateCachedRow(new RowCacheKey(cfId, key));
invalidateCachedPartition(new RowCacheKey(cfId, key));
}
public ClockAndCount getCachedCounter(ByteBuffer partitionKey, CellName cellName)
public ClockAndCount getCachedCounter(ByteBuffer partitionKey, Clustering clustering, ColumnDefinition column, CellPath path)
{
if (CacheService.instance.counterCache.getCapacity() == 0L) // counter cache disabled.
return null;
return CacheService.instance.counterCache.get(CounterCacheKey.create(metadata.cfId, partitionKey, cellName));
return CacheService.instance.counterCache.get(CounterCacheKey.create(metadata.cfId, partitionKey, clustering, column, path));
}
public void putCachedCounter(ByteBuffer partitionKey, CellName cellName, ClockAndCount clockAndCount)
public void putCachedCounter(ByteBuffer partitionKey, Clustering clustering, ColumnDefinition column, CellPath path, ClockAndCount clockAndCount)
{
if (CacheService.instance.counterCache.getCapacity() == 0L) // counter cache disabled.
return;
CacheService.instance.counterCache.put(CounterCacheKey.create(metadata.cfId, partitionKey, cellName), clockAndCount);
CacheService.instance.counterCache.put(CounterCacheKey.create(metadata.cfId, partitionKey, clustering, column, path), clockAndCount);
}
public void forceMajorCompaction() throws InterruptedException, ExecutionException
@ -2830,7 +2259,7 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean
return view.sstables.isEmpty() && view.getCurrentMemtable().getOperations() == 0 && view.liveMemtables.size() <= 1 && view.flushingMemtables.size() == 0;
}
private boolean isRowCacheEnabled()
public boolean isRowCacheEnabled()
{
return metadata.getCaching().rowCache.isEnabled() && CacheService.instance.rowCache.getCapacity() > 0;
}

230
src/java/org/apache/cassandra/db/ColumnIndex.java
View File

@ -18,15 +18,15 @@
package org.apache.cassandra.db;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.util.*;
import com.google.common.annotations.VisibleForTesting;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.db.composites.Composite;
import org.apache.cassandra.db.rows.*;
import org.apache.cassandra.io.sstable.IndexHelper;
import org.apache.cassandra.io.util.DataOutputPlus;
import org.apache.cassandra.io.sstable.format.Version;
import org.apache.cassandra.io.util.SequentialWriter;
import org.apache.cassandra.utils.ByteBufferUtil;
public class ColumnIndex
@ -42,6 +42,14 @@ public class ColumnIndex
this.columnsIndex = columnsIndex;
}
public static ColumnIndex writeAndBuildIndex(UnfilteredRowIterator iterator, SequentialWriter output, SerializationHeader header, Version version) throws IOException
{
assert !iterator.isEmpty() && version.storeRows();
Builder builder = new Builder(iterator, output, header, version.correspondingMessagingVersion());
return builder.build();
}
@VisibleForTesting
public static ColumnIndex nothing()
{
@ -52,192 +60,114 @@ public class ColumnIndex
* Help to create an index for a column family based on size of columns,
* and write said columns to disk.
*/
public static class Builder
private static class Builder
{
private final UnfilteredRowIterator iterator;
private final SequentialWriter writer;
private final SerializationHeader header;
private final int version;
private final ColumnIndex result;
private final long indexOffset;
private final long initialPosition;
private long startPosition = -1;
private long endPosition = 0;
private long blockSize;
private OnDiskAtom firstColumn;
private OnDiskAtom lastColumn;
private OnDiskAtom lastBlockClosing;
private final DataOutputPlus output;
private final RangeTombstone.Tracker tombstoneTracker;
private int atomCount;
private final ByteBuffer key;
private final DeletionInfo deletionInfo; // only used for serializing and calculating row header size
private final OnDiskAtom.Serializer atomSerializer;
private int written;
public Builder(ColumnFamily cf,
ByteBuffer key,
DataOutputPlus output)
private ClusteringPrefix firstClustering;
private final ReusableClusteringPrefix lastClustering;
private DeletionTime openMarker;
public Builder(UnfilteredRowIterator iterator,
SequentialWriter writer,
SerializationHeader header,
int version)
{
assert cf != null;
assert key != null;
assert output != null;
this.iterator = iterator;
this.writer = writer;
this.header = header;
this.version = version;
this.key = key;
deletionInfo = cf.deletionInfo();
this.indexOffset = rowHeaderSize(key, deletionInfo);
this.result = new ColumnIndex(new ArrayList<IndexHelper.IndexInfo>());
this.output = output;
this.tombstoneTracker = new RangeTombstone.Tracker(cf.getComparator());
this.atomSerializer = cf.getComparator().onDiskAtomSerializer();
this.initialPosition = writer.getFilePointer();
this.lastClustering = new ReusableClusteringPrefix(iterator.metadata().clusteringColumns().size());
}
/**
* Returns the number of bytes between the beginning of the row and the
* first serialized column.
*/
private static long rowHeaderSize(ByteBuffer key, DeletionInfo delInfo)
private void writePartitionHeader(UnfilteredRowIterator iterator) throws IOException
{
TypeSizes typeSizes = TypeSizes.NATIVE;
// TODO fix constantSize when changing the nativeconststs.
int keysize = key.remaining();
return typeSizes.sizeof((short) keysize) + keysize // Row key
+ DeletionTime.serializer.serializedSize(delInfo.getTopLevelDeletion(), typeSizes);
ByteBufferUtil.writeWithShortLength(iterator.partitionKey().getKey(), writer.stream);
DeletionTime.serializer.serialize(iterator.partitionLevelDeletion(), writer.stream);
if (header.hasStatic())
UnfilteredSerializer.serializer.serialize(iterator.staticRow(), header, writer.stream, version);
}
public RangeTombstone.Tracker tombstoneTracker()
public ColumnIndex build() throws IOException
{
return tombstoneTracker;
writePartitionHeader(iterator);
while (iterator.hasNext())
add(iterator.next());
return close();
}
public int writtenAtomCount()
private long currentPosition()
{
return atomCount + tombstoneTracker.writtenAtom();
return writer.getFilePointer() - initialPosition;
}
/**
* Serializes the index into in-memory structure with all required components
* such as Bloom Filter, index block size, IndexInfo list
*
* @param cf Column family to create index for
*
* @return information about index - it's Bloom Filter, block size and IndexInfo list
*/
public ColumnIndex build(ColumnFamily cf) throws IOException
private void addIndexBlock()
{
// cf has disentangled the columns and range tombstones, we need to re-interleave them in comparator order
Comparator<Composite> comparator = cf.getComparator();
DeletionInfo.InOrderTester tester = cf.deletionInfo().inOrderTester();
Iterator<RangeTombstone> rangeIter = cf.deletionInfo().rangeIterator();
RangeTombstone tombstone = rangeIter.hasNext() ? rangeIter.next() : null;
IndexHelper.IndexInfo cIndexInfo = new IndexHelper.IndexInfo(firstClustering,
lastClustering.get().takeAlias(),
startPosition,
currentPosition() - startPosition,
openMarker);
result.columnsIndex.add(cIndexInfo);
firstClustering = null;
}
for (Cell c : cf)
private void add(Unfiltered unfiltered) throws IOException
{
lastClustering.copy(unfiltered.clustering());
boolean isMarker = unfiltered.kind() == Unfiltered.Kind.RANGE_TOMBSTONE_MARKER;
if (firstClustering == null)
{
while (tombstone != null && comparator.compare(c.name(), tombstone.min) >= 0)
{
// skip range tombstones that are shadowed by partition tombstones
if (!cf.deletionInfo().getTopLevelDeletion().isDeleted(tombstone))
add(tombstone);
tombstone = rangeIter.hasNext() ? rangeIter.next() : null;
}
// We can skip any cell if it's shadowed by a tombstone already. This is a more
// general case than was handled by CASSANDRA-2589.
if (!tester.isDeleted(c))
add(c);
// Beginning of an index block. Remember the start and position
firstClustering = lastClustering.get().takeAlias();
startPosition = currentPosition();
}
while (tombstone != null)
UnfilteredSerializer.serializer.serialize(unfiltered, header, writer.stream, version);
++written;
if (isMarker)
{
add(tombstone);
tombstone = rangeIter.hasNext() ? rangeIter.next() : null;
RangeTombstoneMarker marker = (RangeTombstoneMarker) unfiltered;
openMarker = marker.isOpen(false) ? marker.openDeletionTime(false) : null;
}
ColumnIndex index = build();
maybeWriteEmptyRowHeader();
return index;
}
/**
* The important distinction wrt build() is that we may be building for a row that ends up
* being compacted away entirely, i.e., the input consists only of expired tombstones (or
* columns shadowed by expired tombstone). Thus, it is the caller's responsibility
* to decide whether to write the header for an empty row.
*/
public ColumnIndex buildForCompaction(Iterator<OnDiskAtom> columns) throws IOException
{
while (columns.hasNext())
{
OnDiskAtom c = columns.next();
add(c);
}
return build();
}
public void add(OnDiskAtom column) throws IOException
{
atomCount++;
if (firstColumn == null)
{
firstColumn = column;
startPosition = endPosition;
// TODO: have that use the firstColumn as min + make sure we optimize that on read
endPosition += tombstoneTracker.writeOpenedMarker(firstColumn, output, atomSerializer);
blockSize = 0; // We don't count repeated tombstone marker in the block size, to avoid a situation
// where we wouldn't make any progress because a block is filled by said marker
}
long size = atomSerializer.serializedSizeForSSTable(column);
endPosition += size;
blockSize += size;
// if we hit the column index size that we have to index after, go ahead and index it.
if (blockSize >= DatabaseDescriptor.getColumnIndexSize())
{
IndexHelper.IndexInfo cIndexInfo = new IndexHelper.IndexInfo(firstColumn.name(), column.name(), indexOffset + startPosition, endPosition - startPosition);
result.columnsIndex.add(cIndexInfo);
firstColumn = null;
lastBlockClosing = column;
}
maybeWriteRowHeader();
atomSerializer.serializeForSSTable(column, output);
// TODO: Should deal with removing unneeded tombstones
tombstoneTracker.update(column, false);
lastColumn = column;
if (currentPosition() - startPosition >= DatabaseDescriptor.getColumnIndexSize())
addIndexBlock();
}
private void maybeWriteRowHeader() throws IOException
private ColumnIndex close() throws IOException
{
if (lastColumn == null)
{
ByteBufferUtil.writeWithShortLength(key, output);
DeletionTime.serializer.serialize(deletionInfo.getTopLevelDeletion(), output);
}
}
UnfilteredSerializer.serializer.writeEndOfPartition(writer.stream);
public ColumnIndex build()
{
// all columns were GC'd after all
if (lastColumn == null)
// It's possible we add no rows, just a top level deletion
if (written == 0)
return ColumnIndex.EMPTY;
// the last column may have fallen on an index boundary already. if not, index it explicitly.
if (result.columnsIndex.isEmpty() || lastBlockClosing != lastColumn)
{
IndexHelper.IndexInfo cIndexInfo = new IndexHelper.IndexInfo(firstColumn.name(), lastColumn.name(), indexOffset + startPosition, endPosition - startPosition);
result.columnsIndex.add(cIndexInfo);
}
if (firstClustering != null)
addIndexBlock();
// we should always have at least one computed index block, but we only write it out if there is more than that.
assert result.columnsIndex.size() > 0;
return result;
}
public void maybeWriteEmptyRowHeader() throws IOException
{
if (!deletionInfo.isLive())
maybeWriteRowHeader();
}
}
}

188
src/java/org/apache/cassandra/db/ColumnSerializer.java
View File

@ -1,188 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.db;
import java.io.DataInput;
import java.io.IOException;
import java.nio.ByteBuffer;
import org.apache.cassandra.db.composites.CellName;
import org.apache.cassandra.db.composites.CellNameType;
import org.apache.cassandra.io.ISerializer;
import org.apache.cassandra.io.FSReadError;
import org.apache.cassandra.io.util.DataOutputPlus;
import org.apache.cassandra.io.util.FileDataInput;
import org.apache.cassandra.io.util.FileUtils;
import org.apache.cassandra.utils.ByteBufferUtil;
public class ColumnSerializer implements ISerializer<Cell>
{
public final static int DELETION_MASK = 0x01;
public final static int EXPIRATION_MASK = 0x02;
public final static int COUNTER_MASK = 0x04;
public final static int COUNTER_UPDATE_MASK = 0x08;
public final static int RANGE_TOMBSTONE_MASK = 0x10;
/**
* Flag affecting deserialization behavior.
* - LOCAL: for deserialization of local data (Expired columns are
* converted to tombstones (to gain disk space)).
* - FROM_REMOTE: for deserialization of data received from remote hosts
* (Expired columns are converted to tombstone and counters have
* their delta cleared)
* - PRESERVE_SIZE: used when no transformation must be performed, i.e,
* when we must ensure that deserializing and reserializing the
* result yield the exact same bytes. Streaming uses this.
*/
public static enum Flag
{
LOCAL, FROM_REMOTE, PRESERVE_SIZE;
}
private final CellNameType type;
public ColumnSerializer(CellNameType type)
{
this.type = type;
}
public void serialize(Cell cell, DataOutputPlus out) throws IOException
{
assert !cell.name().isEmpty();
type.cellSerializer().serialize(cell.name(), out);
try
{
out.writeByte(cell.serializationFlags());
if (cell instanceof CounterCell)
{
out.writeLong(((CounterCell) cell).timestampOfLastDelete());
}
else if (cell instanceof ExpiringCell)
{
out.writeInt(((ExpiringCell) cell).getTimeToLive());
out.writeInt(cell.getLocalDeletionTime());
}
out.writeLong(cell.timestamp());
ByteBufferUtil.writeWithLength(cell.value(), out);
}
catch (IOException e)
{
throw new RuntimeException(e);
}
}
public Cell deserialize(DataInput in) throws IOException
{
return deserialize(in, Flag.LOCAL);
}
/*
* For counter columns, we must know when we deserialize them if what we
* deserialize comes from a remote host. If it does, then we must clear
* the delta.
*/
public Cell deserialize(DataInput in, ColumnSerializer.Flag flag) throws IOException
{
return deserialize(in, flag, Integer.MIN_VALUE);
}
public Cell deserialize(DataInput in, ColumnSerializer.Flag flag, int expireBefore) throws IOException
{
CellName name = type.cellSerializer().deserialize(in);
int b = in.readUnsignedByte();
return deserializeColumnBody(in, name, b, flag, expireBefore);
}
Cell deserializeColumnBody(DataInput in, CellName name, int mask, ColumnSerializer.Flag flag, int expireBefore) throws IOException
{
if ((mask & COUNTER_MASK) != 0)
{
long timestampOfLastDelete = in.readLong();
long ts = in.readLong();
ByteBuffer value = ByteBufferUtil.readWithLength(in);
return BufferCounterCell.create(name, value, ts, timestampOfLastDelete, flag);
}
else if ((mask & EXPIRATION_MASK) != 0)
{
int ttl = in.readInt();
int expiration = in.readInt();
long ts = in.readLong();
ByteBuffer value = ByteBufferUtil.readWithLength(in);
return BufferExpiringCell.create(name, value, ts, ttl, expiration, expireBefore, flag);
}
else
{
long ts = in.readLong();
ByteBuffer value = ByteBufferUtil.readWithLength(in);
return (mask & COUNTER_UPDATE_MASK) != 0
? new BufferCounterUpdateCell(name, value, ts)
: ((mask & DELETION_MASK) == 0
? new BufferCell(name, value, ts)
: new BufferDeletedCell(name, value, ts));
}
}
void skipColumnBody(DataInput in, int mask) throws IOException
{
if ((mask & COUNTER_MASK) != 0)
FileUtils.skipBytesFully(in, 16);
else if ((mask & EXPIRATION_MASK) != 0)
FileUtils.skipBytesFully(in, 16);
else
FileUtils.skipBytesFully(in, 8);
int length = in.readInt();
FileUtils.skipBytesFully(in, length);
}
public long serializedSize(Cell cell, TypeSizes typeSizes)
{
return cell.serializedSize(type, typeSizes);
}
public static class CorruptColumnException extends IOException
{
public CorruptColumnException(String s)
{
super(s);
}
public static CorruptColumnException create(DataInput in, ByteBuffer name)
{
assert name.remaining() <= 0;
String format = "invalid column name length %d%s";
String details = "";
if (in instanceof FileDataInput)
{
FileDataInput fdis = (FileDataInput)in;
long remaining;
try
{
remaining = fdis.bytesRemaining();
}
catch (IOException e)
{
throw new FSReadError(e, fdis.getPath());
}
details = String.format(" (%s, %d bytes remaining)", fdis.getPath(), remaining);
}
return new CorruptColumnException(String.format(format, name.remaining(), details));
}
}
}

535
src/java/org/apache/cassandra/db/Columns.java Normal file
View File

@ -0,0 +1,535 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.db;
import java.io.DataInput;
import java.io.IOException;
import java.util.*;
import java.nio.ByteBuffer;
import java.security.MessageDigest;
import com.google.common.collect.AbstractIterator;
import com.google.common.collect.Iterators;
import org.apache.cassandra.config.CFMetaData;
import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.db.marshal.UTF8Type;
import org.apache.cassandra.db.marshal.MapType;
import org.apache.cassandra.io.util.DataOutputPlus;
import org.apache.cassandra.utils.ByteBufferUtil;
/**
* An immutable and sorted list of (non-PK) columns for a given table.
* <p>
* Note that in practice, it will either store only static columns, or only regular ones. When
* we need both type of columns, we use a {@link PartitionColumns} object.
*/
public class Columns implements Iterable<ColumnDefinition>
{
public static final Serializer serializer = new Serializer();
public static final Columns NONE = new Columns(new ColumnDefinition[0], 0);
public final ColumnDefinition[] columns;
public final int complexIdx; // Index of the first complex column
private Columns(ColumnDefinition[] columns, int complexIdx)
{
assert complexIdx <= columns.length;
this.columns = columns;
this.complexIdx = complexIdx;
}
private Columns(ColumnDefinition[] columns)
{
this(columns, findFirstComplexIdx(columns));
}
/**
* Creates a {@code Columns} holding only the one column provided.
*
* @param c the column for which to create a {@code Columns} object.
*
* @return the newly created {@code Columns} containing only {@code c}.
*/
public static Columns of(ColumnDefinition c)
{
ColumnDefinition[] columns = new ColumnDefinition[]{ c };
return new Columns(columns, c.isComplex() ? 0 : 1);
}
/**
* Returns a new {@code Columns} object holing the same columns than the provided set.
*
* @param param s the set from which to create the new {@code Columns}.
*
* @return the newly created {@code Columns} containing the columns from {@code s}.
*/
public static Columns from(Set<ColumnDefinition> s)
{
ColumnDefinition[] columns = s.toArray(new ColumnDefinition[s.size()]);
Arrays.sort(columns);
return new Columns(columns, findFirstComplexIdx(columns));
}
private static int findFirstComplexIdx(ColumnDefinition[] columns)
{
for (int i = 0; i < columns.length; i++)
if (columns[i].isComplex())
return i;
return columns.length;
}
/**
* Whether this columns is empty.
*
* @return whether this columns is empty.
*/
public boolean isEmpty()
{
return columns.length == 0;
}
/**
* The number of simple columns in this object.
*
* @return the number of simple columns in this object.
*/
public int simpleColumnCount()
{
return complexIdx;
}
/**
* The number of complex columns (non-frozen collections, udts, ...) in this object.
*
* @return the number of complex columns in this object.
*/
public int complexColumnCount()
{
return columns.length - complexIdx;
}
/**
* The total number of columns in this object.
*
* @return the total number of columns in this object.
*/
public int columnCount()
{
return columns.length;
}
/**
* Whether this objects contains simple columns.
*
* @return whether this objects contains simple columns.
*/
public boolean hasSimple()
{
return complexIdx > 0;
}
/**
* Whether this objects contains complex columns.
*
* @return whether this objects contains complex columns.
*/
public boolean hasComplex()
{
return complexIdx < columns.length;
}
/**
* Returns the ith simple column of this object.
*
* @param i the index for the simple column to fectch. This must
* satisfy {@code 0 <= i < simpleColumnCount()}.
*
* @return the {@code i}th simple column in this object.
*/
public ColumnDefinition getSimple(int i)
{
return columns[i];
}
/**
* Returns the ith complex column of this object.
*
* @param i the index for the complex column to fectch. This must
* satisfy {@code 0 <= i < complexColumnCount()}.
*
* @return the {@code i}th complex column in this object.
*/
public ColumnDefinition getComplex(int i)
{
return columns[complexIdx + i];
}
/**
* The index of the provided simple column in this object (if it contains
* the provided column).
*
* @param c the simple column for which to return the index of.
* @param from the index to start the search from.
*
* @return the index for simple column {@code c} if it is contains in this
* object (starting from index {@code from}), {@code -1} otherwise.
*/
public int simpleIdx(ColumnDefinition c, int from)
{
assert !c.isComplex();
for (int i = from; i < complexIdx; i++)
// We know we only use "interned" ColumnIdentifier so == is ok.
if (columns[i].name == c.name)
return i;
return -1;
}
/**
* The index of the provided complex column in this object (if it contains
* the provided column).
*
* @param c the complex column for which to return the index of.
* @param from the index to start the search from.
*
* @return the index for complex column {@code c} if it is contains in this
* object (starting from index {@code from}), {@code -1} otherwise.
*/
public int complexIdx(ColumnDefinition c, int from)
{
assert c.isComplex();
for (int i = complexIdx + from; i < columns.length; i++)
// We know we only use "interned" ColumnIdentifier so == is ok.
if (columns[i].name == c.name)
return i - complexIdx;
return -1;
}
/**
* Whether the provided column is contained by this object.
*
* @param c the column to check presence of.
*
* @return whether {@code c} is contained by this object.
*/
public boolean contains(ColumnDefinition c)
{
return c.isComplex() ? complexIdx(c, 0) >= 0 : simpleIdx(c, 0) >= 0;
}
/**
* Whether or not there is some counter columns within those columns.
*
* @return whether or not there is some counter columns within those columns.
*/
public boolean hasCounters()
{
for (int i = 0; i < complexIdx; i++)
{
if (columns[i].type.isCounter())
return true;
}
for (int i = complexIdx; i < columns.length; i++)
{
// We only support counter in maps because that's all we need for now (and we need it for the sake of thrift super columns of counter)
if (columns[i].type instanceof MapType && (((MapType)columns[i].type).valueComparator().isCounter()))
return true;
}
return false;
}
/**
* Returns the result of merging this {@code Columns} object with the
* provided one.
*
* @param other the other {@code Columns} to merge this object with.
*
* @return the result of merging/taking the union of {@code this} and
* {@code other}. The returned object may be one of the operand and that
* operand is a subset of the other operand.
*/
public Columns mergeTo(Columns other)
{
if (this == other || other == NONE)
return this;
if (this == NONE)
return other;
int i = 0, j = 0;
int size = 0;
while (i < columns.length && j < other.columns.length)
{
++size;
int cmp = columns[i].compareTo(other.columns[j]);
if (cmp == 0)
{
++i;
++j;
}
else if (cmp < 0)
{
++i;
}
else
{
++j;
}
}
// If every element was always counted on both array, we have the same
// arrays for the first min elements
if (i == size && j == size)
{
// We've exited because of either c1 or c2 (or both). The array that
// made us stop is thus a subset of the 2nd one, return that array.
return i == columns.length ? other : this;
}
size += i == columns.length ? other.columns.length - j : columns.length - i;
ColumnDefinition[] result = new ColumnDefinition[size];
i = 0;
j = 0;
for (int k = 0; k < size; k++)
{
int cmp = i >= columns.length ? 1
: (j >= other.columns.length ? -1 : columns[i].compareTo(other.columns[j]));
if (cmp == 0)
{
result[k] = columns[i];
++i;
++j;
}
else if (cmp < 0)
{
result[k] = columns[i++];
}
else
{
result[k] = other.columns[j++];
}
}
return new Columns(result, findFirstComplexIdx(result));
}
/**
* Whether this object is a subset of the provided other {@code Columns object}.
*
* @param other the othere object to test for inclusion in this object.
*
* @return whether all the columns of {@code other} are contained by this object.
*/
public boolean contains(Columns other)
{
if (other.columns.length > columns.length)
return false;
int j = 0;
int cmp = 0;
for (ColumnDefinition def : other.columns)
{
while (j < columns.length && (cmp = columns[j].compareTo(def)) < 0)
j++;
if (j >= columns.length || cmp > 0)
return false;
// cmp == 0, we've found the definition. Ce can bump j once more since
// we know we won't need to compare that element again
j++;
}
return true;
}
/**
* Iterator over the simple columns of this object.
*
* @return an iterator over the simple columns of this object.
*/
public Iterator<ColumnDefinition> simpleColumns()
{
return new ColumnIterator(0, complexIdx);
}
/**
* Iterator over the complex columns of this object.
*
* @return an iterator over the complex columns of this object.
*/
public Iterator<ColumnDefinition> complexColumns()
{
return new ColumnIterator(complexIdx, columns.length);
}
/**
* Iterator over all the columns of this object.
*
* @return an iterator over all the columns of this object.
*/
public Iterator<ColumnDefinition> iterator()
{
return Iterators.forArray(columns);
}
/**
* An iterator that returns the columns of this object in "select" order (that
* is in global alphabetical order, where the "normal" iterator returns simple
* columns first and the complex second).
*
* @return an iterator returning columns in alphabetical order.
*/
public Iterator<ColumnDefinition> selectOrderIterator()
{
// In wildcard selection, we want to return all columns in alphabetical order,
// irregarding of whether they are complex or not
return new AbstractIterator<ColumnDefinition>()
{
private int regular;
private int complex = complexIdx;
protected ColumnDefinition computeNext()
{
if (complex >= columns.length)
return regular >= complexIdx ? endOfData() : columns[regular++];
if (regular >= complexIdx)
return columns[complex++];
return ByteBufferUtil.compareUnsigned(columns[regular].name.bytes, columns[complex].name.bytes) < 0
? columns[regular++]
: columns[complex++];
}
};
}
/**
* Returns the equivalent of those columns but with the provided column removed.
*
* @param column the column to remove.
*
* @return newly allocated columns containing all the columns of {@code this} expect
* for {@code column}.
*/
public Columns without(ColumnDefinition column)
{
int idx = column.isComplex() ? complexIdx(column, 0) : simpleIdx(column, 0);
if (idx < 0)
return this;
int realIdx = column.isComplex() ? complexIdx + idx : idx;
ColumnDefinition[] newColumns = new ColumnDefinition[columns.length - 1];
System.arraycopy(columns, 0, newColumns, 0, realIdx);
System.arraycopy(columns, realIdx + 1, newColumns, realIdx, newColumns.length - realIdx);
return new Columns(newColumns);
}
public void digest(MessageDigest digest)
{
for (ColumnDefinition c : this)
digest.update(c.name.bytes.duplicate());
}
@Override
public boolean equals(Object other)
{
if (!(other instanceof Columns))
return false;
Columns that = (Columns)other;
return this.complexIdx == that.complexIdx && Arrays.equals(this.columns, that.columns);
}
@Override
public int hashCode()
{
return Objects.hash(complexIdx, Arrays.hashCode(columns));
}
@Override
public String toString()
{
StringBuilder sb = new StringBuilder();
boolean first = true;
for (ColumnDefinition def : this)
{
if (first) first = false; else sb.append(" ");
sb.append(def.name);
}
return sb.toString();
}
private class ColumnIterator extends AbstractIterator<ColumnDefinition>
{
private final int to;
private int idx;
private ColumnIterator(int from, int to)
{
this.idx = from;
this.to = to;
}
protected ColumnDefinition computeNext()
{
if (idx >= to)
return endOfData();
return columns[idx++];
}
}
public static class Serializer
{
public void serialize(Columns columns, DataOutputPlus out) throws IOException
{
out.writeShort(columns.columnCount());
for (ColumnDefinition column : columns)
ByteBufferUtil.writeWithShortLength(column.name.bytes, out);
}
public long serializedSize(Columns columns, TypeSizes sizes)
{
long size = sizes.sizeof((short)columns.columnCount());
for (ColumnDefinition column : columns)
size += sizes.sizeofWithShortLength(column.name.bytes);
return size;
}
public Columns deserialize(DataInput in, CFMetaData metadata) throws IOException
{
int length = in.readUnsignedShort();
ColumnDefinition[] columns = new ColumnDefinition[length];
for (int i = 0; i < length; i++)
{
ByteBuffer name = ByteBufferUtil.readWithShortLength(in);
ColumnDefinition column = metadata.getColumnDefinition(name);
if (column == null)
{
// If we don't find the definition, it could be we have data for a dropped column, and we shouldn't
// fail deserialization because of that. So we grab a "fake" ColumnDefinition that ensure proper
// deserialization. The column will be ignore later on anyway.
column = metadata.getDroppedColumnDefinition(name);
if (column == null)
throw new RuntimeException("Unknown column " + UTF8Type.instance.getString(name) + " during deserialization");
}
columns[i] = column;
}
return new Columns(columns);
}
}
}

176
src/java/org/apache/cassandra/db/CompactTables.java Normal file
View File

@ -0,0 +1,176 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.db;
import java.nio.ByteBuffer;
import java.util.*;
import org.apache.cassandra.config.CFMetaData;
import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.db.marshal.*;
import org.apache.cassandra.utils.ByteBufferUtil;
/**
* Small utility methods pertaining to the encoding of COMPACT STORAGE tables.
*
* COMPACT STORAGE tables exists mainly for the sake of encoding internally thrift tables (as well as
* exposing those tables through CQL). Note that due to these constraints, the internal representation
* of compact tables does *not* correspond exactly to their CQL definition.
*
* The internal layout of such tables is such that it can encode any thrift table. That layout is as follow:
* CREATE TABLE compact (
* key [key_validation_class],
* [column_metadata_1] [type1] static,
* ...,
* [column_metadata_n] [type1] static,
* column [comparator],
* value [default_validation_class]
* PRIMARY KEY (key, column)
* )
* More specifically, the table:
* - always has a clustering column and a regular value, which are used to store the "dynamic" thrift columns name and value.
* Those are always present because we have no way to know in advance if "dynamic" columns will be inserted or not. Note
* that when declared from CQL, compact tables may not have any clustering: in that case, we still have a clustering
* defined internally, it is just ignored as far as interacting from CQL is concerned.
* - have a static column for every "static" column defined in the thrift "column_metadata". Note that when declaring a compact
* table from CQL without any clustering (but some non-PK columns), the columns ends up static internally even though they are
* not in the declaration
*
* On variation is that if the table comparator is a CompositeType, then the underlying table will have one clustering column by
* element of the CompositeType, but the rest of the layout is as above.
*
* As far as thrift is concerned, one exception to this is super column families, which have a different layout. Namely, a super
* column families is encoded with:
* CREATE TABLE super (
* key [key_validation_class],
* super_column_name [comparator],
* [column_metadata_1] [type1],
* ...,
* [column_metadata_n] [type1],
* "" map<[sub_comparator], [default_validation_class]>
* PRIMARY KEY (key, super_column_name)
* )
* In other words, every super column is encoded by a row. That row has one column for each defined "column_metadata", but it also
* has a special map column (whose name is the empty string as this is guaranteed to never conflict with a user-defined
* "column_metadata") which stores the super column "dynamic" sub-columns.
*/
public abstract class CompactTables
{
// We use an empty value for the 1) this can't conflict with a user-defined column and 2) this actually
// validate with any comparator which makes it convenient for columnDefinitionComparator().
public static final ByteBuffer SUPER_COLUMN_MAP_COLUMN = ByteBufferUtil.EMPTY_BYTE_BUFFER;
public static final String SUPER_COLUMN_MAP_COLUMN_STR = UTF8Type.instance.compose(SUPER_COLUMN_MAP_COLUMN);
private CompactTables() {}
public static ColumnDefinition getCompactValueColumn(PartitionColumns columns, boolean isSuper)
{
if (isSuper)
{
for (ColumnDefinition column : columns.regulars)
if (column.name.bytes.equals(SUPER_COLUMN_MAP_COLUMN))
return column;
throw new AssertionError("Invalid super column table definition, no 'dynamic' map column");
}
assert columns.regulars.simpleColumnCount() == 1 && columns.regulars.complexColumnCount() == 0;
return columns.regulars.getSimple(0);
}
public static AbstractType<?> columnDefinitionComparator(ColumnDefinition.Kind kind, boolean isSuper, AbstractType<?> rawComparator, AbstractType<?> rawSubComparator)
{
if (isSuper)
return kind == ColumnDefinition.Kind.REGULAR ? rawSubComparator : UTF8Type.instance;
else
return kind == ColumnDefinition.Kind.STATIC ? rawComparator : UTF8Type.instance;
}
public static boolean hasEmptyCompactValue(CFMetaData metadata)
{
return metadata.compactValueColumn().type instanceof EmptyType;
}
public static boolean isSuperColumnMapColumn(ColumnDefinition column)
{
return column.kind == ColumnDefinition.Kind.REGULAR && column.name.bytes.equals(SUPER_COLUMN_MAP_COLUMN);
}
public static DefaultNames defaultNameGenerator(Set<String> usedNames)
{
return new DefaultNames(new HashSet<String>(usedNames));
}
public static DefaultNames defaultNameGenerator(Iterable<ColumnDefinition> defs)
{
Set<String> usedNames = new HashSet<>();
for (ColumnDefinition def : defs)
usedNames.add(def.name.toString());
return new DefaultNames(usedNames);
}
public static class DefaultNames
{
private static final String DEFAULT_PARTITION_KEY_NAME = "key";
private static final String DEFAULT_CLUSTERING_NAME = "column";
private static final String DEFAULT_COMPACT_VALUE_NAME = "value";
private final Set<String> usedNames;
private int partitionIndex = 0;
private int clusteringIndex = 1;
private int compactIndex = 0;
private DefaultNames(Set<String> usedNames)
{
this.usedNames = usedNames;
}
public String defaultPartitionKeyName()
{
while (true)
{
// For compatibility sake, we call the first alias 'key' rather than 'key1'. This
// is inconsistent with column alias, but it's probably not worth risking breaking compatibility now.
String candidate = partitionIndex == 0 ? DEFAULT_PARTITION_KEY_NAME : DEFAULT_PARTITION_KEY_NAME + (partitionIndex + 1);
++partitionIndex;
if (usedNames.add(candidate))
return candidate;
}
}
public String defaultClusteringName()
{
while (true)
{
String candidate = DEFAULT_CLUSTERING_NAME + clusteringIndex;
++clusteringIndex;
if (usedNames.add(candidate))
return candidate;
}
}
public String defaultCompactValueName()
{
while (true)
{
String candidate = compactIndex == 0 ? DEFAULT_COMPACT_VALUE_NAME : DEFAULT_COMPACT_VALUE_NAME + compactIndex;
++compactIndex;
if (usedNames.add(candidate))
return candidate;
}
}
}
}

79
src/java/org/apache/cassandra/db/Conflicts.java Normal file
View File

@ -0,0 +1,79 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.db;
import java.nio.ByteBuffer;
import org.apache.cassandra.db.context.CounterContext;
public abstract class Conflicts
{
private Conflicts() {}
public enum Resolution { LEFT_WINS, MERGE, RIGHT_WINS };
public static Resolution resolveRegular(long leftTimestamp,
boolean leftLive,
int leftLocalDeletionTime,
ByteBuffer leftValue,
long rightTimestamp,
boolean rightLive,
int rightLocalDeletionTime,
ByteBuffer rightValue)
{
if (leftTimestamp != rightTimestamp)
return leftTimestamp < rightTimestamp ? Resolution.RIGHT_WINS : Resolution.LEFT_WINS;
if (leftLive != rightLive)
return leftLive ? Resolution.RIGHT_WINS : Resolution.LEFT_WINS;
int c = leftValue.compareTo(rightValue);
if (c < 0)
return Resolution.RIGHT_WINS;
else if (c > 0)
return Resolution.LEFT_WINS;
// Prefer the longest ttl if relevant
return leftLocalDeletionTime < rightLocalDeletionTime ? Resolution.RIGHT_WINS : Resolution.LEFT_WINS;
}
public static Resolution resolveCounter(long leftTimestamp,
boolean leftLive,
ByteBuffer leftValue,
long rightTimestamp,
boolean rightLive,
ByteBuffer rightValue)
{
// No matter what the counter cell's timestamp is, a tombstone always takes precedence. See CASSANDRA-7346.
if (!leftLive)
// left is a tombstone: it has precedence over right if either right is not a tombstone, or left has a greater timestamp
return rightLive || leftTimestamp > rightTimestamp ? Resolution.LEFT_WINS : Resolution.RIGHT_WINS;
// If right is a tombstone, since left isn't one, it has precedence
if (!rightLive)
return Resolution.RIGHT_WINS;
return Resolution.MERGE;
}
public static ByteBuffer mergeCounterValues(ByteBuffer left, ByteBuffer right)
{
return CounterContext.instance().merge(left, right);
}
}

44
src/java/org/apache/cassandra/db/CounterCell.java
View File

@ -1,44 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.db;
import org.apache.cassandra.config.CFMetaData;
import org.apache.cassandra.db.context.CounterContext;
import org.apache.cassandra.utils.concurrent.OpOrder;
import org.apache.cassandra.utils.memory.AbstractAllocator;
import org.apache.cassandra.utils.memory.MemtableAllocator;
/**
* A column that represents a partitioned counter.
*/
public interface CounterCell extends Cell
{
static final CounterContext contextManager = CounterContext.instance();
public long timestampOfLastDelete();
public long total();
public boolean hasLegacyShards();
public Cell markLocalToBeCleared();
CounterCell localCopy(CFMetaData metadata, AbstractAllocator allocator);
CounterCell localCopy(CFMetaData metaData, MemtableAllocator allocator, OpOrder.Group opGroup);
}

225
src/java/org/apache/cassandra/db/CounterMutation.java
View File

@ -19,7 +19,6 @@ package org.apache.cassandra.db;
import java.io.DataInput;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.util.*;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.Lock;
@ -27,12 +26,15 @@ import java.util.concurrent.locks.Lock;
import com.google.common.base.Function;
import com.google.common.base.Objects;
import com.google.common.collect.Iterables;
import com.google.common.collect.Iterators;
import com.google.common.collect.PeekingIterator;
import com.google.common.util.concurrent.Striped;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.db.composites.CellName;
import org.apache.cassandra.db.rows.*;
import org.apache.cassandra.db.filter.*;
import org.apache.cassandra.db.partitions.*;
import org.apache.cassandra.db.context.CounterContext;
import org.apache.cassandra.db.filter.NamesQueryFilter;
import org.apache.cassandra.exceptions.WriteTimeoutException;
import org.apache.cassandra.io.IVersionedSerializer;
import org.apache.cassandra.io.util.DataOutputPlus;
@ -41,6 +43,7 @@ import org.apache.cassandra.net.MessagingService;
import org.apache.cassandra.service.CacheService;
import org.apache.cassandra.tracing.Tracing;
import org.apache.cassandra.utils.*;
import org.apache.cassandra.utils.concurrent.OpOrder;
public class CounterMutation implements IMutation
{
@ -67,9 +70,9 @@ public class CounterMutation implements IMutation
return mutation.getColumnFamilyIds();
}
public Collection<ColumnFamily> getColumnFamilies()
public Collection<PartitionUpdate> getPartitionUpdates()
{
return mutation.getColumnFamilies();
return mutation.getPartitionUpdates();
}
public Mutation getMutation()
@ -77,7 +80,7 @@ public class CounterMutation implements IMutation
return mutation;
}
public ByteBuffer key()
public DecoratedKey key()
{
return mutation.key();
}
@ -111,19 +114,14 @@ public class CounterMutation implements IMutation
Mutation result = new Mutation(getKeyspaceName(), key());
Keyspace keyspace = Keyspace.open(getKeyspaceName());
int count = 0;
for (ColumnFamily cf : getColumnFamilies())
count += cf.getColumnCount();
List<Lock> locks = new ArrayList<>(count);
Tracing.trace("Acquiring {} counter locks", count);
List<Lock> locks = new ArrayList<>();
Tracing.trace("Acquiring counter locks");
try
{
grabCounterLocks(keyspace, locks);
for (ColumnFamily cf : getColumnFamilies())
result.add(processModifications(cf));
for (PartitionUpdate upd : getPartitionUpdates())
result.add(processModifications(upd));
result.apply();
updateCounterCache(result, keyspace);
return result;
}
finally
@ -160,141 +158,144 @@ public class CounterMutation implements IMutation
*/
private Iterable<Object> getCounterLockKeys()
{
return Iterables.concat(Iterables.transform(getColumnFamilies(), new Function<ColumnFamily, Iterable<Object>>()
return Iterables.concat(Iterables.transform(getPartitionUpdates(), new Function<PartitionUpdate, Iterable<Object>>()
{
public Iterable<Object> apply(final ColumnFamily cf)
public Iterable<Object> apply(final PartitionUpdate update)
{
return Iterables.transform(cf, new Function<Cell, Object>()
return Iterables.concat(Iterables.transform(update, new Function<Row, Iterable<Object>>()
{
public Object apply(Cell cell)
public Iterable<Object> apply(final Row row)
{
return Objects.hashCode(cf.id(), key(), cell.name());
return Iterables.concat(Iterables.transform(row, new Function<Cell, Object>()
{
public Object apply(final Cell cell)
{
return Objects.hashCode(update.metadata().cfId, key(), row.clustering(), cell.column(), cell.path());
}
}));
}
});
}));
}
}));
}
// Replaces all the CounterUpdateCell-s with updated regular CounterCell-s
private ColumnFamily processModifications(ColumnFamily changesCF)
private PartitionUpdate processModifications(PartitionUpdate changes)
{
ColumnFamilyStore cfs = Keyspace.open(getKeyspaceName()).getColumnFamilyStore(changesCF.id());
ColumnFamilyStore cfs = Keyspace.open(getKeyspaceName()).getColumnFamilyStore(changes.metadata().cfId);
ColumnFamily resultCF = changesCF.cloneMeShallow();
List<CounterUpdateCell> counterUpdateCells = new ArrayList<>(changesCF.getColumnCount());
for (Cell cell : changesCF)
{
if (cell instanceof CounterUpdateCell)
counterUpdateCells.add((CounterUpdateCell)cell);
else
resultCF.addColumn(cell);
}
if (counterUpdateCells.isEmpty())
return resultCF; // only DELETEs
ClockAndCount[] currentValues = getCurrentValues(counterUpdateCells, cfs);
for (int i = 0; i < counterUpdateCells.size(); i++)
{
ClockAndCount currentValue = currentValues[i];
CounterUpdateCell update = counterUpdateCells.get(i);
long clock = currentValue.clock + 1L;
long count = currentValue.count + update.delta();
resultCF.addColumn(new BufferCounterCell(update.name(),
CounterContext.instance().createGlobal(CounterId.getLocalId(), clock, count),
update.timestamp()));
}
return resultCF;
}
// Attempt to load the current values(s) from cache. If that fails, read the rest from the cfs.
private ClockAndCount[] getCurrentValues(List<CounterUpdateCell> counterUpdateCells, ColumnFamilyStore cfs)
{
ClockAndCount[] currentValues = new ClockAndCount[counterUpdateCells.size()];
int remaining = counterUpdateCells.size();
List<PartitionUpdate.CounterMark> marks = changes.collectCounterMarks();
if (CacheService.instance.counterCache.getCapacity() != 0)
{
Tracing.trace("Fetching {} counter values from cache", counterUpdateCells.size());
remaining = getCurrentValuesFromCache(counterUpdateCells, cfs, currentValues);
if (remaining == 0)
return currentValues;
Tracing.trace("Fetching {} counter values from cache", marks.size());
updateWithCurrentValuesFromCache(marks, cfs);
if (marks.isEmpty())
return changes;
}
Tracing.trace("Reading {} counter values from the CF", remaining);
getCurrentValuesFromCFS(counterUpdateCells, cfs, currentValues);
Tracing.trace("Reading {} counter values from the CF", marks.size());
updateWithCurrentValuesFromCFS(marks, cfs);
return currentValues;
// What's remain is new counters
for (PartitionUpdate.CounterMark mark : marks)
updateWithCurrentValue(mark, ClockAndCount.BLANK, cfs);
return changes;
}
private void updateWithCurrentValue(PartitionUpdate.CounterMark mark, ClockAndCount currentValue, ColumnFamilyStore cfs)
{
long clock = currentValue.clock + 1L;
long count = currentValue.count + CounterContext.instance().total(mark.value());
mark.setValue(CounterContext.instance().createGlobal(CounterId.getLocalId(), clock, count));
// Cache the newly updated value
cfs.putCachedCounter(key().getKey(), mark.clustering(), mark.column(), mark.path(), ClockAndCount.create(clock, count));
}
// Returns the count of cache misses.
private int getCurrentValuesFromCache(List<CounterUpdateCell> counterUpdateCells,
ColumnFamilyStore cfs,
ClockAndCount[] currentValues)
private void updateWithCurrentValuesFromCache(List<PartitionUpdate.CounterMark> marks, ColumnFamilyStore cfs)
{
int cacheMisses = 0;
for (int i = 0; i < counterUpdateCells.size(); i++)
Iterator<PartitionUpdate.CounterMark> iter = marks.iterator();
while (iter.hasNext())
{
ClockAndCount cached = cfs.getCachedCounter(key(), counterUpdateCells.get(i).name());
PartitionUpdate.CounterMark mark = iter.next();
ClockAndCount cached = cfs.getCachedCounter(key().getKey(), mark.clustering(), mark.column(), mark.path());
if (cached != null)
currentValues[i] = cached;
else
cacheMisses++;
{
updateWithCurrentValue(mark, cached, cfs);
iter.remove();
}
}
return cacheMisses;
}
// Reads the missing current values from the CFS.
private void getCurrentValuesFromCFS(List<CounterUpdateCell> counterUpdateCells,
ColumnFamilyStore cfs,
ClockAndCount[] currentValues)
private void updateWithCurrentValuesFromCFS(List<PartitionUpdate.CounterMark> marks, ColumnFamilyStore cfs)
{
SortedSet<CellName> names = new TreeSet<>(cfs.metadata.comparator);
for (int i = 0; i < currentValues.length; i++)
if (currentValues[i] == null)
names.add(counterUpdateCells.get(i).name());
ReadCommand cmd = new SliceByNamesReadCommand(getKeyspaceName(), key(), cfs.metadata.cfName, Long.MIN_VALUE, new NamesQueryFilter(names));
Row row = cmd.getRow(cfs.keyspace);
ColumnFamily cf = row == null ? null : row.cf;
for (int i = 0; i < currentValues.length; i++)
ColumnFilter.Builder builder = ColumnFilter.selectionBuilder();
NavigableSet<Clustering> names = new TreeSet<>(cfs.metadata.comparator);
for (PartitionUpdate.CounterMark mark : marks)
{
if (currentValues[i] != null)
continue;
Cell cell = cf == null ? null : cf.getColumn(counterUpdateCells.get(i).name());
if (cell == null || !cell.isLive()) // absent or a tombstone.
currentValues[i] = ClockAndCount.BLANK;
names.add(mark.clustering().takeAlias());
if (mark.path() == null)
builder.add(mark.column());
else
currentValues[i] = CounterContext.instance().getLocalClockAndCount(cell.value());
builder.select(mark.column(), mark.path());
}
int nowInSec = FBUtilities.nowInSeconds();
ClusteringIndexNamesFilter filter = new ClusteringIndexNamesFilter(names, false);
SinglePartitionReadCommand cmd = SinglePartitionReadCommand.create(cfs.metadata, nowInSec, key(), builder.build(), filter);
PeekingIterator<PartitionUpdate.CounterMark> markIter = Iterators.peekingIterator(marks.iterator());
try (OpOrder.Group op = cfs.readOrdering.start(); RowIterator partition = UnfilteredRowIterators.filter(cmd.queryMemtableAndDisk(cfs, op), nowInSec))
{
updateForRow(markIter, partition.staticRow(), cfs);
while (partition.hasNext())
{
if (!markIter.hasNext())
return;
updateForRow(markIter, partition.next(), cfs);
}
}
}
private void updateCounterCache(Mutation applied, Keyspace keyspace)
private int compare(Clustering c1, Clustering c2, ColumnFamilyStore cfs)
{
if (CacheService.instance.counterCache.getCapacity() == 0)
if (c1 == Clustering.STATIC_CLUSTERING)
return c2 == Clustering.STATIC_CLUSTERING ? 0 : -1;
if (c2 == Clustering.STATIC_CLUSTERING)
return 1;
return cfs.getComparator().compare(c1, c2);
}
private void updateForRow(PeekingIterator<PartitionUpdate.CounterMark> markIter, Row row, ColumnFamilyStore cfs)
{
int cmp = 0;
// If the mark is before the row, we have no value for this mark, just consume it
while (markIter.hasNext() && (cmp = compare(markIter.peek().clustering(), row.clustering(), cfs)) < 0)
markIter.next();
if (!markIter.hasNext())
return;
for (ColumnFamily cf : applied.getColumnFamilies())
while (cmp == 0)
{
ColumnFamilyStore cfs = keyspace.getColumnFamilyStore(cf.id());
for (Cell cell : cf)
if (cell instanceof CounterCell)
cfs.putCachedCounter(key(), cell.name(), CounterContext.instance().getLocalClockAndCount(cell.value()));
}
}
PartitionUpdate.CounterMark mark = markIter.next();
Cell cell = mark.path() == null ? row.getCell(mark.column()) : row.getCell(mark.column(), mark.path());
if (cell != null)
{
updateWithCurrentValue(mark, CounterContext.instance().getLocalClockAndCount(cell.value()), cfs);
markIter.remove();
}
if (!markIter.hasNext())
return;
public void addAll(IMutation m)
{
if (!(m instanceof CounterMutation))
throw new IllegalArgumentException();
CounterMutation cm = (CounterMutation)m;
mutation.addAll(cm.mutation);
cmp = compare(markIter.peek().clustering(), row.clustering(), cfs);
}
}
public long getTimeout()

488
src/java/org/apache/cassandra/db/DataRange.java
View File

@ -6,7 +6,6 @@
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
@ -17,302 +16,391 @@
*/
package org.apache.cassandra.db;
import java.io.DataInput;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
import com.google.common.base.Objects;
import org.apache.cassandra.config.CFMetaData;
import org.apache.cassandra.db.columniterator.IdentityQueryFilter;
import org.apache.cassandra.db.composites.Composite;
import org.apache.cassandra.db.composites.Composites;
import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.db.filter.*;
import org.apache.cassandra.db.marshal.AbstractType;
import org.apache.cassandra.db.marshal.CompositeType;
import org.apache.cassandra.dht.*;
import org.apache.cassandra.io.util.DataOutputPlus;
import org.apache.cassandra.net.MessagingService;
/**
* Groups key range and column filter for range queries.
*
* The main "trick" of this class is that the column filter can only
* be obtained by providing the row key on which the column filter will
* be applied (which we always know before actually querying the columns).
*
* This allows the paging DataRange to return a filter for most rows but a
* potentially different ones for the starting and stopping key. Could
* allow more fancy stuff in the future too, like column filters that
* depend on the actual key value :)
* Groups both the range of partitions to query, and the clustering index filter to
* apply for each partition (for a (partition) range query).
* <p>
* The main "trick" is that the clustering index filter can only be obtained by
* providing the partition key on which the filter will be applied. This is
* necessary when paging range queries, as we might need a different filter
* for the starting key than for other keys (because the previous page we had
* queried may have ended in the middle of a partition).
*/
public class DataRange
{
protected final AbstractBounds<RowPosition> keyRange;
protected IDiskAtomFilter columnFilter;
protected final boolean selectFullRow;
public static final Serializer serializer = new Serializer();
public DataRange(AbstractBounds<RowPosition> range, IDiskAtomFilter columnFilter)
private final AbstractBounds<PartitionPosition> keyRange;
protected final ClusteringIndexFilter clusteringIndexFilter;
/**
* Creates a {@code DataRange} given a range of partition keys and a clustering index filter. The
* return {@code DataRange} will return the same filter for all keys.
*
* @param range the range over partition keys to use.
* @param clusteringIndexFilter the clustering index filter to use.
*/
public DataRange(AbstractBounds<PartitionPosition> range, ClusteringIndexFilter clusteringIndexFilter)
{
this.keyRange = range;
this.columnFilter = columnFilter;
this.selectFullRow = columnFilter instanceof SliceQueryFilter
? isFullRowSlice((SliceQueryFilter)columnFilter)
: false;
}
public static boolean isFullRowSlice(SliceQueryFilter filter)
{
return filter.slices.length == 1
&& filter.start().isEmpty()
&& filter.finish().isEmpty()
&& filter.count == Integer.MAX_VALUE;
this.clusteringIndexFilter = clusteringIndexFilter;
}
/**
* Creates a {@code DataRange} to query all data (over the whole ring).
*
* @param partitioner the partitioner in use for the table.
*
* @return the newly create {@code DataRange}.
*/
public static DataRange allData(IPartitioner partitioner)
{
return forTokenRange(new Range<Token>(partitioner.getMinimumToken(), partitioner.getMinimumToken()));
}
public static DataRange forTokenRange(Range<Token> keyRange)
/**
* Creates a {@code DataRange} to query all rows over the provided token range.
*
* @param tokenRange the (partition key) token range to query.
*
* @return the newly create {@code DataRange}.
*/
public static DataRange forTokenRange(Range<Token> tokenRange)
{
return forKeyRange(Range.makeRowRange(keyRange));
return forKeyRange(Range.makeRowRange(tokenRange));
}
public static DataRange forKeyRange(Range<RowPosition> keyRange)
/**
* Creates a {@code DataRange} to query all rows over the provided key range.
*
* @param keyRange the (partition key) range to query.
*
* @return the newly create {@code DataRange}.
*/
public static DataRange forKeyRange(Range<PartitionPosition> keyRange)
{
return new DataRange(keyRange, new IdentityQueryFilter());
return new DataRange(keyRange, new ClusteringIndexSliceFilter(Slices.ALL, false));
}
public AbstractBounds<RowPosition> keyRange()
/**
* Creates a {@code DataRange} to query all partitions of the ring using the provided
* clustering index filter.
*
* @param partitioner the partitioner in use for the table queried.
* @param filter the clustering index filter to use.
*
* @return the newly create {@code DataRange}.
*/
public static DataRange allData(IPartitioner partitioner, ClusteringIndexFilter filter)
{
return new DataRange(Range.makeRowRange(new Range<Token>(partitioner.getMinimumToken(), partitioner.getMinimumToken())), filter);
}
/**
* The range of partition key queried by this {@code DataRange}.
*
* @return the range of partition key queried by this {@code DataRange}.
*/
public AbstractBounds<PartitionPosition> keyRange()
{
return keyRange;
}
public RowPosition startKey()
/**
* The start of the partition key range queried by this {@code DataRange}.
*
* @return the start of the partition key range queried by this {@code DataRange}.
*/
public PartitionPosition startKey()
{
return keyRange.left;
}
public RowPosition stopKey()
/**
* The end of the partition key range queried by this {@code DataRange}.
*
* @return the end of the partition key range queried by this {@code DataRange}.
*/
public PartitionPosition stopKey()
{
return keyRange.right;
}
/**
* Returns true if tombstoned partitions should not be included in results or count towards the limit.
* See CASSANDRA-8490 for more details on why this is needed (and done this way).
* */
public boolean ignoredTombstonedPartitions()
* Whether the underlying clustering index filter is a names filter or not.
*
* @return Whether the underlying clustering index filter is a names filter or not.
*/
public boolean isNamesQuery()
{
if (!(columnFilter instanceof SliceQueryFilter))
return false;
return ((SliceQueryFilter) columnFilter).compositesToGroup == SliceQueryFilter.IGNORE_TOMBSTONED_PARTITIONS;
return clusteringIndexFilter instanceof ClusteringIndexNamesFilter;
}
// Whether the bounds of this DataRange actually wraps around.
/**
* Whether the range queried by this {@code DataRange} actually wraps around.
*
* @return whether the range queried by this {@code DataRange} actually wraps around.
*/
public boolean isWrapAround()
{
// On range can ever wrap
// Only range can ever wrap
return keyRange instanceof Range && ((Range<?>)keyRange).isWrapAround();
}
public boolean contains(RowPosition pos)
/**
* Whether the provided ring position is covered by this {@code DataRange}.
*
* @return whether the provided ring position is covered by this {@code DataRange}.
*/
public boolean contains(PartitionPosition pos)
{
return keyRange.contains(pos);
}
public int getLiveCount(ColumnFamily data, long now)
/**
* Whether this {@code DataRange} queries everything (has no restriction neither on the
* partition queried, nor within the queried partition).
*
* @return Whether this {@code DataRange} queries everything.
*/
public boolean isUnrestricted()
{
return columnFilter instanceof SliceQueryFilter
? ((SliceQueryFilter)columnFilter).lastCounted()
: columnFilter.getLiveCount(data, now);
}
public boolean selectsFullRowFor(ByteBuffer rowKey)
{
return selectFullRow;
return startKey().isMinimum() && stopKey().isMinimum() && clusteringIndexFilter.selectsAllPartition();
}
/**
* Returns a column filter that should be used for a particular row key. Note that in the case of paging,
* slice starts and ends may change depending on the row key.
* The clustering index filter to use for the provided key.
* <p>
* This may or may not be the same filter for all keys (that is, paging range
* use a different filter for their start key).
*
* @param key the partition key for which we want the clustering index filter.
*
* @return the clustering filter to use for {@code key}.
*/
public IDiskAtomFilter columnFilter(ByteBuffer rowKey)
public ClusteringIndexFilter clusteringIndexFilter(DecoratedKey key)
{
return columnFilter;
return clusteringIndexFilter;
}
/**
* Sets a new limit on the number of (grouped) cells to fetch. This is currently only used when the query limit applies
* to CQL3 rows.
* Returns a new {@code DataRange} for use when paging {@code this} range.
*
* @param range the range of partition keys to query.
* @param comparator the comparator for the table queried.
* @param lastReturned the clustering for the last result returned by the previous page, i.e. the result we want to start our new page
* from. This last returned must <b>must</b> correspond to left bound of {@code range} (in other words, {@code range.left} must be the
* partition key for that {@code lastReturned} result).
* @param inclusive whether or not we want to include the {@code lastReturned} in the newly returned page of results.
*
* @return a new {@code DataRange} suitable for paging {@code this} range given the {@code lastRetuned} result of the previous page.
*/
public void updateColumnsLimit(int count)
public DataRange forPaging(AbstractBounds<PartitionPosition> range, ClusteringComparator comparator, Clustering lastReturned, boolean inclusive)
{
columnFilter.updateColumnsLimit(count);
return new Paging(range, clusteringIndexFilter, comparator, lastReturned, inclusive);
}
public static class Paging extends DataRange
/**
* Returns a new {@code DataRange} equivalent to {@code this} one but restricted to the provided sub-range.
*
* @param range the sub-range to use for the newly returned data range. Note that assumes that {@code range} is a proper
* sub-range of the initial range but doesn't validate it. You should make sure to only provided sub-ranges however or this
* might throw off the paging case (see Paging.forSubRange()).
*
* @return a new {@code DataRange} using {@code range} as partition key range and the clustering index filter filter from {@code this}.
*/
public DataRange forSubRange(AbstractBounds<PartitionPosition> range)
{
// The slice of columns that we want to fetch for each row, ignoring page start/end issues.
private final SliceQueryFilter sliceFilter;
return new DataRange(range, clusteringIndexFilter);
}
private final CFMetaData cfm;
public String toString(CFMetaData metadata)
{
return String.format("range=%s pfilter=%s", keyRange.getString(metadata.getKeyValidator()), clusteringIndexFilter.toString(metadata));
}
private final Comparator<Composite> comparator;
public String toCQLString(CFMetaData metadata)
{
if (isUnrestricted())
return "UNRESTRICTED";
// used to restrict the start of the slice for the first partition in the range
private final Composite firstPartitionColumnStart;
StringBuilder sb = new StringBuilder();
// used to restrict the end of the slice for the last partition in the range
private final Composite lastPartitionColumnFinish;
boolean needAnd = false;
if (!startKey().isMinimum())
{
appendClause(startKey(), sb, metadata, true, keyRange.isStartInclusive());
needAnd = true;
}
if (!stopKey().isMinimum())
{
if (needAnd)
sb.append(" AND ");
appendClause(stopKey(), sb, metadata, false, keyRange.isEndInclusive());
needAnd = true;
}
// tracks the last key that we updated the filter for to avoid duplicating work
private ByteBuffer lastKeyFilterWasUpdatedFor;
String filterString = clusteringIndexFilter.toCQLString(metadata);
if (!filterString.isEmpty())
sb.append(needAnd ? " AND " : "").append(filterString);
private Paging(AbstractBounds<RowPosition> range, SliceQueryFilter filter, Composite firstPartitionColumnStart,
Composite lastPartitionColumnFinish, CFMetaData cfm, Comparator<Composite> comparator)
return sb.toString();
}
private void appendClause(PartitionPosition pos, StringBuilder sb, CFMetaData metadata, boolean isStart, boolean isInclusive)
{
sb.append("token(");
sb.append(ColumnDefinition.toCQLString(metadata.partitionKeyColumns()));
sb.append(") ").append(getOperator(isStart, isInclusive)).append(" ");
if (pos instanceof DecoratedKey)
{
sb.append("token(");
appendKeyString(sb, metadata.getKeyValidator(), ((DecoratedKey)pos).getKey());
sb.append(")");
}
else
{
sb.append(((Token.KeyBound)pos).getToken());
}
}
private static String getOperator(boolean isStart, boolean isInclusive)
{
return isStart
? (isInclusive ? ">=" : ">")
: (isInclusive ? "<=" : "<");
}
// TODO: this is reused in SinglePartitionReadCommand but this should not really be here. Ideally
// we need a more "native" handling of composite partition keys.
public static void appendKeyString(StringBuilder sb, AbstractType<?> type, ByteBuffer key)
{
if (type instanceof CompositeType)
{
CompositeType ct = (CompositeType)type;
ByteBuffer[] values = ct.split(key);
for (int i = 0; i < ct.types.size(); i++)
sb.append(i == 0 ? "" : ", ").append(ct.types.get(i).getString(values[i]));
}
else
{
sb.append(type.getString(key));
}
}
/**
* Specialized {@code DataRange} used for the paging case.
* <p>
* It uses the clustering of the last result of the previous page to restrict the filter on the
* first queried partition (the one for that last result) so it only fetch results that follow that
* last result. In other words, this makes sure this resume paging where we left off.
*/
private static class Paging extends DataRange
{
private final ClusteringComparator comparator;
private final Clustering lastReturned;
private final boolean inclusive;
private Paging(AbstractBounds<PartitionPosition> range,
ClusteringIndexFilter filter,
ClusteringComparator comparator,
Clustering lastReturned,
boolean inclusive)
{
super(range, filter);
// When using a paging range, we don't allow wrapped ranges, as it's unclear how to handle them properly.
// This is ok for now since we only need this in range slice queries, and the range are "unwrapped" in that case.
// This is ok for now since we only need this in range queries, and the range are "unwrapped" in that case.
assert !(range instanceof Range) || !((Range<?>)range).isWrapAround() || range.right.isMinimum() : range;
assert lastReturned != null;
this.sliceFilter = filter;
this.cfm = cfm;
this.comparator = comparator;
this.firstPartitionColumnStart = firstPartitionColumnStart;
this.lastPartitionColumnFinish = lastPartitionColumnFinish;
this.lastKeyFilterWasUpdatedFor = null;
}
public Paging(AbstractBounds<RowPosition> range, SliceQueryFilter filter, Composite columnStart, Composite columnFinish, CFMetaData cfm)
{
this(range, filter, columnStart, columnFinish, cfm, filter.isReversed() ? cfm.comparator.reverseComparator() : cfm.comparator);
this.lastReturned = lastReturned;
this.inclusive = inclusive;
}
@Override
public boolean selectsFullRowFor(ByteBuffer rowKey)
public ClusteringIndexFilter clusteringIndexFilter(DecoratedKey key)
{
// If we initial filter is not the full filter, don't bother
if (!selectFullRow)
return false;
if (!equals(startKey(), rowKey) && !equals(stopKey(), rowKey))
return true;
return isFullRowSlice((SliceQueryFilter)columnFilter(rowKey));
}
private boolean equals(RowPosition pos, ByteBuffer rowKey)
{
return pos instanceof DecoratedKey && ((DecoratedKey)pos).getKey().equals(rowKey);
return key.equals(startKey())
? clusteringIndexFilter.forPaging(comparator, lastReturned, inclusive)
: clusteringIndexFilter;
}
@Override
public IDiskAtomFilter columnFilter(ByteBuffer rowKey)
public DataRange forSubRange(AbstractBounds<PartitionPosition> range)
{
/*
* We have that ugly hack that for slice queries, when we ask for
* the live count, we reach into the query filter to get the last
* counter number of columns to avoid recounting.
* Maybe we should just remove that hack, but in the meantime, we
* need to keep a reference the last returned filter.
*/
if (equals(startKey(), rowKey) || equals(stopKey(), rowKey))
// This is called for subrange of the initial range. So either it's the beginning of the initial range,
// and we need to preserver lastReturned, or it's not, and we don't care about it anymore.
return range.left.equals(keyRange().left)
? new Paging(range, clusteringIndexFilter, comparator, lastReturned, inclusive)
: new DataRange(range, clusteringIndexFilter);
}
@Override
public boolean isUnrestricted()
{
return false;
}
}
public static class Serializer
{
public void serialize(DataRange range, DataOutputPlus out, int version, CFMetaData metadata) throws IOException
{
AbstractBounds.rowPositionSerializer.serialize(range.keyRange, out, version);
ClusteringIndexFilter.serializer.serialize(range.clusteringIndexFilter, out, version);
boolean isPaging = range instanceof Paging;
out.writeBoolean(isPaging);
if (isPaging)
{
if (!rowKey.equals(lastKeyFilterWasUpdatedFor))
{
this.lastKeyFilterWasUpdatedFor = rowKey;
columnFilter = sliceFilter.withUpdatedSlices(slicesForKey(rowKey));
}
Clustering.serializer.serialize(((Paging)range).lastReturned, out, version, metadata.comparator.subtypes());
out.writeBoolean(((Paging)range).inclusive);
}
}
public DataRange deserialize(DataInput in, int version, CFMetaData metadata) throws IOException
{
AbstractBounds<PartitionPosition> range = AbstractBounds.rowPositionSerializer.deserialize(in, MessagingService.globalPartitioner(), version);
ClusteringIndexFilter filter = ClusteringIndexFilter.serializer.deserialize(in, version, metadata);
if (in.readBoolean())
{
ClusteringComparator comparator = metadata.comparator;
Clustering lastReturned = Clustering.serializer.deserialize(in, version, comparator.subtypes());
boolean inclusive = in.readBoolean();
return new Paging(range, filter, comparator, lastReturned, inclusive);
}
else
{
columnFilter = sliceFilter;
return new DataRange(range, filter);
}
return columnFilter;
}
/** Returns true if the slice includes static columns, false otherwise. */
private boolean sliceIncludesStatics(ColumnSlice slice, boolean reversed, CFMetaData cfm)
public long serializedSize(DataRange range, int version, CFMetaData metadata)
{
return cfm.hasStaticColumns() &&
slice.includes(reversed ? cfm.comparator.reverseComparator() : cfm.comparator, cfm.comparator.staticPrefix().end());
}
long size = AbstractBounds.rowPositionSerializer.serializedSize(range.keyRange, version)
+ ClusteringIndexFilter.serializer.serializedSize(range.clusteringIndexFilter, version)
+ 1; // isPaging boolean
private ColumnSlice[] slicesForKey(ByteBuffer key)
{
// Also note that firstPartitionColumnStart and lastPartitionColumnFinish, when used, only "restrict" the filter slices,
// it doesn't expand on them. As such, we can ignore the case where they are empty and we do
// as it screw up with the logic below (see #6592)
Composite newStart = equals(startKey(), key) && !firstPartitionColumnStart.isEmpty() ? firstPartitionColumnStart : null;
Composite newFinish = equals(stopKey(), key) && !lastPartitionColumnFinish.isEmpty() ? lastPartitionColumnFinish : null;
// in the common case, we'll have the same number of slices
List<ColumnSlice> newSlices = new ArrayList<>(sliceFilter.slices.length);
// Check our slices to see if any fall before the page start (in which case they can be removed) or
// if they contain the page start (in which case they should start from the page start). However, if the
// slices would include static columns, we need to ensure they are also fetched, and so a separate
// slice for the static columns may be required.
// Note that if the query is reversed, we can't handle statics by simply adding a separate slice here, so
// the reversed case is handled by SliceFromReadCommand instead. See CASSANDRA-8502 for more details.
for (ColumnSlice slice : sliceFilter.slices)
if (range instanceof Paging)
{
if (newStart != null)
{
if (slice.isBefore(comparator, newStart))
{
if (!sliceFilter.reversed && sliceIncludesStatics(slice, false, cfm))
newSlices.add(new ColumnSlice(Composites.EMPTY, cfm.comparator.staticPrefix().end()));
continue;
}
if (slice.includes(comparator, newStart))
{
if (!sliceFilter.reversed && sliceIncludesStatics(slice, false, cfm) && !newStart.equals(Composites.EMPTY))
newSlices.add(new ColumnSlice(Composites.EMPTY, cfm.comparator.staticPrefix().end()));
slice = new ColumnSlice(newStart, slice.finish);
}
// once we see a slice that either includes the page start or is after it, we can stop checking
// against the page start (because the slices are ordered)
newStart = null;
}
assert newStart == null;
if (newFinish != null && !slice.isBefore(comparator, newFinish))
{
if (slice.includes(comparator, newFinish))
newSlices.add(new ColumnSlice(slice.start, newFinish));
// In any case, we're done
break;
}
newSlices.add(slice);
size += Clustering.serializer.serializedSize(((Paging)range).lastReturned, version, metadata.comparator.subtypes(), TypeSizes.NATIVE);
size += 1; // inclusive boolean
}
return newSlices.toArray(new ColumnSlice[newSlices.size()]);
}
@Override
public void updateColumnsLimit(int count)
{
columnFilter.updateColumnsLimit(count);
sliceFilter.updateColumnsLimit(count);
}
@Override
public String toString()
{
return Objects.toStringHelper(this)
.add("keyRange", keyRange)
.add("sliceFilter", sliceFilter)
.add("columnFilter", columnFilter)
.add("firstPartitionColumnStart", firstPartitionColumnStart == null ? "null" : cfm.comparator.getString(firstPartitionColumnStart))
.add("lastPartitionColumnFinish", lastPartitionColumnFinish == null ? "null" : cfm.comparator.getString(lastPartitionColumnFinish))
.toString();
return size;
}
}
}

10
src/java/org/apache/cassandra/db/DecoratedKey.java
View File

@ -36,7 +36,7 @@ import org.apache.cassandra.utils.IFilter.FilterKey;
* if this matters, you can subclass RP to use a stronger hash, or use a non-lossy tokenization scheme (as in the
* OrderPreservingPartitioner classes).
*/
public abstract class DecoratedKey implements RowPosition, FilterKey
public abstract class DecoratedKey implements PartitionPosition, FilterKey
{
public static final Comparator<DecoratedKey> comparator = new Comparator<DecoratedKey>()
{
@ -72,7 +72,7 @@ public abstract class DecoratedKey implements RowPosition, FilterKey
return ByteBufferUtil.compareUnsigned(getKey(), other.getKey()) == 0; // we compare faster than BB.equals for array backed BB
}
public int compareTo(RowPosition pos)
public int compareTo(PartitionPosition pos)
{
if (this == pos)
return 0;
@ -86,7 +86,7 @@ public abstract class DecoratedKey implements RowPosition, FilterKey
return cmp == 0 ? ByteBufferUtil.compareUnsigned(getKey(), otherKey.getKey()) : cmp;
}
public static int compareTo(IPartitioner partitioner, ByteBuffer key, RowPosition position)
public static int compareTo(IPartitioner partitioner, ByteBuffer key, PartitionPosition position)
{
// delegate to Token.KeyBound if needed
if (!(position instanceof DecoratedKey))
@ -113,9 +113,9 @@ public abstract class DecoratedKey implements RowPosition, FilterKey
return false;
}
public RowPosition.Kind kind()
public PartitionPosition.Kind kind()
{
return RowPosition.Kind.ROW_KEY;
return PartitionPosition.Kind.ROW_KEY;
}
@Override

30
src/java/org/apache/cassandra/db/DeletedCell.java
View File

@ -1,30 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.db;
import org.apache.cassandra.config.CFMetaData;
import org.apache.cassandra.utils.concurrent.OpOrder;
import org.apache.cassandra.utils.memory.AbstractAllocator;
import org.apache.cassandra.utils.memory.MemtableAllocator;
public interface DeletedCell extends Cell
{
DeletedCell localCopy(CFMetaData metadata, AbstractAllocator allocator);
DeletedCell localCopy(CFMetaData metaData, MemtableAllocator allocator, OpOrder.Group opGroup);
}

300
src/java/org/apache/cassandra/db/DeletionInfo.java
View File

@ -17,40 +17,32 @@
*/
package org.apache.cassandra.db;
import java.io.DataInput;
import java.io.IOException;
import java.security.MessageDigest;
import java.util.Comparator;
import java.util.Iterator;
import com.google.common.base.Objects;
import com.google.common.collect.Iterators;
import org.apache.cassandra.cache.IMeasurableMemory;
import org.apache.cassandra.db.composites.CType;
import org.apache.cassandra.db.composites.Composite;
import org.apache.cassandra.io.IVersionedSerializer;
import org.apache.cassandra.io.util.DataOutputPlus;
import org.apache.cassandra.utils.ByteBufferUtil;
import org.apache.cassandra.db.rows.Cell;
import org.apache.cassandra.utils.ObjectSizes;
import org.apache.cassandra.utils.memory.AbstractAllocator;
/**
* A combination of a top-level (or row) tombstone and range tombstones describing the deletions
* within a {@link ColumnFamily} (or row).
* A combination of a top-level (partition) tombstone and range tombstones describing the deletions
* within a partition.
*/
public class DeletionInfo implements IMeasurableMemory
{
private static final long EMPTY_SIZE = ObjectSizes.measure(new DeletionInfo(0, 0));
/**
* This represents a deletion of the entire row. We can't represent this within the RangeTombstoneList, so it's
* kept separately. This also slightly optimizes the common case of a full row deletion.
* This represents a deletion of the entire partition. We can't represent this within the RangeTombstoneList, so it's
* kept separately. This also slightly optimizes the common case of a full partition deletion.
*/
private DeletionTime topLevel;
private DeletionTime partitionDeletion;
/**
* A list of range tombstones within the row. This is left as null if there are no range tombstones
* A list of range tombstones within the partition. This is left as null if there are no range tombstones
* (to save an allocation (since it's a common case).
*/
private RangeTombstoneList ranges;
@ -65,28 +57,23 @@ public class DeletionInfo implements IMeasurableMemory
{
// Pre-1.1 node may return MIN_VALUE for non-deleted container, but the new default is MAX_VALUE
// (see CASSANDRA-3872)
this(new DeletionTime(markedForDeleteAt, localDeletionTime == Integer.MIN_VALUE ? Integer.MAX_VALUE : localDeletionTime));
this(new SimpleDeletionTime(markedForDeleteAt, localDeletionTime == Integer.MIN_VALUE ? Integer.MAX_VALUE : localDeletionTime));
}
public DeletionInfo(DeletionTime topLevel)
public DeletionInfo(DeletionTime partitionDeletion)
{
this(topLevel, null);
this(partitionDeletion, null);
}
public DeletionInfo(Composite start, Composite end, Comparator<Composite> comparator, long markedForDeleteAt, int localDeletionTime)
public DeletionInfo(ClusteringComparator comparator, Slice slice, long markedForDeleteAt, int localDeletionTime)
{
this(DeletionTime.LIVE, new RangeTombstoneList(comparator, 1));
ranges.add(start, end, markedForDeleteAt, localDeletionTime);
ranges.add(slice.start(), slice.end(), markedForDeleteAt, localDeletionTime);
}
public DeletionInfo(RangeTombstone rangeTombstone, Comparator<Composite> comparator)
public DeletionInfo(DeletionTime partitionDeletion, RangeTombstoneList ranges)
{
this(rangeTombstone.min, rangeTombstone.max, comparator, rangeTombstone.data.markedForDeleteAt, rangeTombstone.data.localDeletionTime);
}
private DeletionInfo(DeletionTime topLevel, RangeTombstoneList ranges)
{
this.topLevel = topLevel;
this.partitionDeletion = partitionDeletion.takeAlias();
this.ranges = ranges;
}
@ -100,17 +87,16 @@ public class DeletionInfo implements IMeasurableMemory
public DeletionInfo copy()
{
return new DeletionInfo(topLevel, ranges == null ? null : ranges.copy());
return new DeletionInfo(partitionDeletion, ranges == null ? null : ranges.copy());
}
public DeletionInfo copy(AbstractAllocator allocator)
{
RangeTombstoneList rangesCopy = null;
if (ranges != null)
rangesCopy = ranges.copy(allocator);
return new DeletionInfo(topLevel, rangesCopy);
return new DeletionInfo(partitionDeletion, rangesCopy);
}
/**
@ -118,106 +104,31 @@ public class DeletionInfo implements IMeasurableMemory
*/
public boolean isLive()
{
return topLevel.isLive() && (ranges == null || ranges.isEmpty());
return partitionDeletion.isLive() && (ranges == null || ranges.isEmpty());
}
/**
* Return whether a given cell is deleted by the container having this deletion info.
* Return whether a given cell is deleted by this deletion info.
*
* @param clustering the clustering for the cell to check.
* @param cell the cell to check.
* @return true if the cell is deleted, false otherwise
*/
public boolean isDeleted(Cell cell)
private boolean isDeleted(Clustering clustering, Cell cell)
{
// We do rely on this test: if topLevel.markedForDeleteAt is MIN_VALUE, we should not
// consider the column deleted even if timestamp=MIN_VALUE, otherwise this break QueryFilter.isRelevant
// If we're live, don't consider anything deleted, even if the cell ends up having as timestamp Long.MIN_VALUE
// (which shouldn't happen in practice, but it would invalid to consider it deleted if it does).
if (isLive())
return false;
if (cell.timestamp() <= topLevel.markedForDeleteAt)
if (cell.livenessInfo().timestamp() <= partitionDeletion.markedForDeleteAt())
return true;
// No matter what the counter cell's timestamp is, a tombstone always takes precedence. See CASSANDRA-7346.
if (!topLevel.isLive() && cell instanceof CounterCell)
if (!partitionDeletion.isLive() && cell.isCounterCell())
return true;
return ranges != null && ranges.isDeleted(cell);
}
/**
* Returns a new {@link InOrderTester} in forward order.
*/
public InOrderTester inOrderTester()
{
return inOrderTester(false);
}
/**
* Returns a new {@link InOrderTester} given the order in which
* columns will be passed to it.
*/
public InOrderTester inOrderTester(boolean reversed)
{
return new InOrderTester(reversed);
}
/**
* Purge every tombstones that are older than {@code gcbefore}.
*
* @param gcBefore timestamp (in seconds) before which tombstones should be purged
*/
public void purge(int gcBefore)
{
topLevel = topLevel.localDeletionTime < gcBefore ? DeletionTime.LIVE : topLevel;
if (ranges != null)
{
ranges.purge(gcBefore);
if (ranges.isEmpty())
ranges = null;
}
}
/**
* Evaluates difference between this deletion info and superset for read repair
*
* @return the difference between the two, or LIVE if no difference
*/
public DeletionInfo diff(DeletionInfo superset)
{
RangeTombstoneList rangeDiff = superset.ranges == null || superset.ranges.isEmpty()
? null
: ranges == null ? superset.ranges : ranges.diff(superset.ranges);
return topLevel.markedForDeleteAt != superset.topLevel.markedForDeleteAt || rangeDiff != null
? new DeletionInfo(superset.topLevel, rangeDiff)
: DeletionInfo.live();
}
/**
* Digests deletion info. Used to trigger read repair on mismatch.
*/
public void updateDigest(MessageDigest digest)
{
if (topLevel.markedForDeleteAt != Long.MIN_VALUE)
digest.update(ByteBufferUtil.bytes(topLevel.markedForDeleteAt));
if (ranges != null)
ranges.updateDigest(digest);
}
/**
* Returns true if {@code purge} would remove the top-level tombstone or any of the range
* tombstones, false otherwise.
* @param gcBefore timestamp (in seconds) before which tombstones should be purged
*/
public boolean hasPurgeableTombstones(int gcBefore)
{
if (topLevel.localDeletionTime < gcBefore)
return true;
return ranges != null && ranges.hasPurgeableTombstones(gcBefore);
return ranges != null && ranges.isDeleted(clustering, cell);
}
/**
@ -227,11 +138,11 @@ public class DeletionInfo implements IMeasurableMemory
*/
public void add(DeletionTime newInfo)
{
if (topLevel.markedForDeleteAt < newInfo.markedForDeleteAt)
topLevel = newInfo;
if (newInfo.supersedes(partitionDeletion))
partitionDeletion = newInfo;
}
public void add(RangeTombstone tombstone, Comparator<Composite> comparator)
public void add(RangeTombstone tombstone, ClusteringComparator comparator)
{
if (ranges == null)
ranges = new RangeTombstoneList(comparator, 1);
@ -248,7 +159,7 @@ public class DeletionInfo implements IMeasurableMemory
*/
public DeletionInfo add(DeletionInfo newInfo)
{
add(newInfo.topLevel);
add(newInfo.partitionDeletion);
if (ranges == null)
ranges = newInfo.ranges == null ? null : newInfo.ranges.copy();
@ -258,53 +169,30 @@ public class DeletionInfo implements IMeasurableMemory
return this;
}
/**
* Returns the minimum timestamp in any of the range tombstones or the top-level tombstone.
*/
public long minTimestamp()
public DeletionTime getPartitionDeletion()
{
return ranges == null
? topLevel.markedForDeleteAt
: Math.min(topLevel.markedForDeleteAt, ranges.minMarkedAt());
}
/**
* Returns the maximum timestamp in any of the range tombstones or the top-level tombstone.
*/
public long maxTimestamp()
{
return ranges == null
? topLevel.markedForDeleteAt
: Math.max(topLevel.markedForDeleteAt, ranges.maxMarkedAt());
}
/**
* Returns the top-level (or "row") tombstone.
*/
public DeletionTime getTopLevelDeletion()
{
return topLevel;
return partitionDeletion;
}
// Use sparingly, not the most efficient thing
public Iterator<RangeTombstone> rangeIterator()
public Iterator<RangeTombstone> rangeIterator(boolean reversed)
{
return ranges == null ? Iterators.<RangeTombstone>emptyIterator() : ranges.iterator();
return ranges == null ? Iterators.<RangeTombstone>emptyIterator() : ranges.iterator(reversed);
}
public Iterator<RangeTombstone> rangeIterator(Composite start, Composite finish)
public Iterator<RangeTombstone> rangeIterator(Slice slice, boolean reversed)
{
return ranges == null ? Iterators.<RangeTombstone>emptyIterator() : ranges.iterator(start, finish);
return ranges == null ? Iterators.<RangeTombstone>emptyIterator() : ranges.iterator(slice, reversed);
}
public RangeTombstone rangeCovering(Composite name)
public RangeTombstone rangeCovering(Clustering name)
{
return ranges == null ? null : ranges.search(name);
}
public int dataSize()
{
int size = TypeSizes.NATIVE.sizeof(topLevel.markedForDeleteAt);
int size = TypeSizes.NATIVE.sizeof(partitionDeletion.markedForDeleteAt());
return size + (ranges == null ? 0 : ranges.dataSize());
}
@ -323,45 +211,43 @@ public class DeletionInfo implements IMeasurableMemory
*/
public boolean mayModify(DeletionInfo delInfo)
{
return topLevel.compareTo(delInfo.topLevel) > 0 || hasRanges();
return partitionDeletion.compareTo(delInfo.partitionDeletion) > 0 || hasRanges();
}
@Override
public String toString()
{
if (ranges == null || ranges.isEmpty())
return String.format("{%s}", topLevel);
return String.format("{%s}", partitionDeletion);
else
return String.format("{%s, ranges=%s}", topLevel, rangesAsString());
return String.format("{%s, ranges=%s}", partitionDeletion, rangesAsString());
}
private String rangesAsString()
{
assert !ranges.isEmpty();
StringBuilder sb = new StringBuilder();
CType type = (CType)ranges.comparator();
assert type != null;
Iterator<RangeTombstone> iter = rangeIterator();
ClusteringComparator cc = ranges.comparator();
Iterator<RangeTombstone> iter = rangeIterator(false);
while (iter.hasNext())
{
RangeTombstone i = iter.next();
sb.append("[");
sb.append(type.getString(i.min)).append("-");
sb.append(type.getString(i.max)).append(", ");
sb.append(i.data);
sb.append("]");
sb.append(i.deletedSlice().toString(cc));
sb.append("@");
sb.append(i.deletionTime());
}
return sb.toString();
}
// Updates all the timestamp of the deletion contained in this DeletionInfo to be {@code timestamp}.
public void updateAllTimestamp(long timestamp)
public DeletionInfo updateAllTimestamp(long timestamp)
{
if (topLevel.markedForDeleteAt != Long.MIN_VALUE)
topLevel = new DeletionTime(timestamp, topLevel.localDeletionTime);
if (partitionDeletion.markedForDeleteAt() != Long.MIN_VALUE)
partitionDeletion = new SimpleDeletionTime(timestamp, partitionDeletion.localDeletionTime());
if (ranges != null)
ranges.updateAllTimestamp(timestamp);
return this;
}
@Override
@ -370,100 +256,18 @@ public class DeletionInfo implements IMeasurableMemory
if(!(o instanceof DeletionInfo))
return false;
DeletionInfo that = (DeletionInfo)o;
return topLevel.equals(that.topLevel) && Objects.equal(ranges, that.ranges);
return partitionDeletion.equals(that.partitionDeletion) && Objects.equal(ranges, that.ranges);
}
@Override
public final int hashCode()
{
return Objects.hashCode(topLevel, ranges);
return Objects.hashCode(partitionDeletion, ranges);
}
@Override
public long unsharedHeapSize()
{
return EMPTY_SIZE + topLevel.unsharedHeapSize() + (ranges == null ? 0 : ranges.unsharedHeapSize());
}
public static class Serializer implements IVersionedSerializer<DeletionInfo>
{
private final RangeTombstoneList.Serializer rtlSerializer;
public Serializer(CType type)
{
this.rtlSerializer = new RangeTombstoneList.Serializer(type);
}
public void serialize(DeletionInfo info, DataOutputPlus out, int version) throws IOException
{
DeletionTime.serializer.serialize(info.topLevel, out);
rtlSerializer.serialize(info.ranges, out, version);
}
public DeletionInfo deserialize(DataInput in, int version) throws IOException
{
DeletionTime topLevel = DeletionTime.serializer.deserialize(in);
RangeTombstoneList ranges = rtlSerializer.deserialize(in, version);
return new DeletionInfo(topLevel, ranges);
}
public long serializedSize(DeletionInfo info, TypeSizes typeSizes, int version)
{
long size = DeletionTime.serializer.serializedSize(info.topLevel, typeSizes);
return size + rtlSerializer.serializedSize(info.ranges, typeSizes, version);
}
public long serializedSize(DeletionInfo info, int version)
{
return serializedSize(info, TypeSizes.NATIVE, version);
}
}
/**
* This object allow testing whether a given column (name/timestamp) is deleted
* or not by this DeletionInfo, assuming that the columns given to this
* object are passed in forward or reversed comparator sorted order.
*
* This is more efficient that calling DeletionInfo.isDeleted() repeatedly
* in that case.
*/
public class InOrderTester
{
/*
* Note that because because range tombstone are added to this DeletionInfo while we iterate,
* `ranges` may be null initially and we need to wait for the first range to create the tester (once
* created the test will pick up new tombstones however). We are guaranteed that a range tombstone
* will be added *before* we test any column that it may delete, so this is ok.
*/
private RangeTombstoneList.InOrderTester tester;
private final boolean reversed;
private InOrderTester(boolean reversed)
{
this.reversed = reversed;
}
public boolean isDeleted(Cell cell)
{
if (cell.timestamp() <= topLevel.markedForDeleteAt)
return true;
// No matter what the counter cell's timestamp is, a tombstone always takes precedence. See CASSANDRA-7346.
if (!topLevel.isLive() && cell instanceof CounterCell)
return true;
/*
* We don't optimize the reversed case for now because RangeTombstoneList
* is always in forward sorted order.
*/
if (reversed)
return DeletionInfo.this.isDeleted(cell);
// Maybe create the tester if we hadn't yet and we now have some ranges (see above).
if (tester == null && ranges != null)
tester = ranges.inOrderTester();
return tester != null && tester.isDeleted(cell);
}
return EMPTY_SIZE + partitionDeletion.unsharedHeapSize() + (ranges == null ? 0 : ranges.unsharedHeapSize());
}
}

92
src/java/org/apache/cassandra/db/DeletionTime.java
View File

@ -19,58 +19,56 @@ package org.apache.cassandra.db;
import java.io.DataInput;
import java.io.IOException;
import java.security.MessageDigest;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Objects;
import org.apache.cassandra.cache.IMeasurableMemory;
import org.apache.cassandra.io.ISerializer;
import org.apache.cassandra.io.util.DataOutputPlus;
import org.apache.cassandra.io.util.FileUtils;
import org.apache.cassandra.utils.FBUtilities;
import org.apache.cassandra.utils.ObjectSizes;
import org.codehaus.jackson.annotate.JsonIgnore;
/**
* A top-level (row) tombstone.
* Information on deletion of a storage engine object.
*/
public class DeletionTime implements Comparable<DeletionTime>, IMeasurableMemory
public abstract class DeletionTime implements Comparable<DeletionTime>, IMeasurableMemory, Aliasable<DeletionTime>
{
private static final long EMPTY_SIZE = ObjectSizes.measure(new DeletionTime(0, 0));
private static final long EMPTY_SIZE = ObjectSizes.measure(new SimpleDeletionTime(0, 0));
/**
* A special DeletionTime that signifies that there is no top-level (row) tombstone.
*/
public static final DeletionTime LIVE = new DeletionTime(Long.MIN_VALUE, Integer.MAX_VALUE);
public static final DeletionTime LIVE = new SimpleDeletionTime(Long.MIN_VALUE, Integer.MAX_VALUE);
public static final Serializer serializer = new Serializer();
/**
* A timestamp (typically in microseconds since the unix epoch, although this is not enforced) after which
* data should be considered deleted. If set to Long.MIN_VALUE, this implies that the data has not been marked
* for deletion at all.
*/
public final long markedForDeleteAt;
public abstract long markedForDeleteAt();
/**
* The local server timestamp, in seconds since the unix epoch, at which this tombstone was created. This is
* only used for purposes of purging the tombstone after gc_grace_seconds have elapsed.
*/
public final int localDeletionTime;
public static final Serializer serializer = new Serializer();
@VisibleForTesting
public DeletionTime(long markedForDeleteAt, int localDeletionTime)
{
this.markedForDeleteAt = markedForDeleteAt;
this.localDeletionTime = localDeletionTime;
}
public abstract int localDeletionTime();
/**
* Returns whether this DeletionTime is live, that is deletes no columns.
*/
@JsonIgnore
public boolean isLive()
{
return markedForDeleteAt == Long.MIN_VALUE && localDeletionTime == Integer.MAX_VALUE;
return markedForDeleteAt() == Long.MIN_VALUE && localDeletionTime() == Integer.MAX_VALUE;
}
public void digest(MessageDigest digest)
{
FBUtilities.updateWithLong(digest, markedForDeleteAt());
FBUtilities.updateWithInt(digest, localDeletionTime());
}
@Override
@ -79,48 +77,58 @@ public class DeletionTime implements Comparable<DeletionTime>, IMeasurableMemory
if(!(o instanceof DeletionTime))
return false;
DeletionTime that = (DeletionTime)o;
return markedForDeleteAt == that.markedForDeleteAt && localDeletionTime == that.localDeletionTime;
return markedForDeleteAt() == that.markedForDeleteAt() && localDeletionTime() == that.localDeletionTime();
}
@Override
public final int hashCode()
{
return Objects.hashCode(markedForDeleteAt, localDeletionTime);
return Objects.hashCode(markedForDeleteAt(), localDeletionTime());
}
@Override
public String toString()
{
return String.format("deletedAt=%d, localDeletion=%d", markedForDeleteAt, localDeletionTime);
return String.format("deletedAt=%d, localDeletion=%d", markedForDeleteAt(), localDeletionTime());
}
public int compareTo(DeletionTime dt)
{
if (markedForDeleteAt < dt.markedForDeleteAt)
if (markedForDeleteAt() < dt.markedForDeleteAt())
return -1;
else if (markedForDeleteAt > dt.markedForDeleteAt)
else if (markedForDeleteAt() > dt.markedForDeleteAt())
return 1;
else if (localDeletionTime < dt.localDeletionTime)
else if (localDeletionTime() < dt.localDeletionTime())
return -1;
else if (localDeletionTime > dt.localDeletionTime)
else if (localDeletionTime() > dt.localDeletionTime())
return -1;
else
return 0;
}
public boolean isGcAble(int gcBefore)
{
return localDeletionTime < gcBefore;
}
public boolean isDeleted(OnDiskAtom atom)
{
return atom.timestamp() <= markedForDeleteAt;
}
public boolean supersedes(DeletionTime dt)
{
return this.markedForDeleteAt > dt.markedForDeleteAt;
return markedForDeleteAt() > dt.markedForDeleteAt() || (markedForDeleteAt() == dt.markedForDeleteAt() && localDeletionTime() > dt.localDeletionTime());
}
public boolean isPurgeable(long maxPurgeableTimestamp, int gcBefore)
{
return markedForDeleteAt() < maxPurgeableTimestamp && localDeletionTime() < gcBefore;
}
public boolean deletes(LivenessInfo info)
{
return deletes(info.timestamp());
}
public boolean deletes(long timestamp)
{
return timestamp <= markedForDeleteAt();
}
public int dataSize()
{
return 12;
}
public long unsharedHeapSize()
@ -132,8 +140,8 @@ public class DeletionTime implements Comparable<DeletionTime>, IMeasurableMemory
{
public void serialize(DeletionTime delTime, DataOutputPlus out) throws IOException
{
out.writeInt(delTime.localDeletionTime);
out.writeLong(delTime.markedForDeleteAt);
out.writeInt(delTime.localDeletionTime());
out.writeLong(delTime.markedForDeleteAt());
}
public DeletionTime deserialize(DataInput in) throws IOException
@ -142,7 +150,7 @@ public class DeletionTime implements Comparable<DeletionTime>, IMeasurableMemory
long mfda = in.readLong();
return mfda == Long.MIN_VALUE && ldt == Integer.MAX_VALUE
? LIVE
: new DeletionTime(mfda, ldt);
: new SimpleDeletionTime(mfda, ldt);
}
public void skip(DataInput in) throws IOException
@ -152,8 +160,8 @@ public class DeletionTime implements Comparable<DeletionTime>, IMeasurableMemory
public long serializedSize(DeletionTime delTime, TypeSizes typeSizes)
{
return typeSizes.sizeof(delTime.localDeletionTime)
+ typeSizes.sizeof(delTime.markedForDeleteAt);
return typeSizes.sizeof(delTime.localDeletionTime())
+ typeSizes.sizeof(delTime.markedForDeleteAt());
}
}
}

153
src/java/org/apache/cassandra/db/DeletionTimeArray.java Normal file
View File

@ -0,0 +1,153 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.db;
import java.util.Arrays;
import org.apache.cassandra.utils.ObjectSizes;
/**
* Utility class to store an array of deletion times a bit efficiently.
*/
public class DeletionTimeArray
{
private long[] markedForDeleteAts;
private int[] delTimes;
public DeletionTimeArray(int initialCapacity)
{
this.markedForDeleteAts = new long[initialCapacity];
this.delTimes = new int[initialCapacity];
clear();
}
public void clear(int i)
{
markedForDeleteAts[i] = Long.MIN_VALUE;
delTimes[i] = Integer.MAX_VALUE;
}
public void set(int i, DeletionTime dt)
{
this.markedForDeleteAts[i] = dt.markedForDeleteAt();
this.delTimes[i] = dt.localDeletionTime();
}
public int size()
{
return markedForDeleteAts.length;
}
public void resize(int newSize)
{
int prevSize = size();
markedForDeleteAts = Arrays.copyOf(markedForDeleteAts, newSize);
delTimes = Arrays.copyOf(delTimes, newSize);
Arrays.fill(markedForDeleteAts, prevSize, newSize, Long.MIN_VALUE);
Arrays.fill(delTimes, prevSize, newSize, Integer.MAX_VALUE);
}
public boolean supersedes(int i, DeletionTime dt)
{
return markedForDeleteAts[i] > dt.markedForDeleteAt();
}
public boolean supersedes(int i, int j)
{
return markedForDeleteAts[i] > markedForDeleteAts[j];
}
public void swap(int i, int j)
{
long m = markedForDeleteAts[j];
int l = delTimes[j];
move(i, j);
markedForDeleteAts[i] = m;
delTimes[i] = l;
}
public void move(int i, int j)
{
markedForDeleteAts[j] = markedForDeleteAts[i];
delTimes[j] = delTimes[i];
}
public boolean isLive(int i)
{
return markedForDeleteAts[i] > Long.MIN_VALUE;
}
public void clear()
{
Arrays.fill(markedForDeleteAts, Long.MIN_VALUE);
Arrays.fill(delTimes, Integer.MAX_VALUE);
}
public int dataSize()
{
return 12 * markedForDeleteAts.length;
}
public long unsharedHeapSize()
{
return ObjectSizes.sizeOfArray(markedForDeleteAts)
+ ObjectSizes.sizeOfArray(delTimes);
}
public void copy(DeletionTimeArray other)
{
assert size() == other.size();
for (int i = 0; i < size(); i++)
{
markedForDeleteAts[i] = other.markedForDeleteAts[i];
delTimes[i] = other.delTimes[i];
}
}
public static class Cursor extends DeletionTime
{
private DeletionTimeArray array;
private int i;
public Cursor setTo(DeletionTimeArray array, int i)
{
this.array = array;
this.i = i;
return this;
}
public long markedForDeleteAt()
{
return array.markedForDeleteAts[i];
}
public int localDeletionTime()
{
return array.delTimes[i];
}
public DeletionTime takeAlias()
{
return new SimpleDeletionTime(markedForDeleteAt(), localDeletionTime());
}
}
}

44
src/java/org/apache/cassandra/db/ExpiringCell.java
View File

@ -1,44 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.db;
import org.apache.cassandra.config.CFMetaData;
import org.apache.cassandra.utils.concurrent.OpOrder;
import org.apache.cassandra.utils.memory.AbstractAllocator;
import org.apache.cassandra.utils.memory.MemtableAllocator;
/**
* Alternative to Cell that have an expiring time.
* ExpiringCell is immutable (as Cell is).
*
* Note that ExpiringCell does not override Cell.getMarkedForDeleteAt,
* which means that it's in the somewhat unintuitive position of being deleted (after its expiration)
* without having a time-at-which-it-became-deleted. (Because ttl is a server-side measurement,
* we can't mix it with the timestamp field, which is client-supplied and whose resolution we
* can't assume anything about.)
*/
public interface ExpiringCell extends Cell
{
public static final int MAX_TTL = 20 * 365 * 24 * 60 * 60; // 20 years in seconds
public int getTimeToLive();
ExpiringCell localCopy(CFMetaData metadata, AbstractAllocator allocator);
ExpiringCell localCopy(CFMetaData metaData, MemtableAllocator allocator, OpOrder.Group opGroup);
}

265
src/java/org/apache/cassandra/db/HintedHandOffManager.java
View File

@ -30,7 +30,6 @@ import javax.management.MBeanServer;
import javax.management.ObjectName;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.collect.ImmutableSortedSet;
import com.google.common.collect.Lists;
import com.google.common.util.concurrent.RateLimiter;
import com.google.common.util.concurrent.Uninterruptibles;
@ -39,17 +38,15 @@ import org.slf4j.LoggerFactory;
import org.apache.cassandra.concurrent.JMXEnabledScheduledThreadPoolExecutor;
import org.apache.cassandra.concurrent.NamedThreadFactory;
import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.config.Schema;
import org.apache.cassandra.db.compaction.CompactionManager;
import org.apache.cassandra.db.composites.*;
import org.apache.cassandra.db.rows.*;
import org.apache.cassandra.db.partitions.*;
import org.apache.cassandra.db.filter.*;
import org.apache.cassandra.db.marshal.Int32Type;
import org.apache.cassandra.db.marshal.UUIDType;
import org.apache.cassandra.dht.IPartitioner;
import org.apache.cassandra.dht.Range;
import org.apache.cassandra.dht.Token;
import org.apache.cassandra.exceptions.WriteFailureException;
import org.apache.cassandra.exceptions.WriteTimeoutException;
import org.apache.cassandra.gms.ApplicationState;
import org.apache.cassandra.gms.FailureDetector;
@ -61,6 +58,7 @@ import org.apache.cassandra.net.MessageOut;
import org.apache.cassandra.net.MessagingService;
import org.apache.cassandra.service.*;
import org.apache.cassandra.utils.*;
import org.apache.cassandra.utils.concurrent.OpOrder;
import org.cliffc.high_scale_lib.NonBlockingHashSet;
/**
@ -91,7 +89,8 @@ public class HintedHandOffManager implements HintedHandOffManagerMBean
public static final HintedHandOffManager instance = new HintedHandOffManager();
private static final Logger logger = LoggerFactory.getLogger(HintedHandOffManager.class);
private static final int PAGE_SIZE = 128;
private static final int MAX_SIMULTANEOUSLY_REPLAYED_HINTS = 128;
private static final int LARGE_NUMBER = 65536; // 64k nodes ought to be enough for anybody.
public final HintedHandoffMetrics metrics = new HintedHandoffMetrics();
@ -110,6 +109,8 @@ public class HintedHandOffManager implements HintedHandOffManagerMBean
private final ColumnFamilyStore hintStore = Keyspace.open(SystemKeyspace.NAME).getColumnFamilyStore(SystemKeyspace.HINTS);
private static final ColumnDefinition hintColumn = SystemKeyspace.Hints.compactValueColumn();
/**
* Returns a mutation representing a Hint to be sent to <code>targetId</code>
* as soon as it becomes available again.
@ -127,11 +128,20 @@ public class HintedHandOffManager implements HintedHandOffManagerMBean
UUID hintId = UUIDGen.getTimeUUID();
// serialize the hint with id and version as a composite column name
CellName name = SystemKeyspace.Hints.comparator.makeCellName(hintId, MessagingService.current_version);
PartitionUpdate upd = new PartitionUpdate(SystemKeyspace.Hints,
StorageService.getPartitioner().decorateKey(UUIDType.instance.decompose(targetId)),
PartitionColumns.of(hintColumn),
1);
Row.Writer writer = upd.writer();
Rows.writeClustering(SystemKeyspace.Hints.comparator.make(hintId, MessagingService.current_version), writer);
ByteBuffer value = ByteBuffer.wrap(FBUtilities.serialize(mutation, Mutation.serializer, MessagingService.current_version));
ColumnFamily cf = ArrayBackedSortedColumns.factory.create(Schema.instance.getCFMetaData(SystemKeyspace.NAME, SystemKeyspace.HINTS));
cf.addColumn(name, value, now, ttl);
return new Mutation(SystemKeyspace.NAME, UUIDType.instance.decompose(targetId), cf);
writer.writeCell(hintColumn, false, value, SimpleLivenessInfo.forUpdate(now, ttl, FBUtilities.nowInSeconds(), SystemKeyspace.Hints), null);
writer.endOfRow();
return new Mutation(upd);
}
/*
@ -142,12 +152,11 @@ public class HintedHandOffManager implements HintedHandOffManagerMBean
public static int calculateHintTTL(Mutation mutation)
{
int ttl = maxHintTTL;
for (ColumnFamily cf : mutation.getColumnFamilies())
ttl = Math.min(ttl, cf.metadata().getGcGraceSeconds());
for (PartitionUpdate upd : mutation.getPartitionUpdates())
ttl = Math.min(ttl, upd.metadata().getGcGraceSeconds());
return ttl;
}
public void start()
{
MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
@ -172,11 +181,17 @@ public class HintedHandOffManager implements HintedHandOffManagerMBean
executor.scheduleWithFixedDelay(runnable, 10, 10, TimeUnit.MINUTES);
}
private static void deleteHint(ByteBuffer tokenBytes, CellName columnName, long timestamp)
private static void deleteHint(ByteBuffer tokenBytes, Clustering clustering, long timestamp)
{
Mutation mutation = new Mutation(SystemKeyspace.NAME, tokenBytes);
mutation.delete(SystemKeyspace.HINTS, columnName, timestamp);
mutation.applyUnsafe(); // don't bother with commitlog since we're going to flush as soon as we're done with delivery
DecoratedKey dk = StorageService.getPartitioner().decorateKey(tokenBytes);
PartitionUpdate upd = new PartitionUpdate(SystemKeyspace.Hints, dk, PartitionColumns.of(hintColumn), 1);
Row.Writer writer = upd.writer();
Rows.writeClustering(clustering, writer);
Cells.writeTombstone(writer, hintColumn, timestamp, FBUtilities.nowInSeconds());
new Mutation(upd).applyUnsafe(); // don't bother with commitlog since we're going to flush as soon as we're done with delivery
}
public void deleteHintsForEndpoint(final String ipOrHostname)
@ -198,9 +213,8 @@ public class HintedHandOffManager implements HintedHandOffManagerMBean
if (!StorageService.instance.getTokenMetadata().isMember(endpoint))
return;
UUID hostId = StorageService.instance.getTokenMetadata().getHostId(endpoint);
ByteBuffer hostIdBytes = ByteBuffer.wrap(UUIDGen.decompose(hostId));
final Mutation mutation = new Mutation(SystemKeyspace.NAME, hostIdBytes);
mutation.delete(SystemKeyspace.HINTS, System.currentTimeMillis());
DecoratedKey dk = StorageService.getPartitioner().decorateKey(ByteBuffer.wrap(UUIDGen.decompose(hostId)));
final Mutation mutation = new Mutation(PartitionUpdate.fullPartitionDelete(SystemKeyspace.Hints, dk, System.currentTimeMillis(), FBUtilities.nowInSeconds()));
// execute asynchronously to avoid blocking caller (which may be processing gossip)
Runnable runnable = new Runnable()
@ -266,13 +280,6 @@ public class HintedHandOffManager implements HintedHandOffManagerMBean
}
}
private static boolean pagingFinished(ColumnFamily hintColumnFamily, Composite startColumn)
{
// done if no hints found or the start column (same as last column processed in previous iteration) is the only one
return hintColumnFamily == null
|| (!startColumn.isEmpty() && hintColumnFamily.getSortedColumns().size() == 1 && hintColumnFamily.getColumn((CellName)startColumn) != null);
}
private int waitForSchemaAgreement(InetAddress endpoint) throws TimeoutException
{
Gossiper gossiper = Gossiper.instance;
@ -335,6 +342,27 @@ public class HintedHandOffManager implements HintedHandOffManagerMBean
}
doDeliverHintsToEndpoint(endpoint);
// Flush all the tombstones to disk
hintStore.forceBlockingFlush();
}
private boolean checkDelivered(InetAddress endpoint, List<WriteResponseHandler<Mutation>> handlers, AtomicInteger rowsReplayed)
{
for (WriteResponseHandler<Mutation> handler : handlers)
{
try
{
handler.get();
}
catch (WriteTimeoutException e)
{
logger.info("Failed replaying hints to {}; aborting ({} delivered), error : {}",
endpoint, rowsReplayed, e.getMessage());
return false;
}
}
return true;
}
/*
@ -352,10 +380,6 @@ public class HintedHandOffManager implements HintedHandOffManagerMBean
DecoratedKey epkey = StorageService.getPartitioner().decorateKey(hostIdBytes);
final AtomicInteger rowsReplayed = new AtomicInteger(0);
Composite startColumn = Composites.EMPTY;
int pageSize = calculatePageSize();
logger.debug("Using pageSize of {}", pageSize);
// rate limit is in bytes per second. Uses Double.MAX_VALUE if disabled (set to 0 in cassandra.yaml).
// max rate is scaled by the number of nodes in the cluster (CASSANDRA-5272).
@ -363,55 +387,38 @@ public class HintedHandOffManager implements HintedHandOffManagerMBean
/ (StorageService.instance.getTokenMetadata().getAllEndpoints().size() - 1);
RateLimiter rateLimiter = RateLimiter.create(throttleInKB == 0 ? Double.MAX_VALUE : throttleInKB * 1024);
delivery:
while (true)
int nowInSec = FBUtilities.nowInSeconds();
try (OpOrder.Group op = hintStore.readOrdering.start();
RowIterator iter = UnfilteredRowIterators.filter(SinglePartitionReadCommand.fullPartitionRead(SystemKeyspace.Hints, nowInSec, epkey).queryMemtableAndDisk(hintStore, op), nowInSec))
{
long now = System.currentTimeMillis();
QueryFilter filter = QueryFilter.getSliceFilter(epkey,
SystemKeyspace.HINTS,
startColumn,
Composites.EMPTY,
false,
pageSize,
now);
ColumnFamily hintsPage = ColumnFamilyStore.removeDeleted(hintStore.getColumnFamily(filter), (int) (now / 1000));
if (pagingFinished(hintsPage, startColumn))
{
logger.info("Finished hinted handoff of {} rows to endpoint {}", rowsReplayed, endpoint);
break;
}
// check if node is still alive and we should continue delivery process
if (!FailureDetector.instance.isAlive(endpoint))
{
logger.info("Endpoint {} died during hint delivery; aborting ({} delivered)", endpoint, rowsReplayed);
break;
}
List<WriteResponseHandler<Mutation>> responseHandlers = Lists.newArrayList();
for (final Cell hint : hintsPage)
while (iter.hasNext())
{
// check if node is still alive and we should continue delivery process
if (!FailureDetector.instance.isAlive(endpoint))
{
logger.info("Endpoint {} died during hint delivery; aborting ({} delivered)", endpoint, rowsReplayed);
return;
}
// check if hints delivery has been paused during the process
if (hintedHandOffPaused)
{
logger.debug("Hints delivery process is paused, aborting");
break delivery;
return;
}
// Skip tombstones:
// if we iterate quickly enough, it's possible that we could request a new page in the same millisecond
// in which the local deletion timestamp was generated on the last column in the old page, in which
// case the hint will have no columns (since it's deleted) but will still be included in the resultset
// since (even with gcgs=0) it's still a "relevant" tombstone.
if (!hint.isLive())
continue;
// Wait regularly on the endpoint acknowledgment. If we timeout on it, the endpoint is probably dead so stop delivery
if (responseHandlers.size() > MAX_SIMULTANEOUSLY_REPLAYED_HINTS && !checkDelivered(endpoint, responseHandlers, rowsReplayed))
return;
startColumn = hint.name();
final Row hint = iter.next();
int version = Int32Type.instance.compose(hint.clustering().get(1));
Cell cell = hint.getCell(hintColumn);
int version = Int32Type.instance.compose(hint.name().get(1));
DataInputStream in = new DataInputStream(ByteBufferUtil.inputStream(hint.value()));
final long timestamp = cell.livenessInfo().timestamp();
DataInputStream in = new DataInputStream(ByteBufferUtil.inputStream(cell.value()));
Mutation mutation;
try
{
@ -420,7 +427,7 @@ public class HintedHandOffManager implements HintedHandOffManagerMBean
catch (UnknownColumnFamilyException e)
{
logger.debug("Skipping delivery of hint for deleted table", e);
deleteHint(hostIdBytes, hint.name(), hint.timestamp());
deleteHint(hostIdBytes, hint.clustering(), timestamp);
continue;
}
catch (IOException e)
@ -430,7 +437,7 @@ public class HintedHandOffManager implements HintedHandOffManagerMBean
for (UUID cfId : mutation.getColumnFamilyIds())
{
if (hint.timestamp() <= SystemKeyspace.getTruncatedAt(cfId))
if (timestamp <= SystemKeyspace.getTruncatedAt(cfId))
{
logger.debug("Skipping delivery of hint for truncated table {}", cfId);
mutation = mutation.without(cfId);
@ -439,7 +446,7 @@ public class HintedHandOffManager implements HintedHandOffManagerMBean
if (mutation.isEmpty())
{
deleteHint(hostIdBytes, hint.name(), hint.timestamp());
deleteHint(hostIdBytes, hint.clustering(), timestamp);
continue;
}
@ -450,7 +457,7 @@ public class HintedHandOffManager implements HintedHandOffManagerMBean
public void run()
{
rowsReplayed.incrementAndGet();
deleteHint(hostIdBytes, hint.name(), hint.timestamp());
deleteHint(hostIdBytes, hint.clustering(), timestamp);
}
};
WriteResponseHandler<Mutation> responseHandler = new WriteResponseHandler<>(endpoint, WriteType.SIMPLE, callback);
@ -458,38 +465,10 @@ public class HintedHandOffManager implements HintedHandOffManagerMBean
responseHandlers.add(responseHandler);
}
for (WriteResponseHandler<Mutation> handler : responseHandlers)
{
try
{
handler.get();
}
catch (WriteTimeoutException|WriteFailureException e)
{
logger.info("Failed replaying hints to {}; aborting ({} delivered), error : {}",
endpoint, rowsReplayed, e.getMessage());
break delivery;
}
}
// Wait on the last handlers
if (checkDelivered(endpoint, responseHandlers, rowsReplayed))
logger.info("Finished hinted handoff of {} rows to endpoint {}", rowsReplayed, endpoint);
}
// Flush all the tombstones to disk
hintStore.forceBlockingFlush();
}
// read less columns (mutations) per page if they are very large
private int calculatePageSize()
{
int meanColumnCount = hintStore.getMeanColumns();
if (meanColumnCount <= 0)
return PAGE_SIZE;
int averageColumnSize = (int) (hintStore.metric.meanRowSize.getValue() / meanColumnCount);
if (averageColumnSize <= 0)
return PAGE_SIZE;
// page size of 1 does not allow actual paging b/c of >= behavior on startColumn
return Math.max(2, Math.min(PAGE_SIZE, 4 * 1024 * 1024 / averageColumnSize));
}
/**
@ -505,18 +484,26 @@ public class HintedHandOffManager implements HintedHandOffManagerMBean
// to deliver to).
compact();
IPartitioner p = StorageService.getPartitioner();
RowPosition minPos = p.getMinimumToken().minKeyBound();
Range<RowPosition> range = new Range<>(minPos, minPos);
IDiskAtomFilter filter = new NamesQueryFilter(ImmutableSortedSet.<CellName>of());
List<Row> rows = hintStore.getRangeSlice(range, null, filter, Integer.MAX_VALUE, System.currentTimeMillis());
for (Row row : rows)
ReadCommand cmd = new PartitionRangeReadCommand(hintStore.metadata,
FBUtilities.nowInSeconds(),
ColumnFilter.all(hintStore.metadata),
RowFilter.NONE,
DataLimits.cqlLimits(Integer.MAX_VALUE, 1),
DataRange.allData(StorageService.getPartitioner()));
try (ReadOrderGroup orderGroup = cmd.startOrderGroup(); UnfilteredPartitionIterator iter = cmd.executeLocally(orderGroup))
{
UUID hostId = UUIDGen.getUUID(row.key.getKey());
InetAddress target = StorageService.instance.getTokenMetadata().getEndpointForHostId(hostId);
// token may have since been removed (in which case we have just read back a tombstone)
if (target != null)
scheduleHintDelivery(target, false);
while (iter.hasNext())
{
try (UnfilteredRowIterator partition = iter.next())
{
UUID hostId = UUIDGen.getUUID(partition.partitionKey().getKey());
InetAddress target = StorageService.instance.getTokenMetadata().getEndpointForHostId(hostId);
// token may have since been removed (in which case we have just read back a tombstone)
if (target != null)
scheduleHintDelivery(target, false);
}
}
}
logger.debug("Finished scheduleAllDeliveries");
@ -572,42 +559,20 @@ public class HintedHandOffManager implements HintedHandOffManagerMBean
// Extract the keys as strings to be reported.
LinkedList<String> result = new LinkedList<>();
for (Row row : getHintsSlice(1))
ReadCommand cmd = PartitionRangeReadCommand.allDataRead(SystemKeyspace.Hints, FBUtilities.nowInSeconds());
try (ReadOrderGroup orderGroup = cmd.startOrderGroup(); UnfilteredPartitionIterator iter = cmd.executeLocally(orderGroup))
{
if (row.cf != null) //ignore removed rows
result.addFirst(tokenFactory.toString(row.key.getToken()));
while (iter.hasNext())
{
try (UnfilteredRowIterator partition = iter.next())
{
// We don't delete by range on the hints table, so we don't have to worry about the
// iterator returning only range tombstone marker
if (partition.hasNext())
result.addFirst(tokenFactory.toString(partition.partitionKey().getToken()));
}
}
}
return result;
}
private List<Row> getHintsSlice(int columnCount)
{
// Get count # of columns...
SliceQueryFilter predicate = new SliceQueryFilter(ColumnSlice.ALL_COLUMNS_ARRAY,
false,
columnCount);
// From keys "" to ""...
IPartitioner partitioner = StorageService.getPartitioner();
RowPosition minPos = partitioner.getMinimumToken().minKeyBound();
Range<RowPosition> range = new Range<>(minPos, minPos);
try
{
RangeSliceCommand cmd = new RangeSliceCommand(SystemKeyspace.NAME,
SystemKeyspace.HINTS,
System.currentTimeMillis(),
predicate,
range,
null,
LARGE_NUMBER);
return StorageProxy.getRangeSlice(cmd, ConsistencyLevel.ONE);
}
catch (Exception e)
{
logger.info("HintsCF getEPPendingHints timed out.");
throw new RuntimeException(e);
}
}
}

7
src/java/org/apache/cassandra/db/IMutation.java
View File

@ -21,13 +21,14 @@ import java.nio.ByteBuffer;
import java.util.Collection;
import java.util.UUID;
import org.apache.cassandra.db.partitions.PartitionUpdate;
public interface IMutation
{
public String getKeyspaceName();
public Collection<UUID> getColumnFamilyIds();
public ByteBuffer key();
public DecoratedKey key();
public long getTimeout();
public String toString(boolean shallow);
public void addAll(IMutation m);
public Collection<ColumnFamily> getColumnFamilies();
public Collection<PartitionUpdate> getPartitionUpdates();
}

121
src/java/org/apache/cassandra/db/IndexExpression.java
View File

@ -1,121 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package org.apache.cassandra.db;
import java.io.DataInput;
import java.io.IOException;
import java.nio.ByteBuffer;
import com.google.common.base.Objects;
import org.apache.cassandra.cql3.Operator;
import org.apache.cassandra.io.util.DataOutputPlus;
import org.apache.cassandra.utils.ByteBufferUtil;
public final class IndexExpression
{
public final ByteBuffer column;
public final Operator operator;
public final ByteBuffer value;
public IndexExpression(ByteBuffer column, Operator operator, ByteBuffer value)
{
this.column = column;
this.operator = operator;
this.value = value;
}
/**
* Checks if the operator of this <code>IndexExpression</code> is a <code>CONTAINS</code> operator.
*
* @return <code>true</code> if the operator of this <code>IndexExpression</code> is a <code>CONTAINS</code>
* operator, <code>false</code> otherwise.
*/
public boolean isContains()
{
return Operator.CONTAINS == operator;
}
/**
* Checks if the operator of this <code>IndexExpression</code> is a <code>CONTAINS_KEY</code> operator.
*
* @return <code>true</code> if the operator of this <code>IndexExpression</code> is a <code>CONTAINS_KEY</code>
* operator, <code>false</code> otherwise.
*/
public boolean isContainsKey()
{
return Operator.CONTAINS_KEY == operator;
}
@Override
public String toString()
{
return String.format("%s %s %s", ByteBufferUtil.bytesToHex(column), operator, ByteBufferUtil.bytesToHex(value));
}
@Override
public boolean equals(Object o)
{
if (this == o)
return true;
if (!(o instanceof IndexExpression))
return false;
IndexExpression ie = (IndexExpression) o;
return Objects.equal(this.column, ie.column)
&& Objects.equal(this.operator, ie.operator)
&& Objects.equal(this.value, ie.value);
}
@Override
public int hashCode()
{
return Objects.hashCode(column, operator, value);
}
/**
* Write the serialized version of this <code>IndexExpression</code> to the specified output.
*
* @param output the output to write to
* @throws IOException if an I/O problem occurs while writing to the specified output
*/
public void writeTo(DataOutputPlus output) throws IOException
{
ByteBufferUtil.writeWithShortLength(column, output);
operator.writeTo(output);
ByteBufferUtil.writeWithShortLength(value, output);
}
/**
* Deserializes an <code>IndexExpression</code> instance from the specified input.
*
* @param input the input to read from
* @return the <code>IndexExpression</code> instance deserialized
* @throws IOException if a problem occurs while deserializing the <code>IndexExpression</code> instance.
*/
public static IndexExpression readFrom(DataInput input) throws IOException
{
return new IndexExpression(ByteBufferUtil.readWithShortLength(input),
Operator.readFrom(input),
ByteBufferUtil.readWithShortLength(input));
}
}

61
src/java/org/apache/cassandra/db/Keyspace.java
View File

@ -30,18 +30,19 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.cassandra.config.*;
import org.apache.cassandra.db.rows.UnfilteredRowIterator;
import org.apache.cassandra.db.commitlog.CommitLog;
import org.apache.cassandra.db.commitlog.ReplayPosition;
import org.apache.cassandra.db.compaction.CompactionManager;
import org.apache.cassandra.db.filter.QueryFilter;
import org.apache.cassandra.db.index.SecondaryIndex;
import org.apache.cassandra.db.index.SecondaryIndexManager;
import org.apache.cassandra.db.partitions.PartitionUpdate;
import org.apache.cassandra.io.sstable.format.SSTableReader;
import org.apache.cassandra.locator.AbstractReplicationStrategy;
import org.apache.cassandra.service.StorageService;
import org.apache.cassandra.service.pager.QueryPagers;
import org.apache.cassandra.tracing.Tracing;
import org.apache.cassandra.utils.concurrent.OpOrder;
import org.apache.cassandra.utils.FBUtilities;
import org.apache.cassandra.metrics.KeyspaceMetrics;
/**
@ -49,8 +50,6 @@ import org.apache.cassandra.metrics.KeyspaceMetrics;
*/
public class Keyspace
{
private static final int DEFAULT_PAGE_SIZE = 10000;
private static final Logger logger = LoggerFactory.getLogger(Keyspace.class);
private static final String TEST_FAIL_WRITES_KS = System.getProperty("cassandra.test.fail_writes_ks", "");
@ -145,6 +144,11 @@ public class Keyspace
}
}
public static ColumnFamilyStore openAndGetStore(CFMetaData cfm)
{
return open(cfm.ksName).getColumnFamilyStore(cfm.cfId);
}
/**
* Removes every SSTable in the directory from the appropriate Tracker's view.
* @param directory the unreadable directory, possibly with SSTables in it, but not necessarily.
@ -347,13 +351,6 @@ public class Keyspace
}
}
public Row getRow(QueryFilter filter)
{
ColumnFamilyStore cfStore = getColumnFamilyStore(filter.getColumnFamilyName());
ColumnFamily columnFamily = cfStore.getColumnFamily(filter);
return new Row(filter.key, columnFamily);
}
public void apply(Mutation mutation, boolean writeCommitLog)
{
apply(mutation, writeCommitLog, true);
@ -372,6 +369,7 @@ public class Keyspace
if (TEST_FAIL_WRITES && metadata.name.equals(TEST_FAIL_WRITES_KS))
throw new RuntimeException("Testing write failures");
int nowInSec = FBUtilities.nowInSeconds();
try (OpOrder.Group opGroup = writeOrder.start())
{
// write the mutation to the commitlog and memtables
@ -382,21 +380,21 @@ public class Keyspace
replayPosition = CommitLog.instance.add(mutation);
}
DecoratedKey key = StorageService.getPartitioner().decorateKey(mutation.key());
for (ColumnFamily cf : mutation.getColumnFamilies())
DecoratedKey key = mutation.key();
for (PartitionUpdate upd : mutation.getPartitionUpdates())
{
ColumnFamilyStore cfs = columnFamilyStores.get(cf.id());
ColumnFamilyStore cfs = columnFamilyStores.get(upd.metadata().cfId);
if (cfs == null)
{
logger.error("Attempting to mutate non-existant table {}", cf.id());
logger.error("Attempting to mutate non-existant table {}", upd.metadata().cfId);
continue;
}
Tracing.trace("Adding to {} memtable", cf.metadata().cfName);
Tracing.trace("Adding to {} memtable", upd.metadata().cfName);
SecondaryIndexManager.Updater updater = updateIndexes
? cfs.indexManager.updaterFor(key, cf, opGroup)
? cfs.indexManager.updaterFor(upd, opGroup, nowInSec)
: SecondaryIndexManager.nullUpdater;
cfs.apply(key, cf, updater, opGroup, replayPosition);
cfs.apply(upd, updater, opGroup, replayPosition);
}
}
}
@ -408,30 +406,21 @@ public class Keyspace
/**
* @param key row to index
* @param cfs ColumnFamily to index row in
* @param cfs ColumnFamily to index partition in
* @param idxNames columns to index, in comparator order
*/
public static void indexRow(DecoratedKey key, ColumnFamilyStore cfs, Set<String> idxNames)
public static void indexPartition(DecoratedKey key, ColumnFamilyStore cfs, Set<String> idxNames)
{
if (logger.isDebugEnabled())
logger.debug("Indexing row {} ", cfs.metadata.getKeyValidator().getString(key.getKey()));
logger.debug("Indexing partition {} ", cfs.metadata.getKeyValidator().getString(key.getKey()));
try (OpOrder.Group opGroup = cfs.keyspace.writeOrder.start())
Set<SecondaryIndex> indexes = cfs.indexManager.getIndexesByNames(idxNames);
SinglePartitionReadCommand cmd = SinglePartitionReadCommand.fullPartitionRead(cfs.metadata, FBUtilities.nowInSeconds(), key);
try (OpOrder.Group opGroup = cfs.keyspace.writeOrder.start();
UnfilteredRowIterator partition = cmd.queryMemtableAndDisk(cfs, opGroup))
{
Set<SecondaryIndex> indexes = cfs.indexManager.getIndexesByNames(idxNames);
Iterator<ColumnFamily> pager = QueryPagers.pageRowLocally(cfs, key.getKey(), DEFAULT_PAGE_SIZE);
while (pager.hasNext())
{
ColumnFamily cf = pager.next();
ColumnFamily cf2 = cf.cloneMeShallow();
for (Cell cell : cf)
{
if (cfs.indexManager.indexes(cell.name(), indexes))
cf2.addColumn(cell);
}
cfs.indexManager.indexRow(key.getKey(), cf2, opGroup);
}
cfs.indexManager.indexPartition(partition, opGroup, indexes, cmd.nowInSec());
}
}

Some files were not shown because too many files have changed in this diff Show More