natas_hw
/
cassandra
mirror of https://github.com/apache/cassandra
0
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

CEP-20: Dynamic Data Masking 

Allows to attach the native masking functions and UDFs to the definitions of
table columns in the schema, as defined by CEP-20.

The functions masking a column can be specified on CREATE TABLE queries, right
at the end of the column definition. The mask of a column can also be changed
or dropped with an ALTER TABLE query. Once a column is masked, SELECT queries
will always return the masked value of the column. That masking is done on the
coordinator, at the end of the query execution. Thus, masking won't affect any
filters or ordering, which would be based on the clear values of the masked
columns. Column masks are stored on the table system_schema.column_masks.

A new UNMASK permission allows to see the clear data of columns with an
attached mask. Also, a new SELECT_MASKED permission allows to run SELECT
queries restricting the clear values of masked columns. Superusers have both
permissions by default, whereas regular users don't have them.

Dynamic data masking can be enabled/disabled with the config property
dynamic_data_masking_enabled in cassandra.yaml. It is disabled by default.

This is the combination of multiple tickets:

 * Add masking functions to column metadata (CASSANDRA-18068)
 * Add UNMASK permission (CASSANDRA-18069)
 * Add SELECT_MASKED permission (CASSANDRA-18070)
 * Add support for using UDFs as masking functions (CASSANDRA-18071)
 * Add feature flag for dynamic data masking (CASSANDRA-18316)

patch by Andrés de la Peña; reviewed by Benjamin Lerer and Berenguer Blasi for CASSANDRA-17940
This commit is contained in:
Andrés de la Peña 2022-12-07 17:28:24 +00:00
parent de1ff6d4d1
commit 2e2a49fcdc
107 changed files with 4916 additions and 386 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
CHANGES.txt
View File

@ -1,4 +1,5 @@
5.0
* CEP-20: Dynamic Data Masking (CASSANDRA-17940)
* Add system_views.snapshots virtual table (CASSANDRA-18102)
* Update OpenHFT dependencies (chronicle-queue, chronicle-core, chronicle-bytes, chronicle-wire, chronicle-threads) (CASSANDRA-18049)
* Remove org.apache.cassandra.hadoop code (CASSANDRA-18323)

8
NEWS.txt
View File

@ -132,6 +132,14 @@ New features
- Added local read/write ratio to tablestats.
- Added system_views.max_sstable_size and system_views.max_sstable_duration tables.
- Added virtual table system_views.snapshots to see all snapshots from CQL shell.
- Added support for attaching CQL dynamic data masking functions to table columns on the schema. These masking
functions can be attached to or dettached from columns with CREATE/ALTER TABLE statements. The functions obscure
the masked data during queries, but they don't change the stored data.
- Added new UNMASK permission. It allows to see the clear data of columns with an attached mask. Superusers have it
by default, whereas regular users don't have it by default.
- Added new SELECT_MASKED permission. It allows to run SELECT queries selecting the clear values of masked columns.
Superusers have it by default, whereas regular users don't have it by default.
- Added support for using UDFs as masking functions attached to table columns on the schema.
Upgrading
---------

8
conf/cassandra.yaml
View File

@ -1660,6 +1660,14 @@ report_unconfirmed_repaired_data_mismatches: false
# warming of auth caches prior to node completing startup. See CASSANDRA-16958
# auth_cache_warming_enabled: false
# If enabled, dynamic data masking allows to attach CQL masking functions to the columns of a table.
# Users without the UNMASK permission will see an obscured version of the values of the columns with an attached mask.
# If dynamic data masking is disabled it won't be allowed to create new column masks, although it will still be possible
# to drop any previously existing masks. Also, any existing mask will be ignored at query time, so all users will see
# the clear values of the masked columns.
# Defaults to false to disable dynamic data masking.
# dynamic_data_masking_enabled: false
#########################
# EXPERIMENTAL FEATURES #
#########################

28
doc/cql3/CQL.textile
View File

@ -252,9 +252,11 @@ bc(syntax)..
'(' <column-definition> ( ',' <column-definition> )* ')'
( WITH <option> ( AND <option>)* )?
<column-definition> ::= <identifier> <type> ( STATIC )? ( PRIMARY KEY )?
<column-definition> ::= <identifier> <type> ( STATIC )? ( <column_mask> )? ( PRIMARY KEY )?
| PRIMARY KEY '(' <partition-key> ( ',' <identifier> )* ')'
<column-mask> ::= MASKED WITH ( DEFAULT | <function> '(' ( <term> (',' <term>)* )? ')' )
<partition-key> ::= <identifier>
| '(' <identifier> (',' <identifier> )* ')'
@ -410,11 +412,15 @@ __Syntax:__
bc(syntax)..
<alter-table-stmt> ::= ALTER (TABLE | COLUMNFAMILY) (IF NOT EXISTS)? <tablename> <instruction>
<instruction> ::= ADD (IF NOT EXISTS)? ( <identifier> <type> ( , <identifier> <type> )* )
<instruction> ::= ADD (IF NOT EXISTS)? ( <identifier> <type> ( <column-mask> )?
( , <identifier> <type> ( <column-mask> )? )* )
| DROP (IF EXISTS)? ( <identifier> ( , <identifier> )* )
| ALTER <identifier> ( <column-mask> | DROP MASKED )
| RENAME (IF EXISTS)? <identifier> to <identifier> (AND <identifier> to <identifier>)*
| DROP COMPACT STORAGE
| WITH <option> ( AND <option> )*
<column-mask> ::= MASKED WITH ( DEFAULT | <function> '(' ( <term> (',' <term>)* )? ')' )
p.
__Sample:__
@ -434,6 +440,7 @@ If the table does not exist, the statement will return an error, unless @IF EXIS
The @<tablename>@ is the table name optionally preceded by the keyspace name. The @<instruction>@ defines the alteration to perform:
* @ADD@: Adds a new column to the table. The @<identifier>@ for the new column must not conflict with an existing column. Moreover, columns cannot be added to tables defined with the @COMPACT STORAGE@ option. If the new column already exists, the statement will return an error, unless @IF NOT EXISTS@ is used in which case the operation is a no-op.
* @DROP@: Removes a column from the table. Dropped columns will immediately become unavailable in the queries and will not be included in compacted sstables in the future. If a column is readded, queries won't return values written before the column was last dropped. It is assumed that timestamps represent actual time, so if this is not your case, you should NOT readd previously dropped columns. Columns can't be dropped from tables defined with the @COMPACT STORAGE@ option. If the dropped column does not already exist, the statement will return an error, unless @IF EXISTS@ is used in which case the operation is a no-op.
* @ALTER@: Alters an existing column. It can be used to set its data mask, or to set it as unmasked. The data mask is any function meant to obscure the real values of the column.
* @RENAME@ a primary key column of a table. Non primary key columns cannot be renamed. Furthermore, renaming a column to another name which already exists isn't allowed. It's important to keep in mind that renamed columns shouldn't have dependent secondary indexes. If the renamed column does not already exist, the statement will return an error, unless @IF EXISTS@ is used in which case the operation is a no-op.
* @DROP COMPACT STORAGE@: Removes Thrift compatibility mode from the table.
* @WITH@: Allows to update the options of the table. The "supported @<option>@":#createTableOptions (and syntax) are the same as for the @CREATE TABLE@ statement except that @COMPACT STORAGE@ is not supported. Note that setting any @compaction@ sub-options has the effect of erasing all previous @compaction@ options, so you need to re-specify all the sub-options if you want to keep them. The same note applies to the set of @compression@ sub-options.
@ -1518,6 +1525,8 @@ The full set of available permissions is:
* @AUTHORIZE@
* @DESCRIBE@
* @EXECUTE@
* @UNMASK@
* @SELECT_MASKED@
Not all permissions are applicable to every type of resource. For instance, @EXECUTE@ is only relevant in the context of functions or mbeans; granting @EXECUTE@ on a resource representing a table is nonsensical. Attempting to @GRANT@ a permission on resource to which it cannot be applied results in an error response. The following illustrates which permissions can be granted on which types of resource, and which statements are enabled by that permission.
@ -1577,6 +1586,12 @@ Not all permissions are applicable to every type of resource. For instance, @EXE
| @EXECUTE@ | @ALL MBEANS@ |Execute operations on any mbean|
| @EXECUTE@ | @MBEANS@ |Execute operations on any mbean matching a wildcard pattern|
| @EXECUTE@ | @MBEAN@ |Execute operations on named mbean|
| @UNMASK@ | @ALL KEYSPACES@ |See the clear contents of masked columns on any table|
| @UNMASK@ | @KEYSPACE@ |See the clear contents of masked columns on any table in keyspace|
| @UNMASK@ | @TABLE@ |See the clear contents of masked columns on the specified table|
| @SELECT_MASKED@ | @ALL KEYSPACES@ |Select restricting masked columns on any table|
| @SELECT_MASKED@ | @KEYSPACE@ |Select restricting masked columns on any table in keyspace|
| @SELECT_MASKED@ | @TABLE@ |Select restricting masked columns on the specified table|
h3(#grantPermissionsStmt). GRANT PERMISSION
@ -1586,7 +1601,7 @@ __Syntax:__
bc(syntax)..
<grant-permission-stmt> ::= GRANT ( ALL ( PERMISSIONS )? | <permission> ( PERMISSION )? (, PERMISSION)* ) ON <resource> TO <identifier>
<permission> ::= CREATE | ALTER | DROP | SELECT | MODIFY | AUTHORIZE | DESRIBE | EXECUTE
<permission> ::= CREATE | ALTER | DROP | SELECT | MODIFY | AUTHORIZE | DESRIBE | EXECUTE | UNMASK | SELECT_MASKED
<resource> ::= ALL KEYSPACES
| KEYSPACE <identifier>
@ -1643,7 +1658,7 @@ __Syntax:__
bc(syntax)..
<revoke-permission-stmt> ::= REVOKE ( ALL ( PERMISSIONS )? | <permission> ( PERMISSION )? (, PERMISSION)* ) ON <resource> FROM <identifier>
<permission> ::= CREATE | ALTER | DROP | SELECT | MODIFY | AUTHORIZE | DESRIBE | EXECUTE
<permission> ::= CREATE | ALTER | DROP | SELECT | MODIFY | AUTHORIZE | DESRIBE | EXECUTE | UNMASK | SELECT_MASKED
<resource> ::= ALL KEYSPACES
| KEYSPACE <identifier>
@ -2484,6 +2499,7 @@ CQL distinguishes between _reserved_ and _non-reserved_ keywords. Reserved keywo
| @LIST@ | no |
| @LOGIN@ | no |
| @MAP@ | no |
| @MASKED@ | no |
| @MATERIALIZED@ | yes |
| @MBEAN@ | yes |
| @MBEANS@ | yes |
@ -2513,6 +2529,7 @@ CQL distinguishes between _reserved_ and _non-reserved_ keywords. Reserved keywo
| @ROLES@ | no |
| @SCHEMA@ | yes |
| @SELECT@ | yes |
| @SELECT_MASKED@| no |
| @SET@ | yes |
| @SFUNC@ | no |
| @SMALLINT@ | no |
@ -2534,6 +2551,7 @@ CQL distinguishes between _reserved_ and _non-reserved_ keywords. Reserved keywo
| @TUPLE@ | no |
| @TYPE@ | no |
| @UNLOGGED@ | yes |
| @UNMASK@ | no |
| @UNSET@ | yes |
| @UPDATE@ | yes |
| @USE@ | yes |
@ -2548,7 +2566,7 @@ CQL distinguishes between _reserved_ and _non-reserved_ keywords. Reserved keywo
| @WHERE@ | yes |
| @WITH@ | yes |
| @WRITETIME@ | no |
| @MAXWRITETIME@ | no |
| @MAXWRITETIME@ | no |
h2(#appendixB). Appendix B: CQL Reserved Types

5
doc/modules/cassandra/examples/BNF/alter_table.bnf
View File

@ -1,5 +1,8 @@
alter_table_statement::= ALTER TABLE [ IF EXISTS ] table_name alter_table_instruction
alter_table_instruction::= ADD [ IF NOT EXISTS ] column_name cql_type ( ',' column_name cql_type )*
alter_table_instruction::= ADD [ IF NOT EXISTS ] column_definition ( ',' column_definition)*
| DROP [ IF EXISTS ] column_name ( ',' column_name )*
| RENAME [ IF EXISTS ] column_name to column_name (AND column_name to column_name)*
| ALTER [ IF EXISTS ] column_name ( column_mask | DROP MASKED )
| WITH options
column_definition::= column_name cql_type [ column_mask]
column_mask::= MASKED WITH ( DEFAULT | function_name '(' term ( ',' term )* ')' )

5
doc/modules/cassandra/examples/BNF/create_table.bnf
View File

@ -2,11 +2,12 @@ create_table_statement::= CREATE TABLE [ IF NOT EXISTS ] table_name '('
column_definition ( ',' column_definition )*
[ ',' PRIMARY KEY '(' primary_key ')' ]
')' [ WITH table_options ]
column_definition::= column_name cql_type [ STATIC ] [ PRIMARY KEY]
column_definition::= column_name cql_type [ STATIC ] [ column_mask ] [ PRIMARY KEY]
column_mask::= MASKED WITH ( DEFAULT | function_name '(' term ( ',' term )* ')' )
primary_key::= partition_key [ ',' clustering_columns ]
partition_key::= column_name | '(' column_name ( ',' column_name )* ')'
clustering_columns::= column_name ( ',' column_name )*
table_options:=: COMPACT STORAGE [ AND table_options ]
table_options::= COMPACT STORAGE [ AND table_options ]
| CLUSTERING ORDER BY '(' clustering_order ')'
[ AND table_options ] | options
clustering_order::= column_name (ASC | DESC) ( ',' column_name (ASC | DESC) )*

2
doc/modules/cassandra/examples/BNF/grant_permission_statement.bnf
View File

@ -1,6 +1,6 @@
grant_permission_statement ::= GRANT permissions ON resource TO role_name
permissions ::= ALL [ PERMISSIONS ] | permission [ PERMISSION ]
permission ::= CREATE | ALTER | DROP | SELECT | MODIFY | AUTHORIZE | DESCRIBE | EXECUTE
permission ::= CREATE | ALTER | DROP | SELECT | MODIFY | AUTHORIZE | DESCRIBE | EXECUTE | UNMASK | SELECT_MASKED
resource ::= ALL KEYSPACES
| KEYSPACE keyspace_name
| [ TABLE ] table_name

1
doc/modules/cassandra/examples/CQL/ddm_alter_mask.cql Normal file
View File

@ -0,0 +1 @@
ALTER TABLE patients ALTER name MASKED WITH mask_default();

5
doc/modules/cassandra/examples/CQL/ddm_create_table.cql Normal file
View File

@ -0,0 +1,5 @@
CREATE TABLE patients (
id timeuuid PRIMARY KEY,
name text MASKED WITH mask_inner(1, null),
birth date MASKED WITH mask_default()
);

11
doc/modules/cassandra/examples/CQL/ddm_create_table_with_udf.cql Normal file
View File

@ -0,0 +1,11 @@
CREATE FUNCTION redact(input text)
CALLED ON NULL INPUT
RETURNS text
LANGUAGE java
AS 'return "redacted";
CREATE TABLE patients (
id timeuuid PRIMARY KEY,
name text MASKED WITH redact(),
birth date
);

6
doc/modules/cassandra/examples/CQL/ddm_create_users.cql Normal file
View File

@ -0,0 +1,6 @@
CREATE USER privileged WITH PASSWORD 'xyz';
GRANT SELECT ON TABLE patients TO privileged;
GRANT UNMASK ON TABLE patients TO privileged;
CREATE USER unprivileged WITH PASSWORD 'xyz';
GRANT SELECT ON TABLE patients TO unprivileged;

1
doc/modules/cassandra/examples/CQL/ddm_drop_mask.cql Normal file
View File

@ -0,0 +1 @@
ALTER TABLE patients ALTER name DROP MASKED;

2
doc/modules/cassandra/examples/CQL/ddm_insert_data.cql Normal file
View File

@ -0,0 +1,2 @@
INSERT INTO patients(id, name, birth) VALUES (now(), 'alice', '1984-01-02');
INSERT INTO patients(id, name, birth) VALUES (now(), 'bob', '1982-02-03');

1
doc/modules/cassandra/examples/CQL/ddm_revoke_unmask.cql Normal file
View File

@ -0,0 +1 @@
REVOKE UNMASK ON TABLE patients FROM privileged;

6
doc/modules/cassandra/examples/CQL/ddm_select_with_masked_columns.cql Normal file
View File

@ -0,0 +1,6 @@
SELECT name, birth FROM patients;
// name | birth
// -------+------------
// a**** | 1970-01-01
// b** | 1970-01-01

8
doc/modules/cassandra/examples/CQL/ddm_select_with_select_masked.cql Normal file
View File

@ -0,0 +1,8 @@
CREATE USER trusted_user WITH PASSWORD 'xyz';
GRANT SELECT, SELECT_MASKED ON TABLE patients TO trusted_user;
LOGIN trusted_user
SELECT name, birth FROM patients WHERE name = 'Alice' ALLOW FILTERING;
// name | birth
// -------+------------
// a**** | 1970-01-01

7
doc/modules/cassandra/examples/CQL/ddm_select_with_unmask_permission.cql Normal file
View File

@ -0,0 +1,7 @@
LOGIN privileged
SELECT name, birth FROM patients;
// name | birth
// -------+------------
// alice | 1984-01-02
// bob | 1982-02-03

6
doc/modules/cassandra/examples/CQL/ddm_select_without_select_masked.cql Normal file
View File

@ -0,0 +1,6 @@
CREATE USER untrusted_user WITH PASSWORD 'xyz';
GRANT SELECT ON TABLE patients TO untrusted_user;
LOGIN untrusted_user
SELECT name, birth FROM patients WHERE name = 'Alice' ALLOW FILTERING;
// Unauthorized: Error from server: code=2100 [Unauthorized] message="User untrusted_user has no UNMASK nor SELECT_UNMASK permission on table k.patients"

7
doc/modules/cassandra/examples/CQL/ddm_select_without_unmask_permission.cql Normal file
View File

@ -0,0 +1,7 @@
LOGIN unprivileged
SELECT name, birth FROM patients;
// name | birth
// -------+------------
// a**** | 1970-01-01
// b** | 1970-01-01

15
doc/modules/cassandra/examples/CQL/select_with_mask_functions.cql Normal file
View File

@ -0,0 +1,15 @@
CREATE TABLE patients (
id timeuuid PRIMARY KEY,
name text,
birth date
);
INSERT INTO patients(id, name, birth) VALUES (now(), 'alice', '1982-01-02');
INSERT INTO patients(id, name, birth) VALUES (now(), 'bob', '1982-01-02');
SELECT mask_inner(name, 1, null), mask_default(birth) FROM patients;
// system.mask_inner(name, 1, NULL) | system.mask_default(birth)
// -----------------------------------+----------------------------
// b** | 1970-01-01
// a**** | 1970-01-01

1
doc/modules/cassandra/nav.adoc
View File

@ -39,6 +39,7 @@
*** xref:cql/functions.adoc[Functions]
*** xref:cql/json.adoc[JSON]
*** xref:cql/security.adoc[Security]
*** xref:cql/dynamic_data_masking.adoc[Dynamic data masking]
*** xref:cql/triggers.adoc[Triggers]
*** xref:cql/appendices.adoc[Appendices]
*** xref:cql/changes.adoc[Changes]

3
doc/modules/cassandra/pages/cql/appendices.adoc
View File

@ -82,6 +82,7 @@ or not.
|`LIST` |no
|`LOGIN` |no
|`MAP` |no
|`MASKED` |no
|`MODIFY` |yes
|`NAN` |yes
|`NOLOGIN` |no
@ -106,6 +107,7 @@ or not.
|`ROLES` |no
|`SCHEMA` |yes
|`SELECT` |yes
|`SELECT_MASKED` |no
|`SET` |yes
|`SFUNC` |no
|`SMALLINT` |no
@ -127,6 +129,7 @@ or not.
|`TUPLE` |no
|`TYPE` |no
|`UNLOGGED` |yes
|`UNMASK` |no
|`UPDATE` |yes
|`USE` |yes
|`USER` |no

4
doc/modules/cassandra/pages/cql/changes.adoc
View File

@ -5,6 +5,10 @@ The following describes the changes in each version of CQL.
== 3.4.7
* Remove deprecated functions `dateOf` and `unixTimestampOf`, replaced by `toTimestamp` and `toUnixTimestamp` (`18328`)
* Added support for attaching masking functions to table columns (`18068`)
* Add UNMASK permission (`18069`)
* Add SELECT_MASKED permission (`18070`)
* Add support for using UDFs as masking functions (`18071`)
== 3.4.6

21
doc/modules/cassandra/pages/cql/cql_singlefile.adoc
View File

@ -2250,6 +2250,8 @@ The full set of available permissions is:
* `AUTHORIZE`
* `DESCRIBE`
* `EXECUTE`
* `UNMASK`
* `SELECT_MASKED`
Not all permissions are applicable to every type of resource. For
instance, `EXECUTE` is only relevant in the context of functions or
@ -2408,6 +2410,18 @@ use of function in `CREATE AGGREGATE` | | |
wildcard pattern | | |
|`EXECUTE` |`MBEAN` |Execute operations on named mbean | | |
|`UNMASK` |`ALL KEYSPACES` |See the clear contents of masked columns on any table | | |
|`UNMASK` |`KEYSPACE` |See the clear contents of masked columns on any table in keyspace | | |
|`UNMASK` |`TABLE` |See the clear contents of masked columns on the specified table | | |
|`SELECT_MASKED` | `ALL KEYSPACES` | `SELECT` restricting masked columns on any table | | |
|`SELECT_MASKED` | `KEYSPACE` | `SELECT` restricting masked columns on any table in specified keyspace | | |
|`SELECT_MASKED` | `TABLE` | `SELECT` restricting masked columns on the specified table | | |
|===
[[grantPermissionsStmt]]
@ -2418,7 +2432,7 @@ _Syntax:_
bc(syntax).. +
::= GRANT ( ALL ( PERMISSIONS )? | ( PERMISSION )? ) ON TO
::= CREATE | ALTER | DROP | SELECT | MODIFY | AUTHORIZE | DESRIBE |
::= CREATE | ALTER | DROP | SELECT | MODIFY | AUTHORIZE | DESRIBE | UNMASK | SELECT_MASKED
EXECUTE
::= ALL KEYSPACES +
@ -2490,7 +2504,7 @@ _Syntax:_
bc(syntax).. +
::= REVOKE ( ALL ( PERMISSIONS )? | ( PERMISSION )? ) ON FROM
::= CREATE | ALTER | DROP | SELECT | MODIFY | AUTHORIZE | DESRIBE |
::= CREATE | ALTER | DROP | SELECT | MODIFY | AUTHORIZE | DESRIBE | UNMASK | SELECT_MASKED
EXECUTE
::= ALL KEYSPACES +
@ -3594,6 +3608,7 @@ or not.
|`LIST` |no
|`LOGIN` |no
|`MAP` |no
|`MASKED` |no
|`MATERIALIZED` |yes
|`MBEAN` |yes
|`MBEANS` |yes
@ -3623,6 +3638,7 @@ or not.
|`ROLES` |no
|`SCHEMA` |yes
|`SELECT` |yes
|`SELECT_MASKED` |no
|`SET` |yes
|`SFUNC` |no
|`SMALLINT` |no
@ -3644,6 +3660,7 @@ or not.
|`TUPLE` |no
|`TYPE` |no
|`UNLOGGED` |yes
|`UNMASK` |no
|`UNSET` |yes
|`UPDATE` |yes
|`USE` |yes

178
doc/modules/cassandra/pages/cql/dynamic_data_masking.adoc Normal file
View File

@ -0,0 +1,178 @@
= Dynamic Data Masking
Dynamic data masking (DDM) allows to obscure sensitive information while still allowing access to the masked columns.
DDM doesn't change the stored data. Instead, it just presents the data on their obscured form during `SELECT` queries.
This aims to provide some degree of protection against accidental data exposure. However, it's important to know that
anyone with direct access to the sstable files will be able to read the clear data.
== Masking functions
DDM is based on a set of CQL native functions that obscure sensitive information. The available functions are:
include::partial$masking_functions.adoc[]
Those functions can be discretionarily used on `SELECT` queries to get an obscured view of the data. For example:
[source,cql]
----
include::example$CQL/select_with_mask_functions.cql[]
----
== Attaching masking functions to table columns
The masking functions can be permanently attached to the columns of a table.
In that case, `SELECT` queries will always return the column values in their masked form.
The masking will be transparent for the users running `SELECT` queries,
so their only way to know that a column is masked will be consulting the table definition.
This is an optional feature that should be enabled with the `dynamic_data_masking_enabled` property in `cassandra.yaml`,
since it's disabled by default.
The masks of the columns of a table can be defined on `CREATE TABLE` queries:
[source,cql]
----
include::example$CQL/ddm_create_table.cql[]
----
Note that in the example above we are referencing the `mask_inner` function with two arguments.
However, that CQL function actually has three arguments when explicitely used on `SELECT` queries.
The first argument is always ommitted when attaching the function to a schema column.
The value of that first argument is always interpreted as the value of the masked column, in this case a `text` column.
For the same reason the call to `mask_default` attached to the column doesn't have any argument,
even when that function requires one argument when explicitely used on `SELECT` queries.
Data can be inserted into the masked table as usual. For example:
[source,cql]
----
include::example$CQL/ddm_insert_data.cql[]
----
The attached column masks will make `SELECT` queries automatically return masked data,
without the need of including the masking function on the query:
[source,cql]
----
include::example$CQL/ddm_select_with_masked_columns.cql[]
----
The masking function attached to a column can be changed with an `ALTER TABLE` query:
[source,cql]
----
include::example$CQL/ddm_alter_mask.cql[]
----
In a similar way, a masking function can be dettached from a column with an `ALTER TABLE` query:
[source,cql]
----
include::example$CQL/ddm_drop_mask.cql[]
----
== Permissions
The `UNMASK` permission allows users to retrieve the unmasked values of masked columns.
The masks will only be applied to the results of a `SELECT` query if the user doesn't have the `UNMASK` permission.
Ordinary users are created without the `UNMASK` permission, whereas superusers do have it.
As an example, suppose that we have a table with masked columns:
[source,cql]
----
include::example$CQL/ddm_create_table.cql[]
----
And we insert some data into the table:
[source,cql]
----
include::example$CQL/ddm_insert_data.cql[]
----
[source,cql]
----
include::example$CQL/ddm_select_without_unmask_permission.cql[]
----
Then we create two users with `SELECT` permission for the table, but we only grant the `UNMASK` permission to one of
the users:
[source,cql]
----
include::example$CQL/ddm_create_users.cql[]
----
We can now see that the user with the `UNMASK` permission can see the clear data, without any masking:
[source,cql]
----
include::example$CQL/ddm_select_with_unmask_permission.cql[]
----
However, the user without the `UNMASK` permission can only see the masked data:
[source,cql]
----
include::example$CQL/ddm_select_without_unmask_permission.cql[]
----
The `UNMASK` permission works as any other permission. Thus, it can be revoked in any moment:
[source,cql]
----
include::example$CQL/ddm_revoke_unmask.cql[]
----
Please note that the anonymous user that is used when authentication is disabled has all the permissions.
Since it includes the `UNMASK` permission, that anonymous user will always see the clear data.
In other words, attaching data masking functions to columns only makes sense if authentication is enabled.
Users without the `UNMASK` permission are not allowed to use masked columns in the `WHERE` clause of a `SELECT` query.
This prevents malicious users from figuring out the clear data by running exhaustive queries. For instance:
[source,cql]
----
include::example$CQL/ddm_select_without_select_masked.cql[]
----
However, there are some use cases where trusted database users just need a useful way to produce masked data
that will be served to untrusted external users.
For example, a trusted app can connect to the database and extract masked data that will be served to its end users.
In that case the trusted user (the app) can be given the `SELECT_MASKED` permission.
That permission allows to use masked columns in the `WHERE` clause of a `SELECT` query,
while still seeing the masked data in the query results. For instance:
[source,cql]
----
include::example$CQL/ddm_select_with_select_masked.cql[]
----
== Custom functions
xref:cql/functions.adoc#user-defined-scalar-functions[User-defined functions (UDFs)] can be attached to a table column.
The UDFs used for masking should belong to the same keyspace as the masked table.
The column value to mask will be passed as the first argument of the attached UDF.
Thus, the UDFs attached to a column should have at least one argument,
and that argument should have the same type as the masked column.
Also, the attached UDF should return values of the same type as the maked column. For instance:
[source,cql]
----
include::example$CQL/ddm_create_table_with_udf.cql[]
----
This creates a dependency between the table schema and the functions.
Any attempt to drop the function will be rejected while this dependency exists.
Thus, to drop the function you should first drop the mask.
This can be done with:
[source,cql]
----
include::example$CQL/ddm_drop_mask.cql[]
----
Dropping the column, or its containing table, or its containing keyspace would also remove the dependency.
xref:cql/functions.adoc#aggregate-functions[Aggregate functions] cannot be used as masking functions.

63
doc/modules/cassandra/pages/cql/functions.adoc
View File

@ -277,71 +277,12 @@ A number of functions are provided to operate on collection columns.
| `collection_avg` | numeric `set` or `list` | Computes the average of the elements in the collection argument. The average of an empty collection returns zero. The returned value is of the same type as the input collection elements, which might include rounding and truncations. For example `collection_avg([1, 2])` returns `1` instead of `1.5`.
|===
[[data-masking-functions]]
===== Data masking functions
A number of functions allow to obscure the real contents of a column containing sensitive data.
[cols=",",options="header",]
|===
|Function | Description
| `mask_null(value)` | Replaces the first argument by a `null` column. The returned value is always an absent column, as it didn't exist, and not a not-null column representing a `null` value.
Examples:
`mask_null('Alice')` -> `null`
`mask_null(123)` -> `null`
| `mask_default(value)` | Replaces its argument by an arbitrary, fixed default value of the same type. This will be `\***\***` for text values, zero for numeric values, `false` for booleans, etc.
Examples:
`mask_default('Alice')` -> `'\****'`
`mask_default(123)` -> `0`
| `mask_replace(value, replacement])` | Replaces the first argument by the replacement value on the second argument. The replacement value needs to have the same type as the replaced value.
Examples:
`mask_replace('Alice', 'REDACTED')` -> `'REDACTED'`
`mask_replace(123, -1)` -> `-1`
| `mask_inner(value, begin, end, [padding])` | Returns a copy of the first `text`, `varchar` or `ascii` argument, replacing each character except the first and last ones by a padding character. The 2nd and 3rd arguments are the size of the exposed prefix and suffix. The optional 4th argument is the padding character, `\*` by default.
Examples:
`mask_inner('Alice', 1, 2)` -> `'A**ce'`
`mask_inner('Alice', 1, null)` -> `'A****'`
`mask_inner('Alice', null, 2)` -> `'***ce'`
`mask_inner('Alice', 2, 1, '\#')` -> `'Al##e'`
| `mask_outer(value, begin, end, [padding])` | Returns a copy of the first `text`, `varchar` or `ascii` argument, replacing the first and last character by a padding character. The 2nd and 3rd arguments are the size of the exposed prefix and suffix. The optional 4th argument is the padding character, `\*` by default.
Examples:
`mask_outer('Alice', 1, 2)` -> `'*li**'`
`mask_outer('Alice', 1, null)` -> `'*lice'`
`mask_outer('Alice', null, 2)` -> `'Ali**'`
`mask_outer('Alice', 2, 1, '\#')` -> `'##ic#'`
| `mask_hash(value, [algorithm])` | Returns a `blob` containing the hash of the first argument. The optional 2nd argument is the hashing algorithm to be used, according the available Java security provider. The default hashing algorithm is `SHA-256`.
Examples:
`mask_hash('Alice')`
`mask_hash('Alice', 'SHA-512')`
|===
include::partial$masking_functions.adoc[]
[[user-defined-scalar-functions]]
==== User-defined functions

1
doc/modules/cassandra/pages/cql/index.adoc
View File

@ -19,6 +19,7 @@ For that reason, when used in this document, these terms (tables, rows and colum
* xref:cql/functions.adoc[Functions]
* xref:cql/json.adoc[JSON]
* xref:cql/security.adoc[CQL security]
* xref:cql/dynamic_data_masking.adoc[Dynamic data masking]
* xref:cql/triggers.adoc[Triggers]
* xref:cql/appendices.adoc[Appendices]
* xref:cql/changes.adoc[Changes]

14
doc/modules/cassandra/pages/cql/security.adoc
View File

@ -352,6 +352,8 @@ The full set of available permissions is:
* `AUTHORIZE`
* `DESCRIBE`
* `EXECUTE`
* `UNMASK`
* `SELECT_MASKED`
Not all permissions are applicable to every type of resource. For
instance, `EXECUTE` is only relevant in the context of functions or
@ -477,6 +479,18 @@ and use of any function in keyspace in `CREATE AGGREGATE`
| `EXECUTE` | `MBEANS` | Execute operations on any mbean matching a wildcard pattern
| `EXECUTE` | `MBEAN` | Execute operations on named mbean
|`UNMASK` |`ALL KEYSPACES` | See the clear contents of masked columns on any table
|`UNMASK` |`KEYSPACE` | See the clear contents of masked columns on any table in keyspace
|`UNMASK` |`TABLE` | See the clear contents of masked columns on the specified table
|`SELECT_MASKED` | `ALL KEYSPACES` | `SELECT` restricting masked columns on any table
|`SELECT_MASKED` | `KEYSPACE` | `SELECT` restricting masked columns on any table in specified keyspace
|`SELECT_MASKED` | `TABLE` | `SELECT` restricting masked columns on the specified table
|===
[[grant-permission-statement]]

61
doc/modules/cassandra/partials/masking_functions.adoc Normal file
View File

@ -0,0 +1,61 @@
[cols=",",options="header",]
|===
|Function | Description
| `mask_null(value)` | Replaces the first argument by a `null` column. The returned value is always an absent column, as it didn't exist, and not a not-null column representing a `null` value.
Examples:
`mask_null('Alice')` -> `null`
`mask_null(123)` -> `null`
| `mask_default(value)` | Replaces its argument by an arbitrary, fixed default value of the same type. This will be `\***\***` for text values, zero for numeric values, `false` for booleans, etc.
Examples:
`mask_default('Alice')` -> `'\****'`
`mask_default(123)` -> `0`
| `mask_replace(value, replacement])` | Replaces the first argument by the replacement value on the second argument. The replacement value needs to have the same type as the replaced value.
Examples:
`mask_replace('Alice', 'REDACTED')` -> `'REDACTED'`
`mask_replace(123, -1)` -> `-1`
| `mask_inner(value, begin, end, [padding])` | Returns a copy of the first `text`, `varchar` or `ascii` argument, replacing each character except the first and last ones by a padding character. The 2nd and 3rd arguments are the size of the exposed prefix and suffix. The optional 4th argument is the padding character, `\*` by default.
Examples:
`mask_inner('Alice', 1, 2)` -> `'A**ce'`
`mask_inner('Alice', 1, null)` -> `'A****'`
`mask_inner('Alice', null, 2)` -> `'***ce'`
`mask_inner('Alice', 2, 1, '\#')` -> `'Al##e'`
| `mask_outer(value, begin, end, [padding])` | Returns a copy of the first `text`, `varchar` or `ascii` argument, replacing the first and last character by a padding character. The 2nd and 3rd arguments are the size of the exposed prefix and suffix. The optional 4th argument is the padding character, `\*` by default.
Examples:
`mask_outer('Alice', 1, 2)` -> `'*li**'`
`mask_outer('Alice', 1, null)` -> `'*lice'`
`mask_outer('Alice', null, 2)` -> `'Ali**'`
`mask_outer('Alice', 2, 1, '\#')` -> `'##ic#'`
| `mask_hash(value, [algorithm])` | Returns a `blob` containing the hash of the first argument. The optional 2nd argument is the hashing algorithm to be used, according the available Java security provider. The default hashing algorithm is `SHA-256`.
Examples:
`mask_hash('Alice')`
`mask_hash('Alice', 'SHA-512')`
|===

7
pylib/cqlshlib/cql3handling.py
View File

@ -305,7 +305,9 @@ JUNK ::= /([ \t\r\f\v]+|(--|[/][/])[^\n\r]*([\n\r]|$)|[/][*].*?[*][/])/ ;
<userType> ::= utname=<cfOrKsName> ;
<storageType> ::= <simpleStorageType> | <collectionType> | <frozenCollectionType> | <userType> ;
<storageType> ::= ( <simpleStorageType> | <collectionType> | <frozenCollectionType> | <userType> ) ( <column_mask> )? ;
<column_mask> ::= "MASKED" "WITH" ( "DEFAULT" | <functionName> <selectionFunctionArguments> );
# Note: autocomplete for frozen collection types does not handle nesting past depth 1 properly,
# but that's a lot of work to fix for little benefit.
@ -1418,6 +1420,7 @@ syntax_rules += r'''
| "WITH" <cfamProperty> ( "AND" <cfamProperty> )*
| "RENAME" ("IF" "EXISTS")? existcol=<cident> "TO" newcol=<cident>
( "AND" existcol=<cident> "TO" newcol=<cident> )*
| "ALTER" ("IF" "EXISTS")? existcol=<cident> ( <column_mask> | "DROP" "MASKED" )
;
<alterUserTypeStatement> ::= "ALTER" "TYPE" ("IF" "EXISTS")? ut=<userTypeName>
@ -1530,6 +1533,8 @@ syntax_rules += r'''
| "MODIFY"
| "DESCRIBE"
| "EXECUTE"
| "UNMASK"
| "SELECT_MASKED"
;
<permissionExpr> ::= ( [newpermission]=<permission> "PERMISSION"? ( "," [newpermission]=<permission> "PERMISSION"? )* )

31
pylib/cqlshlib/test/test_cqlsh_completion.py
View File

@ -620,7 +620,12 @@ class TestCqlshCompletion(CqlshCompletionCase):
self.trycompletions(prefix + ' new_table (col_a ine',
immediate='t ')
self.trycompletions(prefix + ' new_table (col_a int ',
choices=[',', 'PRIMARY'])
choices=[',', 'MASKED', 'PRIMARY'])
self.trycompletions(prefix + ' new_table (col_a int M',
immediate='ASKED WITH ')
self.trycompletions(prefix + ' new_table (col_a int MASKED WITH ',
choices=['DEFAULT', self.cqlsh.keyspace + '.', 'system.'],
other_choices_ok=True)
self.trycompletions(prefix + ' new_table (col_a int P',
immediate='RIMARY KEY ')
self.trycompletions(prefix + ' new_table (col_a int PRIMARY KEY ',
@ -876,7 +881,7 @@ class TestCqlshCompletion(CqlshCompletionCase):
self.trycompletions("GR",
immediate='ANT ')
self.trycompletions("GRANT ",
choices=['ALL', 'ALTER', 'AUTHORIZE', 'CREATE', 'DESCRIBE', 'DROP', 'EXECUTE', 'MODIFY', 'SELECT'],
choices=['ALL', 'ALTER', 'AUTHORIZE', 'CREATE', 'DESCRIBE', 'DROP', 'EXECUTE', 'MODIFY', 'SELECT', 'UNMASK', 'SELECT_MASKED'],
other_choices_ok=True)
self.trycompletions("GRANT MODIFY ",
choices=[',', 'ON', 'PERMISSION'])
@ -885,7 +890,7 @@ class TestCqlshCompletion(CqlshCompletionCase):
self.trycompletions("GRANT MODIFY PERMISSION ",
choices=[',', 'ON'])
self.trycompletions("GRANT MODIFY PERMISSION, ",
choices=['ALTER', 'AUTHORIZE', 'CREATE', 'DESCRIBE', 'DROP', 'EXECUTE', 'SELECT'])
choices=['ALTER', 'AUTHORIZE', 'CREATE', 'DESCRIBE', 'DROP', 'EXECUTE', 'SELECT', 'UNMASK', 'SELECT_MASKED'])
self.trycompletions("GRANT MODIFY PERMISSION, D",
choices=['DESCRIBE', 'DROP'])
self.trycompletions("GRANT MODIFY PERMISSION, DR",
@ -907,7 +912,7 @@ class TestCqlshCompletion(CqlshCompletionCase):
self.trycompletions("RE",
immediate='VOKE ')
self.trycompletions("REVOKE ",
choices=['ALL', 'ALTER', 'AUTHORIZE', 'CREATE', 'DESCRIBE', 'DROP', 'EXECUTE', 'MODIFY', 'SELECT'],
choices=['ALL', 'ALTER', 'AUTHORIZE', 'CREATE', 'DESCRIBE', 'DROP', 'EXECUTE', 'MODIFY', 'SELECT', 'UNMASK', 'SELECT_MASKED'],
other_choices_ok=True)
self.trycompletions("REVOKE MODIFY ",
choices=[',', 'ON', 'PERMISSION'])
@ -916,7 +921,7 @@ class TestCqlshCompletion(CqlshCompletionCase):
self.trycompletions("REVOKE MODIFY PERMISSION ",
choices=[',', 'ON'])
self.trycompletions("REVOKE MODIFY PERMISSION, ",
choices=['ALTER', 'AUTHORIZE', 'CREATE', 'DESCRIBE', 'DROP', 'EXECUTE', 'SELECT'])
choices=['ALTER', 'AUTHORIZE', 'CREATE', 'DESCRIBE', 'DROP', 'EXECUTE', 'SELECT', 'UNMASK', 'SELECT_MASKED'])
self.trycompletions("REVOKE MODIFY PERMISSION, D",
choices=['DESCRIBE', 'DROP'])
self.trycompletions("REVOKE MODIFY PERMISSION, DR",
@ -954,9 +959,23 @@ class TestCqlshCompletion(CqlshCompletionCase):
self.trycompletions('ALTER TABLE IF EXISTS new_table ADD ', choices=['<new_column_name>', 'IF'])
self.trycompletions('ALTER TABLE IF EXISTS new_table ADD IF NOT EXISTS ', choices=['<new_column_name>'])
self.trycompletions('ALTER TABLE new_table ADD IF NOT EXISTS ', choices=['<new_column_name>'])
self.trycompletions('ALTER TABLE new_table ADD col int ', choices=[';', 'MASKED', 'static'])
self.trycompletions('ALTER TABLE new_table ADD col int M', immediate='ASKED WITH ')
self.trycompletions('ALTER TABLE new_table ADD col int MASKED WITH ',
choices=['DEFAULT', self.cqlsh.keyspace + '.', 'system.'],
other_choices_ok=True)
self.trycompletions('ALTER TABLE IF EXISTS new_table RENAME ', choices=['IF', '<quotedName>', '<identifier>'])
self.trycompletions('ALTER TABLE new_table RENAME ', choices=['IF', '<quotedName>', '<identifier>'])
self.trycompletions('ALTER TABLE IF EXISTS new_table DROP ', choices=['IF', '<quotedName>', '<identifier>'])
self.trycompletions('ALTER TABLE IF EXISTS new_table ALTER ', choices=['IF', '<quotedName>', '<identifier>'])
self.trycompletions('ALTER TABLE IF EXISTS new_table ALTER IF E', immediate='XISTS ')
self.trycompletions('ALTER TABLE IF EXISTS new_table ALTER IF EXISTS col ', choices=['MASKED', 'DROP'])
self.trycompletions('ALTER TABLE IF EXISTS new_table ALTER IF EXISTS col M', immediate='ASKED WITH ')
self.trycompletions('ALTER TABLE IF EXISTS new_table ALTER IF EXISTS col MASKED WITH ',
choices=['DEFAULT', self.cqlsh.keyspace + '.', 'system.'],
other_choices_ok=True)
self.trycompletions('ALTER TABLE IF EXISTS new_table ALTER IF EXISTS col D', immediate='ROP MASKED ;')
self.trycompletions('ALTER TABLE IF EXISTS new_table ALTER IF EXISTS col DROP M', immediate='ASKED ;')
def test_complete_in_alter_type(self):
self.trycompletions('ALTER TYPE I', immediate='F EXISTS ')
@ -991,7 +1010,7 @@ class TestCqlshCompletion(CqlshCompletionCase):
def test_complete_in_list(self):
self.trycompletions('LIST ', choices=['ALL', 'AUTHORIZE', 'DESCRIBE', 'EXECUTE', 'ROLES', 'USERS', 'ALTER', 'CREATE', 'DROP', 'MODIFY', 'SELECT'])
self.trycompletions('LIST ', choices=['ALL', 'AUTHORIZE', 'DESCRIBE', 'EXECUTE', 'ROLES', 'USERS', 'ALTER', 'CREATE', 'DROP', 'MODIFY', 'SELECT', 'UNMASK', 'SELECT_MASKED'])
# Non-CQL Shell Commands

1
src/antlr/Cql.g
View File

@ -39,6 +39,7 @@ import Parser,Lexer;
import org.apache.cassandra.auth.*;
import org.apache.cassandra.cql3.conditions.*;
import org.apache.cassandra.cql3.functions.*;
import org.apache.cassandra.cql3.functions.masking.*;
import org.apache.cassandra.cql3.restrictions.CustomIndexExpression;
import org.apache.cassandra.cql3.selection.*;
import org.apache.cassandra.cql3.statements.*;

4
src/antlr/Lexer.g
View File

@ -218,6 +218,10 @@ K_DEFAULT: D E F A U L T;
K_UNSET: U N S E T;
K_LIKE: L I K E;
K_MASKED: M A S K E D;
K_UNMASK: U N M A S K;
K_SELECT_MASKED: S E L E C T '_' M A S K E D;
// Case-insensitive alpha characters
fragment A: ('a'|'A');
fragment B: ('b'|'B');

31
src/antlr/Parser.g
View File

@ -782,11 +782,21 @@ tableDefinition[CreateTableStatement.Raw stmt]
tableColumns[CreateTableStatement.Raw stmt]
@init { boolean isStatic = false; }
: k=ident v=comparatorType (K_STATIC { isStatic = true; })? { $stmt.addColumn(k, v, isStatic); }
: k=ident v=comparatorType (K_STATIC { isStatic = true; })? (mask=columnMask)? { $stmt.addColumn(k, v, isStatic, mask); }
(K_PRIMARY K_KEY { $stmt.setPartitionKeyColumn(k); })?
| K_PRIMARY K_KEY '(' tablePartitionKey[stmt] (',' c=ident { $stmt.markClusteringColumn(c); } )* ')'
;
columnMask returns [ColumnMask.Raw mask]
@init { List<Term.Raw> arguments = new ArrayList<>(); }
: K_MASKED K_WITH name=functionName columnMaskArguments[arguments] { $mask = new ColumnMask.Raw(name, arguments); }
| K_MASKED K_WITH K_DEFAULT { $mask = new ColumnMask.Raw(FunctionName.nativeFunction("mask_default"), arguments); }
;
columnMaskArguments[List<Term.Raw> arguments]
: '(' ')' | '(' c=term { arguments.add(c); } (',' cn=term { arguments.add(cn); })* ')'
;
tablePartitionKey[CreateTableStatement.Raw stmt]
@init {List<ColumnIdentifier> l = new ArrayList<ColumnIdentifier>();}
@after{ $stmt.setPartitionKeyColumns(l); }
@ -929,7 +939,9 @@ alterKeyspaceStatement returns [AlterKeyspaceStatement.Raw stmt]
/**
* ALTER TABLE <table> ALTER <column> TYPE <newtype>;
* ALTER TABLE [IF EXISTS] <table> ADD [IF NOT EXISTS] <column> <newtype>; | ALTER TABLE [IF EXISTS] <table> ADD [IF NOT EXISTS] (<column> <newtype>,<column1> <newtype1>..... <column n> <newtype n>)
* ALTER TABLE [IF EXISTS] <table> ALTER [IF EXISTS] <column> MASKED WITH <maskFunction>);
* ALTER TABLE [IF EXISTS] <table> ALTER [IF EXISTS] <column> DROP MASKED;
* ALTER TABLE [IF EXISTS] <table> ADD [IF NOT EXISTS] <column> <newtype> <maskFunction>; | ALTER TABLE [IF EXISTS] <table> ADD [IF NOT EXISTS] (<column> <newtype> <maskFunction>, <column1> <newtype1> <maskFunction1>..... <column n> <newtype n> <maskFunction n>)
* ALTER TABLE [IF EXISTS] <table> DROP [IF EXISTS] <column>; | ALTER TABLE [IF EXISTS] <table> DROP [IF EXISTS] ( <column>,<column1>.....<column n>)
* ALTER TABLE [IF EXISTS] <table> RENAME [IF EXISTS] <column> TO <column>;
* ALTER TABLE [IF EXISTS] <table> WITH <property> = <value>;
@ -941,10 +953,14 @@ alterTableStatement returns [AlterTableStatement.Raw stmt]
(
K_ALTER id=cident K_TYPE v=comparatorType { $stmt.alter(id, v); }
| K_ALTER ( K_IF K_EXISTS { $stmt.ifColumnExists(true); } )? id=cident
( mask=columnMask { $stmt.mask(id, mask); }
| K_DROP K_MASKED { $stmt.mask(id, null); } )
| K_ADD ( K_IF K_NOT K_EXISTS { $stmt.ifColumnNotExists(true); } )?
( id=ident v=comparatorType b=isStaticColumn { $stmt.add(id, v, b); }
| ('(' id1=ident v1=comparatorType b1=isStaticColumn { $stmt.add(id1, v1, b1); }
( ',' idn=ident vn=comparatorType bn=isStaticColumn { $stmt.add(idn, vn, bn); } )* ')') )
( id=ident v=comparatorType b=isStaticColumn (m=columnMask)? { $stmt.add(id, v, b, m); }
| ('(' id1=ident v1=comparatorType b1=isStaticColumn (m1=columnMask)? { $stmt.add(id1, v1, b1, m1); }
( ',' idn=ident vn=comparatorType bn=isStaticColumn (mn=columnMask)? { $stmt.add(idn, vn, bn, mn); mn=null; } )* ')') )
| K_DROP ( K_IF K_EXISTS { $stmt.ifColumnExists(true); } )?
( id=ident { $stmt.drop(id); }
@ -1112,7 +1128,7 @@ listPermissionsStatement returns [ListPermissionsStatement stmt]
;
permission returns [Permission perm]
: p=(K_CREATE | K_ALTER | K_DROP | K_SELECT | K_MODIFY | K_AUTHORIZE | K_DESCRIBE | K_EXECUTE)
: p=(K_CREATE | K_ALTER | K_DROP | K_SELECT | K_MODIFY | K_AUTHORIZE | K_DESCRIBE | K_EXECUTE | K_UNMASK | K_SELECT_MASKED)
{ $perm = Permission.valueOf($p.text.toUpperCase()); }
;
@ -1940,5 +1956,8 @@ basic_unreserved_keyword returns [String str]
| K_MBEANS
| K_REPLACE
| K_UNSET
| K_MASKED
| K_UNMASK
| K_SELECT_MASKED
) { $str = $k.text; }
;

2
src/java/org/apache/cassandra/auth/CassandraRoleManager.java
View File

@ -82,7 +82,7 @@ public class CassandraRoleManager implements IRoleManager
private static final Logger logger = LoggerFactory.getLogger(CassandraRoleManager.class);
public static final String DEFAULT_SUPERUSER_NAME = "cassandra";
static final String DEFAULT_SUPERUSER_PASSWORD = "cassandra";
public static final String DEFAULT_SUPERUSER_PASSWORD = "cassandra";
/**
* We need to treat the default superuser as a special case since during initial node startup, we may end up with

12
src/java/org/apache/cassandra/auth/DataResource.java
View File

@ -46,7 +46,9 @@ public class DataResource implements IResource
Permission.DROP,
Permission.SELECT,
Permission.MODIFY,
Permission.AUTHORIZE);
Permission.AUTHORIZE,
Permission.UNMASK,
Permission.SELECT_MASKED);
// permissions which may be granted on all tables of a given keyspace
private static final Set<Permission> ALL_TABLES_LEVEL_PERMISSIONS = Sets.immutableEnumSet(Permission.CREATE,
@ -54,7 +56,9 @@ public class DataResource implements IResource
Permission.DROP,
Permission.SELECT,
Permission.MODIFY,
Permission.AUTHORIZE);
Permission.AUTHORIZE,
Permission.UNMASK,
Permission.SELECT_MASKED);
// permissions which may be granted on one or all keyspaces
private static final Set<Permission> KEYSPACE_LEVEL_PERMISSIONS = Sets.immutableEnumSet(Permission.CREATE,
@ -62,7 +66,9 @@ public class DataResource implements IResource
Permission.DROP,
Permission.SELECT,
Permission.MODIFY,
Permission.AUTHORIZE);
Permission.AUTHORIZE,
Permission.UNMASK,
Permission.SELECT_MASKED);
private static final String ROOT_NAME = "data";
private static final DataResource ROOT_RESOURCE = new DataResource(Level.ROOT, null, null);

8
src/java/org/apache/cassandra/auth/Permission.java
View File

@ -61,9 +61,13 @@ public enum Permission
DESCRIBE, // required on the root-level RoleResource to list all Roles
// UDF permissions
EXECUTE; // required to invoke any user defined function or aggregate
EXECUTE, // required to invoke any user defined function or aggregate
UNMASK, // required to see masked data
SELECT_MASKED; // required for SELECT on a table with restictions on masked columns
public static final Set<Permission> ALL =
Sets.immutableEnumSet(EnumSet.range(Permission.CREATE, Permission.EXECUTE));
Sets.immutableEnumSet(EnumSet.range(Permission.CREATE, Permission.SELECT_MASKED));
public static final Set<Permission> NONE = ImmutableSet.of();
}

2
src/java/org/apache/cassandra/config/Config.java
View File

@ -618,6 +618,8 @@ public class Config
*/
public boolean allow_extra_insecure_udfs = false;
public boolean dynamic_data_masking_enabled = false;
/**
* Time in milliseconds after a warning will be emitted to the log and to the client that a UDF runs too long.
* (Only valid, if user_defined_functions_threads_enabled==true)

14
src/java/org/apache/cassandra/config/DatabaseDescriptor.java
View File

@ -4676,4 +4676,18 @@ public class DatabaseDescriptor
{
return Objects.requireNonNull(sstableFormatFactories, "Forgot to initialize DatabaseDescriptor?");
}
public static boolean getDynamicDataMaskingEnabled()
{
return conf.dynamic_data_masking_enabled;
}
public static void setDynamicDataMaskingEnabled(boolean enabled)
{
if (enabled != conf.dynamic_data_masking_enabled)
{
logger.info("Setting dynamic_data_masking_enabled to {}", enabled);
conf.dynamic_data_masking_enabled = enabled;
}
}
}

6
src/java/org/apache/cassandra/cql3/QueryProcessor.java
View File

@ -483,7 +483,7 @@ public class QueryProcessor implements QueryHandler
.map(m -> MessagingService.instance().<ReadResponse>sendWithResult(m, address))
.collect(Collectors.toList()));
ResultSetBuilder result = new ResultSetBuilder(select.getResultMetadata(), select.getSelection().newSelectors(options), null);
ResultSetBuilder result = new ResultSetBuilder(select.getResultMetadata(), select.getSelection().newSelectors(options), false);
return future.map(list -> {
int i = 0;
for (Message<ReadResponse> m : list)
@ -611,17 +611,19 @@ public class QueryProcessor implements QueryHandler
return select.executeRawInternal(makeInternalOptionsWithNowInSec(prepared.statement, nowInSec, values), internalQueryState().getClientState(), nowInSec);
}
@VisibleForTesting
public static UntypedResultSet resultify(String query, RowIterator partition)
{
return resultify(query, PartitionIterators.singletonIterator(partition));
}
@VisibleForTesting
public static UntypedResultSet resultify(String query, PartitionIterator partitions)
{
try (PartitionIterator iter = partitions)
{
SelectStatement ss = (SelectStatement) getStatement(query, null);
ResultSet cqlRows = ss.process(iter, FBUtilities.nowInSeconds());
ResultSet cqlRows = ss.process(iter, FBUtilities.nowInSeconds(), true);
return UntypedResultSet.create(cqlRows);
}
}

4
src/java/org/apache/cassandra/cql3/UntypedResultSet.java
View File

@ -221,7 +221,7 @@ public abstract class UntypedResultSet implements Iterable<UntypedResultSet.Row>
try (ReadExecutionController executionController = pager.executionController();
PartitionIterator iter = pager.fetchPageInternal(pageSize, executionController))
{
currentPage = select.process(iter, nowInSec).rows.iterator();
currentPage = select.process(iter, nowInSec, true).rows.iterator();
}
}
return new Row(metadata, currentPage.next());
@ -286,7 +286,7 @@ public abstract class UntypedResultSet implements Iterable<UntypedResultSet.Row>
try (PartitionIterator iter = pager.fetchPage(pageSize, cl, clientState, nanoTime()))
{
currentPage = select.process(iter, nowInSec).rows.iterator();
currentPage = select.process(iter, nowInSec, true).rows.iterator();
}
}
return new Row(metadata, currentPage.next());

35
src/java/org/apache/cassandra/cql3/functions/FunctionResolver.java
View File

@ -32,6 +32,7 @@ import org.apache.cassandra.cql3.ColumnSpecification;
import org.apache.cassandra.schema.Schema;
import org.apache.cassandra.db.marshal.AbstractType;
import org.apache.cassandra.exceptions.InvalidRequestException;
import org.apache.cassandra.schema.UserFunctions;
import static java.util.stream.Collectors.joining;
import static org.apache.cassandra.cql3.statements.RequestValidations.invalidRequest;
@ -68,7 +69,31 @@ public final class FunctionResolver
AbstractType<?> receiverType)
throws InvalidRequestException
{
Collection<Function> candidates = collectCandidates(keyspace, name, receiverKeyspace, receiverTable, providedArgs, receiverType);
return get(keyspace, name, providedArgs, receiverKeyspace, receiverTable, receiverType, UserFunctions.none());
}
/**
* @param keyspace the current keyspace
* @param name the name of the function
* @param providedArgs the arguments provided for the function call
* @param receiverKeyspace the receiver's keyspace
* @param receiverTable the receiver's table
* @param receiverType if the receiver type is known (during inserts, for example), this should be the type of
* the receiver
* @param functions a set of user functions that is not yet available in the schema, used during startup when those
* functions might not be yet available
*/
@Nullable
public static Function get(String keyspace,
FunctionName name,
List<? extends AssignmentTestable> providedArgs,
String receiverKeyspace,
String receiverTable,
AbstractType<?> receiverType,
UserFunctions functions)
throws InvalidRequestException
{
Collection<Function> candidates = collectCandidates(keyspace, name, receiverKeyspace, receiverTable, providedArgs, receiverType, functions);
if (candidates.isEmpty())
return null;
@ -89,13 +114,15 @@ public final class FunctionResolver
String receiverKeyspace,
String receiverTable,
List<? extends AssignmentTestable> providedArgs,
AbstractType<?> receiverType)
AbstractType<?> receiverType,
UserFunctions functions)
{
Collection<Function> candidates = new ArrayList<>();
if (name.hasKeyspace())
{
// function name is fully qualified (keyspace + name)
candidates.addAll(functions.get(name));
candidates.addAll(Schema.instance.getUserFunctions(name));
candidates.addAll(NativeFunctions.instance.getFunctions(name));
candidates.addAll(NativeFunctions.instance.getFactories(name).stream()
@ -107,7 +134,9 @@ public final class FunctionResolver
{
// function name is not fully qualified
// add 'current keyspace' candidates
candidates.addAll(Schema.instance.getUserFunctions(new FunctionName(keyspace, name.name)));
FunctionName userName = new FunctionName(keyspace, name.name);
candidates.addAll(functions.get(userName));
candidates.addAll(Schema.instance.getUserFunctions(userName));
// add 'SYSTEM' (native) candidates
FunctionName nativeName = name.asNativeFunction();
candidates.addAll(NativeFunctions.instance.getFunctions(nativeName));

314
src/java/org/apache/cassandra/cql3/functions/masking/ColumnMask.java Normal file
View File

@ -0,0 +1,314 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.cql3.functions.masking;
import java.nio.ByteBuffer;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
import java.util.Objects;
import com.google.common.collect.ImmutableList;
import org.apache.commons.lang3.StringUtils;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.cql3.AssignmentTestable;
import org.apache.cassandra.cql3.CQL3Type;
import org.apache.cassandra.cql3.ColumnIdentifier;
import org.apache.cassandra.cql3.CqlBuilder;
import org.apache.cassandra.cql3.Term;
import org.apache.cassandra.cql3.Terms;
import org.apache.cassandra.cql3.functions.Function;
import org.apache.cassandra.cql3.functions.FunctionName;
import org.apache.cassandra.cql3.functions.FunctionResolver;
import org.apache.cassandra.cql3.functions.ScalarFunction;
import org.apache.cassandra.db.marshal.AbstractType;
import org.apache.cassandra.db.marshal.ReversedType;
import org.apache.cassandra.exceptions.InvalidRequestException;
import org.apache.cassandra.transport.ProtocolVersion;
import static java.lang.String.format;
import static org.apache.cassandra.cql3.statements.RequestValidations.invalidRequest;
/**
* Dynamic data mask that can be applied to a schema column.
* <p>
* It consists on a partial application of a certain {@link MaskingFunction} to the values of a column, with the
* precondition that the type of any masked column is compatible with the type of the first argument of the function.
* <p>
* This partial application is meant to be associated to specific columns in the schema, acting as a mask for the values
* of those columns. It's associated to queries such as:
* <pre>
* CREATE TABLE t (k int PRIMARY KEY, v int MASKED WITH mask_inner(1, 1));
* ALTER TABLE t ALTER v MASKED WITH mask_inner(2, 1);
* ALTER TABLE t ALTER v DROP MASKED;
* </pre>
* Note that in the example above we are referencing the {@code mask_inner} function with two arguments. However, that
* CQL function actually has three arguments. The first argument is always ommitted when attaching the function to a
* schema column. The value of that first argument is always the value of the masked column, in this case an int.
*/
public abstract class ColumnMask
{
public static final String DISABLED_ERROR_MESSAGE = "Cannot mask columns because dynamic data masking is not " +
"enabled. You can enable it with the " +
"dynamic_data_masking_enabled property on cassandra.yaml";
/** The CQL function used for masking. */
public final ScalarFunction function;
/** The values of the arguments of the partially applied masking function. */
protected final ByteBuffer[] partialArgumentValues;
private ColumnMask(ScalarFunction function, ByteBuffer... partialArgumentValues)
{
assert function.argTypes().size() == partialArgumentValues.length + 1;
this.function = function;
this.partialArgumentValues = partialArgumentValues;
}
public static ColumnMask build(ScalarFunction function, ByteBuffer... partialArgumentValues)
{
return function.isNative()
? new Native((MaskingFunction) function, partialArgumentValues)
: new Custom(function, partialArgumentValues);
}
/**
* @return The types of the arguments of the partially applied masking function, as an unmodifiable list.
*/
public List<AbstractType<?>> partialArgumentTypes()
{
List<AbstractType<?>> argTypes = function.argTypes();
return argTypes.size() == 1
? Collections.emptyList()
: Collections.unmodifiableList(argTypes.subList(1, argTypes.size()));
}
/**
* @return The values of the arguments of the partially applied masking function, as an unmodifiable list that can
* contain nulls.
*/
public List<ByteBuffer> partialArgumentValues()
{
return Collections.unmodifiableList(Arrays.asList(partialArgumentValues));
}
/**
* @return A copy of this mask for a version of its masked column that has its type reversed.
*/
public ColumnMask withReversedType()
{
AbstractType<?> reversed = ReversedType.getInstance(function.argTypes().get(0));
List<AbstractType<?>> args = ImmutableList.<AbstractType<?>>builder()
.add(reversed)
.addAll(partialArgumentTypes())
.build();
Function newFunction = FunctionResolver.get(function.name().keyspace, function.name(), args, null, null, null);
assert newFunction != null;
return build((ScalarFunction) newFunction, partialArgumentValues);
}
/**
* @param protocolVersion the used version of the transport protocol
* @param value a column value to be masked
* @return the specified value after having been masked by the masked function
*/
public ByteBuffer mask(ProtocolVersion protocolVersion, ByteBuffer value)
{
if (!DatabaseDescriptor.getDynamicDataMaskingEnabled())
return value;
return maskInternal(protocolVersion, value);
}
protected abstract ByteBuffer maskInternal(ProtocolVersion protocolVersion, ByteBuffer value);
public static void ensureEnabled()
{
if (!DatabaseDescriptor.getDynamicDataMaskingEnabled())
throw new InvalidRequestException(DISABLED_ERROR_MESSAGE);
}
@Override
public boolean equals(Object o)
{
if (this == o)
return true;
if (o == null || getClass() != o.getClass())
return false;
ColumnMask mask = (ColumnMask) o;
return function.name().equals(mask.function.name())
&& Arrays.equals(partialArgumentValues, mask.partialArgumentValues);
}
@Override
public int hashCode()
{
return Objects.hash(function.name(), Arrays.hashCode(partialArgumentValues));
}
@Override
public String toString()
{
List<AbstractType<?>> types = partialArgumentTypes();
List<String> arguments = new ArrayList<>(types.size());
for (int i = 0; i < types.size(); i++)
{
CQL3Type type = types.get(i).asCQL3Type();
ByteBuffer value = partialArgumentValues[i];
arguments.add(type.toCQLLiteral(value, ProtocolVersion.CURRENT));
}
return format("%s(%s)", function.name(), StringUtils.join(arguments, ", "));
}
public void appendCqlTo(CqlBuilder builder)
{
builder.append(" MASKED WITH ").append(toString());
}
/**
* {@link ColumnMask} for native masking functions.
*/
private static class Native extends ColumnMask
{
private final MaskingFunction.Masker masker;
public Native(MaskingFunction function, ByteBuffer... partialArgumentValues)
{
super(function, partialArgumentValues);
masker = function.masker(partialArgumentValues);
}
@Override
protected ByteBuffer maskInternal(ProtocolVersion protocolVersion, ByteBuffer value)
{
return masker.mask(value);
}
}
/**
* {@link ColumnMask} for user-defined masking functions.
*/
private static class Custom extends ColumnMask
{
public Custom(ScalarFunction function, ByteBuffer... partialArgumentValues)
{
super(function, partialArgumentValues);
}
@Override
protected ByteBuffer maskInternal(ProtocolVersion protocolVersion, ByteBuffer value)
{
List<ByteBuffer> argumentValues;
int numPartialArgs = partialArgumentValues.length;
if (numPartialArgs == 0)
{
argumentValues = Collections.singletonList(value);
}
else
{
ByteBuffer[] args = new ByteBuffer[numPartialArgs + 1];
args[0] = value;
System.arraycopy(partialArgumentValues, 0, args, 1, numPartialArgs);
argumentValues = Arrays.asList(args);
}
return function.execute(protocolVersion, argumentValues);
}
}
/**
* A parsed but not prepared column mask.
*/
public final static class Raw
{
public final FunctionName name;
public final List<Term.Raw> rawPartialArguments;
public Raw(FunctionName name, List<Term.Raw> rawPartialArguments)
{
this.name = name;
this.rawPartialArguments = rawPartialArguments;
}
public ColumnMask prepare(String keyspace, String table, ColumnIdentifier column, AbstractType<?> type)
{
ScalarFunction function = findMaskingFunction(keyspace, table, column, type);
ByteBuffer[] partialArguments = preparePartialArguments(keyspace, function);
return ColumnMask.build(function, partialArguments);
}
private ScalarFunction findMaskingFunction(String keyspace, String table, ColumnIdentifier column, AbstractType<?> type)
{
List<AssignmentTestable> args = new ArrayList<>(rawPartialArguments.size() + 1);
args.add(type);
args.addAll(rawPartialArguments);
Function function = FunctionResolver.get(keyspace, name, args, keyspace, table, type);
if (function == null)
throw invalidRequest("Unable to find masking function for %s, " +
"no declared function matches the signature %s",
column, this);
if (function.isAggregate())
throw invalidRequest("Aggregate function %s cannot be used for masking table columns", this);
if (function.isNative() && !(function instanceof MaskingFunction))
throw invalidRequest("Not-masking function %s cannot be used for masking table columns", this);
if (!function.isNative() && !function.name().keyspace.equals(keyspace))
throw invalidRequest("Masking function %s doesn't belong to the same keyspace as the table %s.%s",
this, keyspace, table);
CQL3Type returnType = function.returnType().asCQL3Type();
CQL3Type expectedType = type.asCQL3Type();
if (!returnType.equals(expectedType))
throw invalidRequest("Masking function %s return type is %s. " +
"This is different to the type of the masked column %s of type %s. " +
"Masking functions can only be attached to table columns " +
"if they return the same data type as the masked column.",
this, returnType, column, expectedType);
return (ScalarFunction) function;
}
private ByteBuffer[] preparePartialArguments(String keyspace, ScalarFunction function)
{
// Note that there could be null arguments
ByteBuffer[] arguments = new ByteBuffer[rawPartialArguments.size()];
for (int i = 0; i < rawPartialArguments.size(); i++)
{
String term = rawPartialArguments.get(i).toString();
AbstractType<?> type = function.argTypes().get(i + 1);
arguments[i] = Terms.asBytes(keyspace, term, type);
}
return arguments;
}
@Override
public String toString()
{
return format("%s(%s)", name, StringUtils.join(rawPartialArguments, ", "));
}
}
}

25
src/java/org/apache/cassandra/cql3/functions/masking/DefaultMaskingFunction.java
View File

@ -26,7 +26,6 @@ import org.apache.cassandra.cql3.functions.FunctionName;
import org.apache.cassandra.cql3.functions.FunctionParameter;
import org.apache.cassandra.cql3.functions.NativeFunction;
import org.apache.cassandra.db.marshal.AbstractType;
import org.apache.cassandra.transport.ProtocolVersion;
/**
* A {@link MaskingFunction} that returns a fixed replacement value for the data type of its single argument.
@ -41,18 +40,34 @@ public class DefaultMaskingFunction extends MaskingFunction
{
public static final String NAME = "default";
private final ByteBuffer defaultValue;
private final Masker masker;
private <T> DefaultMaskingFunction(FunctionName name, AbstractType<T> inputType)
{
super(name, inputType, inputType);
this.defaultValue = inputType.getMaskedValue();
masker = new Masker(inputType);
}
@Override
public final ByteBuffer execute(ProtocolVersion protocolVersion, List<ByteBuffer> parameters)
public Masker masker(ByteBuffer... parameters)
{
return defaultValue;
return masker;
}
private static class Masker implements MaskingFunction.Masker
{
private final ByteBuffer defaultValue;
private Masker(AbstractType<?> inputType)
{
defaultValue = inputType.getMaskedValue();
}
@Override
public ByteBuffer mask(ByteBuffer value)
{
return defaultValue;
}
}
/** @return a {@link FunctionFactory} to build new {@link DefaultMaskingFunction}s. */

35
src/java/org/apache/cassandra/cql3/functions/masking/HashMaskingFunction.java
View File

@ -38,7 +38,6 @@ import org.apache.cassandra.db.marshal.BytesType;
import org.apache.cassandra.db.marshal.StringType;
import org.apache.cassandra.db.marshal.UTF8Type;
import org.apache.cassandra.exceptions.InvalidRequestException;
import org.apache.cassandra.transport.ProtocolVersion;
import org.apache.cassandra.utils.ByteBufferUtil;
/**
@ -78,20 +77,34 @@ public class HashMaskingFunction extends MaskingFunction
}
@Override
public final ByteBuffer execute(ProtocolVersion protocolVersion, List<ByteBuffer> parameters)
public Masker masker(ByteBuffer... parameters)
{
MessageDigest digest;
if (algorithmArgumentType == null || parameters.get(1) == null)
return new Masker(algorithmArgumentType, parameters);
}
private static class Masker implements MaskingFunction.Masker
{
private final MessageDigest digest;
private Masker(StringType algorithmArgumentType, ByteBuffer... parameters)
{
digest = DEFAULT_DIGEST.get();
}
else
{
String algorithm = algorithmArgumentType.compose(parameters.get(1));
digest = messageDigest(algorithm);
if (algorithmArgumentType == null || parameters[0] == null)
{
digest = DEFAULT_DIGEST.get();
}
else
{
String algorithm = algorithmArgumentType.compose(parameters[0]);
digest = messageDigest(algorithm);
}
}
@Override
public ByteBuffer mask(ByteBuffer value)
{
return HashMaskingFunction.hash(digest, value);
}
return hash(digest, parameters.get(0));
}
@VisibleForTesting

31
src/java/org/apache/cassandra/cql3/functions/masking/MaskingFunction.java
View File

@ -18,6 +18,9 @@
package org.apache.cassandra.cql3.functions.masking;
import java.nio.ByteBuffer;
import java.util.List;
import com.google.common.collect.ObjectArrays;
import org.apache.cassandra.cql3.functions.FunctionFactory;
@ -25,6 +28,7 @@ import org.apache.cassandra.cql3.functions.FunctionName;
import org.apache.cassandra.cql3.functions.FunctionParameter;
import org.apache.cassandra.cql3.functions.NativeScalarFunction;
import org.apache.cassandra.db.marshal.AbstractType;
import org.apache.cassandra.transport.ProtocolVersion;
/**
* A {@link NativeScalarFunction} that totally or partially replaces the original value of a column value,
@ -52,6 +56,33 @@ public abstract class MaskingFunction extends NativeScalarFunction
super(name.name, outputType, ObjectArrays.concat(inputType, argsType));
}
@Override
public final ByteBuffer execute(ProtocolVersion protocolVersion, List<ByteBuffer> parameters)
{
ByteBuffer[] partialParameters = new ByteBuffer[parameters.size() - 1];
for (int i = 0; i < partialParameters.length; i++)
partialParameters[i] = parameters.get(i + 1);
return masker(partialParameters).mask(parameters.get(0));
}
/**
* Returns a new {@link Masker} for the specified masking parameters.
* This is meant to be used by {@link ColumnMask}, so it doesn't need to evaluate the arguments on every call.
*
* @param parameters the masking parameters in the function call.
* @return a new {@link Masker} using the specified masking arguments
*/
public abstract Masker masker(ByteBuffer... parameters);
/**
* Class that actually makes the masking of the first function parameter according to the masking arguments.
*/
public interface Masker
{
public ByteBuffer mask(ByteBuffer value);
}
protected static abstract class Factory extends FunctionFactory
{
public Factory(String name, FunctionParameter... parameters)

15
src/java/org/apache/cassandra/cql3/functions/masking/NullMaskingFunction.java
View File

@ -26,7 +26,6 @@ import org.apache.cassandra.cql3.functions.FunctionName;
import org.apache.cassandra.cql3.functions.FunctionParameter;
import org.apache.cassandra.cql3.functions.NativeFunction;
import org.apache.cassandra.db.marshal.AbstractType;
import org.apache.cassandra.transport.ProtocolVersion;
/**
* A {@link MaskingFunction} that always returns a {@code null} column. The returned value is always an absent column,
@ -38,6 +37,7 @@ import org.apache.cassandra.transport.ProtocolVersion;
public class NullMaskingFunction extends MaskingFunction
{
public static final String NAME = "null";
private static final Masker MASKER = new Masker();
private NullMaskingFunction(FunctionName name, AbstractType<?> inputType)
{
@ -45,9 +45,18 @@ public class NullMaskingFunction extends MaskingFunction
}
@Override
public final ByteBuffer execute(ProtocolVersion protocolVersion, List<ByteBuffer> parameters)
public Masker masker(ByteBuffer... parameters)
{
return null;
return MASKER;
}
private static class Masker implements MaskingFunction.Masker
{
@Override
public ByteBuffer mask(ByteBuffer value)
{
return null;
}
}
/** @return a {@link FunctionFactory} to build new {@link NullMaskingFunction}s. */

63
src/java/org/apache/cassandra/cql3/functions/masking/PartialMaskingFunction.java
View File

@ -37,7 +37,6 @@ import org.apache.cassandra.db.marshal.AbstractType;
import org.apache.cassandra.db.marshal.Int32Type;
import org.apache.cassandra.db.marshal.UTF8Type;
import org.apache.cassandra.exceptions.InvalidRequestException;
import org.apache.cassandra.transport.ProtocolVersion;
/**
* A {@link MaskingFunction} applied to a {@link org.apache.cassandra.db.marshal.StringType} value that,
@ -88,37 +87,53 @@ public class PartialMaskingFunction extends MaskingFunction
}
@Override
public final ByteBuffer execute(ProtocolVersion protocolVersion, List<ByteBuffer> parameters)
public Masker masker(ByteBuffer... parameters)
{
// Parse the beginning and end positions. No validation is needed since the masker accepts negatives,
// but we should consider that the arguments migh be null.
int begin = parameters.get(1) == null ? 0 : Int32Type.instance.compose(parameters.get(1));
int end = parameters.get(2) == null ? 0 : Int32Type.instance.compose(parameters.get(2));
return new Masker(parameters);
}
// Parse the padding character. The type of the argument is a string of any length because we don't have a
// character type in CQL, so we should verify that the passed string argument is single-character.
char padding = DEFAULT_PADDING_CHAR;
if (hasPaddingArgument && parameters.get(3) != null)
private class Masker implements MaskingFunction.Masker
{
private final int begin, end;
private final char padding;
private Masker(ByteBuffer... parameters)
{
String parameter = UTF8Type.instance.compose(parameters.get(3));
if (parameter.length() != 1)
// Parse the beginning and end positions. No validation is needed since the masker accepts negatives,
// but we should consider that the arguments migh be null.
begin = parameters[0] == null ? 0 : Int32Type.instance.compose(parameters[0]);
end = parameters[1] == null ? 0 : Int32Type.instance.compose(parameters[1]);
// Parse the padding character. The type of the argument is a string of any length because we don't have a
// character type in CQL, so we should verify that the passed string argument is single-character.
if (hasPaddingArgument && parameters[2] != null)
{
throw new InvalidRequestException(String.format("The padding argument for function %s should " +
"be single-character, but '%s' has %d characters.",
name(), parameter, parameter.length()));
String parameter = UTF8Type.instance.compose(parameters[2]);
if (parameter.length() != 1)
{
throw new InvalidRequestException(String.format("The padding argument for function %s should " +
"be single-character, but '%s' has %d characters.",
name(), parameter, parameter.length()));
}
padding = parameter.charAt(0);
}
else
{
padding = DEFAULT_PADDING_CHAR;
}
padding = parameter.charAt(0);
}
// Null column values aren't masked
ByteBuffer value = parameters.get(0);
if (value == null)
return null;
@Override
public ByteBuffer mask(ByteBuffer value)
{
// Null column values aren't masked
if (value == null)
return null;
// We mask the string representation of the column value, even if the type of the column is not a string.
String stringValue = inputType.compose(value);
String maskedValue = type.mask(stringValue, begin, end, padding);
return inputType.decompose(maskedValue);
String stringValue = inputType.compose(value);
String maskedValue = type.mask(stringValue, begin, end, padding);
return inputType.decompose(maskedValue);
}
}
public enum Type

21
src/java/org/apache/cassandra/cql3/functions/masking/ReplaceMaskingFunction.java
View File

@ -26,7 +26,6 @@ import org.apache.cassandra.cql3.functions.FunctionName;
import org.apache.cassandra.cql3.functions.FunctionParameter;
import org.apache.cassandra.cql3.functions.NativeFunction;
import org.apache.cassandra.db.marshal.AbstractType;
import org.apache.cassandra.transport.ProtocolVersion;
/**
* A {@link MaskingFunction} that replaces the specified column value by a certain replacement value.
@ -45,9 +44,25 @@ public class ReplaceMaskingFunction extends MaskingFunction
}
@Override
public final ByteBuffer execute(ProtocolVersion protocolVersion, List<ByteBuffer> parameters)
public Masker masker(ByteBuffer... parameters)
{
return parameters.get(1);
return new Masker(parameters[0]);
}
private static class Masker implements MaskingFunction.Masker
{
private final ByteBuffer replacement;
private Masker(ByteBuffer replacement)
{
this.replacement = replacement;
}
@Override
public ByteBuffer mask(ByteBuffer value)
{
return replacement;
}
}
/** @return a {@link FunctionFactory} to build new {@link ReplaceMaskingFunction}s. */

15
src/java/org/apache/cassandra/cql3/selection/ResultSetBuilder.java
View File

@ -47,6 +47,11 @@ public final class ResultSetBuilder
*/
private final GroupMaker groupMaker;
/**
* Whether masked columns should be unmasked.
*/
private final boolean unmask;
/*
* We'll build CQL3 row one by one.
*/
@ -55,16 +60,17 @@ public final class ResultSetBuilder
private long size = 0;
private boolean sizeWarningEmitted = false;
public ResultSetBuilder(ResultMetadata metadata, Selectors selectors)
public ResultSetBuilder(ResultMetadata metadata, Selectors selectors, boolean unmask)
{
this(metadata, selectors, null);
this(metadata, selectors, unmask, null);
}
public ResultSetBuilder(ResultMetadata metadata, Selectors selectors, GroupMaker groupMaker)
public ResultSetBuilder(ResultMetadata metadata, Selectors selectors, boolean unmask, GroupMaker groupMaker)
{
this.resultSet = new ResultSet(metadata.copy(), new ArrayList<List<ByteBuffer>>());
this.resultSet = new ResultSet(metadata.copy(), new ArrayList<>());
this.selectors = selectors;
this.groupMaker = groupMaker;
this.unmask = unmask;
}
private void addSize(List<ByteBuffer> row)
@ -139,6 +145,7 @@ public final class ResultSetBuilder
{
inputRow = new Selector.InputRow(protocolVersion,
columns,
unmask,
selectors.collectWritetimes(),
selectors.collectTTLs());
}

13
src/java/org/apache/cassandra/cql3/selection/Selectable.java
View File

@ -79,7 +79,7 @@ public interface Selectable extends AssignmentTestable
*/
public default boolean processesSelection()
{
// ColumnMetadata is the only case that returns false and override this
// ColumnMetadata is the only case that returns false (if the column is not masked) and overrides this
return true;
}
@ -322,7 +322,7 @@ public interface Selectable extends AssignmentTestable
@Override
public WritetimeOrTTL prepare(TableMetadata table)
{
return new WritetimeOrTTL(column.prepare(table), selected.prepare(table), kind);
return new WritetimeOrTTL((ColumnMetadata) column.prepare(table), selected.prepare(table), kind);
}
}
}
@ -1240,12 +1240,17 @@ public interface Selectable extends AssignmentTestable
this.quoted = quoted;
}
@Override
public ColumnMetadata prepare(TableMetadata cfm)
public ColumnMetadata columnMetadata(TableMetadata cfm)
{
return cfm.getExistingColumn(ColumnIdentifier.getInterned(text, quoted));
}
@Override
public Selectable prepare(TableMetadata cfm)
{
return columnMetadata(cfm);
}
public FieldIdentifier toFieldIdentifier()
{
return quoted ? FieldIdentifier.forQuoted(text)

26
src/java/org/apache/cassandra/cql3/selection/Selection.java
View File

@ -103,15 +103,20 @@ public abstract class Selection
*/
public Integer getOrderingIndex(ColumnMetadata c)
{
if (!isJson)
return getResultSetIndex(c);
// If we order post-query in json, the first and only column that we ship to the client is the json column.
// In that case, we should keep ordering columns around to perform the ordering, then these columns will
// be placed after the json column. As a consequence of where the colums are placed, we should give the
// ordering index a value based on their position in the json encoding and discard the original index.
// (CASSANDRA-14286)
return orderingColumns.indexOf(c) + 1;
if (isJson)
return orderingColumns.indexOf(c) + 1;
// If the column is masked it might appear twice, once masked in the selected column and once unmasked in
// the ordering columns. For ordering we are interested in that second unmasked value.
if (c.isMasked())
return columns.lastIndexOf(c);
return getResultSetIndex(c);
}
public ResultSet.ResultMetadata getResultMetadata()
@ -133,15 +138,16 @@ public abstract class Selection
return new SimpleSelection(table, all, Collections.emptySet(), true, isJson, returnStaticContentOnPartitionWithNoRows);
}
public static Selection wildcardWithGroupBy(TableMetadata table,
VariableSpecifications boundNames,
boolean isJson,
boolean returnStaticContentOnPartitionWithNoRows)
public static Selection wildcardWithGroupByOrMaskedColumns(TableMetadata table,
VariableSpecifications boundNames,
Set<ColumnMetadata> orderingColumns,
boolean isJson,
boolean returnStaticContentOnPartitionWithNoRows)
{
return fromSelectors(table,
Lists.newArrayList(table.allColumnsInSelectOrder()),
boundNames,
Collections.emptySet(),
orderingColumns,
Collections.emptySet(),
true,
isJson,
@ -225,7 +231,7 @@ public abstract class Selection
for (ColumnMetadata orderingColumn : orderingColumns)
{
int index = selectedColumns.indexOf(orderingColumn);
if (index >= 0 && factories.indexOfSimpleSelectorFactory(index) >= 0)
if (index >= 0 && factories.indexOfSimpleSelectorFactory(index) >= 0 && !orderingColumn.isMasked())
continue;
filteredOrderingColumns.add(orderingColumn);

12
src/java/org/apache/cassandra/cql3/selection/Selector.java
View File

@ -306,6 +306,7 @@ public abstract class Selector
{
private final ProtocolVersion protocolVersion;
private final List<ColumnMetadata> columns;
private final boolean unmask;
private final boolean collectWritetimes;
private final boolean collectTTLs;
@ -314,18 +315,20 @@ public abstract class Selector
private RowTimestamps ttls;
private int index;
public InputRow(ProtocolVersion protocolVersion, List<ColumnMetadata> columns)
public InputRow(ProtocolVersion protocolVersion, List<ColumnMetadata> columns, boolean unmask)
{
this(protocolVersion, columns, false, false);
this(protocolVersion, columns, unmask, false, false);
}
public InputRow(ProtocolVersion protocolVersion,
List<ColumnMetadata> columns,
boolean unmask,
boolean collectWritetimes,
boolean collectTTLs)
{
this.protocolVersion = protocolVersion;
this.columns = columns;
this.unmask = unmask;
this.collectWritetimes = collectWritetimes;
this.collectTTLs = collectTTLs;
@ -347,6 +350,11 @@ public abstract class Selector
return protocolVersion;
}
public boolean unmask()
{
return unmask;
}
public void add(ByteBuffer v)
{
values[index] = v;

2
src/java/org/apache/cassandra/cql3/selection/SelectorFactories.java
View File

@ -147,7 +147,7 @@ final class SelectorFactories implements Iterable<Selector.Factory>
*/
public void addSelectorForOrdering(ColumnMetadata def, int index)
{
factories.add(SimpleSelector.newFactory(def, index));
factories.add(SimpleSelector.newFactory(def, index, true));
}
/**

31
src/java/org/apache/cassandra/cql3/selection/SimpleSelector.java
View File

@ -22,6 +22,7 @@ import java.nio.ByteBuffer;
import com.google.common.base.Objects;
import org.apache.cassandra.cql3.functions.masking.ColumnMask;
import org.apache.cassandra.schema.ColumnMetadata;
import org.apache.cassandra.schema.TableMetadata;
import org.apache.cassandra.cql3.ColumnSpecification;
@ -45,7 +46,7 @@ public final class SimpleSelector extends Selector
ByteBuffer columnName = ByteBufferUtil.readWithVIntLength(in);
ColumnMetadata column = metadata.getColumn(columnName);
int idx = in.readInt();
return new SimpleSelector(column, idx);
return new SimpleSelector(column, idx, false);
}
};
@ -55,13 +56,14 @@ public final class SimpleSelector extends Selector
public static final class SimpleSelectorFactory extends Factory
{
private final int idx;
private final ColumnMetadata column;
private final boolean useForPostOrdering;
private SimpleSelectorFactory(int idx, ColumnMetadata def)
private SimpleSelectorFactory(int idx, ColumnMetadata def, boolean useForPostOrdering)
{
this.idx = idx;
this.column = def;
this.useForPostOrdering = useForPostOrdering;
}
@Override
@ -84,7 +86,7 @@ public final class SimpleSelector extends Selector
@Override
public Selector newInstance(QueryOptions options)
{
return new SimpleSelector(column, idx);
return new SimpleSelector(column, idx, useForPostOrdering);
}
@Override
@ -117,14 +119,15 @@ public final class SimpleSelector extends Selector
public final ColumnMetadata column;
private final int idx;
private final boolean useForPostOrdering;
private ByteBuffer current;
private ColumnTimestamps writetimes;
private ColumnTimestamps ttls;
private boolean isSet;
public static Factory newFactory(final ColumnMetadata def, final int idx)
public static Factory newFactory(final ColumnMetadata def, final int idx, boolean useForPostOrdering)
{
return new SimpleSelectorFactory(idx, def);
return new SimpleSelectorFactory(idx, def, useForPostOrdering);
}
@Override
@ -139,9 +142,18 @@ public final class SimpleSelector extends Selector
if (!isSet)
{
isSet = true;
current = input.getValue(idx);
writetimes = input.getWritetimes(idx);
ttls = input.getTtls(idx);
/*
We apply the column mask of the column unless:
- The column doesn't have a mask
- This selector is for a query with ORDER BY post-ordering, indicated by this.useForPostOrdering
- The input row is for a user with UNMASK permission, indicated by input.unmask()
*/
ColumnMask mask = useForPostOrdering || input.unmask() ? null : column.getMask();
ByteBuffer value = input.getValue(idx);
current = mask == null ? value : mask.mask(input.getProtocolVersion(), value);
}
}
@ -184,11 +196,12 @@ public final class SimpleSelector extends Selector
return column.name.toString();
}
private SimpleSelector(ColumnMetadata column, int idx)
private SimpleSelector(ColumnMetadata column, int idx, boolean useForPostOrdering)
{
super(Kind.SIMPLE_SELECTOR);
this.column = column;
this.idx = idx;
this.useForPostOrdering = useForPostOrdering;
}
@Override
@ -228,6 +241,6 @@ public final class SimpleSelector extends Selector
protected void serialize(DataOutputPlus out, int version) throws IOException
{
ByteBufferUtil.writeWithVIntLength(column.name.bytes, out);
out.writeInt(idx);;
out.writeInt(idx);
}
}

2
src/java/org/apache/cassandra/cql3/statements/ModificationStatement.java
View File

@ -659,7 +659,7 @@ public abstract class ModificationStatement implements CQLStatement.SingleKeyspa
}
Selectors selectors = selection.newSelectors(options);
ResultSetBuilder builder = new ResultSetBuilder(selection.getResultMetadata(), selectors);
ResultSetBuilder builder = new ResultSetBuilder(selection.getResultMetadata(), selectors, false);
SelectStatement.forSelection(metadata, selection)
.processPartition(partition, options, builder, nowInSeconds);

73
src/java/org/apache/cassandra/cql3/statements/SelectStatement.java
View File

@ -22,6 +22,8 @@ import java.util.*;
import java.util.stream.Collectors;
import java.util.concurrent.TimeUnit;
import javax.annotation.concurrent.ThreadSafe;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.MoreObjects;
import com.google.common.collect.ImmutableMap;
@ -80,6 +82,7 @@ import org.apache.cassandra.utils.NoSpamLogger;
import org.apache.commons.lang3.builder.ToStringBuilder;
import org.apache.commons.lang3.builder.ToStringStyle;
import static java.lang.String.format;
import static org.apache.cassandra.cql3.statements.RequestValidations.checkFalse;
import static org.apache.cassandra.cql3.statements.RequestValidations.checkNotNull;
import static org.apache.cassandra.cql3.statements.RequestValidations.checkNull;
@ -96,7 +99,10 @@ import static org.apache.cassandra.utils.Clock.Global.nanoTime;
* many of these are made accessible for the benefit of custom
* QueryHandler implementations, so before reducing their accessibility
* due consideration should be given.
*
* Note that select statements can be accessed by multiple threads, so we cannot rely on mutable attributes.
*/
@ThreadSafe
public class SelectStatement implements CQLStatement.SingleKeyspaceCqlStatement
{
private static final Logger logger = LoggerFactory.getLogger(SelectStatement.class);
@ -236,6 +242,21 @@ public class SelectStatement implements CQLStatement.SingleKeyspaceCqlStatement
for (Function function : getFunctions())
state.ensurePermission(Permission.EXECUTE, function);
if (!state.hasTablePermission(table, Permission.UNMASK) &&
!state.hasTablePermission(table, Permission.SELECT_MASKED))
{
List<ColumnMetadata> queriedMaskedColumns = table.columns()
.stream()
.filter(ColumnMetadata::isMasked)
.filter(restrictions::isRestricted)
.collect(Collectors.toList());
if (!queriedMaskedColumns.isEmpty())
throw new UnauthorizedException(format("User %s has no UNMASK nor SELECT_MASKED permission on table %s.%s, " +
"cannot query masked columns %s",
state.getUser().getName(), keyspace(), table(), queriedMaskedColumns));
}
}
public void validate(ClientState state) throws InvalidRequestException
@ -256,6 +277,7 @@ public class SelectStatement implements CQLStatement.SingleKeyspaceCqlStatement
int userLimit = getLimit(options);
int userPerPartitionLimit = getPerPartitionLimit(options);
int pageSize = options.getPageSize();
boolean unmask = !table.hasMaskedColumns() || state.getClientState().hasTablePermission(table, Permission.UNMASK);
Selectors selectors = selection.newSelectors(options);
AggregationSpecification aggregationSpec = getAggregationSpec(options);
@ -268,7 +290,7 @@ public class SelectStatement implements CQLStatement.SingleKeyspaceCqlStatement
if (aggregationSpec == null && (pageSize <= 0 || (query.limits().count() <= pageSize)))
{
rows = execute(query, options, state.getClientState(), selectors, nowInSec, userLimit, null, queryStartNanoTime);
rows = execute(query, options, state.getClientState(), selectors, nowInSec, userLimit, null, queryStartNanoTime, unmask);
}
else
{
@ -282,7 +304,8 @@ public class SelectStatement implements CQLStatement.SingleKeyspaceCqlStatement
nowInSec,
userLimit,
aggregationSpec,
queryStartNanoTime);
queryStartNanoTime,
unmask);
}
if (!SchemaConstants.isSystemKeyspace(table.keyspace))
ClientRequestSizeMetrics.recordReadResponseMetrics(rows, restrictions, selection);
@ -335,11 +358,12 @@ public class SelectStatement implements CQLStatement.SingleKeyspaceCqlStatement
int nowInSec,
int userLimit,
AggregationSpecification aggregationSpec,
long queryStartNanoTime)
long queryStartNanoTime,
boolean unmask)
{
try (PartitionIterator data = query.execute(options.getConsistency(), state, queryStartNanoTime))
{
return processResults(data, options, selectors, nowInSec, userLimit, aggregationSpec);
return processResults(data, options, selectors, nowInSec, userLimit, aggregationSpec, unmask);
}
}
@ -424,7 +448,8 @@ public class SelectStatement implements CQLStatement.SingleKeyspaceCqlStatement
int nowInSec,
int userLimit,
AggregationSpecification aggregationSpec,
long queryStartNanoTime)
long queryStartNanoTime,
boolean unmask)
{
Guardrails.pageSize.guard(pageSize, table(), false, state.getClientState());
@ -453,7 +478,7 @@ public class SelectStatement implements CQLStatement.SingleKeyspaceCqlStatement
ResultMessage.Rows msg;
try (PartitionIterator page = pager.fetchPage(pageSize, queryStartNanoTime))
{
msg = processResults(page, options, selectors, nowInSec, userLimit, aggregationSpec);
msg = processResults(page, options, selectors, nowInSec, userLimit, aggregationSpec, unmask);
}
// Please note that the isExhausted state of the pager only gets updated when we've closed the page, so this
@ -475,9 +500,10 @@ public class SelectStatement implements CQLStatement.SingleKeyspaceCqlStatement
Selectors selectors,
int nowInSec,
int userLimit,
AggregationSpecification aggregationSpec) throws RequestValidationException
AggregationSpecification aggregationSpec,
boolean unmask) throws RequestValidationException
{
ResultSet rset = process(partitions, options, selectors, nowInSec, userLimit, aggregationSpec);
ResultSet rset = process(partitions, options, selectors, nowInSec, userLimit, aggregationSpec, unmask);
return new ResultMessage.Rows(rset);
}
@ -494,6 +520,7 @@ public class SelectStatement implements CQLStatement.SingleKeyspaceCqlStatement
int userLimit = getLimit(options);
int userPerPartitionLimit = getPerPartitionLimit(options);
int pageSize = options.getPageSize();
boolean unmask = state.getClientState().hasTablePermission(table, Permission.UNMASK);
Selectors selectors = selection.newSelectors(options);
AggregationSpecification aggregationSpec = getAggregationSpec(options);
@ -512,7 +539,7 @@ public class SelectStatement implements CQLStatement.SingleKeyspaceCqlStatement
{
try (PartitionIterator data = query.executeInternal(executionController))
{
return processResults(data, options, selectors, nowInSec, userLimit, null);
return processResults(data, options, selectors, nowInSec, userLimit, null, unmask);
}
}
@ -526,7 +553,8 @@ public class SelectStatement implements CQLStatement.SingleKeyspaceCqlStatement
nowInSec,
userLimit,
aggregationSpec,
queryStartNanoTime);
queryStartNanoTime,
unmask);
}
}
@ -584,11 +612,11 @@ public class SelectStatement implements CQLStatement.SingleKeyspaceCqlStatement
}
}
public ResultSet process(PartitionIterator partitions, int nowInSec) throws InvalidRequestException
public ResultSet process(PartitionIterator partitions, int nowInSec, boolean unmask) throws InvalidRequestException
{
QueryOptions options = QueryOptions.DEFAULT;
Selectors selectors = selection.newSelectors(options);
return process(partitions, options, selectors, nowInSec, getLimit(options), getAggregationSpec(options));
return process(partitions, options, selectors, nowInSec, getLimit(options), getAggregationSpec(options), unmask);
}
@Override
@ -894,10 +922,11 @@ public class SelectStatement implements CQLStatement.SingleKeyspaceCqlStatement
Selectors selectors,
int nowInSec,
int userLimit,
AggregationSpecification aggregationSpec) throws InvalidRequestException
AggregationSpecification aggregationSpec,
boolean unmask) throws InvalidRequestException
{
GroupMaker groupMaker = aggregationSpec == null ? null : aggregationSpec.newGroupMaker();
ResultSetBuilder result = new ResultSetBuilder(getResultMetadata(), selectors, groupMaker);
ResultSetBuilder result = new ResultSetBuilder(getResultMetadata(), selectors, unmask, groupMaker);
while (partitions.hasNext())
{
@ -1170,10 +1199,14 @@ public class SelectStatement implements CQLStatement.SingleKeyspaceCqlStatement
if (hasGroupBy)
Guardrails.groupByEnabled.ensureEnabled(state);
boolean isJson = parameters.isJson;
boolean returnStaticContentOnPartitionWithNoRows = restrictions.returnStaticContentOnPartitionWithNoRows();
if (selectables.isEmpty()) // wildcard query
{
return hasGroupBy ? Selection.wildcardWithGroupBy(table, boundNames, parameters.isJson, restrictions.returnStaticContentOnPartitionWithNoRows())
: Selection.wildcard(table, parameters.isJson, restrictions.returnStaticContentOnPartitionWithNoRows());
return hasGroupBy || table.hasMaskedColumns()
? Selection.wildcardWithGroupByOrMaskedColumns(table, boundNames, resultSetOrderingColumns, isJson, returnStaticContentOnPartitionWithNoRows)
: Selection.wildcard(table, isJson, returnStaticContentOnPartitionWithNoRows);
}
return Selection.fromSelectors(table,
@ -1182,8 +1215,8 @@ public class SelectStatement implements CQLStatement.SingleKeyspaceCqlStatement
resultSetOrderingColumns,
restrictions.nonPKRestrictedColumns(false),
hasGroupBy,
parameters.isJson,
restrictions.returnStaticContentOnPartitionWithNoRows());
isJson,
returnStaticContentOnPartitionWithNoRows);
}
/**
@ -1397,8 +1430,8 @@ public class SelectStatement implements CQLStatement.SingleKeyspaceCqlStatement
if (!restrictions.keyIsInRelation())
return null;
List<Integer> idToSort = new ArrayList<Integer>(orderingColumns.size());
List<Comparator<ByteBuffer>> sorters = new ArrayList<Comparator<ByteBuffer>>(orderingColumns.size());
List<Integer> idToSort = new ArrayList<>(orderingColumns.size());
List<Comparator<ByteBuffer>> sorters = new ArrayList<>(orderingColumns.size());
for (ColumnMetadata orderingColumn : orderingColumns.keySet())
{

119
src/java/org/apache/cassandra/cql3/statements/schema/AlterTableStatement.java
View File

@ -23,9 +23,12 @@ import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.concurrent.TimeUnit;
import javax.annotation.Nullable;
import com.google.common.base.Splitter;
import com.google.common.collect.ImmutableSet;
@ -40,6 +43,7 @@ import org.apache.cassandra.cql3.CQL3Type;
import org.apache.cassandra.cql3.CQLStatement;
import org.apache.cassandra.cql3.ColumnIdentifier;
import org.apache.cassandra.cql3.QualifiedName;
import org.apache.cassandra.cql3.functions.masking.ColumnMask;
import org.apache.cassandra.db.Keyspace;
import org.apache.cassandra.db.guardrails.Guardrails;
import org.apache.cassandra.db.marshal.AbstractType;
@ -158,6 +162,78 @@ public abstract class AlterTableStatement extends AlterSchemaStatement
}
}
/**
* ALTER TABLE [IF EXISTS] <table> ALTER [IF EXISTS] <column> ( MASKED WITH <newMask> | DROP MASKED )
*/
public static class MaskColumn extends AlterTableStatement
{
private final ColumnIdentifier columnName;
@Nullable
private final ColumnMask.Raw rawMask;
private final boolean ifColumnExists;
MaskColumn(String keyspaceName,
String tableName,
ColumnIdentifier columnName,
@Nullable ColumnMask.Raw rawMask,
boolean ifTableExists,
boolean ifColumnExists)
{
super(keyspaceName, tableName, ifTableExists);
this.columnName = columnName;
this.rawMask = rawMask;
this.ifColumnExists = ifColumnExists;
}
@Override
public void validate(ClientState state)
{
super.validate(state);
// we don't allow creating masks if they are disabled, but we still allow dropping them
if (rawMask != null)
ColumnMask.ensureEnabled();
}
@Override
public KeyspaceMetadata apply(KeyspaceMetadata keyspace, TableMetadata table)
{
ColumnMetadata column = table.getColumn(columnName);
if (column == null)
{
if (!ifColumnExists)
throw ire("Column with name '%s' doesn't exist on table '%s'", columnName, tableName);
return keyspace;
}
ColumnMask oldMask = table.getColumn(columnName).getMask();
ColumnMask newMask = rawMask == null ? null : rawMask.prepare(keyspace.name, table.name, columnName, column.type);
if (Objects.equals(oldMask, newMask))
return keyspace;
TableMetadata.Builder tableBuilder = table.unbuild();
tableBuilder.alterColumnMask(columnName, newMask);
TableMetadata newTable = tableBuilder.build();
newTable.validate();
// Update any reference on materialized views, so the mask is consistent among the base table and its views.
Views.Builder viewsBuilder = keyspace.views.unbuild();
for (ViewMetadata view : keyspace.views.forTable(table.id))
{
if (view.includes(columnName))
{
viewsBuilder.put(viewsBuilder.get(view.name()).withNewColumnMask(columnName, newMask));
}
}
return keyspace.withSwapped(keyspace.tables.withSwapped(newTable))
.withSwapped(viewsBuilder.build());
}
}
/**
* ALTER TABLE [IF EXISTS] <table> ADD [IF NOT EXISTS] <column> <newtype>
* ALTER TABLE [IF EXISTS] <table> ADD [IF NOT EXISTS] (<column> <newtype>, <column1> <newtype1>, ... <columnn> <newtypen>)
@ -169,12 +245,15 @@ public abstract class AlterTableStatement extends AlterSchemaStatement
private final ColumnIdentifier name;
private final CQL3Type.Raw type;
private final boolean isStatic;
@Nullable
private final ColumnMask.Raw mask;
Column(ColumnIdentifier name, CQL3Type.Raw type, boolean isStatic)
Column(ColumnIdentifier name, CQL3Type.Raw type, boolean isStatic, @Nullable ColumnMask.Raw mask)
{
this.name = name;
this.type = type;
this.isStatic = isStatic;
this.mask = mask;
}
}
@ -188,12 +267,6 @@ public abstract class AlterTableStatement extends AlterSchemaStatement
this.ifColumnNotExists = ifColumnNotExists;
}
@Override
public void validate(ClientState state)
{
super.validate(state);
}
public KeyspaceMetadata apply(KeyspaceMetadata keyspace, TableMetadata table)
{
Guardrails.alterTableEnabled.ensureEnabled("ALTER TABLE changing columns", state);
@ -220,6 +293,7 @@ public abstract class AlterTableStatement extends AlterSchemaStatement
ColumnIdentifier name = column.name;
AbstractType<?> type = column.type.prepare(keyspaceName, keyspace.types).getType();
boolean isStatic = column.isStatic;
ColumnMask mask = column.mask == null ? null : column.mask.prepare(keyspaceName, tableName, name, type);
if (null != tableBuilder.getColumn(name)) {
if (!ifColumnNotExists)
@ -260,9 +334,9 @@ public abstract class AlterTableStatement extends AlterSchemaStatement
}
if (isStatic)
tableBuilder.addStaticColumn(name, type);
tableBuilder.addStaticColumn(name, type, mask);
else
tableBuilder.addRegularColumn(name, type);
tableBuilder.addRegularColumn(name, type, mask);
if (!isStatic)
{
@ -270,7 +344,8 @@ public abstract class AlterTableStatement extends AlterSchemaStatement
{
if (view.includeAllColumns)
{
ColumnMetadata viewColumn = ColumnMetadata.regularColumn(view.metadata, name.bytes, type);
ColumnMetadata viewColumn = ColumnMetadata.regularColumn(view.metadata, name.bytes, type)
.withNewMask(mask);
viewsBuilder.put(viewsBuilder.get(view.name()).withAddedRegularColumn(viewColumn));
}
}
@ -582,7 +657,13 @@ public abstract class AlterTableStatement extends AlterSchemaStatement
{
private enum Kind
{
ALTER_COLUMN, ADD_COLUMNS, DROP_COLUMNS, RENAME_COLUMNS, ALTER_OPTIONS, DROP_COMPACT_STORAGE
ALTER_COLUMN,
MASK_COLUMN,
ADD_COLUMNS,
DROP_COLUMNS,
RENAME_COLUMNS,
ALTER_OPTIONS,
DROP_COMPACT_STORAGE
}
private final QualifiedName name;
@ -595,6 +676,10 @@ public abstract class AlterTableStatement extends AlterSchemaStatement
// ADD
private final List<AddColumns.Column> addedColumns = new ArrayList<>();
// ALTER MASK
private ColumnIdentifier maskedColumn = null;
private ColumnMask.Raw rawMask = null;
// DROP
private final Set<ColumnIdentifier> droppedColumns = new HashSet<>();
private Long timestamp = null; // will use execution timestamp if not provided by query
@ -619,6 +704,7 @@ public abstract class AlterTableStatement extends AlterSchemaStatement
switch (kind)
{
case ALTER_COLUMN: return new AlterColumn(keyspaceName, tableName, ifTableExists);
case MASK_COLUMN: return new MaskColumn(keyspaceName, tableName, maskedColumn, rawMask, ifTableExists, ifColumnExists);
case ADD_COLUMNS: return new AddColumns(keyspaceName, tableName, addedColumns, ifTableExists, ifColumnNotExists);
case DROP_COLUMNS: return new DropColumns(keyspaceName, tableName, droppedColumns, ifTableExists, ifColumnExists, timestamp);
case RENAME_COLUMNS: return new RenameColumns(keyspaceName, tableName, renamedColumns, ifTableExists, ifColumnExists);
@ -634,10 +720,17 @@ public abstract class AlterTableStatement extends AlterSchemaStatement
kind = Kind.ALTER_COLUMN;
}
public void add(ColumnIdentifier name, CQL3Type.Raw type, boolean isStatic)
public void mask(ColumnIdentifier name, ColumnMask.Raw mask)
{
kind = Kind.MASK_COLUMN;
maskedColumn = name;
rawMask = mask;
}
public void add(ColumnIdentifier name, CQL3Type.Raw type, boolean isStatic, @Nullable ColumnMask.Raw mask)
{
kind = Kind.ADD_COLUMNS;
addedColumns.add(new AddColumns.Column(name, type, isStatic));
addedColumns.add(new AddColumns.Column(name, type, isStatic, mask));
}
public void drop(ColumnIdentifier name)

158
src/java/org/apache/cassandra/cql3/statements/schema/CreateTableStatement.java
View File

@ -19,6 +19,8 @@ package org.apache.cassandra.cql3.statements.schema;
import java.util.*;
import javax.annotation.Nullable;
import com.google.common.collect.ImmutableSet;
import org.apache.commons.lang3.StringUtils;
@ -33,6 +35,7 @@ import org.apache.cassandra.auth.IResource;
import org.apache.cassandra.auth.Permission;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.cql3.*;
import org.apache.cassandra.cql3.functions.masking.ColumnMask;
import org.apache.cassandra.db.Keyspace;
import org.apache.cassandra.db.guardrails.Guardrails;
import org.apache.cassandra.db.marshal.*;
@ -54,7 +57,7 @@ public final class CreateTableStatement extends AlterSchemaStatement
private static final Logger logger = LoggerFactory.getLogger(CreateTableStatement.class);
private final String tableName;
private final Map<ColumnIdentifier, CQL3Type.Raw> rawColumns;
private final Map<ColumnIdentifier, ColumnProperties.Raw> rawColumns;
private final Set<ColumnIdentifier> staticColumns;
private final List<ColumnIdentifier> partitionKeyColumns;
private final List<ColumnIdentifier> clusteringColumns;
@ -67,15 +70,12 @@ public final class CreateTableStatement extends AlterSchemaStatement
public CreateTableStatement(String keyspaceName,
String tableName,
Map<ColumnIdentifier, CQL3Type.Raw> rawColumns,
Map<ColumnIdentifier, ColumnProperties.Raw> rawColumns,
Set<ColumnIdentifier> staticColumns,
List<ColumnIdentifier> partitionKeyColumns,
List<ColumnIdentifier> clusteringColumns,
LinkedHashMap<ColumnIdentifier, Boolean> clusteringOrder,
TableAttributes attrs,
boolean ifNotExists,
boolean useCompactStorage)
{
@ -151,6 +151,13 @@ public final class CreateTableStatement extends AlterSchemaStatement
Guardrails.compactTablesEnabled.ensureEnabled(state);
validateDefaultTimeToLive(attrs.asNewTableParams());
// Verify that dynamic data masking is enabled if there are masked columns
for (ColumnProperties.Raw raw : rawColumns.values())
{
if (raw.rawMask != null)
ColumnMask.ensureEnabled();
}
}
SchemaChange schemaChangeEvent(KeyspacesDiff diff)
@ -186,15 +193,16 @@ public final class CreateTableStatement extends AlterSchemaStatement
TableParams params = attrs.asNewTableParams();
// use a TreeMap to preserve ordering across JDK versions (see CASSANDRA-9492) - important for stable unit tests
Map<ColumnIdentifier, CQL3Type> columns = new TreeMap<>(comparing(o -> o.bytes));
rawColumns.forEach((column, type) -> columns.put(column, type.prepare(keyspaceName, types)));
Map<ColumnIdentifier, ColumnProperties> columns = new TreeMap<>(comparing(o -> o.bytes));
rawColumns.forEach((column, properties) -> columns.put(column, properties.prepare(keyspaceName, tableName, column, types)));
// check for nested non-frozen UDTs or collections in a non-frozen UDT
columns.forEach((column, type) ->
columns.forEach((column, properties) ->
{
if (type.isUDT() && type.getType().isMultiCell())
AbstractType<?> type = properties.type;
if (type.isUDT() && type.isMultiCell())
{
((UserType) type.getType()).fieldTypes().forEach(field ->
((UserType) type).fieldTypes().forEach(field ->
{
if (field.isMultiCell())
throw ire("Non-frozen UDTs with nested non-frozen collections are not supported");
@ -209,45 +217,47 @@ public final class CreateTableStatement extends AlterSchemaStatement
HashSet<ColumnIdentifier> primaryKeyColumns = new HashSet<>();
concat(partitionKeyColumns, clusteringColumns).forEach(column ->
{
CQL3Type type = columns.get(column);
if (null == type)
ColumnProperties properties = columns.get(column);
if (null == properties)
throw ire("Unknown column '%s' referenced in PRIMARY KEY for table '%s'", column, tableName);
if (!primaryKeyColumns.add(column))
throw ire("Duplicate column '%s' in PRIMARY KEY clause for table '%s'", column, tableName);
if (type.getType().isMultiCell())
AbstractType<?> type = properties.type;
if (type.isMultiCell())
{
CQL3Type cqlType = properties.cqlType;
if (type.isCollection())
throw ire("Invalid non-frozen collection type %s for PRIMARY KEY column '%s'", type, column);
throw ire("Invalid non-frozen collection type %s for PRIMARY KEY column '%s'", cqlType, column);
else
throw ire("Invalid non-frozen user-defined type %s for PRIMARY KEY column '%s'", type, column);
throw ire("Invalid non-frozen user-defined type %s for PRIMARY KEY column '%s'", cqlType, column);
}
if (type.getType().isCounter())
if (type.isCounter())
throw ire("counter type is not supported for PRIMARY KEY column '%s'", column);
if (type.getType().referencesDuration())
if (type.referencesDuration())
throw ire("duration type is not supported for PRIMARY KEY column '%s'", column);
if (staticColumns.contains(column))
throw ire("Static column '%s' cannot be part of the PRIMARY KEY", column);
});
List<AbstractType<?>> partitionKeyTypes = new ArrayList<>();
List<AbstractType<?>> clusteringTypes = new ArrayList<>();
List<ColumnProperties> partitionKeyColumnProperties = new ArrayList<>();
List<ColumnProperties> clusteringColumnProperties = new ArrayList<>();
partitionKeyColumns.forEach(column ->
{
CQL3Type type = columns.remove(column);
partitionKeyTypes.add(type.getType());
ColumnProperties columnProperties = columns.remove(column);
partitionKeyColumnProperties.add(columnProperties);
});
clusteringColumns.forEach(column ->
{
CQL3Type type = columns.remove(column);
ColumnProperties columnProperties = columns.remove(column);
boolean reverse = !clusteringOrder.getOrDefault(column, true);
clusteringTypes.add(reverse ? ReversedType.getInstance(type.getType()) : type.getType());
clusteringColumnProperties.add(reverse ? columnProperties.withReversedType() : columnProperties);
});
if (clusteringOrder.size() > clusteringColumns.size())
@ -270,7 +280,7 @@ public final class CreateTableStatement extends AlterSchemaStatement
// For COMPACT STORAGE, we reject any "feature" that we wouldn't be able to translate back to thrift.
if (useCompactStorage)
{
validateCompactTable(clusteringTypes, columns);
validateCompactTable(clusteringColumnProperties, columns);
}
else
{
@ -283,12 +293,12 @@ public final class CreateTableStatement extends AlterSchemaStatement
* Counter table validation
*/
boolean hasCounters = rawColumns.values().stream().anyMatch(CQL3Type.Raw::isCounter);
boolean hasCounters = rawColumns.values().stream().anyMatch(c -> c.rawType.isCounter());
if (hasCounters)
{
// We've handled anything that is not a PRIMARY KEY so columns only contains NON-PK columns. So
// if it's a counter table, make sure we don't have non-counter types
if (columns.values().stream().anyMatch(t -> !t.getType().isCounter()))
if (columns.values().stream().anyMatch(t -> !t.type.isCounter()))
throw ire("Cannot mix counter and non counter columns in the same table");
if (params.defaultTimeToLive > 0)
@ -308,38 +318,44 @@ public final class CreateTableStatement extends AlterSchemaStatement
.params(params);
for (int i = 0; i < partitionKeyColumns.size(); i++)
builder.addPartitionKeyColumn(partitionKeyColumns.get(i), partitionKeyTypes.get(i));
{
ColumnProperties properties = partitionKeyColumnProperties.get(i);
builder.addPartitionKeyColumn(partitionKeyColumns.get(i), properties.type, properties.mask);
}
for (int i = 0; i < clusteringColumns.size(); i++)
builder.addClusteringColumn(clusteringColumns.get(i), clusteringTypes.get(i));
{
ColumnProperties properties = clusteringColumnProperties.get(i);
builder.addClusteringColumn(clusteringColumns.get(i), properties.type, properties.mask);
}
if (useCompactStorage)
{
fixupCompactTable(clusteringTypes, columns, hasCounters, builder);
fixupCompactTable(clusteringColumnProperties, columns, hasCounters, builder);
}
else
{
columns.forEach((column, type) -> {
columns.forEach((column, properties) -> {
if (staticColumns.contains(column))
builder.addStaticColumn(column, type.getType());
builder.addStaticColumn(column, properties.type, properties.mask);
else
builder.addRegularColumn(column, type.getType());
builder.addRegularColumn(column, properties.type, properties.mask);
});
}
return builder;
}
private void validateCompactTable(List<AbstractType<?>> clusteringTypes,
Map<ColumnIdentifier, CQL3Type> columns)
private void validateCompactTable(List<ColumnProperties> clusteringColumnProperties,
Map<ColumnIdentifier, ColumnProperties> columns)
{
boolean isDense = !clusteringTypes.isEmpty();
boolean isDense = !clusteringColumnProperties.isEmpty();
if (columns.values().stream().anyMatch(c -> c.getType().isMultiCell()))
if (columns.values().stream().anyMatch(c -> c.type.isMultiCell()))
throw ire("Non-frozen collections and UDTs are not supported with COMPACT STORAGE");
if (!staticColumns.isEmpty())
throw ire("Static columns are not supported in COMPACT STORAGE tables");
if (clusteringTypes.isEmpty())
if (clusteringColumnProperties.isEmpty())
{
// It's a thrift "static CF" so there should be some columns definition
if (columns.isEmpty())
@ -361,8 +377,8 @@ public final class CreateTableStatement extends AlterSchemaStatement
}
}
private void fixupCompactTable(List<AbstractType<?>> clusteringTypes,
Map<ColumnIdentifier, CQL3Type> columns,
private void fixupCompactTable(List<ColumnProperties> clusteringTypes,
Map<ColumnIdentifier, ColumnProperties> columns,
boolean hasCounters,
TableMetadata.Builder builder)
{
@ -381,12 +397,12 @@ public final class CreateTableStatement extends AlterSchemaStatement
builder.flags(flags);
columns.forEach((name, type) -> {
columns.forEach((name, properties) -> {
// Note that for "static" no-clustering compact storage we use static for the defined columns
if (staticColumns.contains(name) || isStaticCompact)
builder.addStaticColumn(name, type.getType());
builder.addStaticColumn(name, properties.type, properties.mask);
else
builder.addRegularColumn(name, type.getType());
builder.addRegularColumn(name, properties.type, properties.mask);
});
DefaultNames names = new DefaultNames(builder.columnNames());
@ -471,7 +487,7 @@ public final class CreateTableStatement extends AlterSchemaStatement
private final boolean ifNotExists;
private boolean useCompactStorage = false;
private final Map<ColumnIdentifier, CQL3Type.Raw> rawColumns = new HashMap<>();
private final Map<ColumnIdentifier, ColumnProperties.Raw> rawColumns = new HashMap<>();
private final Set<ColumnIdentifier> staticColumns = new HashSet<>();
private final List<ColumnIdentifier> clusteringColumns = new ArrayList<>();
@ -495,15 +511,12 @@ public final class CreateTableStatement extends AlterSchemaStatement
return new CreateTableStatement(keyspaceName,
name.getName(),
rawColumns,
staticColumns,
partitionKeyColumns,
clusteringColumns,
clusteringOrder,
attrs,
ifNotExists,
useCompactStorage);
}
@ -524,9 +537,10 @@ public final class CreateTableStatement extends AlterSchemaStatement
return name.getName();
}
public void addColumn(ColumnIdentifier column, CQL3Type.Raw type, boolean isStatic)
public void addColumn(ColumnIdentifier column, CQL3Type.Raw type, boolean isStatic, ColumnMask.Raw mask)
{
if (null != rawColumns.put(column, type))
if (null != rawColumns.put(column, new ColumnProperties.Raw(type, mask)))
throw ire("Duplicate column '%s' declaration for table '%s'", column, name);
if (isStatic)
@ -562,4 +576,52 @@ public final class CreateTableStatement extends AlterSchemaStatement
throw ire("Duplicate column '%s' in CLUSTERING ORDER BY clause for table '%s'", column, name);
}
}
/**
* Class encapsulating the properties of a column, which are its type and mask.
*/
private final static class ColumnProperties
{
public final AbstractType<?> type;
public final CQL3Type cqlType; // we keep the original CQL type for printing fully qualified user type names
@Nullable
public final ColumnMask mask;
public ColumnProperties(AbstractType<?> type, CQL3Type cqlType, @Nullable ColumnMask mask)
{
this.type = type;
this.cqlType = cqlType;
this.mask = mask;
}
public ColumnProperties withReversedType()
{
return new ColumnProperties(ReversedType.getInstance(type),
cqlType,
mask == null ? null : mask.withReversedType());
}
public final static class Raw
{
public final CQL3Type.Raw rawType;
@Nullable
public final ColumnMask.Raw rawMask;
public Raw(CQL3Type.Raw rawType, @Nullable ColumnMask.Raw rawMask)
{
this.rawType = rawType;
this.rawMask = rawMask;
}
public ColumnProperties prepare(String keyspace, String table, ColumnIdentifier column, Types udts)
{
CQL3Type cqlType = rawType.prepare(keyspace, udts);
AbstractType<?> type = cqlType.getType();
ColumnMask mask = rawMask == null ? null : rawMask.prepare(keyspace, table, column, type);
return new ColumnProperties(type, cqlType, mask);
}
}
}
}

25
src/java/org/apache/cassandra/cql3/statements/schema/CreateViewStatement.java
View File

@ -188,7 +188,8 @@ public final class CreateViewStatement extends AlterSchemaStatement
throw ire("Can only select columns by name when defining a materialized view (got %s)", selector.selectable);
// will throw IRE if the column doesn't exist in the base table
ColumnMetadata column = (ColumnMetadata) selector.selectable.prepare(table);
Selectable.RawIdentifier rawIdentifier = (Selectable.RawIdentifier) selector.selectable;
ColumnMetadata column = rawIdentifier.columnMetadata(table);
selectedColumns.add(column.name);
});
@ -325,12 +326,18 @@ public final class CreateViewStatement extends AlterSchemaStatement
builder.params(attrs.asNewTableParams())
.kind(TableMetadata.Kind.VIEW);
partitionKeyColumns.forEach(name -> builder.addPartitionKeyColumn(name, getType(table, name)));
clusteringColumns.forEach(name -> builder.addClusteringColumn(name, getType(table, name)));
partitionKeyColumns.stream()
.map(table::getColumn)
.forEach(column -> builder.addPartitionKeyColumn(column.name, getType(column), column.getMask()));
clusteringColumns.stream()
.map(table::getColumn)
.forEach(column -> builder.addClusteringColumn(column.name, getType(column), column.getMask()));
selectedColumns.stream()
.filter(name -> !primaryKeyColumns.contains(name))
.forEach(name -> builder.addRegularColumn(name, getType(table, name)));
.map(table::getColumn)
.forEach(column -> builder.addRegularColumn(column.name, getType(column), column.getMask()));
ViewMetadata view = new ViewMetadata(table.id, table.name, rawColumns.isEmpty(), whereClause, builder.build());
view.metadata.validate();
@ -348,15 +355,15 @@ public final class CreateViewStatement extends AlterSchemaStatement
client.ensureTablePermission(keyspaceName, tableName, Permission.ALTER);
}
private AbstractType<?> getType(TableMetadata table, ColumnIdentifier name)
private AbstractType<?> getType(ColumnMetadata column)
{
AbstractType<?> type = table.getColumn(name).type;
if (clusteringOrder.containsKey(name))
AbstractType<?> type = column.type;
if (clusteringOrder.containsKey(column.name))
{
boolean reverse = !clusteringOrder.get(name);
boolean reverse = !clusteringOrder.get(column.name);
if (type.isReversed() && !reverse)
return ((ReversedType) type).baseType;
return ((ReversedType<?>) type).baseType;
if (!type.isReversed() && reverse)
return ReversedType.getInstance(type);

7
src/java/org/apache/cassandra/cql3/statements/schema/DropFunctionStatement.java
View File

@ -120,6 +120,13 @@ public final class DropFunctionStatement extends AlterSchemaStatement
if (!dependentAggregates.isEmpty())
throw ire("Function '%s' is still referenced by aggregates %s", name, dependentAggregates);
String dependentTables = keyspace.tablesUsingFunction(function)
.map(table -> table.name)
.collect(joining(", "));
if (!dependentTables.isEmpty())
throw ire("Function '%s' is still referenced by column masks in tables %s", name, dependentTables);
return schema.withAddedOrUpdated(keyspace.withSwapped(keyspace.userFunctions.without(function)));
}

6
src/java/org/apache/cassandra/db/Columns.java
View File

@ -61,7 +61,8 @@ public class Columns extends AbstractCollection<ColumnMetadata> implements Colle
ColumnIdentifier.getInterned(ByteBufferUtil.EMPTY_BYTE_BUFFER, UTF8Type.instance),
SetType.getInstance(UTF8Type.instance, true),
ColumnMetadata.NO_POSITION,
ColumnMetadata.Kind.STATIC);
ColumnMetadata.Kind.STATIC,
null);
public static final ColumnMetadata FIRST_COMPLEX_REGULAR =
new ColumnMetadata("",
@ -69,7 +70,8 @@ public class Columns extends AbstractCollection<ColumnMetadata> implements Colle
ColumnIdentifier.getInterned(ByteBufferUtil.EMPTY_BYTE_BUFFER, UTF8Type.instance),
SetType.getInstance(UTF8Type.instance, true),
ColumnMetadata.NO_POSITION,
ColumnMetadata.Kind.REGULAR);
ColumnMetadata.Kind.REGULAR,
null);
private final Object[] columns;
private final int complexIdx; // Index of the first complex column

4
src/java/org/apache/cassandra/db/aggregation/GroupMaker.java
View File

@ -172,7 +172,7 @@ public abstract class GroupMaker
{
super(comparator, clusteringPrefixSize, state);
this.selector = selector;
this.input = new Selector.InputRow(ProtocolVersion.CURRENT, columns);
this.input = new Selector.InputRow(ProtocolVersion.CURRENT, columns, false);
this.lastOutput = lastClustering == null ? null :
executeSelector(lastClustering.bufferAt(clusteringPrefixSize - 1));
}
@ -184,7 +184,7 @@ public abstract class GroupMaker
{
super(comparator, clusteringPrefixSize);
this.selector = selector;
this.input = new Selector.InputRow(ProtocolVersion.CURRENT, columns);
this.input = new Selector.InputRow(ProtocolVersion.CURRENT, columns, false);
}
@Override

78
src/java/org/apache/cassandra/schema/ColumnMetadata.java
View File

@ -21,11 +21,14 @@ import java.nio.ByteBuffer;
import java.util.*;
import java.util.function.Predicate;
import javax.annotation.Nullable;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.MoreObjects;
import com.google.common.collect.Collections2;
import org.apache.cassandra.cql3.*;
import org.apache.cassandra.cql3.functions.masking.ColumnMask;
import org.apache.cassandra.cql3.selection.Selectable;
import org.apache.cassandra.cql3.selection.Selector;
import org.apache.cassandra.cql3.selection.SimpleSelector;
@ -95,6 +98,12 @@ public final class ColumnMetadata extends ColumnSpecification implements Selecta
*/
private final long comparisonOrder;
/**
* Masking function used to dynamically mask the contents of this column.
*/
@Nullable
private final ColumnMask mask;
private static long comparisonOrder(Kind kind, boolean isComplex, long position, ColumnIdentifier name)
{
assert position >= 0 && position < 1 << 12;
@ -106,52 +115,58 @@ public final class ColumnMetadata extends ColumnSpecification implements Selecta
public static ColumnMetadata partitionKeyColumn(TableMetadata table, ByteBuffer name, AbstractType<?> type, int position)
{
return new ColumnMetadata(table, name, type, position, Kind.PARTITION_KEY);
return new ColumnMetadata(table, name, type, position, Kind.PARTITION_KEY, null);
}
public static ColumnMetadata partitionKeyColumn(String keyspace, String table, String name, AbstractType<?> type, int position)
{
return new ColumnMetadata(keyspace, table, ColumnIdentifier.getInterned(name, true), type, position, Kind.PARTITION_KEY);
return new ColumnMetadata(keyspace, table, ColumnIdentifier.getInterned(name, true), type, position, Kind.PARTITION_KEY, null);
}
public static ColumnMetadata clusteringColumn(TableMetadata table, ByteBuffer name, AbstractType<?> type, int position)
{
return new ColumnMetadata(table, name, type, position, Kind.CLUSTERING);
return new ColumnMetadata(table, name, type, position, Kind.CLUSTERING, null);
}
public static ColumnMetadata clusteringColumn(String keyspace, String table, String name, AbstractType<?> type, int position)
{
return new ColumnMetadata(keyspace, table, ColumnIdentifier.getInterned(name, true), type, position, Kind.CLUSTERING);
return new ColumnMetadata(keyspace, table, ColumnIdentifier.getInterned(name, true), type, position, Kind.CLUSTERING, null);
}
public static ColumnMetadata regularColumn(TableMetadata table, ByteBuffer name, AbstractType<?> type)
{
return new ColumnMetadata(table, name, type, NO_POSITION, Kind.REGULAR);
return new ColumnMetadata(table, name, type, NO_POSITION, Kind.REGULAR, null);
}
public static ColumnMetadata regularColumn(String keyspace, String table, String name, AbstractType<?> type)
{
return new ColumnMetadata(keyspace, table, ColumnIdentifier.getInterned(name, true), type, NO_POSITION, Kind.REGULAR);
return new ColumnMetadata(keyspace, table, ColumnIdentifier.getInterned(name, true), type, NO_POSITION, Kind.REGULAR, null);
}
public static ColumnMetadata staticColumn(TableMetadata table, ByteBuffer name, AbstractType<?> type)
{
return new ColumnMetadata(table, name, type, NO_POSITION, Kind.STATIC);
return new ColumnMetadata(table, name, type, NO_POSITION, Kind.STATIC, null);
}
public static ColumnMetadata staticColumn(String keyspace, String table, String name, AbstractType<?> type)
{
return new ColumnMetadata(keyspace, table, ColumnIdentifier.getInterned(name, true), type, NO_POSITION, Kind.STATIC);
return new ColumnMetadata(keyspace, table, ColumnIdentifier.getInterned(name, true), type, NO_POSITION, Kind.STATIC, null);
}
public ColumnMetadata(TableMetadata table, ByteBuffer name, AbstractType<?> type, int position, Kind kind)
public ColumnMetadata(TableMetadata table,
ByteBuffer name,
AbstractType<?> type,
int position,
Kind kind,
@Nullable ColumnMask mask)
{
this(table.keyspace,
table.name,
ColumnIdentifier.getInterned(name, UTF8Type.instance),
type,
position,
kind);
kind,
mask);
}
@VisibleForTesting
@ -160,18 +175,26 @@ public final class ColumnMetadata extends ColumnSpecification implements Selecta
ColumnIdentifier name,
AbstractType<?> type,
int position,
Kind kind)
Kind kind,
@Nullable ColumnMask mask)
{
super(ksName, cfName, name, type);
assert name != null && type != null && kind != null;
assert (position == NO_POSITION) == !kind.isPrimaryKeyKind(); // The position really only make sense for partition and clustering columns (and those must have one),
// so make sure we don't sneak it for something else since it'd breaks equals()
// The propagation of system distributed keyspaces at startup can be problematic for old nodes without DDM,
// since those won't know what to do with the mask mutations. Thus, we don't support DDM on those keyspaces.
if (mask != null && SchemaConstants.isReplicatedSystemKeyspace(ksName))
throw new AssertionError("DDM is not supported on system distributed keyspaces");
this.kind = kind;
this.position = position;
this.cellPathComparator = makeCellPathComparator(kind, type);
this.cellComparator = cellPathComparator == null ? ColumnData.comparator : (a, b) -> cellPathComparator.compare(a.path(), b.path());
this.asymmetricCellPathComparator = cellPathComparator == null ? null : (a, b) -> cellPathComparator.compare(((Cell<?>)a).path(), (CellPath) b);
this.comparisonOrder = comparisonOrder(kind, isComplex(), Math.max(0, position), name);
this.mask = mask;
}
private static Comparator<CellPath> makeCellPathComparator(Kind kind, AbstractType<?> type)
@ -203,17 +226,22 @@ public final class ColumnMetadata extends ColumnSpecification implements Selecta
public ColumnMetadata copy()
{
return new ColumnMetadata(ksName, cfName, name, type, position, kind);
return new ColumnMetadata(ksName, cfName, name, type, position, kind, mask);
}
public ColumnMetadata withNewName(ColumnIdentifier newName)
{
return new ColumnMetadata(ksName, cfName, newName, type, position, kind);
return new ColumnMetadata(ksName, cfName, newName, type, position, kind, mask);
}
public ColumnMetadata withNewType(AbstractType<?> newType)
{
return new ColumnMetadata(ksName, cfName, name, newType, position, kind);
return new ColumnMetadata(ksName, cfName, name, newType, position, kind, mask);
}
public ColumnMetadata withNewMask(@Nullable ColumnMask newMask)
{
return new ColumnMetadata(ksName, cfName, name, type, position, kind, newMask);
}
public boolean isPartitionKey()
@ -231,6 +259,11 @@ public final class ColumnMetadata extends ColumnSpecification implements Selecta
return kind == Kind.STATIC;
}
public boolean isMasked()
{
return mask != null;
}
public boolean isRegular()
{
return kind == Kind.REGULAR;
@ -249,6 +282,12 @@ public final class ColumnMetadata extends ColumnSpecification implements Selecta
return position;
}
@Nullable
public ColumnMask getMask()
{
return mask;
}
@Override
public boolean equals(Object o)
{
@ -269,7 +308,8 @@ public final class ColumnMetadata extends ColumnSpecification implements Selecta
&& kind == other.kind
&& position == other.position
&& ksName.equals(other.ksName)
&& cfName.equals(other.cfName);
&& cfName.equals(other.cfName)
&& Objects.equals(mask, other.mask);
}
Optional<Difference> compare(ColumnMetadata other)
@ -299,6 +339,7 @@ public final class ColumnMetadata extends ColumnSpecification implements Selecta
result = 31 * result + (type == null ? 0 : type.hashCode());
result = 31 * result + (kind == null ? 0 : kind.hashCode());
result = 31 * result + position;
result = 31 * result + (mask == null ? 0 : mask.hashCode());
hash = result;
}
return result;
@ -334,7 +375,7 @@ public final class ColumnMetadata extends ColumnSpecification implements Selecta
@Override
public boolean processesSelection()
{
return false;
return isMasked();
}
/**
@ -433,6 +474,9 @@ public final class ColumnMetadata extends ColumnSpecification implements Selecta
if (isStatic())
builder.append(" static");
if (isMasked())
mask.appendCqlTo(builder);
}
public static String toCQLString(Iterable<ColumnMetadata> defs)
@ -488,7 +532,7 @@ public final class ColumnMetadata extends ColumnSpecification implements Selecta
public Selector.Factory newSelectorFactory(TableMetadata table, AbstractType<?> expectedType, List<ColumnMetadata> defs, VariableSpecifications boundNames) throws InvalidRequestException
{
return SimpleSelector.newFactory(this, addAndGetIndex(this, defs));
return SimpleSelector.newFactory(this, addAndGetIndex(this, defs), false);
}
public AbstractType<?> getExactTypeIfKnown(String keyspace)

11
src/java/org/apache/cassandra/schema/KeyspaceMetadata.java
View File

@ -20,6 +20,7 @@ package org.apache.cassandra.schema;
import java.util.HashSet;
import java.util.Optional;
import java.util.Set;
import java.util.stream.Stream;
import javax.annotation.Nullable;
@ -30,6 +31,7 @@ import com.google.common.collect.Iterables;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.cql3.CqlBuilder;
import org.apache.cassandra.cql3.SchemaElement;
import org.apache.cassandra.cql3.functions.Function;
import org.apache.cassandra.cql3.functions.UDAggregate;
import org.apache.cassandra.cql3.functions.UDFunction;
import org.apache.cassandra.db.marshal.UserType;
@ -173,6 +175,15 @@ public final class KeyspaceMetadata implements SchemaElement
return any(tables, t -> t.indexes.has(indexName));
}
/**
* @param function a user function
* @return a stream of tables within this keyspace that have column masks using the specified user function
*/
public Stream<TableMetadata> tablesUsingFunction(Function function)
{
return tables.stream().filter(table -> table.dependsOn(function));
}
public String findAvailableIndexName(String baseName)
{
if (!hasIndex(baseName))

156
src/java/org/apache/cassandra/schema/SchemaKeyspace.java
View File

@ -35,6 +35,7 @@ import org.antlr.runtime.RecognitionException;
import org.apache.cassandra.config.*;
import org.apache.cassandra.cql3.*;
import org.apache.cassandra.cql3.functions.*;
import org.apache.cassandra.cql3.functions.masking.ColumnMask;
import org.apache.cassandra.cql3.statements.schema.CreateTableStatement;
import org.apache.cassandra.db.*;
import org.apache.cassandra.db.filter.ColumnFilter;
@ -113,7 +114,7 @@ public final class SchemaKeyspace
+ "extensions frozen<map<text, blob>>,"
+ "flags frozen<set<text>>," // SUPER, COUNTER, DENSE, COMPOUND
+ "gc_grace_seconds int,"
+ "incremental_backups boolean,"
+ "incremental_backups boolean,"
+ "id uuid,"
+ "max_index_interval int,"
+ "memtable_flush_period_in_ms int,"
@ -139,6 +140,20 @@ public final class SchemaKeyspace
+ "type text,"
+ "PRIMARY KEY ((keyspace_name), table_name, column_name))");
private static final TableMetadata ColumnMasks =
parse(COLUMN_MASKS,
"column dynamic data masks",
"CREATE TABLE %s ("
+ "keyspace_name text,"
+ "table_name text,"
+ "column_name text,"
+ "function_keyspace text,"
+ "function_name text,"
+ "function_argument_types frozen<list<text>>,"
+ "function_argument_values frozen<list<text>>,"
+ "function_argument_nulls frozen<list<boolean>>," // arguments that are null
+ "PRIMARY KEY ((keyspace_name), table_name, column_name))");
private static final TableMetadata DroppedColumns =
parse(DROPPED_COLUMNS,
"dropped column registry",
@ -182,7 +197,7 @@ public final class SchemaKeyspace
+ "default_time_to_live int,"
+ "extensions frozen<map<text, blob>>,"
+ "gc_grace_seconds int,"
+ "incremental_backups boolean,"
+ "incremental_backups boolean,"
+ "id uuid,"
+ "include_all_columns boolean,"
+ "max_index_interval int,"
@ -244,8 +259,17 @@ public final class SchemaKeyspace
+ "state_type text,"
+ "PRIMARY KEY ((keyspace_name), aggregate_name, argument_types))");
private static final List<TableMetadata> ALL_TABLE_METADATA =
ImmutableList.of(Keyspaces, Tables, Columns, Triggers, DroppedColumns, Views, Types, Functions, Aggregates, Indexes);
private static final List<TableMetadata> ALL_TABLE_METADATA = ImmutableList.of(Keyspaces,
Tables,
Columns,
ColumnMasks,
Triggers,
DroppedColumns,
Views,
Types,
Functions,
Aggregates,
Indexes);
private static TableMetadata parse(String name, String description, String cql)
{
@ -688,7 +712,7 @@ public final class SchemaKeyspace
{
AbstractType<?> type = column.type;
if (type instanceof ReversedType)
type = ((ReversedType) type).baseType;
type = ((ReversedType<?>) type).baseType;
builder.update(Columns)
.row(table.name, column.name.toString())
@ -697,6 +721,52 @@ public final class SchemaKeyspace
.add("position", column.position())
.add("clustering_order", column.clusteringOrder().toString().toLowerCase())
.add("type", type.asCQL3Type().toString());
ColumnMask mask = column.getMask();
if (SchemaConstants.isReplicatedSystemKeyspace(table.keyspace))
{
// The propagation of system distributed keyspaces at startup can be problematic for old nodes without DDM,
// since those won't know what to do with the mask mutations. Thus, we don't support DDM on those keyspaces.
assert mask == null : "Dynamic data masking shouldn't be used on system distributed keyspaces";
}
else
{
Row.SimpleBuilder maskBuilder = builder.update(ColumnMasks).row(table.name, column.name.toString());
if (mask == null)
{
maskBuilder.delete();
}
else
{
FunctionName maskFunctionName = mask.function.name();
// Some arguments of the masking function can be null, but the CQL's list type that stores them doesn't
// accept nulls, so we use a parallel list of booleans to store what arguments are null.
List<AbstractType<?>> partialTypes = mask.partialArgumentTypes();
List<ByteBuffer> partialValues = mask.partialArgumentValues();
int numArgs = partialTypes.size();
List<String> types = new ArrayList<>(numArgs);
List<String> values = new ArrayList<>(numArgs);
List<Boolean> nulls = new ArrayList<>(numArgs);
for (int i = 0; i < numArgs; i++)
{
AbstractType<?> argType = partialTypes.get(i);
types.add(argType.asCQL3Type().toString());
ByteBuffer argValue = partialValues.get(i);
boolean isNull = argValue == null;
nulls.add(isNull);
values.add(isNull ? "" : argType.getString(argValue));
}
maskBuilder.add("function_keyspace", maskFunctionName.keyspace)
.add("function_name", maskFunctionName.name)
.add("function_argument_types", types)
.add("function_argument_values", values)
.add("function_argument_nulls", nulls);
}
}
}
private static void dropColumnFromSchemaMutation(TableMetadata table, ColumnMetadata column, Mutation.SimpleBuilder builder)
@ -876,9 +946,9 @@ public final class SchemaKeyspace
{
KeyspaceParams params = fetchKeyspaceParams(keyspaceName);
Types types = fetchTypes(keyspaceName);
Tables tables = fetchTables(keyspaceName, types);
Views views = fetchViews(keyspaceName, types);
UserFunctions functions = fetchFunctions(keyspaceName, types);
Tables tables = fetchTables(keyspaceName, types, functions);
Views views = fetchViews(keyspaceName, types, functions);
return KeyspaceMetadata.create(keyspaceName, params, tables, views, types, functions);
}
@ -907,7 +977,7 @@ public final class SchemaKeyspace
return types.build();
}
private static Tables fetchTables(String keyspaceName, Types types)
private static Tables fetchTables(String keyspaceName, Types types, UserFunctions functions)
{
String query = format("SELECT table_name FROM %s.%s WHERE keyspace_name = ?", SchemaConstants.SCHEMA_KEYSPACE_NAME, TABLES);
@ -917,7 +987,7 @@ public final class SchemaKeyspace
String tableName = row.getString("table_name");
try
{
tables.add(fetchTable(keyspaceName, tableName, types));
tables.add(fetchTable(keyspaceName, tableName, types, functions));
}
catch (MissingColumns exc)
{
@ -946,7 +1016,7 @@ public final class SchemaKeyspace
return tables.build();
}
private static TableMetadata fetchTable(String keyspaceName, String tableName, Types types)
private static TableMetadata fetchTable(String keyspaceName, String tableName, Types types, UserFunctions functions)
{
String query = String.format("SELECT * FROM %s.%s WHERE keyspace_name = ? AND table_name = ?", SchemaConstants.SCHEMA_KEYSPACE_NAME, TABLES);
UntypedResultSet rows = query(query, keyspaceName, tableName);
@ -958,7 +1028,7 @@ public final class SchemaKeyspace
return TableMetadata.builder(keyspaceName, tableName, TableId.fromUUID(row.getUUID("id")))
.flags(flags)
.params(createTableParamsFromRow(row))
.addColumns(fetchColumns(keyspaceName, tableName, types))
.addColumns(fetchColumns(keyspaceName, tableName, types, functions))
.droppedColumns(fetchDroppedColumns(keyspaceName, tableName))
.indexes(fetchIndexes(keyspaceName, tableName))
.triggers(fetchTriggers(keyspaceName, tableName))
@ -1002,7 +1072,7 @@ public final class SchemaKeyspace
return builder.build();
}
private static List<ColumnMetadata> fetchColumns(String keyspace, String table, Types types)
private static List<ColumnMetadata> fetchColumns(String keyspace, String table, Types types, UserFunctions functions)
{
String query = format("SELECT * FROM %s.%s WHERE keyspace_name = ? AND table_name = ?", SchemaConstants.SCHEMA_KEYSPACE_NAME, COLUMNS);
UntypedResultSet columnRows = query(query, keyspace, table);
@ -1010,7 +1080,7 @@ public final class SchemaKeyspace
throw new MissingColumns("Columns not found in schema table for " + keyspace + '.' + table);
List<ColumnMetadata> columns = new ArrayList<>();
columnRows.forEach(row -> columns.add(createColumnFromRow(row, types)));
columnRows.forEach(row -> columns.add(createColumnFromRow(row, types, functions)));
if (columns.stream().noneMatch(ColumnMetadata::isPartitionKey))
throw new MissingColumns("No partition key columns found in schema table for " + keyspace + "." + table);
@ -1019,7 +1089,7 @@ public final class SchemaKeyspace
}
@VisibleForTesting
static ColumnMetadata createColumnFromRow(UntypedResultSet.Row row, Types types)
public static ColumnMetadata createColumnFromRow(UntypedResultSet.Row row, Types types, UserFunctions functions)
{
String keyspace = row.getString("keyspace_name");
String table = row.getString("table_name");
@ -1035,7 +1105,53 @@ public final class SchemaKeyspace
ColumnIdentifier name = new ColumnIdentifier(row.getBytes("column_name_bytes"), row.getString("column_name"));
return new ColumnMetadata(keyspace, table, name, type, position, kind);
ColumnMask mask = null;
String query = format("SELECT * FROM %s.%s WHERE keyspace_name = ? AND table_name = ? AND column_name = ?",
SchemaConstants.SCHEMA_KEYSPACE_NAME, COLUMN_MASKS);
UntypedResultSet columnMasks = query(query, keyspace, table, name.toString());
if (!columnMasks.isEmpty())
{
UntypedResultSet.Row maskRow = columnMasks.one();
FunctionName functionName = new FunctionName(maskRow.getString("function_keyspace"), maskRow.getString("function_name"));
List<String> partialArgumentTypes = maskRow.getFrozenList("function_argument_types", UTF8Type.instance);
List<AbstractType<?>> argumentTypes = new ArrayList<>(1 + partialArgumentTypes.size());
argumentTypes.add(type);
for (String argumentType : partialArgumentTypes)
{
argumentTypes.add(CQLTypeParser.parse(keyspace, argumentType, types));
}
Function function = FunctionResolver.get(keyspace, functionName, argumentTypes, null, null, null, functions);
if (function == null)
{
throw new AssertionError(format("Unable to find masking function %s(%s) for column %s.%s.%s",
functionName, argumentTypes, keyspace, table, name));
}
else if (!(function instanceof ScalarFunction))
{
throw new AssertionError(format("Column %s.%s.%s is unexpectedly masked with function %s " +
"which is not a scalar masking function",
keyspace, table, name, function));
}
// Some arguments of the masking function can be null, but the CQL's list type that stores them doesn't
// accept nulls, so we use a parallel list of booleans to store what arguments are null.
List<Boolean> nulls = maskRow.getFrozenList("function_argument_nulls", BooleanType.instance);
List<String> valuesAsCQL = maskRow.getFrozenList("function_argument_values", UTF8Type.instance);
ByteBuffer[] values = new ByteBuffer[valuesAsCQL.size()];
for (int i = 0; i < valuesAsCQL.size(); i++)
{
if (nulls.get(i))
values[i] = null;
else
values[i] = argumentTypes.get(i + 1).fromString(valuesAsCQL.get(i));
}
mask = ColumnMask.build((ScalarFunction) function, values);
}
return new ColumnMetadata(keyspace, table, name, type, position, kind, mask);
}
private static Map<ByteBuffer, DroppedColumn> fetchDroppedColumns(String keyspace, String table)
@ -1067,7 +1183,7 @@ public final class SchemaKeyspace
assert kind == ColumnMetadata.Kind.REGULAR || kind == ColumnMetadata.Kind.STATIC
: "Unexpected dropped column kind: " + kind;
ColumnMetadata column = new ColumnMetadata(keyspace, table, ColumnIdentifier.getInterned(name, true), type, ColumnMetadata.NO_POSITION, kind);
ColumnMetadata column = new ColumnMetadata(keyspace, table, ColumnIdentifier.getInterned(name, true), type, ColumnMetadata.NO_POSITION, kind, null);
long droppedTime = TimeUnit.MILLISECONDS.toMicros(row.getLong("dropped_time"));
return new DroppedColumn(column, droppedTime);
}
@ -1103,17 +1219,17 @@ public final class SchemaKeyspace
return new TriggerMetadata(name, classOption);
}
private static Views fetchViews(String keyspaceName, Types types)
private static Views fetchViews(String keyspaceName, Types types, UserFunctions functions)
{
String query = format("SELECT view_name FROM %s.%s WHERE keyspace_name = ?", SchemaConstants.SCHEMA_KEYSPACE_NAME, VIEWS);
Views.Builder views = org.apache.cassandra.schema.Views.builder();
for (UntypedResultSet.Row row : query(query, keyspaceName))
views.put(fetchView(keyspaceName, row.getString("view_name"), types));
views.put(fetchView(keyspaceName, row.getString("view_name"), types, functions));
return views.build();
}
private static ViewMetadata fetchView(String keyspaceName, String viewName, Types types)
private static ViewMetadata fetchView(String keyspaceName, String viewName, Types types, UserFunctions functions)
{
String query = String.format("SELECT * FROM %s.%s WHERE keyspace_name = ? AND view_name = ?", SchemaConstants.SCHEMA_KEYSPACE_NAME, VIEWS);
UntypedResultSet rows = query(query, keyspaceName, viewName);
@ -1126,7 +1242,7 @@ public final class SchemaKeyspace
boolean includeAll = row.getBoolean("include_all_columns");
String whereClauseString = row.getString("where_clause");
List<ColumnMetadata> columns = fetchColumns(keyspaceName, viewName, types);
List<ColumnMetadata> columns = fetchColumns(keyspaceName, viewName, types, functions);
TableMetadata metadata =
TableMetadata.builder(keyspaceName, viewName, TableId.fromUUID(row.getUUID("id")))

4
src/java/org/apache/cassandra/schema/SchemaKeyspaceTables.java
View File

@ -24,6 +24,7 @@ public class SchemaKeyspaceTables
public static final String KEYSPACES = "keyspaces";
public static final String TABLES = "tables";
public static final String COLUMNS = "columns";
public static final String COLUMN_MASKS = "column_masks";
public static final String DROPPED_COLUMNS = "dropped_columns";
public static final String TRIGGERS = "triggers";
public static final String VIEWS = "views";
@ -45,7 +46,8 @@ public class SchemaKeyspaceTables
*
* See CASSANDRA-12213 for more details.
*/
public static final ImmutableList<String> ALL = ImmutableList.of(COLUMNS,
public static final ImmutableList<String> ALL = ImmutableList.of(COLUMN_MASKS,
COLUMNS,
DROPPED_COLUMNS,
TRIGGERS,
TYPES,

127
src/java/org/apache/cassandra/schema/TableMetadata.java
View File

@ -47,6 +47,8 @@ import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.cql3.ColumnIdentifier;
import org.apache.cassandra.cql3.CqlBuilder;
import org.apache.cassandra.cql3.SchemaElement;
import org.apache.cassandra.cql3.functions.Function;
import org.apache.cassandra.cql3.functions.masking.ColumnMask;
import org.apache.cassandra.db.Clustering;
import org.apache.cassandra.db.ClusteringComparator;
import org.apache.cassandra.db.Columns;
@ -442,6 +444,35 @@ public class TableMetadata implements SchemaElement
return !staticColumns().isEmpty();
}
/**
* @return {@code true} if the table has any masked column, {@code false} otherwise.
*/
public boolean hasMaskedColumns()
{
for (ColumnMetadata column : columns.values())
{
if (column.isMasked())
return true;
}
return false;
}
/**
* @param function a user function
* @return {@code true} if the table has any masked column depending on the specified user function,
* {@code false} otherwise.
*/
public boolean dependsOn(Function function)
{
for (ColumnMetadata column : columns.values())
{
ColumnMask mask = column.getMask();
if (mask != null && mask.function.name().equals(function.name()))
return true;
}
return false;
}
public void validate()
{
if (!isNameValid(keyspace))
@ -931,44 +962,84 @@ public class TableMetadata implements SchemaElement
return this;
}
public Builder addPartitionKeyColumn(String name, AbstractType type)
public Builder addPartitionKeyColumn(String name, AbstractType<?> type)
{
return addPartitionKeyColumn(ColumnIdentifier.getInterned(name, false), type);
return addPartitionKeyColumn(name, type, null);
}
public Builder addPartitionKeyColumn(ColumnIdentifier name, AbstractType type)
public Builder addPartitionKeyColumn(String name, AbstractType<?> type, @Nullable ColumnMask mask)
{
return addColumn(new ColumnMetadata(keyspace, this.name, name, type, partitionKeyColumns.size(), ColumnMetadata.Kind.PARTITION_KEY));
return addPartitionKeyColumn(ColumnIdentifier.getInterned(name, false), type, mask);
}
public Builder addClusteringColumn(String name, AbstractType type)
public Builder addPartitionKeyColumn(ColumnIdentifier name, AbstractType<?> type)
{
return addClusteringColumn(ColumnIdentifier.getInterned(name, false), type);
return addPartitionKeyColumn(name, type, null);
}
public Builder addClusteringColumn(ColumnIdentifier name, AbstractType type)
public Builder addPartitionKeyColumn(ColumnIdentifier name, AbstractType<?> type, @Nullable ColumnMask mask)
{
return addColumn(new ColumnMetadata(keyspace, this.name, name, type, clusteringColumns.size(), ColumnMetadata.Kind.CLUSTERING));
return addColumn(new ColumnMetadata(keyspace, this.name, name, type, partitionKeyColumns.size(), ColumnMetadata.Kind.PARTITION_KEY, mask));
}
public Builder addRegularColumn(String name, AbstractType type)
public Builder addClusteringColumn(String name, AbstractType<?> type)
{
return addRegularColumn(ColumnIdentifier.getInterned(name, false), type);
return addClusteringColumn(name, type, null);
}
public Builder addRegularColumn(ColumnIdentifier name, AbstractType type)
public Builder addClusteringColumn(String name, AbstractType<?> type, @Nullable ColumnMask mask)
{
return addColumn(new ColumnMetadata(keyspace, this.name, name, type, ColumnMetadata.NO_POSITION, ColumnMetadata.Kind.REGULAR));
return addClusteringColumn(ColumnIdentifier.getInterned(name, false), type, mask);
}
public Builder addStaticColumn(String name, AbstractType type)
public Builder addClusteringColumn(ColumnIdentifier name, AbstractType<?> type)
{
return addStaticColumn(ColumnIdentifier.getInterned(name, false), type);
return addClusteringColumn(name, type, null);
}
public Builder addStaticColumn(ColumnIdentifier name, AbstractType type)
public Builder addClusteringColumn(ColumnIdentifier name, AbstractType<?> type, @Nullable ColumnMask mask)
{
return addColumn(new ColumnMetadata(keyspace, this.name, name, type, ColumnMetadata.NO_POSITION, ColumnMetadata.Kind.STATIC));
return addColumn(new ColumnMetadata(keyspace, this.name, name, type, clusteringColumns.size(), ColumnMetadata.Kind.CLUSTERING, mask));
}
public Builder addRegularColumn(String name, AbstractType<?> type)
{
return addRegularColumn(name, type, null);
}
public Builder addRegularColumn(String name, AbstractType<?> type, @Nullable ColumnMask mask)
{
return addRegularColumn(ColumnIdentifier.getInterned(name, false), type, mask);
}
public Builder addRegularColumn(ColumnIdentifier name, AbstractType<?> type)
{
return addRegularColumn(name, type, null);
}
public Builder addRegularColumn(ColumnIdentifier name, AbstractType<?> type, @Nullable ColumnMask mask)
{
return addColumn(new ColumnMetadata(keyspace, this.name, name, type, ColumnMetadata.NO_POSITION, ColumnMetadata.Kind.REGULAR, mask));
}
public Builder addStaticColumn(String name, AbstractType<?> type)
{
return addStaticColumn(name, type, null);
}
public Builder addStaticColumn(String name, AbstractType<?> type, @Nullable ColumnMask mask)
{
return addStaticColumn(ColumnIdentifier.getInterned(name, false), type, mask);
}
public Builder addStaticColumn(ColumnIdentifier name, AbstractType<?> type)
{
return addStaticColumn(name, type, null);
}
public Builder addStaticColumn(ColumnIdentifier name, AbstractType<?> type, @Nullable ColumnMask mask)
{
return addColumn(new ColumnMetadata(keyspace, this.name, name, type, ColumnMetadata.NO_POSITION, ColumnMetadata.Kind.STATIC, mask));
}
public Builder addColumn(ColumnMetadata column)
@ -1093,6 +1164,19 @@ public class TableMetadata implements SchemaElement
return this;
}
public Builder alterColumnMask(ColumnIdentifier name, @Nullable ColumnMask mask)
{
ColumnMetadata column = columns.get(name.bytes);
if (column == null)
throw new IllegalArgumentException();
ColumnMetadata newColumn = column.withNewMask(mask);
updateColumn(column, newColumn);
return this;
}
Builder alterColumnType(ColumnIdentifier name, AbstractType<?> type)
{
ColumnMetadata column = columns.get(name.bytes);
@ -1101,6 +1185,13 @@ public class TableMetadata implements SchemaElement
ColumnMetadata newColumn = column.withNewType(type);
updateColumn(column, newColumn);
return this;
}
private void updateColumn(ColumnMetadata column, ColumnMetadata newColumn)
{
switch (column.kind)
{
case PARTITION_KEY:
@ -1117,8 +1208,6 @@ public class TableMetadata implements SchemaElement
}
columns.put(column.name.bytes, newColumn);
return this;
}
}
@ -1543,7 +1632,7 @@ public class TableMetadata implements SchemaElement
for (ColumnMetadata c : regularAndStaticColumns)
{
if (c.isStatic())
columns.add(new ColumnMetadata(c.ksName, c.cfName, c.name, c.type, -1, ColumnMetadata.Kind.REGULAR));
columns.add(new ColumnMetadata(c.ksName, c.cfName, c.name, c.type, -1, ColumnMetadata.Kind.REGULAR, c.getMask()));
}
otherColumns = columns.iterator();
}

12
src/java/org/apache/cassandra/schema/ViewMetadata.java
View File

@ -20,10 +20,13 @@ package org.apache.cassandra.schema;
import java.nio.ByteBuffer;
import java.util.Optional;
import javax.annotation.Nullable;
import org.apache.commons.lang3.builder.HashCodeBuilder;
import org.apache.commons.lang3.builder.ToStringBuilder;
import org.apache.cassandra.cql3.*;
import org.apache.cassandra.cql3.functions.masking.ColumnMask;
import org.apache.cassandra.db.marshal.UserType;
public final class ViewMetadata implements SchemaElement
@ -158,6 +161,15 @@ public final class ViewMetadata implements SchemaElement
metadata.unbuild().addColumn(column).build());
}
public ViewMetadata withNewColumnMask(ColumnIdentifier name, @Nullable ColumnMask mask)
{
return new ViewMetadata(baseTableId,
baseTableName,
includeAllColumns,
whereClause,
metadata.unbuild().alterColumnMask(name, mask).build());
}
public void appendCqlTo(CqlBuilder builder,
boolean internals,
boolean ifNotExists)

21
src/java/org/apache/cassandra/service/ClientState.java
View File

@ -414,6 +414,27 @@ public class ClientState
ensurePermission(table.keyspace, perm, table.resource);
}
public boolean hasTablePermission(TableMetadata table, Permission perm)
{
if (isInternal)
return true;
validateLogin();
if (!DatabaseDescriptor.getAuthorizer().requireAuthorization())
return true;
List<? extends IResource> resources = Resources.chain(table.resource);
if (DatabaseDescriptor.getAuthFromRoot())
resources = Lists.reverse(resources);
for (IResource r : resources)
if (authorize(r).contains(perm))
return true;
return false;
}
private void ensurePermission(String keyspace, Permission perm, DataResource resource)
{
validateKeyspace(keyspace);

4
src/java/org/apache/cassandra/utils/NativeSSTableLoaderClient.java
View File

@ -212,7 +212,7 @@ public class NativeSSTableLoaderClient extends SSTableLoader.Client
int position = row.getInt("position");
org.apache.cassandra.schema.ColumnMetadata.Kind kind = ColumnMetadata.Kind.valueOf(row.getString("kind").toUpperCase());
return new ColumnMetadata(keyspace, table, name, type, position, kind);
return new ColumnMetadata(keyspace, table, name, type, position, kind, null);
}
private static DroppedColumn createDroppedColumnFromRow(Row row, String keyspace, String table)
@ -220,7 +220,7 @@ public class NativeSSTableLoaderClient extends SSTableLoader.Client
String name = row.getString("column_name");
AbstractType<?> type = CQLTypeParser.parse(keyspace, row.getString("type"), Types.none());
ColumnMetadata.Kind kind = ColumnMetadata.Kind.valueOf(row.getString("kind").toUpperCase());
ColumnMetadata column = new ColumnMetadata(keyspace, table, ColumnIdentifier.getInterned(name, true), type, ColumnMetadata.NO_POSITION, kind);
ColumnMetadata column = new ColumnMetadata(keyspace, table, ColumnIdentifier.getInterned(name, true), type, ColumnMetadata.NO_POSITION, kind, null);
long droppedTime = row.getTimestamp("dropped_time").getTime();
return new DroppedColumn(column, droppedTime);
}

200
test/distributed/org/apache/cassandra/distributed/test/ColumnMaskTest.java Normal file
View File

@ -0,0 +1,200 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.distributed.test;
import java.io.IOException;
import java.net.InetAddress;
import java.util.function.Consumer;
import org.junit.Test;
import com.datastax.driver.core.ConsistencyLevel;
import com.datastax.driver.core.ResultSet;
import com.datastax.driver.core.Session;
import com.datastax.driver.core.SimpleStatement;
import com.datastax.driver.core.Statement;
import com.datastax.driver.core.policies.LoadBalancingPolicy;
import org.apache.cassandra.auth.CassandraRoleManager;
import org.apache.cassandra.distributed.Cluster;
import org.apache.cassandra.distributed.api.IInvokableInstance;
import org.apache.cassandra.distributed.impl.RowUtil;
import org.apache.cassandra.distributed.util.SingleHostLoadBalancingPolicy;
import static com.datastax.driver.core.Cluster.Builder;
import static java.lang.String.format;
import static java.util.concurrent.TimeUnit.MINUTES;
import static java.util.concurrent.TimeUnit.SECONDS;
import static org.apache.cassandra.auth.CassandraRoleManager.DEFAULT_SUPERUSER_NAME;
import static org.apache.cassandra.auth.CassandraRoleManager.DEFAULT_SUPERUSER_PASSWORD;
import static org.apache.cassandra.distributed.api.Feature.GOSSIP;
import static org.apache.cassandra.distributed.api.Feature.NATIVE_PROTOCOL;
import static org.apache.cassandra.distributed.shared.AssertUtils.assertRows;
import static org.apache.cassandra.distributed.shared.AssertUtils.row;
import static org.awaitility.Awaitility.await;
/**
* Tests for dynamic data masking.
*/
public class ColumnMaskTest extends TestBaseImpl
{
private static final String SELECT = withKeyspace("SELECT * FROM %s.t");
private static final String USERNAME = "ddm_user";
private static final String PASSWORD = "ddm_password";
/**
* Tests that column masks are propagated to all nodes in the cluster.
*/
@Test
public void testMaskPropagation() throws Throwable
{
try (Cluster cluster = createClusterWithAuhentication(3))
{
cluster.schemaChange(withKeyspace("CREATE TABLE %s.t (k int PRIMARY KEY, v text MASKED WITH DEFAULT) WITH read_repair='NONE'"));
cluster.get(1).executeInternal(withKeyspace("INSERT INTO %s.t(k, v) VALUES (1, 'secret1')"));
cluster.get(2).executeInternal(withKeyspace("INSERT INTO %s.t(k, v) VALUES (2, 'secret2')"));
cluster.get(3).executeInternal(withKeyspace("INSERT INTO %s.t(k, v) VALUES (3, 'secret3')"));
assertRowsInAllCoordinators(cluster, row(1, "****"), row(2, "****"), row(3, "****"));
cluster.schemaChange(withKeyspace("ALTER TABLE %s.t ALTER v DROP MASKED"));
assertRowsInAllCoordinators(cluster, row(1, "secret1"), row(2, "secret2"), row(3, "secret3"));
cluster.schemaChange(withKeyspace("ALTER TABLE %s.t ALTER v MASKED WITH mask_inner(null, 1)"));
assertRowsInAllCoordinators(cluster, row(1, "******1"), row(2, "******2"), row(3, "******3"));
cluster.schemaChange(withKeyspace("ALTER TABLE %s.t ALTER v MASKED WITH mask_inner(3, null)"));
assertRowsInAllCoordinators(cluster, row(1, "sec****"), row(2, "sec****"), row(3, "sec****"));
}
}
/**
* Tests that column masks are properly loaded at startup.
*/
@Test
public void testMaskLoading() throws Throwable
{
try (Cluster cluster = createClusterWithAuhentication(1))
{
IInvokableInstance node = cluster.get(1);
cluster.schemaChange(withKeyspace("CREATE FUNCTION %s.custom_mask(column text, replacement text) " +
"RETURNS NULL ON NULL INPUT " +
"RETURNS text " +
"LANGUAGE java " +
"AS 'return replacement;'"));
cluster.schemaChange(withKeyspace("CREATE TABLE %s.t (" +
"a text MASKED WITH DEFAULT, " +
"b text MASKED WITH mask_replace('redacted'), " +
"c text MASKED WITH mask_inner(null, 1), " +
"d text MASKED WITH mask_inner(3, null), " +
"e text MASKED WITH %<s.custom_mask('obscured'), " +
"PRIMARY KEY (a, b))"));
String insert = withKeyspace("INSERT INTO %s.t(a, b, c, d, e) VALUES (?, ?, ?, ?, ?)");
node.executeInternal(insert, "secret1", "secret1", "secret1", "secret1", "secret1");
node.executeInternal(insert, "secret2", "secret2", "secret2", "secret2", "secret2");
assertRowsWithRestart(node,
row("****", "redacted", "******1", "sec****", "obscured"),
row("****", "redacted", "******2", "sec****", "obscured"));
cluster.schemaChange(withKeyspace("ALTER TABLE %s.t ALTER a DROP MASKED"));
cluster.schemaChange(withKeyspace("ALTER TABLE %s.t ALTER b MASKED WITH mask_null()"));
cluster.schemaChange(withKeyspace("ALTER TABLE %s.t ALTER c MASKED WITH mask_inner(null, null, '#')"));
cluster.schemaChange(withKeyspace("ALTER TABLE %s.t ALTER d MASKED WITH mask_inner(3, 1, '#')"));
cluster.schemaChange(withKeyspace("ALTER TABLE %s.t ALTER e MASKED WITH %<s.custom_mask('censored')"));
assertRowsWithRestart(node,
row("secret1", null, "#######", "sec###1", "censored"),
row("secret2", null, "#######", "sec###2", "censored"));
}
}
private static Cluster createClusterWithAuhentication(int nodeCount) throws IOException
{
Cluster cluster = init(Cluster.build()
.withNodes(nodeCount)
.withConfig(conf -> conf.with(GOSSIP, NATIVE_PROTOCOL)
.set("dynamic_data_masking_enabled", "true")
.set("user_defined_functions_enabled", "true")
.set("authenticator", "PasswordAuthenticator")
.set("authorizer", "CassandraAuthorizer"))
.start());
// create a user without UNMASK permission
withAuthenticatedSession(cluster.get(1), DEFAULT_SUPERUSER_NAME, DEFAULT_SUPERUSER_PASSWORD, session -> {
session.execute(format("CREATE USER IF NOT EXISTS %s WITH PASSWORD '%s'", USERNAME, PASSWORD));
session.execute(format("GRANT ALL ON KEYSPACE %s TO %s", KEYSPACE, USERNAME));
session.execute(format("REVOKE UNMASK ON KEYSPACE %s FROM %s", KEYSPACE, USERNAME));
});
return cluster;
}
private static void assertRowsInAllCoordinators(Cluster cluster, Object[]... expectedRows)
{
for (int i = 1; i < cluster.size(); i++)
{
assertRowsWithAuthentication(cluster.get(i), expectedRows);
}
}
private static void assertRowsWithRestart(IInvokableInstance node, Object[]... expectedRows) throws Throwable
{
// test querying with in-memory column definitions
assertRowsWithAuthentication(node, expectedRows);
// restart the nodes to reload the column definitions from disk
node.shutdown().get();
node.startup();
// test querying with the column definitions loaded from disk
assertRowsWithAuthentication(node, expectedRows);
}
private static void assertRowsWithAuthentication(IInvokableInstance node, Object[]... expectedRows)
{
withAuthenticatedSession(node, USERNAME, PASSWORD, session -> {
Statement statement = new SimpleStatement(SELECT).setConsistencyLevel(ConsistencyLevel.ALL);
ResultSet resultSet = session.execute(statement);
assertRows(RowUtil.toObjects(resultSet), expectedRows);
});
}
private static void withAuthenticatedSession(IInvokableInstance instance, String username, String password, Consumer<Session> consumer)
{
// wait for existing roles
await().pollDelay(1, SECONDS)
.pollInterval(1, SECONDS)
.atMost(1, MINUTES)
.until(() -> instance.callOnInstance(CassandraRoleManager::hasExistingRoles));
// use a load balancing policy that ensures that we actually connect to the desired node
InetAddress address = instance.broadcastAddress().getAddress();
LoadBalancingPolicy lbc = new SingleHostLoadBalancingPolicy(address);
Builder builder = com.datastax.driver.core.Cluster.builder()
.addContactPoints(address)
.withLoadBalancingPolicy(lbc)
.withCredentials(username, password);
try (com.datastax.driver.core.Cluster cluster = builder.build();
Session session = cluster.connect())
{
consumer.accept(session);
}
}
}

100
test/distributed/org/apache/cassandra/distributed/util/SingleHostLoadBalancingPolicy.java Normal file
View File

@ -0,0 +1,100 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.distributed.util;
import java.net.InetAddress;
import java.util.Collection;
import java.util.Iterator;
import java.util.List;
import java.util.concurrent.CopyOnWriteArrayList;
import com.google.common.collect.Iterators;
import com.datastax.driver.core.Cluster;
import com.datastax.driver.core.Host;
import com.datastax.driver.core.HostDistance;
import com.datastax.driver.core.Statement;
import com.datastax.driver.core.policies.LoadBalancingPolicy;
/**
* A CQL driver {@link LoadBalancingPolicy} that only connects to a fixed host.
*/
public class SingleHostLoadBalancingPolicy implements LoadBalancingPolicy
{
private final InetAddress address;
private Host host;
public SingleHostLoadBalancingPolicy(InetAddress address)
{
this.address = address;
}
protected final List<Host> hosts = new CopyOnWriteArrayList<>();
@Override
public void init(Cluster cluster, Collection<Host> hosts)
{
host = hosts.stream()
.filter(h -> h.getBroadcastAddress().equals(address)).findFirst()
.orElseThrow(() -> new AssertionError("The host should be a contact point"));
this.hosts.add(host);
}
@Override
public HostDistance distance(Host host)
{
return HostDistance.LOCAL;
}
@Override
public Iterator<Host> newQueryPlan(String loggedKeyspace, Statement statement)
{
return Iterators.singletonIterator(host);
}
@Override
public void onAdd(Host host)
{
// no-op
}
@Override
public void onUp(Host host)
{
// no-op
}
@Override
public void onDown(Host host)
{
// no-op
}
@Override
public void onRemove(Host host)
{
// no-op
}
@Override
public void close()
{
// no-op
}
}

8
test/microbench/org/apache/cassandra/test/microbench/btree/AtomicBTreePartitionUpdateBench.java
View File

@ -561,7 +561,13 @@ public class AtomicBTreePartitionUpdateBench
private static ColumnMetadata[] columns(AbstractType<?> type, ColumnMetadata.Kind kind, int count, String prefix)
{
return IntStream.range(0, count)
.mapToObj(i -> new ColumnMetadata("", "", new ColumnIdentifier(prefix + i, true), type, kind != ColumnMetadata.Kind.REGULAR ? i : ColumnMetadata.NO_POSITION, kind))
.mapToObj(i -> new ColumnMetadata("",
"",
new ColumnIdentifier(prefix + i, true),
type,
kind != ColumnMetadata.Kind.REGULAR ? i : ColumnMetadata.NO_POSITION,
kind,
null))
.toArray(ColumnMetadata[]::new);
}

6
test/unit/org/apache/cassandra/SchemaLoader.java
View File

@ -299,7 +299,8 @@ public class SchemaLoader
ColumnIdentifier.getInterned(IntegerType.instance.fromString("42"), IntegerType.instance),
UTF8Type.instance,
ColumnMetadata.NO_POSITION,
ColumnMetadata.Kind.REGULAR);
ColumnMetadata.Kind.REGULAR,
null);
}
public static ColumnMetadata utf8Column(String ksName, String cfName)
@ -309,7 +310,8 @@ public class SchemaLoader
ColumnIdentifier.getInterned("fortytwo", true),
UTF8Type.instance,
ColumnMetadata.NO_POSITION,
ColumnMetadata.Kind.REGULAR);
ColumnMetadata.Kind.REGULAR,
null);
}
public static TableMetadata perRowIndexedCFMD(String ksName, String cfName)

46
test/unit/org/apache/cassandra/auth/CreateAndAlterRoleTest.java
View File

@ -21,11 +21,9 @@ package org.apache.cassandra.auth;
import org.junit.BeforeClass;
import org.junit.Test;
import com.datastax.driver.core.exceptions.AuthenticationException;
import org.apache.cassandra.Util;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.cql3.CQLTester;
import static org.junit.Assert.assertTrue;
import static org.mindrot.jbcrypt.BCrypt.gensalt;
import static org.mindrot.jbcrypt.BCrypt.hashpw;
@ -34,6 +32,10 @@ public class CreateAndAlterRoleTest extends CQLTester
@BeforeClass
public static void setUpClass()
{
DatabaseDescriptor.setPermissionsValidity(0);
DatabaseDescriptor.setRolesValidity(0);
DatabaseDescriptor.setCredentialsValidity(0);
CQLTester.setUpClass();
requireAuthentication();
requireNetwork();
@ -62,11 +64,11 @@ public class CreateAndAlterRoleTest extends CQLTester
useUser(user1, plainTextPwd);
executeNetWithAuthSpin("SELECT key FROM system.local");
executeNet("SELECT key FROM system.local");
useUser(user2, plainTextPwd);
executeNetWithAuthSpin("SELECT key FROM system.local");
executeNet("SELECT key FROM system.local");
useSuperUser();
@ -78,11 +80,11 @@ public class CreateAndAlterRoleTest extends CQLTester
useUser(user1, plainTextPwd2);
executeNetWithAuthSpin("SELECT key FROM system.local");
executeNet("SELECT key FROM system.local");
useUser(user2, plainTextPwd2);
executeNetWithAuthSpin("SELECT key FROM system.local");
executeNet("SELECT key FROM system.local");
}
@Test
@ -105,11 +107,11 @@ public class CreateAndAlterRoleTest extends CQLTester
useUser(user1, plainTextPwd);
executeNetWithAuthSpin("SELECT key FROM system.local");
executeNet("SELECT key FROM system.local");
useUser(user2, plainTextPwd);
executeNetWithAuthSpin("SELECT key FROM system.local");
executeNet("SELECT key FROM system.local");
useSuperUser();
@ -118,32 +120,10 @@ public class CreateAndAlterRoleTest extends CQLTester
useUser(user1, plainTextPwd2);
executeNetWithAuthSpin("SELECT key FROM system.local");
executeNet("SELECT key FROM system.local");
useUser(user2, plainTextPwd2);
executeNetWithAuthSpin("SELECT key FROM system.local");
}
/**
* Altering or creating auth may take some time to be effective
*
* @param query
*/
void executeNetWithAuthSpin(String query)
{
Util.spinAssertEquals(true, () -> {
try
{
executeNet(query);
return true;
}
catch (Throwable e)
{
assertTrue("Unexpected exception: " + e, e instanceof AuthenticationException);
reinitializeNetwork();
return false;
}
}, 10);
executeNet("SELECT key FROM system.local");
}
}

28
test/unit/org/apache/cassandra/cql3/CQLTester.java
View File

@ -1276,6 +1276,11 @@ public abstract class CQLTester
return sessionNet().execute(new SimpleStatement(formatQuery(query)).setFetchSize(pageSize));
}
protected com.datastax.driver.core.ResultSet executeNetWithoutPaging(String query)
{
return executeNetWithPaging(query, Integer.MAX_VALUE);
}
protected Session sessionNet()
{
return sessionNet(getDefaultVersion());
@ -1721,6 +1726,24 @@ public abstract class CQLTester
}
}
protected void assertColumnNames(ResultSet result, String... expectedColumnNames)
{
if (result == null)
{
Assert.fail("No rows returned by query.");
return;
}
ColumnDefinitions columnDefinitions = result.getColumnDefinitions();
Assert.assertEquals("Got less columns than expected.", expectedColumnNames.length, columnDefinitions.size());
for (int i = 0, m = columnDefinitions.size(); i < m; i++)
{
String columnName = columnDefinitions.getName(i);
Assert.assertEquals(expectedColumnNames[i], columnName);
}
}
protected void assertAllRows(Object[]... rows) throws Throwable
{
assertRows(execute("SELECT * FROM %s"), rows);
@ -1731,6 +1754,11 @@ public abstract class CQLTester
return expected;
}
public static Object[][] rows(Object[]... rows)
{
return rows;
}
protected void assertEmpty(UntypedResultSet result) throws Throwable
{
if (result != null && !result.isEmpty())

267
test/unit/org/apache/cassandra/cql3/functions/masking/ColumnMaskInAnyPositionTester.java Normal file
View File

@ -0,0 +1,267 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.cql3.functions.masking;
import java.nio.ByteBuffer;
import java.util.List;
import org.apache.commons.lang3.StringUtils;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.apache.cassandra.db.marshal.AbstractType;
import static java.lang.String.format;
/**
* {@link ColumnMaskTester} verifying that masks can be applied to columns in any position (partition key columns,
* clustering key columns, static columns and regular columns). The columns of any depending materialized views should
* be udpated accordingly.
*/
@RunWith(Parameterized.class)
public abstract class ColumnMaskInAnyPositionTester extends ColumnMaskTester
{
/** The column mask as expressed in CQL statements right after the {@code MASKED WITH} keywords. */
@Parameterized.Parameter
public String mask;
/** The type of the masked column */
@Parameterized.Parameter(1)
public String type;
/** The types of the tested masking function partial arguments. */
@Parameterized.Parameter(2)
public List<AbstractType<?>> argumentTypes;
/** The serialized values of the tested masking function partial arguments. */
@Parameterized.Parameter(3)
public List<ByteBuffer> argumentValues;
@Test
public void testCreateTableWithMaskedColumns() throws Throwable
{
// Nothing is masked
createTable("CREATE TABLE %s (k int, c int, r int, s int static, PRIMARY KEY(k, c))");
assertTableColumnsAreNotMasked("k", "c", "r", "s");
// Masked partition key
createTable(format("CREATE TABLE %%s (k %s MASKED WITH %s PRIMARY KEY, r int)", type, mask));
assertTableColumnsAreMasked("k");
assertTableColumnsAreNotMasked("r");
// Masked partition key component
createTable(format("CREATE TABLE %%s (k1 int, k2 %s MASKED WITH %s, r int, PRIMARY KEY(k1, k2))", type, mask));
assertTableColumnsAreMasked("k2");
assertTableColumnsAreNotMasked("k1", "r");
// Masked clustering key
createTable(format("CREATE TABLE %%s (k int, c %s MASKED WITH %s, r int, PRIMARY KEY (k, c))", type, mask));
assertTableColumnsAreMasked("c");
assertTableColumnsAreNotMasked("k", "r");
// Masked clustering key with reverse order
createTable(format("CREATE TABLE %%s (k int, c %s MASKED WITH %s, r int, PRIMARY KEY (k, c)) " +
"WITH CLUSTERING ORDER BY (c DESC)", type, mask));
assertTableColumnsAreMasked("c");
assertTableColumnsAreNotMasked("k", "r");
// Masked clustering key component
createTable(format("CREATE TABLE %%s (k int, c1 int, c2 %s MASKED WITH %s, r int, PRIMARY KEY (k, c1, c2))", type, mask));
assertTableColumnsAreMasked("c2");
assertTableColumnsAreNotMasked("k", "c1", "r");
// Masked regular column
createTable(format("CREATE TABLE %%s (k int PRIMARY KEY, r1 %s MASKED WITH %s, r2 int)", type, mask));
assertTableColumnsAreMasked("r1");
assertTableColumnsAreNotMasked("k", "r2");
// Masked static column
createTable(format("CREATE TABLE %%s (k int, c int, r int, s %s STATIC MASKED WITH %s, PRIMARY KEY (k, c))", type, mask));
assertTableColumnsAreMasked("s");
assertTableColumnsAreNotMasked("k", "c", "r");
// Multiple masked columns
createTable(format("CREATE TABLE %%s (" +
"k1 int, k2 %s MASKED WITH %s, " +
"c1 int, c2 %s MASKED WITH %s, " +
"r1 int, r2 %s MASKED WITH %s, " +
"s1 int static, s2 %s static MASKED WITH %s, " +
"PRIMARY KEY((k1, k2), c1, c2))",
type, mask, type, mask, type, mask, type, mask));
assertTableColumnsAreMasked("k2", "c2", "r2", "s2");
assertTableColumnsAreNotMasked("k1", "c1", "r1", "s1");
}
@Test
public void testCreateTableWithMaskedColumnsAndMaterializedView() throws Throwable
{
createTable(format("CREATE TABLE %%s (" +
"k1 int, k2 %s MASKED WITH %s, " +
"c1 int, c2 %s MASKED WITH %s, " +
"r1 int, r2 %s MASKED WITH %s, " +
"s1 int static, s2 %s static MASKED WITH %s, " +
"PRIMARY KEY((k1, k2), c1, c2))",
type, mask, type, mask, type, mask, type, mask));
createView("CREATE MATERIALIZED VIEW %s AS SELECT k1, k2, c1, c2, r1, r2 FROM %s " +
"WHERE k1 IS NOT NULL AND k2 IS NOT NULL " +
"AND c1 IS NOT NULL AND c2 IS NOT NULL " +
"AND r1 IS NOT NULL AND r2 IS NOT NULL " +
"PRIMARY KEY (r2, c2, c1, k2, k1)");
assertTableColumnsAreMasked("k2", "c2", "r2", "s2");
assertTableColumnsAreNotMasked("k1", "c1", "r1", "s1");
assertViewColumnsAreMasked("k2", "c2", "r2");
assertViewColumnsAreNotMasked("k1", "c1", "r1");
}
@Test
public void testAlterTableWithMaskedColumns() throws Throwable
{
// Create the table to be altered
createTable(format("CREATE TABLE %%s (k %s, c %<s, r1 %<s, r2 %<s MASKED WITH %s, r3 %s, s %<s static, " +
"PRIMARY KEY (k, c))", type, mask, type));
assertTableColumnsAreMasked("r2");
assertTableColumnsAreNotMasked("k", "c", "r1", "r3", "s");
// Add new masked column
alterTable(format("ALTER TABLE %%s ADD r4 %s MASKED WITH %s", type, mask));
assertTableColumnsAreMasked("r2", "r4");
assertTableColumnsAreNotMasked("k", "c", "r1", "r3", "s");
// Set mask for an existing but unmasked column
alterTable(format("ALTER TABLE %%s ALTER r1 MASKED WITH %s", mask));
assertTableColumnsAreMasked("r1", "r2", "r4");
// Unmask a masked column
alterTable("ALTER TABLE %s ALTER r1 DROP MASKED");
alterTable("ALTER TABLE %s ALTER r2 DROP MASKED");
assertTableColumnsAreMasked("r4");
assertTableColumnsAreNotMasked("r1", "r2", "r3");
// Mask and disable mask for primary key
alterTable(format("ALTER TABLE %%s ALTER k MASKED WITH %s", mask));
assertTableColumnsAreMasked("k");
alterTable("ALTER TABLE %s ALTER k DROP MASKED");
assertTableColumnsAreNotMasked("k");
// Mask and disable mask for clustering key
alterTable(format("ALTER TABLE %%s ALTER c MASKED WITH %s", mask));
assertTableColumnsAreMasked("c");
alterTable("ALTER TABLE %s ALTER c DROP MASKED");
assertTableColumnsAreNotMasked("c");
// Mask and disable mask for static column
alterTable(format("ALTER TABLE %%s ALTER s MASKED WITH %s", mask));
assertTableColumnsAreMasked("s");
alterTable("ALTER TABLE %s ALTER s DROP MASKED");
assertTableColumnsAreNotMasked("s");
// Add multiple masked columns within the same query
alterTable(format("ALTER TABLE %%s ADD (" +
"r5 %s MASKED WITH %s, " +
"r6 %s, " +
"r7 %s MASKED WITH %s, " +
"r8 %s)",
type, mask, type, type, mask, type));
assertTableColumnsAreMasked("r5", "r7");
assertTableColumnsAreNotMasked("r6", "r8");
}
@Test
public void testAlterTableWithMaskedColumnsAndMaterializedView() throws Throwable
{
createTable(format("CREATE TABLE %%s (" +
"k %s, c %<s, r1 %<s, r2 %<s, s1 %<s static, s2 %<s static, " +
"PRIMARY KEY(k, c))", type));
createView("CREATE MATERIALIZED VIEW %s AS SELECT k, c, r2 FROM %s " +
"WHERE k IS NOT NULL AND c IS NOT NULL " +
"AND r1 IS NOT NULL AND r2 IS NOT NULL " +
"PRIMARY KEY (r2, c, k)");
// Adding a column to the table doesn't have an effect on the view
alterTable(format("ALTER TABLE %%s ADD r3 %s MASKED WITH %s", type, mask));
assertTableColumnsAreMasked("r3");
assertTableColumnsAreNotMasked("k", "c", "r1", "r2", "s1", "s2");
assertViewColumnsAreNotMasked("k", "c", "r2");
// Masking a column that is not part of the view doesn't have an effect on the view
alterTable(format("ALTER TABLE %%s ALTER r1 MASKED WITH %s", mask));
alterTable(format("ALTER TABLE %%s ALTER s1 MASKED WITH %s", mask));
assertTableColumnsAreMasked("r1", "r3", "s1");
assertTableColumnsAreNotMasked("k", "c", "r2", "s2");
assertViewColumnsAreNotMasked("k", "c", "r2");
// Masking a column that is part of the view should have an effect on the view
alterTable(format("ALTER TABLE %%s ALTER r2 MASKED WITH %s", mask));
assertTableColumnsAreMasked("r1", "r2", "r3", "s1");
assertTableColumnsAreNotMasked("k", "c", "s2");
assertViewColumnsAreMasked("r2");
assertViewColumnsAreNotMasked("k", "c");
// Mask the rest of the columns
alterTable(format("ALTER TABLE %%s ALTER k MASKED WITH %s", mask));
alterTable(format("ALTER TABLE %%s ALTER c MASKED WITH %s", mask));
alterTable(format("ALTER TABLE %%s ALTER s2 MASKED WITH %s", mask));
assertTableColumnsAreMasked("k", "c", "r1", "r2", "r3", "s1", "s2");
assertViewColumnsAreMasked("k", "c", "r2");
// Unmask a column that is part of the view
alterTable("ALTER TABLE %s ALTER r2 DROP MASKED");
assertTableColumnsAreMasked("k", "c", "r1", "r3", "s1", "s2");
assertTableColumnsAreNotMasked("r2");
assertViewColumnsAreMasked("k", "c");
assertViewColumnsAreNotMasked("r2");
// Unmask the rest of the columns
alterTable("ALTER TABLE %s ALTER k DROP MASKED");
alterTable("ALTER TABLE %s ALTER c DROP MASKED");
alterTable("ALTER TABLE %s ALTER r1 DROP MASKED");
alterTable("ALTER TABLE %s ALTER r3 DROP MASKED");
alterTable("ALTER TABLE %s ALTER s1 DROP MASKED");
alterTable("ALTER TABLE %s ALTER s2 DROP MASKED");
assertTableColumnsAreNotMasked("k", "c", "r1", "r2", "r3", "s1", "s2");
assertViewColumnsAreNotMasked("k", "c", "r2");
}
private String functionName()
{
if (mask.equals("DEFAULT"))
return "mask_default";
return StringUtils.remove(StringUtils.substringBefore(mask, "("), KEYSPACE + ".");
}
private void assertTableColumnsAreMasked(String... columns) throws Throwable
{
for (String column : columns)
{
assertColumnIsMasked(currentTable(), column, functionName(), argumentTypes, argumentValues);
}
}
private void assertViewColumnsAreMasked(String... columns) throws Throwable
{
for (String column : columns)
{
assertColumnIsMasked(currentView(), column, functionName(), argumentTypes, argumentValues);
}
}
}

45
test/unit/org/apache/cassandra/cql3/functions/masking/ColumnMaskInAnyPositionWithDefaultTest.java Normal file
View File

@ -0,0 +1,45 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.cql3.functions.masking;
import java.util.Arrays;
import java.util.Collection;
import org.junit.runners.Parameterized;
import static java.util.Collections.emptyList;
/**
* {@link ColumnMaskInAnyPositionTester} for {@link DefaultMaskingFunction}.
*/
public class ColumnMaskInAnyPositionWithDefaultTest extends ColumnMaskInAnyPositionTester
{
@Parameterized.Parameters(name = "mask={0}, type={1}")
public static Collection<Object[]> options()
{
return Arrays.asList(new Object[][]{
{ "DEFAULT", "int", emptyList(), emptyList() },
{ "DEFAULT", "text", emptyList(), emptyList() },
{ "DEFAULT", "frozen<list<uuid>>", emptyList(), emptyList() },
{ "mask_default()", "int", emptyList(), emptyList() },
{ "mask_default()", "text", emptyList(), emptyList() },
{ "mask_default()", "frozen<list<uuid>>", emptyList(), emptyList() },
});
}
}

42
test/unit/org/apache/cassandra/cql3/functions/masking/ColumnMaskInAnyPositionWithNullTest.java Normal file
View File

@ -0,0 +1,42 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.cql3.functions.masking;
import java.util.Arrays;
import java.util.Collection;
import org.junit.runners.Parameterized;
import static java.util.Collections.emptyList;
/**
* {@link ColumnMaskInAnyPositionTester} for {@link NullMaskingFunction}.
*/
public class ColumnMaskInAnyPositionWithNullTest extends ColumnMaskInAnyPositionTester
{
@Parameterized.Parameters(name = "mask={0}, type={1}")
public static Collection<Object[]> options()
{
return Arrays.asList(new Object[][]{
{ "mask_null()", "int", emptyList(), emptyList() },
{ "mask_null()", "text", emptyList(), emptyList() },
{ "mask_null()", "frozen<list<uuid>>", emptyList(), emptyList() },
});
}
}

54
test/unit/org/apache/cassandra/cql3/functions/masking/ColumnMaskInAnyPositionWithPartialTest.java Normal file
View File

@ -0,0 +1,54 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.cql3.functions.masking;
import java.util.Arrays;
import java.util.Collection;
import org.junit.runners.Parameterized;
import org.apache.cassandra.db.marshal.Int32Type;
import org.apache.cassandra.db.marshal.UTF8Type;
import static java.util.Arrays.asList;
/**
* {@link ColumnMaskInAnyPositionTester} for {@link PartialMaskingFunction}.
*/
public class ColumnMaskInAnyPositionWithPartialTest extends ColumnMaskInAnyPositionTester
{
@Parameterized.Parameters(name = "mask={0}, type={1}")
public static Collection<Object[]> options()
{
return Arrays.asList(new Object[][]{
{ "mask_inner(1, 2)", "text",
asList(Int32Type.instance, Int32Type.instance),
asList(Int32Type.instance.decompose(1), Int32Type.instance.decompose(2)) },
{ "mask_outer(1, 2)", "text",
asList(Int32Type.instance, Int32Type.instance),
asList(Int32Type.instance.decompose(1), Int32Type.instance.decompose(2)) },
{ "mask_inner(1, 2, '#')", "text",
asList(Int32Type.instance, Int32Type.instance, UTF8Type.instance),
asList(Int32Type.instance.decompose(1), Int32Type.instance.decompose(2), UTF8Type.instance.decompose("#")) },
{ "mask_outer(1, 2, '#')", "text",
asList(Int32Type.instance, Int32Type.instance, UTF8Type.instance),
asList(Int32Type.instance.decompose(1), Int32Type.instance.decompose(2), UTF8Type.instance.decompose("#")) }
});
}
}

53
test/unit/org/apache/cassandra/cql3/functions/masking/ColumnMaskInAnyPositionWithReplaceTest.java Normal file
View File

@ -0,0 +1,53 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.cql3.functions.masking;
import java.util.Arrays;
import java.util.Collection;
import org.junit.runners.Parameterized;
import org.apache.cassandra.db.marshal.Int32Type;
import org.apache.cassandra.db.marshal.ListType;
import org.apache.cassandra.db.marshal.UTF8Type;
import static java.util.Collections.emptyList;
import static java.util.Collections.singletonList;
/**
* {@link ColumnMaskInAnyPositionTester} for {@link ReplaceMaskingFunction}.
*/
public class ColumnMaskInAnyPositionWithReplaceTest extends ColumnMaskInAnyPositionTester
{
@Parameterized.Parameters(name = "mask={0}, type={1}")
public static Collection<Object[]> options()
{
return Arrays.asList(new Object[][]{
{ "mask_replace(0)", "int",
singletonList(Int32Type.instance),
singletonList(Int32Type.instance.decompose(0)) },
{ "mask_replace('redacted')", "text",
singletonList(UTF8Type.instance),
singletonList(UTF8Type.instance.decompose("redacted")) },
{ "mask_replace([])", "frozen<list<int>>",
singletonList(ListType.getInstance(Int32Type.instance, false)),
singletonList(ListType.getInstance(Int32Type.instance, false).decompose(emptyList())) },
});
}
}

71
test/unit/org/apache/cassandra/cql3/functions/masking/ColumnMaskInAnyPositionWithUDFTest.java Normal file
View File

@ -0,0 +1,71 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.cql3.functions.masking;
import java.util.Arrays;
import java.util.Collection;
import org.junit.Before;
import org.junit.runners.Parameterized;
import org.apache.cassandra.db.marshal.Int32Type;
import org.apache.cassandra.db.marshal.UTF8Type;
import static java.util.Collections.singletonList;
/**
* {@link ColumnMaskInAnyPositionTester} for user-defined functions.
*/
public class ColumnMaskInAnyPositionWithUDFTest extends ColumnMaskInAnyPositionTester
{
private static final String TEXT_REPLACE_FUNCTION = KEYSPACE + ".mask_text_replace";
private static final String INT_REPLACE_FUNCTION = KEYSPACE + ".mask_int_replace";
@Before
public void setupSchema() throws Throwable
{
createFunction(KEYSPACE,
"text, text",
"CREATE FUNCTION IF NOT EXISTS " + TEXT_REPLACE_FUNCTION + " (column text, replacement text) " +
"CALLED ON NULL INPUT " +
"RETURNS text " +
"LANGUAGE java " +
"AS 'return replacement;'");
createFunction(KEYSPACE,
"int, int",
"CREATE FUNCTION IF NOT EXISTS " + INT_REPLACE_FUNCTION + " (column int, replacement int) " +
"CALLED ON NULL INPUT " +
"RETURNS int " +
"LANGUAGE java " +
"AS 'return replacement;'");
}
@Parameterized.Parameters(name = "mask={0}, type={1}")
public static Collection<Object[]> options()
{
return Arrays.asList(new Object[][]{
{ TEXT_REPLACE_FUNCTION + "('redacted')", "text",
singletonList(UTF8Type.instance),
singletonList(UTF8Type.instance.decompose("redacted")) },
{ INT_REPLACE_FUNCTION + "(0)", "int",
singletonList(Int32Type.instance),
singletonList(Int32Type.instance.decompose(0)) }
});
}
}

89
test/unit/org/apache/cassandra/cql3/functions/masking/ColumnMaskNativeTypesTest.java Normal file
View File

@ -0,0 +1,89 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.cql3.functions.masking;
import java.util.ArrayList;
import java.util.Collection;
import java.util.List;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.apache.cassandra.cql3.CQL3Type;
import static java.lang.String.format;
import static java.util.Collections.emptyList;
/**
* {@link ColumnMaskTester} verifying that we can attach column masks to table columns with any native data type.
*/
@RunWith(Parameterized.class)
public class ColumnMaskNativeTypesTest extends ColumnMaskTester
{
/** The type of the column. */
@Parameterized.Parameter
public CQL3Type.Native type;
@Parameterized.Parameters(name = "type={0}")
public static Collection<Object[]> options()
{
List<Object[]> parameters = new ArrayList<>();
for (CQL3Type.Native type : CQL3Type.Native.values())
{
if (type != CQL3Type.Native.EMPTY)
parameters.add(new Object[]{ type });
}
return parameters;
}
@Test
public void testNativeDataTypes() throws Throwable
{
String def = format("%s MASKED WITH DEFAULT", type);
String keyDef = type == CQL3Type.Native.COUNTER || type == CQL3Type.Native.DURATION
? "int MASKED WITH DEFAULT" : def;
String staticDef = format("%s STATIC MASKED WITH DEFAULT", type);
// Create table with masks
String table = createTable(format("CREATE TABLE %%s (k %s, c %<s, r %s, s %s, PRIMARY KEY(k, c))", keyDef, def, staticDef));
assertColumnIsMasked(table, "k", "mask_default", emptyList(), emptyList());
assertColumnIsMasked(table, "c", "mask_default", emptyList(), emptyList());
assertColumnIsMasked(table, "r", "mask_default", emptyList(), emptyList());
assertColumnIsMasked(table, "s", "mask_default", emptyList(), emptyList());
// Alter column masks
alterTable("ALTER TABLE %s ALTER k MASKED WITH mask_null()");
alterTable("ALTER TABLE %s ALTER c MASKED WITH mask_null()");
alterTable("ALTER TABLE %s ALTER r MASKED WITH mask_null()");
alterTable("ALTER TABLE %s ALTER s MASKED WITH mask_null()");
assertColumnIsMasked(table, "k", "mask_null", emptyList(), emptyList());
assertColumnIsMasked(table, "c", "mask_null", emptyList(), emptyList());
assertColumnIsMasked(table, "r", "mask_null", emptyList(), emptyList());
assertColumnIsMasked(table, "s", "mask_null", emptyList(), emptyList());
// Drop masks
alterTable("ALTER TABLE %s ALTER k DROP MASKED");
alterTable("ALTER TABLE %s ALTER c DROP MASKED");
alterTable("ALTER TABLE %s ALTER r DROP MASKED");
alterTable("ALTER TABLE %s ALTER s DROP MASKED");
assertTableColumnsAreNotMasked("k", "c", "r", "s");
}
}

128
test/unit/org/apache/cassandra/cql3/functions/masking/ColumnMaskQueryTester.java Normal file
View File

@ -0,0 +1,128 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.cql3.functions.masking;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import com.datastax.driver.core.ResultSet;
import static java.lang.String.format;
/**
* Test queries on columns that have attached a dynamic data masking function.
*/
@RunWith(Parameterized.class)
public abstract class ColumnMaskQueryTester extends ColumnMaskTester
{
@Parameterized.Parameter
public String order;
@Parameterized.Parameter(1)
public String mask;
@Parameterized.Parameter(2)
public String columnType;
@Parameterized.Parameter(3)
public Object columnValue;
@Parameterized.Parameter(4)
public Object maskedValue;
@Before
public void setupSchema() throws Throwable
{
createTable("CREATE TABLE %s (" +
format("k1 %s, k2 %<s MASKED WITH %s, ", columnType, mask) +
format("c1 %s, c2 %<s MASKED WITH %s, ", columnType, mask) +
format("r1 %s, r2 %<s MASKED WITH %s, ", columnType, mask) +
format("s1 %s static, s2 %<s static MASKED WITH %s, ", columnType, mask) +
format("PRIMARY KEY((k1, k2), c1, c2)) WITH CLUSTERING ORDER BY (c1 %s, c2 %<s)", order));
createView("CREATE MATERIALIZED VIEW %s AS SELECT k2, k1, c2, c1, r2, r1 FROM %s " +
"WHERE k1 IS NOT NULL AND k2 IS NOT NULL " +
"AND c1 IS NOT NULL AND c2 IS NOT NULL " +
"AND r1 IS NOT NULL AND r2 IS NOT NULL " +
"PRIMARY KEY ((c2, c1), k2, k1)");
executeNet("INSERT INTO %s(k1, k2, c1, c2, r1, r2, s1, s2) VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
columnValue, columnValue, columnValue, columnValue, columnValue, columnValue, columnValue, columnValue);
}
@Test
public void testSelectWithWilcard() throws Throwable
{
ResultSet rs = executeNet("SELECT * FROM %s");
assertColumnNames(rs, "k1", "k2", "c1", "c2", "s1", "s2", "r1", "r2");
assertRowsNet(rs, row(columnValue, maskedValue,
columnValue, maskedValue,
columnValue, maskedValue,
columnValue, maskedValue));
rs = executeNet(format("SELECT * FROM %s.%s", KEYSPACE, currentView()));
assertColumnNames(rs, "c2", "c1", "k2", "k1", "r1", "r2");
assertRowsNet(rs, row(maskedValue, columnValue,
maskedValue, columnValue,
columnValue, maskedValue));
}
@Test
public void testSelectWithAllColumnNames() throws Throwable
{
ResultSet rs = executeNet("SELECT c2, c1, k2, k1, r2, r1, s2, s1 FROM %s");
assertColumnNames(rs, "c2", "c1", "k2", "k1", "r2", "r1", "s2", "s1");
assertRowsNet(rs, row(maskedValue, columnValue,
maskedValue, columnValue,
maskedValue, columnValue,
maskedValue, columnValue));
rs = executeNet(format("SELECT c2, c1, k2, k1, r2, r1 FROM %s.%s", KEYSPACE, currentView()));
assertColumnNames(rs, "c2", "c1", "k2", "k1", "r2", "r1");
assertRowsNet(rs, row(maskedValue, columnValue,
maskedValue, columnValue,
maskedValue, columnValue));
}
@Test
public void testSelectOnlyMaskedColumns() throws Throwable
{
ResultSet rs = executeNet("SELECT k2, c2, s2, r2 FROM %s");
assertColumnNames(rs, "k2", "c2", "s2", "r2");
assertRowsNet(rs, row(maskedValue, maskedValue, maskedValue, maskedValue));
rs = executeNet(format("SELECT k2, c2, r2 FROM %s.%s", KEYSPACE, currentView()));
assertColumnNames(rs, "k2", "c2", "r2");
assertRowsNet(rs, row(maskedValue, maskedValue, maskedValue));
}
@Test
public void testSelectOnlyNotMaskedColumns() throws Throwable
{
ResultSet rs = executeNet("SELECT k1, c1, s1, r1 FROM %s");
assertColumnNames(rs, "k1", "c1", "s1", "r1");
assertRowsNet(rs, row(columnValue, columnValue, columnValue, columnValue));
rs = executeNet(format("SELECT k1, c1, r1 FROM %s.%s", KEYSPACE, currentView()));
assertColumnNames(rs, "k1", "c1", "r1");
assertRowsNet(rs, row(columnValue, columnValue, columnValue));
}
}

46
test/unit/org/apache/cassandra/cql3/functions/masking/ColumnMaskQueryWithDefaultTest.java Normal file
View File

@ -0,0 +1,46 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.cql3.functions.masking;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.List;
import org.junit.runners.Parameterized;
/**
* {@link ColumnMaskQueryTester} for {@link DefaultMaskingFunction}.
*/
public class ColumnMaskQueryWithDefaultTest extends ColumnMaskQueryTester
{
@Parameterized.Parameters(name = "order={0}, mask={1}, type={2}, value={3}")
public static Collection<Object[]> options()
{
List<Object[]> options = new ArrayList<>();
for (String order : Arrays.asList("ASC", "DESC"))
{
options.add(new Object[]{ order, "DEFAULT", "text", "abc", "****" });
options.add(new Object[]{ order, "DEFAULT", "int", 123, 0 });
options.add(new Object[]{ order, "mask_default()", "text", "abc", "****" });
options.add(new Object[]{ order, "mask_default()", "int", 123, 0, });
}
return options;
}
}

44
test/unit/org/apache/cassandra/cql3/functions/masking/ColumnMaskQueryWithNullTest.java Normal file
View File

@ -0,0 +1,44 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.cql3.functions.masking;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.List;
import org.junit.runners.Parameterized;
/**
* {@link ColumnMaskQueryTester} for {@link NullMaskingFunction}.
*/
public class ColumnMaskQueryWithNullTest extends ColumnMaskQueryTester
{
@Parameterized.Parameters(name = "order={0}, mask={1}, type={2}, value={3}")
public static Collection<Object[]> options()
{
List<Object[]> options = new ArrayList<>();
for (String order : Arrays.asList("ASC", "DESC"))
{
options.add(new Object[]{ order, "mask_null()", "text", "abc", null });
options.add(new Object[]{ order, "mask_null()", "int", 123, null });
}
return options;
}
}

48
test/unit/org/apache/cassandra/cql3/functions/masking/ColumnMaskQueryWithPartialTest.java Normal file
View File

@ -0,0 +1,48 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.cql3.functions.masking;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.List;
import org.junit.runners.Parameterized;
/**
* {@link ColumnMaskQueryTester} for {@link PartialMaskingFunction}.
*/
public class ColumnMaskQueryWithPartialTest extends ColumnMaskQueryTester
{
@Parameterized.Parameters(name = "order={0}, mask={1}, type={2}, value={3}")
public static Collection<Object[]> options()
{
List<Object[]> options = new ArrayList<>();
for (String order : Arrays.asList("ASC", "DESC"))
{
options.add(new Object[]{ order, "mask_inner(null, null)", "text", "abcdef", "******" });
options.add(new Object[]{ order, "mask_inner(1, 2)", "text", "abcdef", "a***ef" });
options.add(new Object[]{ order, "mask_inner(1, 2, '#')", "text", "abcdef", "a###ef" });
options.add(new Object[]{ order, "mask_outer(1, null)", "text", "abcdef", "*bcdef" });
options.add(new Object[]{ order, "mask_outer(1, 2)", "text", "abcdef", "*bcd**" });
options.add(new Object[]{ order, "mask_outer(1, 2, '#')", "text", "abcdef", "#bcd##", });
}
return options;
}
}

49
test/unit/org/apache/cassandra/cql3/functions/masking/ColumnMaskQueryWithReplaceTest.java Normal file
View File

@ -0,0 +1,49 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.cql3.functions.masking;
import java.math.BigInteger;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.List;
import org.junit.runners.Parameterized;
/**
* {@link ColumnMaskQueryTester} for {@link ReplaceMaskingFunction}.
*/
public class ColumnMaskQueryWithReplaceTest extends ColumnMaskQueryTester
{
@Parameterized.Parameters(name = "order={0}, mask={1}, type={2}, value={3}")
public static Collection<Object[]> options()
{
List<Object[]> options = new ArrayList<>();
for (String order : Arrays.asList("ASC", "DESC"))
{
options.add(new Object[]{ order, "mask_replace(null)", "int", 123, null });
options.add(new Object[]{ order, "mask_replace('redacted')", "ascii", "abc", "redacted" });
options.add(new Object[]{ order, "mask_replace('redacted')", "text", "abc", "redacted" });
options.add(new Object[]{ order, "mask_replace(0)", "int", 123, 0 });
options.add(new Object[]{ order, "mask_replace(0)", "bigint", 123L, 0L });
options.add(new Object[]{ order, "mask_replace(0)", "varint", BigInteger.valueOf(123), BigInteger.ZERO });
}
return options;
}
}

71
test/unit/org/apache/cassandra/cql3/functions/masking/ColumnMaskQueryWithUDFTest.java Normal file
View File

@ -0,0 +1,71 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.cql3.functions.masking;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.List;
import org.junit.Before;
import org.junit.runners.Parameterized;
/**
* {@link ColumnMaskQueryTester} for user-defined functions.
*/
public class ColumnMaskQueryWithUDFTest extends ColumnMaskQueryTester
{
private static final String TEXT_REPLACE_FUNCTION = KEYSPACE + ".mask_text_replace";
private static final String INT_REPLACE_FUNCTION = KEYSPACE + ".mask_int_replace";
@Before
@Override
public void setupSchema() throws Throwable
{
createFunction(KEYSPACE,
"text, text",
"CREATE FUNCTION IF NOT EXISTS " + TEXT_REPLACE_FUNCTION + " (column text, replacement text) " +
"CALLED ON NULL INPUT " +
"RETURNS text " +
"LANGUAGE java " +
"AS 'return replacement;'");
createFunction(KEYSPACE,
"int, int",
"CREATE FUNCTION IF NOT EXISTS " + INT_REPLACE_FUNCTION + " (column int, replacement int) " +
"CALLED ON NULL INPUT " +
"RETURNS int " +
"LANGUAGE java " +
"AS 'return replacement;'");
super.setupSchema();
}
@Parameterized.Parameters(name = "order={0}, mask={1}, type={2}, value={3}")
public static Collection<Object[]> options()
{
List<Object[]> options = new ArrayList<>();
for (String order : Arrays.asList("ASC", "DESC"))
{
options.add(new Object[]{ order, TEXT_REPLACE_FUNCTION + "('redacted')", "text", "abc", "redacted" });
options.add(new Object[]{ order, TEXT_REPLACE_FUNCTION + "('secret')", "text", "abc", "secret" });
options.add(new Object[]{ order, INT_REPLACE_FUNCTION + "(0)", "int", 123, 0 });
options.add(new Object[]{ order, INT_REPLACE_FUNCTION + "(-1)", "int", 123, -1 });
}
return options;
}
}

580
test/unit/org/apache/cassandra/cql3/functions/masking/ColumnMaskTest.java Normal file
View File

@ -0,0 +1,580 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.cql3.functions.masking;
import java.nio.ByteBuffer;
import java.util.Arrays;
import java.util.List;
import org.junit.Test;
import com.datastax.driver.core.BoundStatement;
import com.datastax.driver.core.PreparedStatement;
import com.datastax.driver.core.Session;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.cql3.CQL3Type;
import org.apache.cassandra.cql3.functions.FunctionFactory;
import org.apache.cassandra.cql3.functions.FunctionParameter;
import org.apache.cassandra.cql3.functions.NativeFunction;
import org.apache.cassandra.cql3.functions.NativeFunctions;
import org.apache.cassandra.db.marshal.AbstractType;
import org.apache.cassandra.db.marshal.Int32Type;
import org.apache.cassandra.exceptions.InvalidRequestException;
import static java.lang.String.format;
import static java.util.Collections.emptyList;
import static org.apache.cassandra.cql3.functions.masking.ColumnMask.DISABLED_ERROR_MESSAGE;
/**
* Tests schema altering queries ({@code CREATE TABLE}, {@code ALTER TABLE}, etc.) that attach/dettach dynamic data
* masking functions to column definitions.
*/
public class ColumnMaskTest extends ColumnMaskTester
{
@Test
public void testCollections() throws Throwable
{
// Create table with masks
String table = createTable("CREATE TABLE %s (k int PRIMARY KEY, " +
"s set<int> MASKED WITH DEFAULT, " +
"l list<int> MASKED WITH DEFAULT, " +
"m map<int, int> MASKED WITH DEFAULT, " +
"fs frozen<set<int>> MASKED WITH DEFAULT, " +
"fl frozen<list<int>> MASKED WITH DEFAULT, " +
"fm frozen<map<int, int>> MASKED WITH DEFAULT)");
assertColumnIsMasked(table, "s", "mask_default", emptyList(), emptyList());
assertColumnIsMasked(table, "l", "mask_default", emptyList(), emptyList());
assertColumnIsMasked(table, "m", "mask_default", emptyList(), emptyList());
assertColumnIsMasked(table, "fs", "mask_default", emptyList(), emptyList());
assertColumnIsMasked(table, "fl", "mask_default", emptyList(), emptyList());
assertColumnIsMasked(table, "fm", "mask_default", emptyList(), emptyList());
// Alter column masks
alterTable("ALTER TABLE %s ALTER s MASKED WITH mask_null()");
alterTable("ALTER TABLE %s ALTER l MASKED WITH mask_null()");
alterTable("ALTER TABLE %s ALTER m MASKED WITH mask_null()");
alterTable("ALTER TABLE %s ALTER fs MASKED WITH mask_null()");
alterTable("ALTER TABLE %s ALTER fl MASKED WITH mask_null()");
alterTable("ALTER TABLE %s ALTER fm MASKED WITH mask_null()");
assertColumnIsMasked(table, "s", "mask_null", emptyList(), emptyList());
assertColumnIsMasked(table, "l", "mask_null", emptyList(), emptyList());
assertColumnIsMasked(table, "m", "mask_null", emptyList(), emptyList());
assertColumnIsMasked(table, "fs", "mask_null", emptyList(), emptyList());
assertColumnIsMasked(table, "fl", "mask_null", emptyList(), emptyList());
assertColumnIsMasked(table, "fm", "mask_null", emptyList(), emptyList());
// Drop masks
alterTable("ALTER TABLE %s ALTER s DROP MASKED");
alterTable("ALTER TABLE %s ALTER l DROP MASKED");
alterTable("ALTER TABLE %s ALTER m DROP MASKED");
alterTable("ALTER TABLE %s ALTER fs DROP MASKED");
alterTable("ALTER TABLE %s ALTER fl DROP MASKED");
alterTable("ALTER TABLE %s ALTER fm DROP MASKED");
assertTableColumnsAreNotMasked("s", "l", "m", "fs", "fl", "fm");
}
@Test
public void testUDTs() throws Throwable
{
String type = createType("CREATE TYPE %s (a1 varint, a2 varint, a3 varint);");
// Create table with mask
String table = createTable(format("CREATE TABLE %%s (k int PRIMARY KEY, v %s MASKED WITH DEFAULT)", type));
assertColumnIsMasked(table, "v", "mask_default", emptyList(), emptyList());
// Alter column mask
alterTable("ALTER TABLE %s ALTER v MASKED WITH mask_null()");
assertColumnIsMasked(table, "v", "mask_null", emptyList(), emptyList());
// Drop mask
alterTable("ALTER TABLE %s ALTER v DROP MASKED");
assertTableColumnsAreNotMasked("v");
}
@Test
public void testAlterTableAddMaskingToNonExistingColumn() throws Throwable
{
String table = createTable("CREATE TABLE %s (k int PRIMARY KEY, v text)");
execute("ALTER TABLE %s ALTER IF EXISTS unknown MASKED WITH DEFAULT");
assertInvalidMessage(format("Column with name 'unknown' doesn't exist on table '%s'", table),
formatQuery("ALTER TABLE %s ALTER unknown MASKED WITH DEFAULT"));
}
@Test
public void testAlterTableRemoveMaskingFromNonExistingColumn() throws Throwable
{
String table = createTable("CREATE TABLE %s (k int PRIMARY KEY, v text)");
execute("ALTER TABLE %s ALTER IF EXISTS unknown DROP MASKED");
assertInvalidMessage(format("Column with name 'unknown' doesn't exist on table '%s'", table),
formatQuery("ALTER TABLE %s ALTER unknown DROP MASKED"));
}
@Test
public void testAlterTableRemoveMaskFromUnmaskedColumn() throws Throwable
{
createTable("CREATE TABLE %s (k int PRIMARY KEY, v text)");
execute("ALTER TABLE %s ALTER v DROP MASKED");
assertTableColumnsAreNotMasked("v");
}
@Test
public void testInvalidMaskingFunctionName() throws Throwable
{
// create table
createTableName();
assertInvalidMessage("Unable to find masking function for v, no declared function matches the signature mask_missing()",
formatQuery("CREATE TABLE %s (k int PRIMARY KEY, v int MASKED WITH mask_missing())"));
// alter table
createTable("CREATE TABLE %s (k int PRIMARY KEY, v int)");
assertInvalidMessage("Unable to find masking function for v, no declared function matches the signature mask_missing()",
"ALTER TABLE %s ALTER v MASKED WITH mask_missing()");
assertTableColumnsAreNotMasked("k", "v");
}
@Test
public void testInvalidMaskingFunctionArguments() throws Throwable
{
// create table
createTableName();
assertInvalidMessage("Invalid number of arguments for function system.mask_default(any)",
formatQuery("CREATE TABLE %s (k int PRIMARY KEY, v int MASKED WITH mask_default(1))"));
// alter table
createTable("CREATE TABLE %s (k int PRIMARY KEY, v int)");
assertInvalidMessage("Invalid number of arguments for function system.mask_default(any)",
"ALTER TABLE %s ALTER v MASKED WITH mask_default(1)");
assertTableColumnsAreNotMasked("k", "v");
}
@Test
public void testInvalidMaskingFunctionArgumentTypes() throws Throwable
{
// create table
createTableName();
assertInvalidMessage("Function system.mask_inner requires an argument of type int, but found argument 'a' of type ascii",
formatQuery("CREATE TABLE %s (k int PRIMARY KEY, v text MASKED WITH mask_inner('a', 'b'))"));
// alter table
createTable("CREATE TABLE %s (k int PRIMARY KEY, v text)");
assertInvalidMessage("Function system.mask_inner requires an argument of type int, but found argument 'a' of type ascii",
"ALTER TABLE %s ALTER v MASKED WITH mask_inner('a', 'b')");
assertTableColumnsAreNotMasked("k", "v");
}
@Test
public void testColumnMaskingWithNotMaskingFunction() throws Throwable
{
// create table
createTableName();
assertInvalidMessage("Not-masking function tojson() cannot be used for masking table columns",
formatQuery("CREATE TABLE %s (k int PRIMARY KEY, v text MASKED WITH tojson())"));
// alter table
createTable("CREATE TABLE %s (k int PRIMARY KEY, v text)");
assertInvalidMessage("Not-masking function tojson() cannot be used for masking table columns",
"ALTER TABLE %s ALTER v MASKED WITH tojson()");
assertTableColumnsAreNotMasked("k", "v");
}
@Test
@SuppressWarnings("resource")
public void testPreparedStatement() throws Throwable
{
createTable("CREATE TABLE %s (k int PRIMARY KEY, v text MASKED WITH DEFAULT)");
execute("INSERT INTO %s (k, v) VALUES (0, 'sensitive')");
Session session = sessionNet();
PreparedStatement prepared = session.prepare(formatQuery("SELECT v FROM %s WHERE k = ?"));
BoundStatement bound = prepared.bind(0);
assertRowsNet(session.execute(bound), row("****"));
alterTable("ALTER TABLE %s ALTER v DROP MASKED");
assertRowsNet(session.execute(bound), row("sensitive"));
alterTable("ALTER TABLE %s ALTER v MASKED WITH mask_replace('redacted')");
assertRowsNet(session.execute(bound), row("redacted"));
}
@Test
public void testViews() throws Throwable
{
createTable("CREATE TABLE %s (k int, c int, v text MASKED WITH mask_replace('redacted'), PRIMARY KEY (k, c))");
execute("INSERT INTO %s (k, c, v) VALUES (0, 0, 'sensitive')");
String view = createView("CREATE MATERIALIZED VIEW %s AS SELECT * FROM %s " +
"WHERE k IS NOT NULL AND c IS NOT NULL AND v IS NOT NULL " +
"PRIMARY KEY (v, k, c)");
waitForViewMutations();
assertRowsNet(executeNet(format("SELECT v FROM %s.%s", KEYSPACE, view)), row("redacted"));
assertRowsNet(executeNet(format("SELECT v FROM %s.%s WHERE v='sensitive'", KEYSPACE, view)), row("redacted"));
assertRowsNet(executeNet(format("SELECT v FROM %s.%s WHERE v='redacted'", KEYSPACE, view)));
alterTable("ALTER TABLE %s ALTER v DROP MASKED");
assertRowsNet(executeNet(format("SELECT v FROM %s.%s", KEYSPACE, view)), row("sensitive"));
assertRowsNet(executeNet(format("SELECT v FROM %s.%s WHERE v='sensitive'", KEYSPACE, view)), row("sensitive"));
assertRowsNet(executeNet(format("SELECT v FROM %s.%s WHERE v='redacted'", KEYSPACE, view)));
}
@Test
@SuppressWarnings("resource")
public void testPreparedStatementOnView() throws Throwable
{
createTable("CREATE TABLE %s (k int, c int, v text MASKED WITH DEFAULT, PRIMARY KEY (k, c))");
execute("INSERT INTO %s (k, c, v) VALUES (0, 0, 'sensitive')");
String view = createView("CREATE MATERIALIZED VIEW %s AS SELECT * FROM %s " +
"WHERE k IS NOT NULL AND c IS NOT NULL AND v IS NOT NULL " +
"PRIMARY KEY (v, k, c)");
waitForViewMutations();
Session session = sessionNet();
PreparedStatement prepared = session.prepare(format("SELECT v FROM %s.%s WHERE v=?", KEYSPACE, view));
BoundStatement bound = prepared.bind("sensitive");
assertRowsNet(session.execute(bound), row("****"));
alterTable("ALTER TABLE %s ALTER v DROP MASKED");
assertRowsNet(session.execute(bound), row("sensitive"));
alterTable("ALTER TABLE %s ALTER v MASKED WITH mask_replace('redacted')");
assertRowsNet(session.execute(bound), row("redacted"));
}
@Test
public void testGroupBy() throws Throwable
{
createTable("CREATE TABLE %s (k int, c int, v text, PRIMARY KEY (k, c))");
execute("INSERT INTO %s (k, c, v) VALUES (0, 0, 'sensitive')");
execute("INSERT INTO %s (k, c, v) VALUES (0, 1, 'sensitive')");
execute("INSERT INTO %s (k, c, v) VALUES (1, 0, 'sensitive')");
execute("INSERT INTO %s (k, c, v) VALUES (1, 1, 'sensitive')");
// without masks
String query = "SELECT * FROM %s GROUP BY k";
assertRowsNet(executeNet(query), row(1, 0, "sensitive"), row(0, 0, "sensitive"));
// with masked regular column
alterTable("ALTER TABLE %s ALTER v MASKED WITH mask_replace('redacted')");
assertRowsNet(executeNet(query), row(1, 0, "redacted"), row(0, 0, "redacted"));
// with masked clustering key
alterTable("ALTER TABLE %s ALTER c MASKED WITH mask_replace(-1)");
assertRowsNet(executeNet(query), row(1, -1, "redacted"), row(0, -1, "redacted"));
// with masked partition key
alterTable("ALTER TABLE %s ALTER k MASKED WITH mask_replace(-1)");
assertRowsNet(executeNet(query), row(-1, -1, "redacted"), row(-1, -1, "redacted"));
// again without masks
alterTable("ALTER TABLE %s ALTER k DROP MASKED");
alterTable("ALTER TABLE %s ALTER c DROP MASKED");
alterTable("ALTER TABLE %s ALTER v DROP MASKED");
assertRowsNet(executeNet(query), row(1, 0, "sensitive"), row(0, 0, "sensitive"));
}
@Test
public void testPaging() throws Throwable
{
createTable("CREATE TABLE %s (k int, c int, v text, PRIMARY KEY (k, c))");
execute("INSERT INTO %s (k, c, v) VALUES (0, 0, 'sensitive')");
execute("INSERT INTO %s (k, c, v) VALUES (0, 1, 'sensitive')");
execute("INSERT INTO %s (k, c, v) VALUES (1, 0, 'sensitive')");
// without masks
assertRowsWithPaging("SELECT * FROM %s", row(1, 0, "sensitive"), row(0, 0, "sensitive"), row(0, 1, "sensitive"));
assertRowsWithPaging("SELECT * FROM %s WHERE k = 1", row(1, 0, "sensitive"));
assertRowsWithPaging("SELECT * FROM %s WHERE k = 0", row(0, 0, "sensitive"), row(0, 1, "sensitive"));
assertRowsWithPaging("SELECT * FROM %s WHERE k = 0 AND c = 1", row(0, 1, "sensitive"));
// with masked regular column
alterTable("ALTER TABLE %s ALTER v MASKED WITH mask_replace('redacted')");
assertRowsWithPaging("SELECT * FROM %s", row(1, 0, "redacted"), row(0, 0, "redacted"), row(0, 1, "redacted"));
assertRowsWithPaging("SELECT * FROM %s WHERE k = 1", row(1, 0, "redacted"));
assertRowsWithPaging("SELECT * FROM %s WHERE k = 0", row(0, 0, "redacted"), row(0, 1, "redacted"));
assertRowsWithPaging("SELECT * FROM %s WHERE k = 0 AND c = 1", row(0, 1, "redacted"));
// with masked clustering key
alterTable("ALTER TABLE %s ALTER c MASKED WITH mask_replace(-1)");
assertRowsWithPaging("SELECT * FROM %s", row(1, -1, "redacted"), row(0, -1, "redacted"), row(0, -1, "redacted"));
assertRowsWithPaging("SELECT * FROM %s WHERE k = 1", row(1, -1, "redacted"));
assertRowsWithPaging("SELECT * FROM %s WHERE k = 0", row(0, -1, "redacted"), row(0, -1, "redacted"));
assertRowsWithPaging("SELECT * FROM %s WHERE k = 0 AND c = 1", row(0, -1, "redacted"));
// with masked partition key
alterTable("ALTER TABLE %s ALTER k MASKED WITH mask_replace(-1)");
assertRowsWithPaging("SELECT * FROM %s", row(-1, -1, "redacted"), row(-1, -1, "redacted"), row(-1, -1, "redacted"));
assertRowsWithPaging("SELECT * FROM %s WHERE k = 1", row(-1, -1, "redacted"));
assertRowsWithPaging("SELECT * FROM %s WHERE k = 0", row(-1, -1, "redacted"), row(-1, -1, "redacted"));
assertRowsWithPaging("SELECT * FROM %s WHERE k = 0 AND c = 1", row(-1, -1, "redacted"));
// again without masks
alterTable("ALTER TABLE %s ALTER k DROP MASKED");
alterTable("ALTER TABLE %s ALTER c DROP MASKED");
alterTable("ALTER TABLE %s ALTER v DROP MASKED");
assertRowsWithPaging("SELECT * FROM %s", row(1, 0, "sensitive"), row(0, 0, "sensitive"), row(0, 1, "sensitive"));
assertRowsWithPaging("SELECT * FROM %s WHERE k = 1", row(1, 0, "sensitive"));
assertRowsWithPaging("SELECT * FROM %s WHERE k = 0", row(0, 0, "sensitive"), row(0, 1, "sensitive"));
assertRowsWithPaging("SELECT * FROM %s WHERE k = 0 AND c = 1", row(0, 1, "sensitive"));
}
/**
* Tests that rows are always ordered according to the clear values of the columns, even for the post-ordering done
* for queries with {@code IN} restrictions and {@code ORDER BY} clauses.
*/
@Test
public void testPostOrdering() throws Throwable
{
createTable("CREATE TABLE %s (k int, c int, v int, PRIMARY KEY (k, c))");
execute("INSERT INTO %s (k, c, v) VALUES (0, 0, 0)");
execute("INSERT INTO %s (k, c, v) VALUES (0, 1, 1)");
execute("INSERT INTO %s (k, c, v) VALUES (1, 0, 3)");
execute("INSERT INTO %s (k, c, v) VALUES (1, 1, 4)");
execute("INSERT INTO %s (k, c, v) VALUES (2, 0, 6)");
execute("INSERT INTO %s (k, c, v) VALUES (2, 1, 7)");
NativeFunctions.instance.add(NEGATIVE);
// Test ordering without masking, just for reference
assertRowsNet(executeNet("SELECT * FROM %s WHERE k IN (0, 1, 2)"),
row(0, 0, 0),
row(0, 1, 1),
row(1, 0, 3),
row(1, 1, 4),
row(2, 0, 6),
row(2, 1, 7));
assertRowsNet(executeNetWithoutPaging("SELECT * FROM %s WHERE k IN (0, 1, 2) ORDER BY c ASC"),
row(0, 0, 0),
row(1, 0, 3),
row(2, 0, 6),
row(0, 1, 1),
row(1, 1, 4),
row(2, 1, 7));
assertRowsNet(executeNetWithoutPaging("SELECT * FROM %s WHERE k IN (0, 1, 2) ORDER BY c DESC"),
row(0, 1, 1),
row(1, 1, 4),
row(2, 1, 7),
row(0, 0, 0),
row(1, 0, 3),
row(2, 0, 6));
// Test ordering with manually applied masking function, just for reference
assertRowsNet(executeNet("SELECT k, mask_negative(c), v FROM %s WHERE k IN (0, 1, 2)"),
row(0, -0, 0), // (0, 0, 0)
row(0, -1, 1), // (0, 1, 1)
row(1, -0, 3), // (1, 0, 3)
row(1, -1, 4), // (1, 1, 4)
row(2, -0, 6), // (2, 0, 6)
row(2, -1, 7)); // (2, 1, 7)
assertRowsNet(executeNetWithoutPaging("SELECT k, mask_negative(c), v FROM %s WHERE k IN (0, 1, 2) ORDER BY c ASC"),
row(0, -0, 0), // (0, 0, 0)
row(1, -0, 3), // (1, 0, 3)
row(2, -0, 6), // (2, 0, 6)
row(0, -1, 1), // (0, 1, 1)
row(1, -1, 4), // (1, 1, 4)
row(2, -1, 7)); // (2, 1, 7)
assertRowsNet(executeNetWithoutPaging("SELECT k, mask_negative(c), v FROM %s WHERE k IN (0, 1, 2) ORDER BY c DESC"),
row(0, -1, 1), // (0, 1, 1)
row(1, -1, 4), // (1, 1, 4)
row(2, -1, 7), // (2, 1, 7)
row(0, -0, 0), // (0, 0, 0)
row(1, -0, 3), // (1, 0, 3)
row(2, -0, 6)); // (2, 0, 6)
alterTable("ALTER TABLE %s ALTER c MASKED WITH mask_negative()");
// Test ordering of wildcard queries with masked column
assertRowsNet(executeNet("SELECT * FROM %s WHERE k IN (0, 1, 2)"),
row(0, -0, 0), // (0, 0, 0)
row(0, -1, 1), // (0, 1, 1)
row(1, -0, 3), // (1, 0, 3)
row(1, -1, 4), // (1, 1, 4)
row(2, -0, 6), // (2, 0, 6)
row(2, -1, 7)); // (2, 1, 7)
assertRowsNet(executeNetWithoutPaging("SELECT * FROM %s WHERE k IN (0, 1, 2) ORDER BY c ASC"),
row(0, -0, 0), // (0, 0, 0)
row(1, -0, 3), // (1, 0, 3)
row(2, -0, 6), // (2, 0, 6)
row(0, -1, 1), // (0, 1, 1)
row(1, -1, 4), // (1, 1, 4)
row(2, -1, 7)); // (2, 1, 7)
assertRowsNet(executeNetWithoutPaging("SELECT * FROM %s WHERE k IN (0, 1, 2) ORDER BY c DESC"),
row(0, -1, 1), // (0, 1, 1)
row(1, -1, 4), // (1, 1, 4)
row(2, -1, 7), // (2, 1, 7)
row(0, -0, 0), // (0, 0, 0)
row(1, -0, 3), // (1, 0, 3)
row(2, -0, 6)); // (2, 0, 6)
// Test ordering of column selection queries with masked column
assertRowsNet(executeNet("SELECT k, c, v FROM %s WHERE k IN (0, 1, 2)"),
row(0, -0, 0), // (0, 0, 0)
row(0, -1, 1), // (0, 1, 1)
row(1, -0, 3), // (1, 0, 3)
row(1, -1, 4), // (1, 1, 4)
row(2, -0, 6), // (2, 0, 6)
row(2, -1, 7)); // (2, 1, 7)
assertRowsNet(executeNetWithoutPaging("SELECT k, c, v FROM %s WHERE k IN (0, 1, 2) ORDER BY c ASC"),
row(0, -0, 0), // (0, 0, 0)
row(1, -0, 3), // (1, 0, 3)
row(2, -0, 6), // (2, 0, 6)
row(0, -1, 1), // (0, 1, 1)
row(1, -1, 4), // (1, 1, 4)
row(2, -1, 7)); // (2, 1, 7)
assertRowsNet(executeNetWithoutPaging("SELECT k, c, v FROM %s WHERE k IN (0, 1, 2) ORDER BY c DESC"),
row(0, -1, 1), // (0, 1, 1)
row(1, -1, 4), // (1, 1, 4)
row(2, -1, 7), // (2, 1, 7)
row(0, -0, 0), // (0, 0, 0)
row(1, -0, 3), // (1, 0, 3)
row(2, -0, 6)); // (2, 0, 6)
// Test ordering of column selection queries with masked column, without selecting the ordered column
assertRowsNet(executeNet("SELECT k, v FROM %s WHERE k IN (0, 1, 2)"),
row(0, 0), // (0, 0, 0)
row(0, 1), // (0, 1, 1)
row(1, 3), // (1, 0, 3)
row(1, 4), // (1, 1, 4)
row(2, 6), // (2, 0, 6)
row(2, 7)); // (2, 1, 7)
assertRowsNet(executeNetWithoutPaging("SELECT k, v FROM %s WHERE k IN (0, 1, 2) ORDER BY c ASC"),
row(0, 0), // (0, 0, 0)
row(1, 3), // (1, 0, 3)
row(2, 6), // (2, 0, 6)
row(0, 1), // (0, 1, 1)
row(1, 4), // (1, 1, 4)
row(2, 7)); // (2, 1, 7)
assertRowsNet(executeNetWithoutPaging("SELECT k, v FROM %s WHERE k IN (0, 1, 2) ORDER BY c DESC"),
row(0, 1), // (0, 1, 1)
row(1, 4), // (1, 1, 4)
row(2, 7), // (2, 1, 7)
row(0, 0), // (0, 0, 0)
row(1, 3), // (1, 0, 3)
row(2, 6)); // (2, 0, 6)
// Test ordering of wildcard JSON queries with masked column
assertRowsNet(executeNet("SELECT JSON * FROM %s WHERE k IN (0, 1, 2)"),
row("{\"k\": 0, \"c\": 0, \"v\": 0}"), // (0, 0, 0)
row("{\"k\": 0, \"c\": -1, \"v\": 1}"), // (0, 1, 1)
row("{\"k\": 1, \"c\": 0, \"v\": 3}"), // (1, 0, 3)
row("{\"k\": 1, \"c\": -1, \"v\": 4}"), // (1, 1, 4)
row("{\"k\": 2, \"c\": 0, \"v\": 6}"), // (2, 0, 6)
row("{\"k\": 2, \"c\": -1, \"v\": 7}")); // (2, 1, 7)
assertRowsNet(executeNetWithoutPaging("SELECT JSON * FROM %s WHERE k IN (0, 1, 2) ORDER BY c ASC"),
row("{\"k\": 0, \"c\": 0, \"v\": 0}"), // (0, 0, 0)
row("{\"k\": 1, \"c\": 0, \"v\": 3}"), // (1, 0, 3)
row("{\"k\": 2, \"c\": 0, \"v\": 6}"), // (2, 0, 6)
row("{\"k\": 0, \"c\": -1, \"v\": 1}"), // (0, 1, 1)
row("{\"k\": 1, \"c\": -1, \"v\": 4}"), // (1, 1, 4)
row("{\"k\": 2, \"c\": -1, \"v\": 7}")); // (2, 1, 7)
assertRowsNet(executeNetWithoutPaging("SELECT JSON * FROM %s WHERE k IN (0, 1, 2) ORDER BY c DESC"),
row("{\"k\": 0, \"c\": -1, \"v\": 1}"), // (0, 1, 1)
row("{\"k\": 1, \"c\": -1, \"v\": 4}"), // (1, 1, 4)
row("{\"k\": 2, \"c\": -1, \"v\": 7}"), // (2, 1, 7)
row("{\"k\": 0, \"c\": 0, \"v\": 0}"), // (0, 0, 0)
row("{\"k\": 1, \"c\": 0, \"v\": 3}"), // (1, 0, 3)
row("{\"k\": 2, \"c\": 0, \"v\": 6}")); // (2, 0, 6)
}
@Test
public void testEnableFlag() throws Throwable
{
// verify that we cannot create tables with masked columns if DDM is disabled
DatabaseDescriptor.setDynamicDataMaskingEnabled(false);
assertInvalidThrowMessage(DISABLED_ERROR_MESSAGE,
InvalidRequestException.class,
"CREATE TABLE t (k int PRIMARY KEY, v text MASKED WITH DEFAULT)");
assertInvalidThrowMessage(DISABLED_ERROR_MESSAGE,
InvalidRequestException.class,
"CREATE TABLE t (k int MASKED WITH DEFAULT PRIMARY KEY, v text)");
assertInvalidThrowMessage(DISABLED_ERROR_MESSAGE,
InvalidRequestException.class,
"CREATE TABLE t (k int, c int MASKED WITH DEFAULT, v text, PRIMARY KEY(k, c))");
// verify that we cannot mask an existing column if DDM is disabled
createTable("CREATE TABLE %s (k int, c int, s int static, r int, PRIMARY KEY(k, c))");
assertInvalidThrowMessage(DISABLED_ERROR_MESSAGE,
InvalidRequestException.class,
"ALTER TABLE %s ALTER k MASKED WITH DEFAULT");
assertInvalidThrowMessage(DISABLED_ERROR_MESSAGE,
InvalidRequestException.class,
"ALTER TABLE %s ALTER c MASKED WITH DEFAULT");
assertInvalidThrowMessage(DISABLED_ERROR_MESSAGE,
InvalidRequestException.class,
"ALTER TABLE %s ALTER s MASKED WITH DEFAULT");
assertInvalidThrowMessage(DISABLED_ERROR_MESSAGE,
InvalidRequestException.class,
"ALTER TABLE %s ALTER r MASKED WITH DEFAULT");
// enable DDM and add some masked data
DatabaseDescriptor.setDynamicDataMaskingEnabled(true);
alterTable("ALTER TABLE %s ALTER k MASKED WITH DEFAULT");
alterTable("ALTER TABLE %s ALTER c MASKED WITH DEFAULT");
alterTable("ALTER TABLE %s ALTER s MASKED WITH DEFAULT");
alterTable("ALTER TABLE %s ALTER r MASKED WITH DEFAULT");
execute("INSERT INTO %s (k, c, s, r) VALUES (1, 2, 3, 4)");
assertRowsNet(executeNet("SELECT * FROM %s"), row(0, 0, 0, 0));
// verify that column masks are not applied if DDM is disabled
DatabaseDescriptor.setDynamicDataMaskingEnabled(false);
assertRowsNet(executeNet("SELECT * FROM %s"), row(1, 2, 3, 4));
// verify that we can drop column masks even if DDM is disabled
alterTable("ALTER TABLE %s ALTER k DROP MASKED");
alterTable("ALTER TABLE %s ALTER c DROP MASKED");
alterTable("ALTER TABLE %s ALTER s DROP MASKED");
alterTable("ALTER TABLE %s ALTER r DROP MASKED");
DatabaseDescriptor.setDynamicDataMaskingEnabled(true);
assertRowsNet(executeNet("SELECT * FROM %s"), row(1, 2, 3, 4));
}
private void assertRowsWithPaging(String query, Object[]... rows)
{
for (int pageSize : Arrays.asList(1, 2, 3, 4, 5, 100))
{
assertRowsNet(executeNetWithPaging(query, pageSize), rows);
for (int limit : Arrays.asList(1, 2, 3, 4, 5, 100))
{
assertRowsNet(executeNetWithPaging(query + " LIMIT " + limit, pageSize),
Arrays.copyOfRange(rows, 0, Math.min(limit, rows.length)));
}
}
}
private static final FunctionFactory NEGATIVE = new FunctionFactory("mask_negative", FunctionParameter.fixed(CQL3Type.Native.INT))
{
@Override
protected NativeFunction doGetOrCreateFunction(List<AbstractType<?>> argTypes, AbstractType<?> receiverType)
{
return new MaskingFunction(name, argTypes.get(0), argTypes.get(0))
{
@Override
public Masker masker(ByteBuffer... parameters)
{
return bb -> {
if (bb == null)
return null;
Integer value = Int32Type.instance.compose(bb) * -1;
return Int32Type.instance.decompose(value);
};
}
};
}
};
}

158
test/unit/org/apache/cassandra/cql3/functions/masking/ColumnMaskTester.java Normal file
View File

@ -0,0 +1,158 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.cql3.functions.masking;
import java.nio.ByteBuffer;
import java.util.Collections;
import java.util.List;
import javax.annotation.Nullable;
import org.junit.Before;
import org.junit.BeforeClass;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.cql3.CQLTester;
import org.apache.cassandra.cql3.ColumnIdentifier;
import org.apache.cassandra.cql3.UntypedResultSet;
import org.apache.cassandra.cql3.functions.ScalarFunction;
import org.apache.cassandra.db.Keyspace;
import org.apache.cassandra.db.marshal.AbstractType;
import org.apache.cassandra.db.marshal.ReversedType;
import org.apache.cassandra.schema.ColumnMetadata;
import org.apache.cassandra.schema.KeyspaceMetadata;
import org.apache.cassandra.schema.Schema;
import org.apache.cassandra.schema.SchemaConstants;
import org.apache.cassandra.schema.SchemaKeyspace;
import org.apache.cassandra.schema.SchemaKeyspaceTables;
import org.apache.cassandra.schema.TableMetadata;
import org.apache.cassandra.schema.UserFunctions;
import static java.lang.String.format;
import static org.assertj.core.api.Assertions.assertThat;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertNull;
import static org.junit.Assert.fail;
/**
* Tests table columns with attached dynamic data masking functions.
*/
public class ColumnMaskTester extends CQLTester
{
protected static final String USERNAME = "ddm_user";
protected static final String PASSWORD = "ddm_password";
@BeforeClass
public static void beforeClass()
{
CQLTester.setUpClass();
requireAuthentication();
requireNetwork();
}
@Before
public void before() throws Throwable
{
DatabaseDescriptor.setDynamicDataMaskingEnabled(true);
useSuperUser();
executeNet(format("CREATE USER IF NOT EXISTS %s WITH PASSWORD '%s'", USERNAME, PASSWORD));
executeNet(format("GRANT ALL ON KEYSPACE %s TO %s", KEYSPACE, USERNAME));
executeNet(format("REVOKE UNMASK ON KEYSPACE %s FROM %s", KEYSPACE, USERNAME));
useUser(USERNAME, PASSWORD);
}
protected void assertTableColumnsAreNotMasked(String... columns) throws Throwable
{
for (String column : columns)
{
assertColumnIsNotMasked(currentTable(), column);
}
}
protected void assertViewColumnsAreNotMasked(String... columns) throws Throwable
{
for (String column : columns)
{
assertColumnIsNotMasked(currentView(), column);
}
}
protected void assertColumnIsNotMasked(String table, String column) throws Throwable
{
ColumnMask mask = getColumnMask(table, column);
assertNull(format("Mask for column '%s'", column), mask);
assertRows(execute(format("SELECT * FROM %s.%s WHERE keyspace_name = ? AND table_name = ? AND column_name = ?",
SchemaConstants.SCHEMA_KEYSPACE_NAME, SchemaKeyspaceTables.COLUMN_MASKS),
KEYSPACE, table, column));
}
protected void assertColumnIsMasked(String table,
String column,
String functionName,
List<AbstractType<?>> partialArgumentTypes,
List<ByteBuffer> partialArgumentValues) throws Throwable
{
KeyspaceMetadata keyspaceMetadata = Keyspace.open(KEYSPACE).getMetadata();
TableMetadata tableMetadata = keyspaceMetadata.getTableOrViewNullable(table);
assertNotNull(tableMetadata);
ColumnMetadata columnMetadata = tableMetadata.getColumn(ColumnIdentifier.getInterned(column, false));
assertNotNull(columnMetadata);
AbstractType<?> columnType = columnMetadata.type;
// Verify the column mask in the in-memory schema
ColumnMask mask = getColumnMask(table, column);
assertNotNull(mask);
assertThat(mask.partialArgumentTypes()).isEqualTo(columnType.isReversed() && functionName.equals("mask_replace")
? Collections.singletonList(ReversedType.getInstance(partialArgumentTypes.get(0)))
: partialArgumentTypes);
assertThat(mask.partialArgumentValues()).isEqualTo(partialArgumentValues);
// Verify the function in the column mask
ScalarFunction function = mask.function;
assertNotNull(function);
assertThat(function.name().name).isEqualTo(functionName);
assertThat(function.argTypes().get(0).asCQL3Type()).isEqualTo(columnMetadata.type.asCQL3Type());
assertThat(function.argTypes().size()).isEqualTo(partialArgumentTypes.size() + 1);
// Retrieve the persisted column metadata
UntypedResultSet columnRows = execute("SELECT * FROM system_schema.columns " +
"WHERE keyspace_name = ? AND table_name = ? AND column_name = ?",
KEYSPACE, table, column);
ColumnMetadata persistedColumn = SchemaKeyspace.createColumnFromRow(columnRows.one(), keyspaceMetadata.types, UserFunctions.none());
// Verify the column mask in the persisted schema
ColumnMask savedMask = persistedColumn.getMask();
assertNotNull(savedMask);
assertThat(mask).isEqualTo(savedMask);
assertThat(mask.function.argTypes()).isEqualTo(savedMask.function.argTypes());
}
@Nullable
protected ColumnMask getColumnMask(String table, String column)
{
TableMetadata tableMetadata = Schema.instance.getTableMetadata(KEYSPACE, table);
assertNotNull(tableMetadata);
ColumnMetadata columnMetadata = tableMetadata.getColumn(ColumnIdentifier.getInterned(column, false));
if (columnMetadata == null)
fail(format("Unknown column '%s'", column));
return columnMetadata.getMask();
}
}

148
test/unit/org/apache/cassandra/cql3/functions/masking/ColumnMaskWithTypeAlteringFunctionTest.java Normal file
View File

@ -0,0 +1,148 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.cql3.functions.masking;
import java.util.Arrays;
import java.util.Collection;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.apache.cassandra.cql3.CQL3Type;
import static java.lang.String.format;
import static org.apache.cassandra.cql3.CQL3Type.Native.ASCII;
import static org.apache.cassandra.cql3.CQL3Type.Native.BIGINT;
import static org.apache.cassandra.cql3.CQL3Type.Native.BLOB;
import static org.apache.cassandra.cql3.CQL3Type.Native.INT;
import static org.apache.cassandra.cql3.CQL3Type.Native.TEXT;
import static org.apache.cassandra.cql3.CQL3Type.Native.VARINT;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
/**
* {@link ColumnMaskTester} for masks using functions that might return values with a type different to the type of the
* masked column. Those queries should fail.
*/
@RunWith(Parameterized.class)
public class ColumnMaskWithTypeAlteringFunctionTest extends ColumnMaskTester
{
/** The column mask as expressed in CQL statements right after the {@code MASKED WITH} keywords. */
@Parameterized.Parameter
public String mask;
/** The type of the masked column. */
@Parameterized.Parameter(1)
public CQL3Type.Native type;
/** The type returned by the tested masking function. */
@Parameterized.Parameter(2)
public CQL3Type returnedType;
private boolean shouldSucceed;
private String errorMessage;
@Parameterized.Parameters(name = "mask={0} type={1}")
public static Collection<Object[]> options()
{
return Arrays.asList(new Object[][]{
{ "mask_default()", INT, INT },
{ "mask_default()", TEXT, TEXT },
{ "mask_null()", INT, INT },
{ "mask_null()", TEXT, TEXT },
{ "mask_hash()", INT, BLOB },
{ "mask_hash('SHA-512')", INT, BLOB },
{ "mask_inner(1,2)", TEXT, TEXT },
{ "mask_outer(1,2)", TEXT, TEXT },
{ "mask_replace(0)", INT, INT },
{ "mask_replace(0)", BIGINT, BIGINT },
{ "mask_replace(0)", VARINT, VARINT },
{ "mask_replace('redacted')", ASCII, ASCII },
{ "mask_replace('redacted')", TEXT, TEXT } });
}
@Before
public void setupExpectedResults()
{
shouldSucceed = returnedType == type;
errorMessage = shouldSucceed ? null : format("Masking function %s return type is %s.", mask, returnedType);
}
@Test
public void testTypeAlteringFunctionOnCreateTable()
{
testOnCreateTable("CREATE TABLE %s.%s (k int PRIMARY KEY, v %s MASKED WITH %s)");
testOnCreateTable("CREATE TABLE %s.%s (k %s MASKED WITH %s PRIMARY KEY, v int)");
testOnCreateTable("CREATE TABLE %s.%s (k1 int, k2 %s MASKED WITH %s, PRIMARY KEY((k1, k2)))");
testOnCreateTable("CREATE TABLE %s.%s (k int, c %s MASKED WITH %s, PRIMARY KEY(k, c)) WITH CLUSTERING ORDER BY (c ASC)");
testOnCreateTable("CREATE TABLE %s.%s (k int, c %s MASKED WITH %s, PRIMARY KEY(k, c)) WITH CLUSTERING ORDER BY (c DESC)");
testOnCreateTable("CREATE TABLE %s.%s (k int, c int, s %s STATIC MASKED WITH %s, PRIMARY KEY(k, c))");
}
private void testOnCreateTable(String query)
{
String formattedQuery = format(query, KEYSPACE, createTableName(), type, mask);
if (shouldSucceed)
createTable(formattedQuery);
else
assertThatThrownBy(() -> execute(formattedQuery)).hasMessageContaining(errorMessage);
}
@Test
public void testTypeAlteringFunctionOnMaskColumn()
{
testOnAlterColumn("CREATE TABLE %s.%s (k int PRIMARY KEY, v %s)",
"ALTER TABLE %s.%s ALTER v MASKED WITH %s");
testOnAlterColumn("CREATE TABLE %s.%s (k %s PRIMARY KEY, v int)",
"ALTER TABLE %s.%s ALTER k MASKED WITH %s");
testOnAlterColumn("CREATE TABLE %s.%s (k1 int, k2 %s, PRIMARY KEY((k1, k2)))",
"ALTER TABLE %s.%s ALTER k2 MASKED WITH %s");
testOnAlterColumn("CREATE TABLE %s.%s (k int, c %s, PRIMARY KEY(k, c)) WITH CLUSTERING ORDER BY (c ASC)",
"ALTER TABLE %s.%s ALTER c MASKED WITH %s");
testOnAlterColumn("CREATE TABLE %s.%s (k int, c %s, PRIMARY KEY(k, c)) WITH CLUSTERING ORDER BY (c DESC)",
"ALTER TABLE %s.%s ALTER c MASKED WITH %s");
testOnAlterColumn("CREATE TABLE %s.%s (k int, c int, s %s STATIC, PRIMARY KEY(k, c))",
"ALTER TABLE %s.%s ALTER s MASKED WITH %s");
}
private void testOnAlterColumn(String createQuery, String alterQuery)
{
String table = createTableName();
createTable(format(createQuery, KEYSPACE, table, type));
String formattedQuery = format(alterQuery, KEYSPACE, table, mask);
if (shouldSucceed)
alterTable(formattedQuery);
else
assertThatThrownBy(() -> execute(formattedQuery)).hasMessageContaining(errorMessage);
}
@Test
public void testTypeAlteringFunctionOnAddColumn()
{
String table = createTable(format("CREATE TABLE %%s (k int PRIMARY KEY, v %s)", type));
String query = format("ALTER TABLE %s.%s ADD n %s MASKED WITH %s", KEYSPACE, table, type, mask);
if (shouldSucceed)
alterTable(query);
else
assertThatThrownBy(() -> execute(query)).hasMessageContaining(errorMessage);
}
}

284
test/unit/org/apache/cassandra/cql3/functions/masking/ColumnMaskWithUDFTest.java Normal file
View File

@ -0,0 +1,284 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.cql3.functions.masking;
import org.junit.Test;
import com.datastax.driver.core.BoundStatement;
import com.datastax.driver.core.Session;
import org.apache.cassandra.exceptions.InvalidRequestException;
import org.assertj.core.api.Assertions;
import static java.lang.String.format;
import static org.junit.Assert.assertNull;
/**
* {@link ColumnMaskTester} for column masks using a UDF.
*/
public class ColumnMaskWithUDFTest extends ColumnMaskTester
{
@Test
@SuppressWarnings("resource")
public void testUDF() throws Throwable
{
// create a table masked with a UDF and with a materialized view
String function = createAddFunction();
createTable(format("CREATE TABLE %%s (k int, c int, v int MASKED WITH %s(7), PRIMARY KEY (k, c))", function));
String view = createView("CREATE MATERIALIZED VIEW %s AS SELECT * FROM %s " +
"WHERE k IS NOT NULL AND c IS NOT NULL AND v IS NOT NULL " +
"PRIMARY KEY (v, k, c)");
// add some data
execute("INSERT INTO %s (k, c, v) VALUES (0, 0, 10)");
execute("INSERT INTO %s (k, c, v) VALUES (0, 1, 20)");
Object[][] clearRows = rows(row(0, 0, 10), row(0, 1, 20));
Object[][] maskedRows = rows(row(0, 0, 17), row(0, 1, 27));
waitForViewMutations();
Session session = sessionNet();
// query the table and verify that the column mask is applied
String tableQuery = "SELECT k, c, v FROM %s";
assertRowsNet(executeNet(tableQuery), maskedRows);
BoundStatement tablePrepared = session.prepare(formatQuery(tableQuery)).bind();
assertRowsNet(session.execute(tablePrepared), maskedRows);
// query the view and verify that the column mask is applied
String viewQuery = format("SELECT k, c, v FROM %s.%s", KEYSPACE, view);
BoundStatement viewPrepared = session.prepare(formatQuery(tableQuery)).bind();
assertRowsNet(executeNet(viewQuery), maskedRows);
assertRowsNet(session.execute(viewPrepared), maskedRows);
// drop the column mask and verify that the column is not masked anymore
alterTable("ALTER TABLE %s ALTER v DROP MASKED");
assertRowsNet(executeNet(tableQuery), clearRows);
assertRowsNet(executeNet(viewQuery), clearRows);
assertRowsNet(session.execute(tablePrepared), clearRows);
assertRowsNet(session.execute(viewPrepared), clearRows);
// mask the column again, but this time with a different argument
alterTable(format("ALTER TABLE %%s ALTER v MASKED WITH %s(8)", function));
maskedRows = rows(row(0, 0, 18), row(0, 1, 28));
assertRowsNet(executeNet(tableQuery), maskedRows);
assertRowsNet(executeNet(viewQuery), maskedRows);
assertRowsNet(session.execute(tablePrepared), maskedRows);
assertRowsNet(session.execute(viewPrepared), maskedRows);
}
@Test
public void testUDFWithReversed() throws Throwable
{
// create a table masked with a UDF and with a materialized view
String function = createAddFunction();
createTable(format("CREATE TABLE %%s (k int, c int MASKED WITH %s(100), v int, PRIMARY KEY (k, c)) " +
"WITH CLUSTERING ORDER BY (c DESC)", function));
String view = createView("CREATE MATERIALIZED VIEW %s AS SELECT * FROM %s " +
"WHERE k IS NOT NULL AND c IS NOT NULL AND v IS NOT NULL " +
"PRIMARY KEY (v, c, K) " +
"WITH CLUSTERING ORDER BY (c DESC, k ASC)");
// add some data
execute("INSERT INTO %s (k, c, v) VALUES (0, 1, 0)");
execute("INSERT INTO %s (k, c, v) VALUES (0, 2, 0)");
Object[][] clearRows = rows(row(0, 2, 0), row(0, 1, 0));
Object[][] maskedRows = rows(row(0, 102, 0), row(0, 101, 0));
waitForViewMutations();
// query the table and verify that the column mask is applied
String tableQuery = "SELECT k, c, v FROM %s";
assertRowsNet(executeNet(tableQuery), maskedRows);
// query the view and verify that the column mask is applied
String viewQuery = format("SELECT k, c, v FROM %s.%s", KEYSPACE, view);
assertRowsNet(executeNet(viewQuery), maskedRows);
// drop the column mask and verify that the column is not masked anymore
alterTable("ALTER TABLE %s ALTER c DROP MASKED");
assertRowsNet(executeNet(tableQuery), clearRows);
assertRowsNet(executeNet(viewQuery), clearRows);
// mask the column again, but this time with a different argument
alterTable(format("ALTER TABLE %%s ALTER c MASKED WITH %s(1000)", function));
maskedRows = rows(row(0, 1002, 0), row(0, 1001, 0));
assertRowsNet(executeNet(tableQuery), maskedRows);
assertRowsNet(executeNet(viewQuery), maskedRows);
}
/**
* Verifies that queries dropping a UDF that is used for masking columns are rejected.
*/
@Test
public void testDropUDFWithDependentMasks() throws Throwable
{
String function = createAddFunction();
String mask = format("MASKED WITH %s(7)", function);
String table1 = createTable(format("CREATE TABLE %%s (k int %s PRIMARY KEY, v int %<s)", mask));
String table2 = createTable(format("CREATE TABLE %%s (k int %s, c int %<s, v int %<s, s int static %<s, PRIMARY KEY (k, c))", mask));
String dropFunctionQuery = format("DROP FUNCTION %s", function);
String message = format("Function '%s' is still referenced by column masks in tables %s, %s", function, table1, table2);
assertInvalidThrowMessage(message, InvalidRequestException.class, dropFunctionQuery);
alterTable(format("ALTER TABLE %s.%s ALTER k DROP MASKED", KEYSPACE, table1));
alterTable(format("ALTER TABLE %s.%s ALTER k DROP MASKED", KEYSPACE, table2));
assertInvalidThrowMessage(message, InvalidRequestException.class, dropFunctionQuery);
alterTable(format("ALTER TABLE %s.%s ALTER v DROP MASKED", KEYSPACE, table1));
alterTable(format("ALTER TABLE %s.%s ALTER v DROP MASKED", KEYSPACE, table2));
message = format("Function '%s' is still referenced by column masks in tables %s", function, table2);
assertInvalidThrowMessage(message, InvalidRequestException.class, dropFunctionQuery);
alterTable(format("ALTER TABLE %s.%s ALTER c DROP MASKED", KEYSPACE, table2));
assertInvalidThrowMessage(message, InvalidRequestException.class, dropFunctionQuery);
alterTable(format("ALTER TABLE %s.%s ALTER s DROP MASKED", KEYSPACE, table2));
schemaChange(dropFunctionQuery);
}
@Test
public void testMissingUDF() throws Throwable
{
assertMaskingFails("Unable to find masking function for v, no declared function matches the signature %s()",
"missing_udf");
}
@Test
public void testUDFWithNoArguments() throws Throwable
{
assertMaskingFails("Invalid number of arguments in call to function %s: 0 required but 1 provided",
createFunction(KEYSPACE,
"",
"CREATE FUNCTION %s () " +
"CALLED ON NULL INPUT " +
"RETURNS int " +
"LANGUAGE java " +
"AS 'return 42;'"));
}
@Test
public void testUDFWithWrongArgumentCount() throws Throwable
{
assertMaskingFails("Invalid number of arguments in call to function %s: 2 required but 3 provided",
createAddFunction(),
"(1, 3)");
}
@Test
public void testUDFWithWrongArgumentType() throws Throwable
{
assertMaskingFails("Type error: org.apache.cassandra.db.marshal.Int32Type " +
"cannot be passed as argument 0 of function %s of type text",
createFunction(KEYSPACE,
"text",
"CREATE FUNCTION %s (a text) " +
"CALLED ON NULL INPUT " +
"RETURNS text " +
"LANGUAGE java " +
"AS 'return \"redacted\";'"));
}
@Test
public void testUDFWithWrongReturnType() throws Throwable
{
assertMaskingFails("Masking function %s() return type is text",
createFunction(KEYSPACE,
"int",
"CREATE FUNCTION %s (a int) " +
"CALLED ON NULL INPUT " +
"RETURNS text " +
"LANGUAGE java " +
"AS 'return \"redacted\";'"));
}
@Test
public void testUDFInOtherKeyspace() throws Throwable
{
assertMaskingFails("Masking function %s() doesn't belong to the same keyspace as the table",
createFunction(KEYSPACE_PER_TEST,
"int",
"CREATE FUNCTION %s (input int) " +
"CALLED ON NULL INPUT " +
"RETURNS int " +
"LANGUAGE java " +
"AS 'return Integer.valueOf(input);'"));
}
@Test
public void testUDA() throws Throwable
{
String fState = createAddFunction();
String fFinal = createIdentityFunction();
assertMaskingFails("Aggregate function %s() cannot be used for masking table columns",
createAggregate(KEYSPACE,
"int",
"CREATE AGGREGATE %s(int) " +
"SFUNC " + shortFunctionName(fState) + " " +
"STYPE int " +
"FINALFUNC " + shortFunctionName(fFinal) + " " +
"INITCOND 42"));
}
private void assertMaskingFails(String message, String function) throws Throwable
{
assertMaskingFails(message, function, "()");
}
private void assertMaskingFails(String message, String function, String arguments) throws Throwable
{
message = format(message, function);
// create table should fail
Assertions.assertThatThrownBy(() -> execute(format("CREATE TABLE %s.%s (k int PRIMARY KEY, v int MASKED WITH %s%s)",
KEYSPACE, createTableName(), function, arguments)))
.isInstanceOf(InvalidRequestException.class)
.hasMessageContaining(message);
assertNull(currentTableMetadata());
// alter table should fail
String table = createTable("CREATE TABLE %s (k int PRIMARY KEY, v int)");
Assertions.assertThatThrownBy(() -> execute(format("ALTER TABLE %s.%s ALTER v MASKED WITH %s%s",
KEYSPACE, table, function, arguments)))
.isInstanceOf(InvalidRequestException.class)
.hasMessageContaining(message);
assertColumnIsNotMasked(table, "v");
}
private String createAddFunction() throws Throwable
{
return createFunction(KEYSPACE,
"int, int",
"CREATE FUNCTION %s (a int, b int) " +
"CALLED ON NULL INPUT " +
"RETURNS int " +
"LANGUAGE java " +
"AS 'return Integer.valueOf(a) + Integer.valueOf(b);'");
}
private String createIdentityFunction() throws Throwable
{
return createFunction(KEYSPACE,
"int, int",
"CREATE FUNCTION %s (a int) " +
"CALLED ON NULL INPUT " +
"RETURNS int " +
"LANGUAGE java " +
"AS 'return a;'");
}
}

8
test/unit/org/apache/cassandra/cql3/functions/masking/MaskingFunctionTester.java
View File

@ -192,7 +192,7 @@ public abstract class MaskingFunctionTester extends CQLTester
* Tests the native masking function for the specified column type and values on all possible types of column.
* That is, when the column is part of the primary key, or a regular column, or a static column.
*
* @param type the type of the tested column
* @param type the type of the tested column
* @param values the values of the tested column
*/
private void testMaskingOnAllColumns(CQL3Type type, Object... values) throws Throwable
@ -222,7 +222,7 @@ public abstract class MaskingFunctionTester extends CQLTester
* Tests the native masking function for the specified column type and values when the column isn't part of the
* primary key. That is, when the column is either a regular column or a static column.
*
* @param type the type of the tested column
* @param type the type of the tested column
* @param values the values of the tested column
*/
private void testMaskingOnNotKeyColumns(CQL3Type type, Object... values) throws Throwable
@ -265,8 +265,8 @@ public abstract class MaskingFunctionTester extends CQLTester
* Tests the native masking function for the specified column type and value.
* This assumes that the table is already created.
*
* @param name the name of the tested column
* @param type the type of the tested column
* @param name the name of the tested column
* @param type the type of the tested column
* @param value the value of the tested column
*/
protected abstract void testMaskingOnColumn(String name, CQL3Type type, Object value) throws Throwable;

367
test/unit/org/apache/cassandra/cql3/functions/masking/SelectMaskedPermissionTest.java Normal file
View File

@ -0,0 +1,367 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.cql3.functions.masking;
import org.junit.After;
import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Test;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.cql3.CQLTester;
import org.awaitility.core.ThrowingRunnable;
import static java.lang.String.format;
/**
* Tests the {@link org.apache.cassandra.auth.Permission#SELECT_MASKED} permission.
*/
public class SelectMaskedPermissionTest extends CQLTester
{
private static final Object[] CLEAR_ROW = row(7, 7, 7, 7);
private static final String USER = "ddm_user"; // user that will have their permissions changed
private static final String PASSWORD = "ddm_password";
@BeforeClass
public static void beforeClass()
{
DatabaseDescriptor.setDynamicDataMaskingEnabled(true);
DatabaseDescriptor.setPermissionsValidity(0);
DatabaseDescriptor.setRolesValidity(0);
CQLTester.setUpClass();
requireAuthentication();
requireNetwork();
}
@Before
public void before() throws Throwable
{
useSuperUser();
createTable("CREATE TABLE %s (k int, c int, s int static, v int, PRIMARY KEY (k, c))");
executeNet("INSERT INTO %s(k, c, s, v) VALUES (?, ?, ?, ?)", CLEAR_ROW);
executeNet(format("CREATE USER IF NOT EXISTS %s WITH PASSWORD '%s'", USER, PASSWORD));
executeNet(format("GRANT SELECT ON ALL KEYSPACES TO %s", USER));
}
@After
public void after() throws Throwable
{
useSuperUser();
executeNet("DROP USER IF EXISTS " + USER);
alterTable("ALTER TABLE %s ALTER k DROP MASKED");
alterTable("ALTER TABLE %s ALTER c DROP MASKED");
alterTable("ALTER TABLE %s ALTER s DROP MASKED");
alterTable("ALTER TABLE %s ALTER v DROP MASKED");
}
@Test
public void testPartitionKeyColumn() throws Throwable
{
alterTable("ALTER TABLE %s ALTER k MASKED WITH DEFAULT");
Object[] maskedRow = row(0, 7, 7, 7);
// test queries with default permissions (no UNMASK nor SELECT_MASKED)
testPartitionKeyColumnWithDefaultPermissions(maskedRow);
// test queries with only SELECT_MASKED permission
executeNet(format("GRANT SELECT_MASKED ON ALL KEYSPACES TO %s", USER));
testPartitionKeyColumnWithOnlySelectMasked(maskedRow);
// test queries with only UNMASK permission (which includes SELECT_MASKED)
executeNet(format("REVOKE SELECT_MASKED ON ALL KEYSPACES FROM %s", USER));
executeNet(format("GRANT UNMASK ON ALL KEYSPACES TO %s", USER));
testPartitionKeyColumnWithUnmask();
// test queries with both UNMASK and SELECT_MASKED permissions
executeNet(format("GRANT UNMASK, SELECT_MASKED ON ALL KEYSPACES TO %s", USER));
testPartitionKeyColumnWithUnmask();
// test queries again without both UNMASK and SELECT_MASKED permissions
executeNet(format("REVOKE UNMASK, SELECT_MASKED ON ALL KEYSPACES FROM %s", USER));
testPartitionKeyColumnWithDefaultPermissions(maskedRow);
}
private void testPartitionKeyColumnWithDefaultPermissions(Object[] maskedRow) throws Throwable
{
assertWithUser(() -> {
assertAuthorizedQuery("SELECT * FROM %s", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE c = 7 ALLOW FILTERING", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE v = 7 ALLOW FILTERING", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE s = 7 ALLOW FILTERING", maskedRow);
assertUnauthorizedQuery("SELECT * FROM %s WHERE k = 7", "[k]");
assertUnauthorizedQuery("SELECT * FROM %s WHERE k >= 7 ALLOW FILTERING", "[k]");
assertUnauthorizedQuery("SELECT * FROM %s WHERE k = 7 AND c = 7", "[k]");
assertUnauthorizedQuery("SELECT * FROM %s WHERE k = 7 AND v = 7 ALLOW FILTERING", "[k]");
assertUnauthorizedQuery("SELECT * FROM %s WHERE token(k) = token(7)", "[k]");
assertUnauthorizedQuery("SELECT * FROM %s WHERE token(k) >= token(7)", "[k]");
});
}
private void testPartitionKeyColumnWithOnlySelectMasked(Object[] maskedRow) throws Throwable
{
assertWithUser(() -> {
assertAuthorizedQuery("SELECT * FROM %s", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE c = 7 ALLOW FILTERING", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE v = 7 ALLOW FILTERING", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE s = 7 ALLOW FILTERING", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE k = 7", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE k >= 7 ALLOW FILTERING", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE k = 7 AND c = 7", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE k = 7 AND v = 7 ALLOW FILTERING", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE token(k) = token(7)", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE token(k) >= token(7)", maskedRow);
});
}
private void testPartitionKeyColumnWithUnmask() throws Throwable
{
assertWithUser(() -> {
assertAuthorizedQuery("SELECT * FROM %s", CLEAR_ROW);
assertAuthorizedQuery("SELECT * FROM %s WHERE c = 7 ALLOW FILTERING", CLEAR_ROW);
assertAuthorizedQuery("SELECT * FROM %s WHERE v = 7 ALLOW FILTERING", CLEAR_ROW);
assertAuthorizedQuery("SELECT * FROM %s WHERE s = 7 ALLOW FILTERING", CLEAR_ROW);
assertAuthorizedQuery("SELECT * FROM %s WHERE k = 7", CLEAR_ROW);
assertAuthorizedQuery("SELECT * FROM %s WHERE k >= 7 ALLOW FILTERING", CLEAR_ROW);
assertAuthorizedQuery("SELECT * FROM %s WHERE k = 7 AND c = 7", CLEAR_ROW);
assertAuthorizedQuery("SELECT * FROM %s WHERE k = 7 AND v = 7 ALLOW FILTERING", CLEAR_ROW);
assertAuthorizedQuery("SELECT * FROM %s WHERE token(k) = token(7)", CLEAR_ROW);
assertAuthorizedQuery("SELECT * FROM %s WHERE token(k) >= token(7)", CLEAR_ROW);
});
}
@Test
public void testClusteringKeyColumn() throws Throwable
{
alterTable("ALTER TABLE %s ALTER c MASKED WITH DEFAULT");
Object[] maskedRow = row(7, 0, 7, 7);
// test queries with default permissions (no UNMASK nor SELECT_MASKED)
testClusteringKeyColumnWithDefaultPermissions(maskedRow);
// test queries with only SELECT_MASKED permission
executeNet(format("GRANT SELECT_MASKED ON ALL KEYSPACES TO %s", USER));
testClusteringKeyColumnWithOnlySelectMasked(maskedRow);
// test queries with only UNMASK permission (which includes SELECT_MASKED)
executeNet(format("REVOKE SELECT_MASKED ON ALL KEYSPACES FROM %s", USER));
executeNet(format("GRANT UNMASK ON ALL KEYSPACES TO %s", USER));
testClusteringKeyColumnWithUnmask();
// test queries with both UNMASK and SELECT_MASKED permissions
executeNet(format("GRANT UNMASK, SELECT_MASKED ON ALL KEYSPACES TO %s", USER));
testClusteringKeyColumnWithUnmask();
// test queries again without both UNMASK and SELECT_MASKED permissions
executeNet(format("REVOKE UNMASK, SELECT_MASKED ON ALL KEYSPACES FROM %s", USER));
testClusteringKeyColumnWithDefaultPermissions(maskedRow);
}
private void testClusteringKeyColumnWithDefaultPermissions(Object[] maskedRow) throws Throwable
{
assertWithUser(() -> {
assertAuthorizedQuery("SELECT * FROM %s", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE k = 7", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE v = 7 ALLOW FILTERING", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE s = 7 ALLOW FILTERING", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE token(k) = token(7)", maskedRow);
assertUnauthorizedQuery("SELECT * FROM %s WHERE c = 7 ALLOW FILTERING", "[c]");
assertUnauthorizedQuery("SELECT * FROM %s WHERE k = 7 AND c = 7", "[c]");
});
}
private void testClusteringKeyColumnWithOnlySelectMasked(Object[] maskedRow) throws Throwable
{
assertWithUser(() -> {
assertAuthorizedQuery("SELECT * FROM %s", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE k = 7", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE v = 7 ALLOW FILTERING", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE s = 7 ALLOW FILTERING", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE token(k) = token(7)", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE c = 7 ALLOW FILTERING", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE k = 7 AND c = 7", maskedRow);
});
}
private void testClusteringKeyColumnWithUnmask() throws Throwable
{
assertWithUser(() -> {
assertAuthorizedQuery("SELECT * FROM %s", CLEAR_ROW);
assertAuthorizedQuery("SELECT * FROM %s WHERE k = 7", CLEAR_ROW);
assertAuthorizedQuery("SELECT * FROM %s WHERE v = 7 ALLOW FILTERING", CLEAR_ROW);
assertAuthorizedQuery("SELECT * FROM %s WHERE s = 7 ALLOW FILTERING", CLEAR_ROW);
assertAuthorizedQuery("SELECT * FROM %s WHERE token(k) = token(7)", CLEAR_ROW);
assertAuthorizedQuery("SELECT * FROM %s WHERE c = 7 ALLOW FILTERING", CLEAR_ROW);
assertAuthorizedQuery("SELECT * FROM %s WHERE k = 7 AND c = 7", CLEAR_ROW);
});
}
@Test
public void testStaticColumn() throws Throwable
{
alterTable("ALTER TABLE %s ALTER s MASKED WITH DEFAULT");
Object[] maskedRow = row(7, 7, 0, 7);
// test queries with default permissions (no UNMASK nor SELECT_MASKED)
testStaticColumnWithDefaultPermissions(maskedRow);
// test queries with only SELECT_MASKED permission
executeNet(format("GRANT SELECT_MASKED ON ALL KEYSPACES TO %s", USER));
testStaticColumnWithOnlySelectMasked(maskedRow);
// test queries with only UNMASK permission (which includes SELECT_MASKED)
executeNet(format("REVOKE SELECT_MASKED ON ALL KEYSPACES FROM %s", USER));
executeNet(format("GRANT UNMASK ON ALL KEYSPACES TO %s", USER));
testStaticColumnWithUnmask();
// test queries with both UNMASK and SELECT_MASKED permissions
executeNet(format("GRANT UNMASK, SELECT_MASKED ON ALL KEYSPACES TO %s", USER));
testStaticColumnWithUnmask();
// test queries again without both UNMASK and SELECT_MASKED permissions
executeNet(format("REVOKE UNMASK, SELECT_MASKED ON ALL KEYSPACES FROM %s", USER));
testStaticColumnWithDefaultPermissions(maskedRow);
}
private void testStaticColumnWithDefaultPermissions(Object[] maskedRow) throws Throwable
{
assertWithUser(() -> {
assertAuthorizedQuery("SELECT * FROM %s", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE token(k) = token(7)", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE k = 7", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE k = 7 AND c = 7", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE c = 7 ALLOW FILTERING", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE v = 7 ALLOW FILTERING", maskedRow);
assertUnauthorizedQuery("SELECT * FROM %s WHERE s = 7 ALLOW FILTERING", "[s]");
});
}
private void testStaticColumnWithOnlySelectMasked(Object[] maskedRow) throws Throwable
{
assertWithUser(() -> {
assertAuthorizedQuery("SELECT * FROM %s", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE token(k) = token(7)", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE k = 7", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE k = 7 AND c = 7", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE c = 7 ALLOW FILTERING", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE s = 7 ALLOW FILTERING", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE v = 7 ALLOW FILTERING", maskedRow);
});
}
private void testStaticColumnWithUnmask() throws Throwable
{
assertWithUser(() -> {
assertAuthorizedQuery("SELECT * FROM %s", CLEAR_ROW);
assertAuthorizedQuery("SELECT * FROM %s WHERE token(k) = token(7)", CLEAR_ROW);
assertAuthorizedQuery("SELECT * FROM %s WHERE k = 7", CLEAR_ROW);
assertAuthorizedQuery("SELECT * FROM %s WHERE k = 7 AND c = 7", CLEAR_ROW);
assertAuthorizedQuery("SELECT * FROM %s WHERE c = 7 ALLOW FILTERING", CLEAR_ROW);
assertAuthorizedQuery("SELECT * FROM %s WHERE s = 7 ALLOW FILTERING", CLEAR_ROW);
assertAuthorizedQuery("SELECT * FROM %s WHERE v = 7 ALLOW FILTERING", CLEAR_ROW);
});
}
@Test
public void testCollections() throws Throwable
{
alterTable("ALTER TABLE %s ALTER v MASKED WITH DEFAULT");
Object[] maskedRow = row(7, 7, 7, 0);
// test queries with default permissions (no UNMASK nor SELECT_MASKED)
testCollectionsWithDefaultPermissions(maskedRow);
// test queries with only SELECT_MASKED permission
executeNet(format("GRANT SELECT_MASKED ON ALL KEYSPACES TO %s", USER));
testCollectionsWithOnlySelectMasked(maskedRow);
// test queries with only UNMASK permission (which includes SELECT_MASKED)
executeNet(format("REVOKE SELECT_MASKED ON ALL KEYSPACES FROM %s", USER));
executeNet(format("GRANT UNMASK ON ALL KEYSPACES TO %s", USER));
testCollectionsWithUnmask();
// test queries with both UNMASK and SELECT_MASKED permissions
executeNet(format("GRANT UNMASK, SELECT_MASKED ON ALL KEYSPACES TO %s", USER));
testCollectionsWithUnmask();
// test queries again without both UNMASK and SELECT_MASKED permissions
executeNet(format("REVOKE UNMASK, SELECT_MASKED ON ALL KEYSPACES FROM %s", USER));
testCollectionsWithDefaultPermissions(maskedRow);
}
private void testCollectionsWithDefaultPermissions(Object[] maskedRow) throws Throwable
{
assertWithUser(() -> {
assertAuthorizedQuery("SELECT * FROM %s", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE token(k) = token(7)", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE k = 7", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE k = 7 AND c = 7", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE c = 7 ALLOW FILTERING", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE s = 7 ALLOW FILTERING", maskedRow);
assertUnauthorizedQuery("SELECT * FROM %s WHERE v = 7 ALLOW FILTERING", "[v]");
});
}
private void testCollectionsWithOnlySelectMasked(Object[] maskedRow) throws Throwable
{
assertWithUser(() -> {
assertAuthorizedQuery("SELECT * FROM %s", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE token(k) = token(7)", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE k = 7", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE k = 7 AND c = 7", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE c = 7 ALLOW FILTERING", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE s = 7 ALLOW FILTERING", maskedRow);
assertAuthorizedQuery("SELECT * FROM %s WHERE v = 7 ALLOW FILTERING", maskedRow);
});
}
private void testCollectionsWithUnmask() throws Throwable
{
assertWithUser(() -> {
assertAuthorizedQuery("SELECT * FROM %s", CLEAR_ROW);
assertAuthorizedQuery("SELECT * FROM %s WHERE token(k) = token(7)", CLEAR_ROW);
assertAuthorizedQuery("SELECT * FROM %s WHERE k = 7", CLEAR_ROW);
assertAuthorizedQuery("SELECT * FROM %s WHERE k = 7 AND c = 7", CLEAR_ROW);
assertAuthorizedQuery("SELECT * FROM %s WHERE c = 7 ALLOW FILTERING", CLEAR_ROW);
assertAuthorizedQuery("SELECT * FROM %s WHERE s = 7 ALLOW FILTERING", CLEAR_ROW);
assertAuthorizedQuery("SELECT * FROM %s WHERE v = 7 ALLOW FILTERING", CLEAR_ROW);
});
}
private void assertAuthorizedQuery(String query, Object[]... rows) throws Throwable
{
assertRowsNet(executeNet(query), rows);
}
private void assertUnauthorizedQuery(String query, String unauthorizedColumns) throws Throwable
{
assertInvalidMessageNet(format("User %s has no UNMASK nor SELECT_MASKED permission on table %s.%s, " +
"cannot query masked columns %s",
USER, KEYSPACE, currentTable(), unauthorizedColumns), query);
}
private void assertWithUser(ThrowingRunnable assertion) throws Throwable
{
useUser(USER, PASSWORD);
assertion.run();
useSuperUser();
}
}

223
test/unit/org/apache/cassandra/cql3/functions/masking/UnmaskPermissionTest.java Normal file
View File

@ -0,0 +1,223 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.cql3.functions.masking;
import java.util.Arrays;
import org.junit.After;
import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Test;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.cql3.CQLTester;
import static java.lang.String.format;
/**
* Tests the {@link org.apache.cassandra.auth.Permission#UNMASK} permission.
* <p>
* The permission is tested for a regular user with the {@code UNMASK} permissions on different resources,
* while also verifying the absence of side effects on other ordinary users, superusers and internal queries.
*/
public class UnmaskPermissionTest extends CQLTester
{
private static final String CREATE_KEYSPACE = "CREATE KEYSPACE IF NOT EXISTS %s WITH replication = " +
"{'class': 'SimpleStrategy', 'replication_factor': '1'}";
private static final String CREATE_TABLE = "CREATE TABLE IF NOT EXISTS %s.%s " +
"(k int, c int, v text MASKED WITH mask_replace('redacted'), " +
"PRIMARY KEY (k, c))";
private static final String INSERT = "INSERT INTO %s.%s (k, c, v) VALUES (?, ?, ?)";
private static final String SELECT_WILDCARD = "SELECT * FROM %s.%s";
private static final String SELECT_COLUMNS = "SELECT k, c, v FROM %s.%s";
private static final Object[] CLEAR_ROW = row(0, 0, "sensitive");
private static final Object[] MASKED_ROW = row(0, 0, "redacted");
private static final String KEYSPACE_1 = "mask_keyspace_1";
private static final String KEYSPACE_2 = "mask_keyspace_2";
private static final String TABLE_1 = "mask_table_1";
private static final String TABLE_2 = "mask_table_2";
private static final String USER = "ddm_user"; // user that will have their permissions changed
private static final String OTHER_USER = "ddm_ordinary_user"; // user that won't have their permissions altered
private static final String PASSWORD = "ddm_password";
@BeforeClass
public static void beforeClass()
{
DatabaseDescriptor.setDynamicDataMaskingEnabled(true);
DatabaseDescriptor.setPermissionsValidity(0);
DatabaseDescriptor.setRolesValidity(0);
CQLTester.setUpClass();
requireAuthentication();
requireNetwork();
}
@Before
public void before() throws Throwable
{
useSuperUser();
schemaChange(format(CREATE_KEYSPACE, KEYSPACE_1));
schemaChange(format(CREATE_KEYSPACE, KEYSPACE_2));
createTable(format(CREATE_TABLE, KEYSPACE_1, TABLE_1));
createTable(format(CREATE_TABLE, KEYSPACE_1, TABLE_2));
createTable(format(CREATE_TABLE, KEYSPACE_2, TABLE_1));
execute(format(INSERT, KEYSPACE_1, TABLE_1), CLEAR_ROW);
execute(format(INSERT, KEYSPACE_1, TABLE_2), CLEAR_ROW);
execute(format(INSERT, KEYSPACE_2, TABLE_1), CLEAR_ROW);
for (String user : Arrays.asList(USER, OTHER_USER))
{
executeNet(format("CREATE USER IF NOT EXISTS %s WITH PASSWORD '%s'", user, PASSWORD));
executeNet(format("GRANT SELECT ON ALL KEYSPACES TO %s", user));
}
}
@After
public void after() throws Throwable
{
useSuperUser();
executeNet("DROP USER IF EXISTS " + USER);
}
@Test
public void testUnmaskDefaults() throws Throwable
{
// ordinary user without changed permissions should see masked data
useUser(OTHER_USER, PASSWORD);
assertMasked(KEYSPACE_1, TABLE_1);
assertMasked(KEYSPACE_1, TABLE_2);
assertMasked(KEYSPACE_2, TABLE_1);
// super user should see unmasked data
useSuperUser();
assertClear(KEYSPACE_1, TABLE_1);
assertClear(KEYSPACE_1, TABLE_2);
assertClear(KEYSPACE_2, TABLE_1);
// internal user should see unmasked data
assertClearInternal(KEYSPACE_1, TABLE_1);
assertClearInternal(KEYSPACE_1, TABLE_2);
assertClearInternal(KEYSPACE_2, TABLE_1);
}
@Test
public void testUnmaskOnAllKeyspaces() throws Throwable
{
assertPermissions(format("GRANT UNMASK ON ALL KEYSPACES TO %s", USER), () -> {
assertClear(KEYSPACE_1, TABLE_1);
assertClear(KEYSPACE_1, TABLE_2);
assertClear(KEYSPACE_2, TABLE_1);
});
assertPermissions(format("REVOKE UNMASK ON ALL KEYSPACES FROM %s", USER), () -> {
assertMasked(KEYSPACE_1, TABLE_1);
assertMasked(KEYSPACE_1, TABLE_2);
assertMasked(KEYSPACE_2, TABLE_1);
});
}
@Test
public void testUnmaskOnKeyspace() throws Throwable
{
assertPermissions(format("GRANT UNMASK ON KEYSPACE %s TO %s", KEYSPACE_1, USER), () -> {
assertClear(KEYSPACE_1, TABLE_1);
assertClear(KEYSPACE_1, TABLE_2);
assertMasked(KEYSPACE_2, TABLE_1);
});
assertPermissions(format("REVOKE UNMASK ON KEYSPACE %s FROM %s", KEYSPACE_1, USER), () -> {
assertMasked(KEYSPACE_1, TABLE_1);
assertMasked(KEYSPACE_1, TABLE_2);
assertMasked(KEYSPACE_2, TABLE_1);
});
}
@Test
public void testUnmaskOnTable() throws Throwable
{
assertPermissions(format("GRANT UNMASK ON TABLE %s.%s TO %s", KEYSPACE_1, TABLE_1, USER), () -> {
assertClear(KEYSPACE_1, TABLE_1);
assertMasked(KEYSPACE_1, TABLE_2);
assertMasked(KEYSPACE_2, TABLE_1);
});
assertPermissions(format("REVOKE UNMASK ON TABLE %s.%s FROM %s", KEYSPACE_1, TABLE_1, USER), () -> {
assertMasked(KEYSPACE_1, TABLE_1);
assertMasked(KEYSPACE_1, TABLE_2);
assertMasked(KEYSPACE_2, TABLE_1);
});
}
private void assertPermissions(String alterPermissionsQuery, ThrowingRunnable assertion) throws Throwable
{
// alter permissions as superuser
useSuperUser();
executeNet(alterPermissionsQuery);
// verify the tested user permissions
useUser(USER, PASSWORD);
assertion.run();
// the ordinary user without modified permissions should keep seeing masked data
useUser(OTHER_USER, PASSWORD);
assertMasked(KEYSPACE_1, TABLE_1);
assertMasked(KEYSPACE_1, TABLE_2);
assertMasked(KEYSPACE_2, TABLE_1);
// super user should keep seeing unmasked data
useSuperUser();
assertClear(KEYSPACE_1, TABLE_1);
assertClear(KEYSPACE_1, TABLE_2);
assertClear(KEYSPACE_2, TABLE_1);
// internal user should keep seeing unmasked data
assertClearInternal(KEYSPACE_1, TABLE_1);
assertClearInternal(KEYSPACE_1, TABLE_2);
assertClearInternal(KEYSPACE_2, TABLE_1);
}
private void assertMasked(String keyspace, String table) throws Throwable
{
assertRowsNet(executeNet(format(SELECT_WILDCARD, keyspace, table)), MASKED_ROW);
assertRowsNet(executeNet(format(SELECT_COLUMNS, keyspace, table)), MASKED_ROW);
}
private void assertClear(String keyspace, String table) throws Throwable
{
assertRowsNet(executeNet(format(SELECT_WILDCARD, keyspace, table)), CLEAR_ROW);
assertRowsNet(executeNet(format(SELECT_COLUMNS, keyspace, table)), CLEAR_ROW);
}
private void assertClearInternal(String keyspace, String table) throws Throwable
{
assertRows(execute(format(SELECT_WILDCARD, keyspace, table)), CLEAR_ROW);
assertRows(execute(format(SELECT_COLUMNS, keyspace, table)), CLEAR_ROW);
}
@FunctionalInterface
private interface ThrowingRunnable
{
void run() throws Throwable;
}
}

57
test/unit/org/apache/cassandra/cql3/statements/DescribeStatementTest.java
View File

@ -39,12 +39,14 @@ import org.apache.cassandra.locator.InetAddressAndPort;
import org.apache.cassandra.locator.TokenMetadata;
import org.apache.cassandra.schema.Schema;
import org.apache.cassandra.schema.TableId;
import org.apache.cassandra.schema.TableMetadata;
import org.apache.cassandra.service.StorageService;
import org.apache.cassandra.transport.ProtocolVersion;
import static java.lang.String.format;
import static org.apache.cassandra.schema.SchemaConstants.*;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
@ -757,6 +759,56 @@ public class DescribeStatementTest extends CQLTester
row(KEYSPACE_PER_TEST, "index", indexWithOptions, expectedIndexStmtWithOptions));
}
@Test
public void testDescribeTableWithColumnMasks() throws Throwable
{
requireNetwork();
DatabaseDescriptor.setDynamicDataMaskingEnabled(true);
String table = createTable(KEYSPACE_PER_TEST,
"CREATE TABLE %s (" +
" pk1 text, " +
" pk2 int MASKED WITH DEFAULT, " +
" ck1 int, " +
" ck2 int MASKED WITH mask_default()," +
" s1 decimal static, " +
" s2 decimal static MASKED WITH mask_null(), " +
" v1 text, " +
" v2 text MASKED WITH mask_inner(1, null), " +
"PRIMARY KEY ((pk1, pk2), ck1, ck2 ))");
TableMetadata tableMetadata = Schema.instance.getTableMetadata(KEYSPACE_PER_TEST, table);
assertNotNull(tableMetadata);
String tableCreateStatement = "CREATE TABLE " + KEYSPACE_PER_TEST + "." + table + " (\n" +
" pk1 text,\n" +
" pk2 int MASKED WITH system.mask_default(),\n" +
" ck1 int,\n" +
" ck2 int MASKED WITH system.mask_default(),\n" +
" s1 decimal static,\n" +
" s2 decimal static MASKED WITH system.mask_null(),\n" +
" v1 text,\n" +
" v2 text MASKED WITH system.mask_inner(1, null),\n" +
" PRIMARY KEY ((pk1, pk2), ck1, ck2)\n" +
") WITH ID = " + tableMetadata.id + "\n" +
" AND CLUSTERING ORDER BY (ck1 ASC, ck2 ASC)\n" +
" AND " + tableParametersCql();
assertRowsNet(executeDescribeNet("DESCRIBE TABLE " + KEYSPACE_PER_TEST + "." + table + " WITH INTERNALS"),
row(KEYSPACE_PER_TEST,
"table",
table,
tableCreateStatement));
// masks should be listed even if DDM is disabled
DatabaseDescriptor.setDynamicDataMaskingEnabled(false);
assertRowsNet(executeDescribeNet("DESCRIBE TABLE " + KEYSPACE_PER_TEST + "." + table + " WITH INTERNALS"),
row(KEYSPACE_PER_TEST,
"table",
table,
tableCreateStatement));
}
private static String allTypesTable()
{
return "CREATE TABLE test.has_all_types (\n" +
@ -918,9 +970,4 @@ public class DescribeStatementTest extends CQLTester
executeNet(v, "USE " + useKs);
return v;
}
private static Object[][] rows(Object[]... rows)
{
return rows;
}
}

3
test/unit/org/apache/cassandra/db/CellTest.java
View File

@ -76,7 +76,8 @@ public class CellTest
ColumnIdentifier.getInterned(name, false),
type,
ColumnMetadata.NO_POSITION,
ColumnMetadata.Kind.REGULAR);
ColumnMetadata.Kind.REGULAR,
null);
}
@Test

2
test/unit/org/apache/cassandra/db/ColumnsTest.java
View File

@ -496,7 +496,7 @@ public class ColumnsTest
private static ColumnMetadata def(String name, AbstractType<?> type, ColumnMetadata.Kind kind)
{
return new ColumnMetadata(TABLE_METADATA, bytes(name), type, ColumnMetadata.NO_POSITION, kind);
return new ColumnMetadata(TABLE_METADATA, bytes(name), type, ColumnMetadata.NO_POSITION, kind, null);
}
private static TableMetadata mock(Columns columns)

Some files were not shown because too many files have changed in this diff Show More