Add io.netty.tryReflectionSetAccessible=true to j11 server options in order to enable netty to use Unsafe direct byte buffer construction

Patch by Sam Tunnicliffe; reviewed by Brandon Williams, Ekaterina Dimitrova and Berenguer Blasi for CASSANDRA-16493

CHANGES.txt

@@ -1,4 +1,5 @@
 4.0-beta5
+ * Add io.netty.tryReflectionSetAccessible=true to j11 server options in order to enable netty to use Unsafe direct byte buffer construction (CASSANDRA-16493)
  * Make cassandra-stress -node support host:port notation (CASSANDRA-16529)
  * Better handle legacy gossip application states during (and after) upgrades (CASSANDRA-16525)
  * Mark StreamingMetrics.ActiveOutboundStreams as deprecated (CASSANDRA-11174)

build.xml

@@ -213,6 +213,9 @@
       make tests stable on low resource environments, but in java 11 we need to make it unlimited
       (don't define MaxMetaspaceSize) and disable class unloading in CMS outside of a
       stop-the-world pause.
+
+      In java 11 we also need to set a system property to enable netty to use Unsafe direct byte
+      buffer construction (see CASSANDRA-16493)
     -->
     <resources id="_jvm8_test_arg_items">
       <!-- TODO see CASSANDRA-16212 - we seem to OOM non stop now after CASSANDRA-16212, so to have clean CI while this gets looked into, disabling limiting metaspace
@@ -224,6 +227,7 @@
     <pathconvert property="_jvm8_test_arg_items_concat" refid="_jvm8_test_arg_items" pathsep=" "/>
     <resources id="_jvm11_test_arg_items">
         <string>-XX:-CMSClassUnloadingEnabled</string>
+        <string>-Dio.netty.tryReflectionSetAccessible=true</string>
     </resources>
     <pathconvert property="_jvm11_test_arg_items_concat" refid="_jvm11_test_arg_items" pathsep=" "/>
     <condition property="test-jvmargs" value="${_jvm11_test_arg_items_concat}" else="${_jvm8_test_arg_items_concat}">

conf/jvm11-server.options

@@ -93,4 +93,11 @@
 # -XX:+PrintPromotionFailure            maps to 'promotion' with level 'trace'
 # -XX:PrintFLSStatistics=1              maps to 'freelist' with level 'trace'
 
+### Netty Options
+
+# On Java >= 9 Netty requires the io.netty.tryReflectionSetAccessible system property to be set to true to enable
+# creation of direct buffers using Unsafe. Without it, this falls back to ByteBuffer.allocateDirect which has
+# inferior performance and risks exceeding MaxDirectMemory
+-Dio.netty.tryReflectionSetAccessible=true
+
 # The newline in the end of file is intentional