cassandra/.claude/skills/cassandra-injvm-dtest/references/classloader_guide.md

Classloader Deep Dive

Comprehensive guide to understanding and debugging classloader issues in the Cassandra in-JVM dtest framework.

Classloader Architecture

The Three ClassLoader Layers

The in-JVM dtest framework uses a three-tier classloader hierarchy:

┌─────────────────────────────────────────┐
│     Test ClassLoader (Your Tests)      │
│  - JUnit test code                      │
│  - Test utilities                       │
│  - Direct Cluster/Instance access      │
└─────────────────────────────────────────┘
                 ↓
┌─────────────────────────────────────────┐
│      Shared ClassLoader                 │
│  - Primitives (String, Integer, etc.)   │
│  - Config objects (@Shared)             │
│  - API interfaces (ICoordinator, etc.)  │
│  - Serializable contracts               │
└─────────────────────────────────────────┘
                 ↓
┌─────────────────────────────────────────┐
│   Instance ClassLoaders (Per Node)      │
│  - DatabaseDescriptor                   │
│  - StorageService                       │
│  - All Cassandra internal classes       │
│  - Logging configuration                │
│  - Each instance gets own copy          │
└─────────────────────────────────────────┘

What Gets Loaded Where

Shared ClassLoader (available to all):

• Java primitives: String, Integer, Long, etc.
• Config objects: InstanceConfig, IInstanceConfig
• API interfaces: ICoordinator, IInstance, IInvokableInstance
• ByteBuffer and related classes
• Classes marked with @Shared annotation
• Serialization infrastructure

Instance ClassLoader (per-instance isolation):

• DatabaseDescriptor (each instance has own static state)
• StorageService (separate singleton per instance)
• All Cassandra core classes (schema, storage, etc.)
• Logging configuration (logback per instance)
• Classes marked with @Isolated annotation
• Everything else in org.apache.cassandra.*

Why This Matters

Each instance needs its own isolated copy of Cassandra's static state:

• DatabaseDescriptor holds configuration → each node needs different config
• StorageService is a singleton → each node needs separate instance
• Schema metadata is static → each node may have different schema versions

Code Transfer Mechanism

How Lambdas Cross Boundaries

When you call runOnInstance(() -> { ... }):

1. Serialization (Test ClassLoader):

   Lambda → Java Serialization → byte[]
   

2. Transfer (Across Boundary):

   byte[] passed to IsolatedExecutor
   

3. Deserialization (Instance ClassLoader):

   byte[] → Java Deserialization → Lambda (in instance classloader)
   

4. Execution (Instance Context):

   Lambda runs with access to instance's Cassandra classes
   

What Can Be Captured

Safe to Capture (automatically serialized):

// Primitives
final int value = 42;
final String name = "test";
final long timestamp = System.currentTimeMillis();

cluster.get(1).runOnInstance(() -> {
    System.out.println(value);    // OK
    System.out.println(name);     // OK
    System.out.println(timestamp); // OK
});

Unsafe to Capture (serialization will fail):

// Non-serializable objects from test
FileOutputStream fos = new FileOutputStream("test.txt");
AtomicInteger counter = new AtomicInteger(0);
DatabaseDescriptor descriptor = ...;  // From test classloader

cluster.get(1).runOnInstance(() -> {
    fos.write(...);          // FAIL: NotSerializableException
    counter.incrementAndGet(); // FAIL: NotSerializableException
    descriptor.get...();     // FAIL: Even if serializable, wrong classloader
});

Common Issues and Solutions

Issue 1: ClassCastException

Symptom:

java.lang.ClassCastException: org.apache.cassandra.dht.Murmur3Partitioner$LongToken cannot be cast to org.apache.cassandra.dht.Token

Cause: Object created in test classloader, trying to use in instance classloader. Even though the classes have the same name, they're loaded by different classloaders and are considered different types.

Wrong:

// Create token in test classloader
Token token = new Murmur3Partitioner.LongToken(0);

// Try to use in instance
cluster.get(1).runOnInstance(() -> {
    DatabaseDescriptor.setPartitionerUnsafe(token); // FAIL: ClassCastException
});

Right:

// Create token inside instance classloader
cluster.get(1).runOnInstance(() -> {
    Token token = new Murmur3Partitioner.LongToken(0);
    DatabaseDescriptor.setPartitionerUnsafe(token); // OK
});

Why This Happens:

Test CL: Token.class  → Object ID 0x1234
Instance CL: Token.class → Object ID 0x5678

0x1234 cannot be cast to 0x5678 even if identical bytecode

Issue 2: NotSerializableException

Symptom:

java.io.NotSerializableException: MyCustomClass

Cause: Lambda captured non-serializable object from test scope.

Wrong:

List<String> names = new ArrayList<>();  // ArrayList is Serializable
Thread thread = new Thread(() -> {});    // Thread is NOT Serializable

cluster.get(1).runOnInstance(() -> {
    names.add("test");  // OK - ArrayList is Serializable
    thread.start();     // FAIL - Thread is not Serializable
});

Right:

// Only capture serializable primitives
final String threadName = "test-thread";

cluster.get(1).runOnInstance(() -> {
    Thread thread = new Thread(() -> {}, threadName);  // Create inside
    thread.start();  // OK
});

Issue 3: Static State Not Shared

Symptom: Static fields modified in one instance don't affect others.

Cause: Each instance has separate copy of static state.

Wrong Assumption:

// Set on instance 1
cluster.get(1).runOnInstance(() -> {
    DatabaseDescriptor.setPartitioner(...);
});

// Expect to see on instance 2
cluster.get(2).runOnInstance(() -> {
    // DatabaseDescriptor is DIFFERENT instance here
    Partitioner p = DatabaseDescriptor.getPartitioner();  // Won't see change from node 1
});

Right Approach:

// Configure each instance separately
cluster.stream().forEach(instance -> {
    instance.runOnInstance(() -> {
        DatabaseDescriptor.setPartitioner(...);
    });
});

Issue 4: ClassNotFoundException

Symptom:

java.lang.ClassNotFoundException: com.mycompany.TestUtils

Cause: Class from test classpath not available in instance classloader.

Wrong:

// TestUtils only in test classpath
import com.mycompany.TestUtils;

cluster.get(1).runOnInstance(() -> {
    TestUtils.doSomething();  // FAIL: ClassNotFoundException
});

Right:

// Either:
// 1. Move TestUtils to Cassandra dependencies
// 2. Mark TestUtils as @Shared
// 3. Use only Cassandra classes inside runOnInstance

cluster.get(1).runOnInstance(() -> {
    // Use only Cassandra classes
    StorageService.instance.method();  // OK
});

Issue 5: Lambda Capture Confusion

Symptom: Unexpected behavior or null values inside lambda.

Cause: Captured variables must be effectively final.

Wrong:

int counter = 0;

for (int i = 0; i < 3; i++) {
    cluster.get(i + 1).runOnInstance(() -> {
        System.out.println(counter);  // FAIL: counter must be final
    });
    counter++;
}

Right:

for (int i = 0; i < 3; i++) {
    final int nodeId = i + 1;  // Effectively final
    cluster.get(nodeId).runOnInstance(() -> {
        System.out.println("Node: " + nodeId);  // OK
    });
}

Debugging Techniques

Technique 1: Identify ClassLoader

Print which classloader loaded a class:

cluster.get(1).runOnInstance(() -> {
    System.out.println("Current thread CL: " +
        Thread.currentThread().getContextClassLoader());

    System.out.println("Token CL: " +
        Token.class.getClassLoader());

    System.out.println("DatabaseDescriptor CL: " +
        DatabaseDescriptor.class.getClassLoader());
});

Expected output:

Current thread CL: IsolatedClassLoader@abc123
Token CL: IsolatedClassLoader@abc123
DatabaseDescriptor CL: IsolatedClassLoader@abc123

Technique 2: Verify Serialization

Test if object can be serialized before using in lambda:

public static boolean isSerializable(Object obj) {
    try {
        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        ObjectOutputStream oos = new ObjectOutputStream(baos);
        oos.writeObject(obj);
        oos.close();
        return true;
    } catch (NotSerializableException e) {
        System.out.println("NOT serializable: " + e.getMessage());
        return false;
    } catch (IOException e) {
        throw new RuntimeException(e);
    }
}

// Usage
MyObject obj = new MyObject();
if (isSerializable(obj)) {
    // Safe to capture in lambda
    final MyObject capturedObj = obj;
    cluster.get(1).runOnInstance(() -> {
        // Use capturedObj
    });
} else {
    // Must create inside runOnInstance
}

Technique 3: Explicit Transfer

Use IsolatedExecutor.transfer() to explicitly move objects:

import org.apache.cassandra.distributed.impl.IsolatedExecutor;

// Create in test classloader
MySerializableObject obj = new MySerializableObject();

// Explicitly transfer to instance classloader
cluster.get(1).runOnInstance(() -> {
    MySerializableObject transferred = (MySerializableObject)
        IsolatedExecutor.transferAdhoc(obj, Thread.currentThread().getContextClassLoader());

    // Now 'transferred' is in correct classloader
    useObject(transferred);
});

Technique 4: Inspect Lambda Captures

Use reflection to see what a lambda captured:

import java.lang.reflect.Field;

Consumer<Object> lambda = (obj) -> {
    System.out.println(obj);
};

// Inspect captured fields
for (Field field : lambda.getClass().getDeclaredFields()) {
    field.setAccessible(true);
    System.out.println("Captured: " + field.getName() +
                      " = " + field.get(lambda));
}

Technique 5: Compare Class Identity

Check if two class references are actually the same:

Class<?> testToken = Token.class;  // From test classloader

cluster.get(1).runOnInstance(() -> {
    Class<?> instanceToken = Token.class;  // From instance classloader

    // These will be DIFFERENT
    System.out.println("Test CL Token: " + System.identityHashCode(testToken));
    System.out.println("Instance CL Token: " + System.identityHashCode(instanceToken));
    System.out.println("Same class? " + (testToken == instanceToken));  // false!
});

Annotations: @Shared and @Isolated

@Shared Annotation

Mark classes to be loaded in shared classloader:

import org.apache.cassandra.utils.Shared;

@Shared
public class MySharedData implements Serializable {
    private String value;

    // Can be used across classloader boundaries
    public MySharedData(String value) {
        this.value = value;
    }
}

Use @Shared for:

• Config/DTO classes
• Data transfer objects
• Immutable value classes
• Things that need to cross boundaries

@Isolated Annotation

Mark classes to be loaded per-instance:

import org.apache.cassandra.utils.Isolated;

@Isolated
public class MyInstanceState {
    private static MyInstanceState instance;

    // Each Cassandra instance gets own copy
    public static MyInstanceState getInstance() {
        if (instance == null)
            instance = new MyInstanceState();
        return instance;
    }
}

Use @Isolated for:

• Stateful singletons
• Classes with static state
• Instance-specific configuration

Best Practices

1. Minimize Lambda Captures

Bad:

List<String> results = new ArrayList<>();
Map<String, Integer> counts = new HashMap<>();
AtomicInteger total = new AtomicInteger();

cluster.get(1).runOnInstance(() -> {
    // Captures 3 objects
    results.add("test");
    counts.put("key", 1);
    total.incrementAndGet();
});

Good:

cluster.get(1).runOnInstance(() -> {
    // Create everything inside
    List<String> results = new ArrayList<>();
    results.add("test");
    // Process locally
});

2. Use Return Values

Bad:

AtomicInteger result = new AtomicInteger();

cluster.get(1).runOnInstance(() -> {
    int count = countSomething();
    result.set(count);  // Won't work - different classloader
});

Good:

int result = cluster.get(1).callOnInstance(() -> {
    return countSomething();  // Return primitive
});

3. Create Objects in Target ClassLoader

Bad:

Token token = new Murmur3Partitioner.LongToken(0);

cluster.get(1).runOnInstance(() -> {
    useToken(token);  // ClassCastException
});

Good:

final long tokenValue = 0;

cluster.get(1).runOnInstance(() -> {
    Token token = new Murmur3Partitioner.LongToken(tokenValue);
    useToken(token);  // OK
});

4. Use Primitives for Communication

Bad:

MyComplexObject obj = new MyComplexObject(...);

cluster.get(1).runOnInstance(() -> {
    process(obj);  // May fail serialization
});

Good:

final int id = obj.getId();
final String name = obj.getName();

cluster.get(1).runOnInstance(() -> {
    MyComplexObject obj = new MyComplexObject(id, name);
    process(obj);  // OK - created in instance classloader
});

5. Coordinate via CQL

Bad:

// Try to share objects between instances
Object sharedState = cluster.get(1).callOnInstance(() -> {
    return getState();
});

cluster.get(2).runOnInstance(() -> {
    setState(sharedState);  // ClassCastException
});

Good:

// Share via CQL
cluster.coordinator(1).execute(
    "INSERT INTO shared.state (id, value) VALUES (?, ?)",
    ConsistencyLevel.QUORUM, 1, "state-value");

// Read on other instance
Object[][] rows = cluster.coordinator(2).execute(
    "SELECT value FROM shared.state WHERE id = ?",
    ConsistencyLevel.QUORUM, 1);

Advanced: Understanding IsolatedExecutor

Serialization Process

// In test classloader
SerializableCallable<String> lambda = () -> "Hello";

// IsolatedExecutor.transfer():
byte[] serialized = serializeOneObject(lambda);  // Test CL → bytes

// Transfer to instance classloader
Object deserialized = deserializeOneObject(serialized);  // bytes → Instance CL

// Now deserialized is in instance classloader

Custom Transfer

For complex scenarios:

// Serialize in test classloader
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(baos);
oos.writeObject(myObject);
byte[] bytes = baos.toByteArray();

// Transfer and deserialize in instance classloader
cluster.get(1).runOnInstance(() -> {
    ByteArrayInputStream bais = new ByteArrayInputStream(bytes);
    ObjectInputStream ois = new ObjectInputStream(bais) {
        @Override
        protected Class<?> resolveClass(ObjectStreamClass desc)
                throws IOException, ClassNotFoundException {
            // Force resolution in instance classloader
            return Class.forName(desc.getName(), false,
                               Thread.currentThread().getContextClassLoader());
        }
    };

    Object obj = ois.readObject();
    // obj is now in instance classloader
});

Troubleshooting Checklist

When encountering classloader issues:

1. Check what you're capturing

   • Is it a primitive? → OK
   • Is it Serializable? → Maybe OK
   • Is it a Cassandra class? → Create inside runOnInstance

2. Verify serialization

   • Use isSerializable() helper
   • Check for non-serializable fields

3. Identify the classloader

   • Print classloader info
   • Compare class identity

4. Simplify the lambda

   • Remove captures one by one
   • Create objects inside

5. Use primitives

   • Extract primitive values
   • Pass only primitives

6. Return, don't modify

   • Return results from callOnInstance
   • Don't try to modify captured objects