Cassandra timeout during SIMPLE write query at consistency LOCAL

I am using spark-sql 3.0.2 with spark-cassandra-connector_2.12:3.1.0 along with java8 in my spark-project.

While writing to cassandra table I am facing below error

latest exception was
Cassandra timeout during SIMPLE write query at consistency LOCAL_ONE (1 replica were required but only 0 acknowledged the write)

Please check the executor logs for more exceptions and information at com.datastax.spark.connector.writer.AsyncStatementWriter.$anonfun$close$2(TableWriter.scala:282) at scala.Option.map(Option.scala:230) at com.datastax.spark.connector.writer.AsyncStatementWriter.close(TableWriter.scala:277) at com.datastax.spark.connector.datasource.CassandraDriverDataWriter.commit(CasssandraDriverDataWriterFactory.scala:46) at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.$anonfun$run$7(WriteToDataSourceV2Exec.scala:450) at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1411) at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:477) at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:385) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at org.apache.spark.scheduler.Task.run(Task.scala:127) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:462) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:465) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Suppressed: java.io.IOException: Failed to write statements to kspace.table_cassandra

I am using below configuration

concurrentWrites: 2056 #1024 #64
outputBatchSizeBytes: 2048
outputBatchGroupingKey: partition
outputConsistencyLevel: LOCAL_ONE
outputBatchGroupingBufferSize: 3000
outputThroughput_mb_per_sec: 1024 #256
connectionKeep_alive_ms: 30000
readTimeout_ms: 600000
ignoreNulls: true
readConsistencyLevel: LOCAL_ONE

CodePudding user response：

This usually happens because Spark Cassandra Connector is trying to write data as fast as possible, and often it overloads individual Cassandra nodes, so they are becoming not responsive (maybe because of Java GC, compaction, etc.).

You can tune write configuration parameters, like, decreasing spark.cassandra.output.concurrent.writes, spark.cassandra.output.throughputMBPerSec, etc., but very often it could be fixed by not using LOCAL_ONE consistency level (spark.cassandra.output.consistency.level), but TWO, or LOCAL_QUORUM.

Your configuration is basically pushed nodes to the limit

CodePudding user response：

Write timeouts happen because your cluster is under-sized and cannot keep up with writes.

You will need to throttle the load from your Spark app as Alex Ott advised so it doesn't overload your cluster. Alternatively, you should consider increasing the capacity of your cluster by adding more nodes. Cheers!