Troubleshooting

The Uploader or Downloader fails to process large files with OutOfMemoryError. This can occur when generating the MD5 checksum if the file size exceeds free memory in the Kafka Connect JVM. To turn off MD5 checksum checks for large files, configure file.maximum.size.bytes.for.md5 to a value that is lower than the free memory in the JVM; on the Uploader (or on the Uploader and Downloader).

The Downloader (sink connector) starts but does not process topic data while the Uploader (source connector) sends data This can occur if the source connector is mis-configured with the wrong value serializer. The connector must use ByteArrayConverter; not JsonConverter; because it must be able to stream any file format; including binary files. This applies for both the uploader and downloader. Ensure these that the value.converter and key.converter configuration properties are set:

value.converter=org.apache.kafka.connect.converters.ByteArrayConverter
key.converter=org.apache.kafka.connect.converters.ByteArrayConverter

You may also need to add these; depending on your config: key.converter.schemas.enable=false value.converter.schemas.enable=false value.converter.schema.registry.url=http://localhost:8081 key.converter.schema.registry.url=http://localhost:8081

These will apply to all connectors running on the server, so if it is not possible to modify Kafka connect worker properties, then modify them in the submit JSON instead; for both the downloader and uploader jobs.

Add these to the connector JSON:

"value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter"
"key.converter": "org.apache.kafka.connect.converters.ByteArrayConverter"

As above; it may also be necessary to add the key.converter.schemas.enable etc to the JSON.

org.apache.kafka.common.errors.RecordTooLargeException: The request included a message larger than the max message size the server will accept. The value specified for binary.chunk.size.bytes exceeds the maximum message size (message.max.bytes) for this Kafka cluster. Specify a smaller value.

Reflections scanner could not find any classes for URLs or Scanner SubTypesScanner was not configured There is an unexpected file in the plugins directory. Do not place the plugins "zipfile" in the plugins directory - instead unzip it, so that the plugins directory contains the subdirectories markteehan-file-chunk-source-2.x (containing lib, etc and doc) and similar for the sink.