Release Notes
NEW - version 2.8 (Oct 2024)
Source & Sink: Repackaging of docker containers from cp-connect to OneCricketeer/apache-kafka-connect-docker] to reduce the container to 330MB (from 2500MB) Source & Sink: docker: fixed a premature exit if the curl POST DELETE return=7 becuase the job does not exist size from 2500MB to ~330MB Source: Bug fix for short filenames (for example "a") where the filename may also exist inside the file path causing "java.io.FileNotFoundException" when deleting a .PROCESSING file.
version 2.7 (Sept 2024)
Bug fix for file ordering: (rarely) chunk files are uploaded out of order; causing merge failure (for single-task)
version 2.6 (June-2024)
Sink connector configuration property "binary.chunk.size.bytes" has been deprecated. The binary chunk size bytes used by the source connector is included in the Kafka header, which is read by the sink connector. Sink connector configuration property "topic.partitions" has been deprecated. The topic.partitions used by the uploader is included in the Kafka header, which is read by the sink connector.
Docker support: a containerized uploader and downloader are now available on Docker Hub. These containers use a Kafka Connect Standalone server to run the source or sink connector. Multiple uploader containers streaming to one Kafka Cluster can be started - ensure that the input.file.pattern and/or files.dir do not overlap. Multiple downloader containers can be started; streaming from the same Kafka Cluster - remember that each container will download all chunked files so they not be writing streamed files to the same destination filesystem.
version 2.5 (24-June-2024)
Demo Video https://www.youtube.com/@Streamsend-dp4cg/videos
Plugin packaging has changed to "streamsend": this changes the "connector.class"
New source connector configuration properties "finished.file.retention.mins" and "error.file.retention.mins" which automate cleanup of uploaded files after the number of minutes specified
New Source & Sink Connector configuration properties "topic.partitions" to inform the connectors of the partition count to distribute messages to. This configuration property should be set to the partition count of the topic, and it should be set for both the source and the sink connector.
New - version 2.4 (03-May-2024)
New Source Connector configuration property "files.dir" (deprecates properties "input.path:, "error.path" and "finished.path") New Sink Connector configuration property "files.dir" (deprecates property "output.path") Source: new configuration properties "finished.file.retention.mins" & "error.file.retention.mins" for automated retention & cleanup of uploaded files
The file-chunk connector plugins are now available on Confluent Hub. To install the connectors, run
confluent hub install markteehan/file-chunk-source:latest
confluent hub install markteehan/file-chunk-sink:latest