Skip to main content

Sink Connector

files.dir​

The directory to write files that have been processed. This directory must exist and be writable by the user running Kafka Connect. The connector will automatically create subdirectories for builds, chunks, locked and merged. The completed files will be in the merged subdriectory. The other three subdirectories are self-managed by the connector. If the source and sink connectors are running on the same machine (for testing) then ensure that the files.dir property is not set to the same directory for both connectors.

  • Importance: HIGH
  • Type: STRING
warning

The Sink connector does not delete Merged Files from the files.dir. Removal of files from this directory must be down by the downstream process that uses the merged file.

topics​

The topic to consume data from a paired File Chunk Source Connector. Multiple consumers (sink connectors) configured to consume from the same topic is supported. The topic can have one, or multiple partitions. The topic should be created manually prior to startup of the source or sink connectors. A schema registry is not required, as all file-chunk events are serialized as bytestream.

  • Importance: HIGH
  • Type: STRING
  • Default Value: none
halt.on.error​

Stop all tasks if an error is encountered while processing file merges

  • Importance: HIGH
  • Type: BOOLEAN
  • Default: true
topic.partitions​

(Deprecated in release 2.6): the topic.partitions used by the uploader is now passed to the downloader in the message header. (Deprecated in release 2.6): the binary.chunk.size.bytes used by the uploader is now passed to the downloader in the message header.