Kafka cluster is the group of brokers that are hosting the data in various topics and their partitions. Producer is not part of the cluster. It is a client of the cluster, and produces messages on to the topics.
Application development is about writing producers and consumers. Kafka cluster is what you have as your great start to work against. You have a big system of all kinds of information events swimming around in the kafka cluster, neatly organized in topics and partitions. Now, it is on you to derive some value out of it by meaningful interactions.
A producer would have to have the connectivity details of the cluster to be able to interact with the cluster, and then the topic name where it wants to write to.In java, KafkaProducer is the class that abstracts a kafka producer. Once it is able to connect to the cluster, and point to the topic, next thing the producer needs is the ability to represent the event that it wants to write.
Producer Record
This is the abstraction for holding the key value pair that the producer would like to send to the topic. ProducerRecord<K,V> is a generic class and allows you to specify the type for the key K, and the type for the value V, and has multiple overloaded methods to create a record against a specific topic, and a specific partition, and more variants there of
ProducerRecord(String topic, Integer partition, K key, V value)
Creates a record to be sent to a specified topic and partition
Producers and partitioning
A topic can be partitioned. The basic abstraction of an event as a key value pair implies that when key is specified, the corresponding values would all need to sit together in it’s own partition. When you are tracking a package that you ordered, you want to see what is happening to it as it is coming to you. If it is Christmas, and you have ordered multiple gifts for your family, you would want to be able to track these individually. Producer enables this. It is the producer that puts Boban as the key on the tracking events of the first gift, and Molly as the key on the tracking events of the second gift. Partitioning is producer’s business. If you like your events well organized, write producer code well.
