{"id":945,"date":"2020-01-24T07:25:14","date_gmt":"2020-01-24T06:25:14","guid":{"rendered":"http:\/\/www.klaushaller.net\/?page_id=945"},"modified":"2020-01-24T10:02:09","modified_gmt":"2020-01-24T09:02:09","slug":"part-2-kafka-fault-tolerance-using-replica","status":"publish","type":"page","link":"https:\/\/www.klaushaller.net\/?page_id=945","title":{"rendered":"Apache Kafka Tutorial, Part 2"},"content":{"rendered":"\n

Part 2: Kafka Fault\nTolerance using Replica <\/h1>\n\n\n\n

If you replicate events on multiple Kafka servers, you still have all the data when one of the server crashes. The other server stores the data as well and is ready to deliver it to the consumers. This is the idea behind replica in Kafka. Note, however, that the overall platform is blocked if there is no  Zookeeper up and running. <\/p>\n\n\n\n

The concept\nis relatively simple for replica: There is one leader. It is the only one\naccepting new messages and events from producers and it is the only one\ndelivering events and messages to consumers. Replica servers \u201cjust\u201d get a copy\nof the events and messages from the leader.<\/p>\n\n\n\n

To try out this functionality by ourselves, we extend our infrastructure by increasing the number of Kafka nodes from one to three and adding a new topic using the Kafka replica feature as shown in the figure below.<\/p>\n\n\n\n

\"\"
Figure 2: Kafka platform with topic replication <\/figcaption><\/figure>\n\n\n\n

2.1 Configuring Additional\nKafka Nodes<\/h2>\n\n\n\n

We copy the file server.properties<\/code> twice to get two new files, server1.properties<\/code> and server2.properties<\/code>. In these files, we set the broker id parameter to 1 and 2. Also, we have to configure new log file directories for each of them, since we run all of them on our local machine in this tutorial.<\/p>\n\n\n\n

So, the file server.properties<\/code> should contain:<\/p>\n\n\n\n

# The id of the broker. This must be set to a unique integer for each broker.\nbroker.id=0\n[\u2026]\n# listeners=PLAINTEXT:\/\/:9092\n[\u2026]\n# A comma separated list of directories under which to store log files\nlog.dirs=C:\\Users\\post\\Kafka\\kafka_2.13-2.4.0\\logs_server0<\/code><\/pre>\n\n\n\n
The file server1.properties<\/code> should contain:<\/p>\n\n\n\n
# The id of the broker. This must be set to a unique integer for each broker.\nbroker.id=1\n[\u2026]\nlisteners=PLAINTEXT:\/\/:9093\n[\u2026]\n# returned from java.net.InetAddress.getCanonicalHostName().\nadvertised.listeners=PLAINTEXT:\/\/localhost:9093\n[\u2026]\n# A comma separated list of directories under which to store log files\nlog.dirs=C:\\Users\\post\\Kafka\\kafka_2.13-2.4.0\\logs_server1<\/code><\/pre>\n\n\n\n
The file server2.properties<\/code> should contain:<\/p>\n\n\n\n
# The id of the broker. This must be set to a unique integer for each broker.\nbroker.id=2\n[\u2026]\nlisteners=PLAINTEXT:\/\/:9094\n[\u2026]\n# returned from java.net.InetAddress.getCanonicalHostName().\nadvertised.listeners=PLAINTEXT:\/\/localhost:9094\n[\u2026]\n# A comma separated list of directories under which to store log files\nlog.dirs=C:\\Users\\post\\Kafka\\kafka_2.13-2.4.0\\logs_server2\n<\/code><\/pre>\n\n\n\n
2.2 Start the Additional\nKafka Nodes<\/h2>\n\n\n\n
The next\nsteps are straight-forward:<\/p>\n\n\n\n
  1. Open a new command shell.<\/li>
  2. Start a Kafka server using configuration file server1.properties (see Part 1 of the tutorial if you cannot remember the command).<\/li>
  3. If everything went well, you should see the following: INFO [KafkaServer id=1] started (kafka.server.KafkaServer)<\/code><\/li>
  4. Open another new command shell.<\/li>
  5. Start a Kafka server using configuration file server2.properties (see Part 1 of the tutorial if you cannot remember the command).<\/li>
  6. If everything went well, you should see the following: INFO [KafkaServer id=2] started (kafka.server.KafkaServer)<\/code><\/li><\/ol>\n\n\n\n
    At this moment, you should have four open command shell windows: one for the Zookeeper and three Kafka server nodes. All of them must be up and running to continue.<\/p>\n\n\n\n
    2.3 Create New Topic with\nReplication<\/h2>\n\n\n\n
    Now the\nsystem is set-up so that we can create a new topic myReplicaChannel, this time using the replica\nfeatures. We want to have one leader and two replica. We open another new\ncommand shell and execute the following command:<\/p>\n\n\n\n
    kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic myReplicaChannel<\/code><\/p>\n\n\n\n
    Do you\nremember in which folder you have to execute this command?<\/p>\n\n\n\n
    Now, we\ncreate a new producer and consumer in two separate, new command shell windows\nusing these two commands:<\/p>\n\n\n\n
    kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic myReplicaChannel<\/code><\/p>\n\n\n\n
    and<\/p>\n\n\n\n
    kafka-console-producer.bat --broker-list localhost:9092 --topic myReplicaChannel<\/code><\/p>\n\n\n\n
    In the producer\ncommand shell, we type in \u201cReplication is great!\u201d and press the return key. Directly\nafterwards, we see this sentence in the consumer command shell window. So, what\ndoes this mean?<\/p>\n\n\n\n
    The implication is simple: There is no change for users respectively producers and consumers if we use the replica feature.<\/p>\n\n\n\n
    2.4 Checking the Kafka\nSystem State <\/h2>\n\n\n\n
    Using the command line interface, we get an overview of the overall system state (in a later part of the tutorial, we look on more production-like monitoring solutions). More precise, we can use the Kafka topics command with the list option:<\/p>\n\n\n\n
    kafka-topics.bat --list --zookeeper localhost:2181<\/code><\/p>\n\n\n\n
    As a result, we get a list with our two topics: myFirstChannel<\/code>and myReplicaChannel<\/code>.<\/p>\n\n\n\n
    Let\u2019s\nassume we do not only have one or two topics and \u201cjust\u201d three Kafka nodes, but\n50 topics and 15 nodes. We might be interested in understanding which server is\ninvolved in which topic \u2013 and how the actual configuration looks like. Did we\nspecify 3 replica, or did we not consider this because we were experimenting with\nthis topic for a while and forgot about this issue? <\/p>\n\n\n\n
    Here, we\ncan use the topics command, this time with the \u201cdescribe\u201d option:<\/p>\n\n\n\n
    kafka-topics.bat --describe --zookeeper localhost:2181 --topic myReplicaChannel<\/code><\/p>\n\n\n\n
    Before continue reading, you might want to compare the result for the myReplicaChannel <\/code>with the one for the myFirstChannel<\/code>. When we submit the command above, we get this result:<\/p>\n\n\n\n
    Topic: myReplicaChannel    PartitionCount: 1       ReplicationFactor: 3    Configs:<\/code><\/p>\n\n\n\n
    Topic: myReplicaChannel    Partition: 0    Leader: 0       Replicas: 0,2,1 Isr: 0,2,1<\/code><\/p>\n\n\n\n
    So, what does this mean? <\/p>\n\n\n\n