文档名称 文档密级

2017-4-12 华为保密信息,未经授权禁止扩散 第 1页, 共 18页

Bitnami Kafka for Huawei Enterprise

Cloud

Description

Apache Kafka is publish-subscribe messaging rethought as a distributed

commit log.

How to start or stop the services?

Each Bitnami stack includes a control script that lets you easily stop, start and

restart services. The script is located at /opt/bitnami/ctlscript.sh. Call it without

any service name arguments to start all services:

sudo /opt/bitnami/ctlscript.sh start

Or use it to restart a single service, such as Apache only, by passing the

service name as argument:

sudo /opt/bitnami/ctlscript.sh restart apache

Use this script to stop all services:

sudo /opt/bitnami/ctlscript.sh stop

Restart the services by running the script without any arguments:

sudo /opt/bitnami/ctlscript.sh restart

Obtain a list of available services and operations by running the script without

any arguments:

sudo /opt/bitnami/ctlscript.sh

How to upload files to the server with SFTP?

NOTE: Bitnami applications can be found in /opt/bitnami/apps.

文档名称 文档密级

2017-4-12 华为保密信息,未经授权禁止扩散 第 2页, 共 18页

When uploading files via SFTP, you must have the server's IP address,

username and password or, if you chose to deploy the server with an SSH

key instead of a password, you must have the server username and

corresponding SSH private key in .ppk format (for Windows) or in .pem format

(for Linux and Mac OS X).

To obtain the IP address for your server, follow these steps:

Log in to the Huawei Cloud Server Console.

Navigate to the "Computing -> Elastic Cloud Server" section.

Note the public IP address for the server you wish to connect to.

Once you have confirmed that you have the IP address and correct SSH

credentials for your server, follow the instructions below depending on the

platform you wish to use.

Although you can use any SFTP/SCP client to transfer files to your server,

this guide documents FileZilla (Windows, Linux and Mac OS X), WinSCP

(Windows) and Cyberduck (Mac OS X).

Using an SSH Key

Once you have your server's SSH key, choose your preferred application and

follow the steps below to connect to the server using SFTP.

FileZilla

IMPORTANT: To use FileZilla, your server private key should be in PPK

format.

Follow these steps:

Download and install FileZilla.

文档名称 文档密级

2017-4-12 华为保密信息,未经授权禁止扩散 第 3页, 共 18页

Launch FileZilla and use the "Edit -> Settings" command to bring up

FileZilla's configuration settings.

Within the "Connection -> SFTP" section, use the "Add keyfile" command

to select the private key file for the server. FileZilla will use this private key

to log in to the server.

Use the "File -> Site Manager -> New Site" command to bring up the

FileZilla Site Manager, where you can set up a connection to your server.

Enter your server host name and specify bitnami as the user name.

文档名称 文档密级

2017-4-12 华为保密信息,未经授权禁止扩散 第 4页, 共 18页

Select "SFTP" as the protocol and "Ask for password" as the logon type.

Use the "Connect" button to connect to the server and begin an SFTP

session. You might need to accept the server key, by clicking "Yes" or

"OK" to proceed.

You should now be logged into the /home/bitnami directory on the server. You

can now transfer files by dragging and dropping them from the local server

window to the remote server window.

If you have problems accessing your server, get extra information by use the

"Edit -> Settings -> Debug" menu to activate FileZilla's debug log.

文档名称 文档密级

2017-4-12 华为保密信息,未经授权禁止扩散 第 5页, 共 18页

WinSCP

IMPORTANT: To use WinSCP, your server private key should be in PPK

format.

Follow these steps:

Download and install WinSCP.

Launch WinSCP and in the "Session" panel, select "SFTP" as the file

protocol.

Enter your server host name and specify bitnami as the user name.

文档名称 文档密级

2017-4-12 华为保密信息,未经授权禁止扩散 第 6页, 共 18页

Click the "Advanced…" button and within the "SSH -> Authentication ->

Authentication parameters" section, select the private key file for the

server. WinSCP will use this private key to log in to the server.

From the "Session" panel, use the "Login" button to connect to the server

and begin an SCP session.

You should now be logged into the /home/bitnami directory on the server. You

can now transfer files by dragging and dropping them from the local server

window to the remote server window.

If you need to upload files to a location where the bitnami user doesn't have

write permissions, you have two options:

Once you have configured WinSCP as described above, click the

"Advanced…" button and within the "Environment -> Shell" panel, select

sudo su - as your shell. This will allow you to upload files using the

文档名称 文档密级

2017-4-12 华为保密信息,未经授权禁止扩散 第 7页, 共 18页

administrator account.

Upload the files to the /home/bitnami directory as usual. Then, connect via

SSH and move the files to the desired location with the sudo command, as

shown below:

sudo mv /home/bitnami/uploaded-file /path/to/desired/location/

Cyberduck

IMPORTANT: To use Cyberduck, your server private key should be in PEM

format.

Follow these steps:

文档名称 文档密级

2017-4-12 华为保密信息,未经授权禁止扩散 第 8页, 共 18页

Select the "Open Connection" command and specify "SFTP" as the

connection protocol.

In the connection details panel, under the "More Options" section, enable

the "Use Public Key Authentication" option and specify the path to the

private key file for the server.

Use the "Connect" button to connect to the server and begin an SFTP

session.

You should now be logged into the /home/bitnami directory on the server. You

can now transfer files by dragging and dropping them from the local server

window to the remote server window.

文档名称 文档密级

2017-4-12 华为保密信息,未经授权禁止扩散 第 9页, 共 18页

Using a Password

Once you have your server's SSH credentials, choose your preferred

application and follow the steps below to connect to the server using SFTP.

FileZilla

Follow these steps:

Download and install FileZilla.

Launch FileZilla and use the "File -> Site Manager -> New Site" command

to bring up the FileZilla Site Manager, where you can set up a connection

to your server.

Enter your server host name.

Select "SFTP" as the protocol and "Ask for password" as the logon type.

Use bitnami as the server username and the password generated during

the server deployment process.

Use the "Connect" button to connect to the server and begin an SFTP

session. You might need to accept the server key, by clicking "Yes" or

"OK" to proceed.

文档名称 文档密级

2017-4-12 华为保密信息,未经授权禁止扩散 第 10页, 共 18页

You should now be logged into the /home/bitnami directory on the server. You

can now transfer files by dragging and dropping them from the local server

window to the remote server window.

If you have problems accessing your server, get extra information by use the

"Edit -> Settings -> Debug" menu to activate FileZilla's debug log.

WinSCP

Follow these steps:

Download and install WinSCP.

Launch WinSCP and in the "Session" panel, select "SFTP" as the file

protocol.

文档名称 文档密级

2017-4-12 华为保密信息,未经授权禁止扩散 第 11页, 共 18页

Enter your server host name and set bitnami as the server username.

From the "Session" panel, use the "Login" button to connect to the server

and begin an SCP session. Enter the password when prompted.

You should now be logged into the /home/bitnami directory on the server. You

can now transfer files by dragging and dropping them from the local server

window to the remote server window.

Cyberduck

Follow these steps:

文档名称 文档密级

2017-4-12 华为保密信息,未经授权禁止扩散 第 12页, 共 18页

Select the "Open Connection" command and specify "SFTP" as the

connection protocol.

In the connection details panel, enter the server IP address, bitnami as the

username, and the password generated during the deployment process.

文档名称 文档密级

2017-4-12 华为保密信息,未经授权禁止扩散 第 13页, 共 18页

Use the "Connect" button to connect to the server and begin an SFTP

session.

You should now be logged into the /home/bitnami directory on the server. You

can now transfer files by dragging and dropping them from the local server

window to the remote server window.

What is the default configuration?

By default, the Zookeeper server runs on port 2181 and the Kafka server has

a single broker running on port 9092. Only conections from localhost are

permitted.

How to connect to Kafka from a different

machine?

To connect to Kafka and Zookeeper from a different machine, you must open

ports 9092 and 2181 for remote access. Refer to the FAQ for more information

on this.

Once you have added the firewall rule and opened the ports, perform these

additional steps:

Edit your Zookeeper configuration file

(/opt/bitnami/zookeeper/conf/zoo.cfg) and comment out the following line:

clientPortAddress=localhost

Edit your Kafka configuration file

(/opt/bitnami/kafka/config/server.properties). If necessary, uncomment the

following line and change the value of the parameter to the public IP

address of the server:

#advertised.host.name=<hostname routable by clients>

Restart the server to reload the configuration files.

sudo /opt/bitnami/ctlscript.sh restart

文档名称 文档密级

2017-4-12 华为保密信息,未经授权禁止扩散 第 14页, 共 18页

How to run a Kafka producer and consumer

from the server itself?

You can run the following example to publish and collect your first message:

Declare a new topic. The Kafka server is configured to use the server's

public IP address:

/opt/bitnami/kafka/bin/kafka-topics.sh --create --zookeeper 127.0.0.

1:2181 --replication-factor 1 --partitions 1 --topic test

We use –replication-factor to indicate how many servers are going to

have a copy of the logs, and –partitions to choose the number of

partitions for the topic we are creating.

Start a new producer on the same Kafka server and generates a message

in the topic. Remember to replace SERVER-IP with your server's public IP

address. Enter CTRL-D to end the message.

/opt/bitnami/kafka/bin/kafka-console-producer.sh --broker-list SERVE

R-IP:9092 --topic test

this is my first message

Collect and display the first message in the consumer:

/opt/bitnami/kafka/bin/kafka-console-consumer.sh --zookeeper 127.0.

0.1:2181 --topic test --from-beginning

How to create a Kafka multi-broker cluster?

This section describes the creation of a multi-broker Kafka cluster with

brokers located on different hosts. In this scenario:

One server hosts the Zookeeper server and a Kafka broker

The second server hosts a a second Kafka broker

The third server hosts a producer and a consumer

文档名称 文档密级

2017-4-12 华为保密信息,未经授权禁止扩散 第 15页, 共 18页

NOTE: Before beginning, ensure that ports 2181 (Zookeeper) and 9092

(Kafka) are open on the first server and port 9092 (Kafka) is open on the

second server. Also ensure that remote connections are possible between

the three servers (instructions).

Configuring the first server (Zookeeper manager and Kafka

broker)

The default configuration may be used as is. However, you must perform the

steps below:

Delete the contents of the Zookeeper and Kafka temporary directories

sudo rm -rf /opt/bitnami/kafka/tmp/kafka-logs

sudo rm -rf /opt/bitnami/zookeeper/tmp/zookeeper

Restart the Kafka and Zookeeper services.

sudo /opt/bitnami/ctlscript.sh restart kafka

sudo /opt/bitnami/ctlscript.sh restart zookeeper

Configuring the second server (Kafka broker)

Edit the /opt/bitnami/kafka/config/server.properties configuration file and

update the broker.id parameter.

文档名称 文档密级

2017-4-12 华为保密信息,未经授权禁止扩散 第 16页, 共 18页

broker.id = 1

This broker id must be unique in the Kafka ecosystem.

In the same file, update the zookeeper.connect parameter to reflect the

public IP address of the first server.

zookeeper.connect=PUBLIC_IP_ADDRESS_OF_ZOOKEEPER_MANAGER:2181

Delete the contents of the Zookeeper and Kafka temporary directories

sudo rm -rf /opt/bitnami/kafka/tmp/kafka-logs

sudo rm -rf /opt/bitnami/zookeeper/tmp/zookeeper

Stop the Zookeeper service.

sudo /opt/bitnami/ctlscript.sh stop zookeeper

Restart the Kafka service.

sudo /opt/bitnami/ctlscript.sh restart kafka

Configuring the third server (Kafka message

producer/consumer)

Edit the /opt/bitnami/kafka/config/producer.properties file and update the

metadata.broker.list parameter with the public IP addresses of the two

brokers:

metadata.broker.list=PUBLIC_IP_ADDRESS_OF_FIRST_KAFKA_BROKER:9092,

PUBLIC_IP_ADDRESS_OF_SECOND_KAFKA_BROKER:9092

Edit the /opt/bitnami/kafka/config/consumer.properties file and update the

zookeeper.connect parameter to reflect the public IP address of the first

server.

zookeeper.connect=PUBLIC_IP_ADDRESS_OF_ZOOKEEPER_MANAGER:2181

Since this host only serves as a producer and a consumer, stop the Kafka

and Zookeeper services:

文档名称 文档密级

2017-4-12 华为保密信息,未经授权禁止扩散 第 17页, 共 18页

sudo /opt/bitnami/ctlscript.sh stop kafka

sudo /opt/bitnami/ctlscript.sh stop zookeeper

Testing the cluster

NOTE: The following commands should be executed on the third server

(Kafka message producer/consumer).

Create a new topic.

/opt/bitnami/kafka/bin/kafka-topics.sh --create --zookeeper PUBLIC_I

P_ADDRESS_OF_FIRST_KAFKA_BROKER:2181 --replication-factor 2 --partit

ions 1 --topic multiBroker

Produce some messages by running the command below and then

entering some messages, each on a separate line. Enter Ctrl-C to end.

/opt/bitnami/kafka/bin/kafka-console-producer.sh --broker-list PUBLI

C_IP_ADDRESS_OF_FIRST_KAFKA_BROKER:9092 --topic multiBroker

this is a message

this is another message

^C

Consume the messages. The consumer will connect to the cluster and

retrieve and display the messages you entered in the previous step.

/opt/bitnami/kafka/bin/kafka-console-consumer.sh --zookeeper PUBLIC_

IP_ADDRESS_OF_FIRST_KAFKA_BROKER:2181 --topic multiBroker --from-beg

inning

this is a message

this is another message

^C

How to debug Kafka and Zookeeper errors?

The main Kafka log file is created at /opt/bitnami/kafka/logs/server.log.

文档名称 文档密级

2017-4-12 华为保密信息,未经授权禁止扩散 第 18页, 共 18页

The main Zookeeper log file is created at

/opt/bitnami/zookeeper/tmp/zookeeper.out.