Kafka源码环境搭建
安装Java|Scala|Gradle
首先确定系统的JDK版本至少是8。如果不是,那么就需要更换至相应的版本。可以通过java -version
来查看当前系统的JDK版本。
java -version
java version "1.8.0_261"
Java(TM) SE Runtime Environment (build 1.8.0_261-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.261-b12, mixed mode)
下载Scala安装包并进行解压,下载地址为:https://www.scala-lang.org/download/2.13.5.html。
wget https://downloads.lightbend.com/scala/2.13.5/scala-2.13.5.tgz
tar -zxvf scala-2.13.5.tgz
mv scala-2.13.5 /Users/luowei/devtools/scala-2.13.5
然后将$SCALA_HOME/bin
的路径添加到环境变量PATH中, vim ~/.bash_profile
:
export SCALA_HOME=/Users/luowei/devtools/scala-2.13.5
export PATH=${PATH}:$SCALA_HOME/bin
".bash_profile" 8L, 365B
使环境变量生效
source ~/.bash_profile
然后关闭终端重新开启一个,使用scala -version
来验证是否配置完成
scala -version
Scala code runner version 2.13.5 -- Copyright 2002-2020, LAMP/EPFL and Lightbend, Inc.
同样方式,下载并安装Gradle,然后将$GRADLE_HOME/bin
的路径添加到环境变量PATH中。然后使用gradle -v
来验证是否配置完成
gradle -v
------------------------------------------------------------
Gradle 7.4
------------------------------------------------------------
Build time: 2022-02-08 09:58:38 UTC
Revision: f0d9291c04b90b59445041eaa75b2ee744162586
Kotlin: 1.5.31
Groovy: 3.0.9
Ant: Apache Ant(TM) version 1.10.11 compiled on July 10 2021
JVM: 1.8.0_261 (Oracle Corporation 25.261-b12)
OS: Mac OS X 10.16 x86_64
下载Kafka源码
-
从地址https://github.com/apache/kafka/archive/refs/tags/2.8.2.zip下载kafka-2.8.2.zip,然后解压缩并修改目录为Kafka-2.8.2
-
进入目录,修改gradle.properties文件,这里主要修改其中scalaVersion的值,使其与之前安装的scala版本保持一致。
group=org.apache.kafka # NOTE: When you change this version number, you should also make sure to update # the version numbers in # - docs/js/templateData.js # - tests/kafkatest/__init__.py # - tests/kafkatest/version.py (variable DEV_VERSION) # - kafka-merge-pr.py version=2.5.0 scalaVersion=2.13.5 task=build org.gradle.jvmargs=-Xmx1024m -Xss2m ~ "gradle.properties" 26L, 1131B
-
接着采用命令
gradlew idea
将Kafka源码工程导入到IDEA中。$ ./gradlew idea #如果你用的是Eclipse,执行./gradlew eclipse即可
接着,你需要打开 IDEA,选择“打开工程”,然后再选择 kafka 目录即可。至此,我们就在 IDEA 中搭建了 Kafka 源码环境。
运行Kafka服务程序
-
添加启动项,Edit Configurations->点击+号->Application添加一个启动项,然后填写如下内容
-
拷贝log4j.properties至core目录下
-
启动并运行,出现slf4j找不到的错误信息
11:33:57: Executing ':core:Kafka.main()'... > Configure project : Building project 'core' with Scala version 2.13.5 Building project 'streams-scala' with Scala version 2.13.5 > Task :raft:processResources UP-TO-DATE > Task :raft:determineCommitId UP-TO-DATE > Task :core:processResources UP-TO-DATE > Task :raft:createVersionFile > Task :generator:compileJava UP-TO-DATE > Task :generator:processResources NO-SOURCE > Task :generator:classes UP-TO-DATE > Task :metadata:processResources UP-TO-DATE > Task :clients:processResources UP-TO-DATE > Task :raft:processMessages MessageGenerator: processed 1 Kafka message JSON files(s). > Task :core:processMessages MessageGenerator: processed 6 Kafka message JSON files(s). > Task :metadata:processMessages MessageGenerator: processed 14 Kafka message JSON files(s). > Task :clients:processMessages MessageGenerator: processed 136 Kafka message JSON files(s). > Task :clients:compileJava UP-TO-DATE > Task :clients:classes UP-TO-DATE > Task :clients:determineCommitId UP-TO-DATE > Task :clients:createVersionFile > Task :clients:jar UP-TO-DATE > Task :metadata:compileJava UP-TO-DATE > Task :metadata:classes UP-TO-DATE > Task :metadata:jar UP-TO-DATE > Task :raft:compileJava UP-TO-DATE > Task :raft:classes UP-TO-DATE > Task :raft:jar UP-TO-DATE > Task :core:compileJava UP-TO-DATE > Task :core:compileScala UP-TO-DATE > Task :core:classes UP-TO-DATE > Task :core:Kafka.main() log4j:WARN No appenders could be found for logger (kafka.utils.Log4jControllerRegistration$). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
-
先查看项目可引用的slf4j包
-
在
build.gradle
的子项目core
中需要添加这两个依赖:project(':core') { println "Building project 'core' with Scala version ${versions.scala}" ... dependencies { ... // 添加以下两个依赖 // https://mvnrepository.com/artifact/org.slf4j/slf4j-api compile group: 'org.slf4j', name: 'slf4j-api', version: '1.7.30' // https://mvnrepository.com/artifact/org.slf4j/slf4j-log4j12 compile group: 'org.slf4j', name: 'slf4j-log4j12', version: '1.7.30' }
首先要添加slf4j的依赖,还要添加基于slf4j的log4j12的依赖。注意这里一定要用
compile
而不是testCompile
,因为后者的意思是只在test环境下编译. -
先启动zookeeper服务
-
如此便可顺利地运行kafka服务,部分日志如下:
[2023-02-10 13:30:53,519] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$) [2023-02-10 13:30:53,904] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util) [2023-02-10 13:30:54,011] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler) [2023-02-10 13:30:54,015] INFO starting (kafka.server.KafkaServer) [2023-02-10 13:30:54,015] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer) [2023-02-10 13:30:54,037] INFO [ZooKeeperClient Kafka server] Initializing a new session to localhost:2181. (kafka.zookeeper.ZooKeeperClient) [2023-02-10 13:30:54,044] INFO Client environment:zookeeper.version=3.5.9-83df9301aa5c2a5d284a9940177808c01bc35cef, built on 01/06/2021 20:03 GMT (org.apache.zookeeper.ZooKeeper) [2023-02-10 13:30:54,044] INFO Client environment:host.name=localhost (org.apache.zookeeper.ZooKeeper) [2023-02-10 13:30:54,044] INFO Client environment:java.version=1.8.0_261 (org.apache.zookeeper.ZooKeeper) [2023-02-10 13:30:54,044] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper) [2023-02-10 13:30:54,044] INFO Client environment:java.home=/Library/Java/JavaVirtualMachines/jdk1.8.0_261.jdk/Contents/Home/jre (org.apache.zookeeper.ZooKeeper) [2023-02-10 13:30:54,044] INFO Client environment:java.class.path=/Users/luowei/opensourceProjects/kafka-2.8.2/core/build/classes/java/main:/Users/luowei/opensourceProjects/kafka-2.8.2/core/build/classes/scala/main:/Users/luowei/opensourceProjects/kafka-2.8.2/core/build/resources/main:/Users/luowei/opensourceProjects/kafka-2.8.2/raft/build/libs/kafka-raft-2.8.2.jar:/Users/luowei/opensourceProjects/kafka-2.8.2/metadata/build/libs/kafka-metadata-2.8.2.jar:/Users/luowei/opensourceProjects/kafka-2.8.2/clients/build/libs/kafka-clients-2.8.2.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/net.sourceforge.argparse4j/argparse4j/0.7.0/6f0621d0c3888de39e0f06d01f37ba53a798e657/argparse4j-0.7.0.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.module/jackson-module-scala_2.13/2.10.5/f8e550e7b1a937e805e9b3c8ce16aeb16bc56a66/jackson-module-scala_2.13-2.10.5.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-csv/2.10.5/2fdba33036a74540f59ec21f956a3a5427e1c9db/jackson-dataformat-csv-2.10.5.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.datatype/jackson-datatype-jdk8/2.10.5/2e145b4eeccdfe0efc8f4aa38b80d493465f7064/jackson-datatype-jdk8-2.10.5.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.module/jackson-module-paranamer/2.10.5/3907141093e7a5523becedba1dd27f7448caf0c/jackson-module-paranamer-2.10.5.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.10.5.1/7ff756c3af1fe95cb3cddba9158fc3289ca06387/jackson-databind-2.10.5.1.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/net.sf.jopt-simple/jopt-simple/5.0.4/4fdac2fbe92dfad86aa6e9301736f6b4342a3f5c/jopt-simple-5.0.4.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/com.yammer.metrics/metrics-core/2.2.0/f82c035cfa786d3cbec362c38c22a5f5b1bc8724/metrics-core-2.2.0.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/org.scala-lang.modules/scala-collection-compat_2.13/2.3.0/c992f4285e6d3a9375a8c41788db4bef5649c777/scala-collection-compat_2.13-2.3.0.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/org.scala-lang.modules/scala-java8-compat_2.13/0.9.1/970d8d65f42a76c2fad104ea7f50e8f1daf38b8/scala-java8-compat_2.13-0.9.1.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/com.typesafe.scala-logging/scala-logging_2.13/3.9.2/6986314976f55419819ca7ae9f9d077ba070fe42/scala-logging_2.13-3.9.2.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/org.scala-lang/scala-reflect/2.13.5/8dee4d8c5374920f2db2f5f55d9dd6ecd4194cc/scala-reflect-2.13.5.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/org.scala-lang/scala-library/2.13.5/d0df6e78b6ccb0102cb0395bfa8cb806e9e81b61/scala-library-2.13.5.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/org.apache.zookeeper/zookeeper/3.5.9/3b0e8eaff97fce87075c02fd764e4f2d04996f31/zookeeper-3.5.9.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-log4j12/1.7.30/c21f55139d8141d2231214fb1feaf50a1edca95e/slf4j-log4j12-1.7.30.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.30/b5a4b6d16ab13e34a88fae84c35cd5d68cac922c/slf4j-api-1.7.30.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/com.github.luben/zstd-jni/1.4.9-1/7dc25611870925d32752c5b45d7d23ae01702f43/zstd-jni-1.4.9-1.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/org.lz4/lz4-java/1.7.1/c4d931ef8ad2c9c35d65b231a33e61428472d0da/lz4-java-1.7.1.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/org.xerial.snappy/snappy-java/1.1.8.1/d30aaf4d41d6ff0c760c8931d3b8dafc0293c91a/snappy-java-1.1.8.1.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.10.5/33298de8da86f92f8ccd61ced214d3b16f8c531e/jackson-annotations-2.10.5.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.10.5/db2ba27938de7f2d478a97d6abcdaa17cbbd3cea/jackson-core-2.10.5.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/com.thoughtworks.paranamer/paranamer/2.8/619eba74c19ccf1da8ebec97a2d7f8ba05773dd6/paranamer-2.8.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/org.apache.zookeeper/zookeeper-jute/3.5.9/9f14e67a10bfe1ff02347c02116ab4ca03febb6f/zookeeper-jute-3.5.9.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/org.apache.yetus/audience-annotations/0.5.0/55762d3191a8d6610ef46d11e8cb70c7667342a3/audience-annotations-0.5.0.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.73.Final/1a2231c0074f88254865c3769a4b5842939ea04d/netty-handler-4.1.73.Final.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-epoll/4.1.73.Final/763cf37f7475764113ba2764f08dbadd5356d23b/netty-transport-native-epoll-4.1.73.Final.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.73.Final/9496a30a349863a4c6fa10d5c36b4f3b495d3a31/netty-codec-4.1.73.Final.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-classes-epoll/4.1.73.Final/2a5682520fc756efc921da78b0f00c35196b6708/netty-transport-classes-epoll-4.1.73.Final.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-unix-common/4.1.73.Final/4701063d36f390e02da6da85c13e32a0e78349d2/netty-transport-native-unix-common-4.1.73.Final.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.73.Final/abb155ddff196ccedfe85b810d4b9375ef85fcfa/netty-transport-4.1.73.Final.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.73.Final/bfe83710f0c1739019613e81a06101020ca65def/netty-resolver-4.1.73.Final.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.73.Final/244a569c9aae973f6f485ac9801d79c1eca36daa/netty-buffer-4.1.73.Final.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.73.Final/27731b58d741b6faa6a00fa3285e7a55cc47be01/netty-common-4.1.73.Final.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-classes/2.0.46.Final/9937a832d9c19861822d345b48ced388b645aa5f/netty-tcnative-classes-2.0.46.Final.jar:/Users/luowei/.gradle/caches/modules-2/files-2.1/log4j/log4j/1.2.17/5af35056b4d257e4b64b9e8069c0746e8b08629f/log4j-1.2.17.jar (org.apache.zookeeper.ZooKeeper) [2023-02-10 13:30:54,045] INFO Client environment:java.library.path=/Users/luowei/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:. (org.apache.zookeeper.ZooKeeper) [2023-02-10 13:30:54,045] INFO Client environment:java.io.tmpdir=/var/folders/08/ksqlltqj43b454gls77qbbzc0000gn/T/ (org.apache.zookeeper.ZooKeeper) [2023-02-10 13:30:54,045] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper) [2023-02-10 13:30:54,045] INFO Client environment:os.name=Mac OS X (org.apache.zookeeper.ZooKeeper) [2023-02-10 13:30:54,045] INFO Client environment:os.arch=x86_64 (org.apache.zookeeper.ZooKeeper) [2023-02-10 13:30:54,045] INFO Client environment:os.version=10.16 (org.apache.zookeeper.ZooKeeper) [2023-02-10 13:30:54,045] INFO Client environment:user.name=luowei (org.apache.zookeeper.ZooKeeper) [2023-02-10 13:30:54,045] INFO Client environment:user.home=/Users/luowei (org.apache.zookeeper.ZooKeeper) [2023-02-10 13:30:54,045] INFO Client environment:user.dir=/Users/luowei/opensourceProjects/kafka-2.8.2 (org.apache.zookeeper.ZooKeeper) [2023-02-10 13:30:54,045] INFO Client environment:os.memory.free=198MB (org.apache.zookeeper.ZooKeeper) [2023-02-10 13:30:54,045] INFO Client environment:os.memory.max=3641MB (org.apache.zookeeper.ZooKeeper) [2023-02-10 13:30:54,045] INFO Client environment:os.memory.total=245MB (org.apache.zookeeper.ZooKeeper) [2023-02-10 13:30:54,048] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=18000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@5a5a729f (org.apache.zookeeper.ZooKeeper) [2023-02-10 13:30:54,053] INFO jute.maxbuffer value is 4194304 Bytes (org.apache.zookeeper.ClientCnxnSocket) [2023-02-10 13:30:54,058] INFO zookeeper.request.timeout value is 0. feature enabled= (org.apache.zookeeper.ClientCnxn)
小结
至此,kafka就通过IDEA运行起来了,在整个环境搭建过程中一定要注意kafka和java、scala、gradle等对应关系。很多时候,在搭建过程中会遇到各种错误,比如scala版本不一致或者slf4j相关错误。