pulsar apache

[长期更新]pulsar问题列表

Posted by 梁远鹏 on 2021-05-04 | 阅读 |共949字，阅读约 2 分钟

TOC

说明

本文主要收集 apache pulsar 遇到的一些常见问题,欢迎对本文进行投稿你认为好的场景或问题或是一个实践.

升级 pulsar 客户端 sdk

从 pulsar2.10/pulsar2.11 的 sdk 升级到 pulsar 3.0.0 时由于不是全部组件都升级到统一的 3.0.0 版本,因此遇到了在调用 adminAPI 的时候报错了,应该是 jar 包冲突导致的 noSuchMethod 异常.

于是将所有组件都升级到统一的 pulsar 3.0.0,并且添加一个 pulsar admin API 的集成测试来覆盖这一点.

添加 pulsar-broker 用于测试时依赖冲突

但是遇到一个新问题是下载依赖时一直卡着:

...
Downloaded from central: https://repo.maven.apache.org/maven2/org/slf4j/slf4j-parent/2.0.5/slf4j-parent-2.0.5.pom (16 kB at 1.3 MB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/pulsar/pulsar-transaction-coordinator/3.0.0/pulsar-transaction-coordinator-3.0.0.pom
Progress (1): 4.1/4.8 kB
Progress (1): 4.8 kB    
                    
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/pulsar/pulsar-transaction-coordinator/3.0.0/pulsar-transaction-coordinator-3.0.0.pom (4.8 kB at 371 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/pulsar/pulsar-package-filesystem-storage/3.0.0/pulsar-package-filesystem-storage-3.0.0.pom
Progress (1): 2.0 kB
                    
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/pulsar/pulsar-package-filesystem-storage/3.0.0/pulsar-package-filesystem-storage-3.0.0.pom (2.0 kB at 146 kB/s)
Error: The operation was canceled.

通过添加 debug 日志最终看到是 maven 无法解决依赖冲突导致下载卡住了:

Caused by: org.eclipse.aether.collection.UnsolvableVersionConflictException: Could not resolve version conflict among [org.apache.pulsar:pulsar-broker:jar:3.1.0 -> org.apache.pulsar:managed-ledger:jar:3.1.0 -> org.apache.pulsar:pulsar-metadata:jar:3.1.0 -> io.etcd:jetcd-core:jar:0.7.5 -> io.etcd:jetcd-grpc:jar:0.7.5 -> io.grpc:grpc-core:jar:1.51.0, org.apache.pulsar:pulsar-broker:jar:3.1.0 -> org.apache.pulsar:managed-ledger:jar:3.1.0 -> org.apache.pulsar:pulsar-metadata:jar:3.1.0 -> io.etcd:jetcd-core:jar:0.7.5 -> io.etcd:jetcd-common:jar:0.7.5 -> io.grpc:grpc-core:jar:1.51.0, org.apache.pulsar:pulsar-broker:jar:3.1.0 -> org.apache.bookkeeper:stream-storage-server:jar:4.16.2 -> org.apache.bookkeeper:stream-storage-java-client:jar:4.16.2 -> io.grpc:grpc-core:jar:1.54.1, org.apache.pulsar:pulsar-broker:jar:3.1.0 -> org.apache.bookkeeper:stream-storage-server:jar:4.16.2 -> org.apache.bookkeeper:stream-storage-java-client:jar:4.16.2 -> io.grpc:grpc-grpclb:jar:1.54.1 -> io.grpc:grpc-core:jar:[1.54.1,1.54.1], org.apache.pulsar:pulsar-broker:jar:3.1.0 -> org.apache.bookkeeper:stream-storage-server:jar:4.16.2 -> org.apache.bookkeeper:stream-storage-java-client:jar:4.16.2 -> io.grpc:grpc-netty:jar:1.54.1 -> io.grpc:grpc-core:jar:[1.54.1,1.54.1], org.apache.pulsar:pulsar-broker:jar:3.1.0 -> org.apache.bookkeeper:stream-storage-server:jar:4.16.2 -> org.apache.bookkeeper:stream-storage-java-client:jar:4.16.2 -> io.grpc:grpc-rls:jar:1.54.1 -> io.grpc:grpc-core:jar:[1.54.1,1.54.1], org.apache.pulsar:pulsar-broker:jar:3.1.0 -> org.apache.bookkeeper:stream-storage-server:jar:4.16.2 -> org.apache.bookkeeper:stream-storage-java-client:jar:4.16.2 -> io.grpc:grpc-services:jar:1.54.1 -> io.grpc:grpc-core:jar:[1.54.1,1.54.1], org.apache.pulsar:pulsar-broker:jar:3.1.0 -> org.apache.bookkeeper:stream-storage-server:jar:4.16.2 -> org.apache.bookkeeper:stream-storage-java-client:jar:4.16.2 -> io.grpc:grpc-testing:jar:1.54.1 -> io.grpc:grpc-core:jar:[1.54.1,1.54.1], org.apache.pulsar:pulsar-broker:jar:3.1.0 -> org.apache.bookkeeper:stream-storage-server:jar:4.16.2 -> org.apache.bookkeeper:stream-storage-java-client:jar:4.16.2 -> io.grpc:grpc-xds:jar:1.54.1 -> io.grpc:grpc-core:jar:1.54.1, org.apache.pulsar:pulsar-broker:jar:3.1.0 -> org.apache.pulsar:pulsar-functions-worker:jar:3.1.0 -> org.apache.pulsar:pulsar-functions-runtime:jar:3.1.0 -> org.apache.pulsar:pulsar-functions-instance:jar:3.1.0 -> io.grpc:grpc-all:jar:1.45.1 -> io.grpc:grpc-core:jar:[1.45.1,1.45.1]]
    at org.eclipse.aether.util.graph.transformer.NearestVersionSelector.newFailure (NearestVersionSelector.java:129)
    at org.eclipse.aether.util.graph.transformer.NearestVersionSelector.backtrack (NearestVersionSelector.java:100)
    at org.eclipse.aether.util.graph.transformer.NearestVersionSelector.selectVersion (NearestVersionSelector.java:80)
    at org.eclipse.aether.util.graph.transformer.ConflictResolver.transformGraph (ConflictResolver.java:217)
    at org.eclipse.aether.util.graph.transformer.ChainedDependencyGraphTransformer.transformGraph (ChainedDependencyGraphTransformer.java:71)
    at org.eclipse.aether.internal.impl.collect.DependencyCollectorDelegate.collectDependencies (DependencyCollectorDelegate.java:246)
    at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.collectDependencies (DefaultDependencyCollector.java:87)
    at org.eclipse.aether.internal.impl.DefaultRepositorySystem.collectDependencies (DefaultRepositorySystem.java:305)
    at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve (DefaultProjectDependenciesResolver.java:151)
    at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies (LifecycleDependencyResolver.java:224)
    at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies (LifecycleDependencyResolver.java:136)
    at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved (MojoExecutor.java:355)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:313)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
[ERROR] 
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

最后是在依赖中排除了 pulsar-broker 中的 grpc 依赖解决了问题.

<dependency>
            <groupId>org.apache.pulsar</groupId>
            <artifactId>pulsar-broker</artifactId>
            <version>${pulsar.version}</version>
            <scope>test</scope>
            <exclusions>
                <exclusion>
                    <groupId>io.grpc</groupId>
                    <artifactId>grpc-all</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>io.grpc</groupId>
                    <artifactId>grpc-core</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>io.grpc</groupId>
                    <artifactId>grpc-testing</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>io.grpc</groupId>
                    <artifactId>grpc-auth</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

在这个问题卡着的时候考虑过使用 testcontainers 来代替,也就是每次跑集成测试时直接下载一个 pulsar 容器来启动,也能达到效果.

但是最终还是采用引入 pulsar-broker 依赖的方式在 java 代码中启动一个 pulsar 本地服务来完成测试需求,主要是考虑以下方面:

  • 项目中封装的这个组件主要是和 pulsar 打交道,没有业务代码,因此组件内专注于 pulsar 的内容,通过本地启动 pulsar 服务也能更好的理解 pulsar.
  • testcontainers 直接启动一个 pulsar 对于我这个集成测试需求来说可能太重了,因为一个 pulsar 容器镜像就需要 1.+ G 的大小,并且强依赖 docker 服务

有序的线程池

OrderedExecutor 是一个在 Pulsar 内部被广泛使用的一个线程池,每个 ledger 都在对应的线程上处理,不过使用不当会出现线程卡死导致相同顺序的数据无法得到线程从而导致线程池出现问题.

在我司物联网的业务场景中同样使用了 OrderedExecutor,每一个传感器使用一个与传感器 ID 对应的线程,也是出现过由于线程没有正确结束业务导致线程池整体出现问题.

熟悉 OrderedExecutor 对理解 Pulsar 源码会有很大帮助.

微信公众号

扫描下面的二维码关注我们的微信公众号,第一时间查看最新内容。同时也可以关注我的Github,看看我都在了解什么技术,在页面底部可以找到我的Github。

wechat-qrcode
扫码下面二维码加我微信 :)

开源项目推荐

  • replacer

    K8S webhook的参考实现|Replace container image of kubernetes mutate webhook.

    Language: Golang Forks: * Stars: * Updated: Nov 20, 2021

相关文章