Debezium connector is broken in 2.3.0

[aahmed-se]

Steps to reproduce

docker run -it -p 6650:6650 -p 8080:8080 apachepulsar/pulsar-all:2.3.0 /pulsar/bin/pulsar standalone

Error Log

00:06:01.073 [main] WARN  org.apache.pulsar.functions.utils.io.ConnectorUtils - Failed to load connector from /pulsar/./connectors/pulsar-io-debezium-2.3.0.nar
java.nio.file.NoSuchFileException: /tmp/pulsar-nar/pulsar-io-debezium-2.3.0.nar-unpacked/META-INF/services/pulsar-io.yaml
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) ~[?:1.8.0_181]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_181]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_181]
	at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[?:1.8.0_181]
	at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_181]
	at java.nio.file.Files.newByteChannel(Files.java:407) ~[?:1.8.0_181]
	at java.nio.file.Files.readAllBytes(Files.java:3152) ~[?:1.8.0_181]
	at org.apache.pulsar.common.nar.NarClassLoader.getServiceDefinition(NarClassLoader.java:198) ~[org.apache.pulsar-pulsar-common-2.3.0.jar:2.3.0]
	at org.apache.pulsar.functions.utils.io.ConnectorUtils.getConnectorDefinition(ConnectorUtils.java:103) ~[org.apache.pulsar-pulsar-functions-utils-2.3.0.jar:2.3.0]
	at org.apache.pulsar.functions.utils.io.ConnectorUtils.searchForConnectors(ConnectorUtils.java:123) [org.apache.pulsar-pulsar-functions-utils-2.3.0.jar:2.3.0]
	at org.apache.pulsar.functions.worker.ConnectorsManager.<init>(ConnectorsManager.java:34) [org.apache.pulsar-pulsar-functions-worker-2.3.0.jar:2.3.0]
	at org.apache.pulsar.functions.worker.WorkerService.start(WorkerService.java:159) [org.apache.pulsar-pulsar-functions-worker-2.3.0.jar:2.3.0]
	at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1042) [org.apache.pulsar-pulsar-broker-2.3.0.jar:2.3.0]
	at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:441) [org.apache.pulsar-pulsar-broker-2.3.0.jar:2.3.0]
	at org.apache.pulsar.PulsarStandalone.start(PulsarStandalone.java:305) [org.apache.pulsar-pulsar-broker-2.3.0.jar:2.3.0]
	at org.apache.pulsar.PulsarStandaloneStarter.main(PulsarStandaloneStarter.java:102) [org.apache.pulsar-pulsar-broker-2.3.0.jar:2.3.0]

[sijie]

@aahmed-se I don't think you are using the right connector. you need to use pulsar-io-kafka-connect-adaptor instead of using pulsar-io-debezium. The documentation is here : http://pulsar.apache.org/docs/en/io-cdc/

Additionally I think debezium module should not be packaged as a NAR. it is just a normal module for keeping debezium related code changes and a dependency for pulsar-io-kafka-connect-adaptor.

[aahmed-se]

I am running the docker image it's throwing the error

docker run -it -p 6650:6650 -p 8080:8080 apachepulsar/pulsar-all:2.3.0 /pulsar/bin/pulsar standalone

haven't changed anything.
It looks like a packaging issue

[sijie]

@aahmed-se I see. the problem is pulsar-io-debezium should not be packaged as NAR. should be a simple fix.

[sijie]

sending a fix soon

[merlimat]

Also packaging and downloads are now mentioning and linking to debezium connector. In any case, shouldn’t that be debezium connector rather that be named Kafka connect?

On Thu, Feb 21, 2019 at 6:46 PM Sijie Guo ***@***.***> wrote: sending a fix soon — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3651 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAD0JA16Vh4D_b_U2_gMj0L3ZsgIRKhhks5vP1npgaJpZM4bIlkT> .

-- Matteo Merli <mmerli@apache.org>

[sijie]

Sent pull request #3656

In any case, shouldn’t that be debezium connector rather that be named Kafka connect?

It is a generic kafka-connect adaptor connector that invokes debezium. because debezium is too tied to Kafka connect.

We can add a few wrappers to wrap kafka-connect-adaptor to a debezium connector if needed. /cc @jiazhai to see if he can help with that. However I think keeping a generic kafka-connect-adaptor connector will have its own value.