Exception in thread "main" java.lang.NoClassDefFoundError: org/bytedeco/javacpp/Pointer

[aday00]

Thanks for helping me build tensorflow-java per #94 !

My first tests involve NdArray and Graph, but I naively can't find org/bytedeco/javacpp/Pointer.class to put in my.jar to avoid the following exception:

$ java -jar my.jar tf --verbose
DEBUG 1596849621481: Started ndarray TensorflowJavaTest.
DEBUG 1596849621705: matrix3d rank 3
DEBUG 1596849621706: Finished ndarray TensorflowJavaTest.
DEBUG 1596849621706: Started graph TensorflowJavaTest.
Exception in thread "main" java.lang.NoClassDefFoundError: org/bytedeco/javacpp/Pointer
	at vis.TensorflowJavaTest.run(tf.scala:76)
	at vis.TensorflowJavaTest$.runArgs(tf.scala:56)
	at vis.VIS$.main(vis.scala:171)
	at vis.VIS.main(vis.scala)
Caused by: java.lang.ClassNotFoundException: org.bytedeco.javacpp.Pointer
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 4 more

The code for my simple test is in Scala, which gets packed into my.jar with other code:

      debug("Started ndarray TensorflowJavaTest.")
      // run simple data buffers test as in https://github.com/tensorflow/java/tree/master/ndarray
      val matrix3d = org.tensorflow.ndarray.NdArrays.ofInts( org.tensorflow.ndarray.Shape.of(2, 3, 2) )
      debug("matrix3d rank " + matrix3d.rank)
      debug("Finished ndarray TensorflowJavaTest.")

      debug("Started graph TensorflowJavaTest.")
      // run simple graph and session tests per https://github.com/tensorflow/java/blob/master/tensorflow-core/tensorflow-core-api/src/test/java/org/tensorflow/SessionTest.java
      val graph = new org.tensorflow.Graph() // defined in https://github.com/tensorflow/java/blob/master/tensorflow-core/tensorflow-core-api/src/main/java/org/tensorflow/Graph.java
      val session = new org.tensorflow.Session(graph)
      val tf = org.tensorflow.op.Ops.create(graph)
      debug("Finished graph TensorflowJavaTest.")

Where might the bytedeco classes be? I only see these jars from my build:

(base) tensorflow-java$ find . -name '*jar'|grep -v surefire
./tensorflow-framework/target/tensorflow-framework-0.2.0-SNAPSHOT.jar
./ndarray/target/ndarray-0.2.0-SNAPSHOT.jar
./tensorflow-core/tensorflow-core-generator/target/tensorflow-core-generator-0.2.0-SNAPSHOT.jar
./tensorflow-core/tensorflow-core-generator/target/tensorflow-core-generator-0.2.0-SNAPSHOT-sources.jar
./tensorflow-core/tensorflow-core-api/target/tensorflow-core-api-0.2.0-SNAPSHOT.jar
./tensorflow-core/tensorflow-core-api/target/tensorflow-core-api-0.2.0-SNAPSHOT-linux-x86_64.jar
./tensorflow-core/tensorflow-core-api/target/tensorflow-core-api-0.2.0-SNAPSHOT-sources.jar
./tensorflow-core/tensorflow-core-platform/target/tensorflow-core-platform-0.2.0-SNAPSHOT.jar
./tensorflow-core/tensorflow-core-platform/target/tensorflow-core-platform-0.2.0-SNAPSHOT-sources.jar
./tensorflow-core/tensorflow-core-platform/target/tensorflow-core-platform-0.2.0-SNAPSHOT-javadoc.jar

No bytedeco jars in Maven's .ivy2 cache:

$ find ~/.ivy2/cache -name '*bytedeco*'

In the docker container where I built tensorflow-java, there is a .javacpp directory with bytedeco artifacts, but no .class or .jar files.

root@4f6083770318:~/.javacpp/cache# find|sort
.
./.lock
./javacpp-1.5.3-linux-x86_64.jar
./javacpp-1.5.3-linux-x86_64.jar/org
./javacpp-1.5.3-linux-x86_64.jar/org/bytedeco
./javacpp-1.5.3-linux-x86_64.jar/org/bytedeco/javacpp
./javacpp-1.5.3-linux-x86_64.jar/org/bytedeco/javacpp/linux-x86_64
./javacpp-1.5.3-linux-x86_64.jar/org/bytedeco/javacpp/linux-x86_64/libjnijavacpp.so
./tensorflow-core-api-0.2.0-SNAPSHOT-linux-x86_64.jar
./tensorflow-core-api-0.2.0-SNAPSHOT-linux-x86_64.jar/org
./tensorflow-core-api-0.2.0-SNAPSHOT-linux-x86_64.jar/org/tensorflow
./tensorflow-core-api-0.2.0-SNAPSHOT-linux-x86_64.jar/org/tensorflow/internal
./tensorflow-core-api-0.2.0-SNAPSHOT-linux-x86_64.jar/org/tensorflow/internal/c_api
./tensorflow-core-api-0.2.0-SNAPSHOT-linux-x86_64.jar/org/tensorflow/internal/c_api/linux-x86_64
./tensorflow-core-api-0.2.0-SNAPSHOT-linux-x86_64.jar/org/tensorflow/internal/c_api/linux-x86_64/libjnitensorflow.so
./tensorflow-core-api-0.2.0-SNAPSHOT-linux-x86_64.jar/org/tensorflow/internal/c_api/linux-x86_64/libtensorflow.so
./tensorflow-core-api-0.2.0-SNAPSHOT-linux-x86_64.jar/org/tensorflow/internal/c_api/linux-x86_64/libtensorflow.so.2
./tensorflow-core-api-0.2.0-SNAPSHOT-linux-x86_64.jar/org/tensorflow/internal/c_api/linux-x86_64/libtensorflow_framework.so
./tensorflow-core-api-0.2.0-SNAPSHOT-linux-x86_64.jar/org/tensorflow/internal/c_api/linux-x86_64/libtensorflow_framework.so.2

Would a colleague kindly tell me where the bytedeco Pointer class file is, so I may continue with my tests? Many thanks in advance! Sorry for the basic question. Surely I'm missing something, because I can see bytedeco in various pom.xml files, like this one:

(base) tensorflow-java/tensorflow-core/tensorflow-core-api$ grep -2 bytedeco pom.xml
  <dependencies>
    <dependency>
      <groupId>org.bytedeco</groupId>
      <artifactId>javacpp</artifactId>
      <version>${javacpp.version}</version>
    </dependency>
    <dependency>
      <groupId>org.bytedeco</groupId>
      <artifactId>javacpp</artifactId>
      <version>${javacpp.version}</version>
--
      </plugin>
      <plugin>
        <groupId>org.bytedeco</groupId>
        <artifactId>javacpp</artifactId>
        <version>${javacpp.version}</version>
--
              <quiet>true</quiet>
              <links>
                <link>http://bytedeco.org/javacpp/apidocs</link>
              </links>
            </configuration>

[saudet]

It comes from that artifact you point out:
https://github.com/tensorflow/java/blob/master/tensorflow-core/tensorflow-core-api/pom.xml#L23-L27

[aday00]

Thanks @saudet ! I think I'm missing something fundamental. Please advise where might the bytedeco Pointer.class might be, to fix my java.lang.NoClassDefFoundError: org/bytedeco/javacpp/Pointer? None of these are it:

tensorflow-java$ find . -name '*class'|grep Pointer
./tensorflow-core/tensorflow-core-api/target/classes/org/tensorflow/internal/c_api/Delete_func_Pointer.class
./tensorflow-core/tensorflow-core-api/target/classes/org/tensorflow/internal/c_api/TF_Buffer$Data_deallocator_Pointer_long.class
./tensorflow-core/tensorflow-core-api/target/classes/org/tensorflow/internal/c_api/Listener_BytePointer.class
./tensorflow-core/tensorflow-core-api/target/classes/org/tensorflow/internal/c_api/Compute_func_Pointer_TF_OpKernelContext.class
./tensorflow-core/tensorflow-core-api/target/classes/org/tensorflow/internal/c_api/Deallocator_Pointer_long_Pointer.class

[saudet]

What are you trying to do exactly?

[aday00]

Certainly, I have a my.jar file, which contains all the class files. I ask where Pointer.class is, so I can put Pointer.class in my.jar, and fix the NoClassDefFoundError. The java classloader doesn't know where Pointer.class is. My commandline is java -jar my.jar tf --verbose.

For example, for NdArray, I can put these classes in my.jar, and my tests pass. I can use NdArray without exceptions.

tensorflow-java$ find . -name '*class'|grep NdArray|head
./ndarray/target/classes/org/tensorflow/ndarray/BooleanNdArray.class
./ndarray/target/classes/org/tensorflow/ndarray/ByteNdArray.class
./ndarray/target/classes/org/tensorflow/ndarray/NdArrays.class
./ndarray/target/classes/org/tensorflow/ndarray/DoubleNdArray.class
./ndarray/target/classes/org/tensorflow/ndarray/ShortNdArray.class
./ndarray/target/classes/org/tensorflow/ndarray/impl/AbstractNdArray.class
./ndarray/target/classes/org/tensorflow/ndarray/impl/dense/ByteDenseNdArray.class
./ndarray/target/classes/org/tensorflow/ndarray/impl/dense/AbstractDenseNdArray.class
./ndarray/target/classes/org/tensorflow/ndarray/impl/dense/LongDenseNdArray.class
./ndarray/target/classes/org/tensorflow/ndarray/impl/dense/FloatDenseNdArray.class
...

[saudet]

Your JAR file just doesn't contain all the class files. You'll need to add the missing ones

[aday00]

Indeed, but I don't know where Pointer.class is. It's missing. Where might it be? I would've thought mvn install built it.

[saudet]

Try this: https://maven.apache.org/plugins/maven-shade-plugin/examples/includes-excludes.html

[aday00]

Thanks @saudet ! Having zero maven expertise, I see the mvn install typescript I made has this line:

Downloading from central: https://repo.maven.apache.org/maven2/org/bytedeco/javacpp/1.5.3/javacpp-1.5.3.jar

So I downloaded that manually, and Pointer.class is in there:

$ unzip -l javacpp-1.5.3.jar |grep Pointer.class
     5470  2020-04-13 21:52   org/bytedeco/javacpp/CharPointer.class
     4505  2020-04-13 21:52   org/bytedeco/javacpp/FloatPointer.class
     3378  2020-04-13 21:52   org/bytedeco/javacpp/BoolPointer.class
    11444  2020-04-13 21:52   org/bytedeco/javacpp/PointerPointer.class
      479  2020-04-13 21:52   org/bytedeco/javacpp/FunctionPointer.class
     4522  2020-04-13 21:52   org/bytedeco/javacpp/DoublePointer.class
    11136  2020-04-13 21:52   org/bytedeco/javacpp/BytePointer.class
    15662  2020-04-13 21:52   org/bytedeco/javacpp/Pointer.class
     4053  2020-04-13 21:52   org/bytedeco/javacpp/CLongPointer.class
     4505  2020-04-13 21:52   org/bytedeco/javacpp/ShortPointer.class
     4440  2020-04-13 21:52   org/bytedeco/javacpp/LongPointer.class
     5547  2020-04-13 21:52   org/bytedeco/javacpp/IntPointer.class
     4288  2020-04-13 21:52   org/bytedeco/javacpp/BooleanPointer.class
     4055  2020-04-13 21:52   org/bytedeco/javacpp/SizeTPointer.class

Later, I may need to edit pom.xml as you suggest to whitelist the retention of javacpp-1.5.3.jar or other jars, to keep all the class files I need to make a working uber jar. I'm surprised javacpp-1.5.3.jar isn't in maven's ~/.ivy2 cache.

Pleased I get a different error now, will work on getting *.so in java.library.path, and close this ticket.

$ java -jar vis.tf.jar tf --verbose
DEBUG 1596918557098: Started ndarray TensorflowJavaTest.
DEBUG 1596918557364: matrix3d rank 3
DEBUG 1596918557364: Finished ndarray TensorflowJavaTest.
DEBUG 1596918557365: Started graph TensorflowJavaTest.
Warning: Could not load Loader: java.lang.UnsatisfiedLinkError: no jnijavacpp in java.library.path
Exception in thread "main" java.lang.UnsatisfiedLinkError: no jnitensorflow in java.library.path
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
	at java.lang.Runtime.loadLibrary0(Runtime.java:870)
	at java.lang.System.loadLibrary(System.java:1122)
	at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1631)
	at org.bytedeco.javacpp.Loader.load(Loader.java:1265)
	at org.bytedeco.javacpp.Loader.load(Loader.java:1109)
	at org.tensorflow.internal.c_api.global.tensorflow.<clinit>(tensorflow.java:12)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.bytedeco.javacpp.Loader.load(Loader.java:1176)
	at org.bytedeco.javacpp.Loader.load(Loader.java:1125)
	at org.tensorflow.NativeLibrary.load(NativeLibrary.java:48)
	at org.tensorflow.TensorFlow.init(TensorFlow.java:111)
	at org.tensorflow.TensorFlow.<clinit>(TensorFlow.java:126)
	at org.tensorflow.Graph.<clinit>(Graph.java:730)
	at vis.TensorflowJavaTest.run(tf.scala:76)
	at vis.TensorflowJavaTest$.runArgs(tf.scala:56)
	at vis.VIS$.main(vis.scala:171)
	at vis.VIS.main(vis.scala)
Caused by: java.lang.UnsatisfiedLinkError: no tensorflow in java.library.path
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
	at java.lang.Runtime.loadLibrary0(Runtime.java:870)
	at java.lang.System.loadLibrary(System.java:1122)
	at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1631)
	at org.bytedeco.javacpp.Loader.load(Loader.java:1213)
	... 14 more

Thanks for showing me a path to do this the right way in the future. Hope to get there soon!

[aday00]

Copying over the *.so files, the test completes in the docker container. Thanks again!

root@4f6083770318:tensorflow-java# java -jar my.jar tf --verbose
DEBUG 1596924611892: Started ndarray TensorflowJavaTest.
DEBUG 1596924612277: matrix3d rank 3
DEBUG 1596924612277: Finished ndarray TensorflowJavaTest.
DEBUG 1596924612278: Started graph TensorflowJavaTest.
DEBUG 1596924616479: Finished graph TensorflowJavaTest.