Exception in thread "main" java.lang.IncompatibleClassChangeError: Method 'org.tensorflow.Tensor org.tensorflow.types.TInt32.vectorOf(int[])' must be InterfaceMethodref constant

[aday00]

Thanks for helping me build a working .jar file containing TensorflowJava per #96!

Would a colleague kindly help me with the following runtime error, when I run my tests within a Docker container:

root@4f6083770318:tensorflow-java# java -jar my.jar tf --verbose
DEBUG 1596940815346: Started ndarray TensorflowJavaTest.
DEBUG 1596940815727: matrix3d rank 3
DEBUG 1596940815727: Finished ndarray TensorflowJavaTest.
DEBUG 1596940815728: Started graph TensorflowJavaTest.
DEBUG 1596940819085: fetch_test fetched.data.getInt(0)=3 is 3? true
DEBUG 1596940819087: fetch_test fetched.data.getInt(1)=4 is 4? true
Exception in thread "main" java.lang.IncompatibleClassChangeError: Method 'org.tensorflow.Tensor org.tensorflow.types.TInt32.vectorOf(int[])' must be InterfaceMethodref constant
        at vis.TensorflowJavaTest.feed_test(tf.scala:80)
        at vis.TensorflowJavaTest.run(tf.scala:106)
        at vis.TensorflowJavaTest$.runArgs(tf.scala:58)
        at vis.VIS$.main(vis.scala:171)
        at vis.VIS.main(vis.scala)

My tests are based on runUsingColonSeparatedNames() in SessionTest.java, and I re-wrote some in Scala.

The test body is:

      debug("Started ndarray TensorflowJavaTest.")
      // run simple data buffers test as in https://github.com/tensorflow/java/tree/master/ndarray
      val matrix3d = org.tensorflow.ndarray.NdArrays.ofInts( org.tensorflow.ndarray.Shape.of(2, 3, 2) )
      debug("matrix3d rank " + matrix3d.rank)
      debug("Finished ndarray TensorflowJavaTest.")

      debug("Started graph TensorflowJavaTest.")
      // run simple graph and session tests per https://github.com/tensorflow/java/blob/master/tensorflow-core/tensorflow-core-api/src/test/java/org/tensorflow/SessionTest.java
      val graph = new org.tensorflow.Graph() // defined in https://github.com/tensorflow/java/blob/master/tensorflow-core/tensorflow-core-api/src/main/java/org/tensorflow/Graph.java
      val session = new org.tensorflow.Session(graph)
      val tf = org.tensorflow.op.Ops.create(graph)

      fetch_test(session, tf)
      feed_test(session, tf)

      debug("Finished graph TensorflowJavaTest.")

The fetch test is:

    private def fetch_test(session: org.tensorflow.Session, tf: org.tensorflow.op.Ops): Unit = {
      val split = tf.split( tf.constant(0), tf.array(1, 2, 3, 4), 2l)
      tf.math.add(split.output().get(0), split.output().get(1))
      // Fetch using colon separated names.
      val fetched = session.runner().fetch("Split:1").run().get(0).expect(org.tensorflow.types.TInt32.DTYPE)
      val fetch0 = fetched.data().getInt(0)
      debug("fetch_test fetched.data.getInt(0)=%d is 3? %b".format(fetch0, fetch0 == 3)) // expected to be 3
      val fetch1 = fetched.data().getInt(1)
      debug("fetch_test fetched.data.getInt(1)=%d is 4? %b".format(fetch1, fetch1 == 4)) // expected to be 4
    }

The feed test is:

      import scala.collection.JavaConversions._ // convert java iterator to scala iterator with foreach defined.  Allows TensorflowJava's NdArraySequence to be treated as a Java Iterable, to be treated as a Scala Iterable, so mkString() will work 
...
    private def feed_test(session: org.tensorflow.Session, tf: org.tensorflow.op.Ops): Unit = {
      // Feed using colon separated names.
      val fed = org.tensorflow.types.TInt32.vectorOf(4, 3, 2, 1) // <-- line 80, throws IncompatibleClassChangeError at runtime
      val fetched = session.runner()
                           .feed("Split:0", fed)
                           .feed("Split:1", fed)
                           .fetch("Add")
                           .run()
                           .get(0)
                           .expect(org.tensorflow.types.TInt32.DTYPE)
      val data = fetched.data()
      debug("feed_test fetched.data=%s".format(data.scalars.mkString(",")))
    }

Kindly advise what is wrong with the line val fed = org.tensorflow.types.TInt32.vectorOf(4, 3, 2, 1), which throws the IncompatibleClassChangeError at runtime?

[aday00]

Unfortunately the error persists despite explicit typing:

      val fed: org.tensorflow.Tensor[org.tensorflow.types.TInt32] = org.tensorflow.types.TInt32.vectorOf(4, 3, 2, 1)

[aday00]

If I change the line to:

      val fed: org.tensorflow.Tensor[org.tensorflow.types.TInt32] = org.tensorflow.types.TInt32.tensorOf( org.tensorflow.ndarray.NdArrays.vectorOf(4, 3, 2, 1) )

The runtime error still occurs, albeit with different wording:

root@4f6083770318:tensorflow-java# java -jar my.jar tf --verbose
DEBUG 1596943576230: Started ndarray TensorflowJavaTest.
DEBUG 1596943576558: matrix3d rank 3
DEBUG 1596943576559: Finished ndarray TensorflowJavaTest.
DEBUG 1596943576561: Started graph TensorflowJavaTest.
DEBUG 1596943578485: fetch_test fetched.data.getInt(0)=3 is 3? true
DEBUG 1596943578486: fetch_test fetched.data.getInt(1)=4 is 4? true
Exception in thread "main" java.lang.IncompatibleClassChangeError: Method 'org.tensorflow.Tensor org.tensorflow.types.TInt32.tensorOf(org.tensorflow.ndarray.NdArray)' must be InterfaceMethodref constant
        at vis.TensorflowJavaTest.feed_test(tf.scala:81)
        at vis.TensorflowJavaTest.run(tf.scala:107)
        at vis.TensorflowJavaTest$.runArgs(tf.scala:58)
        at vis.VIS$.main(vis.scala:171)
        at vis.VIS.main(vis.scala)

Would Java 8 vs Java 11 be an issue here? My build environment is Java 8, but the Docker environment by default had Java 11, so I went with that. I suspect it would be best if all environments were Java 8.

[karllessard]

Hi @aday00 , normally it should work with Java 11 but could be good to give it a try on Java 8 to drill down into the problem. Or could the problem be related to Scala?

Can you replace the line val fed = org.tensorflow.types.TInt32.vectorOf(4, 3, 2, 1) by something like that, just to see if that works?

val fed = org.tensorflow.types.TInt32.tensorOf(Shape.of(4)).setInt(4, 0).setInt(3, 1).setInt(2, 2).setInt(1, 3)

[aday00]

Building and running all on java8 fixed it! The problem was building in a java8 environment but running in java11. I kept my build environment as java8, but made a different docker environment that's java8 (instead of java11) for running the test.

root@e6c215dbe790:tensorflow-java# java -jar my.jar tf --verbose
DEBUG 1597197923864: Started ndarray TensorflowJavaTest.
DEBUG 1597197924501: matrix3d rank 3
DEBUG 1597197924502: Finished ndarray TensorflowJavaTest.
DEBUG 1597197924502: Started graph TensorflowJavaTest.
DEBUG 1597197928686: fetch_test fetched.data.getInt(0)=3 is 3? true
DEBUG 1597197928688: fetch_test fetched.data.getInt(1)=4 is 4? true
DEBUG 1597197928779: feed_test fetched.data=org.tensorflow.ndarray.impl.dense.IntDenseNdArray@87b,org.tensorflow.ndarray.impl.dense.IntDenseNdArray@83d,org.tensorflow.ndarray.impl.dense.IntDenseNdArray@7ff,org.tensorflow.ndarray.impl.dense.IntDenseNdArray@7c1
DEBUG 1597197928781: Finished graph TensorflowJavaTest.

Will next fix up my test code to print more sane results, at your suggestion, thanks @karllessard!

[aday00]

Improved feed_test code. Previously, output was IntDenseNdArray references, which is not informative:

DEBUG 1597197928779: feed_test fetched.data=org.tensorflow.ndarray.impl.dense.IntDenseNdArray@87b,org.tensorflow.ndarray.impl.dense.IntDenseNdArray@83d,org.tensorflow.ndarray.impl.dense.IntDenseNdArray@7ff,org.tensorflow.ndarray.impl.dense.IntDenseNdArray@7c1

Now, printing the IntDenseNdArray/Tensor values, which is informative:

DEBUG 1597204232442: feed_test fetched.data=8,6,4,2

To do this, the updated feed_test code involves a map(...) with a data.getInt(...):

 private def feed_test(session: org.tensorflow.Session, tf: org.tensorflow.op.Ops): Unit = {
      // Feed using colon separated names.
      val fed: org.tensorflow.Tensor[org.tensorflow.types.TInt32] = org.tensorflow.types.TInt32.vectorOf(4, 3, 2, 1)
      //val fed: org.tensorflow.Tensor[org.tensorflow.types.TInt32] = org.tensorflow.types.TInt32.tensorOf( org.tensorflow.ndarray.NdArrays.vectorOf(4, 3, 2, 1) ) // functionally equivalent to above
      val fetched = session.runner()
                           .feed("Split:0", fed)
                           .feed("Split:1", fed)
                           .fetch("Add")
                           .run()
                           .get(0)
                           .expect(org.tensorflow.types.TInt32.DTYPE)
      val data = fetched.data()
      debug("feed_test fetched.data=%s".format( (0l until data.shape.size).map( data.getInt(_) ).mkString(",") ))
    }

The full test output is:

root@e6c215dbe790:tensorflow-java# java -jar my.jar tf --verbose
DEBUG 1597204230791: Started ndarray TensorflowJavaTest.
DEBUG 1597204231071: matrix3d rank 3
DEBUG 1597204231071: Finished ndarray TensorflowJavaTest.
DEBUG 1597204231073: Started graph TensorflowJavaTest.
DEBUG 1597204232279: fetch_test fetched.data.getInt(0)=3 is 3? true
DEBUG 1597204232288: fetch_test fetched.data.getInt(1)=4 is 4? true
DEBUG 1597204232442: feed_test fetched.data=8,6,4,2
DEBUG 1597204232443: Finished graph TensorflowJavaTest.

So we see the vector sum: (4,3,2,1) + (4,3,2,1) = (8,6,4,2). Good. Thanks for your help @karllessard !

[Craigacp]

Building and running all on java8 fixed it! The problem was building in a java8 environment but running in java11. I kept my build environment as java8, but made a different docker environment that's java8 (instead of java11) for running the test.

root@e6c215dbe790:tensorflow-java# java -jar my.jar tf --verbose
DEBUG 1597197923864: Started ndarray TensorflowJavaTest.
DEBUG 1597197924501: matrix3d rank 3
DEBUG 1597197924502: Finished ndarray TensorflowJavaTest.
DEBUG 1597197924502: Started graph TensorflowJavaTest.
DEBUG 1597197928686: fetch_test fetched.data.getInt(0)=3 is 3? true
DEBUG 1597197928688: fetch_test fetched.data.getInt(1)=4 is 4? true
DEBUG 1597197928779: feed_test fetched.data=org.tensorflow.ndarray.impl.dense.IntDenseNdArray@87b,org.tensorflow.ndarray.impl.dense.IntDenseNdArray@83d,org.tensorflow.ndarray.impl.dense.IntDenseNdArray@7ff,org.tensorflow.ndarray.impl.dense.IntDenseNdArray@7c1
DEBUG 1597197928781: Finished graph TensorflowJavaTest.

Will next fix up my test code to print more sane results, at your suggestion, thanks @karllessard!

What version of scala are you using? Everything compiled on Java 8 should be upwards compatible, and it's worrying if it's not.

[aday00]

@Craigacp I use scala 2.11.8. All the scala libs get wrapped up in the uber jar, and run with java.

$ scala -version
Scala code runner version 2.11.8 -- Copyright 2002-2016, LAMP/EPFL

Also, my java8 build environment is:

$ java -version
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-1~deb9u1-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

The docker/test-environment java8 version is:

# java -version
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (build 1.8.0_252-8u252-b09-1~deb9u1-b09)
OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)

The docker/test-environment that was java11 and had compatibility issues was:

# java -version
openjdk version "11.0.6" 2020-01-14
OpenJDK Runtime Environment (build 11.0.6+10-post-Debian-1deb10u1)
OpenJDK 64-Bit Server VM (build 11.0.6+10-post-Debian-1deb10u1, mixed mode, sharing)

[aday00]

@karllessard I like your simple SavedModelBundle export(...) https://github.com/karllessard/tensorflow-java/blob/bdb0420fd72264457a6d47e5e6c6f7a2e56b4271/tensorflow-core/tensorflow-core-api/src/test/java/org/tensorflow/SavedModelBundleTest.java#L116
Will this be merged into the mainline soon, or should I use your branch?

I see the mainline SavedModelBundle has the Loader but not the Exporter https://github.com/tensorflow/java/blob/0f0ff7d10fafe2c42ea4768ec13ab5a169da1ce5/tensorflow-core/tensorflow-core-api/src/main/java/org/tensorflow/SavedModelBundle.java.

[karllessard]

Hey @aday00 , I'm working on it but before merging I need to confirm a few things with other members of the SIG as we might prefer to go toward a function-centric approach (TF 2.x) instead of a session-centric one (TF 1.x). But I'll consider your feedback, thanks!

BTW, feel free to participate to the community calls we are having to discuss live about these matters, next one is this Friday (9am PST, August 14th). Let me know if you are interested.

[karllessard]

@Craigacp , do you think it is a problem related to Scala? I'm building my applications on Java8 (using Kotlin) and running them on Java11 and I didn't ran into this issue so far.

[Craigacp]

I don't think Scala 2.11.8 works properly with Java 9+. I think 2.11.12 is the first release with some semblance of Java 9+ support, and something later on in the 2.12 release series would be better still.

[aday00]

@karllessard From my naive perspective, having multiple ways of saving a model, like Python Keras does, can be helpful in practice. I trust you and the Tensorflow-Java community have thought about all this a lot, and that you'll all do the right thing.

@Craigacp You're right, I've used Scala 2.11.8 with Java 9+ with libraries other than Tensorflow-Java, and I run into similar issues too. Sorry for the noise.

[aday00]

Stupid git question: what git command do I run to checkout https://github.com/karllessard/tensorflow-java/blob/bdb0420fd72264457a6d47e5e6c6f7a2e56b4271/tensorflow-core/tensorflow-core-api/src/test/java/org/tensorflow/SavedModelBundleTest.java#L116

The exporter(...) to save a model isn't in the master branch.

Edit: Ah, this must be the save_model branch.

The commands to checkout the save_model branch are:

1. git fetch
2. git checkout -b save_model remotes/origin/save_model

Then we see exporter(...). Good.

tensorflow-java-karl$ grep exporter ./tensorflow-core/tensorflow-core-api/src/test/java/org/tensorflow/SavedModelBundleTest.java
        SavedModelBundle.exporter(testFolder.toString())