I am using intellij, maven 3, and flink 1.15.1 to write a stateful streaming job. I am trying to write unit tests for my custom KeyedProcessFunction and tried following the documentation here along with adding the dependency mentioned here. I'm interested in using the KeyedOneInputStreamOperatorTestHarness class, but I can't find that class in any of my dependencies (redacted pom posted below). The only time I can find the class is by including
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java_2.12</artifactId>
<version>1.14.5</version>
<scope>test</scope>
<type>test-jar</type>
</dependency>
which is from the previous version. Flink >= 1.15.0 doesn't require the scala dependent libraries now, so there is no flink-streaming-java_2.12 for 1.15.1.
Did the class I want to use get moved somewhere else or purposefully excluded? I've tried all the google foo I know, and I can find the class in the flink repo, but not in any of the flink dependencies I've tried so far, outside of that <=1.14.5 one. Am I doing something wrong or missing some documentation? Is using the 1.14.5 library my only option or is there some new type of test utils that I don't know about?
Pom dependencies that don't have the class:
...
<flink.version>1.15.1</flink.version>
...
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-java</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-clients</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-kafka</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-test-utils</artifactId>
<version>${flink.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-test-utils-junit</artifactId>
<version>${flink.version}</version>
<scope>test</scope>
</dependency>
...
CodePudding user response:
It seems that I've figured it out, but my maven knowledge is limited so I'm not sure what the pros and cons are. Also I imagine future versions of flink will change this. The bonus is that I can use the latest flink version without including in older code.
I added these dependencies for testing (for 1.15.1 at least):
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-test-utils</artifactId>
<version>${flink.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java</artifactId>
<version>${flink.version}</version>
<scope>test</scope>
<classifier>tests</classifier>
<type>test-jar</type>
</dependency>
I came across the flink readme about artifacts and the classifier tag, and I knew that the classes I needed were in flink-streaming-java package, specifically in its tests section. So including dependency for flink-streaming-java with the <classifier>tests</classifier>
tag made the test classes available.
For what it's worth, there is a cryptic warning in the doc:
Useful when users should be pulling in Flink tests dependencies. This is mostly for the test harnesses and probably not what you want.
I still needed the flink-test-utils for the mock runner it provides.