0

Is there an example of a self-contained repository showing how to perform SQL unit testing of PyFlink (specifically 1.13.x if possible)?

There is a related SO question here, where it is suggested to use some of the tests from PyFlink itself. The issue I'm running into is that the PyFlink repo assumes that a bunch of things are on the Java classpath and that some Python utility classes are available (they're not distributed via PyPi apache-flink).

I have done the following:

  1. Copied test_case_utils.py and source_sink_utils.py from PyFlink into my project.
  2. Copy an example unit test (this one as suggested by the related SO question.

When I try to run the test, I get an error because the test case cannot determine what version of Avro jars to download (download_apache_avro() fails, because this code tries to evaluate the value of avro.version by running mvn help:evaluate -Dexpression=avro.version)

I then added a dummy pom.xml defining a Maven property of avro.version (with a value of 1.10.0) and my unit test case is loaded.

I now get a new error and my test is skipped:

'flink-table-planner*-tests.jar' is not available. Will skip the related tests.

I don't know how to fix this. I've tried adding flink-table-planner and flink-table-planner-blink dependencies with <type>test-jar</type> to my dummy pom.xml, but it still fails.

This is starting to feel like a real pain to do something that should be trivial: basic TDD of a PyFlink project. Is there a real-world example of a Python project that shows how to set up a testing environment for unit testing SQL with PyFlink?

John
  • 10,837
  • 17
  • 78
  • 141
  • This question could be improved by [edit]ing out the part about looking for an example, and instead simply asking the question you're looking for. As it is, the question seems to solicit links to external libraries or solutions, which is off-topic for stack overflow (as per the answer below). – TheFungusAmongUs Apr 26 '22 at 13:49

1 Answers1

0

You can refer to https://github.com/dianfu/pyflink-faq/tree/main/testing which gives an example on how to write unit tests in an external project.

Dian Fu
  • 211
  • 1
  • 2