2

Background:

  • I'm working with (for me) a reasonably large codebase (eg: I've only got a few of the related projects checked out at the moment, and its > 11000 classes).
  • Build is ant, Tests are JUnit, CI is Jenkins.
  • Running all tests before checkin is not an option, it takes Jenkins hours. Even for some of the individual apps it can be 45 minutes.
  • There are some tests that don't reference individual methods based on reflection, and in some cases don't even directly reference the class of the tested methods, as they interrogate an aggregator class, and are aware of the patterns of pass-through methods in use here. As it's a big codebase, > 10 developers, and I'm not in charge, this is something I can not change for now.

What I want, is the ability to before check-in print out a list of all test classes that are two degrees away (Kevin-Bacon-wise) from any class in the git diff list. This way I can run them all and cut down on angry emails from Jenkins when something I missed eventually gets run and has an error.

The easiest way I can think of to achieve this is to code it myself with a Ruby script or similar, which allows me to account for some of the patterns we're using, but to do it I need to be able to query "which classes reference class X?"

I could parse .java or (easier) .class files to get this info, but I'd rather not :) Is there a way I can make Javac export it in a simple format as it compiles?

Sophistifunk
  • 4,742
  • 4
  • 28
  • 37
  • More info: I need to run this on my workstation, to catch errors I've made before they go upstream, at least until I can demonstrate it's useful. Switching to Git, Maven, and improving the Jenkins process would be nice, but it's not something I can make happen. – Sophistifunk Apr 18 '13 at 05:50
  • There are many tools that analyze dependencies between classes. YOu might be able to use on that has textual output and build a dependency graph. Also look at Sonar, which I don't know anything about but is supposed to be a great tool for any source analysis – Miserable Variable Apr 19 '13 at 00:02

1 Answers1

3

Is there a way I can make Javac export it in a simple format as it compiles?

AFAIK, no.

However, there are other ways to get a list of the dependencies:

(Note however that you are unlikely to get a static tool to extract dependencies resulting from Class.forName(), etcetera. Also note that you cannot infer the complete set of dependencies from bytecode files because of the way that "compile time constants" are handled.)

It strikes me that there are a few problems here:

  • It sounds to me like your build, and indeed your project structure is monolithic. If you could restructure the code base into large-scale modules that build separately (according to their dependencies), and version controlled separately, then you only need to do a full build and run all unit tests when there is a change high up ... in a module that everything else depends on. (Can I suggest the "Maven" word. It really helps for a large codebase, and 11,000 classes is large.)

  • It sounds like you may be suffering from the "branches are hard" problem of classic VCS systems.

  • It sounds like you may need a beefier CI system. If you've got more cores and the build framework is right, you should be able to get faster CI builds. (And if you modularize so that you rebuild less ...)

I think it might be easier to address your slow build/test cycle that way rather than via extra (possibly bespoke) tooling to do dependency analysis.

But I recognize that it may not be up to you to make those decisions.

Community
  • 1
  • 1
Stephen C
  • 698,415
  • 94
  • 811
  • 1,216
  • I'm trying to better catch errors *before* they get to Jenkins. – Sophistifunk Apr 18 '13 at 05:44
  • Hmmm ... have you considered using Jenkins to build your development branch? A beefier CI might be needed, but hardware is cheaper than developer time, especially if the hardware is shared by 10 developers. – Stephen C Apr 18 '13 at 06:36
  • 1
    The other point is that you can do parallel Maven builds on your desktop, assuming it has a bit of grunt. (You can get a 4 core / 16Gb machine for around $1K these days ... assuming you are in the US of A.) – Stephen C Apr 18 '13 at 06:42
  • I've actually been considering setting up set of POMs, but I haven't tackled it as I know how much trouble mvn gives you when you don't follow its demands wrt directory layout, and when looking for how to do so most answers tend to be along the lines of "f*** you, do it Maven's way" – Sophistifunk Apr 18 '13 at 23:33
  • A better characterization would be "Don't make life hell for yourself. Do it Maven's way". A codebase restructure is a once-off cost, and if you are using a decent VCS it won't disrupt your history. Yes it is all extra work, but think of the pay-off. – Stephen C Apr 18 '13 at 23:39
  • Your Honour, the state rests. – Sophistifunk Apr 19 '13 at 03:58