Spawning out new JVM processes is making testing slow on my computer, so I looked at the underlying mechanisms. So, rather than using the current code in Java (or C#), I started a quick hack of Fant code in Fan. Here's what I have so far (executing directly inside a BuildPod script):

Pod.find(podName).types.findAll{it.fits(Test#)}.each |type| {
  log.info(type.name)
  type.methods.findAll{it.name.startsWith("test") && !it.isAbstract}.each {
    log.info(it.name)
    Test test := type.make
    try { 
      test.setup
      it.call(test)
    } catch (Err err) {
      log.error("it failed", err)
    } finally {
      test.teardown
    }     
  }     
}

Sadly, I don't seem to have access to the verifyCount. That's when I learned that Test is really also written in Java, and I don't think I can get easy access to the verifyCount field from Fan. I tried a hack of using [java] fan.sys::Test as JTest, but that gives me sys::UnknownTypeErr: sys::FanObj.

Anyway, running the tests from directly in the same Fan process does seem to speed things up, but is it possible you could make verifyCount a public method or readonly field? Also, it might be nice to implement the class in pure Fan, too, if that's possible.

I can definitely add a hook for getting at test data.

Spawning out new JVM processes is making testing slow on my computer

Is this something you are doing yourself, or are you having a problem with fant itself?

I can definitely add a hook for getting at test data.

Thanks much.

Is this something you are doing yourself, or are you having a problem with fant itself?

Fant works fine. Just that the new JVM exec is slow, and that's how it's designed to run from build scripts right now. Or at least, that's how it seems to be written and behaving.

If I do happen to get all the fant behavior done (which isn't 100% vital for me), I'll submit it for including in Fan, if you'd like.

In most circumstances you have to spawn a new process to test because you have to reload the classes you just recompiled. So it is a bit tricky.

Buildall spawns one process for fant -all as a performance enhancement. Maybe BuildGroup probably should too (if you can assume everything is a pod).

Why not load those classes in separate classloaders? If you need to recompile, throw away the CL and create a new one.

don't know necessarily what long term solution might be, but I added a hook for now to get at Test.verifyCount using dynamic call operator:

test->verifyCount

test->verifyCount

Thanks much. Will that be in the next build? Maybe worth renaming this thread to match this and call that a solution for now. I'll mention if anything else looks missing, but I think this is the main item.

By the way, my test code seems to pick up changes in the current round of compiling. So that's working fine for me.

Why not load those classes in separate classloaders? If you need to recompile, throw away the CL and create a new one.

Concerning potential reloading of pods, it would be sort of cool if each pod could be loaded (via whatever mechanisms for whatever platforms) such that they could be uncached and reloaded on demand. I think that might be possible with classloader per pod in Java, but I'm not a real expert. I've only played with classloaders a little. This topic will likely become important eventually, even if not now, and if not now, I don't feel a need to push it yet, either.

test->verifyCount

Verified this as working for me now. Here's my pure-Fan fant implementation (missing command-line features and pattern filtering and so on):

override Void test() {
  // Redone to avoid new JVM spawns. Too slow that way.
  startTime := DateTime.now
  failureCount := 0
  failureNames := Str[,]
  methodCount := 0
  verifyCount := 0
  echo("")
  testTypes := Pod.find(podName).types.findAll{it.fits(Test#)}
  testTypes.each |type| {
    methods := type.methods.findAll{it.name.startsWith("test") && !it.isAbstract}
    methodCount += methods.size
    methods.each |method| {
      testName := "${type.qname}.${method.name}"
      try {
        echo("-- Run: $testName ...")
        Test? test
        try {
          test = type.make
        } catch (Err err) {
          echo("")
          echo("ERROR: Cannot make test $type")
          err.trace
          throw FailureErr()
        }
        try {
          test.setup
          method.call(test)
        } catch (Err err) {
          echo("")
          echo("TEST FAILED")
          err.trace
          throw FailureErr()
        } finally {
          try {
            test.teardown
          } catch (Err err) {
            err.trace
          }
        }
        verifyCount += test->verifyCount
        echo("   Pass: ${testName} [${test->verifyCount}]")
      } catch (FailureErr err) {
        failureCount++
        failureNames.add(testName)
      }
    }
  }
  echo("")
  echo("Time: ${DateTime.now - startTime}")
  echo("")
  if (!failureNames.isEmpty) {
    echo("Failed:")
    failureNames.each {
      echo("  $it")
    }
    echo("")
  }
  echo("***")
  echo(
    "*** " +
    (failureCount == 0 ? "All tests passed!" : failureCount + " FAILURES") +
    " [${testTypes.size} tests, $methodCount methods, $verifyCount verifies]"
  )
  echo("***")
}

Apologies on the evil end-line curlies.

Apologies on the evil end-line curlies.

That's not the problem, actually. The problem is you don't use newlines after closing curlies. Look at this code:

// blah blah
  verifyCount += test->verifyCount
  echo("   Pass: ${testName} [${test->verifyCount}]")
} catch (FailureErr err) {
  failureCount++
  failureNames.add(testName)

When scanning this code quickly, one misses the "catch" because it looks just like another statement (sometimes one will see the curly brace, other times not).

But you're close to formatting heaven:

// blah blah
  verifyCount += test->verifyCount
  echo("   Pass: ${testName} [${test->verifyCount}]")
}
catch (FailureErr err) {
  failureCount++
  failureNames.add(testName)

Ta da! Now there's no mistaking that catch is a separate construct.

These ideas covered in Code Complete, among other sources.

Looks like a bug in Fandoc. Formatting's all screwed up.

I was thinking, it would be cool if Flux had a little add-in, that didn't take up much display space, just sat in the corner and ran tests as a background task.

It could check the test's pod for dependencies, and re-schedule tests whenever a file changes; show a green light when all's good and yellow/red with popup details for failures. Ideally links in the walkback jump to marks in the breaking source.

Then without having to do anything, you'd know within seconds, when you break something. Would help with the "what in the world did I do to cause that?" problem.

When scanning this code quickly, one misses the "catch" because it looks just like another statement

Actually, I always put the catch on the same line after the curly specifically so it doesn't look like just another statement. I like it obviously tied to the previous block. But, just goes to show that we all have different styles.

I skimmed some of Code Complete once upon a time (about 10 years ago, I think). I can't remember what I thought about it, but I at least didn't get back to it again.