I've reached a major milestone - unit tests can be written in Fan itself! I've kept the compiler self tests written in Java which you can run via "fant -self". However now fant is designed primarily to run the Fan unit test framework:

Usage:
  fant [options] <pod>
  fant [options] <pod>::<test>
  fant [options] <pod>::<test>.<method>
Options:
  -help, -h, -?  print usage help
  -v             verbose mode
  -self <name>   run compiler self tests

The fan test framework is modeled pretty much just like JUnit, NUnit, RUnit, etc:

1. Create test cases by subclassing sys::Test
2. Override setup() or teardown() for init/cleanup
3. Create individual test methods by starting your method name with "test"
4. Use verify methods to assert conditions (verify, verifyFalse, verifyEq, verifyNotEq)
5. Run the tests via fant

The test framework uses reflection to find all the Test classes and their testFoo methods. Note fan tests work like JUnit - a new Test instance is created and setup before running each test method. With this simple test framework and all the shorthand Fan syntax makes writing tests in Fan much cleaner than all the syntax that Java requires.

Other developments (in order to get to this point):

• I stubbed out the sysTest pod which I being using for sys/compiler tests
• I decided that sys pod will be completely written in native language, however it will be completely stubbed out in fan to generate docs and fcode for reflection (I can't come up a decent hybrid solution)
• I got basic inheritance working
• I got pod loading from zip working
• I decided on sys::Obj.echo() as our System.out.println convenience
• I got multiple class definitions in one source file working (Fan won't support nested classes, but it won't require classes to be in specific filenames with)

I may be covering old territory here; how about having a test facet that could be used to identify which methods are test methods - just like newer versions of JUnit.

i.e.

@test
Void someMethod()
{
	…
}

This post (from 2006) is a bit dated, so just to be sure I want to point out the current documentation is under docTools.

The reason I didn't use a facet was because I wanted to force the convention to have methods start with "test". The only reason to use a facet is to let people use other conventions, and I'm not sure why breaking such a simple convention is a good idea.

I don't know the history of JUnit - did they change from naming conventions to annotations for a good reason, or was more because they just wanted use the latest fanciest stuff?

I strongly suggest going with the facet. You shouldn't have to modify your method names just to give special meaning to your methods for a specific tool. It also opens the door for a lot of neat features, such as defining groups for tests.

For example, with TestNG:

@Test(groups = { "fast", "database" }, timeOut = 10000)
public void verifyInsert() {
  ...
}

Take a look at the Javadoc to get an idea of what's possible.

By the way, TestNG used annotations first, JUnit 4 just copied us a few years later :-)

-- Cedric

I prefer our current convention. IMO, your test suite should be separate from your production code, and therefore I see no reason to use facets. A good test suite can get very complicated, and should not be muddying up your production code.

Not sure I see the connection between mixing test and production code and the code snippet I pasted above?

Tests should always be separated from your production code, no argument there.

The point is just that "this method is a test" is by definition a facet, just like "this method is remote" or "this method represents column foo in the database", so the method name is not the right location to carry this information.

My point was since its a separate focused set of methods, it seems perfectly reasonable to use the testXXX convention.

Though one of my major gripes with the current system (as you've noted maybe indirectly) is that I can't arbitrarily groups tests. Grouping by Type is usually not sufficient when trying to test specific things. So in that case, being able to assign tests to a group would be very useful (at both the type and method level).

The timeOut is interesting as well, there's probably alot we can glean from existing unit testing frameworks.

Whether or not the @test facet denotes the method as a test method, or if you're required to use testXXX I could probably go either way on. I tend to prefer convention and structure, which is why I lean toward testXXX.

Here are a few more examples, which will hopefully convince you that testXXX is a bad idea:

@Test(invocationCount = 1000, threadPoolSize = 5, groups = {"fast", "database"})
public void databaseShouldNotLockUp() { ... }

// This method will provide data to any test method that declares 
// that its Data Provider is named "test1"
@DataProvider(name = "test1")
public Object[][] createData1() {
 return new Object[][] {
   { "Cedric", new Integer(36) },
   { "Anne", new Integer(37)}, 
 };
}

// This test method declares that its data should be supplied by the Data Provider
// named "test1".  It will print "Cedric 37" and then "Anne 37"
@Test(dataProvider = "test1")
public void verifyData1(String n1, Integer n2) {
  System.out.println(n1 + " " + n2);
} 

// This test will fail it it doesn't throw the expected exception
@Test(expectedExceptions = ParseErrorException.class)
public void parserShouldNotAcceptIllegalExpressions() {
  parse("1 ++ 2");
}

Folks,

In fact, both testXXX and @test facet is a convention - nothing else. The talk is about which convention you'd prefer to use.

From my standpoint annotations is better, just for simple reason: testXXX convention assumes that you need to change method name when following convention (enabling/disabling a test, etc). Changing name of the method is much more unsafe operation (comparing to change of annotation), which sometimes require fixing references to that method (e.g. a kind of test suite referring testXXX methods) and so on...

Thank you, Andrey

That sort of hurts my head, and seems much simpler (and clearer) to just implement in normal Fan code:

// The data provider example was a bit contrived, so this 
// doesn't match up exactly
Void testData()
{
  verifyData("Cedric", 36)
  verifyData("Anne",   37)
}

Void verifyData(Str name, Int age)
{
  p := makePerson(name, age)
  verifyEq(p.name, name)
  verifyEq(p.age,  age)
}

Void testParser() 
{
  verifyErr(ParserErrorException.type) |,| { parse("1 ++ 2") }
  verifyErr(ParserErrorException.type) |,| { parse(...) }
  verifyErr(ParserErrorException.type) |,| { parse(...) }
  verifyErr(SomeOtherErr.type) |,| { ... }
}

I still think there is room for facets to do some things like grouping tests - but I much prefer the very simple API we have currently.

Here are a few more examples, which will hopefully convince you that testXXX is a bad idea

Cedric, I don't see why those example methods couldn't be prefixed with test just as easily (which I would prefer by convention anyhow). I'm not against adding facets to make sys::Test do more. But in this case the convention works fine, and forcing the use of a facet doesn't add any value (unless you count not using conventions as "value").

For the purposes of this discussion let's separate the notion of using @test to indicate a test versus adding new features to the test harness. For example I like the grouping idea, but I don't see that being predicated on removing the naming convention.

I've used old JUnit, JUnit 4, and TestNG. I'm not an expert on the fancy features of any of them. I've just done simple unit tests.

I like @test much better than test... names. Two reasons:

1. When I see a list of tests, I don't like the annoying test... in front of all of them. I just want to name my tests whatever I want to name them.
2. As a newbie (from personal experience), it's easy to write helper methods to check certain conditions and call these testWhatever without thinking about how this registers them as individual tests. It's much harder to accidentally put @test in front of a method name.

And the amount of typing isn't much different for either form.

I have used both extensively, and prefer convention over configuration. Many times I've forgotten to add an annotation to a method, and the only thing that saved me was TDD (see the red before you see the green). Inspecting other developers' code, I see test methods with no annotations that is clearly designed to be called -- but which isn't being called because the developers forgot to prepend an annotation to the method.

Prepending test to methods also helps one to develop good names. Instead of @Test isInside (where the name of the test corresponds to the name of the method being tested -- a horrible but widespread practice), you might write, "testThatIsInsideReturnsTrueForPointInsideRect". So common is testThat in my own code I would actually prefer a unit testing framework to require that test methods start with testThat.

Problem with the namaing convention is that it does not allow you, like in TestNG to do some test groups nor, and that's more important, test dependencies!

If you could argue that test grouping could be managed somehome using packaging or classes, there is no simple way to recréate test dependencies and interaction using naming convention.

By this aspect, naming convention reduce the overall coverage and maintenance ot your test suite. IMHO, the example JohnGD gave with the "testThatIsInsideReturnsTrueForPointInsideRect" test is really what should NOT happen when you do TDD, especially for unit testing. Your unit should be small enough, or decoupled engouh to be able to fully test it in ONE test, especially if it's a true/false method. If you complain that the setup of the test takes to long, the you should really have a look about TestNG solves this problem with it's annoation and way of organising the tests.

When I see a list of tests, I don't like the annoying test... in front of all of them. I just want to name my tests whatever I want to name them.

I am the opposite, I want to see testFoo. Methods are verbs and to me this is saying this method "tests foo", where test is the strong verb.

As a newbie (from personal experience), it's easy to write helper methods to check certain conditions and call these testWhatever without thinking about how this

Convention is that testFoo is the test method, and verifyFoo are the helpers.

Problem with the naming convention is that it does not allow you, like in TestNG to do some test groups nor, and that's more important, test dependencies!

I don't agree with this. A feature like grouping is orthogonal to identifying tests. In fact if I were to approach this feature it would have nothing to do with testing, but rather be a general purpose feature for grouping pods, types, or slots which could then be used for testing, IDE navigation, reflection querying, etc.

In fact I really like the grouping idea - I'll put that on the roadmap as a feature to work out.

Like I said before, I think testing features are orthogonal to identifying tests. There is nothing to say that a specific feature is predicated on having a test facet. That could be true when designing a specific feature, but you have to take each feature and design it first. For example, I would design grouping as general purpose, so the feature would probably use a facet called "groups".

So it might be more productive to discussion specific features.

I have used both extensively, and prefer convention over configuration.

I am strongly in this position right now. Right now the only reason I see to use a facet instead of a naming convention is to break convention. And I fail to see why forcing a convention isn't a good thing. But please let's make a distinction between test features versus test identification.

Problem with the namaing convention is that it does not allow you, like in TestNG to do some test groups nor, and that's more important, test dependencies!

As Brian has pointed out, these are orthogonal issues. Requiring that test methods be identified by prefixing with test does not forbid facets used to group tests.

By this aspect, naming convention reduce the overall coverage and maintenance ot your test suite. IMHO, the example JohnGD gave with the "testThatIsInsideReturnsTrueForPointInsideRect" test is really what should NOT happen when you do TDD, especially for unit testing.

Ummm, TDD is NOT unit testing, and it's impossible to test-drive the development of ANY method using a single test, unless that method returns a constant. Why? Because in TDD, you're supposed to pass the failing testcase in the simplest possible way, which -- for a single testcase -- involves hardcoding the expected return value.

Methinks you're confusing unit testing with TDD. A rather common mistake to make.

I have used both old JUnit and TestNG. Overall I prefer TestNG, however I find the @Test annotation annoying - I prefer the convention approach of starting methods with test.

However, couldn't the two appraches just be combined? All methods starting with test are test methods PLUS any method annotated with @test.

Two issues:

• To refuel an old discussion: I prefer the @Test annotation over a naming convention. If I misspell a method tsetXXX, the test won't get run. Yes, I know I should make the test fail before making it succeed, so I would know when it doesn't run. But this requires programmer discipline, which is not always 100% there. Misspelling @Tset will result in a compiler error, so I know there is something wrong.
• fant (at least in the NetBeans plugin 1.6.4) seems to only run the tests in the first class found in the xxxTest.fan file. I'm a bit of a noob here, so any assistance is appreciated.

To add more logs to the fire. @Test annotations would probably be easier to implement in an IDE since the testXXX convention needs to be coded and checked by special means, while @Test will be supported when regular facets are implemented.