I have noticed a lot of Fan vs Scala lately. In some ways I guess that is to be expected since both languages are statically typed alternatives on the JVM (unlike the dynamic languages such as Groovy, JRuby, Jython, and Clojure). But just remember that we are all on the same team trying to advance the state of JVM languages - we share the common goal of convincing Java programmers that there are alternatives to the Java language itself.

I think Scala is a really great language and I'm always on the lookout of good stuff to steal from it. But I do believe there are some key philosophical differences which I haven't really seen talked about. Note I haven't used Scala to write any production code, so I am not an expert by any means - but I do try to keep up with it.

Both Fan and Scala are trying to take software scalability to the next level, but I think we are taking different paths to get there. Scala's focus is clearly on a sophisticated type system which can catch as many errors as possible at compile time. It is also focused on mixing functional style with OO style (more so than Fan which only uses functional programming primarily as an enabler of better APIs).

Fan uses a static type system also - but as just one tool in the toolbox. Rather much of Fan's effort has been focused on overall architecture - more of a "cathedral" approach to providing a unified solution for all the different pieces required to build, test, and deploy software (yes I know IDE solutions are lacking, but I think this will take care of itself with time). This philosophy on the entire platform and the end-to-end programming experience versus just the language is reflected in Fan's features and even this site itself.

Pods

Fan has a built-in module system called "pods" that is seriously baked into the core. This enables a simple, consistent namespace of types based on pod::type. Pods are intertwined into Fan's reflection APIs to enable all sorts of things that are impossible to do in Java (and I assume Scala) such iterate all the installed types:

Pod.list.each |pod| 
{ 
  echo("--- pod $p.name $p.version ---")
  pod.types.each |t| { if (t.isPublic) echo(t) }
}

If I were to pick the absolute most important feature required for scalable software, I personally would pick a first class module system. I think this is a key difference between Fan and Scala. Scala follows more of the Java model of packages, with an unspecified module system. From what I can tell, OSGi seems to be the preferred Scala solution for modularity, but it isn't integrated tightly into the language like Fan.

Slot Reflection

One of major design decisions Fan made a long time ago was to disallow method overloading. Method overloading is the kiss of death for easy reflection and meta-programming. In Fan any slot can be identified with the qualified name pod::Type.slot. Slot literals are built into the language:

m := List#join  // same as Slot.findMethod("sys::List.join")

Languages like Java or Scala which support method overloading require the parameter types in order to perform method reflection. Identifying methods strictly by name is also a key enabler for efficient dynamic dispatch. This is a trade-off Fan and Scala have made differently - Fan took away a Java feature in order to simplify reflection and dynamic dispatch.

Type Database

Having a built-in module system enables all sorts of other really awesome features. One of the most important is Fan's type database. Because Fan knows about all the installed modules and each module's types, we can index the installed types to provide a unified solution for reflecting things like plugins. For example, in Fan to query the handler for a given URI scheme:

UriScheme scheme := Type.findByFacet(@uriScheme, "file").first.make

More Dynamic Type System

The example above illustrates the "auto-cast" of Fan. Typical Fan code uses reflection like this a lot, and this sort of meta-programming style does not lend itself to static type checking. This is why Fan will let you use Obj as a wildcard whenever a more explicit type is expected. Note the wildcard feature does not really weaken the static type system much, because it typically only kicks in with reflection APIs (which would have have required an explicit cast anyways). As an example, I've just performed a huge amount of refactoring on the code base for symbols and virtually everything was caught at compile time. But this is a case where Fan's type system makes a trade-off towards a weaker static type system, but easier for meta-programming and reflection.

Fan also supports the "->" operator to perform a dynamic call. I know you can do similar things with Scala's support for arbitrary method names, but it isn't quite the same as building it into the language. This operator is designed to play nicely with Fan's type system via auto-casting. More importantly it is a built-in language construct that we can map to the new invokedynamic opcode on the JVM. I don't use this operator a lot myself in production code because I prefer the safety and performance of static calls, but I use it a ton in my tests - I couldn't live without it.

Build Engine

Fan includes a built-in engine for writing build scripts as Fan scripts. Sure it is super simple and doesn't have all the bells and whistles of other Java-land solutions. But it follows the Fan philosophy of having everything ready to use and tightly integrated into Fan's overall architecture. Build scripts leverage Fan's declarative style, but the ability to easily write a new method when you need it. This is opposed to the Java-land way of doing declarative stuff in XML, and then having to write functions separately (something that I've always found awkward).

Furthermore much the build tooling is available via the reflection APIs to easily create new tools. For example to get the fandoc for a method and format as HTML:

fandoc := Str#split.doc
FandocParser().parseStr(fandoc).write(HtmlDocWriter())

Immutability

Fan's type system is by no means as flexible as Scala's. But I do think the notion of immutability is more hardwired into Fan via the notion of const types and const fields. The notion of immutability is also baked into all the reflection and concurrency code. This makes it almost impossible to share mutable state between actors (threads). The only way to share mutable state is via the Unsafe class which exists only as an escape hatch.

Perhaps the prototypical example is the Obj.isImmutable method - the ability to reflectively query any object's mutability.

Portability and JavaScript

Fan started life as portable language for both the JVM and .NET CLR. That forced us to create a clean separation between Fan code and the underlying platform. It turns out that having that underlying philosophy in the architecture is paying huge dividends as we work to make Fan code run seamlessly in the browser by compiling to JavaScript. Unlike GWT which is a separate solution from Java, Fan's JavaScript support is being integrated into the whole experience including tooling and deployment. There is one portable system, graphics, and widget library that just works everywhere.

Some might argue that Fan's attention to portability is a weakness compared to Scala's focus on the JVM. It is certainly a trade off, but I think in the end both Fan and Scala will make for first class JVM languages. And I think Fan's portability will really pay off in the long run.

APIs

Because of Fan's attention to portability, we're trying to create clean, elegant core system APIs required by most typical apps. This lets us define APIs that use Fan idioms such as closures, immutable, and serialization. But it also lets us really focus on an awesome API experience. Fan is much more about its APIs than the language itself - the language is designed to support the APIs, not vise versa.

True, Fan will never have the breadth of APIs available Java - but those are available via the Java FFI. But by focusing on common APIs in Fan such as IO, DateTime, Weblets, etc, Fan code shares common definitions for key abstractions. I think Fan and Scala differ here quite a bit. Scala tends to use implicit conversion - for example to map from plain Java String to rich string. Fan attempts to normalize core concepts like strings, DateTime, IO, etc into powerful, centralized classes which don't require conversions.

Serialization

One of Fan's pivotal features is its human readable JSON like serialization syntax. This format supports Fan's type system (which is predicated on a clean module/namespace infrastructure) so that object trees can be round-tripped via IO with no loss of fidelity. This serialization infrastructure is widely used in all sorts of features such as facets, symbol configuration, messaging between actors, etc. For example to read an list of ints from a string:

"[0, 1, 2]".in.readObj

When you look at typical Java system architectures, there are huge amounts of wasted effort which go into things like XML configuration files and solving all the associated impedance mismatch that follows. The notion of declarative configuration in Fan is best illustrated with the upcoming symbols feature. Now when you have a system wide setting, you just create a symbol:

pod acme
{
  ** Some fandoc describing symbol - type inferred as Duration
   virtual timeout := 10sec

  ** List of server host names - type inferred as Str[]
   virtual servers := ["localhost"]
}

When you want to use the symbol in your code, you use a symbol literal which provides a statically typed value:

socket.options.receiveTimeout = @timeout.val

Then you can override the symbol's value at deployment time using a configuration file:

// {repo}/etc/acme/pod.fansym config file 
timeout=30sec
servers=["alpha", "bravo", "charlie"]

Everything is integrated in Fan's module subsystem, static type system, reflection, and documentation. This feature is also used for statically typed localization. It is precisely this sort of source-to-deployment model that we are really trying to make a great experience in Fan!

(Side note - this feature is only possible to efficiently implement with caching because we have such a comprehensive approach to immutability).

Conclusion

I don't particularly like Fan vs Scala debates, and my intention is not to diss Scala. But I did want to shed some light on my philosophy of Fan and how I think we have made different trade-offs. A lot of it boils down to Scala's focus on its static type system versus Fan's focus on modularity and meta-programming. There is no right or wrong, just different trade-offs that will suit different tastes. There is also definitely an element of cathedral versus bazaar - Fan is trying to bake more into a unified whole (which is something I can understand might not appeal to those who favor the bazaar model of gluing lots of pieces together).

I know the Fan vs Scala debates are probably only just beginning. I would just ask that those of you who participate (especially on this site) keep the conversation professional and constructive. Both Scala and Fan are pretty awesome technologies (if I saw so myself :-).

Generally, I think this was a very good Fan vs Scala article (probably the best I've read), focusing primarily on Fan's strengths. There are a few things I would like to point out though.

First, you entirely glossed over the issue of Java interop. Fan can use any Java library (more or less), but it has to do it through a separate FFI. In Scala, Java libraries are Scala libraries. There is no disparity. This has led to some unfortunate compromises (like null), but on the whole I think it was the right decision. The interop is the language, particularly on the JVM or the CLR.

Scala's JSON support is almost as good as Fan's. Scala does have a built-in library for JSON, but it isn't as good as literaljson. You should read a bit about the library, it's very good, certainly good enough to qualify as "human readable".

Fan's hybrid-dynamic type system has been a point of philosophical contention for a while now, but I would like to specifically take umbrage at your statement that "automatic casting doesn't weaken the type system". What you have done is add the following rule to the type system:

t : T2
T1 < T2
------- T-AutoCast
t : T1

That's in addition to the pre-existing subtyping rule (otherwise known as upcasting):

t : T2
T2 < T1
------- T-Sub
t : T1

In other words, we can in two steps go from any type to any other type. This renders the type system all-but meaningless in the hands of a developer who isn't paying attention to such things. I'm in favor of powerful language features, but I don't think this is a feature that I can support.

Finally, I'm not sure how you can seriously claim that overloading impairs performance. I've written a few compilers in my time, and even in the most naive implementation, dynamic dispatch to an overloaded method is absolutely no slower than to a non-overloaded method. It just takes a little bit of name mangling and away you go (see: C++). HotSpot is hardly a naive compiler, and I can assure you that in practical tests, overloading has no appreciable impact on performance.

To put it bluntly, if you're seeing performance issues specifically because of method overloading, then you're doing something wrong with dynamic dispatch.

Coming down off my high horse: I would like to restate that I liked your article. Obviously, your bias would be toward Fan (as mine would be toward Scala), but that didn't impair a well-reasoned and positive comparison. Nice job!

Finally, I'm not sure how you can seriously claim that overloading impairs performance. I've written a few compilers in my time, and even in the most naive implementation, dynamic dispatch to an overloaded method is absolutely no slower than to a non-overloaded method. It just takes a little bit of name mangling and away you go (see: C++).

I think you misunderstood Brian's claim. Quoting:

Method overloading is the kiss of death for easy reflection and meta-programming.

He's only talking about reflection and meta-programming, not performance in general. Nobody here would argue that method overloading impairs the performance of statically compiled code. However when you introduce reflection into the picture, it's a different story.

In Java (and presumably in Scala too, correct me if I'm wrong) you can have multiple foo methods, each with a different signature. So at runtime when you call:

clazz.getMethod("foo", new Class[] { Bar.class, Baz.class })

The Class.getMethod implementation searches for all methods that fit those argument types, and attempts to decide which one is the best fit. Naturally this step incurs a performance penalty.

However in Fan every field/method must be uniquely named. So reflecting that member is as straightforward performance-wise as a hashtable lookup. I believe this difference is the basis of the "kiss of death" claim.

It would be interesting to benchmark reflection-based serialization in Java vs Fan. I'd like to see how well XMLEncoder/XMLDecoder stack up against Out.writeObj/In.readObj, respectively.

Aside: Here's a gem from Class.getMethod javadoc (emphasis, editorial commentary mine):

If more than one such method is found in (step) C, and one of these methods has a return type that is more specific than any of the others, that method is reflected; otherwise one of the methods is chosen arbitrarily. <headdesk/>

@djspiewak

First, you entirely glossed over the issue of Java interop. Fan can use any Java library (more or less), but it has to do it through a separate FFI.

Does that really matter? All the FFI does is specify a special syntax in the using statement. The nice thing about that is that it is clear if you are importing from Java or .NET. I assume that in Scala on .NET you use the exact same syntax when importing .NET namespaces as you do as importing Java packages. I guess my personal taste is that explicit is better.

Scala's JSON support is almost as good as Fan's. Scala does have a built-in library for JSON, but it isn't as good as literaljson.

I checked it out - pretty cool. Although Fan's serialization syntax goes way beyond just JSON. When you serialize a foo::Bar, you get back a foo::Bar. For example:

Sys.out.writeObj(GridPane
  {
    Label { text="Hello"; font=Font("12pt Arial") },
    Label { text="World!"; fg=Color.green }
  }, 
  ["skipDefaults":true, "indent":2])

Actually prints:

fwt::GridPane
{
  fwt::Label
  {
    text="Hello"
    font=gfx::Font("12pt Arial")
  },
  fwt::Label
  {
    text="World!"
    fg=gfx::Color("#00ff00")
  },
}

That is a chunk of text you can output from a GUI builder and can pass straight to the browser/SWT and have it deserialized into a widget. The format is tightly integrated into the language and all the APIs and supports the same using imports and type inference as the language. On the flip side, any serialized object is a valid Fan expression too.

but I would like to specifically take umbrage at your statement that "automatic casting doesn't weaken the type system".

Actually I said "doesn't weaken the type system much" - I totally agree it weakens the type system. The question is does it cause any practical harm in day to day programming? My experience has been no, most things still get caught at compile time. Like all engineering decisions it is a trade-off.

To put it bluntly, if you're seeing performance issues specifically because of method overloading, then you're doing something wrong with dynamic dispatch.

As qualidafial said, my point was only related to reflection of method lookup using a single string key. Even if you hash the types in your lookup, you still have to hash the types and then you can't have a simple method literal syntax. But to actually do a dynamic dispatch is not a simple hashing of types, it requires finding the best fit. For example:

class A {}  class B {}  class C : A {}

void foo(A)
void foo(B)
void foo(Obj)

obj->foo(new C())  // which version of foo gets called?

You actually have to examine the types of the parameters to find the best fit - I don't think you can do that with simple name mangling.

but that didn't impair a well-reasoned and positive comparison. Nice job!

Thanks for the feedback!

In other words, we can in two steps go from any type to any other type.

You only need 1 step in Scala: asInstanceOf. :-)

Auto-casting in Fan is useful precisely at the points where a programmer would insert a manual cast. But auto-casting has the advantage of cleaning up the code a lot. Fan code is much easier to read than Java or Scala.

For the particular language that Fan is, auto-casting is indeed the right tradeoff, and I don't believe it's significantly less safe (in day to day programming) than manual casting.

Now, the surest way to prevent cast errors is to forbid casting (ala Haskell), but to do that, you need a seriously strong type system. Fan isn't trying to be that. Scala is, but it's littered with junk like casting and null and other stuff.

I guess I should apologize for my blog posts if they've been troublesome. Still, thanks, Brian, for giving your opinions on what you think matters most. It's helpful to see your perspective, because it gives context.

@qualidafial

I think you misunderstood Brian's claim.

I think I did too. Very sorry, Brian!

@brian

Does that really matter? All the FFI does is specify a special syntax in the using statement. The nice thing about that is that it is clear if you are importing from Java or .NET. I assume that in Scala on .NET you use the exact same syntax when importing .NET namespaces as you do as importing Java packages. ...

I think it does matter. Interop is only as powerful as it is integrated. As I mentioned, in Scala, there is no disparity at all between Scala and Java: the two are essentially identical. This means that calling out to Java doesn't require speaking a separate language -- or, more specifically, using a separate language feature. Scala's Java interop is literally seamless, in every sense of the word. Fan on the other hand has a very good Java FFI, but it almost slaps you in the face with the fact that you're "shelling out" to another language. It works, but it's not as nice as I would like.

... I guess my personal taste is that explicit is better.

I guess we differ there, at least on matters of interop. :-)

@JohnDG

You only need 1 step in Scala: asInstanceOf. :-)

But it's explicit. That's the point I'm trying to make. Consider the following Fan pseudo-code:

Object bar := foo
AnyType baz := bar

In this code, foo can be of any type, and we've just implicitly mangled it into an arbitrary, and potentially unrelated type. This is assuming of course that Fan actually enforces casting within a hierarchy. If not, then we can simplify even further:

AnyType baz := foo

The type theorist in me is just cringing at this point. I can see the benefit to auto-casting in cleaning up cast-heavy code. For a language like Fan which rejects generics, such a feature would almost be a necessity. However, I really, really prefer to just avoid casting altogether. To me, casting is a sign that I was lazy in modeling my type signatures.

Now, the surest way to prevent cast errors is to forbid casting (ala Haskell), but to do that, you need a seriously strong type system. Fan isn't trying to be that. Scala is, but it's littered with junk like casting and null and other stuff.

Not really true. I almost never cast in Scala. When I do, it's usually just to make some stupid little prototype code work, and I'll come back to it later. Casting is the developer saying that "they know better than the type system". Well, with a really powerful type system (like Scala's), that almost never has to be the case. As long as you are using that type system appropriately, then all of that nasty casting goes away without any loss of type safety. Examples:

• Higher-kinded Map
• Using that map to avoid casting

In this code, foo can be of any type, and we've just implicitly mangled it into an arbitrary, and potentially unrelated type. This is assuming of course that Fan actually enforces casting within a hierarchy.

Fan does enforce casting within a hierarchy.

The point that Brian and I make is that the contrived example you show above doesn't occur very often in real-world coding.

The feature is used where a developer would just be inserting a manual cast anyway. This is why there is no significant loss of type safety.

What is the difference between:

Object bar := foo
AnyType baz := bar

and

Object bar := foo
AnyType baz := (AnyType)bar

aside from that one requires more typing than the other (as well as containing duplicated information on the typing of baz)?

I do grant there's one real danger of auto-casting: it can make developers lazy and just choose to use Obj everywhere (I've seen this in some Fan APIs), and to the extent that happens, static type safety takes a serious blow.

Not really true. I almost never cast in Scala.

Well, I never cast in Haskell. So there. :-)

As long as you are using that type system appropriately, then all of that nasty casting goes away without any loss of type safety.

My personal claim (without formal studies to prove anything) is that full static type safety provides little ROI as long as the runtime also provides dynamic type safety. Note that I do like some enforced type safety. A certain amount makes things more readable and improves performance and toolability.

Pure static and pure dynamic are both fine design decisions, too. But I think some folks see something philosophically wrong with taking a carefully planned middle road. I think a middle road can be fine, personally.

And without formal studies (and often even with), a lot really does come down to personal experience and opinion.

I do grant there's one real danger of auto-casting: it can make developers lazy and just choose to use Obj everywhere

I think that has more to do with lack of general-purpose generics than it has to with autocasting. I think some super lax generics could be nice, but I know that's not going to happen in Fan, so I don't fight it.

@JohnDG

Well, I never cast in Haskell. So there. :-)

:-) Fair enough. I still think that auto-casting only serves to make a dangerous language feature too convenient, but that may be more a personal opinion than an empirical fact.

@tompalmer

My personal claim (without formal studies to prove anything) is that full static type safety provides little ROI as long as the runtime also provides dynamic type safety.

I'm not convinced. Going from Java to Scala, I have found so many situations where I have saved literally hours because my code was correct by construction. Even really simple stuff like the following:

def sum(ls: List[Int]): Int =
  if (ls.empty) 0 else ls.head + sum(ls.tail)

This bit of code is quite respectable, and it's even more concise than the following:

def sum(ls: List[Int]): Int = ls match {
  case hd :: tail => hd + sum(tail)
  case Nil => 0
}

The key advantage to the second version is that the type system will statically guarantee that we are never going to try to access the head of an empty list. That's powerful. It's not hard to imagine how this advantage can be generalized to less contrived tasks.

For the record, I would never implement a sum function as given, it just serves as an example. :-) I would have actually done it like this:

def sum(ls: List[Int]) = ls.foldLeft(0) { _ + _ }

Also statically safe, but not as clear cut.

def sum(ls: List[Int]) = ls.foldLeft(0) { _ + _ }

                                        ^^^^^^^^^

Any language that accepts a moon smiley as valid code, has got to be good. :-)

I sort of like the implicit _ params in Scala. Pros and cons vs. the it of Fan, but overall (lacking tons of experience with either, admittedly), I like Fan's use of it better.

edit: deleted

both languages are statically typed alternatives on the JVM (unlike the dynamic languages such as Groovy....

Small correction: Groovy allows you to use static and/or dynamic typing.

Small correction: Groovy allows you to use static and/or dynamic typing.

I remember reading at some point that they don't generate optimized bytecode based on explicit variable types. And not because they didn't take to time to do that but because they ended up choosing semantics that didn't allow for it. In a quick skim of Groovy's docs on explicit types, I didn't see why they couldn't. Either my skim was too quick, the docs are long out of date, or my prior understanding was wrong. I'm not sure which of the three is the case.

>Small correction: Groovy allows you to use static and/or dynamic typing.

It's part static and part dynamic 8)

See Groovy Does Not Have Optional Static Typing

jfheon, thanks for the pointer.

jfheon

Small correction: Groovy allows you to use static and/or dynamic typing.

It's part static and part dynamic 8)

See Groovy Does Not Have Optional Static Typing

Point taken. Thanks!

@tompalmer:

When first seeing Scala (and having done Groovy in the past), I did like it better than _

But it fails. When reading

def sum(ls: List[Int]) = ls.foldLeft(0) { it + it }

the developer mind (at least mine) would assume it to be the same value / variable, but not _ in the case of

def sum(ls: List[Int]) = ls.foldLeft(0) { _ + _ }

because _ + _ is more like *.* where both * do not mean the same thing but are placeholders. It is not looking like a placeholder.

Cheers

Stephan

http://www.codemonkeyism.com

It should be called thingy instead of it :)

the developer mind (at least mine) would assume it to be the same value / variable

Understood, and in Fan it is the same variable. I had been referring to the whole package of semantics in each language on this matter, not just the syntax of it vs. _. My apologies on being unclear.

@djspiewak

def sum(ls: List[Int]) = ls.foldLeft(0) { _ + _ }

In this case, why wouldn't you use reduceLeft instead? I checked, and both work / give same answer...

scala> def sum(ls: List[Int]) = ls.foldLeft(0) { _ + _ }
sum: (List[Int])Int

scala> def sum2(ls: List[Int]) = ls.reduceLeft { _ + _ } 
sum2: (List[Int])Int

scala> var a = List(1,2,3,4,5) 
a: List[Int] = List(1, 2, 3, 4, 5)

scala> sum(a)
res3: Int = 15

scala> sum2(a)
res4: Int = 15

But IMO reduceLeft is conceptually easier because you don't specify the starting object.

@astubbs: You totally missed the point of djspiewak's first algorithms: The if-clause resp. the case Nil construct.

scala> def sum2(ls: List[Int]) = ls.reduceLeft { _ + _ }
sum2: (List[Int])Int

scala> val a = List[Int]()
a: List[Int] = List()

scala> sum2(a)
java.lang.UnsupportedOperationException: Nil.reduceLeft
        at scala.List.reduceLeft(List.scala:1093)
        at .sum2(<console>:4)
        at .<init>(<console>:7)
        at .<clinit>(<console>)
        at RequestResult$.<init>(<console>:3)
        at RequestResult$.<clinit>(<console>)
        at RequestResult$result(<console>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorI...

Clear indeed: What should reduceLeft return on an empty list? For foldLeft this is simply the seed.

Thanks for taking on Scala's modules! In general, I expect that by dropping static types, Fan will be able to further the frontier in modules. For example, virtual classes look useful for modules, but are murderous to type.

That said, please be aware that the Scala supports modularity in ways far better than Java packages. Also, modules certainly have been considered in Scala's history. To be brief, the Scala approach to modules is to try and make the regular language features usable for large-scale components. A module instance is, in Scala, an object. If you have a module that is parameterizable -- for example, a sorting module parameterized by the ordering function and the element type -- then you'd make either a trait on an abstract class that takes those parts as parameters.

Maybe try a harder challenge? One problem considered in Scala's history was the "extensible expression problem". Maybe try that in Fan and see how it goes? The challenge is to design a module of expressions and operations on expressions. Then, define sub-modules that add new types of expressions, new operations, or both. OO and functional languages are good at one kind of extensibility or the other, but it's really tough to figure out a way to support both.

Lex,

Thanks for swinging by and giving some background on Scala.

Maybe try a harder challenge? One problem considered in Scala's history was the "extensible expression problem".

The way I understand this problem is how to you jointly extend your data classes and operations on those classes.

One aspect of the problem is that you have to do this with compile-time safety. I don't buy this aspect of the problem - large software systems aren't compiled, they are assembled (typically from pre-compiled modules). I don't personally think a compiler is the right tool for managing the assembly of modules. So for discussion sake, let's take the compile time safety requirement off the table (which probably makes the problem much less interesting if you care deeply about static type systems).

The way Fan treats this sort of extensibility is with its type database. Effectively, we redefine the problem, instead of a typing problem we think about it as database problem. All the pods (modules) installed contain types, and those types have relationships which are captured by facets (annotations). This can be indexed into a database which lets us query relationships between types. What types and relationships are available aren't known until deployment time, but it makes for amazingly extensible software systems.

An example in Fan:

class Data {}

class Operation 
{
  static Operation findFor(Data d) { Type.findByFacet(@opOnData, d.type, true) }
  abstract Void operate(Data d)
}

@opOnData=DataA# OperationA { ... }  class DataA {}
@opOnData=DataB# OperationB { ... }  class DataB {}

Operation.findFor(someData).operate(someData)

We are using reflection, casting, and maybe dynamic typing - so we haven't solved the problem with compile time safety. But I can drop in new pods which define either new Data classes or new Operation classes and they will be glued together at runtime. So things are quite extensible.

In my experience a strong static type system works really well "in the small" down at the low levels of a software architecture. But as you rise up to the top of an architecture and the problem becomes modeling data and gluing modules together, static typing becomes a straight jacket. I like Scala's consistent philosophy on the issue, but I definitely have a different philosophy.

There is nothing you can do with reflection, casting, and dynamic typing, that can't be done with a sufficiently powerful static typing, except of course create typing errors, which are hidden bugs.

Before Scala, languages have not been sufficiently advanced to deliver on all the facets.

Dynamically (i.e. un- a/k/a uni-)typed languages have been widely thought to be a "breath of fresh air", relieving the tsuris and limitations of typing complexity, thus providing the flexibility needed to solve specific problems, and with less programmer man-hours. But only up until the required complexity, or runtime speed, of a project outstrips the weaknesses of an untyped language.

There are many reasons that a correctly designed typed language can actually achieve these gains and exceed the productivity of an untyped language.

To solve the Expression Problem, a language must allow new functionality to typesafely interopt with, and without forcing recompilation nor modification of, preexisting code.

Untyped Languages Don't Scale

Modules in untyped languages are easy to extend without recompilation, because they place no compile time constraints on data types. At runtime these unlimited types interact in unlimited ways, thus the implicit assumptions in preexisting and new code are unlimited, i.e. nothing is local and everything is a potential externality. Thus, when new code is run, new errors due to data type assumptions can manifest both in the preexisting code, as well as the new code. In other words, untyped languages initially feel great for speed of writing new code, but they can become error-proliferation Rigor mortis as the complexity of runtime extension interactions increase. This increase is exponential (Reed's Law again), i.e. complexity blows up. Thus the effort that must applied increases exponentially, eventually falling behind the productivity that would be possible with a correctly designed typed language. The only untyped solution is to limit extension to what is anticipated, i.e. ad-hoc, simulated typing at runtime:

http://existentialtype.wordpress.com/2011/03/19/dynamic-languages-are-static-languages/

Thus untyped languages are a beeline for solving specific problems fast, but they don't compose generally and are not generally extensible.

The fork between typed (Java, C++) and untyped (JavaScript, Python, Perl, Ruby, PHP, etc.) languages in the 2000s was necessary to fulfill the exponential increase in demand for customized and mainstream programming driven by the 2^n opportunities for interaction on the internet. The Java typing system was not sufficiently advanced[17] to handle this multifarious programming within the tradeoff of the 80/20 rule for both productivity and learning curve, so the only way to meet the demand for rapid and flexible design, was to forgo typing with an ad-hoc smorgasborg bandaid of unit tests, runtime exceptions, refactoring, and untyped composability programming paradigms, e.g. Ruby on Rails.

Those skimpy beelines expediently solve specific problems, but analogous to the Leaning Tower of Pisa with only 3 meters of foundation in a weak soil, they do not scale with composition, because:

* No untyped program can solve the Expression Problem.

* No untyped program can mix impurity with enforced purity (a/k/a referential transparency), required because no "real world" program is 100% pure, i.e. without memory.

* No impure program can compose generally, without causing a cascade of refactoring.

* No impure, and especially no untyped, program has the local determinism and thus separation-of-concerns (a/k/a SPOT).

* No impure, untrusted program be safely sandboxed.

* No impure program has a provable parallelism.

Thus to scale to Reed's Law, there needs to be a sufficiently advanced typed language with purity enforcement.

The Actor model addresses concurrency, not parallelism:

http://existentialtype.wordpress.com/2011/03/17/parallelism-is-not-concurrency/

Afaics, parallelism is much more important, because we are headed towards massively multicore (doubling # of cores every 18 months or faster).

Unfortunately, Scala is not pure (i.e. enforced immutability) either. Erlang and Fantom are only optionally immutable, which defeats the point. Fantom supports imperative statements, such as for, while, break, continue, etc.. A pure functional language should use tail recursion for all loops. It is impossible to not use a mutable variable or create side-effects when employing a loo. Fantom has a Void function output type, which is impossible in an immutable language, because such a function can only create side-effects. Fan even supports static fields, i.e. global state. That is the anti-pattern of referential transparency and principles of inversion-of-control, D in SOLID, SPOT, single-responsibility, etc:

http://lambda-the-ultimate.org/node/2678

http://gbracha.blogspot.com/2008/02/cutting-out-static.html

There are only three pure languages I know of Haskell, Clean, and Mercury:

http://james-iry.blogspot.com/2010/05/types-la-chart.html

And imho, none of those could ever go mainstream. They are too obtuse. Haskell can't even do multiple inheritance. Bob Harper argues (and I concur) that lazy is inferior:

http://augustss.blogspot.com/2011/05/more-points-for-lazy-evaluation-in.html#4642367335333855323

http://existentialtype.wordpress.com/2011/04/24/the-real-point-of-laziness/

I studied Filinkski's paper.

Fantom's type system has many corner cases, holes, and incomplete. Doesn't even support generics, must less higher-kinds:

http://fantom.org/sidewalk/topic/675#c4549

http://fantom.org/sidewalk/topic/1433

Higher-kinds are essential for implementing the standard library elegantly in a category theory model (i.e. not Scalaz complexity). If you want more on that, locate my post on stackoverflow.

Seems many people have thrown their hands in the air and taken Clojure for its simplicity, in spite of the LISP parenthesis proliferation maze.

I have a solution, but I won't mention it, since this blog page is about comparing Scala and Fantom. Sorry to be so negative, but as application developer involved with (or sole developer of) successful million user commercial software projects, I got tired of waiting (and begging) for someone to write the language I need. Lucky for me, Scala is very close, and can serve as an interim output target for the language I need.

Yeah! This is most clever robot-generated article I ever saw!

Couldn't resisit: xkcd

It is impossible to not use a mutable variable or create side-effects when employing a loo.

Best. Typo. Ever.

Not that I can comment intelligently on the deep issues of the post (and I even know Haskell!).

However, as I have programmed professionally for many years in each of Lisp, C++, Java, Perl and Ruby, I have some familiarity with the actual productivity gains of all the various languages.

I think there are theoretical gains possible with insert your favorite programming modality here but the average programmer is... average in whatever language you give him. I can track down the article of the guy who absolutely programmed rings around everyone he worked with for years and years, using Lisp. And so he believed that Lisp was incredibly productive, and everyone should use it... until he went to work for Google, and discovered that, wow, really good programmers are really good in whatever languages they like best. He was no longer a big fish in a small pond, and even C++ programmers were keeping up with him or doing better.

Another, in some ways more interest post I saw pointed out that it appears that some people think in a functional-programming way, and some don't, so forcing either kind of person to use the kind of language they're not mentally suited to is bound to fail.

So, yeah, best typo ever :-)

yachris, I guess there is a logical typo on the word or, and an extra not could make the statement more clear.

It is impossible to not use a mutable variable and not create side-effects when employing a loop.

Note those side-effects can be isolated to the function where they occur, and so the function can still be referentially transparent, even if it contains a loop. But you are correct, it encourages an imperative mindset, which makes it more difficult to feel comfortable programming in a pure functional, i.e. declarative.

I have some familiarity with the actual productivity gains of all the various languages

But you don't have a pure, eager language experience to compare to. Haskell is a lazy pure language, and laziness introduces space and time indeterminism which can impact productivity. Also Haskell doesn't have multiple inheritance. Haskell IO monad paradigm for interfacing imperative code is imho unnecessary complexity. Imperative code simply needs to go in to outermost callers of the program, i.e. functional reactive programming.

The key point is that wide-scale uncoordinated reuse of code without refactoring (what I call "compositionality") is impossible with imperative (side-effect) programming. Programming in a declarative manner will be much easier, and it makes more sense, because it enables compositionality, parallelism, etc.. Imho, it is just resisted as an accident of history, because no one has created a language with it simplified, unified, and without the unnecessary complexity.

This blog was about comparing Fantom and Scala. The main thing that set me off to comment with such a strong opinion, is that brian is asserting that modularity comes from reducing the power (specifically the "degrees-of-freedom", because dynamic typing is unityped) of the typing system. I disagree emphatically, and sorry about that. I don't wish to criticize anyone's work, especially when it is obvious that they are very talented.

A key point is that not representing a semantic (behavior) of module in a type and not isolating side-effects with immutability, does not make that semantic (behavior) disappear. Instead the semantic leaks across into other modules in a completely uncontrollable manner, i.e. indeterminism a/k/a bugs. There is a common slang for it, "spaghetti code". Someone wouldn't perhaps realize this if they didn't understand the Entropic force ( http://fqxi.org/community/articles/display/132 ), Coase's Theorem, and how this applies via Godel's theorem, the Halting theorem, etc.. I think people have enough experience with spaghetti code to know this is real phenomenon. Of course, we can't type everything, unless we want to prove all our programs with dependent typing (e.g. Epigram, Coq, etc), but imo Scala takes it to level just below full dependent-typing that is the peak in the cost vs. benefit trade-off.

http://james-iry.blogspot.com/2010/05/types-la-chart.html

Perhaps you can try to figure out where Fantom goes on that chart at the above link.

Also I noticed after writing my comment, that Fantom can compile to JavaScript. That is an impressive feature in the here and now. So I got to thinking that I am really talking about the future, because for the moment there is not pure, eager FP in the web browser. There is a valid argument about what is best for today, and what might be best for the future. I was talking more about where we need to go in order to get wide-scale bottom-up compositionality (i.e. modularity). It won't come from reducing the power of type system, employing runtime ad hoc typing, and calling that "modules".

Also there are many uses of programming where wide-scale compositionality is not the main benefit sought. I can not assert with certainty that will remain the case, but I can agree that is the case today.

yachris, I guess there is a logical typo on the word or, and an extra not could make the statement more clear.

Sorry, my childish sense of humor raising its ugly head... the typo in question was the phrase, "employing a loo." I know you meant "loop" as the last word there; however, "loo" is British slang for "toilet", so I couldn't resist. Sorry.

But you don't have a pure, eager language experience to compare to.

Well, so write one! No, really... these days, writing a language interpreter is astoundingly easy, compared to the old YACC/LEX days. You kids get off my lawn :-)

This, to me, is one of the most impressive things about Fantom; Brian and Andy not only wrote a pretty darn good language, they wrote a working darn good language, and then built a successful company around it. Awesome!

Their knowledge and understanding of type theory, etc., is way beyond mine, and I think it's also really cool that there's such an interesting group of people here discussing this stuff. Thanks for participating!

Programming in a declarative manner will be much easier,

Well, easier for you, perhaps. Maybe even me... but almost certainly not.

From 1985 through 1989, I was a full-time Lisp programmer (this was during the storied "AI Summer"). I taught (or attempted to teach) a number of people to program "the Lisp way". Some got it right off, some largely got it after a while, some didn't. And I didn't really understand why.

But the point about each person's individual ability and thought process is what the "academics" in language design REALLY don't understand.

When I was programming in Lisp, I conceived the theory that there is a constant number of Lisp programmers: for every tired professional saying, "FINE! We'll use Java! Have it your way!" there's some fresh-faced kid coming out of CMU ready to turn the world upside down with his awesome tool of choice.

Corollary: The development of Clojure wasn't possible until MIT stopped teaching Lisp to new undergraduates, thus freeing up Lisp programmers to be absorbed by the Clojure community :-)

So yeah, maybe you (or someone else) will come up with the ultimate one true "pure, eager language" (and let's add automatically parallelizing, since we're dreaming) and it will take over the world.

But allow me to predict that it will take over the world of programmers who think that way. The rest of us will muddle along with the tools we have, and like, and get a lot done with.

yachris, I thought you had misquoted me on the "loo" and didn't even go look at what I had written. :) Thanks for appreciating my input, and agreed great to see Daniel Spiewak and Lex Spoon who are well known by the Scala community. James Iry is prolific also, perhaps he is not aware of this blog.

Tangentially, imho, I don't think comparing experience using LISP or Haskell is indicative of whether you will ultimately find immutable FP easy. My current goal is I think there is a way to make pure FP easy for people familiar with the imperative C block style of programming.(1) But that wasn't the main point I wanted to make here, although side-effects are a necessary issue when discussing modular composition.

We are comparing two working languages, Fantom and Scala. Scala has taken a type theoretic approach, and from my limited understanding of Fantom, it appears to have an ad hoc typing approach. I am not at all criticizing the talent of the Brian and Andy. Imo, Scala also has some weaknesses, which is why I pointed out some tangential points about immutability, and compared to Haskell and its weaknesses. My main point here is I don't think this module direction for Fantom is going to achieve wide-scale modularity.

I want to reiterate that my reason for commenting on this page was not primarily about pure FP (i.e. immutable, declarative programming), instead was to disagree with the assertion that untyped modules are modular. Modularity only comes from typing, because semantics (behavior a/k/a invariants) of a module do not cease to exist, when they are not typed. And semantics have to be compatible (i.e invariants enforced, e.g. LSP invariants), in order for modules to interopt without runtime error.

Tangentially this is why I disagree with structural subtyping (an optional Scala feature), because it violates the semantic covariance of the Liskov Substitution Principle. A Set.count() does not have the same semantic meaning as an Iterator.count(). Instead I plan to support partial interfaces, i.e. instead of matching on the type signature of count, match on Set.count or Iterator.count.

Typing is the way of encoding denotational semantics, and enforcing compatibility at compile-time. The tradeoff w.r.t. productivity is the extra effort to shoehorn the semantics into compatible types, versus the eventual degradation of modular composition (i.e. spaghetti code) when such extra effort is not applied to fully type the modules. Brian refers to the granular modularity of typing as a "straight jacket", but this is not necessarily true where types are correctly designed. Typing can actually make it easier to reuse functionality. Afaics, the problem has been ad hoc denotational semantics without any category theory. When we use category theory to design our types, we get benefits of easier extension. For example, the Applicative(1), enables us to automatically lift all functions that operate on types of kind 0 (i.e. non-parametrized types, e.g. Int, String, etc), to functions that operate on functions that operate on higher-kinds, e.g. a function Int -> Int -> Int = (x,y) -> x + y, automatically (no code smell, no boilerplate, needed) becomes a function List[Int] -> List[Int] -> List[Int], and every other kind of type you can make from an Int type parameter. It is almost like magic. It also applies to a type parametrized function of kind 0, e.g. [T >: AddMethodInterface] : T -> T -> T = (x,y) -> x + y. Thus the need for higher-kinds(2), which Fantom doesn't have, but Scala does, yet Scala still has some code smell(2).

Also the way type extension is done is critical. Scala employs implicits and view bounds, which means if a caller has a type NonExtended, and the function (needs an Extended so it) takes a NonExtended with an Extended view bound, then the two can interopt. This is not quite sufficient for the wide-scale uncoordinated modularity that we want. We need for the compiler to generate all permutations of possible view bounds automatically and implicitly (so we never are bothered with this noise).

With those things correctly implemented, the interface (trait in Scala) becomes the modular unit with packages for scoping, then the "straight jacket" should diminish. Then the only reason to avoid typing would be to avoid modularity and embrace escalating spaghetti as module composition scales. So thus I am taking issue with the assertion that encapsulating some semantics into a set of modules with partial ad hoc typing, are anything but avoiding wide-scale modular composition. I agree with brian that wide-scale module composition is not easy. But unless I misunderstand his module solution, it appears he is saying to give up on typing, which is the antithesis of modularity. Typed modularity is not easy if the typing system is not sufficiently powerful (e.g. higher-kinded) and if category theory has not been employed in the design of the types.

Brian mentions not supporting overloaded method names, because of the cost to efficiency for dynamic dispatch. When I was exploring the way to optimally do extension (see prior paragraph on view bounds), I researched vtables(3) and realized that the way to avoid hashing the function signature strings, is to store a copy of of the method pointers for each interface, so that interface dispatch is always an indexed lookup. Obviously this optimization isn't possible with dynamic typing, where function signatures are not known at compile-time, so perhaps Fantom can't use it where dynamic typing and reflection is employed. Note this optimization is necessary because due to linearized multiple inheritance (mixins), the order of interfaces in the vtable isn't determined until the class's total hierarchy is known, because if trait A extends C with D with E and trait B extends D with C, then class F extends A with B would effectively reverse the order of linearized inheritance of and insert B into trait A extends E with B with D with C.(4)

writing a language interpreter is astoundingly easy

btw, I decided to write my own implementation of a LL(k) parser generator. :) Because I couldn't shoehorn JavaCC elegantly, couldn't accept a closed source implementation (e.g. SLK), nor did I want to use those LR grammar generators, nor would I accept parser combinators (they are not provably context-free, nor as fast as table lookup).

(1) http://copute.com/dev/docs/Copute/ref/std/

(2) http://www.codecommit.com/blog/ruby/monads-are-not-metaphors#comment-5453

(3) http://www.research.ibm.com/people/d/dgrove/papers/oopsla01.pdf , Section "4.4 Multiple Inheritance in C++"

http://stackoverflow.com/questions/1504633/what-is-the-point-of-invokeinterface/1505476#1505476

(4) "Why not multiple inheritance?" in chapter on Traits in Programming in Scala.

I don't know much about type theory(though I'd love to know more), but I do know two things:

1. Efficency depends on the people using language not the language itself. You could write every piece of software in assembler. In fact I've seen an operating system written entirely in one.
2. Good language doesn't clash with developer's mental model and consequently takes little time to learn/master. If each time I need to write a type I need to consult a thick book on type theory... than no thanks. Trap operator is mostly a shortcut ->. I view it in same lieu as JavaScripts lax typing (or any other feature). When used correctly it works well. Which leads me too...
3. No amount of language will protect programmers from themselves. Any system designed by humans will be misused by humans.

Speaking of Scala's trait, if you are really keen on purity of types then for the love of god don't use linearized traits. To me that sounds like a great way to lose any vestige of sanity (Ball with Red with Shiny != Ball with Shiny with Red).

DanielFath,

You won't need to know type theory in order to use an intuitively designed standard library based on higher-kinded types.

For example, unfortunately Scala can't encode category theory typeclasses without repeating boilerplate code(2). But with my language or let's think of it as a preprocessor for Scala(1), if you want to create a new class, e.g. MyClass[T], that for example implements an Applicative, then you only need to define two methods in your class, apply[A] : Sub[T -> A] -> Sub[A] and static lift : T -> Sub[T], and inherit from the mixin ApplicativeMap[MyClass,T]. The [MyClass,...] type parameter is telling ApplicativeMap the type of the class that is being created, and this is the higher-kinded generics feature of Scala. The lift function is the unit in the Scala example given(2), and is simply a factory method that constructs an instance of your class, Sub[T] (where you have declared via the higher-kinded feature that Sub is MyClass) from an instance of the type parameter T of your class. For example, if your MyClass[T] is actually a List[T], then Sub is List and T are all the types that can go into your list. You are familiar with T from Java, it is called generics or type parametrization. The apply method inputs a function, T -> A, wrapped inside an instance of your class, and outputs an instance of your class with all the T mapped to A, by calling that function on each T in your class.

By inheriting from and implementing those two methods of Applicative, now your class has a special power. Any function in the entire world that operates on instances of T, can now be automatically applied to operate instead of instances of your class. So for example, let's say there exists a function, add[T >: HasAddMethod]( a : T, b : T ) = a + b, then it can automatically be called as if it had the type signature, MyClass[T] -> MyClass[T] -> MyClass[T]. So in other words, if you had two lists and you wanted a list of the elements added, then you could call the add function on your two lists. You won't actually be writing all those types down. You will simply write:

class MyClass[T] inherits ApplicativeMap[MyClass,T]
{
   Applicative.lift = ...your implementation...
   apply = ...your implementation...
}

1. I wrote most of WordUp (one of the world's first WYSIWYG full featured word processors back in 1980s) in assembly language (about 5 megabytes of source code) before I discovered C. The difference in productivity of switching to C was no less than a factor of 10.
2. Agreed. That is why I provided the example above, to show you that higher-kinded typing doesn't have to interfere, it can actually aid in reuse in a very intuitive and easy-to-understand way. Unfortunately imo, no one has bottled up monads and applicative in a way that is easy and intuitive. I decided to do something about that. I have always believed that programming should be easy, simple, and fun. It requires the language and library designer to be more clever about considering the learning curve of the users. I respect that language designers of Fantom or Kotlin, are trying achieve K.I.S.S..
3. I hear you, but I mostly disagree even though your point applies especially in poorly typed languages. I provided the example above, which will type check and keep you from doing something that breaks the semantics. And I am working on an immutable language that won't let you use loops. But ultimately, too much talk here is meaningless. A working language with many users is the real test. Also again I want to reiterate that my original objective was to point out in this blog, that not typing modules is not a direction to achieve wide-scale modular composition. I wasn't primarily making the point about nuances of type theory. But I guess it is impossible to make that modularity point, without then discussing the potential tsuris of typing. I don't know Fantom well, but from the brief look at it, it appears to be what I was originally intending to create for a language, where it would be possible to mix dynamic and static typing, and mix mutable and immutable coding styles. As I worked through it over past year or so, I refined my opinion to conclude that was not the optimum design to attempt. Fantom, with its mix of intuitive features, might be ideal for certain real-world applications (e.g. apparently this blog is written in Fantom, including the client-side JavaScript!), my main point was to argue against a non-typing as being applicable to wide-scale modular composition. Any way, as you imply, what users adopt is the real world test. I hope I have helped to provide a little detail that can be used to contemplate tradeoffs. There are probably angles of consideration that can be presented for and against any language design decisions.

(1) http://copute.com/dev/docs/Copute/ref/std/Applicative.copu

(2) http://www.codecommit.com/blog/ruby/monads-are-not-metaphors#comment-5453

@Shelby Did you make any progress since 2011?

If there could be a language that provides me compile time guarantees of many failures including type errors, with minimal to no extra effort, then it would be ridiculous not to use it over similarly expressive, simple, easy, fun and performant languages.

The problem is, I've never come across such a language yet. Haskell has difficult time and space complexities to work with, it can be argued to offer a more difficult and complex usage, and it doesn't provide as rich and battle tested an ecosystem as say the JVM.

In the meantime, I find Fantom to offer the right set of trade-offs, to keep the most useful type safety, while enabling a rapid and enjoyable development with a full suite of pragmatic production oriented ecosystem.

But, since you were talking about working on a language which had the above goal in mind, I'm curious to ask, if you've made any progress or have found it too difficult?

@didbus yes just today in fact ZenScript coding was launched. And I've learned so much since then.