#348 Bug: Compiler cause OutOfMemoryError

cgrinds Tue 26 Aug 2008

An OutOfMemoryError was reported by wirianto that I found/fixed. I suspect this is the same issue katox is seeing too.

There are two issues:

  1. (On Windows) I noticed that if you're in the c:\apps\fan\bin directory and you run any of the executables (while in that directory) you'll throw an OutOfMemoryException.
ERROR: cannot compile script
sys::Err: java.lang.OutOfMemoryError: Java heap space
  compiler::TokenVal.make (TokenVal.fan)
  compiler::TokenVal.make (TokenVal.fan)
  compiler::Tokenizer.symbol (Tokenizer.fan:848)
  compiler::Tokenizer.find (Tokenizer.fan:132)
  compiler::Tokenizer.next (Tokenizer.fan:86)
  compiler::Tokenizer.strInterpolation (Tokenizer.fan:462)
  compiler::Tokenizer.str (Tokenizer.fan:376)
  compiler::Tokenizer.find (Tokenizer.fan:122)
  compiler::Tokenizer.next (Tokenizer.fan:86)
  compiler::Tokenizer.tokenize (Tokenizer.fan:67)
  compiler::Tokenize.tokenize (Tokenize.fan:82)
  compiler::Tokenize.runSource (Tokenize.fan:71)
  compiler::InitInput.run (InitInput.fan:71)
  compiler::Compiler.compile (Compiler.fan:60)
  compiler::Main.compileScript (Main.fan:207)
  sun.reflect.NativeMethodAccessorImpl.invoke0 (Unknown)
  sun.reflect.NativeMethodAccessorImpl.invoke (Unknown)
  sun.reflect.DelegatingMethodAccessorImpl.invoke (Unknown)
  java.lang.reflect.Method.invoke (Unknown)
  fan.sys.Method.invoke (Method.java:515)
  fan.sys.Method$MethodFunc.call3 (Method.java:302)
  More...

This happens because when you run, for example, flux.exe, what actually gets passed to the OS is:

C:\apps\fan-1.0.31\bin\flux.exe flux

Notice flux is an argument here, which means when running in the bin directory the bash script named "flux" will be read/parsed/executed.

Which leads to the 2nd problem.

  1. The compiler has a bug, when flux,fan,fansh (or any of the bash scripts) are read they contain a string that trips up the compiler. I distilled that string into a simple example here:
class Bug_Compiler {
  static Void main() {
      Str toCompile := "${/*" 
  }
} 

The Tokenizer sits in a while(true) loop adding tokens to a list, eventually consuming the entire Java heap because the right brace is never found in the string interpolation. The fix is simple:

=== modified file 'fan/parser/Tokenizer.fan'
--- fan/parser/Tokenizer.fan    2008-08-26 18:26:21 +0000
+++ fan/parser/Tokenizer.fan    2008-08-26 18:26:47 +0000
@@ -460,7 +460,7 @@
       {
         if (cur === '"') throw err("Unexpected end of string, missing }")
         tok := next
-        if (tok.kind == Token.rbrace) break
+        if (tok.kind == Token.rbrace || tok.kind == Token.eof) break
         tokens.add(tok)
       }
       tokens.add(makeVirtualToken(Token.rparen))

Also FYI - I've never been able to run the fan.exe since it depends on .NET which I don't have installed. I'm not a huge fan of launcher anyway so I've never much cared - I typically create my own batch files by sucking the contents out of the bash files. I prefer a batch file so I can exactly control the cmd line args, makes it easier to add/remove debug options, profiler options, -Ds, etc.

Anyway I just wanted to mention that others on Windows may also have Java but not .NET - the error you get in that case is:

The application has failed to start because mscoree.dll was not found. Re-installing the application may fix this problem

helium Tue 26 Aug 2008

Anyway I just wanted to mention that others on Windows may also have Java but not .NET

So what?

cgrinds Tue 26 Aug 2008

Hi helium, On Windows if .NET isn't installed - then you won't be able to run Fan out of the box. Was that not clear in my ramble? Or are you saying "so what if Fan doesn't work out of the box" ?

brian Tue 26 Aug 2008

Boy - those are two big problems that are really embarrassing. They both have the potential to make the out of the box Fan experience crappy - that is the last thing we need.

On the compiler infinite loop bug - that makes perfect sense to what some people have been seeing. I checked in a fix along with a test suite for all the end-of-file cases I could think of.

On the fan launcher - it was our intention that the launcher load the .NET DLL all reflectively thru COM so that the launcher would work for Java just fine even if .NET wasn't installed. At some point in the distant past I actually verified this - but it appears that we broke something. We clearly need the launcher to work perfectly even if you don't have .NET installed - so Andy is going to take a look at that.

Thanks for your feedback Chris!

katox Wed 27 Aug 2008

An OutOfMemoryError was reported by wirianto that I found/fixed. I suspect this is the same issue katox is seeing too.

cgrinds is right (and fast!). I rebuilt Fan with his patch applied and this out-of-memory problem went away. The error message now is

flux(9,2): Unexpected end of string
ERROR: cannot compile script

which is correct, I guess. I wouldn't think of a infinite loop in a compiler I thought it were just some arrears in flux skeleton...

This was on jdk6/Linux box, BTW.

helium Wed 27 Aug 2008

@cgrinds: Sorry, got something wrong, never mind.

andy Wed 27 Aug 2008

The application has failed to start because mscoree.dll was not found. Re-installing the application may fix this problem

Yeah, I started to fix that awhile ago, but never finished, and seems to have gotten swept under the rug. Sorry about that, this will definitely be fixed for the next build.

@cgrinds - would you mind emailing me, so I can send you a fan.exe to test to make sure I fixed correctly?

andy Thu 28 Aug 2008

I fixed the launchers to work correctly if .NET is not installed - they'll be in the next build.

Login or Signup to reply.