#875 Add 1-step bootstrap script

tactics Thu 17 Dec 2009

I lost a laptop last week, and so I needed to get a brand new Fantom development environment set up. It was frustrating and it took me about 6 hours to get working! I ran into tons of problems, and I want to detail them here in the hopes we can help clean the setup process up a bit.

For the record, this is all on Windows. I'll try it again on Linux later this week if I have the time.

Mercurial

First, I ran across an issue cloning the hg repository. It was funny. I copied the error message from the console into Google, and the second link was good old Sidewalk (topic #515). I tried it over again all night, and it kept aborting. Luckily, I was able to use Andy's bundle. Unluckily, I was unaware that you need to manually specify the URL to pull. For most of the night, I was working off of old code, and it was only thanks to the change from Fan->Fantom that I realized something was off. Regardless, I was getting similar errors on the old revision and the HEAD.

Setup

The setup for Fantom development was a bit irritating. It was early in the night, and I wasn't being careful, but I managed to make every mistake possible. The documentation is just a little out of date. You're dealing with four config files (2x sys.props and 2x etc/build/pod.fansym) and two of everything else. I think the setup could be greatly simplified with the use of a few well-placed scripts.

Path Names in sys.props

I found out the hard way that C:\Documents and Settings\Michael\fantom was a bad place to put my code. On Windows, launcher.cpp (around line 200) isn't able to handle spaces in paths to the substitute fan.exe. So I had to start over again, moving all my code over to another directory. I changed all the paths to be D:\\data\\projects\\fantom\\fantom-1.0.47... only to find out I had to change the source files on the LHS as well. It's obvious in retrospect, but when you're meeting barrier after barrier, it gets frustrating.

Buildpods.fan

It's been a few hours now. I finally get to run the buildall script. Things were terrible for a long time, but it was due to me using an outdated Mercurial repo, as I explained above. But after I fixed that, still more things went wrong.

Buildall said that everything compiled fine. I tried running flux, and it gave me the strangest error:

D:\data\projects\fantom\fan-1.0\src>..\bin\fan flux
sys::UnknownTypeErr: fwt::WidgetPeer
  fan.sys.Pod.findType (Pod.java:268)
  fan.sys.FanClassLoader.findClass (FanClassLoader.java:191)
  ... (cut some of the lines) ...
  fanx.tools.Fan.execute (Fan.java:39)
  fanx.tools.Fan.run (Fan.java:242)
  -1 More...

I'm not sure how you end up with -1 stackframes, but whatever. I ran buildpods by itself. Same thing. I cd'ed to the src/fwt directory and ran build.fan manually. Same thing. Finally, by accident, I tried running fwt/build.fan from the src/ directory and it worked. Flux ran fine. I realized I also had to compile web, email, testNative, flux, and inet the same way in order for them to pass their fant tests.

The take home messages are that there's some dependency on the current working directory on the fwt build script and that buildpods isn't working correctly for some reason. I don't know how either work well enough to investigate.

So finally, I got my installation working.

It was painful and frustrating, even for someone who has been on the project as long as I have. Thankfully, there's lots of room for improvement. Here are my thoughts:

  • Update the documentation on the website
  • Fix buildpods.fan script
  • Correct the limitations on path names in sys.props
  • Remove the cwd-dependency on individual pod build scripts
  • Create a One-Click setup script*
  • Eventually, I'd love to do away with fansubstitutes entirely

One-Click setup script

For speed of setting up new development environments and getting new developers ready to go, I'd like to see a One-Click setup script distributed with the releases. What I mean by that is a script or pod which automatically configures and installs a working bootstrapped development environment. You might call it:

$ fan setup bootstrap C:\fantom\devel

And it would go through all the steps:

  1. hg clone http://hg.fantom.org/repos/fan-1.0 C:\fantom\devel
  2. Configure C:\fantom\devel\lib\sys.props
  3. Configure C:\fantom\devel\etc\build\pod.fansym
  4. Run the buildall script with tests
  5. Run hello world for good measure or something.

We could filter the output of all the compilation to a file. If there was an error, we could have the developer email us the file to help figure out what went wrong.

Like I said, if I have time later this week, I will try to get it set up on Linux. Hopefully things will go smoother now that I've tripped over just about every possible problem.

DanielFath Thu 17 Dec 2009

After coming through same hassle I have to concur. But the ticket name is a bit misleading.

Anyway how come the bootstrap script only has one parameter? Last time I checked for bootstraping you need two distro dev and rel. And fan should execute from the dev folder, if I recall correctly.

brian Thu 17 Dec 2009

@tatics

I agree bootstrap is painful - it was never my original intention that anybody did this but Andy and I. Although now that many people are working off hg tip it probably behooves us to make this easier.

I don't think documentation is really the issue, because no matter how much I keep adding to that chapter it doesn't seem to make the pain go away :-) I thought I had fixed all the doc problems though - what did you find wrong?

The space in the path issue already has a ticket #525.

I am not sure what is wrong with buildpods and cwd-depends. There is an ordered dependency chain in buildpods, but I am not sure exactly what problem you ran into.

I think a one-click script would be nice, and would alleviate a lot of pain. The only question is what to write it in? Batch/bash won't cut it because a) batch sucks ass and b) we want a single portable script. Would a python script suit everybody?

andy Thu 17 Dec 2009

Seems like the easier process there is to include that script in the builds, then you can write it in Fantom:

  1. Download latest Fantom build for rel
  2. Run bootstrap.fan to setup dev

DanielFath Thu 17 Dec 2009

^This. Preferably there should be some way to coerce rel to extract its stuff in the dev folder without having to set that up.

tactics Thu 17 Dec 2009

I don't think documentation is really the issue, because no matter how much I keep adding to that chapter it doesn't seem to make the pain go away :-) I thought I had fixed all the doc problems though - what did you find wrong?

I think all the information is probably in the doc, but the way it's organized makes it hard to follow correctly. For example, there's a line states that you should include only <buildDevHome>/bin in your path. But it's the second to last line of the file, instead of the second from the top, like it ought to be :-)

I am not sure what is wrong with buildpods and cwd-depends. There is an ordered dependency chain in buildpods, but I am not sure exactly what problem you ran into.

I'm not sure exactly what problem I ran into either. Buildpods was acting very weird, saying it built everything successfully, installing the pods, but then things were broken (it looks like native code wasn't compiling correctly).

The cwd-dependency had to do with the Java compilation step. When running build from the wrong directory, it looks like javac couldn't find the fan.gfx package. (Of course, this led me to being totally confused when I grepped the entire distribution only to find there is no file with the words package fan.gfx!) It looks like somewhere in the Java builder, a relative path is used to walk over the jars needed to build it.

I think a one-click script would be nice, and would alleviate a lot of pain. The only question is what to write it in?

Andy understood how I meant to have the script. We here at Fantom love the taste of our dogfood ;-) The procedure for new devs would be:

  1. Download the latest stable release
  2. Set up the runtime (almost a non-step)
  3. Run a bootstrap.fan script, passing in where you want it installed
  4. Sit back and wait for everything to build and test itself

tcolar Thu 17 Dec 2009

Kinda agree the current process is painful. Python script would be ok but not sire it's too common on a windows platform.

An alternative would be to use my brand new Fantom to Jar tool :) to make a FantomBuilder.jar, then you could just run java -jar FantomBuilder.jar to build the source code (would be ~1MB) Of course that requires Java, but I guess it's even more widespread than python ?

The nice thing is that there would be no risk of getting mixed up between the release runtime (in the jar) and the one being built.

Anyway just an idea, otherwise Andy's proposal is fine as long as you make it easy to not get mixed up between rel and dev (FAN_HOME and the like).

Either way, it could be simpler :)

brian Thu 17 Dec 2009

Renamed from Updating bootstrap docs to Add 1-step bootstrap script

brian Thu 17 Dec 2009

Promoted to ticket #875 and assigned to brian

lbertrand Thu 17 Dec 2009

I like the idea of it being a Fantom script

tactics Fri 18 Dec 2009

I tried the buildpods.fan script again today. It doesn't look to be a cwd dependency after all. But I found that if I build gfx first manually and then fwt manually, it fwt starts passing its tests.

I noticed that there are a few implicit dependencies between pods. Pod gfx requires compilerJs and flux requires icons. We might want to include these in their pod.fan files or (maybe for gfx) change the build script to only build the Javascript natives if compilerJs is installed.

The build script will require changing the current Repos etc/build/pod.fansym so all the build scripts work. Is it possible to alter a symbol's value during the execution of the script? The docs say that symbol values are reloaded automatically when they are changed, but I'm kinda iffy about how it's implemented. And beyond that, there doesn't seem to be a standard method to update the value from within the program.

DanielFath Fri 18 Dec 2009

Yeah, this question bugged me too. My is that since they are making it, they'll probably turn it into something that can be changed from the program or they'll use the old fashioned way.

brian Sun 20 Dec 2009

Ticket resolved in 1.0.48

I have added a script called "adm/bootstrap.fan" which:

  1. Verifies your environment
  2. Clones hg tip or if hg repo exists performs a pull/update
  3. Configures your etc files
  4. Rebuilds your devHome environment from scratch

Won't be able to test this script fully until after the next full build (so might take a couple builds to work the kinks out).

brian Tue 22 Dec 2009

Another fix for the script to make it work with bash script - changeset. Andy said this version of the script worked on his OS X machine.

katox Thu 31 Dec 2009

There is something fishy in the bootscript ... not sure what maybe process spawning / concurrency bug in the core?

I got this from two subsequent runs:

katox@tso:~/git/fantom/dev/adm (fuel)$ fan bootstrap.fan 
[13:18:45 31-Dec-09] [err] [bootstrap] Cannot boot
sys::NullErr: Coerce to non-null
  fan.sys.NullErr.makeCoerce (NullErr.java:38)
  bootstrap_0::Bootstrap.execToStr (/home/kato/git/fantom/dev/adm/bootstrap.fan:225)
  bootstrap_0::Bootstrap.initEnv (/home/kato/git/fantom/dev/adm/bootstrap.fan:70)
  bootstrap_0::Bootstrap.run (/home/kato/git/fantom/dev/adm/bootstrap.fan:51)
  util::AbstractMain.main (AbstractMain.fan:368)

then (changed nothing in fan script or environment or elsewhere):

katox@tso:~/git/fantom/dev/adm (fuel)$ fan bootstrap.fan 

Bootstrap Environment:
  repo:      http://hg.fantom.org/repos/fan-1.0
  hgVer:     Mercurial Distributed SCM (version 1.3.1)
  jdkVer:    javac 1.6.0_15 (need 1.6+)
  jdkHome:   file:/usr/lib/jvm/java-6-sun-1.6.0.15/
  relVer:    1.0.48
  relHome:   file:/home/kato/git/fantom/dev/
  devHome:   file:/home/kato/git/fantom/fan/

Continue with these settings? [y|n] y

hg clone http://hg.fantom.org/repos/fan-1.0 /home/kato/git/fantom/fan

liamstask Tue 5 Jan 2010

Running the bootstrap today on OS X, I'm getting an OOM error:

js [testSys]
  Delete [/Users/liam/Documents/mtcode/fan/fan/src/testSys/temp-js/]
  CreateDir [/Users/liam/Documents/mtcode/fan/fan/src/testSys/temp-js/]
  Compile [testSys]
    FindSourceFiles [56 files]
ERR: Internal compiler error
sys::Err: java.lang.OutOfMemoryError: Java heap space

Just a little annoying, but would be nice if it ran cleanly all the way through.

andy Tue 5 Jan 2010

Running the bootstrap today on OS X, I'm getting an OOM error:

Are you synced up to tip? That was a bug in compilerJs that should be fixed now.

liamstask Tue 5 Jan 2010

Pretty sure - I'm running it through the bootstrap script, which syncs up to the latest tip. I have 1.0.48 as my rel installation, and I'm running

./rel/bin/fan ./rel/adm/bootstrap.fan

brian Tue 5 Jan 2010

Might want to check your default stack size. On Windows's I have the default stack size cranked up to 128M because 64M is too small.

On OS X/Linux you'd have to set this yourself in fanlaunch.

helium Tue 5 Jan 2010

Huh? If the stack is to small you get an stack overflow error, normally because of too deep recursion. If anything he should increase the heap size.

brian Tue 5 Jan 2010

If the stack is to small you get an stack overflow error, normally because of too deep recursion. If anything he should increase the heap size.

Sorry, that is what I meant, heap size. For Windows we use:

-Xmx128M

liamstask Tue 5 Jan 2010

Eh - that got me a little farther, but it popped up again a little later when called from one of the native Java portions (note that doesn't have the 128M heap set):

javaNative [gfx]
  Exec [/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home/bin/java -cp /Users/liam/Documents/mtcode/fan/fan/lib/java/sys.jar -Dfan.home=/Users/liam/Documents/mtcode/fan/fan fanx.tools.Jstub -d /Users/liam/Documents/mtcode/fan/fan/lib/java gfx]
    Java Stub [gfx]
  js [gfx]
    CreateDir [/Users/liam/Documents/mtcode/fan/fan/src/gfx/temp-js/]
    Compile [gfx]
      FindSourceFiles [17 files]
ERR: Internal compiler error
sys::Err: java.lang.OutOfMemoryError: Java heap space

really feeling like I must be missing something silly...

brian Tue 5 Jan 2010

did you increase it in both rel and dev environments?

also try to bump it up to 512M, 1GB, etc

liamstask Tue 5 Jan 2010

Right - had forgotten to do it in both. Thanks.

Upped to 512MB in both and all was well.

DanielFath Thu 28 Jan 2010

First a proposal. Since my mercurial tends to cause some weird problems, I usually update it beforehand, so I don't need bootstrap to hg pull for me. Thus I propose an option such as -skip or -s that skips the pull/update command. There are probably other use cases in which you might not want bootstrap to update your relHome.

Secondly I have another problem using 1.0.49 to perform bootstrap.

Bootstrap Environment:
  repo:      http://hg.fantom.org/repos/fan-1.0
  hgVer:     Mercurial Distributed SCM (version 1.3.1+7cea12e70129)
  jdkVer:    javac 1.6.0_17 (need 1.6+)
  jdkHome:   file:/C:/dev/lang/jdk1.6.0_17/
  relVer:    1.0.48
  relHome:   file:/C:/dev/lang/fantom-1.0.49/
  devHome:   file:/C:/dev/lang/fantom/

Continue with these settings? [y|n] y

hg pull -u http://hg.fantom.org/repos/fan-1.0
pulling from http://hg.fantom.org/repos/fan-1.0
searching for changes
no changes found

Config etcs:
  C:\dev\lang\fantom-1.0.49\etc\build\pod.fansym: buildDevHome=`file:/C:/dev/la
g/fantom/`
  C:\dev\lang\fantom-1.0.49\etc\build\pod.fansym: buildJdkHome=`file:/C:/dev/la
g/jdk1.6.0_17/`
  C:\dev\lang\fantom\etc\build\pod.fansym: buildJdkHome=`file:/C:/dev/lang/jdk1
6.0_17/`

C:\dev\lang\fantom-1.0.49\bin\fan.exe C:\dev\lang\fantom\src\buildall.fan superc
lean
Delete [file:/C:/dev/lang/fantom/lib/fan/]
BUILD SUCCESS [13ms]!

C:\dev\lang\fantom-1.0.49\bin\fan.exe C:\dev\lang\fantom\src\buildboot.fan full
clean [sys]
compile [sys]
  Compile [sys]
    FindSourceFiles [81 files]
    WritePod [file:/C:/dev/lang/fantom/lib/fan/sys.pod]
test [sys]
  Exec [C:\dev\lang\fantom\bin\fant.exe sys]
ERROR: Cannot find Java main fanx/tools/Fant
ERR: Exec failed [C:\dev\lang\fantom\bin\fant.exe sys]
BUILD FAILED [1230ms]!
##
## FATAL: src/buildboot.fan full failed
##

Note: I did change the source of the bootstrap so it defaults to fantom instead of fan. Besides this bug appeared even when I used to set home using the -devHome option.

brian Thu 28 Jan 2010

beforehand, so I don't need bootstrap to hg pull for me. Thus I propose an option such as -skip or -s that skips the pull/update command.

If you are in sync, it should do nothing. I typically do this too. So you don't even want it to check that it is in sync?

Secondly I have another problem using 1.0.49 to perform bootstrap.

You need grab the latest version from hg. That has a fix now that "full" includes "test" by default.

DanielFath Thu 28 Jan 2010

If you are in sync, it should do nothing.

Theoretically, yeah. Thanks to my erratic Internet, it sometimes can't update causing the whole build to puff up in smoke making me wait until my internet is steady enough (usually around 12 a.m.).

You need grab the latest version from hg. That has a fix now that "full" includes "test" by default.

Ok.

C:\dev\lang\fantom-1.0.49\adm\bootstrap.fan(76,21): WARN Deprecated type 'sys::S
ys'
C:\dev\lang\fantom-1.0.49\adm\bootstrap.fan(127,9): WARN Deprecated type 'sys::S
ys'
C:\dev\lang\fantom-1.0.49\adm\bootstrap.fan(128,17): WARN Deprecated type 'sys::
Sys'
C:\dev\lang\fantom-1.0.49\adm\bootstrap.fan(196,17): WARN Deprecated type 'sys::
Sys'

Bootstrap Environment:
  repo:      http://hg.fantom.org/repos/fan-1.0
  hgVer:     Mercurial Distributed SCM (version 1.3.1+7cea12e70129)
  jdkVer:    javac 1.6.0_17 (need 1.6+)
  jdkHome:   file:/C:/dev/lang/jdk1.6.0_17/
  relVer:    1.0.49
  relHome:   file:/C:/dev/lang/fantom-1.0.49/
  devHome:   file:/C:/dev/lang/fantom/

Continue with these settings? [y|n] y

Config etcs:
  C:\dev\lang\fantom-1.0.49\etc\build\pod.fansym: buildDevHome=`file:/C:/dev/lan
g/fantom/`
  C:\dev\lang\fantom-1.0.49\etc\build\pod.fansym: buildJdkHome=`file:/C:/dev/lan
g/jdk1.6.0_17/`
  C:\dev\lang\fantom\etc\build\pod.fansym: buildJdkHome=`file:/C:/dev/lang/jdk1.
6.0_17/`

C:\dev\lang\fantom-1.0.49\bin\fan.exe C:\dev\lang\fantom\src\buildall.fan superc
lean
Delete [file:/C:/dev/lang/fantom/lib/fan/]
BUILD SUCCESS [14ms]!

C:\dev\lang\fantom-1.0.49\bin\fan.exe C:\dev\lang\fantom\src\buildboot.fan full
clean [sys]
compile [sys]
  Compile [sys]
    FindSourceFiles [81 files]
    WritePod [file:/C:/dev/lang/fantom/lib/fan/sys.pod]
test [sys]
  Exec [C:\dev\lang\fantom\bin\fant.exe sys]
ERROR: Cannot find Java main fanx/tools/Fant
ERR: Exec failed [C:\dev\lang\fantom\bin\fant.exe sys]
BUILD FAILED [1389ms]!
##
## FATAL: src/buildboot.fan full failed
##

DevHome was up to date at time of writing and I have changed my FAN_HOME to 1.0.49.

brian Thu 28 Jan 2010

OK, I'll add an option for that

DanielFath Thu 28 Jan 2010

Thanks a lot, brian.

BTW the second problem hasn't been resolved.

brian Fri 29 Jan 2010

Daniel - try out this patch

DanielFath Sat 30 Jan 2010

Yeah it skip/pulls :D I still get the error in bootstrap :(

C:\dev\lang\fantom\bin>fan.exe
ERROR: Cannot find Java main fanx/tools/Fan

C:\dev\lang\fantom\bin>fansh.exe
ERROR: Cannot find Java main fanx/tools/Fan

C:\dev\lang\fantom\bin>fanp.exe
ERROR: Cannot find Java main fanx/tools/Fanp

C:\dev\lang\fantom\bin>fant.exe
ERROR: Cannot find Java main fanx/tools/Fant

None of the .exe in devHome directory seems to work. Is this supposed to happen before the bootstrap has been run?

brian Sat 30 Jan 2010

None of the .exe in devHome directory seems to work. Is this supposed to happen before the bootstrap has been run?

Nothing is going to work unless the bootstrap completes successfully.

Based on your above dumps the telling problem is:

C:\dev\lang\fantom-1.0.49\bin\fan.exe C:\dev\lang\fantom\src\buildboot.fan full

If you are using the patched version it should be calling "compile" target, not "full". Double check line 197 and 198 of your script.

DanielFath Sat 30 Jan 2010

I assumed relHome script is the one being called so I ran the script from fantom-1.0.49. Tbh, I didn't really patch the script, I mean i updated devHome but since I'm running script from relHome I manually patched its bootstrap.fan.

From the 1.0.49 distro I downloaded from the site.

Void build()
{
  runBuild(relHome, `src/buildall.fan`, "superclean")
  runBuild(relHome, `src/buildboot.fan`, "full")
  runBuild(devHome, `src/buildpods.fan`, "full", ["FAN_HOME":devHome.osPath])
}

It works now, having replaced "full" with "compile". So why 1.0.49 can't do a full build?

brian Sat 30 Jan 2010

Your code should read:

runBuild(relHome, `src/buildboot.fan`, "compile")
runBuild(devHome, `src/buildpods.fan`, "compile", ["FAN_HOME":devHome.osPath])

DanielFath Sat 30 Jan 2010

Your code should read:

runBuild(relHome, `src/buildboot.fan`, "compile")
runBuild(devHome, `src/buildpods.fan`, "compile", ["FAN_HOME":devHome.osPath])

Yeah, I know it works great now :D Any explanation at what was the problem?

brian Sat 30 Jan 2010

Yeah, I know it works great now :D Any explanation at what was the problem?

We changed what "full" means - see #883

Login or Signup to reply.