Up to this point I've tried to keep things as simple as possible by just having everything required to build a pod declared in the build script itself. That was super simple, but I think we are reaching a point where a more sophisticated solution is warranted. Here are some problems I'm thinking I'd like to solve:

Duplication of Pod Meta: as part of the fanr feature, we are declaring a lot more pod metadata in the build scripts. Since much of the metadata is common across an entire project, its not very DRY having things like proj, org, license, and vcs data spread across all the pod level build files of the project.

IDE Access: having key meta-data like dependencies, source dirs, etc trapped in a script file versus a data file makes it much harder for IDEs to consume. I know tcolar has been complaining about that for a while :-) Several years ago we actually had a special source file in each pod, but ever since we nuked that idea I've been really reluctant to add any new complexity or special file types.

Multi-Projects: the process of building and distributing a project with dependencies on other projects sometimes needs access to inject version patch numbers or meta-data when rebuilding an external project from source. As an example, when we build SkySpark we pull in the tip of Fantom and Sidewalk as dependencies, but inject meta-data so that we know exactly how the pod was built.

Dependency Versions: currently we've been loosy goosey in core dependencies between pods. For example all core Fantom pods have a generic dependency on "sys 1.0". With the coming of fanr, I'm thinking we need become more strict. For starters I'm thinking that dependency versions will be on the specific build. So when we do build 1.0.60, the dependencies of core Fantom pods on other core pods will all be 1.0.60. Maintaining these dependency versions needs to be pulled into centralized files.

Proposal

What I am thinking for a first step is to leave BuildPod so that it works just like it does today - we won't break any existing build scripts. But instead of configuring all the build/metadata in the script, you can create a peer file called "build.props" that defines all the same stuff. Here would be a simple example for the wisp pod:

podName=wisp
summary=Wisp Web Server
depends=                     \
   sys        $buildVersion, \
   concurrent $buildVersion, \
   inet       $buildVersion, \
   web        $buildVersion
srcDirs=fan/, test/
docSrc=true
include=../build.props 

Then you can change your build script to this one liner:

class Build : build::BuildPod { new make() { initFromBuildProps } }

Originally I had the notion of getting rid of build scripts altogether and replacing with them with a fanb tool that just took a directory and worked off the build.props. That would avoid a lot of fairly useless one line build.fan scripts, but maybe that sort of thing is better left for a future. Comments welcome if everyone would like to figure out to remove build.fan scripts altogether when no new builds targets are required.

Note the "include" key in the wisp example. What I'm thinking is that include would let you pull in project level props from a common file shared by all pods. So for example the Fantom core might have a top level build.props that looks like this:

version=1.0.61
meta.org.name=Fantom
meta.org.uri=http://fantom.org/
meta.proj.name=Fantom Core
meta.proj.uri=http://fantom.org/
meta.license.name=Academic Free License 3.0
meta.vcs.name=Mercurial
meta.vcs.uri=http://hg.fantom.org/fan-1.0/

This gives a simple DRY pattern for configuring all the project wide metadata and current project build number. The build engine would just merge them all (allowing a pod level script to override keys from includes).

Using a simple props file has some nice characteristics versus creating something new or using a more complex format like a serialized Fantom object. Because the basic data model is key/value pairs, it makes it easy to cascade includes and merge them together. And its a super simple file format that is easy for other tools like IDEs to parse (for example we use props for things we want the launcher C/bash code to parse).

Also note how we might do version dependencies. If I am building a project like the fantom.org website that depends on an explicit version of Sidewalk and an explicit version of Fantom, then I really want a single, DRY place to configure that version. By allowing the use of $ to refer to another prop we get a simple macro system that allows you to do something like:

// top-level build.props
sidewalkVer=1.0.23
fantomVer=1.0.62

// pod level build.props
depends=                \
  sys $fantomVer        \
  fwt $fantomVer        \
  webfwt $sidewalkVer   \
  sidewalk $sidewalkVer \

Putting a macro system in there does complicate life of IDE trying to parse this stuff too, but is still pretty simple to implement. And seems like generally useful.

Thinking about tackling this problem tomorrow, so if you have an opinion please let me know what you think.

Instead of creating a second file/format - can we just adjust the current build APIs to allow inheriting some of these field values from a parent script? That seems like a safer short-term solution.

I think there is likely some improvements we can make in the build process - I've considered something like fanb as well. And it might dovetail into improvements in BootEnv/PathEnv use cases also. But we should probably at least frame what the overall design there should look like before we make any intermediate changes.

I will second Andy here - a simple inheritance between build script should be enough.

I like the idea of a fanb as a simple command line tool to create new fantom project directory structure, and automatically uses fanr to manage dependencies of the new project created, ...

Also, I would like this tool to be like sbt for scala with incremental build... And maybe also something that run the test as soon as a change is detected like JUnitMax.

And I still think that there should be distinction between boot env/path env for development and a runtime env for just managing deployment of fantom application.

Keep the good work.

+1 on Andy

I am not a Fantom expert but I personally loved the idea that everything is Fantom... I think it is great that it is a build script not a props file because it allows me to do whatever I want in there... I am no expert in Fantom but I sure looks like it could be really useful...

My impression is that for this to succeed and be as powerful as it should on the long term it will eventually turn into something like C's "make" although it is very powerful it is also hard and annoying... I know this is exaggeration but you get the point... fantom scripts require no additional syntax or anything new to learn... you can do what ever you want in plain Fantom....

May I ask why is this such a big deal for IDEs? It looks like it can be simply done by some reflection... If this is a problem, I think that we can just add a simple method that reflects the script itself and print all values in the above format to the console... that way it can executed from C/Bash/whatever without needing to do anything tricky.

Just up-cast to build::BuildPod, not need reflection.

I don't remember complaining that much :)

But yes, the build file being in Fantom was not ideal, because that's a pretty hefty grammar to need to know just to figure out a few build properties.

In a way in my IDE t's not even so bad because i already have the grammar/parser ... but if anything that's inefficient.

I've went back and forth between using reflection or trying to parse on my own ... and neither worked too well.

When I used reflection it wasn't ideal, for one that means the IDE is tied to a particular Fantom build / implementation (bundled) which is quite fragile.

That type of reflection (from java) has actually changed many times, so I've tried to stay away from that.

Having to bootstrap just to read that is really not efficient(IDE's read that data often) and it causes many issues when the fantom home is not yet set / or improperly set.

I also tried using a regexp but obviously it's a bit ugly and of course that regexp business completely fails once inheritance is involved so it's kinda lame.

So anyway ... a simpler format like properties or basic JSON would be best for tooling, clean and fast to parse ... especially for "lighter" tools wether simple editors or any other tool that would want to interact with a fantom project.

Just to make sure everyone fully understands - if information like dependencies is in the build script, then the only way to absolutely be sure to get the proper results is to actually execute the build script and then reflect the values. So its not an ideal process for an IDE which needs this information to correctly do things like code completion, semantic browsing, etc. Ideally you would like to pull that out into a data/declarative format.

But that doesn't mean that the flexibility of build scripts should go away. I think its always there, just seems like 99% of build scripts are purely declarative and add no additional build targets.

Just to throw a new idea out there, how do people feel about the build-pods being defined in a FOB?

1. The syntax is almost Fantom syntax, so no new formats.
2. It's static, so IDEs can play with it.
3. The documentation is just Fandoc (it's all neatly organized in a familiar format)

About the "building is scripting", I think there's some merit to that. Make and Ant are horribly opaque, and it's really hard to get a good idea of what's going on in an unfamiliar project or to set up fancy build dependencies. I think even if we have a static "core" build plan, anything beyond the basics should easily escape into a script.

Just to throw a new idea out there, how do people feel about the build-pods being defined in a FOB?

You mean serialized objects right?

I do think there are lots of pros and cons for picking format. What I liked about props was easy to parse, easy to merge/cascade name/value pairs, and easy to build simple macro system. Downside your only types are String name/value pairs (but that also can be advantage because you have flexibility to map to static typing at load time).