By the way, I think it should be possible (and if it is, then it should be the default) to have a separate user folder for installed pods rather than in the Fan home area. It would be nice to leave an install/download untouched by later work.

This is something that has always been in the back of my mind. I am pretty open to design suggestions on how we might structure installations.

The key point to remember is that Fan needs (or at least would very much like know) all the pods which are available. This information is then used to enable Pod.list, Pod.find and the type database.

Today I have a drop dead simple design - all pods go into the exact same lib/fan directory and this is how we discover pods. The nice benefit of this simplicity is that it is impossible to have conflicting pods.

So if we want to take a more sophisticated approach, there are all sorts of options. The simplest case would just be to have some "classpath" like construct which searched a series of directories for pods. Is that a fixed set of directories or open ended? If open ended is it defined for the installation or for a specific Fan process? The more complicated the solution, then the more complicated it becomes to maintain the type database. The simplest solution would be to just add something like another fixed libuser directory.

I also think that in order to design a successful solution to this problem, it seems prudent to start thinking about our "package management" solution. Ideally we want something like Ruby Gems so that developers can post pods in the cloud and have tools to download them into local installations as needed. We have the fundamentals: a clean modular architecture with well defined naming, versioning, and dependencies. But I haven't given too much thought to the next level which will be required once the community starts to grow. Do we attempt to use OS package managers or build something like Ruby Gems?

I don't like OS package managers. They always lag. One person can't manage every OS and distro. If you are clever enough to abstract and integrate, then go ahead, but that might be too complex.

Ignoring for a moment the large scale of install from remote pod repos, I think fundamentally the idea of multiple dirs would be nice. You could look in the current dir and parents to see if a repo exist, then in "~/.fan/whatever/" (or "User/Application Data/Fan/whatever/"), then in the fan home repo. Default installs to the "~/.fan/" repo. Automatically managing this common use case (current working directory/project, then user dir, then fan home) would be nice, allowing override of the user dir case by environment variables or by command line options.

I was going to recommend configuring each repo to point specifically at its parents (prototypes?), but a common case might be to use one repo on multiple versions of Fan (different repos), and having to change the configuration would be a pain.

Anyway, the solutions should keep the big picture in mind, but I see these as different issues:

1. User dir (or configurable) install to avoid touching fan home.
2. Current working directory repo discovery.
3. Easy install/update/uninstall from remote repos.
4. Integration with OS package management.

In my own opinion, item 1 is the most immediate need.

It would also be nice to be able to configure Flux without touching the Fan home files. Maybe a full Fan layout (of just changed files or whatnot) instead of a focus on just the pods?

I like the basic idea you've laid out Tom - although I'm not sure I understand the exact details of your proposal.

So you install the Fan distro into fan home or just home. Then you put "home/bin" into your path so that you are running either "fan.exe" or the "fan" bash script. These scripts will continue to work like they today and infer the location of "home/lib" from themselves.

So really what we are talking about is a mechanism to add one or more lib directories into your "pod path". Although I think you make a good point we might want something a little more general purpose with the flux files as an example.

I'm a little wary of inferring things from your current working directory - that seems sort of fragile if you start cd'ing around. So I'm more inclined to set the stuff up based on your user home directory.

I agree about the current path bit. Personally, i don't find java's classpath all that burdensome to set up and it allows for per application settings. That might not be as strictly necessary with named/versions dependencies baked in, though.

Java by default has "." in the classpath, right? This would just be an extension to recognize project structures from anywhere in the project. But then again, they don't recommend "." in the path for \*nix, so I guess we have examples both ways.

Glad to hear about the potential for a user dir pod (or fan) dir. Firefox, for instance, doesn't overwrite its own directory for installing extensions. Rather, they go in your user dir (under subdirs appropriate to the OS) by default.

No, "." is not in the classpath by default. You have to add that manually.

Most systems tend to build some kinds of repositories (including Java with Maven2, Perl, PHP, ...) - a logical way for Fan too. I'd avoid system repository managers (like dpkg, apt) though.

It'd be pretty cool to have proxied repositories (as maven2 does) with structure by convention. I like the idea having a normal filesystem tree with installed pods (using pod descriptor to automatically track dependencies) which can be easily changed to delegate to a remote repository chain.

However, this returns us to the versioning and dependencies debate. I think it is neccessary to have the possibility of multiple versions installed and running (if they are used internally). Otherwise the transitivity (through the auto dependency tracking) would be a very painful process. I think one should not be forced to examine the internals of other pods or the versions of it their respective utility pods just to use them.

Once there is a base of utility libraries we can expect various projects to use various versions. Having to resolve conflicts of such pods would be futile and a never ending task (a problem without a proper solution, as in maven) ...

If you think about the simplest thing you might do, it would be something like a -podpath option to include directories to search for pods. The main problem with something that simple, is we don't have a good "home" directory where we can store the type database. Plus something that simple doesn't necessarily give us a clean way to create alternate versions of various config files in lib or flux (although maybe that ends up being a different problem).

I'm thinking more of the ability where your environment is configured in a file and passed to the runtime:

fan -env mysetup.props foo::Bar ...

// mysetup.props
fan.home=/apps/fan/
pod.path=/dev/foo/;/dev/bar/

Then whatever directory your env file lives in becomes the "env home" for things like caching the type database files.

Then whatever directory your env file lives in becomes the "env home" for things like caching the type database files.

That feels backwards. It would seem to make more sense to specify a user env directory and then find a properties file within it.

In general on this topic, I agree that the base Fan install should be fixed and not-changed. Thus there needs to be one or more user environment directories.

While I don't like maven in general, I do find the concept of a large central repo useful, particularly with a good, clear structure for pods and versions. I think Fan should do the same, but as I've said before, I believe that maven's experience teaches us that structure in the pod name will be needed (com.foo.bar.MyPod, not comFooBarMyPod).

I believe that maven's experience teaches us that structure in the pod name will be needed (com.foo.bar.MyPod, not comFooBarMyPod).

I completely agree. Besides that versioning directly in the name should be considered. This scheme is used for .so libraries for ages. For instance

-rwxr-xr-x 1 root root 1315024 2008-09-29 12:40 /lib/libc-2.8.90.so
lrwxrwxrwx 1 root root      14 2008-11-01 23:15 /lib/libc.so.6 -> libc-2.8.90.so

I do find the concept of a large central repo useful, particularly with a good, clear structure for pods and versions.

It should be also made more robust than m2 repositories. It is easy to break them by uploading a bad or incomplete artifact.

That feels backwards. It would seem to make more sense to specify a user env directory and then find a properties file within it.

I guess I'm coming at it more like IDE project files. How do you guys setup stuff like that? To me the runtime and IDEs should share the same meta-data.

I hear and understand the whole multiple versions and repository issue. My thoughts on stuff like that are that disk space is insanely cheap. For example, the runtime components of Fan are just 13MB now - in a lot of cases it makes a lot more sense to just have a complete image for each project. Simple is usually better. Although I understand that is not a suitable solution for some types of scenerios. I do think having some sort of "project meta-data file" would make it easy to build a multi-version repository layer.

In a framework like Fan I tend to think about two steps: 1) have we enabled the correct hooks to let someone else build this? 2) should we build it into the core? I clearly think the answer to 1 is no right now - which is the first problem we need to solve.

Here is an idea, instead of having a meta-data environment file, we have an open ended API:

class FanEnv
{
  virtual File findPod(Str name) // default searches podDirs
  virtual File[] findPods()      // default searches podDirs
  File[] podDirs := [Sys.homeDir]
}

Then you can pass a script file:

fan -env myenv.fan ....

// myenv.fan
class Env : FanEnv
{
  Void setup() { podDirs.add(someDir) }
}

Or you can pass a type qname:

fan -env fanMaven::Env ...

A solution like that could solve a couple problems:

1. provide simple way to extend your environment with a script
2. allow more sophisticated repository/packaging solutions
3. decouple the runtime from repository/packaging strategies

I dislike the general inflexibility of external configuration files (or just weird syntax, sendmail.cf being the shining example). I think that a single make system (preferably in declarative Fan) is a must. If we thrown in a general repository that'd be really cool - far ahead most current combos of languages and their build/test/deploy systems.

I still think you can't avoid versioning though. SW is too dynamic :/. In order to avoid the horrid scenarios of passing around incompatible versions of various objects... what about forcing a developer to declare pod output types and versions or even interface versioning?

Transitive dependencies require the ability to have multiple versions of the same module, from my experience.

I'm probably not going to work on this in the next few months, but I added to the roadmap as something we need to do.

However, we are in the process of figuring out to provision Fan pods and applications on our AMIs for Amazon, which might bring this topic to the top of the priority queue shortly.

If any has concrete proposals or design ideas please continue to share.

I'd like to see this moved up in priority, if possible. It's the most obvious dip in the road in an otherwise sparkling language design and implementation.

I think your proposal with a file listing the configuration is fine. You could even add a default behavior to look in the current directory for env.fan, so there is an intelligent default.

But if you're not going to work on this soon, can we please have a directory such as /userlib, rather than /lib to hold the pods. At least then, managing new installations and the environment is much simpler. Plus having user pods in their own directory is a reasonable stopgap (at least when compared with putting them in the system lib where system and user pods are all mixed together).

Any higher prioritization of this issue would be much appreciated.

p.s. I'm a new Fan programmer who'd like to recommend it to colleagues, but I hesitate b/c of this one aspect.

I don't have any problem making this feature a priority. We just have to reach a consensus on what the solution is.

My last proposal was for a pluggable FanEnv class which was provided to the runtime to manage pod resolution. It might be too early for that solution.

The simplest thing that will work, is to provide one alternative directory which would mimic the Fan home directory structure. Let's call this appHome - it indicates the directory being used for the current application. This directoy is configured:

1. As command line switch: fan -appHome /dir <main>
2. If no switch then check FAN_APP_HOME environment variable
3. If no environment variable then use "{userHome}/fan"

All pod management would work off both fanHome/lib and appHome/lib.

The type registry would get stored into appHome/lib.

The lib/sys.props would be loaded from fanHome/lib first, then appHome/lib would get a chance to override things.

If we do the symbols feature, then symbol overrides would work off fanHome/symbols and appHome/symbols.

There is a new flag in build::BuildPod which lets you build the pod to the appHome/lib directory.

The sys::Test.tempDir scratch directory would be moved under appHome/test.

All other things would continue to work strictly from fanHome and we could evaluate on a case by case basis about appHome support.

I don't really love the term "appHome", but I feel that "userHome" is a little limiting, as I can see this being used on a per application/environment basis. Another term might be "envHome". Suggestions welcome.

Does that proposal work for everybody?

EDIT: actually I like envHome, so replace appHome with envHome everywhere above

I think this all sounds good, save one detail:

There is a new flag in build::BuildPod which lets you build the pod to the appHome/lib directory.

Seems like that should happen automatically by having the source under appHome vs fanHome?

Thanks for the quick reply. This approach looks like a good, workable solution.

Would love to see it in the next rev. Again, thank you!

Agreed. I think the default should be in appHome. Personally, in my java work, i never put libs in, say, jre/lib/ext which is often (erroneously) recommended as a place to drop global libs. I prefer to have application libs isolated off in the application directory. If I really want a lib to be global, I can manually install it in sysHome. If you're going to use a common area, you need to start having versioned pods so multiple apps can use different versions of the same pods.

Do any of the rest of you have any feedback on this proposal?

This approach is a good step forward from today's setup. So, even if we enhance it further in the future, I'd say go for it.

I like this, and I think we need it.

EDIT: actually I like envHome, so replace appHome with envHome everywhere above

Not sure I like that, though; envHome sounds too much like it should mean Fan environment, which is what I take the fanHome to mean.

I think, fanHome for the Fan distribution; this location stays mostly untouched unless I'm modifying Fan itself; and

appHome to specify a project root location; I might have many different non-interfering Fan projects, but with this, each project can only see itself, and the Fan system.

+1 for fanHome and appHome. Thought projHome might be more applicable if thats the term we'll be using.

Promoted to ticket #425 and assigned to brian

Sounds like no one likes envHome.

Do you guys prefer appHome or projHome? I vote for projHome, which seems a little less generic, and would work well for setting up a Fan IDE environment.

+1 for projHome

If this is a runtime switch I'd say +1 for appHome, since it's only a project on your development machine--once you deploy it, it becomes an app.

Just throwing this out there. It might make sense to make appHome or whatever the default build location. For library writers and core developers, fanHome is where you want your code to end up, but for the average Fan user, it doesn't make sense to clutter up the official pod directory with experiments and throw away apps.

Additionally, to help support those throw-away apps, it might also be useful to have a flag to build a pod to the current directory, with the ability to load pods in the current directory.

Of course, with that last idea, the pods would need to be listed explicitly to prevent abuse, just as in Unix, you have to prepend ./ to executables in the current directory.

It might make sense to make appHome or whatever the default build location

I think pods should build to their context (if under appHome, goes to appHome/lib, and if under fanHome goes to fanHome/lib).

I think pods should build to their context (if under appHome, goes to appHome/lib, and if under fanHome goes to fanHome/lib).

That makes sense. What about pods built outside of the appHome and fanHome directories?

What about pods built outside of the appHome and fanHome directories?

Current Flux build process is, when you start the compile, Flux searches up the parent directory recursively looking for a build.fan file, which has (should have) options for podname and source locations, etc.

If that build.fan location isn't under appHome or fanHome, or if fanHome isn't defined, then maybe the build process should just put the .pod file beside the build.fan that it was generated from.

Then to run it, Fan could use fanHome for Fan system, appHome if specified for additional dependencies, and current directory in case the project has several pods.

That way you could have a single Fan system, multiple separate application/project areas, and also independent applications located wherever you want to put them.

Maybe we should approach this from the other direction, by having each pod specify the target home dir in build.fan?

What about pods built outside of the appHome and fanHome directories?

Should we even allow that?

The fanHome/appHome dichotomy is really just a starting point, as Brian wanted something that could be implemented simply. And I think it's a great starting point.

If we limit dev to these two directories, as Andy asks, I think this should be a temporary constraint. Eventually, as qualidafial suggests, I feel it should be possible to use any directory.

This is not just a personal preference. I suspect it will be very difficult to get Fan adopted in enterprises, if pods cannot follow site conventions on file placement. More so because, Java, .NET, and native executables can be linked with and placed in directories pretty much wherever the developer wants.

If we limit dev to these two directories, as Andy asks, I think this should be a temporary constraint. Eventually, as qualidafial suggests, I feel it should be possible to use any directory.

I think Andy was talking about source directories - and I think a source dir should be located anywhere you want. For now I say anything not under fanHome gets compiled into appHome.

Regarding lib directories, I don't agree they should just go anywhere. That isn't necessarily any more flexible, only more confusing. Unlike Java, one of the most powerful features of Fan is the ability reflect the installed pods and build a database of all the installed types. But the tradeoff of that feature is that Fan must have a well defined directory structure to reflect and store the type database.

I think Andy was talking about source directories - and I think a source dir should be located anywhere you want. For now I say anything not under fanHome gets compiled into appHome.

Yes, you can have as many appHome dirs as you want - I meant that we should require an appHome to build a pod (outside of fanHome). Otherwise you either have to (a) manually copy the pod to run it, or (2) get into classpath-type issues we want to avoid.

I had planned to knock this feature out in May, but I've been dragging my feet because I don't really feel like we've got a good solution. I originally thought that maybe we could just add a temp solution, but I think we really just need to figure out the right design and do it right. Per post 654 cheeser would like to have this for running Fan in GlassFish via his TradeWinds project.

What we are really talking about here is how to structure Fan "repositories" which encompass several things:

• pods
• type database built from installed pods (runtime generated files)
• configuration such as sys.props, flux setup, etc
• possibly localization files

Things to figure out:

1. how are repositories stored locally on disk?
2. how are repositories stored remotely on the cloud?
3. how are repositories installed, shared, managed?
4. how does an application specify which repositories should be used?
5. how do build tools and IDEs work with these repositories?

As part of this feature I definitely want to lay down the design for managing pods in the cloud too - I want to tackle those things together.

I am not familiar with frameworks like Maven or Ruby GEMs so if anyone wants to post a simple overview with pros/cons for how this stuff works for those types of things that would be quite valuable.

I'm not much expert on Maven or RubyGems, though I've used both some. Here are some sample repos:

• http://gems.github.com/
• http://gems.rubyforge.org/
• http://repo1.maven.org/maven2/

I'm not sure how the directory structure looks for RubyGems.

I do agree it would nice to have a unified system for repos. A normal Fan install could look the same as any other repo, perhaps. And repos do need the ability to have multiple versions installed simultaneously (for use by different pods with different requirements perhaps or minimally for redistribution to elsewhere).

If a repo in the clouds looks like a local one, there may be value in having files with pod lists, since directory listing via HTTP generally isn't a good thing. And avoiding the need for custom server software would be nice.

Seems like the namespace API is a good place to plug in classloaders, however I'm not sure how that would translate across to .NET or JS.

Ticket resolved in 1.0.45

Fixed via new repo design