Hi,

how far advanced is a Pod distribution management tool going? What would people think of taking Scala Bazaar System (sbaz) as a model?

As far as names are concerned - can I suggest "Opera"

• meaning literally "creative works" - plural of "Opus"
• and as in "Fantom of the ..." (couldn't resist - sorry)

(something a bit more creative/fun than CFAN or FBaz, anyway. Someone suggested Graveyard, but that seems to me to be more a place where things get forgotten)

Regards,

Matthew (newbie, admittedly)

When coming up with names, it's better to have a unique, Google-able name that is easy to spell.

The original name for Fantom was Fan. That caused a lot of search problems. Taking a lesson from Google's Go language (the least Google-able name you could think of), Brian and Andy agreed to rename the language Fantom. Even still, there are at least two other "Fantom" products: an old fashioned car model and a hard drive company that share the name.

Opera would suffer from similar issues.

Opus would be better.

Graveyard is probably most appropriate from a cynical developer's perspective, but (sadly), it would be pretty bad for marketing.

just a note: Opus in Arabic means "Ouch"

That might not be very bad but I thought I mention it...

just a note: Opus in Arabic means "Ouch"

Ouch indeed. I'm sure I could think up some more. The joys of global branding ;-)

If I were doing it, I would probably start with SBaz, as it seems to make sense, and they've based it on Debian.

However, I'm new to Fantom and package management, and have limited time (which might mean I'm not best qualified to do it anyway, but I wouldn't mind contributing in any way I can - it's always good to have a focus when learning someting new :)

Best Regards,

M.

Speaking of here is another four letter name. Ecto. As in ectoplasm which is component of ghosts/phantoms etc.

Zuul.

Can we name it something descriptive and less scientific? something like:

PodStore

Upodia !!!

Pod Gallery

iPod?

+1 for Ecto

And adding the pod, it becomes EctoPod which sounds good, I think ;-)

Alternatively we could go for a meaningless (to me) like - FanDist or FDist.

Zuul

just don't cross the streams, that would be bad. :)

PodStore

there's a Pod for that... ;-)

How about FanBase ?

+1 to FanBase

Well naming stuff sucks :-)

In this case, we probably need to start discussing requirements and use cases to see what a bare minimum system would include (everybody has their own ideas of what it might look like)

The barrier to entry to discussions about name is very low. On the other hand requirements and use cases are something no one likes to make :-P

On the other hand requirements and use cases are something no one likes to make :-P

Which was partly why I suggested looking at sbaz as a starting point. They may have implemented more than what we want (at least initially)

And possibly why no one responded to that part of my original post (or maybe it's not a great idea, and everyone's too polite to say ;-)

Scala Bazaars System

Scala Bazaars Manual

Scala Bazaar Tutorial

I suppose part of the Fantom way would be to use Fantom fragments for the configuration, rather than XML? But that's implementation detail.

I'm new to Scala, Fantom, and package management, so I thought why not look at all 3 at once? ;-) I don't know Perl either, but I really don't want to go there.

Ok some ideas I'd like to see:

1. Installing/Uninstalling pods should be just copy/paste and deleting. I'd prefer not to remember cryptic commands or be puzzled by the install procedure.
2. Ideally PodStore would be different from working Fantom repository. I often update the sources to get the latest Fantom goodies. In doing so I delete the entire Fantom directory, paste new one from Hg.
3. As for configuration I've been looking a lot on Cedric's blog and I must say I'm taking a liking to YAML. If we are to have configuration files, they should be in YAML.

To keep things simple configuration should be stored in props and/or fog files.

As of DanielFath's 2nd point, having an options flag workingCopy that would warn user like in

WARNING: You're installing to the working copy.
Installed pods will be deleted on next full rebuild.
Do you want to install to the working copy [y/N]

might be sufficient.

Installing/Uninstalling pods...

You're right, installing/uninstalling pods shouldn't be cryptic or puzzling. (I don't use apt-get, but Ubuntu's Update Manager GUI is really easy. sbaz is fairly easy to get your head around) And I don't think installing/uninstalling pods can be as simple as copy/paste and deleting, if we're to take into account dependencies between packages (including versions).

Package management came about for Perl, Linux, BSD, Java libs (using Maven, or Ant+Ivy) etc because this became unmanageable as the number of components increased. For some packages, the dependencies are fairly loose - I can download almost any version of the scala installer from Scala-lang.org, and almost any Java6 version from Oracle, install them where I want, point Scala at my Java, and it will probably just work. On the other hand, nearly 10 years ago, I installed Bugzilla, and the list of components it needed (MySql, Apache, Perl, etc), and the versions of each, and the dependencies of each - including the C compilers and libs, was a nightmare. It was doable (for me - just about), but not my idea of fun. The Ubuntu packages lag behind the latest available versions a bit, but at least when I do get to update them, it should just work. and at least I didn't have to wait for the next version of the distro.

We're not close to that scale yet, but if Fantom takes off like it deserves to, it will help if people have an easy way to contribute.

Ideally PodStore would be different from working Fantom repository

The Fantom devkit/environment itself could be a package, e.g. fan-core, fan-dev. Source could also be released as fan-core-src, etc. If you wanted to get the latest bleeding edge stuff, you could set your package manager to get versions marked unstable, or experimental, while more conservative users get only the stable released packages.

And if you want to get the source from Hg and build it yourself, then you could always do that as well. And I think it would make sense to have that as a separate Fantom environment installed for that.

Looking through the apidocs for Pod and Depend, they look like there already is the support for specifying versioned dependencies between pods, although it doesn't look like it's used much yet. In that case, a package wouldn't really be all that different from a pod, apart from not having any code of its own.

Back to Scala as an example, a Scala install has a meta directory, which describes the installation, with

• available a file containing a list of the current available packages, with their dependencies. This gets updated from the server regularly.
• description,
• installed (what's installed - it would be compared with available to show what can be downloaded and installed, what needs upgrading etc.)
• universe (what "universe" we're set to using - you could be connected to a stable universe, or an experimental universe)
• cache - downloaded packages

sbaz uses the meta directory to manage the installation, and keeps it up to date.

If you copy things into or deleted thing from the installation, then meta is inconsistent and I don't know what happens. Better to have a separate installation to hack around in, in that case.

It looks like sbaz doesn't have a huge take up, judging by the number of packages in the default "universe". Perhaps Scala hasn't reached that critical mass yet, like Perl. Maybe there's a chicken and egg situation: it's not yet your one-stop shop because there's not much in it, and people are not putting much stuff in because it's not your one-stop yet.

Now, where is that White Rabbit ...?

I think that sbaz gets very little use because of Maven / Ivy. It is very easy to build Scala projects with Maven and leverage the dependency management it provides. Alternative Scala build tools like sbt also use Maven / Ivy for dependency management. If you look at Clojure build tools such as leiningen and cake they also leverage Maven / Ivy.

And I don't think installing/uninstalling pods can be as simple as copy/paste and deleting, if we're to take into account dependencies between packages (including versions).

I think it's possible, the problem is, it requires some extra bloat. My idea would be each package get its own folder. Each folder contains necessary libraries for running the package. Before a package starts all it's dependencies are checked to see if they are loaded and if not you try to load your own version. So no shared library, just the ability to check if a similar version can be found in another package.

Also yeah, I'd want some advance dependency information. Eclipse comes off as an example of what to not do. For instance I spend 30min downloading various dependencies only for eclipse to tell me that 1 of my libraries is in conflict. Conflict should be resolved before any downloads are made.

I am not an expert in Package management implications... but a couple of thoughts that IMHO are vital..

1. Please, no cached meta data for existing packages.... what I mean that things are done with copy/paste and that is it... I shouldn't need to update any metadata... it should be able to lookup available pods and do what it needs when it needs to ... that saves TONS of headaches in upgrades between versions... an example that comes to mind is mono... its GAC is a prob... and upgrading to a new version is a problem and requires lots of headaches..that of course if your distro is not officially supported. One of Fantom's great things.. is installation with copy/paste and just add the path to the env ... that is more or less it... please keep this as a strategy... that is a thing that made me seriously think of using fantom... since I can easily upgrade without going through a lot ... just download and copy / paste... All of this of course does not apply on the repo where we cach everything we wish.
2. Automatic pod download would be great... and It looks like it should work without a prob with the previous point... the app should look for the required pod before it runs in the actual dir not in anything cached... of course it could cach the results and check on start if any pod has been touched to refresh the cach... I mean that the dir is the main thing for any pod lookup or download.
3. I think that any config files whatsoever should be in pure Fantom...

Great discussion! Couple of key points on what I am thinking...

1. The unit of packaging will just be the pod. It is the already the basic unit of deployment, versioning, dependency management, and reflection. We only need to build a package manager for Fantom, so I don't think we need to complicate things by creating something new when we already have a great simple "package" embodied by pods. Since pods are really just named, versioned zip files that still provides a lot of flexibility in what happens after they are installed.
2. I don't actually anticipate any configuration files other than directories of pods themselves (and the meta-data already zipped inside them). This keeps things simple with file copy installation and makes it difficult to corrupt things. There might be some config stuff like repository URIs and such, but those should all be props or serialized Fantom objects (which is convention).
3. Wire protocol will be simple REST API that can be easily implemented for new clients and/or servers
4. Client tool will ship with core Fantom distro
5. We will run a default server for the community on this site as part of our "sidewalk" web framework stack (which we eventually plan to open source)

In the end I would like to find the simplest, most elegant solution that solves the key problems without creating a bloated over-engineered monster. To me the key problems are:

• servers in cloud where community can post pods
• easy human navigation/search of pods and their documentation, APIs, etc
• easy machine navigation/search of pods via REST APIs
• simple client tool to download pods to your local machine
• focus on flexible, simple kernel which will empower community to create more advanced solutions layered above

The unit of packaging will just be the pod.

I was thinking about this today, and it looked to me like pods had everything already. If you wanted to, I suppose you could also use an empty pod just to gather a group of pods together through its dependencies.

I was wondering if it might be a good idea to include the version in the pod filename as well (e.g widgets-1.2.jar), for being able to find the pod quickly in the podstore without having to unzip it to check its version in meta.props. (I imagine you would not want to remove old versions of a pod just because another pod needs a newer version, as you may have other pods which still need the older version). This would also help keep the podstore a flat directory. (on the other hand when we have 100s and 1000s of pods, this would start to cause a problem on Windows :( ) The pod would still be referred to in the code the same way as now (eg using widgets, & widgets::WonderWidget, etc - you wouldn't want to put the version in the source code as well) so this would need a change in the pod resolution. If you were running code that wasn't part of a pod, or the pod didn't specify versions for all pods used, there would need to be a "current" or "default" version.

In the end I would like to find the simplest...

I like simple :) having a tool to manage pod downloads simplifies the run-time.

One thing the build process needs to catch is the specification of dependencies which themselves have dependencies which conflict.

What level of security do you want? checking of MD5 sums? Pod signing?

Can I suggest adding a url to the meta.props for the pod's project website? (maven poms also have a load of other stuff like list of maintainer's names and their email addresses, issue tracking url, etc, but this could all be gained from the project website and doesn't need to fill a pod)

servers in cloud

you mean for mirroring, for instance?

We also want to consider unique identification of pods. Since pods have simple names, there is a high potential for naming collisions. What policies do we want to establish for assigning pod instances URIs? Furthermore, how are we going to resolve name collisions on a local box? (For example, I may need Brian's widget pod for one application and Andy's widget pod for another, but both need to reside on the same box.

We also want to consider unique identification of pods. Since pods have simple names, there is a high potential for naming collisions.

Pods are the unique name, there seems to be a misconception that because the core uses simple pod names, that third party pods shouldn't use names that avoid collisions. You should name your pods much like you would Java packages. Pods are the top namespace. So at runtime you can't use two pods with the same name (just like you can't use two different Java packages with the same name).

A different question would be can a web repository allow duplicate names. At first I thought it might convenient, but I think it would just lead to conflicts and encourage bad behavior. So I'm thinking that even a web based repository should require unique pod names. We should stick with the simple technical solution (pods are globally unique) and then there is only the community issue of properly naming your pods.

It's just that Java package names tend to be a lot longer, and the structure is reflected in the source layout So, java.util is not the same package name as org.rocketdyne.util, and both can quite happily co-exist.

(Actually, you can have 2 java packages with the same full name, and it happens in unit testing. the unit tests are in the same package, but in a different jar or directory on the classpath. But they resolve to the same package, and can access package private classes and methods in the source classes (some argue that unit tests should be in a different package to avoid this temptation, and make the unit-tests more blackbox) This is a special case, though, and random unrelated packages shouldn't share the same fully qualified name normally. If both packages had a class with he same name, 1st one in the classpath is the one that gets loaded. It's possible, nothing prevents it, but it's not always a good idea.)

I think the reverse domain convention of java is a simple way of avoiding collisions, and one we could follow, if only by convention:

using org-rocketdyne-widgets // unsure of pod name restrictions using com-bigweld-widgets

(Unlike Java, though, the name doesn't imply a directory structure, so you could say these are simple names, just longer)

And using ... as ... can keep your names short in your code.

Your pods should pretty much follow similar rules as your Java packages - in fact your Fantom pod names are your Java package (just prefixed with "fan."). So if your used org.rocketdyne.widget as your Java packages, your pod name should be orgRocketdyneWidget.

Although I think in general if you have a well known project name that should be safe as a prefix. For example we just prefix our sidewalk pods with sidewalk versus something like orgFantomSidewalk.

In the end you always want to find the right balance b/w short but unique identifiers. But the whole Fantom design is premised on the basis that pod names are unique. What conventions the community establishes to ensure that are really what we should nail down.

See Conventions

Towards the goal of consistency, note that according to that Conventions page you should omit the ".com", implying that it should actually be:

org.rocketdyne.widget => rocketdyneWidget

...at least according to the Conventions page. (Which seems cleaner to me than including the "org", although it does increase the likelihood of name collisions.)

But what about different version of the same pod?

In this case, the pod has the same name, it just exists in version 1 and version 2, and both should be available from the repository and from the local repository too...

At the moment, this is not possible to have the same pod in 2 different versions on your local fantom installation so you can have multiple projects, each using he same pod in different versions in their dependencies.

But what about different version of the same pod?

A repository absolutely must store/manage multiple versions of a given pod.

However, I don't personally think a runtime should ever use multiple versions.

What I was thinking of is something like this:

fantom.org/repo
   foo-1.0.23
   foo-1.0.24
   foo-1.0.25

local/repo
   foo-1.0.23
   foo-1.0.24

local/runtime
   foo-1.0.24

The repo tool would make it easy to sync between web and local repositories (just like hg). But in the end a given runtime picks exactly one version of each pod.

Ah! I hadn't found the conventions page, brilliant :-)

(Here's a thought: smiley interpretation for fandoc :)

For the version, this make sense. Having multiple version install but only 1 loaded at runtime.

But at the moment, the version number is not appended to the pod when built in the lib directory.

Does this mean this will change and therefore we can work on 2 different version of the same pod using the same dev environment? That will be good.

I think that structure is needed above the pod level. The maven/ivy environment should not be ignored, and they have a "group" level that is separate to the "artifact" level. Going against this widely used design would make integration with those toolsets a lot harder. (The "just convention" approach will lead to scaling problems down the line if Fantom is succesful)

Within a name, I can live with no dots (for good syntax reasons), but I would like to see a different symbol used. Its clearer than using a camel case convention. But lets ignore that for now.

This all means a level above a pod, effectively owned by an organisation:

// example syntaxes
fantom::sys::FooBar
comOpengamma::web::FooBar
:fantom:sys::FooBar
:comOpengamma:web::FooBar
[fantom]sys::FooBar
[comOpengamma]web::FooBar

This maps onto distribution tools in a far clearer way. (Some filesystems don't handle large numbers of files in a single directory that well) I also think that having the version number in the pod name is the right way to go.

repo
 : fantom
   : sys-1.34.pod
 : comOpengamma
   : web-2.3.4.pod

And yes, I would expose the "group" level in the API. But the code would still consider this as a pod. (ie. the Pod type would have a name field with the whole name as today, but with two other methods to get the two parts).

On the broader point, the key difference with Fantom is that everything is a "module"/pod. With that, the concept of the class/type is far less important - you compile a pod, not a class. Assuming that all the right meta-data is in the pod, then nothing more is needed for distribution.

Pods must have a license. They must specify dependencies. In fact, everything that maven poms have should be considered for inclusion.

I'd still prefer to see a Pod source file for this data. Its a better version of the current build files. Potentially, code could be compiled and run locally without the Pod.fan file for easy bootstrapping, but in order to be released and distributed in a central repo, it must have one.

On dependencies, its important to add some more advanced kinds. For example, some dependencies should be optional. For example, a library A may need to integrate with a lot of other tools, B, C and D. End user code E uses A, but only wants to use integration B, not C or D. The first approach is to depend on B/C/D, but that bloats the dependency chain (remember that B/C/D may also have dependencies). The second approach is for the author or A to create 4 pods (A-core, A-B-integration, A-C-integration, A-D-integration), which is fine, but a lot of work for author and user. Thus, the "optional" dependency concept is sometimes useful. There, A only has an optional dependency on B/C/D, so that E has to depend on A and B directly. If A tried to use C or D, there would be a runtime error. Sometimes that extra flexibility is needed.

Finally, there may be a role for source repositories. Each git/hg/svn repo is a well defined fixed version. They all have a "version number" of sorts. Perhaps, the released pod should directly link back to the source repo? I'm sure that this would open up new avenues for finding patches to bugs in advanced client tools.

But at the moment, the version number is not appended to the pod when built in the lib directory.

That is by design because it forces the runtime to only have one file to select, fast and simple. Once we move to a repo model, I would like to have easy to way to "roll back" to an old version. But I was thinking more that different versions are stored in a different directory and copied to lib as needed.

Stephen, thanks taking the time to write up your great comments.

he maven/ivy environment should not be ignored, and they have a "group" level that is separate to the "artifact" level.

But how does this work in practice? Can two different groups specify the same artifact name which happens to be the same Java package? If so what happens at runtime?

but I would like to see a different symbol used. Its clearer than using a camel case convention

The problem with any symbol we pick is that complicates the grammar unless a) we restrict to using statements (no fully qualified pods in code) or b) we pick something really new. One thing I considered a long time ago is (don't like it):

using fan:::sys
using skyfoundry:::fresco

But in the end a pod is identified using some globally unique string. The characters allowed in that string are restricted for filename and grammar reasons only. Plus it is community convention to decide (and enforce) how those strings are guaranteed unique.

Pods must have a license. They must specify dependencies. In fact, everything that maven poms have should be considered for inclusion.

I agree - pods should contain any sort of interesting meta-data. We just have to decide and establish standard "tag" names for keying the various data.

I'd still prefer to see a Pod source file for this data. Its a better version of the current build files.

Well we used to have this and have gone back to just having the build script inject it into the compiler at compile time. I think this tends to work pretty good in practice even when I have tons of meta-data to stick in a pod, because in the end build scripts are real scripts that can do anything code can do. But if things get complicated enough we could always codify a way to pull stuff out and have BuildPod compile that file as part of the build process.

On dependencies, its important to add some more advanced kinds. For example, some dependencies should be optional.

I can understand this suggestion - at Tridium we had the notion of ui dependencies, test dependencies, etc that could be stripped out when deploying code to small embedded devices. Although it does add a great deal of complication, so it be really nice to avoid or at least come up with something super simple. Something simple which might be useful would be that "optional" was strictly a compile-time dependency that got omitted in runtime dependency (and failed with runtime exception when class was loaded).

Finally, there may be a role for source repositories. Each git/hg/svn repo is a well defined fixed version.

I have really been thinking about this a lot too. Something like if an hg URI was included in the pod meta-data, then the repository tool could automatically install "tip" as one of the valid versions by pulling from hg and building from source. That would be awesome for easily accessing patches that have been pushed into hg but not bundled in a versioned pod.

Can two different groups specify the same artifact name which happens to be the same Java package? If so what happens at runtime?

Yes, two groups can specify conflicting artifacts. But so can one group with two artifacts. Conflicts happen in Java. Fantom should eliminate that source of failure.

My proposal keeps a single globally unique pod name. It just splits the name into two (or more if you allowed dots/dashes) parts internally, which is useful for distribution. Once you have two levels, its only the first that the conventions apply to. Projects would be free to do what they like with the second artifact level wrt naming.

Personally, I'd remove pods from code and restrict them to using elements (as there are aliases, this is no problem). I might actually go further and only have using at the pod level, as it keeps all dependency management in a single place.

The simplest scheme is to say that all artifacts from a single group must come from a single repository. This then allows Apache to control all Apache group artifacts for example. Thats why this syntax might make sense:

using [org.apache]foo.baz::Type
// pod name "[org.apache]foo.baz"
// group name "org.apache"
// artifact name "foo.baz"
// type name "Type"

because the [] part is the "supplier" of the next part - [java] is the supplier of all classpath driven Java files, but in the future [org.joda] might be configured to point at a maven repository giving this:

using [org.joda]joda-money::org.joda.Money
// pod name "[org.joda]joda-money"
// group name "org.joda"
// artifact name "joda-money"
// type name "org.joda.Money"

Which can access the maven system and pull down the joda-money jar file in the org.joda group finding the org.joda.money type, and knowing it is Java code based on the "org.joda" repo settings.

Notice though that the type is qualified with the package name. This is where the difference between Java and Fantom comes in. In Fantom, the type name is always simple, because its uniquely defined within the pod and the artifact is the pod. In Java, the artifact does not uniquely define a single namespace. There could be two or more Money classes within the artifact. Thus the package name is part of the type, not part of the artifact. Put that way suggests that the current "java package = pod" approach isn't right.

If you stick with "java package = pod", then you'd need a more complex structure for maven derived Java artifacts:

using [org.joda]joda-money:org.joda::Money
// pod name "[org.joda]joda-money:org.joda"
// group name "org.joda"
// artifact name "joda-money:org.joda"
// type name "Money"

Essentially its about defining if a pod is the first namespace above a type or the unit of distribution. What is a Pod?

The optional dependency simply means make it available to compilation, but don't include it in the dependency chain at runtime. Classes can therefore only use those parts of the compiled code if some other pod provides the dependency. Handles tests and integration-type scenarios.

Another point is that a Java class with annotations will load even in the absence of the annotation class at runtime. This is very useful, and Fantom should do the same if possible.

The source code hg/git approach has limitations, as not everyone wants to release their code publicly. But as you say, I'm sure there are a bunch of useful things that can be done with it. In an ideal world, you might remove the whole release concept entirely!

Within a name, I can live with no dots (for good syntax reasons), but I would like to see a different symbol used. Its clearer than using a camel case convention.

I would like to see this, too. (Although, I would prefer that symbol to just be a dot).

When a large project needs to be factored into a bunch of smaller pods, developers end up having to name all of those small pods. If we don't establish a convention, then they will end up with very simple names (display, template, database, etc) where collisions are more likely.

However, if my project is called SuperApp, I can easily name these with a superapp. prefix: superapp.display. superapp.template, superapp.database, etc. Java users will find this dot syntax comfortable, even though it's just part of the name and has no structural significance.

The dot also makes it easy for custom Env loaders to locate pods. If there's Fantom code being used by the web team and the marketing team, we can name our pods web.database and marketing.display or and have those resolve to different directories (like Java) or servers. It just seems to add a lot of flexibility.

Finally, there may be a role for source repositories. Each git/hg/svn repo is a well defined fixed version. They all have a "version number" of sorts. Perhaps, the released pod should directly link back to the source repo? I'm sure that this would open up new avenues for finding patches to bugs in advanced client tools.

This is a great idea. This should lower the barrier for new developers and help speedy adoption of the language.

Consider to postfix the version numbers to pod name.

That is by design because it forces the runtime to only have one file to select, fast and simple.

If I hava some pod files A with different version

A-1.0.pod, A-2.0.pod, A-3.0.pod

B is a pod that depends "A 3.0". Then fantom know to load A-3.0.pod at runtime.

Version Select

See the Depend.version doc:

• versionSimple: returns the version
• versionPlus: returns the version
• versionRange: returns the start version

Version Conflict

And this way, if B depends C and C depends "A 2.0", will be a runtime error.