In the recent post #709, there was a question about importing another script from the file system. This is actually a problem we're facing at SkyFoundry, except the scripts we want to link together don't live on the file system, rather they live a database. The general problem is best expressed as how do we name scripts in Fan's pod namespace and resolve them into pods?

There are two different mechanisms used today:

• FFI: in this case we identify foreign pod namespaces using the syntax [ffi]pod such as [java]java.util
• Sys.compile: this method is used to manage compiling text files into pods and caching the result based on file timestamp. What I do here generate a pod name based on mangling the file name and the timestamped version.

There are a couple different issues in the overall design:

• Naming: how do I name these pods? Today we have simple names for standard pods, auto-generated names for scripts, and FFI names for Java
• Resolution: given a name how to I resolve it a pod? there are two sides to resolution: a runtime reflection aspect, and a compile time aspect
• Caching: how is the pod/script cached for reuse and link dependencies

At the most fundamental level, Fan defines everything using a namespace structured as pod::Type.slot. In the original design way back in 2005, we choose a single identifier for each level of the namespace. But as Fan has grown, the pod name as a global top level identifier has started growing into a mini-syntax of its own (especially when we introduced FFI pod namespaces). The top level of Fan's pod namespace includes:

• actual Fan pods
• script files (in a file system, in a database, etc)
• FFI where a foreign type system is mapped into Fan

I think the time has come where organizing this top level name with a simple string is getting too cluttered. So I propose to replace pod names with pod uris and identify pods using a completely pluggable URI where namespaces are scoped via the URI scheme:

acmeFoo             =>  shortcut for pod:acmeFoo
dir/script.fan      =>  file:dir/script.fan
db script           =>  db:rec-id
[java]java.util     =>  java:java.util
[dotnet]System.Data =>  dotnet:System.data

This change unifies the concepts of FFI and scripts into a naming model based on Uris. Uris are in turn backed in Fan by a very rich infrastructure: literals, a very complete sys::Uri class, and a nice plugin mechanism via sys::UriScheme and sys::UriSpace. Pods may be named via any Uri in standard form, except that two consecutive colons are not permitted (since that would ambiguous with regard to qnames). Existing standard pod modules remain exactly the same as today, except they have an implied Uri of "pod:name".

Now once we name pods with a URI, we can establish a much clearer process for resolving pods and compiling scripts:

1. Pod.find takes a URI
2. If the pod is already loaded, it is returned
3. If the pod is not loaded and the scheme is "pod:" then current repo rules apply
4. The pod's URI is resolved using sys::Uri.get to a sys::File, and then existing file based script/compiler/caching rules apply

As part of this change these APIs would be modified:

Pod.find(Str, Bool)  =>  Pod.find(Uri uri, [Str:Obj]? options := null, Bool checked := true)
Pod.name             =>  Pod.uri
Sys.compile          =>  goes away, functionality merged into Pod.find

The using statement grammar would be changed so that either a single identifier or URI literal would follow:

using [java] java.util  =>  using `java:java.util`
using acmePod               // stays same as shortcut for using `pod:acmePod`
using `utils.fan`           // new functionality to link in a script file

There are tons of future enhancements we can add including custom pod management, multi-level pod naming, script caching etc. But the thing I want to do now is just get the fundamental naming model right so that we cleanly tackle new features in the future. To summarize the proposed changes:

• change the Pod APIs to use Uri instead of Str name
• change the FFI syntax to use Uris
• change the using grammar to allow either single identifier or uri literal
• enable scripts to link to other scripts using any Uri which dereferences to a File

Seems like a nice direction.

As a first question, is there any concern about taking over top-level schemes like java: and dotnet: and so on?

Seems a bit brave, but alternatives seem ugly. For instance, say Fan just takes over fan: for all its uris. Then, you could more safely say things like fan:java:java.util or whatnot. And the normal fan: scheme would need to become something like fan:space:... or something. Ugly, but it feels safer.

You read my mind. +1

As a first question, is there any concern about taking over top-level schemes like java: and dotnet: and so on?

I don't personally think it is a big problem since we are using them internal to our programming language.

I think overall this is headed in the right direction. To second Tom, I think the pod/FFI URI's should be namespaced under fan: (or someother root) to avoid polluting the root.

Couple of details that came to mind...

So what does a qname look like now?

• `pod:sys`::Int#toStr
• `file:script.fan`::Build#targets
• Can I even reflect Java now?

Losing my single flat list of simple str names sucks, but maybe its a necessary casualty to properly support scripting, which I think will start to take a bigger role in Fan.

Something seems off about the URI though - seems like that is a mechanism of how I load a pod. But it serves as the unique identifier for my pod as well. What happens for scripts in sub-directories, that reference a script file with different URIs? I assume the pod would normalize to the canonical name and would cache correctly. But now my code has two different URIs to refer to same pod:

// in source file A
a := Pod.find(`sub\foo.fan`).type("Foo")

// in source file B
b := Pod.find(`..\sub\foo.fan`).type("Foo")

a === b ???

I remember making a conscious decision to not go the Java package in each source file route. But seems like that might help for scripting, but maybe it just makes it worse...

I think I like it... So for things like resources internal to a pod, I guess that normal pod works like now, otherwise it's just a relative uri?

So what does a qname look like now?

// normalized form for all modules stays exactly the same
sys::Int#toStr          

// same as before but any char is allowed now except double colon
file:/dir/script.fan::Build  

Can I even reflect Java now?

Yes:

fansh> Type.find("[java]java.util::Date")
[java]java.util::Date

Will become:

Type.find("java:java.util::Date")

Losing my single flat list of simple str names sucks, but maybe its a necessary casualty to properly support scripting, which I think will start to take a bigger role in Fan.

Providing consistent namespace support for FFI is just as important as scripting for the design.

What happens for scripts in sub-directories, that reference a script file with different URIs?

Relative URIs will be a case where the using statement won't match the actual pod's URI. But I think that is OK, because this is the way URIs work in the real world - the normative URI is always absolute. Relative URIs will use the existing base object mechanism already built into sys::Uri.get.

I remember making a conscious decision to not go the Java package in each source file route.

Statically compiled pods don't change at all from today (other than new FFI syntax). What changes is the ability to link scripts, in which case the model is more akin to Python or Ruby, then Java.

Statically compiled pods don't change at all from today (other than new FFI syntax). What changes is the ability to link scripts, in which case the model is more akin to Python or Ruby, then Java.

I was proposing that in order to solve the script pod-name issue. But thinking about it some more, using the URI will probably suffice.

I really like the ability to use URIs with using.

One interesting use for that, off the top of my head, would be the ability to play with someone's pod on the web without even having to download it to your file system. You could simply do

using `http://example.com/fan/newLibrary.fan`

and it would work. It's quick, it's dirty, but it would let you play with Fan code with the absolute minimum of hassle.

I like all of this, I think that's a good idea.

Tactics: that's a neat idea, though that sounds a bit scary too, nice feature to write a virus :)

Interesting idea. If pointing at a CVS/SVN repo you would have a crude form of versioning, simple and very REST like.

Even if we end up not worrying about the scheme namespace, if java: means Java packages, and dotnet: means .NET namespaces, then I think it's unexpected for fan: to mean something other than Fan pods. That seems more analogous than pod: to me.

then I think it's unexpected for fan: to mean something other than Fan pods. That seems more analogous than pod: to me.

Good point, I don't really like "pod:" much either. One thing regarding terminology is that we are (or have already) moved to a point where the term "pod" means two things: in one case pod means the top-level namespace and Pod class used to reflect that namespace (which includes scripts and Java packages). Then there is the more specific case of the term where we actually mean a pod module loaded by the repo. Having that ambiguous terminology is something I try to avoid in systems I design, but haven't quite figured out the right terminology yet.

So the scheme "pod:" is neither consistent with "java:", nor is it quite clear since all those namespaces are technically pods. Other options would be:

• no scheme - the lack of a scheme always means pod module loaded from repo
• "repo:" - technically accurate, but doesn't seem quite right

My favorite solution is to just codify that inet really is normalized into inet with no scheme.

My favorite solution is to just codify that inet really is normalized into inet with no scheme.

I think I'd prefer a scheme. It might be hard to distinguish from relative URIs otherwise, among other issues.

Also, in the future, if I type "java:java.util" into Flux, what should I expect to see? Just asking to think about the meaning of these URIs in other contexts.

I think I'd prefer a scheme. It might be hard to distinguish from relative URIs otherwise, among other issues.

Well I guess the question is what is the scheme then?

Also, in the future, if I type "java:java.util" into Flux, what should I expect to see?

In HTTP parlance, the URI maps to a resource which is a Java type. How that maps to a UI representation is a user agent issue. For example in Flux the representation might be the JavaDoc for the package or class. But at the low level:

`java:java.util`.get        =>  Pod.find("java:java.util")    
`java:java.util::Date`.get  =>  Type.find("java:java.util::Date")

Ideally we'd like the pod::Type.slot namespace to be symmetrical with the URI namespace - but I need to give that some more thought.

Quick question. I haven't tested or looked in detail. Can fan gather enough info to reflect a Java package as a Fan pod?

I'm reflecting the visible-to-Fan jars now, and that works: I can view all available java packages, classes in each package, and fields/methods in class. Signatures show up as they would be used thru FFI. "Extra" info like fandoc comments of course doesn't magically appear, I'm kind of wishing for a way to link in javadocs.

Actually though, Fan reflection for Java seems to work at the class level -- reflecting a Java type, and querying it for its pod, doesn't work. I got the "package" list by iterating through the jar file.

So I guess, the info is there, but reflection won't give you Java "pods".

Quick question. I haven't tested or looked in detail. Can fan gather enough info to reflect a Java package as a Fan pod?

Java reflection works at the type/slot level. But Java doesn't support the ability to reflect packages or what types a package contains.

The compiler does this by walking all the jars and building a map of all the packages and types. But the runtime reflection doesn't have this capability (although I guess we could potentially add that).

Thanks for the info. My question was to see if it was meaningful to call Java packages (or .NET namespaces) pods. So, back to the terminology question. So, it's a question of whether it's meaningful to expose them through the Pod type.

If so, that's fine. If not, maybe make a less-detailed abstraction called Namespace (since that's now freed up from UriSpace). And Pod would be a subtype for the full Fan concept, while maybe Java packages and others are just shown as "namespaces" with less features.

Then "pod:" comes back to being a meaningful scheme for Fan pods.

If that doesn't make sense, though, then I vote "fan:" (or maybe "repo:" though it seems possible we might want to expose resources outside of pods in repos?) for the Fan pod scheme. If "fan:" for Fan pods then I vote renaming the current UriSpace scheme to "urispace:" or something.

Or maybe go for something awkward like "fanpod:" since usually you'd just say for example inet instead of `fanpod:inet` anyway.

So, it's a question of whether it's meaningful to expose them through the Pod type.

This is an excellent train of thought. When Andy and I were talking about this earlier in the week that was exactly what I was debating. Does it make sense to overload the term "pod" for both namespaces and modules? And should those concepts be broken out into two classes.

My gut says that the term "pod" and the reflective API Pod still makes sense to model the namespace issue. And I can't really see that adding a new reflective class only for pod modules really makes anything more clear or adds much value.

If "fan:" for Fan pods then I vote renaming the current UriSpace scheme to "urispace:" or something.

I was kind of thinking this same thing myself. Although the UriSpace is used extensively, so I'd hate to use such a long scheme name. One option might be sys: (although I'd be more hesitant about using that). But again, most of the use of fan: today is to reference resources inside a pod such as "fan:/sys/pod/icons/x16/save.png". If we repurposed fan: to mean pod module, then those URIs could become "fan:icons/x16/save.png".

Promoted to ticket #720 and assigned to brian

I don't know - I gave it more thought, I'm still thinking that maybe "pod:" is the right scheme name for pods (and we leave "fan:" alone).

Ticket cancelled

Replaced by updated proposal #934