The current design for adding additional jars and native libraries to the Fan Java runtime was born of necessity for packaging up the SWT cleanly.

The current design relies on logic in the launcher code (C code for win32 and Bash for Unix). This makes it hard to maintain and enhance - for example this is why I haven't extended the extension model to repos yet.

I think the solution is to move extension handling into Java into the sys.jar code.

Couple of questions:

1. This problem only immediately needs to be solved for SWT. Would we prefer a limited SWT only solution or a more general purpose solution which can be used for any jar/native lib?
2. Adding jars to Fan's runtime and having Fan handle implicit classpath issues means that visibility to those classes is only to Fan's classloader. This is potentially one piece of the larger issue about how we want Fan classloading to work. What do people think?

I like the idea.

1. I think this should be a general solution, because it will probably be needed for other things than SWT.
2. With the URLClassLoader it should work well, it would be nice if there was a method to load user jars on the fly. As long as the FanClassLoader provides some access methods to the classpath this should not be too limiting.

One of the issue with ClassLoading is dealing with security, some implementation are simplistic (fully open) and some like web containers can be quite complex. Personally at the system/language level I lean toward "open" cause that's just more flexible, and not that worrisome.

Hi Brian,

New extension design looks to be essential for people who want to run Fan code within various Java containers like OSGi or JEE.

For example we did manage to run Fan code inside Eclipse Equinox (OSGi container) and able to develop Eclipse plugins completely in Fan without any line of Java code, which is really cool. However our current approach is extremely ugly due to lack of fan repository support and classloading issues.

Ideally we'd like to be able to deploy each fan pod as OSGi bundle (eclipse plugin), so there would be 1-to-1 mapping (pod<->bundle), but we can't achieve this right now.

Current approach is to dump all the pods and typedb into single eclipse plugin, which will be deployed. This approach completely kills modularity Fan is designed for. Of course there are many other disadvantages related to development and deployment with this approach and it looks lifeless in the long term. I guess people trying to run Fan code within other containers will meet same issues, and of course it would be great if enough level of flexibility for typedb and available/installed pods could be achieved with new repository/extensibility design.

Another big problem is classloading. Most of Java containers (JEE, OSGi, pre-OSGi Spring, pre-OSGi eclipse) offers relatively complex classloading mechanisms, which are different from container to container, but in general case all these containers use dedicated classloader per the unit of deploy (eclipse plugin, J2EE application, etc).

My experience of running Fan inside eclipse using described ugly approach is following:

• Someone developing eclipse extensions in fan, which is defined in some eclipse plugin (say "myplugin").
• eclipse trying to instantiate extension from myplugin when required, using classloader associated with myplugin. myplugin classloaded know about all the dependent plugins and can load all dependent java classes, but...
• we're reusing base Fan VM code (from sys.jar) to instantiate Fan object, and sys.jar is located in shared "fan.core" plugin, which use own classloader. This fan.core classloader do not know about all the dependencies of myplugin, so fails if Fan code from myplugin uses any Java class not present on "fan.core" classloader path.

Temporary and ugly again solution is to add all the potentially required java classes on "fan.core" classloader path, but this is another death bullet into modularity and future extensibility.

We're very anxious to run Fan seamlessly within eclipse, but this would require general purpose solutions for both of your (1) and (2) questions. We'd be happy to help you with extra hands, research, etc to make flexible repository and classloading real.

Thank you, Andrey

I agree with trying for a more general solution.

My current use case for this is the JOGL 3D stuff I was experimenting with. For that there's a set of well-known and trusted (signed) jars and libraries; a different set of libs for each of a number of different platforms.

The security issue is interesting; since the libs are signed (by Sun I think) you can distribute them and use them without worrying about signing your own code. So that's fine.

The problem is, how to do something simple like a 3D demo app, and distribute with it the libraries needed for it to run? I certainly can't be trusted to be dumping files into other people's Fan libs directory; I barely even understand what some of the systems Fan runs on look like.

I think if I could mirror the lib/ext structure in my (strawman) 3Ddemo pod, I could probably manage to include the proper jars, dlls, sos, whatever, in the "normal" subdirectories, so that there'd be at least a reasonable hope of it working on systems that I'm not even sure how to power on.

Kind of scary now that I think about it.

At least this way when I do screw it up, I won't have left my trash scattered through other people's system setup. It'll just be one pod to delete.

Climbing down off my strawhorse...

I like the idea of dynamically-loadable extensions -- that is, not just bringing them by adding them to the launcher's classpath, but being able to call a sys api to load (and unload!) an extension lib. In fact I can think of at least one good reason to make pods work this way: toolbuilders need to load a pod reflect it, then modify/reload.

So: I'm thinking the core system (sys.pod and a very few others) should load and be held locked; any other pod, with any extension libs it contains, should be loadable (and reloadable) through a sys api call.

Ideally we'd like to be able to deploy each fan pod as OSGi bundle (eclipse plugin), so there would be 1-to-1 mapping (pod<->bundle), but we can't achieve this right now.

Andrey, can you provide a set of requirements for Fan in order to achieve this?

I think if I could mirror the lib/ext structure in my (strawman) 3Ddemo pod

There is definitely the issue of whether this done on a per pod basis or a per repo basis. I was definitely thinking repo, but per pod is interesting (maybe just manually libary/jar loading from your etc/{pod} directory.

per pod is interesting

Both jars and other natives (dll, so, js, ...) inside pods would be sweet. I imagine best use case would be someone makes a Fan wrapper around one or more jars/natives as a pod, then other pods could depend on that pod to get the respective native functionality.

Andrey, can you provide a set of requirements for Fan in order to achieve this?

Sure, below is a quick dump from my head:

• REQ1: End-user MUST be able to register/unregister pods with Fan VM (repository) at run time (I guess this is what many users wants).
• REQ2: For each pod registered to repository, end-user CAN provide a classloader. If provided this classloader MUST be used by Fan VM when emitting JVM byte code for each repository.

Looks like two requirements above will be enough for very flexible Fan VM reuse and embedding. Potential design:

• For each pod Fan VM maintains metadata (part of typedb, etc) and classloader (either provided by external system or created automatically by Fan VM using pod metadata). This classloader is responsible to load any class required to emit java bytecode, emit code itself and load resources and native libraries when run this code. (I assume JVM will ask pod's classloader first to load library/resource if pod's code was emitted through pod's classloader).
• Pod's classloader will use required pods' classloaders as parents to forward resolve/load requests if this classloader can't complete request. (This may force code emit for required classes from required pods via parent classloaders, which is fine).

So for the end-user, registration of the pod will look pretty simple, like

Pods.register(`/lib/fan/mypod.pod`)

or low-level Java API for container-managed class loading:

FanVM.registerPod(podUri, classloader);

For our OSGi case, we can simply query container for all the plugins, which provisioning fan pods, and register them with FanVM (using correct classloaders, which are in turn will be able to provision all the required Java classes Fan may need to emit the bytecode).

Other benefits of classloader-per-pod approach:

• pods unloading at runtime becomes real (including garbage collection of emitted JVM classes).
• advanced visibility/security handling: pod's classloader may refuse to provision some internal (non-exported) classes (even with public visibility) to children (in pods dependency chain) classloaders.
• huge flexibility if pod author can customize classloading for her pod
• resolves other classpath related issues, for example now we can not compile Fan code, which use Java FFI on the fly within OSGi container). Point of failure is trivial: current compilerJava build "default" classpath using java.classPath properties, etc, and of course miss all of the classes deployed into our environment. Classloader provided for to-be-compiled pod will resolve this problem (if compilerJava will use proposed classloader per pod approach)

It very likely I missed many positive and negative points, just some thoughts to start discussion.

Thanks, Andrey

Proposal: Pod fragments

Note: this is an addition to classloader-per-pod, also this is not mine ideas but a things we can borrow from original eclipse plugin architecture, which later influenced OSGi a lot (and I guess revived it).

If classloader-per-pod approach looks good for Fan, and pod's classloader will be responsible for native classes (JVM, CRL) and native libraries loading, some pods heavily dependent on target platform may require huge amount of native libraries to be provisioned on any system. For example SWT need to provide binaries for each relevant os-ws-arch combination. Packaging all these libraries within single pod do not look reasonable for many cases. Pod fragments may address this issue.

Let me first introduce platform filters:

Platform filter is constraint defined against os, arch, and ws. Which tells Fan VM if this pod is relevant for current platform. Eclipse use LDAP filter string to define this constraint (do not ask me why LDAP). For example:

(& (osgi.ws=win32) (osgi.os=win32) (osgi.arch=x86))

target is win32-win32-x86

or

(& (osgi.os=linux) (osgi.arch=x86_64))

target is 64-bits Linuxes

So FanVM should refuse any attempt to register irrelevant pods and ignore such pods when scanning repository.

Back to SWT provisioning problem, pod platform filters will not help if someone want to provision Fan based product to subset of os-ws-arch targets (for example win32 and linux only). If FWT pod will include all the libraries - we'll have unused binaries included in the distribution. If we'll create many FWT pods with platform filters - common Fan and SWT Java code will be included in all pods, so we'll provision duplicate code.

Pod fragments may address this problem. Pod fragment is a part of pod content (all content allowed for pod is also allowed for fragment) with fragment-specific platform filter. When Fan VM construct pod classloader (assuming pod is allowed to run on current platform), this classloader must be able to access all the pod contents plus all the contents contributed by this pod's fragments (passed platform filtering). So for FWT pod we'll have:

fwt.pod - platform filter: none
fwt.win32-win32-x86.pod(fragment) - @platform-filter = "(& (osgi.ws=win32) (osgi.os=win32) (osgi.arch=x86))" @fragment-for = "fwt"
...

pod fragments will deliver only platform-dependent content (.dll, .so, etc)

After constructed at runtime, pod content will match target platform perfectly, without any irrelevant content provisioned to the VM.

Please share you thoughts.

-- Andrey

@Andrey

Thanks for taking the time to post your thoughts. I think there were two clear recommendations:

• allow a pluggable class loader per pod
• allow sub-sets of pod native files to be used

Not sure I completely understand the proposal of second issue.

I understand where you are coming from on class-loader per pod. I need to give that some thought. That would offer tremendous flexibility, but likewise a lot potential complexity. I definitely need to give it some thought over the next few weeks.

Is your goal to load Fan pods from within the OSGI runtime as bundles?

s your goal to load Fan pods from within the OSGI runtime as bundles?

From the integration point of view it would be very helpful to have such possibility. It seems that OSGi becomes more and more popular, and a lot of advanced frameworks choose OSGi as a deployment model.

In your OSGi bundles you could probably use an Activator to init the fan subsystem prior to exporting the services from your bundle. I use a similar approach in tradewinds to allow developers to write servlets in fan.

Is your goal to load Fan pods from within the OSGI runtime as bundles?

For my concrete task, yes: running Fan within OSGi runtime would be great. But in general one of attractive Fan features is prospective modularity, which features I hope could be in parity with modern modular frameworks and containers (I mean lightweight containers like OSGi implementations).

As an offtopic: our current technology stack is based on Eclipse OSGi container, Eclipse Modeling Framework and other cool things including Eclipse based tooling. When looked at Fan first I got a feeling that most of the features I love are designed from in the Fan already (modularity, modeling). Among with awesome language and awesome features (actors, platform-independence, etc) Fan is very attractive runtime, and we're dreaming to switch to it.

So I love Fan direction, and IMO next thing which would attract more users to Fan is feature parity with industrial technologies like OSGi for modularity, easy and flexible development, deployment, and software maintenance...

Please do not consider my "requirements" in context of OSGi runtime. We can resolve this task even with current Pod model, so please do not pay attention to my goal. I just hope many users (and I) would be happy to see better Fan in first place, and running Fan within OSGi just another good Fan code deployment use-case.

Kind Regards, Andrey

Hi Cheeser,

Thank you for the suggestion, unfortunately Activator would not help. We're facing deployment problem, you did probably experience too. The problem is following if speaking in terms of J2EE:

Problem definition:

User want to develop Web application in fan. Such application may include fan pods, native Fan classes implemented in Java, supplementary application-specific Java libraries to call-in from Fan with FFI, and a bunch of resources (gifs, jsps, and hypothetical fsps - Fan Server Pages).

Let's skip development step, and look at deployment of this application into JEE container. Basically we need to deploy Fan VM with Fan runtime (Fan Standard Libraries), and User Application. I see two solutions:

Solution #1: Deploy Fan VM + runtime as an integral part of the application. In this case every web application (application.war) will include own Fan runtime copy. That's probably good for JEE since JEE applications are isolated from the beginning (you can't extend an application by contributing some "extension" applications, you can't "reuse" web applications, etc). You can't even pass objects by reference between two web applications:

Even two of your J2EE applications refer to the same physical Java class files - they will be loaded by different classloaders, and passing object of class foo.bar.X from app A to app B (I assume there is a magical way to do this) will cause in app B failure due to foo.bar.X was loaded by different class-loaders.

While this Fan VM per JEE application approach looks good for JEE - having a copy of Fan VM per each of hundreds lightweight plugins is nonsense and has no practical meaning. I want to have a single (or a few) Fan VMs per container (JVM). You may ask why few: someone may want to run incompatible versions of Fan within single JVM, like 1.0 and 2.0 based pods and plugins (and this is doable without any tweaks in Fan, like it's possible to run many isolated Fan VMs within JEE container). And, finally, I'd like to pass Fan objects between deployed units.

Solution #2 (attempt): Let's try to run shared Fan VM for all J2EE application. This would be possible if web application classloaders will share common parent classloader and Fan VM will be loaded by such a parent. To stay JEE vendor-independent, let's add all Fan classes to JRE classes (or modify JEE container launcher to bottstrap with Fan stuff on the classpath). Some containers may offer custom mechanism to share objects/code among web applications, like Weblogic's J2EE libraries: http://download-llnw.oracle.com/docs/cd/E13222_01/wls/docs90/programming/libraries.html#1059206, but this is will not resolve problem.

After Fan Java stuff added to container itself, let's dump all the applications' pods into Fan home to resolve single typedb problem, and make all of them visible to Fan.

Now let's remove pods and Fan Java natives from web-apps (they're already dumped to Fan home and will be useless within web-app).

Let's run the applications, and expected result is... failure! The problem is Fan will not see our Java supplementary application-specific libraries deployed as a part of web-application (e.g. My_Custom_ERP_Connector or My_Propietary_Network_Protocol). The only solution is to push all the applications stuff down to container classloader. Web-applications has no sense anymore, welcome back to monolithic early Java age :).

OK both solutions do not looks like solutions, and I do not see third way until at least 2 things will be done:

* no monolithic typedb - typedb should be dynamically gathered from all the "avaiable" pods (it probably could be cached, or have fast merging from pods metadata algorithm) * flexible classloading - ideally classloader per unit of deploy. And this is common pattern for all containers I know:

1. J2EE has an application/module as unit of deploy and classloader per application/module.
2. OSGi's unit of deploy is bundle (plugin), and OSGi comes with classloader per bundle.

If pod is a unit of deploy we'd definitely need a classloader per pod.

Kind Regards, Andrey

Hi folks,

InfoQ just posted good reading on this theme: http://www.infoq.com/articles/modular-java-what-is-it

OK - guys thanks for all your input on this topic. I need to spent a bit of time digesting it and how it relates to classloaders and plugin-hooks. Then I will propose a specific design for further feedback. Might take a few weeks, but this is something I'd like to fix for next build.

Hi guys,

Alex Blewitt just posted a Draft Scala modularity reqs. Can be worth reading to figure out requirements for Fan. http://article.gmane.org/gmane.comp.lang.scala.internals/1104

Kind Regards, Andrey