I can't run the fantom on Android by JarDist. I compare the java byte code again and again. Finally, I find the fantom not set the ACC_SUPER flag.
My code:
class Widget
{
virtual Void paint()
{
echo("widget")
}
}
class Button : Widget
{
override Void paint()
{
echo("button")
super.paint()
}
}
//in java
Button b = Button.make();
b.paint();
Error:
button //print button
// this exeception occur on invokespecial instructions: super.paint();
java.lang.NoSuchMethodError: Widget.paint
brianSun 9 Oct 2011
Promoted to ticket #1670 and assigned to brian
Okay, I need to investigate that. Have you tried setting that flag in Java fanx.emit code to see if it resolves the problem?
go4Sun 9 Oct 2011
I don't know how to get it fixed. I will verify that if you give a patch.
Thanks.
go4Sun 9 Oct 2011
I find another issue:
invokespecial instruction can only be used to invoke a private method and super method. But the fantom not really have private method. so there is expected invokevirtual instruction.
I don't know why Android is so careful on java byte code. But it will be sweet if fantom could run on Android.
brianMon 10 Oct 2011
I don't think that error has anything to do ACC_SUPER actually now that I've read about it. But if you want to try it, you can set that flag in FTypeEmit.java line 73, something like this:
If you want to try that we can see if it works or not, then take the problem from there.
go4Mon 10 Oct 2011
It bring more problems.
In Java Virtual Machine implementations prior to version JDK 1.02, this instruction was called invokenonvirtual, and was less restrictive than invokespecial - it wasn't limited to invoking only superclass, private or <init> methods. The class access flag ACC_SUPER (see Chapter 4) is used to indicate which semantics are used by a class. In older class files, the ACC_SUPER flag is unset. In all new classes, the ACC_SUPER flag should be set, indicating that the restrictions enforced by invokespecial are obeyed. (In practice, all the common uses of invokenonvirtual continue to be supported by invokespecial, so this change should have little impact on JVM users).
This means that set ACC_SUPER will change the semantics of invokespecial.
brianMon 10 Oct 2011
Your code example is the typical example of where you have to use invokespecial - it is the only way to invoke Widget.paint non-virtually. I don't think ACC_SUPER does anything but make use of invokespecial more restrictive. And I think things already pretty strict because that is why I can't allow super calls to mixins.
What I would suggest is dump the Java bytecode from DistJar for those two classes and see if you notice anything different b/w Java. It should be exactly the same bytecode so not sure why it doesn't work.
go4Mon 10 Oct 2011
The Android don't complain NoSuchMethodError when I set the ACC_SUPER flag. But other issues raised( the fwt can't work correctly ).
The invokespecial is different from invokeNovirtul. invokespecial is used to invoke:
I don't believe we do anything that prevents us from setting the ACC_SUPER flag - we don't use invokespecial except in the three cases you listed (and limit Fantom at the language level as a work around where we truly need an invokenonvirtual opcode).
So when you say fwt doesn't work correctly what does that mean? FWT is all SWT based, how would it ever work in Andriod?
brianMon 10 Oct 2011
I ran thru the entire test suite using the ACC_SUPER flag, and there is only one thing that really makes a difference: using a named super to call a method which isn't your direct parent:
class A { virtual Str x() { "A.x" } }
class B : { override Str x() { "B.x" } }
class C : { override Str x() { "C.x" } }
In Fantom we allow C to call A.super.x to explicitly use class A's version. The ACC_SUPER flag prevents that and forces you to only be able to call B.x.
It does appear that we use that feature in fwt::ContentPane to override add in one place.
go4Tue 11 Oct 2011
invokespecial can be used to call a private method of this.
But, the fantom's private method mapping to java's package method. In fact, have no private protection in the emitted java bytecode. See the fanx/emit/FTypeEmit.java, line 373.
In this case, the Android refuse to call noPrivate method by invokespecial.
go4Sat 15 Oct 2011
I add a CompilerStep to rename the private slot. The private slot foo is renamed to foo$podName$typeName. It works well on Android, although there is little issue on reflection and named super.
Code:
class RenamePrivate : CompilerStep
{
new make(Compiler compiler) : super(compiler) {}
override Void run()
{
log.debug("RenamePrivate")
walk(compiler, VisitDepth.expr)
}
override Void visitMethodDef(MethodDef def)
{
if (needRename(def)) def.name = newName(def)
}
override Void visitFieldDef(FieldDef def)
{
if (needRename(def)) def.name = newName(def)
}
private Bool needRename(CSlot def)
{
if (def.isPrivate && !def.isStatic && !def.isCtor && !def.isNative &&
!def.name.startsWith("instance\$init"))
{
return true
}
return false
}
private Str newName(CSlot def)
{
"$def.name\$$pod.name\$$def.parent.name"
}
override Expr visitExpr(Expr expr)
{
if (expr is CallExpr)
{
c := expr as CallExpr
if (needRename(c.method)) c.name = newName(c.method)
}
return expr
}
}
I mean, if using invokevirtual instead of invokespecial to call a private method, private slot name conflicts will occur. So, I try to do name mangling for private slot.
You know, all things is public in javascrpt. There is also have namespace problem. See the `http://fantom.org/sidewalk/topic/1651#c11356`
brianTue 18 Oct 2011
There are several issues:
it appears that Andriod requires ACC_SUPER which means we must remove support for named supers above your direct base class
it appears that Andriod won't let you use invokespecial on a package scoped method, so we will need to switch to invokevirtual.
It appears that Java correctly dispatches invokevirtual with package private methods. But I couldn't find it in the JVM spec, so I've asked the experts.
Just to list some of the critical issues:
private methods are not emitted as private Java methods because then closures can't access them without method wrappers (what Java does for inner classes)
both internal and private methods have potential for inter-pod naming conflicts which hopefully can be solved with JVM package private dispatch behavior
JS will likely require name mangling, but I'd like to avoid that at compiler level and keep it a target runtime issue
go4Tue 18 Oct 2011
Very exciting summary. Thanks.
Actually, Dalvik(Android VM) is not a JVM. The compiler translate the jar to Dalvik .dex file:
fcode => java bytecode => dex file.
The Dalvik have two opcode for invoke novirtul: invoke-supper and invoke-direct. It is a pity, they have the same limits as invokespecial.
DanielFathTue 18 Oct 2011
So we need another target, no?
fcode => dex files.
brianSat 22 Oct 2011
Just to note: John Rose did confirm the way invokevirtual works with package scoped methods. This means we can continue to emit private/internal methods to package scoped methods in the JVM without resorting to name mangling.
So I think the only thing we lose is the ability to use a named super more than one level above your direct base class. Other than that, we should be able to safely switch to ACC_SUPER and invokevirtual on private method methods.
brianMon 14 Nov 2011
@go4
I have pushed the first two changes required for Andriod:
I have not made the final required change which is to switch non-virtual method calls from invokespecial to invokevirtual. This can be tested in a fairly safe way by changing compiler::CodeAsm.fan around line 1286 like this:
Can you test that out in conjunction with the other changes and see if that gets us where we need to be for Andriod?
That change is mostly safe, but it creates some loop holes in that private methods inside the same pod might not resolve correctly. So if we really go down this path we'll have to tackle the whole clusterfuck of generating public wrapper methods around private methods for closure access so we can actually call private methods nonvirtually. But I don't want to go down that path until you have verified that this will all work on Andriod and we are sure this is the right path.
go4Wed 16 Nov 2011
Yes, It works.
In fact, I have successfully used Fantom on Android more than a month. It works very well， Thanks.
I don't think this is a big problem. just a compiler warning is enough.
Perhaps all the problems are a Android VM bug.
brianWed 16 Nov 2011
Those tests are the ones that actually check that private methods aren't being treated as virtual :-) So that would be expected with the change from CallNonVirtual to CallVirtual.
So other than that, the full test suite passes on Andriod? That is pretty sweet.
I'm going to post a build this week, but probably won't officially make this change until following build. The remaining issue is now this:
We can't make private methods actually be private because then synthetic closure classes can't access the private methods (Java generates wrapper methods for inner classes to handle this); for Fantom I emit private methods as package private
Andriod doesn't let us use invokespecial on package private methods
If we switch to use invokevirtual, then it is possible that a subclass which creates a private method with the same name as a superclass in the same pod will be incorrectly treated an override
Potentially this issue is related to JavaScript method hiding for internal and private methods
There are three basic strategies I see we can use:
Emit Fantom private methods as private Java methods and generate internal wrappers for closure use (like Java does for inner classes)
Disallow subclasses to reuse internal or private method names which would override a method in a parent class in the same pod (treat these as compiler errors and prevent it happening in first place)
Name mangle all private and internal methods (this might have benefit of helping JavaScript issues)
I'm sort of debating between B and C myself. But comments welcome
Hi Brian, I make a new way to fix this. Follow the strategy B:
Disallow subclasses to reuse internal or private method names which would override a method in a parent class in the same pod (treat these as compiler errors and prevent it happening in first place)
This is also great, I'd love to see official Fantom on Android!
brianSun 28 Jan 2018
Looking forward to seeing your designs on the other issues.
I can't remember why I said prevent in the same pod? Seems like you'd have to do for all pods, which in turn would break encapsulation and create a fragile base class problem. But that post is from seven years ago
go4Sun 28 Jan 2018
The Fantom to Java protection modifiers mapping (from FTypeEmit.java line 418):
private => package-private
protected => public
internal => public
The private name is visible in the same pod and internal name is visible for all pods.
BTW: We nedd INVOKESPECIAL to invoke method of superclass's version and INVOKEVIRTUAL for others. There are already CallCtor CallMixin in opcode, so CallSuper would be more easy to distinguish from CallNoVirtual.
go4 Sun 9 Oct 2011
I can't run the fantom on Android by JarDist. I compare the java byte code again and again. Finally, I find the fantom not set the ACC_SUPER flag.
My code:
class Widget { virtual Void paint() { echo("widget") } } class Button : Widget { override Void paint() { echo("button") super.paint() } } //in java Button b = Button.make(); b.paint();Error:
brian Sun 9 Oct 2011
Promoted to ticket #1670 and assigned to brian
Okay, I need to investigate that. Have you tried setting that flag in Java fanx.emit code to see if it resolves the problem?
go4 Sun 9 Oct 2011
I don't know how to get it fixed. I will verify that if you give a patch.
Thanks.
go4 Sun 9 Oct 2011
I find another issue:
invokespecialinstruction can only be used to invoke a private method and super method. But the fantom not really haveprivatemethod. so there is expectedinvokevirtualinstruction.I don't know why Android is so careful on java byte code. But it will be sweet if fantom could run on Android.
brian Mon 10 Oct 2011
I don't think that error has anything to do ACC_SUPER actually now that I've read about it. But if you want to try it, you can set that flag in FTypeEmit.java line 73, something like this:
If you want to try that we can see if it works or not, then take the problem from there.
go4 Mon 10 Oct 2011
It bring more problems.
This means that set ACC_SUPER will change the semantics of
invokespecial.brian Mon 10 Oct 2011
Your code example is the typical example of where you have to use invokespecial - it is the only way to invoke
Widget.paintnon-virtually. I don't think ACC_SUPER does anything but make use of invokespecial more restrictive. And I think things already pretty strict because that is why I can't allowsupercalls to mixins.What I would suggest is dump the Java bytecode from DistJar for those two classes and see if you notice anything different b/w Java. It should be exactly the same bytecode so not sure why it doesn't work.
go4 Mon 10 Oct 2011
The Android don't complain NoSuchMethodError when I set the ACC_SUPER flag. But other issues raised( the fwt can't work correctly ).
The invokespecial is different from invokeNovirtul. invokespecial is used to invoke:
invokespecial reference
All Java compilers since JDK 1.1 always set the ACC_SUPER flag.
Why ACC_SUPER
And Android only support jvm bytecode 1.4-1.6.
brian Mon 10 Oct 2011
I don't believe we do anything that prevents us from setting the ACC_SUPER flag - we don't use invokespecial except in the three cases you listed (and limit Fantom at the language level as a work around where we truly need an invokenonvirtual opcode).
So when you say fwt doesn't work correctly what does that mean? FWT is all SWT based, how would it ever work in Andriod?
brian Mon 10 Oct 2011
I ran thru the entire test suite using the ACC_SUPER flag, and there is only one thing that really makes a difference: using a named super to call a method which isn't your direct parent:
class A { virtual Str x() { "A.x" } } class B : { override Str x() { "B.x" } } class C : { override Str x() { "C.x" } }In Fantom we allow
Cto callA.super.xto explicitly use class A's version. The ACC_SUPER flag prevents that and forces you to only be able to callB.x.It does appear that we use that feature in
fwt::ContentPaneto overrideaddin one place.go4 Tue 11 Oct 2011
invokespecialcan be used to call a private method of this.But, the fantom's
privatemethod mapping to java'spackagemethod. In fact, have no private protection in the emitted java bytecode. See the fanx/emit/FTypeEmit.java, line 373.In this case, the Android refuse to call
noPrivatemethod by invokespecial.go4 Sat 15 Oct 2011
I add a
CompilerStepto rename the private slot. The private slotfoois renamed tofoo$podName$typeName. It works well on Android, although there is little issue on reflection and named super.Code:
class RenamePrivate : CompilerStep { new make(Compiler compiler) : super(compiler) {} override Void run() { log.debug("RenamePrivate") walk(compiler, VisitDepth.expr) } override Void visitMethodDef(MethodDef def) { if (needRename(def)) def.name = newName(def) } override Void visitFieldDef(FieldDef def) { if (needRename(def)) def.name = newName(def) } private Bool needRename(CSlot def) { if (def.isPrivate && !def.isStatic && !def.isCtor && !def.isNative && !def.name.startsWith("instance\$init")) { return true } return false } private Str newName(CSlot def) { "$def.name\$$pod.name\$$def.parent.name" } override Expr visitExpr(Expr expr) { if (expr is CallExpr) { c := expr as CallExpr if (needRename(c.method)) c.name = newName(c.method) } return expr } }compiler/fan/assembler/CodeAsm.fan, line 1287:
go4 Sun 16 Oct 2011
I mean, if using invokevirtual instead of invokespecial to call a private method, private slot name conflicts will occur. So, I try to do name mangling for private slot.
You know, all things is public in javascrpt. There is also have namespace problem. See the `http://fantom.org/sidewalk/topic/1651#c11356`
brian Tue 18 Oct 2011
There are several issues:
It appears that Java correctly dispatches invokevirtual with package private methods. But I couldn't find it in the JVM spec, so I've asked the experts.
Just to list some of the critical issues:
go4 Tue 18 Oct 2011
Very exciting summary. Thanks.
Actually, Dalvik(Android VM) is not a JVM. The compiler translate the jar to Dalvik .dex file:
The Dalvik have two opcode for invoke novirtul: invoke-supper and invoke-direct. It is a pity, they have the same limits as invokespecial.
DanielFath Tue 18 Oct 2011
So we need another target, no?
brian Sat 22 Oct 2011
Just to note: John Rose did confirm the way invokevirtual works with package scoped methods. This means we can continue to emit private/internal methods to package scoped methods in the JVM without resorting to name mangling.
So I think the only thing we lose is the ability to use a named super more than one level above your direct base class. Other than that, we should be able to safely switch to ACC_SUPER and invokevirtual on private method methods.
brian Mon 14 Nov 2011
@go4
I have pushed the first two changes required for Andriod:
I have not made the final required change which is to switch non-virtual method calls from invokespecial to invokevirtual. This can be tested in a fairly safe way by changing compiler::CodeAsm.fan around line 1286 like this:
Can you test that out in conjunction with the other changes and see if that gets us where we need to be for Andriod?
That change is mostly safe, but it creates some loop holes in that private methods inside the same pod might not resolve correctly. So if we really go down this path we'll have to tackle the whole clusterfuck of generating public wrapper methods around private methods for closure access so we can actually call private methods nonvirtually. But I don't want to go down that path until you have verified that this will all work on Andriod and we are sure this is the right path.
go4 Wed 16 Nov 2011
Yes, It works.
In fact, I have successfully used Fantom on Android more than a month. It works very well， Thanks.
There are two test failed:
I don't think this is a big problem. just a compiler warning is enough.
Perhaps all the problems are a Android VM bug.
brian Wed 16 Nov 2011
Those tests are the ones that actually check that private methods aren't being treated as virtual :-) So that would be expected with the change from CallNonVirtual to CallVirtual.
So other than that, the full test suite passes on Andriod? That is pretty sweet.
I'm going to post a build this week, but probably won't officially make this change until following build. The remaining issue is now this:
There are three basic strategies I see we can use:
I'm sort of debating between B and C myself. But comments welcome
go4 Thu 17 Nov 2011
I just run the testSys.
I can't run the test by JarDistEnv:
The some IO failed on Android:
Most of the causes is:
go4 Sat 27 Jan 2018
Hi Brian, I make a new way to fix this. Follow the strategy B:
change set
SlimerDude Sat 27 Jan 2018
This is also great, I'd love to see official Fantom on Android!
brian Sun 28 Jan 2018
Looking forward to seeing your designs on the other issues.
I can't remember why I said prevent in the same pod? Seems like you'd have to do for all pods, which in turn would break encapsulation and create a fragile base class problem. But that post is from seven years ago
go4 Sun 28 Jan 2018
The Fantom to Java protection modifiers mapping (from FTypeEmit.java line 418):
The private name is visible in the same pod and internal name is visible for all pods.
BTW: We nedd INVOKESPECIAL to invoke method of superclass's version and INVOKEVIRTUAL for others. There are already CallCtor CallMixin in opcode, so CallSuper would be more easy to distinguish from CallNoVirtual.