Created small patch to allow read/write of Fan objects using Json pod. Just "continuation" of ideas in original code, existing tests are passing, checked that works correctly with maps, lists, enums and composite objects

diff -r a1ba1fe8606f src/json/fan/JsonParser.fan                                         
--- a/src/json/fan/JsonParser.fan       Mon Oct 12 17:58:10 2009 -0400                   
+++ b/src/json/fan/JsonParser.fan       Mon Oct 19 18:37:38 2009 +0700                   
@@ -29,7 +29,7 @@                                                                        
       return parseObject                                                                
   }                                                                                     
                                                                                         
-  private Str:Obj? parseObject()                                                        
+  private Obj parseObject()                                                             
   {                                                                                     
     pairs := Str:Obj?[:] { ordered = true }                                             
                                                                                         
@@ -40,7 +40,7 @@                                                                        
     while (true)                                                                        
     {                                                                                   
       skipWhitespace                                                                    
-      if (maybe(JsonToken.objectEnd)) return pairs                                      
+      if (maybe(JsonToken.objectEnd)) return checkObject(pairs)                         
                                                                                         
       // FIXIT would like pair to be a 2-tuple                                          
       // OR a map with atom/symbol keys!                                                
@@ -51,9 +51,57 @@                                                                       
                                                                                         
     expect(JsonToken.objectEnd)                                                         
                                                                                         
-    return pairs                                                                        
+    return checkObject(pairs)                                                           
   }                                                                                     
                                                                                         
+  private const static Str fanType := "fanType"                                         
+  private const static Str fanValue := "fanValue"                                       
+                                                                                        
+  ** If this is a fan obj, try to create obj, return map in case of any failure         
+  private Obj? checkObject(Str:Obj? obj)                                                
+  {                                                                                     
+    if(!obj.containsKey(fanType)) return obj                                            
+    type := Type.find(obj[fanType], false)                                              
+    if(type == null) return obj                                                         
+                                                                                        
+    if(type.facet(@simple, false, true))                                                
+    {                                                                                   
+      if(!obj.containsKey(fanValue)) return obj                                         
+      method := type.method("fromStr", false)                                           
+      if(method == null) return obj                                                     
+      return method.call(obj[fanValue])                                                 
+    }
+
+    result := type.make
+    success := obj.eachWhile |v, k|
+    {
+      if(k == fanType) return null //skip field
+      f := type.field(k, false)
+      if(f == null) return obj
+      setField(result, f, v)
+      return null
+    }
+    if(success != null) return obj
+    return result
+  }
+
+  private Void setField(Obj instance, Field f, Obj? value)
+  {
+    if(value == null) { f[instance] = value; return }
+
+    //special handling for containers
+    if([f.of, value.type].all |t| {t.fits(List#)} ||
+       [f.of, value.type].all |t| {t.fits(Map#)} )
+    {
+      if(f[instance] == null) f[instance] = f.of.make //empty container
+      container := f[instance]
+      container->clear
+      container->addAll(value)
+      return
+    }
+
+    f[instance] = value
+  }
   private Void parsePair(Str:Obj? obj)
   {
     skipWhitespace
diff -r a1ba1fe8606f src/json/fan/JsonWriter.fan
--- a/src/json/fan/JsonWriter.fan       Mon Oct 12 17:58:10 2009 -0400
+++ b/src/json/fan/JsonWriter.fan       Mon Oct 19 18:37:38 2009 +0700
@@ -47,7 +47,7 @@

     this.out.writeChar(JsonToken.objectStart)
     writePair("fanType", type.signature)
-    if (type.facet(@simple, null, true))
+    if (type.facet(@simple, false, true))
     {
       this.out.print(",")
       writePair("fanValue", obj.toStr)

While I like the patch, I think this topic warrants further discussion before we do anything.

I would summarize the basic problem as this: How can I round-trip serialize Fan types with JSON and not loose the original Fan type information?

For example today, if I use Fan serialization to write out a DateTime, I get back a DateTime. One of the original requirements for JSON was to have an "enhanced" format which includes enough type information to maintain full fidelity with the type system. I haven't studied the problem to see what conventions might be in use already, but I think we need to have the format discussion before we commit any code.

In JsonWriter code there already was a convention that if we write Fan object which is not a Json-ready type (like map, list, bool, number, string), we write it as map with additional pair "fanType":<fully-qualified type name>. If object is @simple, then it's toStr value put to "fanValue" pair. So simple type like Range serialized like this:

{"fanType":"sys::Range","fanValue":"0..<5"}

For composite objects, JsonWriter uses reflection to write all fields in the same way. So, for example, Fan object below

p := Person 
{
  firstName = "Ivan"
  lastName = "Inozemtsev"
  numbers = [PhoneNumber { phoneType = PhoneType.mobile; number = "+79139494750"}]
}

Will be serialized like this (formatted for readability)

{
  "fanType":"jsonTest::Person",
  "firstName":"Ivan",
  "lastName":"Inozemtsev",
  "numbers": 
    [
      {
        "fanType":"jsontTest::PhoneNumber",
        "number":"+79139494750",
        "phoneType": {"fanType":"jsonTest::PhoneType", "fanValue":"mobile"}
      }
    ]
}

What I did is fixed small error in JsonWriter (which caused NPE when trying to write Fan object) and extended JsonParser to check for "fanType" keys in maps and attempt to create an object if found

Btw, DateTime is also supported automatically, after serialization it looks like this:

{"fanType":"sys::DateTime","fanValue":"2009-10-20T12:58:59.651+07:00 Novosibirsk"}

It might seem that we can skip writing type info for simple types and write them just like strings and then during deserialization check if field type is simple, call field.of.fromStr method. But there might be a problem when there is a hierarchy of simple types and we deserialize object of type which has field of base class. For example:

@simple class A {...}
@simple class B : A {...}
@simple class C : A {...}

//Can't be deserialized from json correctly
class D 
{
  A a := B()
}

The code that supports the "fanType" key was speculative and never really discussed, so don't assume that just because it is there means it is correct.

That said, it would be useful to look at what others are doing - I know .NET and Scala have some decent ideas around this.

My personal preference would be to keep the resultant JSON as clean as possible since JSON is all about simplicity and clarity - it plays very nicely with dynamic languages. I guess my question is this: is round trip serialization support in JSON a good goal to strive for, or is fan native serialization sufficient and if you want to write out JSON you basically just convert your object to a map?

Ok, I was not aware that the current code was working this way:

fansh> Json.write(Sys.out, Date.today)
{"fanType":"sys::Date","fanValue":"2009-10-20"}

I would have expected a simple type like Date to be encoded as just "2009-10-20".

I had originally discussed with Kevin two modes: a clean JSON dynamic encoding that would not support round-tripping, and a bulker type aware encoding that would support round-tripping.

I think for JSON the right move is to output a clean, simple encoding and let's not worry about adding type information for round-tripping. I think that is an abuse of JSON. If you really want full Fan types, then you are better off using Fan serialization or XML.

So here is my proposal:

1. Simples are serialized using toStr
2. Serializables are serialized as JSON objects using normal @transient semantics of which fields should be serialized
3. Anything read back from JSON is only serialized into Maps and Lists, any further mapping to Fan types is outside the scope of the standard JSON API

Promoted to ticket #796 and assigned to brian

Completely agree now, json api should be as clean as possible However, probably it is possible to add one more class to json pod (or some other pod from standard library), which will do Fan type mapping? Because otherwise there might be situation when many people write the same code for breaking obj to map of lists of maps and combining them back to object. This can be done with methods like this:

** break object to map of primitives recursively
Obj? decompose(Obj object, |Type t, Str:Obj? map| mapType := 
     |Type t, Str:Obj? map| { map["fanType"] = t.qname })
{
  ...
}

** create object from map
Obj? compose(Str:Obj? map, |Str:Obj? -> Type| getType := 
     |Str:Obj? m -> Type| { Type.find(m["fanType"])} )
{
  ...
}

@ivan,

that is a great idea, in fact maybe that is the answer to your other post about how to solve setting const fields by reflection, something like a new method on Type that constructs an object and sets all its fields:

Obj Type.compose(Str:Obj? fields)

Then assuming the JSON used the field names, you could do just do this:

MyType#.compose(JSON.read(in))

Yes, I already thought that these topics are related and implementing such method in standard library may provide easy solution for creating const objects.

Unfortunately having just one parameter is not enough - it is necessary to get type info from somewhere (in case class Foo { Base a := Derived() }). Or this method is not supposed to be recursive?

Or this method is not supposed to be recursive?

I think recursively mapping nested objects to types will be tricky. That will definitely be more difficult make the JSON API and the "Batch Reflection" API composable. I'll have to give that one some thought.

That's definitely too tricky in general case, so I suggested to have a function Str:Obj?->Type as a parameter. In this case it's just user's responsibility to provide the correct mapping (depending on task, user can use some values from map to determine the correct type, or, having knowledge about base and derived types he use, determine the type by checking presence of some keys in the map). More precisely, signature for compose may look like this:

** Obj Type.compose(Str:Obj? fields, |Field, Str:Obj?->Type|? mapper := 
                                       |Field f -> Type| { f.of })

I am not sure the "map" operation belongs on the reflection API. Seems like maybe baking too much into the low level.

I am thinking that might be better to require this:

map := Json.read
// tranform map (maybe looking at field types)
obj := MyType#.make([,], map)

Seems like the transformation of the map doesn't belong in a reflection API, but rather is more of a "utility" custom to the application.

Ticket resolved in 1.0.48

OK, I've gone through the JSON APIs and tightened up the semantics with regard to documentation and test suite:

**
** Write the given object as JSON to the given stream.
** The obj must be one of the follow:
**   - null
**   - Bool
**   - Num
**   - Str
**   - Str:Obj?
**   - Obj?[]
**   - [simple]`docLang::Serialization#simple` (written as JSON string)
**   - [serializable]`docLang::Serialization#serializable` (written as JSON object)
**
public static Void write(OutStream out, Obj? obj)

**
** Read a JSON object from the given stream and return
** one of the follow types:
**   - null
**   - Bool
**   - Int
**   - Float
**   - Str
**   - Str:Obj?
**   - Obj?[]
**
public static Obj? read(InStream in)

I also added a Json.writeToStr for convenience. No more "fanType" or "fanValue" keys are written out. Simples are output as normal string, and serializables as normal JSON objects.

I also tweaked the Json methods to work with any object type including the scalars (Bool, Int, Str). Technically only an object or list may be used as a top-level document type. But it seems useful to allow the API to work with the scalar types also.

This is an old thread now, but it's a topic that has come up for me recently. Given that there is so much focus on Javascript performance in browsers, and native JSON support is likely in modern browsers, do you think that there is a performance benefit in providing full fidelity JSON serialization for fantom?

As is, it appears that there is a step in the middle to convert to and from a map, and the JSON serialization process just deals with maps. Is there a significant performance benefit to skipping the mapping step?

do you think that there is a performance benefit in providing full fidelity JSON serialization for fantom

Hooking into the native JSON impl will definitely be more performant. But to expose that object into the Fantom runtime you still need to map it into proper Fantom types. So you still take a hit (but that will be likely (much?) less that the string processing done in pure Fantom).

As is, it appears that there is a step in the middle to convert to and from a map

Poked thru the code, not sure I saw what you were pointing at?