We have a bit of framework in place for creating web applications, as much as was needed for Sidewalk, but its far from complete. One of the largest missing pieces is a way to build web pages easily. Authoring a web page is difficult and awkward primarily because you're always working with at least 3 very different mediums - HTML, CSS, and JavaScript. Throw in your application language and SQL and now you're up to 5.

I don't think we should try to abstract those languages, since I think its important to understand them to write effective code, but having to edit 3 different files to produce a page is both painful and fragile. And while tools like Firebug have greatly improved debugging CSS and JavaScript, it would be much more efficient if we could catch errors at compile time.

My end goal is to be able to write a complete web page in a single Fan class, while still producing 3 distinct files at runtime. There's lots of open issues here (some really big), but these are my initial thoughts.

// myApp pod
using web
using webapp

class EmailForm : Widget {
  override Void onGet() {
    body.form(
      "method='post' action='$req.uri'
       onsubmit='return myApp.EmailForm.validateEmail();'")
    body.input("name='email'")
    body.sumit
    body.formEnd  
  }

  override Void onPost() {
    email := req.form["email"]
    saveEmail(email)
    res.redirect(req.uri)
  }  

  @webappJavascript
  static Bool validateEmail(Document doc) {
    val := doc.elem("div.foo form input").value
    if (val.size > 0 && val.size < 64) {
      return true
    }
    else {
      displayError
      return false
    }
  }

  @webappCss
  const Str:Str style := [
    "form": "padding: 10px;",
    "form input": "background: #ccc;"
  ]  
}

Issue #1

We annotate Javascript and CSS code using facets, and then at compile time or runtime they get pulled out into static files. Obviously the big missing piece here is a mechanism to translate Fan source code or fcode into Javascript, and at least partial support for the sys library in Javascript. We waded in these waters before, but that was a long time ago.

Issue #2

How do those file get generated?

1. Compile time - is that a field on BuildPod or does it always look for those facets?
2. Runtime - ServiceViewStep needs to be enhanced to check if that pod has been processed, and just generate files in a temp directory.

I think we always simply collapse everything into a single file (one for CSS, one for Javascript). Tangent to that we need well known Uri's for those files, ideally independent of Namespace. Maybe just something like:

`/<podname>.js`
`/<podname>.css`

Also, it would be nice here to optionally do validation on those files and definitely minimize them. We also need to go back and add gzip to FileWeblet.

Issue #3

In order to get compile time error checking, we'd really need a DOM API written in Fan. Optionally we can use dynamic invoke:

val := doc->elem("div.foo form input")->value

I don't know if its better to make Fan->JS methods take a Document object as the first parameter, or just have a field/method on Widget to acquire it.

Issue #4

I'm not sure the best way to model CSS, I haven't really given it much thought yet.

I think the design revolves around the Fan-to-JS solution. If that is a source/AST to JS compiler, then the output files would have to be part of the build process and bundled into the pod file. If we generated from fcode, then I think we'd want to do everything at runtime (with caching where appropriate).

You might look into Links, which is a functional programming language that has achieved a similar goal. Here's a classic dictionary suggestion example:

fun lowercase(s) {
  for (var c <- s) [toLower(c)]
}

fun suggest(prefixx) client {
 replaceChildren(
  format(completions(lowercase(prefixx))),
  getNodeById("suggestions") 
 )
} 

fun format(words) {
 for (var w <- words)
  <span>
   <b>{stringToXml(w.word)}</b>
   <i>{stringToXml(w.type)}</i>:
      {stringToXml(w.meaning)}
   <br/>
  </span>
}

fun completions(prefixx) server {
 var wordlist = table "wordlist" with (
   word : String, 
   type : String, 
   meaning : String
 ) from (database "dictionary");

 debug("prefixx: " ++ prefixx);

 if (prefixx == "") []
 else {
  take(10, for (var w <-- wordlist)
            where (w.word ~ /{prefixx}.*/)
            orderby (w.word)
            [w])
 }
}

fun main() {
 <html>
  <head>
   <title>Dictionary suggest</title>
   <style>
     input {{ width:500px }}
     #suggestions {{ text-align:left; width:500px;
                     background-color:#ccccff }}
   </style>
  </head>
 <body>
  <h1>Dictionary suggest</h1>
  <form l:onkeyup="{suggest(prefixx)}">
   <input type="text" l:name="prefixx"
          class="input" autocomplete="off"/>
  </form>
  <div id="suggestions" class="suggestions"/>
  </body>
 </html>
}
main()

This might also be the time to revisit Stephen's suggestion of scrapping embedded DLSs for Fan variables and passing the whole block along with Fan type information to an outside compiler. Could do a lot more than just make Java integration a breeze.

That's interesting, and exactly what I'm after. I'll need to take a closer look.

As John says, my first thought when the requirement was stated was the plugin compiler concept. This would allow the HTML and CSS parts to use their own compilers that are specific to the problems and challenges of those languages. In other words, rather than have a single general purpose language that matches 80% of the syntax of the required DSL, have different ones that match 100% of the goals.

override Void onGet() using htmlPlugin {
  <body>
    <form method="POST" action="$req.uri" onsubmit="return myApp.EmailForm.validateEmail();">
      <input name="email" />
      <input type="submit" />
    </form>
  </body>
}

static Bool validateEmail(Document doc) using javascriptPlugin {
  var val = doc.elem("div.foo form input").value;
  if (val.size > 0 && val.size < 64) {
    return true;
  } else {
    displayError();
    return false;
  }
}

static CSS css() using cssPlugin {
  form {
    padding: 10px;
  }
  form input {
    "background: #ccc;
  }
}

I've been giving this all some more thought the past few days, and I think I've come to the conclusion that my original approach is wrong. I think you can easily break a web site down into two different camps:

1. Static HTML pages (use "static" loosely)
2. Webapps (Gmail, MobileMe, etc)

The static HTML side is very well designed, with great separation of structure (HTML) and presentation (CSS). Webapps on the other hand, explicitly abstract those concepts in order to create a framework that makes it easy to develop applications. The two are fundamentally different, and how you approach the two are very different.

There's a huge gray area between those two though, and I think alot of existing frameworks and sites try to straddle the lines there (myself included), and its the source of their complexity and inadequacy. It may be possible to leverage the beauty of semantic HTML and CSS with the benefits of an application framework, but I've tried twice and didn't really get anywhere, and I have yet to see anyone else do it well either (if you know someone who did though - please send my way!).

One of my main concerns with client-heavy applications has always been the speed of JavaScript and DOM consistency, but there's tons of momentum now to address those issues, and make client-side applications a true alternative. HTML is not an interface design language, so it will always have its sore spots, but you can get pretty close now-a-days. And things will only get better with developments like HTML5.

So I've switched gears, I think we throw away the current webapp::Widget stuff - and leave the close to-the-metal web::Weblet API for that stuff, and focus on a true application framework - ideally closely modeled off the FWT. The Fan-To-JavaScript compiler is still a prerequiste here - and needs to be the first priority. Though I think mixing client and server side code in a single type goes away, which definitely simplifies things a bit.

Andy, could you please specify your intention a bit more in detail?

The original design seems a bit like a glorified PHP, intermixing logic and output + some autogeneration of missing pieces.

There are other approaches to that -- JSF -- an attempt to put all html/css/javascript code behind (quite huge) components which would just be picked by and used by developers from a component library.

Or Wicket-like ("web ORM") approach where the cut is made between presentation and logic. You have (seemingly) static html+css and then pure java components -- event driven (pages are instantinated by new Page(param1, param2) + autogenerated boilerplate javascript and AJAX code. You also pick pre-defined components but is is much easier to make customizations.

Or "good old" (yuck!) approach of JSP+a million of underpowered DSL languages

Or, as always, there is a bare-boned approach piping data from/to sockets (via servlet or similar API).

I don't really get your new idea. Which one of the above is the closest and what should be the major differences? Sorry for this portion of oversimplification but I think we all know these approaches well (and their pitfalls).

Its still a bit hazy, but I think the closest thing would be GWT. If you use the standard widget library you never write a single line of HTML/CSS/Javascript - its 100% Fan.

There's a huge impedance mismatch between what HTML provides and what an "Application" wants. So I think you simply have to throw away to notion of pages and forms, and treat it much more like you would approach a desktop application.

Right now I'm thinking we send the "page" over as a serialized widget tree and its gets built client side via the DOM. So I think the effects how you write custom widgets.

I'll post more details as things solidify in my head.

This means you need to turn fcode into JavaScript (probably not too hard, but if fcode is extremely low level, then performance might suffer), and provide a way to write JavaScript backends to things like FWT.

You must be part of the way there now, since you're supporting both Java and .NET.

Yep, this whole effort hinges on the Fan-to-JavaScript compiler, and that needs to be the first priority.

Turning fcode into JavaScript would be nice, although I think it is easier to compile from source form and use the AST. I've written a Java bytecode to C compiler before, and it can be done - but it is pretty ugly.

I should also say that GMail style web applications aren't the kind I work on. The kind of apps I build are simple RESTful request response where users fill in forms and click like to navigate the app. Very little javascript. No complex "widgets".

As such a GWT solution isn't interesting to me, as I want to actually write the HTML, CSS and javascript. I still suspect that the largest proportion of server apps are still of this style (its just that the most famous are more complex). So, do you focus the efforts of Fan on the shiny lights of a GWT style solution, or the conceptually much simpler (and I believe more widely used) "long tail" of enabling HTML and CSS generation.

I'm not sure it has to be that polarizing. For instance, I would expect this framework would make it easy to write Sidewalk, which I think is reasonably representative of apps on the web today. The difference is you need to be willing to approach the problem differently.

I think the world has moved past simple GET/POST request cycles, which kills the user experience on the web. Take Sidewalk for example. Conceptually its a pretty simple application, but at least half the effort was in seemingly small tasks, like doing preview, or validating your password.

And that's largely because of the huge disconnect between writing HTML on GET, then trying to access the DOM in JavaScript, then sending data over to the server to do something useful, then responding back on the client.

There's a big problem space here, but I'm most interested in fixing these root issues, which are the things that make writing webapps hard.

I'd have to agree that you really want to focus on what you call "webapps" with heavy javascript utilization, especially on top of tools like jquery, mootools, yui, etc. I would also agree that "webapps" are not yet the majority, but they are the (near) future - that's where google, MS, Adobe, etc are going, and it ought to be your primary focus. Sorry to polarize, but it is already happening.