Been trying to put together a proxy Weblet for my WispServer using the WebClient and have had mixed results. It looked like there is some build in support using the proxy property on the WebClient but I have yet to figure out how it's intended to be used. Is there an easy way to define a WebClient as a proxy and subsequently call it from an HttpReq?. I feel like I've exhausted all the parameter tweaks I can think of on both of these at this point and continue to get mixed results. i.e. a post goes through on the WebClient but the HttpReq throws an Uncaught DomException if the proxy is set. If the proxy isn't set, then the WebClient seems to throw an Http 400 error. Just a few examples when trying to use post. I won't list all the issues as I can't remember all the things i've tried but having similar inconsistent results with Get when trying to tweak the parameters of these classes.

Is there a simple way to setup a proxy Weblet that can handle basic GET and POST commands?

the proxy property on the WebClient but I have yet to figure it out

The proxy settings on WebClient are for connecting to an existing Internet Proxy. i.e. when set up, the WebClient will then route all requests through the proxy. Note that HTTP request headers are slightly different when talking to a proxy than when talked to the server direct.

For a reliable method of picking up system Internet Proxy settings, use Java's proxy-vole library. See the Butter Proxies article for info on how to use it in Fantom.

Is there a simple way to setup a proxy Weblet

Creating a proper standards compliant proxy server is quite a challenge.

But it sounds like you just want to use WebClient to make a HTTP request and set the results on the HttpRes.

There is no Weblet to do this because (I imagine) the configuration to set which incoming URLs should map to which outgoing URLs would be bigger than the code itself!

The code would need to do the following:

• set URL, method, and HTTP headers on WebClient from HttpReq
• make the call
• set the StatusCode, HTTP headers on HttpRes from WebClient
• read the WebClient body and set it on HttpRes

BedSheet does this in its ProxyMod class, and Andy's draft mini-framework also does similar in the DevMod class.

I've been trying passing the headers around to do build a proxy but the HttpReq seems to have issues or some settings i'm not aware of to make it work correctly. Trying to go directly to a source URL where a proxy isn't needed, I need to post form data in my use case and if I use postForm method, it always throws:

sys::Err: Failed to execute send on XMLHttpRequest: Failed to load <url>.

There seemed to be a few nuisance errors about unsafe headers so I filtered those out before sending but it still fails with the same issue.

So the flow of data right now is this:

1. A Weblet getes called via the onGet method from a web browser.
2. The headers from the onGet are stored in the envelop.
3. The onGet method builds out HTML using the WebUtil.jsMain method to load the fantom code compiled as javascript.
4. That method passes the headers to an HttpReq object with formData and attempts a post but alwasy receives the error indicated above.

Am I missing a step somewhere?

The proxy should be transparent to your HttpReq - so from that error sounds more like the URI is malformed - whats the URI you are trying to pass?

This was without using a Proxy. Might need to move this to a new thread since this is just using the HttpReq (which would likely be moved to go through a proxy later).

I know it's not the URI as when I use a get to read the URI or pass it into a WebBrowser object to load, it's fine. One thing I can track when doing this with Postman, is that the first response if the post is accepted is a 302 redirect, which is expected. Is it possible the HttpReq is trying and failing to follow the redirect before sending the POST? The reason I ask that question is because from the DeveloperTool window on chrome, I can see the POST going to the redirect URI and not the original URI so it looks to me like the HttpReq is opening as a GET to the originating URI I wanted to POST to, redirects and then tries to POST to that next URI and fails on some unknown error.

I'm also thinking the postForm method may be causing some issues since I can't call that method without losing all the headers I set from the originating caller since it overwrites the headers with it's own content-type. I can set that manually but then I don't have access to the encodeForm method since it's private and then lose the ability to post form data.

Reviewing the timing with wireshark, I see that the postForm method is actually performing as expected but the HttpReq object is then following that 302 location defined in the response which is returning the content as a GET as expected but within the context of the HttpReq.postForm in fanton, I get the error I mentioned.

Is it possible the dom::HttpReq is trying and failing to follow the redirect before sending the POST?

Possible, but from Javascript it's the browser that handles redirects, so there's not much Fantom can do about it.

Note that 302 is a poor and ambiguous response status, if you have any sway I'd return a 303 or 307 instead.

since dom::HttpReq.postForm() overwrites the header with it's own content-type

application/x-www-form-urlencoded is the correct and, as far I know, the only supported content type that should be used with form encoded data. General body content, however, maybe encoded any way you wish.

I don't have access to the encodeForm method since it's private

You could use reflection:

HttpReq#.method("encodeForm").callOn(HttpReq(), params)

Reviewing the timing with wireshark

Wow, I've never need to go to that level - I've always found F12 Developer Tools and / or FireBug to give me all the details I need.

Without a stacktrace it would seem likely that the error is being thrown from dom::HttpReq.makeRes(). It's pretty simple so I'd imagine your response has a dodgy header, or Fantom isn't handling the response headers properly.

Is it possible to see the raw headers that get returned?

I managed to figure out that HttpReq postForm or any type of HttpReq POST doesn't really do a simple POST, it tries to complete the whole process, i.e. following the 302 redirect and then subsequently trying to complete the GET from that instead of stopping at the POST. By redirecting my initial URI to the URI the HttpReq was going to get redirected to, it fixed the HttpReq error and returns the GET response from the 302 redirect.

Unfortunately, the POST i'm trying to do is an authentication to another server and any session that is created by that POST seems to be lost once that POST is complete. I can try to open the URI to the server after the POST is complete via a WebBrowser but the authentication is lost and/or not held onto anymore past that POST.

I can verify all of this works with Postman with chrome but getting the same session to authenticate from a simple POST and be usable when done through fantom so far as proven impossible.

HttpReq POST ... tries to complete the whole process, i.e. following the 302 redirect

As mentioned, redirects are handled by the browser (specifically the XMLHttpRequest class) and there's nothing Fantom can do about it.

See Prevent redirection of Xmlhttprequest for details. Note that PostMan is a Chrome Plugin and has access to lower level APIs.

getting the session to authenticate ... through fantom has proven impossible

Fantom's dom::HttpReq is a really simple wrapper around the XMLHttpRequest class.

As you're already injecting Javascript into the page, why not try using vanilla Javascript and XMLHttpRequest to make the authentication call? If you get that working, it should be simple enough to back track to Fantom and see where (or if) it gets it wrong.

I may be trying to accomplish an impossible task while doing further investigation. I have a web service loaded in a WebBrowser (iframe) but that means it will not share any session or cookie information with the containing document. Since that's the only place I can control the authentication and session/cookie creation, i'm not able to authenticate it the actual iframe session.

You're correct in that Cookies (and hence authentication sessions) are not shared across domains.

Though you may be able to circumvent the issue with Cross-Origin Resource Sharing (CORS). It requires some extra headers to sent / received to / from the servers. See Cross-Domain Cookies on StackOverflow.

It looks like the auth method used by the POST already returns those headers in it's response. What I am not seeing though is any way to modify any of the headers when it comes to working with a WebBrowser which seems necessary to some degree to accomplish this.

Not sure if this helps your use-case or not, but Using CORS on HTML5 Rocks says:

withCredentials

Standard CORS requests do not send or set any cookies by default. In order to include cookies as part of the request, you need to set the XMLHttpRequest’s .withCredentials property to true:

xhr.withCredentials = true;

In order for this to work, the server must also enable credentials by setting the Access-Control-Allow-Credentials response header to true. See the server section for details.

Access-Control-Allow-Credentials: true

The .withCredentials property will include any cookies from the remote domain in the request, and it will also set any cookies from the remote domain. Note that these cookies still honor same-origin policies, so your JavaScript code can’t access the cookies from document.cookie or the response headers. They can only be controlled by the remote domain.

I'm not really sure what is being referred to as the server and what's remote domain in that description.

The question boils down to just a few things I think:

1. The server already responds with the Access-Control headers set accordingly when the POST command is sent.
2. How can I access the xhr properties and methods since fantom uses a wrapper for this but doesn't seem to allow access to those methods?
3. Does this mean the POST call needs to set this property or the WebBrowser after authentication has taken place?
4. Does the WebBrowser need extra headers set or does the POST with the additional withCredentials set to true make this work? Since the WebBrowser doesn't really give me any control over how it loads anything, if I have to add headers to it's load methods, this won't work no matter what the auth headers are set to.

There is a lot of questions/comments here. But it sounds like the real issues aren't really specific to Fantom (if I understand correctly). I think what might be good is to get your app working in vanilla JavaScript. Then if there is something specific you are doing not exposed by Fantom we can add to the dom APIs.

Specially about dom::HttpReq - it does allow you full control of the headers. Its nothing more than a Fantom wrapper around XMLHttpRequest.

I'll try to stick to just a pure fantom question concerning this issue.

It was mentioned to set the property in the request like this:

xhr.withCredentials = true;

xhr being XMLHttpRequest and dom:HttpReq being a wrapper for that, I would assume I could do something like:

HttpReq.withCredentials = true

but this is not the case. So the question becomes, how do I gain access to the XMLHttpRequest methods not exposed explicitly by the dom:HttpReq?

XMLHttpRequest.withCredentials is not currently exposed by dom::HttpReq so I'm not sure you can without writing custom Javascript.

For now, I would suggest monkey patching the JS HttpReq object when you write out your Javascript.

Pushed a fix if you're able to pull from tip

Cloned the repo just not seeing a way to build the all the fantom pods from it. Is this an option so I can point my IDE to the new fantom build or do I have to build each pod individually?

Hopefully you would only need to build the dom pod, depending on what else has changed in tip.

Here's how I just built it:

C:\> cd C:\Temp\fan-1.0\src\dom

C:\Temp\fan-1.0\src\dom> fan build.fan
compile [dom]
  Compile [dom]
    FindSourceFiles [19 files]
    CompileJs
    WritePod [file:/C:/Apps/fantom-1.0.69/lib/fan/dom.pod]
BUILD SUCCESS [1170ms]!

I rebuilt the updated pod and set the withCredentials to true but no luck. Still not able to share cookies or session data between the browser session and WebBrowser (iframe).

You'll probably need todo some setup on the server side as well - cross-site stuff always seems to have a lot of moving parts.

But does it need to be an iframe? If you could throw up a normal HTML page with GET/POST might save you some trouble (pretty much all of my fwt/domkit based stuff has always used vanilla HTML for auth - just ends up be being easier).

The server is another web service which I have no control over so I can't do anything on that end as well as any other web service I would like to integrate into a fantom application. The iframe is necessary (unless there's something else to use) since it needs to open up the URL to the other web service to allow all the interactions built into it to exist but the cross origin makes it pretty impossible to do at this time when it comes to sharing any session data.

cross origin makes it pretty impossible to do at this time when it comes to sharing any session data

I am still fuzzy about the issue. If you can make this work in plain JavaScript, then it should work in Fantom too and if not that is a bug/enhancement we should be able to easily fix. If that is the case, then need to know that.

If you can't make it work in JS, then you might just be trying to do something not allowed.

The problem with that is i'm not a JS developer and once I hand it off to my JS developers, if they make it work, that's where the development will end. We won't double back to fantom to re-develop something a second time we already got working somewhere else.

For a clearer example within a fantom context, suppose I started a Wisp server and built out a web page that when loaded authenticated to a Skyspark instance. Now that browser contains session information but I don't want skyspark to be my web page, i'd just open a new tab for that, I want Skyspark embedded in my web page. An iframe would not have that session information but would need it to be loaded there without requiring a second login.