#2394 JsonOutStream and Newlines

SlimerDude Tue 3 Feb 2015

I've been playing with bulk queries in elastic search and for streaming purposes they mandate where and when \n new lines are used:

{ action: { metadata }}\n
{ request body        }\n
{ action: { metadata }}\n
{ request body        }\n

Only JsonOutStream hardcodes a \n after every Map / Json Obj key value pair:

...
line 69: else writeChar(JsonToken.comma).writeChar('\n')
...
line 82: if (notFirst) writeChar(JsonToken.comma).writeChar('\n')
...

I was just thinking it'd be nice to have some sort of pretty print flag so we could turn the behaviour on and off.

brian Tue 3 Feb 2015

I was just thinking it'd be nice to have some sort of pretty print flag so we could turn the behaviour on and off.

Adding the newline while you are streaming is pretty printing. Without the developer being explicit, that guarantees the most readable output. How else could we do it?

SlimerDude Tue 3 Feb 2015

the developer being explicit

Exactly, so I was thinking of something like:

class JsonOutStream : OutStream {

  new make(OutStream out, Bool prettyPrint := true) : super(out) {
    this.prettyPrint = prettyPrint
  }

  ...
}

That way it preserves existing functionality but you are able to create a non-pretty (ugly?) JsonOutStream if needed.

SlimerDude Tue 3 Feb 2015

Or just a flag, it doesn't need to be a ctor parameter.

andy Tue 3 Feb 2015

The default behavior isn't terribly useful - I end up just hand-cooking my output to work around this (which defeats the purpose of the library ;)

So I would definitely be in favor of addressing this. Not sure what that flag is called though, not sure I like prettyPrint.

SlimerDude Tue 3 Feb 2015

It'd be good to pretty print the JSON properly, complete with indentation, then the prettyPrint flag name would make more sense.

brian Tue 3 Feb 2015

That way it preserves existing functionality but you are able to create a non-pretty

So you are saying you want it to be not pretty, in which case why do you care what it looks like it? You want it to be one huge line? It is only one extra char per map/array item which in the scheme of a fairly verbose doesn't add much overhead.

andy Tue 3 Feb 2015

Its not always one giant request - simple things like maps might be small and are much more useful when treated as one-line string - it might be stored in file or property somewhere - where those newlines cause issues. Thats been my use case.

For HTTP I don't think it matters either way, the browser will always show you the formatted response. But if you want to use JSON in other contexts I think "single-line" makes a better default.

SlimerDude Wed 4 Feb 2015

So you are saying you want it to be not pretty

Exactly.

As stated in the first post, my issue / use case is with the elastic search bulk API where the body of a HTTP POST request may comprise of 1000s of JSON objects.

Elasticsearch uses the \n as a token delimiter so it knows when one JSON obj finishes and another starts. Then the HTTP request body may be parsed as a stream. It's similar to how \n is used in CSV files - it lets you know when you can start parsing the next row / obj.

At the moment I can't use JsonOutStream because it outputs a \n after every key / val pair - which is incorrectly interpreted by elasticsearch.

brian Wed 4 Feb 2015

I pushed a change to get rid of the newlines. So now it just one big line, although you can always insert your own newlines and space for pretty printing.

Login or Signup to reply.