Jackson 2.9 features

Wow. Now that Jackson 2.9.0 is finally out (*) — it took almost 1 full year to get from 2.8.0 (Features) to 2.9.0, whereas previous versions took 4–6 months — it is time time to go over treasure trove of new features included in 2.9.

As usual, the full 2.9 release notes should give 360 degree view of all changes, but I will try to pick most impactful features for a brief overview here. I will also try to blog some more about one of features (non-blocking parsing), usage scenarios (config input merging, handling), and format improvements (Avro reader/writer schema evolution) at a later point, but let’s start with an overview.

(*) in fact 2.9.1 patch version is already out.

Yet More Bigger “Minor” Release

Although 2.8 was already return to the “bigger minor” version (after 2.7), we went for “Evan Moar” mode with 2.9. Reason for this was two-fold:

1. We started with a concise but ambitious list of about dozen major “should have” features, and while working on other things too, really wanted to make sure most of these features will make it in. This tends to fill release more than focusing on date of delivery (but extend timeline obviously).
2. It is very likely that 2.9 will be the last minor version for Jackson 2.x — so any API changes, new features NEED to be included, or wait until big version change of 3.0. This had the result of forcing inclusion of some things that previously might have got moved into the “next minor version” bucket.

Compatibility

No changes since 2.8: JDK 7 is now needed to compile most components (but not annotations, streaming), but all code should be able to run on Java 6 still (with exception of Java8-specific modules).

Modules

New Modules

One new component: jackson-dataformat-ion included in Binary Dataformats repo — used for reading/writing Amazon Ion format documents.
This could be useful for, say, Kindle developers.

Significant Module Improvements

This time biggest improvements were for:

• Avro format module: full support for Schema evolution: ability to specify separate “Reader” and “Writer” schemas. Check AvroSchema method withReaderSchema() for entry point — Writer schema created first, then a new instance with Reader schema created, and this is given to ObjectReader (NOTE: I need to blog more about this)
• CSV format module: misc added fixes, features, including but not limited to CsvParser.Feature.INSERT_NULL_FOR_MISSING_COLUMNS, CsvParser.Feature.ALLOW_TRAILING_COMMA, CsvParser.Feature.FAIL_ON_MISSING_COLUMNS — also, new Exception type, CsvMappingException for better error handling
• Properties format module: better interoperability with System properties (“read” System properties); reading from / writing as java.util.Properties — especially convenient to allow, say, command-line overrides!

Many other modules were improved as well, as per Release Notes.

New Core Functionality

“No Trailing Junk”

One of long-standing features has been ability to detect content where valid JSON Object (or other value) is followed by garbage. By default Jackson only reads enough input needed to create one value of expected type, return that, and allow further reads if necessary; but not check if anything else might follow.

With 2.9 you can configure ObjectMapper or ObjectReader to briefly check that input stream contains nothing after value has been bound:

ObjectMapper mapper = new ObjectMapper()
   .enable(DeserializationFeature.FAIL_ON_TRAILING_TOKENS);
Value v = mapper.reader();
// throws MismatchedInputException if valid tokens found;
// otherwise throws tokenization exception

Note that this effectively calls JsonParser.nextToken() after binding (which would trigger decoding exception, JsonParseException, if non-json content follows); and verifies that method returns null which signals end of content: if this is not true, MismatchedInputException is thrown.

@JsonAlias

Another highly requested feature is the ability to indicate alternate names to be accepted on deserialization (but not change serialization where a single primary name is used). For example, when evolving API definitions it may make sense to rename properties (to better reflect semantics); but allow older clients to still use old property name during transition period. Or sometimes it is necessary to work around client-/server-side issues of wrong names.

Usage is similar to common @JsonProperty annotation (and similar):

public class Bean {
   @JsonAlias({ "id", "_name" })
   public String name;
}

In this case, messages like:

{ "name" : "Alice" }
{ "id" : "Bob" }
{ "_name" : "Fred" }

would all bind seamlessly into Bean while still serializing using primary property name, name.

Alias information is also exposed through Jackson’s Introspection, used for generating JSON Schema, Avro Schemas and Protobuf protoc definitions, so it can help interoperability: this is especially useful with Avro schema evolution.

Custom `@JsonInclude`

Although set of default choices for @JsonInclude (and separation of value vs content inclusion) covers many use cases, it is impossible (and before that, impractical) to try to cover everything with static pre-configured rules. So 2.9 finally introduces simple extension mechanism for using custom strategy objects for deciding what to include and what not.

public class Filtered {
  @JsonInclude(value=Include.CUSTOM,
     valueFilter=MyFilter.class)
  public String value;
}
public class MyFilter {
   @Override
   public boolean equals(Object other) {
     // return 'true' to not write (exclude), 'false' to write
     return "secret".equals(other);
   }
}

Filtering is decided based on calling basic boolean equals() method: if true is returned, value is filtered out (property and value are NOT serialized); if false returned it is included normally.

Note that for Map values (but not arrays or Collections, for now) you can alternatively specify content / contentFilter settings — these apply to contents (values of entries within Map), whereas value / valueFilter apply to the Map value itself.

Null replacement/error/skipping (on deserialization)

Yet another long-standing gap in Jackson functionality has been inability to do much about incoming (read from JSON) null values: unlike other tokens, nulls are NOT passed to JsonDeserializer but typically directly assigned as Java nulls.
This means that the only way to change handling of nulls has been to define a setter in which null value is changed to something else, or assigned is ignored. This is work that is bit tedious, and for 3rd party types also not possible to do directly, in which case one may need to resort to more complex handling (bind to one type, then convert).

Jackson 2.9 adds new properties to existing (but rarely used) @JsonSetter annotation to allow specifying one of common alternatives:

public class Config {
   @JsonSetter(nulls=Nulls.SKIP) // don't override if null
   public Coordinates coordinates = new Coordinates(1, 2);
   @JsonSetter(nulls=Nulls.AS_EMPTY) // null will clear to empty list
   public List<String> names = Collections.emptyList();
   @JsonSetter(nulls=Nulls.AS_EMPTY,
       contentNulls=Nulls.FAIL)
   public Map<String, Extra> attributes;
}

In this example, we have set a suitable default POJO for coordinates, and should incoming content have null for that property, we want to just ignore this null.
Property names just needs to exist as empty List; we could have either used SKIP as above, or force it to be set as “empty” object (relevant for structured types and Strings — deserializer is asked for what that value is via JsonDeserializer.getEmptyValue(context) method).
And finally, there is also another property, contentNulls, which is applied to values of Maps: in this case we make mapper throw JsonMappingException if null value is passed in. This could as well have been SKIP, if we wanted to quietly just ignore such values: this a design choice that some developers feel strong about (defensive vs assertive).

But wait! There’s more. In above example, we applied settings to individual properties. But there are 2 other possibilities too:

1. Specify defaults for a type (possibly overridden by per-property annotations)
2. Specify global defaults to use (unless overridden by type, property)

ObjectMapper mapper = new ObjectMapper();
// Prevent assigned of `null` for any `String` property!
mapper.configOverride(String.class)
   .setSetterInfo(JsonSetter.Value.forValueNulls(Nulls.FAIL));
// Skip all null assignments (unless overridden)
mapper.setDefaultSetterInfo(JsonSetter.Value.forValueNulls(Nulls.SKIP));

So. For all you null-haters out there: you have new tools to discriminate against those pesky null values that may try to sneak in that content.
This feature is of particular importance when using next feature, merging, covered next.

Merging

Jackson has had “shallow” form of merging available for a long time now:

Config config = ... ; // create and initialize
mapper.readerForUpdating(config).readValue(src);

which works fine but only for scalar values: structured values like Maps and POJOs would still be replaced. With 2.9 we will finally support “deep” merge, in which structured values are merged as well.

Support is added using combination of annotations and config overrides, quite similar to how null-handling was specified above. So it a property of types and properties (where should deep merge be applied, where not), and not a Feature to enable. Let’s have a look at augmented example

public class Config {
   @JsonMerge // yes please merge
   @JsonSetter(nulls=Nulls.SKIP) // don't override if null
   public Coordinates coordinates = new Coordinates(1, 2);
   @JsonMerge(OptBoolean.FALSE) // don't try merging Lists
   @JsonSetter(nulls=Nulls.AS_EMPTY)
   public List<String> names = Collections.emptyList();
}

As with null-handling, you can also specify merging defaults by type and globally:

// to allow merging for values of specific type:
mapper.configOverride(MyConfig.class).setMergeable(true);
// or by default, for types for which applicable:
mapper.setDefaultMergeable(true);

There are some nuances to merging, and ability to do so:

1. Scalar types can not be merged: if annotated (or via defaulting), attempts ignored (unless MapperFeature.IGNORE_MERGE_FOR_UNMERGEABLE is disabled)
2. Merging of Collections is simple: entries are appended; no checks for existence applied — use Sets when you want “true” merge
3. Although arrays can not be modified in place, “merge” is performed by constructing a new array instance with contents appended
4. Support is needed by JsonDeserializer so datatype modules need to be retrofitted to support it for relevant types (collections, Maps)

Avro Module: support reader/writer schema

The biggest new feature is probably the ability to support “schema evolution” (fancy word for case where Avro encoder has used one Schema [“writer schema”] but reader wants to use different one [“reader schema” ]and the two are compatible , not identical — compatible meaning it is possible to add, rename and remove properties in limited way).
Simple example should illustrate the usage:

public class XY {
  public int x, y;
}
public class XYZ {
  public int x, y, z;
}
AvroMapper mapper = new AvroMapper();
final AvroSchema xySchema = mapper.schemaFor(XY.class);
// write an XY:
byte[] avro = mapper.writer(xySchema)
  .writeValueAsbytes(new XY(1, 2));
// but "translate" into XYZ via reader/writer schema
final AvroSchema trSchema = xySchema.withReaderSchema(
  mapper.schemaFor(XYZ.class));
XYZ xyz = mapper.readerFor(XYZ.class)
  .with(trSchema)
  .readValue(avro);

It is worth noting that Jackson’s data-binding typically supports such changes (addition and removal of properties; rename via new @JsonAlias) automatically. But Avro encoder/decoder absolutely require use of Reader/Writer schema concept: attempts to “just read” something encoded using Schema 1 with different (even if compatible!) Schema 2 will result either in failure (best case) or data corruption (worst case) — Avro format does not have enough metadata to distinguish these cases.
Schemas used as Reader/Writer schema can come from external files (read with AvroMapper.schemaFrom()), generated from POJOs (AvroMapper.schemaFor()) and even built using Apache’s standard Avro lib if necessary.

Besides support for reader/writer schemas, there is plenty more included in 2.9 Avro module:

• Support for many of annotations from Apache Avro lib (@Stringable, @AvroAlias, @AvroEncode.
• Improved read performance with native decoder (although Apache Avro lib BinaryDecoder is still be available as an alternative as well)
• Ability to write “File Avro” with AvroGenerator.Feature.AVRO_FILE_OUTPUT — this will add header into output, needed by some processing tools and storage systems (note, however, that Jackson does not currently read such format — room for future improvement!)

Non-Blocking Reading (aka Async Parsing)

And last — but not least! — possibly the longest-lasting feature request was finally implemented: ability to parse JSON using non-blocking input source.
Inspiration in this case came from Aalto XML parser which has been able to do this for XML for years now.

The idea is that instead of using a blocking input source like InputStream or Reader input is “fed” (pushed) when it comes available. Minimally intrusive approach (as shown by Aalto which extends Stax with minimal changes) includes just two things:

• Possibility for JsonParser to return JsonToken.NOT_AVAILABLE when it does not have enough input to decode the next token
• Ability to feed more input after NOT_AVAILABLE has been returned

I will have to write a bigger example of usage at a later point, but here is the process:

JsonFactory f = new JsonFactory();
// will create async feeder based on byte[]:
JsonParser asyncP = f.createNonBlockingByteArrayParser();
while (true) {
  JsonToken t;
  while ((t = asyncP.nextToken()) == JsonToken.NOT_AVAILABLE) {
    // need to feed more
    ByteArrayFeeder feeder =
      (ByteArrayFeeder) asyncP.getNonBlockingInputFeeder();
    feeder.feedInput(buffer, offset, length);
  }
  // token available, process:
  // ... 
}

So basically feeding of input occurs when previous content has been decoded and handled. Future versions may (and should) add support for other feeders such as ByteBuffer.

But wait! There is more! 2.9 also includes Async/Non-blocking decoder for Smile format (“binary JSON”); and we hope to provide one for CBOR relatively soon as well. It should in fact be possible to add support for almost all (if not all?) formats eventually, if there is interest.

And perhaps best of all, upcoming Spring 5 release will include support for non-blocking reading, using Jackson, as well.

Misc Improvements

• @JsonValue for fields — formerly it was only allowed on methods (getters)
• @JsonView on classes: if set, annotated View will be used as the default view for properties that are not otherwise annotated — this can reduce need for common View annotations a lot.
• @JsonStreamContext.pathAsPointer(): convenient way to get JsonPointer from context accessed by JsonParser.getParsingContext() and JsonGenerator.getOutputContext(). Useful for diagnostics, error messages, or perhaps even for keeping references to content as JsonPointer expressions can be used both for traversing JsonNodes and filtering JsonParser content.
• Strict vs Lenient parsing of java.util.Dates (and Calendars): using @JsonFormat(lenient = OptBoolean.FALSE) you can force strict handling of Dates (default is lenient handling) — this prevents use of Dates like “35th of February”. NOTE: aside from per-property annotations, you can also use “Config Override” system for type defaults.