Evergreen ILS

Open Source Library Software

by on

Fun with Javascript and JSON

I haven’t broken radio silence in a while so I thought I would have a general discussion of some of the fun code I’ve been exploring. OPAC development has been interesting as of late in part because of its evolution into an almost fully dynamic client side application. Our extensive use of JSON and XMLHTTPRequest gives us this ability to make the site more dynamic and is also a reason in itself why the OPAC has been a joy to code.

Traditional web pages typically use forms for submitting data to a server. Javascript often plays the role of form validator. JS is also good at making a page appear more interesting and dynamic (e.g. DHTML – moving/populating HTML elements dynamically). With the emergence of XMLHTTPRequest (or AJAX), it is no longer necessary to submit a form and download another web page to see the results of the submission. The form data can be analyzed by the javascript and submitted to the server for processing. The server can then respond with some data at which point the javascript can dynamically add the received data to the page (e.g. shove data into a table, etc.). With this simple addition, Javascript has evolved from “helper” code to full blown application code, so much so that you could write a fully functional web site that only has a single web page accompanied by a collection of javascript files that perform all of the work.

We’ve taken XMLHTTPRequest one step further and added JSON to the mix. JSON (mentioned in this blog previously) is a “lightweight data-interchange format” (see json.org). It gives us a way to turn program objects into strings, or serialize them. JSON is great for us because it’s a lot lighter than XML. It allows us to encode our data with practically no extraneous data clogging the lines. As a quick example, an array converted to JSON would look something like this: [ 1, 2, 3]. Whereas in XML it might appear like so: <array><item>1</item><item>2</item><item>3</item><array>. Even with a small dataset you see an immediate difference in the number of characters required to encode the object.

JSON parsers exist in many languages, and we’ve developed our own parsers in C, Perl, and Javascript. Why did we write our own, you ask? You guessed it… we took JSON one step further as well. We added what we call class hints to the JSON format. This allows us to parse a JSON string and determine what type of object we’re looking at based on a hint encoded as a comment within the object. So, for example, the Javascript JSON parser might receive a JSON string from the server that is encoded with a class hint of “user”. The JSON parser will then be able to turn the JSON string into a full blown Javascript “user” object that knows how to access and update the data it contains.

Wrapping it all up with some code

Consider a situation where you have a user object and wish to update that user’s email address. A form has been drawn on the screen and the user is entering the new information…

The onclick for the submit button might look something like the following:

button.onclick = function() {
 
  var new_email = /* retrieve the data from the form and validate */
  userObject.email(new_email);

  /* this is our XMLHTTPRequest wrapper class which turns any objects into JSON
    strings and turns a request into an XMLHTTPRequest with a URL that 
    our Apache knows how to process */
  var request = new RemoteRequest( serviceName, methodName, userObject );
 
  /* requests can be asynchronous, in which case it's necessary 
    to provide a callback for the request */
  request.setCompleteCallback( 
    function(req) {
      var updatedUser = req.getResultObject();
      alert("Email updated: " + updatedUser.email()); /* or what have you... */
    }
  );
 request.send();
}

… And that’s pretty much it. Using this same basic layout, we can retrieve and update any objects that the server allows and we don’t have to concern ourselves with how the objects are constructed. Having such a simple, uniform data interchange framework allows us to spend more time adding content to the web site and less time crafting URLS and reloading entire pages to update a single field like an email address.

For more on XMLHTTPRequest, there’s a good intro here.

Reader Interactions

Comments

  2. NOTE: This is in response to both the comment above, and the blog entry to which it points.

    While I understand your concerns, for I shared them initially, I’ll point out a couple things for you to consider:

    * How do you, and we, define “usablitiy”?
    Open-ILS defines it as “the property of a project that allows work to get done in an efficient manner.” Choosing an optimized wire protocol is part of that because the libraries in our consortium are on rather slow network connections. Our JSON base protocol is more than 50% smaller, sometimes as much as 90%, and the deserializer is significantly faster than an XML parser.

    * Use the right tool for the job
    As a supporting argument to the point above, consider a pure Java application: one would write a JNI based app and then add a SOAP interface to it. Since the main client applications of the Open-ILS backend will be a XUL-based staff client and the DHTML based OPAC, JSON is the tool for the job.

    That being said, here’s a point which should allay your concerns. Since we’ve already written the XML version of everything we’re doing in JSON, as far as data transfer is concerned, we are planning to add support back in to OpenSRF to allow you to choose which format you’d like the result of a method request to be returned.

    One last thing about the “self describing” nature of XML: it’s only self describing to humans, not to machines. With the possible exception of the “semantic web” uses of RDF, an application that uses XML as a messaging protocol can only understand and use tags that it has been told about using an XSD or similar authority document. This is exactly the same position we are in when using JSON, with the exception that you, as a human, need to look up the meaning of a field within a hinted structure. I’ll definitely accept that one extra step of “indirect documentation” as the price for the measured gain in performance we’ve received.

    Of course, the reason I’m posting this response isn’t to fight over XML, or even, really, to defend our design decisions. These decisions are ours to live with, right or wrong. The real reason I am responding is that it’s in the best interest of our project to reevaluate our design any time the opportunity arises. I’m very glad you started the discussion on JSON, since it forced me to double check our design and make sure we’re moving in the right direction. We all thank you, sincerly, for the constructive sharp-stick-in-eye ;), and I, personnally, hope you’ll continue to take enough interest in our project to question our design, and to contribute in other ways down the road.

  3. You have made a strong denfense of your design decision to use JSON in place of XML in client-server communications. For example, libraries in North America are not well known for having high network capacity, so reducing the load on the wire is not a bad idea. My last remaining concern is whether the use of JSON instead of XML will be a barrier to interoperability and extensibility. What I mean is, there are thousands of tools and developers that understand XML, but far fewer that understand JSON. Now, it is not that JSON is hard to understand, parse, or otherwise work with, but I would love to see – in my lifetime – libraries that are compliant with standards in use outside the library world. I would not want JSON to be the next MARC or Z39.50. I agree 100% with Kenton Good in this respect. However, I note that you say that you have built XML flavors of all of your JSON interfaces – will these be maintained throughout your development cycle.

    I am a great champion of your efforts. In fact, I will state publically that I predict the success of this project will drive much needed innovation among the commercial ILS vendors. Increased choice and increased innovation will only be good for our libraries.

    As for helping out – I would love to do a security review of your architecture and code base, if you are interested.

    Cross posted here.

  4. Apart from JSON being faster, cleaner, and more compatible with programming language syntax JSON is more semantically descriptive since both humans and software can differentiate between relationships which are compositional or aggregate while XML cannot; which is clearly a valuable distinction.

    I have been using JSON for a while now in combination with the ECMAScript DOM in order to completely avoid PHP/HTML/XML/CSS/JS language barriers. All of my code is in JS/ASP (which is JS), and it’s much much easier to work with.

    I keep *one* language, where I can keep Objects in the Application Object on the server and then just haul it out and dump it to JSON and then the client gets the Object as it was without any fuss. This is particularly useful when the Application Object code is constantly changing. My use for this technology was an online massively multiplayer strategy game. 😀

  5. I’ve been arguing with the inventor of JSON himself about an optional Javascript identifier at the front of a JSON construct, which could go a long way toward a more “self-describing” JSON. I’ve blogged on it, at first proposing it be called JSON++, but now I have another name in mind, and have completed my first stab at a parser, for Java.

    In the meantime google on “JSON++” and you can find my blog post and the relevant usenet article. Feel free to contact me via email and/or don’t be surprised if you hear from me when I unveil version 0.1.

  6. UBF (properly, UBF(A)) is a binary-data-exchange format broadly similar to JSON that was developed in the Erlang world. It does a couple of things well that I think JSON could borrow: firstly, it supports compact binary data blocks; and secondly, it has the notion of a tag that’s an annotation on the data without being part of the data – essentially an escape into metadata. See http://www.sics.se/~joe/ubf/site/home.html for details about UBF.