This chapter is about bringing the external data to HTML Web pages. In the previous chapters we’ve been using only hard-coded data - our goal was to see how to layout the Web page and how to change the layout in case some events have happened, e.g. the user clicked on the menu button. Now we’ll make sure that our single-page application Save The Child can request the data from the external sources and send them the data too. This comes down to two questions:
-
How can an HTML Web page exchange the data with Web servers?
-
What format to use for presenting the application data?
While there could be different answers to these questions, we’ll be using AJAX techniques as an answer to the first question and JSON data format as an answer to the second one. We’ll start this chapter with explaining why AJAX and JSON are appropriate choice for the Save The Child Web application and many others.
In the early days of the Internet, every new page, whether on the same or separate web site, required a new request and response to the web server. This, in turn, would re-render the entire contents of the new page. If a user points her Web browser to one URL and then changes it to another, the new request will be sent to the new URL, and the new page will arrive and will be rendered by the browser. The URL may have been changed not because the user decided to go to visit a different Web site, but simply because the user selected a menu item that resulted in bringing a new Web page from the same domain. This was pretty much the only way to design Web sites in the 90th.
Back in 1999, Microsoft decided to create a Web version of Outlook - their popular eMail application. Their goal was to be able to modify the Input folder as the new emails arrive, but without refreshing the entire content of the Web page. They created an ActiveX control called XMLHTTP that lived inside Internet Explorer 5 and could make requests to the remote servers receiving data without the need to refresh the entire Web page. Such a Web Outlook client would would make periodic requests to the mail server, and if the new mail arrived the application would insert the new row on the top of the Inbox by direct change of the DOM object from JavaScript.
In early 2000th, other Web browsers implemented their own versions of XMLHTTPRequest. Its Working Draft 6 is published by W3C. Google created their famous email client GMail and Map Web applications. In 2005 Jesse James Garrett wrote an article titled "AJAX: A New Approach to Web Applications". The Web developer community liked the term AJAX, which stand for Asynchronous JavaScript and XML, and this gave a birth to the new breed of Web applications that could update the content of just the portion of a Web page without re-retrieving the entire page. Interestingly enough, the last letter in the AJAX acronym stands for XML, while realistically presenting the data in the XML form is not required, and currently is seldom used as a data format in the client-server exchanges. JSON is used a lot more often to represent the data, but apparently AJAJ didn’t sound as good as AJAX.
Visit the Google Finance or Yahoo! Finance Web pages when the stock market is open, and you’ll see how the price quotes or other financial indicators are changing while the most of the content remains the same. This gives an illusion of the server pushing the data to your Web client. But most likely, it not a data push but rather periodic polling of the server’s data using AJAX. In the modern Web applications we expect to see more of the real server side data push using HTML5 WebSockets API, which is described in details in Chapter 8 of this book.
JSON stands for JavaScript Object Notation. It’s a more compact than XML way to represent the data. Besides, all modern Web Browsers understand and can parse JSON data. After learning the JavaScript object literals in Chapter 1, you’ll see that the presenting the data in JSON notation is almost the same as writing JavaScript object literals.
Anatomy of a Web application depicts a high level view of a typical Web application. All of the code samples from Chapter 1 (and in Appendixes) were where written in HTML, JavaScript and CSS. In this chapter will add to the mix the XMLHttpRequest object that will send and receive the JSON content wrapped into HTTPRequest and HTTPResponse objects.
When a Web page is loaded the user doesn’t know (and doesn’t have to know) that the page content was brought from several servers that can be located thousands miles apart. More often than not, when the user enters the URL requesting an HTML document, the server side code can engage several servers to bring all the data requested by the user. Some of the data are retrieved by making calls to one or more Web Services. The legacy Web services were build using SOAP + XML, but majority of today’s Web services are build using lighter RESTful architecture, and JSON has become a de facto standard data exchange format of the REST Web services.
Imagine a single-page application that needs some data to refreshed in real time. For example, our Save The Child application includes an online auction where people can bid and purchase some goods as a part of a charity event. If John from New York placed a bid on certain auction item, and some time later Mary from Chicago placed the higher bid on the same item, we want to make sure that John knows about it immediately, in real time. This means that the server-side software that received Mary’s bid has to push this data to all users who expressed their interest in the same item.
But the server has to send and the browser has to modify only the new price while the rest of the content of the Web page should remain the same. You can implement behavior using AJAX. But first, the bad news: you can’t implement the real-time server side push with AJAX. You can only emulate this behavior by using polling techniques, when the XMLHttpRequest object sits inside your JavaScript code and periodically sends HTTP requests to the server asking if there were any changes in bids since the last request.
If, for instance, the last request was made at 10:20:00AM, the new bid was placed at 10:20:02AM, and the application makes a new request (and updates the browser’s window) at 10:20:25AM, this means that the user will be notified about the price change with a three second delay. AJAX is still request-response based way of getting the server’s data, and strictly speaking, doesn’t offer a real real-time updates. Some people use the term near real time notifications.
Another bad news is that AJAX uses HTTP protocol for the data communication, which means that a substantial overhead in the form of HTTPResponse header will be added to the new price, and it can be as large as several hundred bytes. This is still better than sending the entire page to the Web browser, but HTTP adds a hefty overhead.
|
Note
|
We’ll implement such an auction in Chapter 8 using a lot more efficient protocol called WebSockets, which supports a real-time data push and adds only several extra bytes to the data load. |
Let’s try to implement AJAX techniques by implementing a data retrieval. The process of making an AJAX request is well defined and consists of the following steps:
-
Create an instance of XMLHttpRequest object.
-
Initialize the request to your data source by invoking the method
open(). -
Assign the a handler to the
onreadystatechangeattribute to process server’s response. -
Make a non-blocking request to the data source by calling
send(). -
In your handler function process the response when it arrives from the server. This is were asynchronous came from - the handler can be invoked at any time whenever the server prepares the response.
-
Modify the DOM elements based on the received data, if need be.
|
Note
|
Most likely you are going to be using one of the popular JavaScript frameworks, which will spare you from knowing all these details, but knowing how it works under the hood can be beneficial. |
In most of the books on AJAX you’ll see browser-specific ways of instantiating the XMLHttpRequest object (a.k.a. XHR). Most likely you’ll be developing your application using some JavaScript library or a framework and all browser specifics in instantiating of the XMLHttpRequest will be hidden from you. Chapters 3 and 4 include such examples, but let’s stick to the standard JavaScript way implemented by all modern browsers:
var xhr = new XMLHttpRequest();
The next step is to initialize a request by invoking the method open(). You need to provide the HTTP method (GET, POST et al.), the URL of the data source. Optionally, you can provide three more arguments: a Boolean variable indicating if you want this request to be processed asynchronously (which is the default), and the user id and password if the authentication is required. Keep in mind, that the following method does not request the data yet.
xhr.open('GET', dataUrl);
|
Tip
|
Always use HTTPS protocol if you need to send the user id and password. Using secure HTTP should be your preferred protocol in general (read more in Chapter 9). |
XHR has an attribute readyState, and as soon as it changes the callback function assigned to the onreadystatechange will be invoked. This callback should contain your application specific code to analyze the response and process it accordingly. Assigning such a callback is pretty simple:
xhr.onreadystatechange = function(){…}
Inside such a callback function you’ll be analyzing the value of the XHR’s attribute readyState, which can have one of the following values:
| Value | State | Description |
|---|---|---|
0 |
UNSENT |
the XHR has been constructed |
1 |
OPENED |
open() was successfully invoked |
2 |
HEADERS_RECEIVED |
All HTTP headers has been received |
3 |
LOADING |
The response body is being received |
4 |
DONE |
the data transfer has been completed |
Finally, send the AJAX request for data. The method send() can be called with or without parameters depending on if you need to send the data to the server or not. In its simplest fore the method send() can be invoked as follows:
xhr.send();
The complete cycle of the readyState transitions is depicted in Transitions of the readyState attribute
Let’s spend a bit more time discussing the completion of the this cycle when server’s response is received and the XHR’s readyState is equal to 4. This means that we’ve got something back, which can be either the data we’ve expected or the error message. We need to handle both scenarios in the function assigned to the onreadystatechange attribute. This is a common way to do it in JavaScript without using frameworks:
xhr.onreadystatechange = function(){
if (xhr.readyState == 4) {
if((xhr.status >=200 && xhr.status <300) || xhr.status===304) {
// We got the data. Get the value from one of the response attributes
// e.g. xhr.responseText and process the data accordingly.
} else {
// We got an error. Process the error code and
// display the content of the statusText attribute.
}
}
};One note about the third line of the code above. Here we’re checking the HTTP status code received from server. W3C splits the HTTP codes into groups. The codes numbered as 1xx are informational, 2xx are successful codes, 3xx are about redirections, 4xx represent bad requests (like infamous 404 for Not Found), and 5xx for server errors. That’s why the above code fragment checks for all 2xx codes and 304 - the data was not modified and taken from cache.
|
Note
|
If your application needs to post the data to the server, you need to open the connection to the server with the var data="This is some data";
xhr.open('POST', dataUrl, true);
xhr.setRequestHeader('Content-type', 'application/x-www-form-urlencoded');
...
xhr.send(data); |
|
Tip
|
XMLHttpRequest Level 2 adds some new functionality like FormData object, timeouts, ArrayBuffers and more. It’s supported by most of the Web browsers. |
AJAX techniques have their pros and cons. You saw how easy it was to create a Web page that didn’t have to refresh itself, but provided the users with the means of communicating with the server. This certainly improves the user experience. The fact that AJAX allows you to lower the amount of data that goes over the wire is important too. Another important advantage of AJAX that it works in a standard HTML/JavaScript environment and is supported by all Web browsers. The JavaScript frameworks hides all the differences in instantiating XMLHttpRequest and simplify making HTTP requests and processing responses. Since the entire page is not reloaded, you can create "fat clients" that keep certain data preloaded once and reused in your JavaScript in different use cases. With AJAX you can lazy load some content as needed rather than loading everything at once. Taken for granted auto-completion feature would not be possible in HTML/JavaScript application without the AJAX.
On the bad side, with AJAX the user loses the functionality of the browser’s Back button, which reloads the previous Web page while the user could expect to see the previous state of the same page.
Since the AJAX brings most of the content dynamically, the search engines might not rank your Web pages as high as it would do if the content was statically embedded in the HTML. If discoverability of your Web application is important, some extra steps should be taken to make it more SEO-friendly, e.g. using an SEO Server.
Increasing the amount of AJAX interactions means that your application will have to send more of the JavaScript code to the Web browser, which increases the complexity of programming and decreases the scalability of your application.
|
Tip
|
Using HTML5 History API (see Chapter 1) will help you in teaching the old dog (the browser’s Back button) new tricks. |
AJAX applications are subject to the same origin policy (the same protocol, host name, and port) allowing XMLHttpRequest make HTTP requests only to the domains where the Web application was loaded from. It’s a security measure to limit the ability of JavaScript code to interact with resources that arrive to the Web browser from a different Web server.
|
Tip
|
W3C has published a working draft of Cross-Origin Resource Sharing (CORS) - a mechanism to enable client-side cross-origin requests. |
To see the first example where we are using AJAX in our Save The Child application run the project-01-donation-ajax-html, where we’ve removed the hard-coded data about countries and states from HTML and saved them in two separate files: data/us-states.html and data/countries.html. In this project the file index.html has two empty comboboxes (<select> elements):
<select name="state" id="state">
<option value="" selected="selected"> - State - </option>
<!-- AJAX will load the rest of content -->
</select>
<select name="country" id="counriesList">
<option value="" selected="selected"> - Country - </option>
<!-- AJAX will load the rest of content -->
</select>The resulting Save The Child page will look the same as the last sample from the previous chapter, but the Countries and States dropdowns are now populated be the data located in these files(later in this chapter in the section on JSON we’ll replace this HTML file with its JSON version). These are the first three lines (out of 241) from the file countries.html:
<option value="United States">United States</option>
<option value="United Kingdom">United Kingdom</option>
<option value="Afghanistan">Afghanistan</option>The JavaScript code that reads countries and states from file (text and HTML markup) and populates the dropdowns comes next. The content of these files is assigned to the innerHTML attribute of the given HTML <select> element.
function loadData(dataUrl, target) {
var xhr = new XMLHttpRequest();
xhr.open('GET', dataUrl, true);
xhr.onreadystatechange = function() {
if (xhr.readyState == 4) {
if((xhr.status >=200 && xhr.status <300) ||
xhr.status===304){
target.innerHTML += xhr.responseText;
} else {
console.log(xhr.statusText);
}
}
}
xhr.send();
}
// Load the countries and states using XHR
loadData('data/us-states.html', statesList);
loadData('data/countries.html', counriesList);|
Note
|
The above code has an issue, which may not be so obvious, but can irritate users. The problem is that it doesn’t handle errors. Yes, we print the error message on the developer’s console, but the end user will never see them. If for some reason the data about countries or states won’t arrive, the dropdowns will be empty, the donation form won’t be valid and the users will become angry that they can’t make a donation without knowing why. Proper error handling and reports are very important for any application so never ignore it. You should display a user-friendly error messages on the Web page. For example the above else statement can display the received message in the page footer
|
else {
console.log(xhr.statusText);
// Show the error message on the Web page
footerContainer.innerHTML += '<p class="error">Error getting ' +
target.name + ": "+ xhr.statusText + ",code: "+
xhr.status + "</p>";
}This code uses the CSS selector error that will show the error message on the red background. you can find it in the file styles.css in the project-02-donation-error-ajax-html. It looks like this:
footer p.error {
background:#d53630;
text-align:left;
padding: 0.9em;
color: #fff;
}The following code fragment shows how to add the received data to a certain area on the Web page.
This code creates an HTML paragraph <p> with the text returned by the server and then adds this paragraph to the <div> with the ID main:
if (xhr.readyState == 4) {
// All status codes between 200 and 300 mean success
// and 304 means Not Modified
if((xhr.status >=200 && xhr.status <300) || xhr.status===304){
var p = document.createElement("p");
p.appendChild(document.createTextNode(myRequest.responseText));
document.getElementById("main").appendChild(p);
}
}In any client-server application one of the important decisions to be made is about the format of the data that go over the network. We are talking about the application-specific data. Someone has to decide how to represent the data about an Auction Item, Customer, Donation et al. The easiest way to represent text data is Comma Separated Format (CSV), but it’s not easily readable by humans, hard to validate, and recreation of JavaScript objects from CSV feed would require additional information about the headers of the data.
Sending the data in XML form addresses the readability and validation issues, but it’s very verbose. Every data element has to be surrounded by an opening and closing tag describing the data. Converting the XML data to/from JavaScript object requires special parsers, and you’d need to use one of the JavaScript libraries for cross-browser compatibility.
Douglas Crockford popularized new data format, which is known as JSON. In today’s Web JSON became the most popular data format. It’s not as verbose as XML, and JSON’s notation is almost the same as JavaScript object literals. It’s easily readable by humans, and every ECMAScript 5 compliant browser includes a native JSON object: window.JSON. Even though the JSON formatted data look like JavaScript object literals, JSON is language independent. Here’s an example of the data in the JSON format:
{
"fname":"Alex",
"lname":"Smith",
"age":30,
"address": {
"street":"123 Main St.",
"city": "New York"}
}Anyone who knows JavaScript understands that this is an object that represents a person, which has a nested object that represents an address. Note the difference with JavaScript literals: the names of the properties are always strings, and every string must be taken into quotes. Representing the same object in XML would need a lot more characters (e.g. <fname>Alex</fname> etc).
There are some other important differences between JSON and XML. The structure of the XML document can be defined using DTD or XML Schema, which simplifies the data validation, but requires additional programming and schema maintenance. On the other hand, JSON data have data types, for example the age attribute in the above example is not only a Number, but will be further evaluated by the JavaScript engine and will be stored as an integer. JSON also supports arrays while XML doesn’t.
For parsing JSON in JavaScript you use the method JSON.parse(), which takes a string and returns JavaScript object, for example:
var customer=JSON.parse('{"fname":"Alex","lname":"Smith"}');
console.log(“Your name is ” + customer.fname + “ “ + customer.lname);For a reverse operation - turning an object into JSON string - do JSON.stringify(custormer). The older browsers didn’t have the JSON object, and there is an alternative way of parsing JSON is with the help of the script json2.js, which creates the JSON property on the global object. The json2.js is freely available on Github. In Chapter 3 you’ve learned about feature detection with Modernizr, and you can automate the loading of this script if needed.
Modernizr.load({
test: window.JSON,
nope: 'json2.js',
complete: function () {
var customer = JSON.parse('{"fname":"Alex","lname":"Smith"}');
}
});Usually, JSON-related articles and blogs are quick to remind you about the evil nature of the JavaScript function eval() that can take an arbitrary JavaScript code and execute it. The JSON.parse() is pictured as a protection against the malicious JavaScript that can be injected into your application’s code and then executed by eval() by the Web browser. The main argument is that JSON.parse() will not be processing the incoming code unless it contains valid JSON data.
Protecting your application code from being infected by means of eval() can be done outside of your application code. Replacing HTTP with secure HTTPS protocol helps a lot in this regard. Some Web applications eliminate the possibility of cross-origin scripting by routing all requests to third-party data sources via proxying such requests through your trusted servers. But proxying all requests through your server may present scalability issues - imagine if thousands of concurrent users will be routed through your server - so do some serious load testing before making this architectural decision.
|
Tip
|
There are several JSON tools useful for developers. To make sure that your JSON data is valid and properly formatted use JSONLint. If you paste an ugly one-line JSON data JSONLint will reformat it into a readable form. There is also an add-on JSONView, available both for Firefox and for Chrome browsers. With JSONView the JSON objects are displayed in a pretty formatted collapsible format. If there are errors in the JSON document they will be reported. At the time of this writing Chrome’s version of JSONView does a better job in reporting errors. |
Earlier in this chapter you’ve seen an example of populating states and countries in the donate form from HTML files. Now you’ll see how to retrieve the JSON data by making an AJAX call. Open in the Web browser the project-04-2-donation-ajax-json - it reads the countries and states from the files countries.json and us_states.json respectively. The beginning of the file countries.json is shown below:
{
"countrieslist": [
{
"name": "Afghanistan",
"code": "AF"
}, {
"name": "Åland Islands",
"code": "AX"
}, {
"name": "Albania",
"code": "AL"
},The JavaScript code that populates the countries and states comboboxes comes next. Note the difference in creating the <option> tags from JSON vs. HTML. In case of HTML, the received data were added to the <select> element as is: target.innerHTML += xhr.responseText; In JSON files the data were not wrapped in to the <option> tags, hence it’s done programmatically.
function loadData(dataUrl, rootElement, target) {
var xhr = new XMLHttpRequest();
xhr.overrideMimeType("application/json");
xhr.open('GET', dataUrl, true);
xhr.onreadystatechange = function() {
if (xhr.readyState == 4) {
if (xhr.status == 200) {
//parse jsoon data
var jsonData = JSON.parse(xhr.responseText);
var optionsHTML = ''
for(var i= 0; i < jsonData[rootElement].length; i++){
optionsHTML+='<option value="'+jsonData[rootElement][i].code+'">'
+ jsonData[rootElement][i].name+'</option>'
}
var targetCurrentHtml = target.innerHTML;
target.innerHTML = targetCurrentHtml + optionsHTML;
} else {
console.log(xhr.statusText);
// Show the error on the Web page
tempContainer.innerHTML += '<p class="error">Error getting ' +
target.name + ": "+ xhr.statusText + ",code: "+ xhr.status + "</p>";
}
}
}
xhr.send();
}
loadData('data/us-states.json', 'usstateslist', statesList);
loadData('data/countries.json', 'countrieslist', counriesList);In the above example we called the method XMLHttpRequest.overrideMimeType() to ensure that the data will be treated by the browser as JSON even if the server won’t report them as such.
JSON supports arrays, and the next example shows you how the information about a customer can be presented in JSON format. A customer can have more than one phone, which are stored in an array.
<script >
var customerJson = '{"fname":"Alex",
"lname":"Smith",
"phones":[
"212-555-1212",
"565-493-0909"
]
}';
var customer=JSON.parse(customerJson);
console.log("Parsed customer data: fname=" + customer.fname +
" lname=" + customer.lname +
" home phone=" + customer.phones[0] +
" cell phone=" + customer.phones[1]);
</script>The code above creates an instance of the JavaScript object referenced by the variable customer. In this example the phones array just holds two strings. But you can store object in JSON array the same way as you’d do it in JavaScript object literal - just don’t forget to put every property name in quotes.
var customerJson = '{"fname":"Alex",
"lname":"Smith",
"phones":[
{"type":"home", "number":"212-555-1212"},
{"type":"work","number":"565-493-0909"}]
}';The last example in Chapter 1 was about displaying various charity events on the Google map using multiple markers. But the data about these events were hard-coded in HTML file. After getting familiar with AJAX and JSON it should not be too difficult to create a separate file with the information about charities in JSON format and load them using XMLHTTPRequest object.
The next version of Save The Child is a modified version of the application that displayed Google map with multiple markers from Chapter 1. But this time we’ll load the information about the charity events from the file campaigndata.json shown next.
{
"campaigns": {
"header": "Nationwide Charity Events",
"timestamp":"12/15/2012",
"items": [
{
"title": "Lawyers for Children",
"description":"Lawyers offering free services for The Children",
"location":"New York,NY"
},
{
"title": "Mothers of Asthmatics",
"description":"Mothers of Asthmatics - nationwide Asthma network",
"location": "Dallas,TX"
},
{
"title": "Friends of Blind Kids",
"description":"Semi-annual charity events for blind kids",
"location":"Miami,FL"
},
{
"title": "A Place Called Home",
"description":"Adoption of The Children",
"location":"Miami,FL"
},
{
"title": "Marathon for Survivors",
"description":"Annual marathon for cancer survivors",
"location":"Fargo, ND"
}
]
}
}Run the project-03-maps-json-data and you’ll see the map with the markers for each of the events loaded from the file campaigndata.json (see Markers built from JSON data). Click on the marker to see an overlay with the event details.
Note that this JSON file contains the object campaigns, which includes the array of objects items representing charity events. XMLHttpRequest object loads the data and the JSON parses it assigning the campaigns object to the variable campaignsData that is used in showCampaignsInfo() with Google Maps API (we’ve omitted the mapping part for brevity).
function showCampaignsInfo(campaigns) {
campaignsCount = campaigns.items.length;
var message = "<h3>" + campaigns.header + "</h3>" +
"On " + campaigns.timestamp +
" we run " + campaignsCount + " campaigns.";
locationUI.innerHTML = message + locationUI.innerHTML;
resizeMapLink.style.visibility = "visible";
createCampaignsMap(campaigns);
}
function loadCampaignsData(dataUrl) {
var xhr = new XMLHttpRequest();
xhr.open('GET', dataUrl);
xhr.onreadystatechange = function() {
if (xhr.readyState == 4) {
if ((xhr.status >= 200 && xhr.status < 300) ||
xhr.status === 304) {
var jsonData = xhr.responseText;
var campaignsData = JSON.parse(jsonData).campaigns;
showCampaignsInfo(campaignsData);
} else {
console.log(xhr.statusText);
tempContainer.innerHTML += '<p class="error">Error getting ' +
target.name + ": "+ xhr.statusText +
",code: "+ xhr.status + "</p>";
}
}
}
xhr.send();
}
var dataUrl = 'data/campaignsdata.json';
loadCampaignsData(dataUrl);|
Tip
|
Some older Web browsers may bring up a File Download popup window when the content type of the server’s response is set to "application/json". Try to use the MIME type "text/html" instead, if you ran into this issue. |
|
Note
|
For simplicity, in this section we’ve been loading JSON formatted data from files, but in the real-world applications the JSON data is created on the server dynamically. For example, a browser makes a RESTFull call to the Java-based server, which queries a database, generates JSON formated result and sends it back to the Web server. |
Large-scale Web applications could be integrated with some Content Management Systems (CMS), which can be supplying content such as charity events, sales promotions, et al. CMS servers can be introduced into the architecture of a Web application to separate the work on preparing the content from the application delivering it as shown in CMS in the picture depicting a diagram of a with a Web application integrated with the CMS server.
The content contributors and editors prepare the information on the charities and donation campaigns using a separate application, not the Save The Child page. The CMS server and the Web application server www.savesickchild.org may be located in the same or separate data centers. The server-side code of the Save The Child is making a call to a CMS server whenever the site visitor is requesting the information about charity events. If you get to pick a CMS for your future Web application make sure it offers data feed in JSON format.
Some time ago one of the authors of this book were helping Mercedes Benz USA in development of their consumer facing Web application where people could search, review and configure their next car. Current Mercedes deals from CMS shows a snapshot taken from the mbusa.com. Three rectangular areas at the bottom were created by the Web designers to display today’s deals and promotions. The up-to-date content for these areas (in a JSON format) was retrieved from a CMS server when the user visited mbusa.com.
|
Tip
|
There’s a side benefit of learning JSON - it’s used as data format in NoSQL databases like MongoDB. |
If a Web browser receives JSON stream from the server the application needs to turn it into JavaScript objects. If a Web client needs to send the JavaScriot objects to the server they can be converted into JSON string. Similar tasks have to be performed on the server side. Our Save The Child application uses Java application server. There is a number of third-party Java libraries that can consume and generate JSON content.
There are several Java libraries to convert Java objects into their JSON representation and back, for example Google’s Gson, Jackson, json-simple.
Google’s Gson is probably the simplest one for use. It provides methods toJson() and fromJson() to convert Java objects to JSON and back. Gson allows pre-existing un-modifiable objects to be converted to and from JSON and Supports Java Generics. Gson works well with complex objects with deep inheritance hierarchies.
Let’s say JavaScript sends to Java the following JSON string:
{"fname": "Alex", "lname":"Smith","skillLevel": 11}
The Java code can turn it into an instance of the Customer object by calling the method Gson.fromJson(). Similarly, Java code can create a JSON string from an object instance. Both of these operations are illustrated below.
public Customer createCustomerFromJson(String jsonString){
Gson myGson = new Gson();
Customer cust = myGson.fromJson(jsonString, Customer.class);
return cust;
}
public String createJsonFromCustomer(Customer cust){
Gson gson = new Gson();
return gson.toJson(cust, Customer.class);
}Of course, the declaration of the Java class Customer must exist in the in the classpath and don’t forget to include gson.jar to your Java project.
JSON data format is often used in non-JavaScript applications. For example, a Java server can exchange the JSON-formatted data with a .Net server.
|
Note
|
The Java EE 7 specification includes JSR 353, which defines a standardized way for parsing and generating JSON. JSR 353 defines the Java API from JSON Processing (JSON-P) that shouldn’t be confused with another acronym JSONP or JSON-P, which is JSON with Padding (we’ll discuss it at the end of this chapter). |
JSON format is more compact than XML and is readable by the human beings. But when you are ready to deploy your application in production, you still want to compress the data so less bytes will travel over the wire to the user’s browser. The server-side libraries that generate JSON will make the data sent to the client compact by removing the tab and the new line characters.
If you want to turn the pretty-print JSON into a more compact one-line format just use such Web sites as JavaScript Compressor or JSON Formatter. For example, after running the 12Kb file countries.json through this compressor, its size was decreased to 9Kb. JSONLint can also compress JSON if you provide this URL: http://jsonlint.com?reformat=compress.
Similarly to most of the content that is being sent to browsers by the Web servers, the JSON data should be compressed. GZip and Deflate are the two main compression methods used in today’s Web. Both use the same compression algorithm deflate, but while with Deflate the compressed data are being streamed to the client, the GZip first compresses the entire file, calculates the size and adds some additional headers to the compressed data. So GZip may need some extra time and memory, but you are more protected from getting incomplete JSON, JavaScript or other content. Both Gzip and Deflate are easily configurable by major Web servers, but it’s hard to say which one is better for your application - set up some tests with each of them and decide which one works faster or take less system resources, but don’t compromise on reliability of the compressed content.
We prefer using GZip, which stands for GNU zip compression. On the server side you’d need to configure the gzip filters on your Web server. You need to refer to your Web server’s documentation to find out how to configure gzipping, which is done by the MIME type. For example, you can request to gzip everything except images (you might want to do this if you’re not sure if all browsers can properly uncompress certain MIME types).
For example, applying the GZip filter to the 9Kb countries.json will reduce its size to 3Kb, which means serious bandwidth savings especially in the Web applications with lots of concurrent users. This is even more important for the mobile Web clients, which may be operating in the areas with slower connections. Web clients usually set the HTTP Request attribute Accept-Encoding: gzip inviting the server to return gzipped content, and the Web server may compress the response if it does support it or unzipped content otherwise. If the server supports gzip, the HTTP response will have the attribute Content-Encoding: gzip, and the browser will know to unzip the response data before use.
Gzip is being used for compressing all types of content: HTML, CSS, JavaScript and more. If your server sends JSON content to the client setting the content type to application/json don’t forget to include this MIME type in your server configuration for Gzip.
Web browsers support the gzipping too, and your application can set Content-Ecoding: gzip in HTTP request while sending the data from the Web client to the server. But Web clients usually don’t send massive amounts of data to the server so the benefits of the compression on the client side may not be as big.
Let’s consider yet another use case for JSON in Save The Child. We want to display charts with statistics about the donations. By now, our application look not exactly as the original mockup from [FIG3-2], but it’s pretty close. There is an empty space in the left to the maps, and the charts showing donation statistics can fit right in. Now we need to decide how to draw the charts using nothing, but HTML5 elements. Note that we are not talking about displaying static images using the <img> element - the goal is to draw the images dynamically in the client’s code. You can accomplish this goal using HTML5 elements <canvas> or <svg>.
The <canvas> element provides a bitmap canvas, where your scripts which can draw graphs, game graphics, or other visual images on the fly without using any plugins like Flash Player or Silverlight. To put it simple, the <canvas> defines a rectangular area that consists of pixels, where you can draw. Keep in mind that the DOM object can’t peek inside the canvas and access specific pixels. So if you are planning to create an area with dynamically changed graphics you might want to consider using <svg>.
The <svg> element supports Scalable Vector Graphics (SVG), which is the XML-based language for describing two-dimensional graphics. Your code has to provide commands to draw the lines, text, images et al.
Let’s review some code fragments from project-04-canvas-pie-chart-json. The HTML section defines <canvas> of 260x240 pixels. If the user’s browser doesn’t support <canvas>, the user won’t see the chart, but will see the text "Your browser does not support HTML5 Canvas" instead. You need to give an ID to your canvas element so your JavaScript code can access it.
<div id="charts-container">
<canvas id="canvas" width="260" height="240">
Your browser does not support HTML5 Canvas
</canvas>
<h3>Donation Stats</h3>
<p> Lorem ipsum dolor sit amet, consectetur</p>
</div>Run the project-04-canvas-pie-chart-json, and you’ll see the chart with donation statistics by city as in Adding a chart. We haven’t style our <canvas> element, but we could’ve added a background color, the border and other bells and whistles if required.
The data to be used for drawing a pie chart in our canvas are stored in the file data/chartdata.json, but in a real-world the server side code can generate it based on the up-to-the-second donation data and send it to the client. For example, you could do it as was explained in the section Json in Java above. This is the content of our file chartdata.json:
{
"ChartData": {
"items": [
{
"donors": 48,
"location":"Chicago, IL"
},
{
"donors": 60,
"location": "New York, NY"
},
{
"donors": 90,
"location":"Dallas, TX"
},
{
"donors": 22,
"location":"Miami, FL"
},
{
"donors": 14,
"location":"Fargo, ND"
},
{
"donors": 44,
"location":"Long Beach, NY"
},
{
"donors": 24,
"location":"Lynbrook, NY"
}
]
}
}Loading of the the charddata.json is done using AJAX techniques as explained earlier. Although in our example we’re loading the chart immediately when the Save The Child loads, the following code could be invoked only when the user requests to see the charts by clicking on some menu item on the page.
function getChartData(dataUrl, canvas) {
var xhr = new XMLHttpRequest();
xhr.open('GET', dataUrl, true);
xhr.onreadystatechange = function() {
if (xhr.readyState == 4) {
if ((xhr.status >= 200 && xhr.status < 300) ||
xhr.status === 304) {
var jsonData = xhr.responseText;
var chartData = JSON.parse(jsonData).ChartData; // (1)
drawPieChart(canvas, chartData, 50, 50, 49); // (2)
} else {
console.log(xhr.statusText);
tempContainer.innerHTML += '<p class="error">Error getting ' +
target.name + ": "+ xhr.statusText +
",code: "+ xhr.status + "</p>";
}
}
}
xhr.send();
}
loadData('data/chartdata.json', document.getElementById("canvas"));-
Parse JSON and create the
ChartDataJavascript object. -
Pass the data to the
drawPieChart()function that will draw the pie in thecanvaselement with the center coordinates x=50 and y=50 pixels. The top left corner of the canvas has coordinates (0,0). The radius of the pie will be 49 pixels. The code of the function that draws the pie on the canvas goes next.
function drawPieChart (canvas, chartData, centerX, centerY, pieRadius) {
var ctx; // The context of canvas
var previousStop = 0; // The end position of the slice
var totalDonors = 0;
var totalCities = chartData.items.length;
// Count total donors
for (var i = 0; i < totalCities; i++) {
totalDonors += chartData.items[i].donors; // (1)
}
ctx = canvas.getContext("2d"); // (2)
ctx.clearRect(0, 0, canvas.width, canvas.heigh);
var colorScheme = ["#2F69BF", "#A2BF2F", "#BF5A2F", // (3)
"#BFA22F", "#772FBF", "#2F94BF", "#c3d4db"];
for (var i = 0; i < totalCities; i++) { // (4)
//draw the sector
ctx.fillStyle = colorScheme[i];
ctx.beginPath();
ctx.moveTo(centerX, centerY);
ctx.arc(centerX, centerY, pieRadius, previousStop, previousStop +
(Math.PI * 2 * (chartData.items[i].donors/totalDonors)),false);
ctx.lineTo(centerX, centerY);
ctx.fill();
// label's bullet
var labelY = 20 * i + 10;
var labelX = pieRadius*2 + 20;
ctx.rect(labelX, labelY, 10, 10);
ctx.fillStyle = colorScheme[i];
ctx.fill();
// label's text
ctx.font = "italic 12px sans-serif";
ctx.fillStyle = "#222";
var txt = chartData.items[i].location + " | " +
chartData.items[i].donors;
ctx.fillText (txt, labelX + 18, labelY + 8);
previousStop += Math.PI * 2 * (chartData.items[i].donors/totalDonors);
}
}-
Count the total number of donors.
-
Get the 2D context of the
<canvas>element. This is the most crucial element to know for drawing on a canvas. -
The color scheme is just a set of colors to be used for painting each slice (sector) of the pie.
-
The for-loop paints one sector on each iteration. This code draws lines, arcs, rectangles, and adds text to the canvas. Describing the details of each method of the context object is out of scope of this book, but you can find the details of the context API in the W3C documentation available online.
|
Tip
|
To minimize the amount of manual coding, consider using one of the JavaScript libraries that helps with visualization (e.g. D3.js). |
What if we want to make this chart dynamic and reflect the changes in donations every 5 minutes? If you’re using <canvas>, you’ll need to redraw each and every pixel of our canvas with the pie. With SVG, each element of the drawing would be the DOM element so we could have redraw only those elements that have changed. If with canvas your script draws using pixels, the SVG drawings are done with vectors.
To implement the same donation statistics pie with the <svg> element, you’d need to replace the <canvas> element with the following markup:
<div id="charts-container">
<svg id="svg-container" xmlns="http://www.w3.org/2000/svg">
</svg>
<h3>Donation Stats</h3>
<p>
Lorem ipsum dolor sit amet, consectetur
</p>
</div>Running the project-05-svg-pie-chart-json would show you pretty much the same pie as it uses the file chartdata.json with the same content, but the pie was produced differently. The code of the new version of the drawPieChart() is shown below. We won’t discuss all the details of the drawing with SVG, but will highlight the a couple of important lines of code that illustrate the difference between drawing on <canvas> vs. <svg>.
function drawPieChart(chartContaner, chartData, centerX, centerY,
pieRadius, chartLegendX, chartLegendY) {
// the XML namespace for svg elements
var namespace = "http://www.w3.org/2000/svg";
var colorScheme = ["#2F69BF", "#A2BF2F", "#BF5A2F", "#BFA22F",
"#772FBF", "#2F94BF", "#c3d4db"];
var totalCities = chartData.items.length;
var totalDonors = 0;
// Count total donors
for (var i = 0; i < totalCities; i++) {
totalDonors += chartData.items[i].donors;
}
// Draw pie sectors
startAngle = 0;
for (var i = 0; i < totalCities; i++) {
// End of the sector = starting angle + sector size
var endAngle = startAngle + chartData.items[i].donors / totalDonors * Math.PI * 2;
var x1 = centerX + pieRadius * Math.sin(startAngle);
var y1 = centerY - pieRadius * Math.cos(startAngle);
var x2 = centerX + pieRadius * Math.sin(endAngle);
var y2 = centerY - pieRadius * Math.cos(endAngle);
// This is a flag for angles larger than than a half circle
// It is required by the SVG arc drawing component
var big = 0;
if (endAngle - startAngle > Math.PI) {
big = 1;
}
//Create the <svg:path> element
var path = document.createElementNS(namespace, "path"); // (1)
// Start at circle center
var pathDetails = "M " + centerX + "," + centerY + // (2)
// Draw line to (x1,y1)
" L " + x1 + "," + y1 +
// Draw an arc of radius
" A " + pieRadius + "," + pieRadius +
// Arc's details
" 0 " + big + " 1 " +
// Arc goes to to (x2,y2)
x2 + "," + y2 +
" Z";
// Close the path at (centerX, centerY)
// Attributes for the <svg:path> element
path.setAttribute("d", pathDetails);
// Sector fill color
path.setAttribute("fill", colorScheme[i]);
chartContaner.appendChild(path); // (3)
// The next sector begins where this one ends
startAngle = endAngle;
// label's bullet
var labelBullet = document.createElementNS(namespace, "rect");
// Bullet's position
labelBullet.setAttribute("x", chartLegendX);
labelBullet.setAttribute("y", chartLegendY + 20 * i);
// Bullet's size
labelBullet.setAttribute("width", 10);
labelBullet.setAttribute("height", 10);
labelBullet.setAttribute("fill", colorScheme[i]);
chartContaner.appendChild(labelBullet); // (3)
// Add the label text
var labelText = document.createElementNS(namespace, "text");
// label position = bullet's width(10px) + padding(8px)
labelText.setAttribute("x", chartLegendX + 18);
labelText.setAttribute("y", chartLegendY + 20 * i + 10);
var txt = document.createTextNode(chartData.items[i].location +
" | "+chartData.items[i].donors);
labelText.appendChild(txt);
chartContaner.appendChild(labelText); // (3)
}
}-
Create the
<svg:path>HTML element, which is the most important SVG element for drawing basic shapes.. It includes a series of commands that produce the required drawing. For example, M 10 10 means movo to the coordinate 10,10 and L 20 30 means draw the line to the coordinate 20,30. -
Fill the details of the
<svg:path>element to draw the pie sector. Run the project-05-svg-pie-chart-json to see the Save The Child page, then right-click on the pie chart and select Inspect Element (this is the name of the menu item in Firefox). The chart content in SVG shows the resulting content of our<svg>element. As you can see, it’s not pixel based but a set of XML-like commands that drew the content of the chart. If you’ll run the previous version of our application (project-04-canvas-pie-chart-json) and right-click on the chart, you will be able to save it as an image, but won’t see the internals of the<canvas>element. -
Adding the internal elements of the chart container to the DOM - path, bullets and text. These elements can be modified if needed without redrawing the entire content of the container.
|
Tip
|
in our code example we have written the path commands manually to process the data dynamically. But Web designers often use tools (Adobe Illustrator, Incscape et al.) to draw and then export images into an SVG format. In this case all paths will be encoded as <svg:path> automatically.
|
Since the SVG is XML-based, its very easy to generate the code shown in The chart content in SVG on the server, and lots of Web applications send ready to display SVG graphics to the users' Web browsers. But in our example we are generating the SVG output in the JavaScript from JSON received from the server, which provides a cleaner separation between the client and the server-side code. The final decision on what to send to the Web browser (ready to render SVG or raw JSON) has to be made after considering various factors such as available bandwidth, the size of data, the number of users, the existing load on the server resources.
|
Tip
|
SVG supports animations and transformation effects, while canvas doesn’t. |
Imagine that a Web page was loaded from the domain abc.com, and it need JSON formatted data from another domain (xyz.com). As mentioned earlier, AJAX has cross-origin restrictions, which prevent this. JSONP is a technique used in to relax the cross-origin restrictions. With JSONP, instead of sending plain JSON data, the server wraps them up into a JavaScript function and then sends it to the Web browser for execution as a callback. The Web page that was originated from abc.com may send the request http://xyz.com?callback=myDataHandler technically requesting the server xyz.com to invoke the JavaScript callback named myDataHandler. This URL is a regular HTTP GET request, which may have other parameters too so you can send some data to the server too.
The server will send to the browser the JavaScript function that may look as follow:
function myDataHandler({"fname": "Alex", "lname":"Smith","skillLevel": 11});
The Web browser will invoke the callback myDataHandler(), which must exist in the Web page. The Web browser will pass the received JSON object as an argument to this callback:
function myDataHandler(data){
// process the content of the argument data - the JSON object
// received from xyz.com
}If all you need is just to retrieve the data from a different domain on page just add the following tag to your HTML page:
<script src="http://xyz.com?callback=myDataHandler">
But what if you need to dynamically make such requests periodically (e.g. get all twits with a hashtag #savisickchild by sending an HTTP GET using Twitter API at http://search.twitter.com/search.json?q=savesickchild&rpp=5&include_entities=true&with_twitter_user_id=true&result_type=mixed)? You add a change handler to the option that is called and passes or grabs the value needed.
You can dynamically add a <script> tag to the DOM object from your JavaScript code. Whenever the browser sees the new <script> element it executes it. Such script injection can be done like this:
var myScriptTag = document.createElement("script");
myScriptTag.src = "http://xyz.com?callback=myDataHandler";
document.getElementsByTagName("body").appendChild(myScriptTag);Your JavaScript can build the URL for the myScriptTag.src dynamically and pass parameters to the server based on some user’s actions.
Of course, this technique presents a danger if there is a chance that the JavaScript code sent by xyz.com is intercepted and replaced by a malicious code (similarly to the JavaScript eval() danger). But it’s not more dangerous that receiving any JavaScript from non-trusted server. Besides, your handler function could always make sure that the received data is a valid object with expected properties, and only after that handle the data.
If you decide to use JSONP don’t forget about error handling. Most likely you’ll be using one of the JavaScript frameworks and they usually offer a standard mechanism for JSONP error handling, dealing with poorly formatted JSON responses, and recovery in cases of network failure. One of such libraries is called jQuery-JSONP.
In this section you’ll see a small code example illustrating the data retrieval from publicly publicly available Open Beer DataBase, which exists to help software developers test their code that makes REST Web service calls and works with JSON and JSONP data. Our Save The Child page won’t display beer bottles, but we want to show that in addition to the retrieval of the donations and charts data from one domain we can get the data from a third-party domain openbeerdatabase.com.
First, enter the URL http://api.openbeerdatabase.com/v1/breweries.json in the address bar of your Web browser, and it’ll return the following JSON data (only 2 out of 7 breweries are shown for brevity):
{
"page": 1,
"pages": 1,
"total": 7,
"breweries": [
{
"id": 1,
"name": "(512) Brewing Company",
"url": "http://512brewing.com",
"created_at": "2010-12-07T02:53:38Z",
"updated_at": "2010-12-07T02:53:38Z"
},
{
"id": 2,
"name": "21st Amendment Brewing",
"url": "http://21st-amendment.com",
"created_at": "2010-12-07T02:53:38Z",
"updated_at": "2010-12-07T02:53:38Z"
}
]
}Now let’s request the same data, but in a JSONP format by adding to the URL a parameter with a callback name myDataHandler. Entering in the browser http://api.openbeerdatabase.com/v1/breweries.json?callback=processBeer will return the following (it’s a short version):
processBeer({"page":1,"pages":1,"total":7,"breweries":[{"id":1,"name":"(512) Brewing Company",
"url":"http://512brewing.com","created_at":"2010-12-07T02:53:38Z",
"updated_at":"2010-12-07T02:53:38Z"},{"id":2,"name":"21st Amendment Brewing",
"url":"http://21st-amendment.com","created_at":"2010-12-07T02:53:38Z",
"updated_at":"2010-12-07T02:53:38Z"}]})Since we haven’t declared the function processBeer() yet, it won’t be invoked. Let’s fix it now. The function will check first if the received data contains the information about the breweries. If it does, the name of the very first brewery will be printed on the JavaScript console. Otherwise the console output will read "Retrieved data has no breweries info".
var processBeer=function (data){
// Uncomment the next line to emulate malicious data
// data="function evilFunction(){alert(' Bad function');}";
if (data.breweries == undefined){
console.log("Retrieved data has no breweries info.");
} else{
console.log("In the processBeer callback. The first brewery is "
+ data.breweries[0].name);
}
}
var myScriptTag = document.createElement("script");
myScriptTag.src =
"http://api.openbeerdatabase.com/v1/breweries.json?callback=processBeer";
var bd = document.getElementsByTagName('body')[0];
bd.appendChild(myScriptTag);The beer has arrived is a screen snapshot taken in the Firebug when it reached the breakpoint placed inside the processBeer callback on the console.log(in the processBeer callback"). You can see the content of the data argument - the beer has arrived.
As a training exercise, try to replace the data retrieval from the beer Web service with adding the data feed from Twitter based on certain hash tags. See if you can find the place in the Save The Child Web page to display (and periodically update) this Twitter stream.
|
Tip
|
json-generator.com is a handy Web site that can generate a file with a JSON or JSONP content based on your template. You can use this service to test AJAX queries - the generated JSON can be saved on this server to help in testing of your Web application. |
In this chapter you’ve learned about using AJAX as a means of communications of your Web browser with the servers. AJAX also deserves a credit for making the JavaScript language popular again by showing a practical way of creating single-page Web applications. Over the years JSON became the standard way of exchanging the data on the Web. The current version of the Save The Child application cleanly separates the code from the data, and you know how to update the content of the Web page without the need to re-retrieve the entire page from the server. In the next chapter you’ll get familiar with the more productive way of developing Web applications with the library called jQuery.