Hello again! Thanks for getting this far, hopefully you enjoyed the previous posts too! The graphing tool is now live for you to have a play with if you haven’t had a chance to yet. I’m currently working hard on making it faster and better which I will talk about in this post.
This is final post of the series and possibly the most important. This post relates to the handling of the data itself and some of the problems I’ve faced along the way. I’ll talk you through how the data handling has evolved and the various tools I’ve used to improve the speed of the site and the accessibility of the data.
When it first came to accessing the data on the web page I went for what I knew – the http GET request. We were running a Node Red server that, when it received the request on a specific url replied with the data from the balloon as stored in a database. This worked locally just as expected and it worked well. The website displayed the data on the screen, all of it. We then moved to testing that across the internet and it didn’t work due to what is known as ‘the same-origin policy’.
The same origin policy dictates that requests can only be made to a server running on the same domain as the website. This was a problem but after a bit of research I found a way round it. Cross-domain requests can be made inside html script tags (like for accessing external libraries such as jQuery), combining this with the JSONP data structure allowed me to access the data cross-domain. JSONP works by wrapping JSON in a function call with the data as the argument, then when the script tag sends the request a function is called (which you define) and is passed the data. Voilà! I have defeated the same-origin policy with this work around but I still needed the data to be live.
Even with all of this in place I still required the user to refresh the page in order to get the newest data. I didn’t like this as I had set out to make the data live and forcing the user to get the data themselves was not efficient or live. To solve this I wrote a function to rewrite the script tag in the html in order to refresh the data periodically. This worked and the site now responded to new incoming data live. However (as you are probably thinking too) this is very inefficient. I quickly realised that although there was not much data at the moment, when it came to launch refreshing the entire dataset repeatedly is not a good way to do this. I thought there has to be a better way – and there was!
This was when I discovered websockets. Why this wasn’t suggested for this task on every message board online I do not know because this solution seems clearly and obviously better. Websockets are like a http GET except the first connection is a handshake and then the channel is kept open in both directions. The client can query the server whenever is wants and the server can send over new data as and when it arrives! We set this up in the server (very quickly indeed with node red) and now the client can query the database directly – Perfect! This allows me to not only request only the data I need but also add new features that take advantage of this and would have taken too long with the old system. Are there any downsides to websockets? Yes, mainly that it is not supported in all browsers. However 85% of users have access to it and we have the old system in place as a backup for non-supported browsers.
Although websockets seem to be the answer there are some alternative solutions you could use. Hosting the website on the same server as the website would eliminate a lot of the cross-domain issues entirely and you could have direct access to the data. There is also a cross-domain protocol being put in place that is very similar to http GET called CORS. This would also work however I found it very difficult to use and seemingly in it’s infancy.
I enjoyed all of these challenges and hope by explaining my mistakes it will help someone else who is maybe battling the same issues. Currently the live site is not running the websockets code yet but I will migrate it soon so that you can check out the code. As always you can email me with questions at lde12@ic.ac.uk.
No comments:
Post a Comment