Moving towards a Universal JavaScript application

Shawn Lim bio photo By Shawn Lim

Introduction

Back in November 2013, the idea of Universal JavaScript came out in the JavaScript community, back then coined “Isomorphic Javascript”. The architecture seeks to achieve SEO, Performance and Maintainability for web applications. It uses JavaScript on the entire stack for the web client, and talks to an external service for data.

This was a unique approach that solves one of the major pains of web development: Rendering markup in two different ways on the server and client side.

We were very inspired and sought to move our Web client, which was built on Backbone.js and Python Flask. While many resources and tutorials have surfaced to build Universal JavaScript applications, it is less obvious how existing web applications can move towards it.

We will be sharing our adventure (and pain) of transforming Carousell Web into a Universal JavaScript application in this blog post so that it may help those who are considering the same move.

Trouble on our first prototype

Like many companies, we initially wanted to adopt technologies already familiar to the rest of our engineers. Python was our language of choice, therefore we built our web server on the Python Flask Framework and rendered templates on the server side with Jinja2. Any kind of dynamic user interface is then sprinkled on the front end with Backbone.js.

This approach served us well for a while and we shipped a working website into the wild. As the source code grew, we started to have these headaches:

  1. Our web server has mixed concerns of api duties, caching, and rendering of the user interface (UI).

  2. Often, when we require a bit of client-side routing (for a snappier interface), we ended up writing the same type of controllers and templates on Flask and Backbone.js. Writing views, controllers and filters in different languages and frameworks is unsurprisingly error prone!

  3. Bootstrapping state from the server side to client side is not straightforward. The server should create the initial UI, and the frontend should take over and attach event handlers. This process is made more challenging when we have two different ways of rendering UI on the server side and client side.

  4. Too much context switching because of the different languages and frameworks involved.

Design

Naturally, we sought to unify the duties of rendering UI under a single language and framework to solve these problems. This forms a UI layer that consistently produce the same markup on the server and client side.

Core to this design is a routing layer that maps pages to different routes. Each page request may fire a data fetching action and the results are then used to prerender markup on the server side. All navigation and interactions after the first page behaves like a Single Page Application (SPA), accumulating state through more data fetching and mutating markup on the fly.

The entire UI layer is designed to be stateless on the server side, fetching all required data from an external service and dehydrating them to the client side. This allows us to scale up the UI layer without worrying about inconsistencies in state.

The existing web server then becomes a cache layer for the UI layer to retrieve data from, and will periodically fetch new data from the database layer.

Carousell Web Universal JS Architecture

Implementation

1. Selecting a common rendering framework

A big part of the design hinges on the ability of the rendering framework. After much research, we chose React for the following reasons:

  • Deterministic way of getting the same UI given the same state and markup.

  • Automatic rebuilding of UI under the hood with performance optimizations to prevent unnecessary DOM redraw.

  • Declarative way of writing UI, versus the imperative style of our Backbone.js code.

2. Trying out React on a small scale

Initially, we were unsure about React so we wanted to try it on something small before commiting more effort to it. We rolled out React on a simple location filter for the Search page.

The new React component was built on top of the existing Flask + Backbone stack. This is one of the nice things of working with React since it plays well with any existing frontend stack. The only drawback which we had to accept was additional page weight of React, which amounts to a whopping 133KB!

At first, JSX seems strange. Some say it looks like an abomination. We have been building web applications in MVC style for a long time, neatly separating out views from logic. Surely JSX is bad?

Maybe, but React is all about composing reusable UI. React forces us to think of composing a page as a tree of smaller components. Each component then contains markup neatly packaged with interactions on the markup. This is a good thing for us. A new hire (or a designer) could view a component and easily tell not only how it looks like, but also how it behaves!

3. Building a whole page with React

Without tearing down our existing Flask + Backbone stack, we built a Sell page with React on Flask like how one might do it SPA-style. We rendered a barebones <body /> tag, sent down scripts with React code, and allowed the client to build the sell page.

This time, the task was more complex since there are more interactions involved and use of 3rd party libraries for image editing. React comes with several lifecycle hooks that we eventually used to add initialization code for all our 3rd party widgets / libraries. Fun fact: Be careful of 3rd party libraries that clone DOM nodes. As part of its own book-keeping, React assigns a unique react-id to DOM nodes so it gets especially mad when it finds multiple DOM nodes with the same react-id.

React code tends to become lengthy due to mixing interactions with markup. To keep ourselves sane, we made use of React components’ composability and wrote many micro components and assembled them together to form the sell page. Many of these components could also be reused elsewhere in the source code (after a teeny bit of refactoring ;] )

After breaking some sweat, we learnt some of the framework quirks and love how easy it is to build complex UI with React. Components are declared with how they should look like under different states, versus writing instructions on how to manipulate the DOM to get our desired UI. This was definitely a great productivity boost compared to other approaches of web development.

4. A baby Universal JavaScript application

With the UI rendering framework out of the way, it was finally time to try out a pared-down version of the final design.

We built a simple Node.js application serving a small page, the Login page. This page was rewritten from the Flask side to the Node.js side. The complexity was small enough for us to ensure the following is working:

Shared code to render UI

The same LoginPage component code can be used on the server and client side to build the UI. React handles the entire lifecycle of the UI from the server to the client. When the LoginPage is built on the server, it does not rebuild the page again on the client side and simply attaches the event handlers.

Simultaneously keeping 2 versions of the web in production

Using a load balancer, we whitelisted certain routes (in our case, the login route) to be directed to our Node.js server, and directed the rest of the traffic to the old Flask server.

Stateless UI rendering server

In the “Design” section, we discussed how our UI layer gets data from an external service. We were able to test this design by building a stateless Node.js server that talks to the Flask server and proxies data from Flask to the browser.

This allowed us to authenticate a user on the Login page and safely redirect the user to other pages, authenticated as though they had logged in on the old login page served by the Flask + Backbone stack.

5. Going full scale

Using the same technique of whitelisting routes, we can safely migrate existing features and new features to the UI layer. Some of the old JSX code that were written on top of the Flask + Backbone stack were easy to port over to the new stack.

As we create more components, many of them start to have the same data dependencies and so we adopted Fluxible to simplify data flow. There are many other alternatives such as Alt but we liked Fluxible because it comes with many utility functions to build Universal JavaScript applications and an extendable API to create useful plugins to support our design.

Outcome

We have built Carousell Web with most of what I’ve described. Developing the web application is no longer as painful as it was. We have:

  • Component-driven development with UI as a function of states. This leads to a lot of predictability and ease of writing complex UI.

  • UI written in the same language meant consistency and no context switching.

  • The same server-side rendering that allows search engines to index our links easily and fast content load for the first page since the entire page is prerendered on the server side.

  • The same client-side rendering that gives SPAs its fast page transitions.

  • A UI rendering service built with JavaScript that talks to other API / Backend services written in other languages.

Not a Silver Bullet

React renders components on the server side at a price of high CPU usage. In general, that is harmful for Node.js applications since React will block the event loop. Should you still build a Universal JavaScript application in React + Node? That depends on how much you value the development speed and maintainability versus raw rendering speed.

Nonetheless, here are some tips to deal with the rendering speed:

  1. Clustering more Node.js processes to take advantage of multi-core servers. Additional processes are capable of handling more rendering jobs so React does not starve the event loops from handling requests.

  2. Caching React components. In sites where there is very little user-specific pages, you may cache React components (or even fully-formed html) in your favorite caches (Redis, Memcached, Varnish, etc.)

  3. Using optimized build of React on server side. React comes with a lot of helpful checks and warnings, which should be turned off on production since they come with additional overhead. This can be done by resolving require("react") to an optimized build of the library (react-with-addons.min) on production environment. See this issue for more information.

Conclusion

Moving an existing web application to Universal JavaScript was extremely rewarding. Our approach represents much of how engineering works in Carousell: experimenting with prototypes and performing incremental work on production systems to better understand new tools. We love writing good software and there are still many interesting ways to optimize our Universal JavaScript setup for performance and scale. If you are interested in working on these, ping us on Carousell Jobs