This was a unique approach that solves one of the major pains of web development: Rendering markup in two different ways on the server and client side.
Trouble on our first prototype
Like many companies, we initially wanted to adopt technologies already familiar to the rest of our engineers. Python was our language of choice, therefore we built our web server on the Python Flask Framework and rendered templates on the server side with Jinja2. Any kind of dynamic user interface is then sprinkled on the front end with Backbone.js.
This approach served us well for a while and we shipped a working website into the wild. As the source code grew, we started to have these headaches:
Our web server has mixed concerns of api duties, caching, and rendering of the user interface (UI).
Often, when we require a bit of client-side routing (for a snappier interface), we ended up writing the same type of controllers and templates on Flask and Backbone.js. Writing views, controllers and filters in different languages and frameworks is unsurprisingly error prone!
Bootstrapping state from the server side to client side is not straightforward. The server should create the initial UI, and the frontend should take over and attach event handlers. This process is made more challenging when we have two different ways of rendering UI on the server side and client side.
Too much context switching because of the different languages and frameworks involved.
Naturally, we sought to unify the duties of rendering UI under a single language and framework to solve these problems. This forms a UI layer that consistently produce the same markup on the server and client side.
Core to this design is a routing layer that maps pages to different routes. Each page request may fire a data fetching action and the results are then used to prerender markup on the server side. All navigation and interactions after the first page behaves like a Single Page Application (SPA), accumulating state through more data fetching and mutating markup on the fly.
The entire UI layer is designed to be stateless on the server side, fetching all required data from an external service and dehydrating them to the client side. This allows us to scale up the UI layer without worrying about inconsistencies in state.
The existing web server then becomes a cache layer for the UI layer to retrieve data from, and will periodically fetch new data from the database layer.
1. Selecting a common rendering framework
A big part of the design hinges on the ability of the rendering framework. After much research, we chose React for the following reasons:
Deterministic way of getting the same UI given the same state and markup.
Automatic rebuilding of UI under the hood with performance optimizations to prevent unnecessary DOM redraw.
Declarative way of writing UI, versus the imperative style of our Backbone.js code.
2. Trying out React on a small scale
Initially, we were unsure about React so we wanted to try it on something small before commiting more effort to it. We rolled out React on a simple location filter for the Search page.
The new React component was built on top of the existing Flask + Backbone stack. This is one of the nice things of working with React since it plays well with any existing frontend stack. The only drawback which we had to accept was additional page weight of React, which amounts to a whopping 133KB!
At first, JSX seems strange. Some say it looks like an abomination. We have been building web applications in MVC style for a long time, neatly separating out views from logic. Surely JSX is bad?
Maybe, but React is all about composing reusable UI. React forces us to think of composing a page as a tree of smaller components. Each component then contains markup neatly packaged with interactions on the markup. This is a good thing for us. A new hire (or a designer) could view a component and easily tell not only how it looks like, but also how it behaves!
3. Building a whole page with React
Without tearing down our existing Flask + Backbone stack, we built a Sell page with React on Flask like how one might do it SPA-style. We rendered a barebones
<body /> tag, sent down scripts with React code, and allowed the client to build the sell page.
This time, the task was more complex since there are more interactions involved and use of 3rd party libraries for image editing. React comes with several lifecycle hooks that we eventually used to add initialization code for all our 3rd party widgets / libraries. Fun fact: Be careful of 3rd party libraries that clone DOM nodes. As part of its own book-keeping, React assigns a unique
react-id to DOM nodes so it gets especially mad when it finds multiple DOM nodes with the same
React code tends to become lengthy due to mixing interactions with markup. To keep ourselves sane, we made use of React components’ composability and wrote many micro components and assembled them together to form the sell page. Many of these components could also be reused elsewhere in the source code (after a teeny bit of refactoring ;] )
After breaking some sweat, we learnt some of the framework quirks and love how easy it is to build complex UI with React. Components are declared with how they should look like under different states, versus writing instructions on how to manipulate the DOM to get our desired UI. This was definitely a great productivity boost compared to other approaches of web development.
With the UI rendering framework out of the way, it was finally time to try out a pared-down version of the final design.
We built a simple Node.js application serving a small page, the Login page. This page was rewritten from the Flask side to the Node.js side. The complexity was small enough for us to ensure the following is working:
Shared code to render UI
LoginPage component code can be used on the server and client side to build the UI. React handles the entire lifecycle of the UI from the server to the client. When the
LoginPage is built on the server, it does not rebuild the page again on the client side and simply attaches the event handlers.
Simultaneously keeping 2 versions of the web in production
Using a load balancer, we whitelisted certain routes (in our case, the login route) to be directed to our Node.js server, and directed the rest of the traffic to the old Flask server.
Stateless UI rendering server
In the “Design” section, we discussed how our UI layer gets data from an external service. We were able to test this design by building a stateless Node.js server that talks to the Flask server and proxies data from Flask to the browser.
This allowed us to authenticate a user on the Login page and safely redirect the user to other pages, authenticated as though they had logged in on the old login page served by the Flask + Backbone stack.
5. Going full scale
Using the same technique of whitelisting routes, we can safely migrate existing features and new features to the UI layer. Some of the old JSX code that were written on top of the Flask + Backbone stack were easy to port over to the new stack.
We have built Carousell Web with most of what I’ve described. Developing the web application is no longer as painful as it was. We have:
Component-driven development with UI as a function of states. This leads to a lot of predictability and ease of writing complex UI.
UI written in the same language meant consistency and no context switching.
The same server-side rendering that allows search engines to index our links easily and fast content load for the first page since the entire page is prerendered on the server side.
The same client-side rendering that gives SPAs its fast page transitions.
Not a Silver Bullet
Nonetheless, here are some tips to deal with the rendering speed:
Clustering more Node.js processes to take advantage of multi-core servers. Additional processes are capable of handling more rendering jobs so React does not starve the event loops from handling requests.
Caching React components. In sites where there is very little user-specific pages, you may cache React components (or even fully-formed html) in your favorite caches (Redis, Memcached, Varnish, etc.)
Using optimized build of React on server side. React comes with a lot of helpful checks and warnings, which should be turned off on production since they come with additional overhead. This can be done by resolving
require("react")to an optimized build of the library (
react-with-addons.min) on production environment. See this issue for more information.