This was a unique approach that solves one of the major pains of web development: Rendering markup in two different ways on the server and client side.
Trouble on our first prototype
Like many companies, we initially wanted to adopt technologies already familiar to the rest of our engineers. Python was our language of choice, therefore we built our web server on the Python Flask Framework and rendered templates on the server side with Jinja2. Any kind of dynamic user interface is then sprinkled on the front end with Backbone.js.
This approach served us well for a while and we shipped a working website into the wild. As the source code grew, we started to have these headaches:
Our web server has mixed concerns of api duties, caching, and rendering of the user interface (UI).
Often, when we require a bit of client-side routing (for a snappier interface), we ended up writing the same type of controllers and templates on Flask and Backbone.js. Writing views, controllers and filters in different languages and frameworks is unsurprisingly error prone!
Bootstrapping state from the server side to client side is not straightforward. The server should create the initial UI, and the frontend should take over and attach event handlers. This process is made more challenging when we have two different ways of rendering UI on the server side and client side.
Too much context switching because of the different languages and frameworks involved.
Naturally, we sought to unify the duties of rendering UI under a single language and framework to solve these problems. This forms a UI layer that consistently produce the same markup on the server and client side.
Core to this design is a routing layer that maps pages to different routes. Each page request may fire a data fetching action and the results are then used to prerender markup on the server side. All navigation and interactions after the first page behaves like a Single Page Application (SPA), accumulating state through more data fetching and mutating markup on the fly.
The entire UI layer is designed to be stateless on the server side, fetching all required data from an external service and dehydrating them to the client side. This allows us to scale up the UI layer without worrying about inconsistencies in state.
The existing web server then becomes a cache layer for the UI layer to retrieve data from, and will periodically fetch new data from the database layer.
1. Selecting a common rendering framework
A big part of the design hinges on the ability of the rendering framework. After much research, we chose React for the following reasons:
Deterministic way of getting the same UI given the same state and markup.
Automatic rebuilding of UI under the hood with performance optimizations to prevent unnecessary DOM redraw.
Declarative way of writing UI, versus the imperative style of our Backbone.js code.
2. Trying out React on a small scale
Initially, we were unsure about React so we wanted to try it on something small before commiting more effort to it. We rolled out React on a simple location filter for the Search page.
The new React component was built on top of the existing Flask + Backbone stack. This is one of the nice things of working with React since it plays well with any existing frontend stack. The only drawback which we had to accept was additional page weight of React, which amounts to a whopping 133KB!
At first, JSX seems strange. Some say it looks like an abomination. We have been building web applications in MVC style for a long time, neatly separating out views from logic. Surely JSX is bad?
Maybe, but React is all about composing reusable UI. React forces us to think of composing a page as a tree of smaller components. Each component then contains markup neatly packaged with interactions on the markup. This is a good thing for us. A new hire (or a designer) could view a component and easily tell not only how it looks like, but also how it behaves!
3. Building a whole page with React
Without tearing down our existing Flask + Backbone stack, we built a Sell page with React on Flask like how one might do it SPA-style. We rendered a barebones
<body />tag, sent down scripts with React code, and allowed the client to build the sell page.
This time, the task was more complex since there are more interactions involved and use of 3rd party libraries for image editing. React comes with several lifecycle hooks that we eventually used to add initialization code for all our 3rd party widgets / libraries. Fun fact: Be careful of 3rd party libraries that clone DOM nodes. As part of its own book-keeping, React assigns a unique
react-idto DOM nodes so it gets especially mad when it finds multiple DOM nodes with the same
React code tends to become lengthy due to mixing interactions with markup. To keep ourselves sane, we made use of React components’ composability and wrote many micro components and assembled them together to form the sell page. Many of these components could also be reused elsewhere in the source code (after a teeny bit of refactoring ;] )
After breaking some sweat, we learnt some of the framework quirks and love how easy it is to build complex UI with React. Components are declared with how they should look like under different states, versus writing instructions on how to manipulate the DOM to get our desired UI. This was definitely a great productivity boost compared to other approaches of web development.
With the UI rendering framework out of the way, it was finally time to try out a pared-down version of the final design.
We built a simple Node.js application serving a small page, the Login page. This page was rewritten from the Flask side to the Node.js side. The complexity was small enough for us to ensure the following is working:
Shared code to render UI
LoginPagecomponent code can be used on the server and client side to build the UI. React handles the entire lifecycle of the UI from the server to the client. When the
LoginPageis built on the server, it does not rebuild the page again on the client side and simply attaches the event handlers.
Simultaneously keeping 2 versions of the web in production
Using a load balancer, we whitelisted certain routes (in our case, the login route) to be directed to our Node.js server, and directed the rest of the traffic to the old Flask server.
Stateless UI rendering server
In the “Design” section, we discussed how our UI layer gets data from an external service. We were able to test this design by building a stateless Node.js server that talks to the Flask server and proxies data from Flask to the browser.
This allowed us to authenticate a user on the Login page and safely redirect the user to other pages, authenticated as though they had logged in on the old login page served by the Flask + Backbone stack.
5. Going full scale
Using the same technique of whitelisting routes, we can safely migrate existing features and new features to the UI layer. Some of the old JSX code that were written on top of the Flask + Backbone stack were easy to port over to the new stack.
We have built Carousell Web with most of what I’ve described. Developing the web application is no longer as painful as it was. We have:
Component-driven development with UI as a function of states. This leads to a lot of predictability and ease of writing complex UI.
UI written in the same language meant consistency and no context switching.
The same server-side rendering that allows search engines to index our links easily and fast content load for the first page since the entire page is prerendered on the server side.
The same client-side rendering that gives SPAs its fast page transitions.
Not a Silver Bullet
Nonetheless, here are some tips to deal with the rendering speed:
Clustering more Node.js processes to take advantage of multi-core servers. Additional processes are capable of handling more rendering jobs so React does not starve the event loops from handling requests.
Caching React components. In sites where there is very little user-specific pages, you may cache React components (or even fully-formed html) in your favorite caches (Redis, Memcached, Varnish, etc.)
Using optimized build of React on server side. React comes with a lot of helpful checks and warnings, which should be turned off on production since they come with additional overhead. This can be done by resolving
require("react")to an optimized build of the library (
react-with-addons.min) on production environment. See this issue for more information.
What began as a hackathon project at Startup Weekend, has now grown into one of the largest marketplace apps in the world since launching our iPhone app in August of 2012 and Android app in January of 2013. For the Carousell team, this is a once in a lifetime opportunity to re-imagine what peer-to-peer e-commerce can become in the era where smartphones are increasingly becoming the main way people access the Internet.
Though Carousell doesn’t move petabytes of data (yet), we facilitate large amount of transactions and are facing many of the challenges that eBay faced back in the early days. This post is for the software engineers who are curious about the types of challenges that we face.
Here are some illustrations of the problems we face and how we are tackling them:
Chat and Communication
When we first built Carousell, we started off with a basic implementation of the existing chat system to determine if it would be a popular feature among our users. Chat messages are stored in PostgreSQL databases and then later processed using Celery with RabbitMQ, allowing us to cope with the huge growth in the number of chat messages exchanged between our users. Today, millions of chat messages are exchanged between our buyers and sellers. The reliability and scalability of the chat system has become an important task for our engineering team. Our immediate plan is to introduce near real-time support to our chat system to improve the user experience for our users.
Search and Discovery
To date, the Carousell community has listed over 8 million new and preloved items for sale in categories ranging from fashion and beauty to even baby products. Search is the most used feature in Carousell to find and discover new items to buy within the user’s specified budget. Our current search backend is powered by a cluster of Elasticsearch servers using Haystack, a Django library to simplify access to Elasticsearch. We chose Elasticsearch as it was relatively easy to setup and provides spatial search capabilities out of the box, allowing users to find items on sale near their current location. Elasticsearch’s built-in replication capabilities also allows us to easily scale up to increased search demand by using zen discovery. New nodes can be added to our Elasticsearch cluster to scale up painlessly with no visible downtime for the users.
Some of our upcoming plans to improve Search and Discovery are to increase the relevancy in search results based on individual tastes, purchase histories and current location, enable real-time search alerts of new items to users who are actively looking for the best deals and also to improve our Popular algorithm to make it easy for our users to discover trending items.
Trust and Safety
With hundreds of thousands of transactions and growing happening at Carousell every month, building a trusted and safe marketplace for buyers and sellers is one of our top priorities at Carousell. Our engineers work closely with our Support and Community teams to build product features and support tools. Feedback from our Support team give us insights into user behaviours on the app as well as insights on how users are reacting to new features that we have built for them. Some of our upcoming plans are to develop a system to proactively identify patterns and suspicious behavior to mitigate fraud and scam cases by building an effective verification, feedback and reputation system to ensure a trusted and safe marketplace. With a growing user base, we will also work with the support team to streamline their work and prioritization of the different types of support tickets together so we can get to our users enquiries in a shorter amount of time. User acquisition, retention and virality.
Building a successful marketplace in every new country that we’re expanding into requires us to find and engineer scalable and repeatable ways to acquire new users, retain existing users, and most importantly help our users to become successful buyers and sellers. This requires us to design our product around the different stages of a user lifecycle: Acquisition, Activation, Retention, Referral, (Revenue); and to tailor our product to users from different countries while keeping in mind of cultures nuances and practices. We’re still in the very early days of our international growth and some of the exciting plans that we have for growth are to scale our user acquisition strategy through SEO, virality and referral system; run multivariate tests to maximize the impact of features; revamp our logging platform with technologies such as Kafka, HDFS, Hive and Presto.
One of our upcoming plans is to integrate payment systems to support transactions happening at Carousell. To date, over 2 million items have been sold on Carousell. As we expand internationally, the next challenge for our engineering team is to seamlessly integrate the various international and local payment providers into the buying experience. We want to provide the simplest way for users to transact by supporting as many payment methods, such as credit cards, PayPal, bank transfers, and so on. In order to ensure the trust and safety of transactions, we’ll also need to implement a secure escrow system to handle the holding and releasing of funds upon successful transactions. Interested in working on these challenges?
As Carousell continues to grow, we will be working on the challenges mentioned above including new technical challenges that will manifest themselves over time. If you are interested in working on these exciting challenges in our engineering team, we will be happy to chat with you over coffee to learn more about you. Drop us a note or check out our job openings. We’d love to hear from you :)