In the following case study, we’ll explore how Spartan Spark was able to successfully bridge the gap between a 15-year-old manually-scaled system and automate scalability to support upgrading a platform with a large database - Spartan Spark's marketing automation platform, SpartanOne. An additional challenge was to determine the best way to upgrade this complex database to current technology so existing clients using the platform could continue to grow their online presence and lead generation efforts.
Each time the database was emailed, a large volume of visitors came to the website, resulting in downtimes to their existing infrastructure, among a host of other problems that impeded the client from growing their online presence. To support larger numbers of clients, the system needed to scale automatically under higher volume use when more resources were called for. This also had to happen at a reasonable cost.
This methodology applies to a wide variety of businesses and digital marketing activities, making it a cost-effective way to integrate various software applications that need to work together to increase scalability. The methodology described will ultimately allow for online marketers to create systems that can support a large number of clients when required, and scale down to save on server costs during low-volume periods.
We sought to upgrade SpartanOne’s business services platform to improve performance, scalability, and availability. Modern standards had progressed exponentially in recent years demanding a change to meet current technological standards.
A major risk was that many of the technologies Spartan Spark considered were so new, documentation and community support were limited. Node JS is a prime example. It required extensive testing to ensure we would not be moving our customers to an unstable foundation.
No SQL technologies each have their own set of features. Identifying the right implementation was challenging. We also faced technological risk maintaining data integrity migrating My SQLto NoSQL, with many features unsupported by NoSQL. This required processes to ensure data integrity and validity.
A further technological risk was related to combining different software tools and their compatibility within our programming schema. A misstep in decisions would cost us in terms of the achievement of performance objectives as well as initial software development and longer term life-cycle support of the code. We needed to ensure best practices were established and maintained across project iterations.
Spartan Spark’s methodology needed to ask a host of high-value questions:
The dynamic nature of the platform proved to be unstable. This led to the development of a Content Management System (CMS), and revision of our Release Management process to allow non-technical people in the company to deploy the entire system.
Spartan Spark made great progress toward these goals by creating a Node JS-based framework extending ExpressJS, and used that framework to deliver the key service required for our Service Orientated Architecture.
We began our systematic investigative process by documenting all system properties, performance criteria, and inter-relationships between data sets resident on multiple platforms.
Being event-driven, Node JS is prone to order of operation errors and its heavy use of callback functions tends toward tight coupling, making troubleshooting difficult. To overcome these challenges, we employed Promises, a decoupling design pattern for easier readability and maintenance. We evaluated jQuery, Promise polyfill, Catline, and bluebird. jQuery was familiar but was not optimized for server-based implementation. Testing showed other options similar in performance but bluebird documentation addressed our use cases. We succeeded in decoupling Node JS so we could eliminate operation errors.
We next addressed the Object Relational Model (ORM) for mapping data requests to the datastore. ORM is complicated and not something we wanted to tackle ourselves. We tested orm-node2, sequelize, bookshelf, orm, and Waterline. Needing to support both No SQL and SQL with table joins, two options failed this requirement. Further testing showed orm-node2 was the best candidate, but its cache implementation was buggy using custom index keys. We worked with the developer to fix these issues but continue to run with caching disabled, which has significant performance costs. We implemented our own caching of server responses to mitigate this. We tested memcached, Redis, and to a lesser degree, DynamoDB. We chose Redis because its driver supported JSON natively, providing major performance gains.
Single and chained queries performance were tested. At low loads, performance difference between My SQL and No SQL was minimal, but the simplicity of single queries versus five queries to glue a user together still made No SQL the right choice. Considerable effort was required to make SQL comparable to No SQL in high load simulations. Although No SQL eliminates typical bottlenecks, its lack of validation requires careful data handling.
We moved to testing data conversion and discovered how dependent we were on standard SQL for enforcing data consistency, something No SQL lacks. Things looked good when restarting the system after every change, but server stability revealed small value differences in the datastore and logs that were hard to reproduce. Our My SQL schema used many ENUM(eration) to enforce specific values and capitalization still returns corrected normalized values. For updates and inserts, No SQL does not normalize so comparisons and matching fail. We invested significant effort ensuring our code was either tolerant to this or enforced specific values.
We hoped to reuse PHP Smarty templates but decided they could only be a guideline. We compared Jade with Mustache and EJS. The last two were too capable, tempting developers to use too much logic in the template instead of the business layer. Underscore works for non HTML use cases because of its speed but lacks features tailored for HTML. After testing we concluded that Jade worked for both full page HTML and Web 2.0, with its minimal expressive syntax and limited programmability.
Our findings during our ORM investigations led to our testing a number of helper libraries. None of these were found adequate so we chose instead to wrap an existing form library, extending it to meet our requirements. Once complete, we reinforced the validation in the ORM layer.
At this point we began exploring application deployment. We needed an automated way to minimize the operational costs of deploying iterations often, and scale horizontally to new clients. We were able to leverage Amazon Web Services (AWS) from deployment to disaster recovery, with a platform that scales automatically based on load. Step by step we replaced 10 years of operational development with new technologies through a cloud-based service oriented architecture (SOA), reducing monthly operational costs while increasing up-time, system performance, and availability, all with a model that does not see massive up-front hardware costs for machines that used to sit mostly idle. By utilizing SOA, we are able to scale individual parts of the application without impacting others.
Unit testing, functional testing, and scalability testing are all critical to have any ability to anticipate problems before they happen.
We were also able to modernize the platform as part of the upgrades to better leverage social media and Single Page Applications.
“After being frustrated with other service providers. Spartan Spark got our website up and linked to a payment system over just a weekend.” -Patricia Marin - Calgary Family Services