Background
The Sparks & Smiles programme is a social impact programme run by GIC that aims to empower Singaporean youths from high-needs households to be changemakers who want to pay it forward in their own communities. Together with GIC’s charity partner Beyond Social Services, students are provided with mentorship training and are connected to a disadvantaged child, youth or family. They are also given opportunities to engage with GIC senior leaders and receive a grant to support their studies.
The Sparks system is a workflow management system that alleviates the administrative overhead of running the GIC Sparks & Smiles programme. Prior to the system being built, the staff from Beyond Social Services had to keep track of the progress of each of their mentees manually. With the Sparks system, all of the data for the Sparks & Smiles programme is centralised in one place. In addition, it also makes the workflow of the Sparks & Smiles program seamless, offering email notifications and reminders to the relevant parties, be it students, school staff, or Beyond Social Services staff, as actions are carried out within the system.
Objectives
1. Redesigning the Application Architecture to Address Slow Load Times
Over the past two years, the Sparks system experienced a gradual slowdown in its loading times. After investigating, we found out that this was due to the application architecture handling data in an inefficient manner. In order to tackle this issue, we had to redesign the entire application architecture to minimise the amount of data being transferred between the frontend and the backend. This involved rewriting a large portion of the codebase to adhere to our redesigned architecture.
After deploying our rewritten application, the Sparks system has seen tremendous improvements in loading times. For context, the slowest part of the system (the dashboard), has seen a roughly 93x increase in load speeds, going from over 35 seconds on the old system, to under half a second on the new system.
2. Minor Enhancements & Bug Fixing
We implemented several minor enhancements which were requested by GIC and Beyond Social Services. These include the ability to recommend students for a renewal of the GIC Sparks & Smiles award, and hiding deactivated Beyond Social Services staff from new deployments, among other things. In addition, we also relooked the workflow of the system to ensure that the email triggers made sense.
On top of that, we fixed all of the bugs which were reported over the past year. This was made possible by our architecture redesign, which resulted in a much less coupled system, allowing us to achieve better separation of concerns. As a result, we were able to easily trace the source of bugs and correct them. At the end of the summer, we had resolved all outstanding issues accumulated since the inception of the project.
3. Fully Containerising the Application
While the Sparks system ran in Docker containers, not every aspect of the application was containerised. This presented a significant challenge in deploying the application onto a different host as things such as the backup cron job and SSL certificates had to be manually set up, making them prone to error.
To counter this issue, we fully embraced the practice of Infrastructure as Code (IaC). This not only allowed us to manage and provision our deployment infrastructure automatically, it also eliminated the configuration drift that is inherent in sufficiently complex manual setup processes. By having all infrastructure-related configurations and scripts version controlled in a repository, this also ensures the transfer of knowledge between CVWO batches.
In addition, we improved the format of the database backups to make use of SQL dumps instead. This allows the database backups to be platform and configuration agnostic as opposed to the previous practice of backing up the raw files on disk.
Process
Investigation of Slow Load Times
Over the past year, Beyond Social Services feedbacked that load times for the Sparks system were getting increasingly longer. As such, our first priority for the summer was to investigate the cause of the slowdown. This led us to discover that the system was poorly designed. Specifically, most of the application logic was being handled in the frontend instead of the backend. This resulted in the backend querying and sending huge amounts of unnecessary data to the frontend. As the amount of data in the system grew, so did the load times.
Application Rewrite
In order to rectify the issue discovered above, we had no choice but to rewrite large portions of the application. Because we had no idea as to the extent of the issue prior to the summer, we had to quickly design an updated application architecture and devise an execution plan within a short timespan. This was particularly challenging as we had to simultaneously familiarise the team with CVWO’s processes as well as handle the building of the Active Ageing Centres system.
The main obstacle we faced while rewriting the application was that the entire codebase was tightly coupled. This meant that a change made in one part of the application had a ripple effect on other parts. As a result, we faced much difficulty in rewriting the application in parallel, being forced to coordinate changes to prevent conflicts in the code which slowed down our progress. During this process, we took the opportunity to enforce proper separation of concerns within the code so as to make the system easier to reason about.
Other than drastically reducing the application load times, the system rewrite has made the application more extensible and robust. This will make the future implementation of features much easier, while reducing the likelihood of regressions being introduced into the system.
Screenshots
Team
- Ian Yong (Project Lead, Year 2)
- Han Geng Ning (Developer, Year 1)
- Jovyn Tan (Developer, Year 1)
- Sivayogasubramanian (Developer, Year 1)
Afterthoughts
Being able to play an active role in training up the software engineering capabilities of everyone in the team has been immensely rewarding. I hope that their experience in CVWO has given them a greater appreciation for the intricacies of software development.
– Ian
CVWO opened my eyes to the complexity and beauty of software engineering. I’m thankful to be able to pick up so many technical and interpersonal skills while working with an inspiring team ( ´ ▽ ` )
– Geng Ning
CVWO was an intense learning experience and I’m grateful for the trust and responsibility given to us as developers!
– Jovyn
It has been an exciting experience for me here at CVWO over the summer. I am grateful for this opportunity to serve the community whilst also learning the technical skills needed for a software engineering project. I am also thankful to my team leads and teammates for helping one another when times were tough.
– Siva
We would like to express our heartfelt gratitude to the following people who have been influential and supportive during our project:
- Prof Ben Leong (NUS) for his advice and guidance.
- GIC for their generous support of the CVWO programme.
- Chiu Ying Yik (Beyond Social Services) for her co-operation and support throughout the project.