Consolidating a three-headed e-commerce platform into one multi-store system

PROBLEM

The retailer's e-commerce platform had been built for a single home market and grown over the years, feature by feature, into something it was never meant to be: a three-headed beast.

The original platform held the product and pricing data but only served the home market. Another platform served multiple foreign markets and held their tax logic, currency formatting and other localisations but depended on the original platform for the source data. Yet another platform served only a single foreign market and was a completely different and separated platform altogether.

Added difficulties were that the platform was started from a CMS but was so extensively modified through the years that almost nothing of it was left. Almost every component and feature had either been stripped or had evolved into a custom-made version, down to a completely bespoke database abstraction layer. An initial push to reduce the tech debt had already been made by partially introducing Symfony, but the effort had stalled.

That is the point where I came into the team, with a clear objective: consolidating all three platforms into the original one and removing as much tech debt as possible while doing so.

CONSTRAINTS

• A rewrite was off the table: the platform was too large and too heavily customised to replace, and the internal team had committed to investing in the original platform instead. Every improvement had to land inside the codebase as it stood.
• All three platforms had to keep selling: the consolidation ran alongside normal trading. Each platform kept serving its own markets throughout, so nothing could go dark and no step could depend on a big-bang cutover. Performance across all stores was closely monitored by the data science team, so any negative impact would have been noticed immediately.
• End-of-year season: Black Friday, Sinterklaas, Christmas and New Year all fall within roughly one month, which is the absolute busiest and most important period of the year. Because of this, a company-wide code freeze comes into effect from one week before Black Friday until the first week of January. Over a two-year project that meant two freeze windows, roughly three months in which nothing shipped at all.

APPROACH

Workflow

From the start of the project, we agreed on keeping releases as small as possible and releasing them often. This allowed us to have each change running in production for a while and spot any issues that made it past the development cycle, before we migrated the first store. It also meant we were never sitting on a large, half-finished change when the freeze came into effect.

Foundation

As I was settling in during the first few weeks, I completed the integration of Symfony, updating it to the latest LTS version in the process. During this effort, I learned the codebase inside and out and a few key decisions were made about the architecture of the project:

• Dependency injection: a remaining separate container/dependency injection mechanism was initially kept, until we'd refactored enough areas to replace it with Symfony's container.
• Database layer: the fully custom-built database layer was also kept, because it did its job well. Migrating to Doctrine would have cost too much time for too little benefit, as it would have meant moving from the custom layer's Active Record pattern to Doctrine's Data Mapper pattern.

Routing

The legacy codebase itself was a challenge along every step of the way, because of the high degree of customisation that had been introduced over the years. One of those challenges was the routing. This was one of the few components of the original framework that was still mostly intact.

As the entire codebase and content was built on top of this system, we didn't want to overhaul it entirely and instead opted to implement the symfony-cmf/routing-bundle's ChainRouter:

cmf_routing:
  chain:
    routers_by_id:
      cmf_routing.dynamic_router:       300
      router.default:                   200
      App\Routing\LegacyDatabaseRouter: 100
  dynamic:
    enabled: true
    route_provider_service_id: 'App\Routing\TranslatableRouteProvider'
    persistence:
      orm:
        enabled: true
        route_class: App\Entity\TranslatableRoute

The ChainRouter was set up so that the new translatable routes would always come first, allowing us to override any existing URLs. After that, the Symfony router got a chance to match a route before the original routing engine (wrapped in a facade to play nice with Symfony) handled the legacy URLs and pages or returned a 404 if nothing would match.

Translations

Due to the project's size, another challenge (albeit a smaller one) proved to be discovering all the places where translations were needed. Various templates and all the variants of them were tedious to discover, but more elusive were all the places where images were shown, as most of them needed to be able to display a different version for each translation.

A combination of scripts and dumps, fallback logging, and manual hunting backed by reports from users testing on acceptance was needed to surface all of the translations.

Product search

A non-codebase related challenge was getting a decision made by management about the implementation path of the overhauled product search functionality. The choice was whether we would set up our own Elasticsearch infrastructure or use a hosted Elasticsearch provider. It dragged on for six months before finally settling on running the in-house infrastructure option.

This ended up severely delaying the product search and the migration of the first store, as the infrastructure needed to be set up first.

RESULT

Over the span of about two years, we overhauled a large part of the existing codebase implementing a new routing infrastructure supporting the multi-store and multilingual capability needed to migrate the stores served on the sister platforms back into the original one.

Despite the delay because of the dragging decision, a new product search was also implemented before the first store migrated.

Shortly after I moved to another project, the first storefront was successfully migrated to the upgraded original platform.