Data applications are no different from regular software and accrue technical debt like all applications over time. They frequently receive little updates because “they work” and continue to provide business value. Over time, they end up being viewed as black boxes for producing valuable data. There’s often an inherent tension between users of the data application who are primarily concerned with continuing to produce data reliably in the short term and the teams responsible for the evolution of the data application who are concerned with adapting it to gracefully serve the business as it scales. We will discuss moving out such an application based on a case study of an application that we have migrated.

We cover the following key regions

• Should you even do this?
  • Where are you migrating from? Where to?
  • What do you gain in the new world? What do your customers gain?
  • Build it from scratch? (every developer’s tendency)
  • Do you feel good about owning the system in its current state?
• Defining a project plan
  • How do you break into phases? What does success look like for each?
• Automating as much as you can
• Showing that this will work
  • Assuaging fears and messaging
  • Harvesting production for tests.
• Doing this without disrupting business operations
  • Getting buy-in from customers
  • Offering “carrots” along the way

This talk will be relevant for members of teams who own such applications, but the lessons learnt can help developers avoid building up unmaintainable software. Familiarity with any existing tools or software is not required for this talk.