PyData Washington DC 2018 - Presentation: Avoiding Bad Database Surprises: Simulation and Scalability

Avoiding Bad Database Surprises: Simulation and Scalability

Audience level:

Novice

Description

There are many stories of developers creating databases that don't operate at scale. The application is good, but the database won't work the realistic volumes of data. It's like a horror movie where they never looked behind the door, ran into the dark forest and night, and discovered the database was the monster killing their application. How can we leverage Python to avoid scaling problems?

Abstract

Background
Some Horror Stories
The results of profiling
The Horror Movie Trope: there's a monster, and it's in your base, and it's killing your app
While you can slay the monster, it would be so much easier to avoid it
Simulate Early and Often
Using Python
Data Model Definitions and ORM
JSON Schema definitions
A metaclass for data objects
A generator
Building collections of objects
Handling Data Domains -- more than simple text
Handling JSON Schema formats -- example dates
Handling Account Numbers, Names and Addresses
specialized generators
Recap
Don't hope or assume
Look at the horror movie tropes
Simulate early and often

Saturday 11:45 AM–12:30 PM in Production DS, Data - Auditorium

Avoiding Bad Database Surprises: Simulation and Scalability

Steven Lott

Description

Abstract

Subscribe to Receive PyData Updates