One common scenario in large enterprise systems is esoteric/inconsistently structured data. This data is crucial to a firm’s success, but cannot be easily read, analyzed or extracted. The data might not have a schema and might only exist in memory. An example of this is C++ code that has classes with strange and inconsistent interfaces which do not have fast human-readable serializations. Programmers are stuck when needing to test and analyze this data. A better solution would be to migrate the data to a common schema-based format and use Python data science libraries to analyze it.

Python (with clang) and Apache Arrow enables you to quickly and easily transform data into the Apache Parquet format, where you can use PyArrow and pandas to analyze it. Attendees will learn how to:

1. Use PyArrow to read CSV, JSON, custom data and hierarchical data.
2. Use pandas to elegantly compare data.
3. Parse their C++ classes using the cindex module and extract all the relevant data accessors and generate a “schema.”
4. Use the schema with a Jinja template to convert from C++ to Parquet using the Apache Arrow C++ API.

Outline

Problem statement

• Go over the problem and show what esoteric data is.

Pandas

• Review pandas role in comparing esoteric data.

Free Data!!

Apache Arrow Overview

• Go over examples of PyArrow and why you want to use Apache Arrow.
  • How to read CSV, JSON, custom and hierarchical data and save it to Parquet.

Put Clang, Jinja, Arrow and Python together to access your data:

• Review a specific example of how to parse C++ classes using Python’s cindex module
• Create the schema from the parsing
• Generate the converter using Jinja templates
  • convert
• Show details of the generated Parquet file