Breaking Biology's Data Wall: Scaling a Multi-disciplinary Data Platform at Basecamp Research

In this session, Keith Kam from Basecamp Research shares practical insights from developing BaseData™, the world's most diverse metagenomic sequence database designed to power the next generation of bio AI foundation models.

In this session, Keith Kam from Basecamp Research shares practical insights from developing BaseData™, the world's most diverse metagenomic sequence database designed to power the next generation of bio AI foundation models. Building a data platform that simultaneously supports bioinformatics, analytics, and machine learning teams presents significant engineering challenges. We'll discuss key lessons learned in managing complexity across multiple dimensions:

Tooling Complexity: Managing diverse scientific tools ranging from simple Python packages to complex GPU-enabled environments across heterogeneous computing requirements. Data Complexity: Scaling development velocity while preventing data quality issues across varied bioinformatics file formats and evolving tabular data structures. Operational Complexity: Handling diverse operational patterns for large datasets including incremental loads, point-in-time snapshots, and complex backfill scenarios.This session provides practical guidance for data engineers and scientific computing teams navigating the complexities of scaling heterogeneous data platforms.