Technical Accomplishments
Astronomical Data Systems
Example E2E System Diagram
Data Management & Curation: Data from most ground-based telescopes are not collected with the intent of providing public access though an archive. Even for the National Optical Astronomy Observatory, which has a public archive of raw and (for some instruments, reduced) data, shortcomings in the legacy data taking systems for all but the most recent few instruments greatly complicate any effort to manage the heterogenious, incomplete, and sometimes inaccurate metadata. Since the representation to the community of the Archive holdings via common searches, and the appropriate scientific interpretation of the datasets, requires accurate metadata, I designed a system of Data Quality Assurance and data remediation for core metadata (see my paper from ADASS 2008 on this topic). Here, “core” means metadata that describe data provenance and pedigree, as well as coverages in the VO-sense: spatial, spectral/bandpass, temporal, and brightness; see my technical report for details. With this restricted set it is possible to validate metadata, and in many cases correct them, using a variety of cross-checks, heuristics, third-party calibration software, etc.
LSST telescope design
LSST: To this day, data systems and archives in even the most sophisticated astronomical observatories are under-appreciated, yet they are becoming more critical to the scientific success of the observatories and the communities they serve. On no project is this more true than the Large Synoptic Survey Telescope, now under construction, which will survey the entire accessible sky roughly once every 3 nights, to a depth of roughly 25th mag. Many Tera-bytes of imaging data will be generated per night (a data volume comparable to the entire Sloan Digital Sky Survey), which will result in several Peta-bytes per year of data, catalogs, and database contents. The bulk of the data management, transport, reduction, calibration, and analysis will have to be fully automated if this project is to succeed.
I participated on the LSST Data Managemetent Design & Development Team, which developed detailed science and technical requirements, prototyped portions of the system (those deemed to be high risk), and which processed simulated and legacy astronomical datasets to evaluate the algorithnic techniques and scientific fidelity of the prototype. This system will process all of the science data, enable evaluation of the observing strategy, the scientific quality of the accumulated data, and measure progress against the goals of the 10 yr survey.
ADASS
ADASS 2006 in Tucson, AZ
I’ve been deeply involved in community efforts to encourage best practices in astronomical data system design and implementation. I spent 15 years on the Program Organizing Committee for the Astronomical Data Analysis Software and Systems (ADASS) conference series, which is the premiere forum for software-intensive systems in astronomy; I served as chair of that group for 4 years. I also chaired the Local Organizing Committee when ADASS XVI was held in Tucson.
Astronomical Data Standards
With the explosive growth in digital astronomical data, the importance of standards for data interchange and interoperability is greatly magnified. I am a member of the IAU interest group (formerly, Technical Working Group) for the development and curation of the FITS (Flexible Image Transport System) standard. From 2006—2008 I was part of a small technical panel to edit and update the standard to FITS v3.0, which was a more up-to-date and readable document than its predecessor. I continued my work with the FITS Technical Working Group from 2014–2016 to fold into the Standard new features (the representation of Time world coordinates), and various of the FITS conventions that have proven successful over time, such as representations of compressed data. The new version (v4.0) is now in final draft form.
FITS is not the end of the story, of course. Standards for the serialization and exchange of structured data are in an advanced state of development within the International Virtual Observatory Alliance. Structured data, as embodied for instance in VOTables, are routinely by many VO-aware applications in use today. And formalisms for the representation of semantic content (in machine-interpretable form) are in development within the VO and larger scientific communities. I have maintained an ongoing interest in the definition of data and service standards, and expect to participate in their development and use in the years to come.