Hubble Lakehouse — Big Data Architecture & Space Weather
Medallion Big Data pipeline, batch and streaming: Hubble telescope observations joined with NOAA space weather, served as 4 business datamarts.
- Client
- M1 Data Engineering & AI academic project - EFREI Paris
- Year
- 2026
- Expertise
- Data & AI
- Services
- Data Engineering, Lakehouse Architecture, Streaming, DevOps
- Tools
- Apache Spark
- Airflow
- HDFS
- Kubernetes
- Power BI
Context
Design an end-to-end Lakehouse architecture to ingest, clean and expose heterogeneous data: a massive history of more than 30 years of observations and a continuous space weather feed.
Our approach
A Big Data pipeline (Medallion architecture) processes, in batch and streaming, the metadata of Hubble telescope observations joined with space weather (NOAA), to feed four business datamarts.
Physicist Datamart
Sky mapping to identify over-observed regions and the blind spots of research.
Chemist Datamart
Spectrographic analysis of narrow-band filters (hydrogen, oxygen, sulphur).
Biologist Datamart
Detection of observations corrupted by solar flares (NOAA space weather).
Engineer Datamart
Maintenance tracking and how instrument calibration times evolve over time.
Deliverables
A fully working end-to-end Big Data pipeline: source code versioned on GitHub, a JWT-secured API, a video demo of the data flows and a Power BI decision report split into four business pages.