ALeRCE broker: current status and demo · Javier Arredondo, Ernesto Castillo, Francisco Forster,...
Transcript of ALeRCE broker: current status and demo · Javier Arredondo, Ernesto Castillo, Francisco Forster,...
![Page 1: ALeRCE broker: current status and demo · Javier Arredondo, Ernesto Castillo, Francisco Forster, Diego Rodríguez, Daniela Ruz, Ignacio Reyes, Camilo Valenzuela. ALeRCE infrastructure](https://reader033.fdocuments.net/reader033/viewer/2022042409/5f24d90a0ac77c2e420ef8e6/html5/thumbnails/1.jpg)
ALeRCE broker: current status and demo
![Page 2: ALeRCE broker: current status and demo · Javier Arredondo, Ernesto Castillo, Francisco Forster, Diego Rodríguez, Daniela Ruz, Ignacio Reyes, Camilo Valenzuela. ALeRCE infrastructure](https://reader033.fdocuments.net/reader033/viewer/2022042409/5f24d90a0ac77c2e420ef8e6/html5/thumbnails/2.jpg)
Outline
● Introduction● Goals● Team / Methodology● Infrastructure● Machine Learning
● Demo○ Web Interfaces○ API○ Jupyter notebooks
![Page 3: ALeRCE broker: current status and demo · Javier Arredondo, Ernesto Castillo, Francisco Forster, Diego Rodríguez, Daniela Ruz, Ignacio Reyes, Camilo Valenzuela. ALeRCE infrastructure](https://reader033.fdocuments.net/reader033/viewer/2022042409/5f24d90a0ac77c2e420ef8e6/html5/thumbnails/3.jpg)
Future time domain astronomy ecosystem
Survey telescopes
Alert brokers/TOMs
Follow up telescopes
+ other brokers/TOMs
![Page 4: ALeRCE broker: current status and demo · Javier Arredondo, Ernesto Castillo, Francisco Forster, Diego Rodríguez, Daniela Ruz, Ignacio Reyes, Camilo Valenzuela. ALeRCE infrastructure](https://reader033.fdocuments.net/reader033/viewer/2022042409/5f24d90a0ac77c2e420ef8e6/html5/thumbnails/4.jpg)
Astronomical infrastructure in Chile
TAOE-ELT
LSST
ALMA
SOAR
La Silla
CTIO
VLT
CTAGMT
CCAT
Magellan
VISTA
Gemini
![Page 5: ALeRCE broker: current status and demo · Javier Arredondo, Ernesto Castillo, Francisco Forster, Diego Rodríguez, Daniela Ruz, Ignacio Reyes, Camilo Valenzuela. ALeRCE infrastructure](https://reader033.fdocuments.net/reader033/viewer/2022042409/5f24d90a0ac77c2e420ef8e6/html5/thumbnails/5.jpg)
![Page 6: ALeRCE broker: current status and demo · Javier Arredondo, Ernesto Castillo, Francisco Forster, Diego Rodríguez, Daniela Ruz, Ignacio Reyes, Camilo Valenzuela. ALeRCE infrastructure](https://reader033.fdocuments.net/reader033/viewer/2022042409/5f24d90a0ac77c2e420ef8e6/html5/thumbnails/6.jpg)
ALeRCE is a Chilean-led initiative to build a community broker for LSST and other large etendue
survey telescopes
![Page 7: ALeRCE broker: current status and demo · Javier Arredondo, Ernesto Castillo, Francisco Forster, Diego Rodríguez, Daniela Ruz, Ignacio Reyes, Camilo Valenzuela. ALeRCE infrastructure](https://reader033.fdocuments.net/reader033/viewer/2022042409/5f24d90a0ac77c2e420ef8e6/html5/thumbnails/7.jpg)
Goals
To facilitate the study of non-moving, variable and transients objects:
● Fast classification of transients, variable stars and active galactic nuclei
● Flexibility to adapt to different science cases (taxonomy, data products)
● Connect survey and follow up resources in Chile and abroad
![Page 8: ALeRCE broker: current status and demo · Javier Arredondo, Ernesto Castillo, Francisco Forster, Diego Rodríguez, Daniela Ruz, Ignacio Reyes, Camilo Valenzuela. ALeRCE infrastructure](https://reader033.fdocuments.net/reader033/viewer/2022042409/5f24d90a0ac77c2e420ef8e6/html5/thumbnails/8.jpg)
Scientific Questions
![Page 9: ALeRCE broker: current status and demo · Javier Arredondo, Ernesto Castillo, Francisco Forster, Diego Rodríguez, Daniela Ruz, Ignacio Reyes, Camilo Valenzuela. ALeRCE infrastructure](https://reader033.fdocuments.net/reader033/viewer/2022042409/5f24d90a0ac77c2e420ef8e6/html5/thumbnails/9.jpg)
ALeRCE Team
![Page 10: ALeRCE broker: current status and demo · Javier Arredondo, Ernesto Castillo, Francisco Forster, Diego Rodríguez, Daniela Ruz, Ignacio Reyes, Camilo Valenzuela. ALeRCE infrastructure](https://reader033.fdocuments.net/reader033/viewer/2022042409/5f24d90a0ac77c2e420ef8e6/html5/thumbnails/10.jpg)
Agile Methodology
● Daily meetings + 2 weeks sprints● Product owner● Kanban / Backlog● Weekly ML/training set meetings
Javier Arredondo, Ernesto Castillo, Francisco Forster, Diego Rodríguez, Daniela Ruz, Ignacio
Reyes, Camilo Valenzuela
![Page 11: ALeRCE broker: current status and demo · Javier Arredondo, Ernesto Castillo, Francisco Forster, Diego Rodríguez, Daniela Ruz, Ignacio Reyes, Camilo Valenzuela. ALeRCE infrastructure](https://reader033.fdocuments.net/reader033/viewer/2022042409/5f24d90a0ac77c2e420ef8e6/html5/thumbnails/11.jpg)
ALeRCE infrastructure
Distributed Storage
Container orchestrator
Distributed messaging
Distributed database
Services
![Page 12: ALeRCE broker: current status and demo · Javier Arredondo, Ernesto Castillo, Francisco Forster, Diego Rodríguez, Daniela Ruz, Ignacio Reyes, Camilo Valenzuela. ALeRCE infrastructure](https://reader033.fdocuments.net/reader033/viewer/2022042409/5f24d90a0ac77c2e420ef8e6/html5/thumbnails/12.jpg)
ALeRCE infrastructure
Distributed Storage
Container orchestrator
Distributed messaging
Distributed database
Services
![Page 13: ALeRCE broker: current status and demo · Javier Arredondo, Ernesto Castillo, Francisco Forster, Diego Rodríguez, Daniela Ruz, Ignacio Reyes, Camilo Valenzuela. ALeRCE infrastructure](https://reader033.fdocuments.net/reader033/viewer/2022042409/5f24d90a0ac77c2e420ef8e6/html5/thumbnails/13.jpg)
ALeRCE pipeline
![Page 14: ALeRCE broker: current status and demo · Javier Arredondo, Ernesto Castillo, Francisco Forster, Diego Rodríguez, Daniela Ruz, Ignacio Reyes, Camilo Valenzuela. ALeRCE infrastructure](https://reader033.fdocuments.net/reader033/viewer/2022042409/5f24d90a0ac77c2e420ef8e6/html5/thumbnails/14.jpg)
ZTF Data Model
![Page 15: ALeRCE broker: current status and demo · Javier Arredondo, Ernesto Castillo, Francisco Forster, Diego Rodríguez, Daniela Ruz, Ignacio Reyes, Camilo Valenzuela. ALeRCE infrastructure](https://reader033.fdocuments.net/reader033/viewer/2022042409/5f24d90a0ac77c2e420ef8e6/html5/thumbnails/15.jpg)
Classification
● Current Classifiers:
○ Convolutional Neural Network (stamps)
○ Random Forest (light curve)
● Future Classifiers:
○ Domain Adaptation○ Data Augmentation○ Outlier Detection○ Forecasting
![Page 16: ALeRCE broker: current status and demo · Javier Arredondo, Ernesto Castillo, Francisco Forster, Diego Rodríguez, Daniela Ruz, Ignacio Reyes, Camilo Valenzuela. ALeRCE infrastructure](https://reader033.fdocuments.net/reader033/viewer/2022042409/5f24d90a0ac77c2e420ef8e6/html5/thumbnails/16.jpg)
Late ClassifierEarly ClassifierConvolutional Neural Network
(using first stamps)Random Forest
(using light curve, at least 5 observations)
Classes: AGN, SN, VS, asteroid, bogusClasses:
- Stochastic: AGN-I, Blazar, CV/Nova- Transients: SNe Ia, SNe Ibc,SNe II,SNe IIn, SLSNe- Periodic stars: EBSD/D, EBC, DSCT, RRL, Ceph, LPV, Periodic Other
![Page 17: ALeRCE broker: current status and demo · Javier Arredondo, Ernesto Castillo, Francisco Forster, Diego Rodríguez, Daniela Ruz, Ignacio Reyes, Camilo Valenzuela. ALeRCE infrastructure](https://reader033.fdocuments.net/reader033/viewer/2022042409/5f24d90a0ac77c2e420ef8e6/html5/thumbnails/17.jpg)
δ
![Page 18: ALeRCE broker: current status and demo · Javier Arredondo, Ernesto Castillo, Francisco Forster, Diego Rodríguez, Daniela Ruz, Ignacio Reyes, Camilo Valenzuela. ALeRCE infrastructure](https://reader033.fdocuments.net/reader033/viewer/2022042409/5f24d90a0ac77c2e420ef8e6/html5/thumbnails/18.jpg)
18
Detection Threshold
Convolutional recurrent neural network
Carrasco-Davis+18PASP
Pclass i (t0)
![Page 19: ALeRCE broker: current status and demo · Javier Arredondo, Ernesto Castillo, Francisco Forster, Diego Rodríguez, Daniela Ruz, Ignacio Reyes, Camilo Valenzuela. ALeRCE infrastructure](https://reader033.fdocuments.net/reader033/viewer/2022042409/5f24d90a0ac77c2e420ef8e6/html5/thumbnails/19.jpg)
19
Detection Threshold
Convolutional recurrent neural network
Carrasco-Davis+18PASP
Pclass i (t0)
![Page 20: ALeRCE broker: current status and demo · Javier Arredondo, Ernesto Castillo, Francisco Forster, Diego Rodríguez, Daniela Ruz, Ignacio Reyes, Camilo Valenzuela. ALeRCE infrastructure](https://reader033.fdocuments.net/reader033/viewer/2022042409/5f24d90a0ac77c2e420ef8e6/html5/thumbnails/20.jpg)
Training vs domain overfitting
Carrasco-Davis+18PASP
![Page 21: ALeRCE broker: current status and demo · Javier Arredondo, Ernesto Castillo, Francisco Forster, Diego Rodríguez, Daniela Ruz, Ignacio Reyes, Camilo Valenzuela. ALeRCE infrastructure](https://reader033.fdocuments.net/reader033/viewer/2022042409/5f24d90a0ac77c2e420ef8e6/html5/thumbnails/21.jpg)
Summary● Goals:
○ Fast classification: transients, variable stars and AGN○ Flexibility to adapt to different science cases○ Connect survey and follow up
● Interdisciplinary team:○ Machine Learning○ Astronomy ○ Computer Science
● Infrastructure:○ Modular / Distributed / Robust
● Machine Learning:○ Classification: CNN / Random Forest○ Outlier Detection○ Domain Adaptation / Data Augmentation○ Forecasting
![Page 23: ALeRCE broker: current status and demo · Javier Arredondo, Ernesto Castillo, Francisco Forster, Diego Rodríguez, Daniela Ruz, Ignacio Reyes, Camilo Valenzuela. ALeRCE infrastructure](https://reader033.fdocuments.net/reader033/viewer/2022042409/5f24d90a0ac77c2e420ef8e6/html5/thumbnails/23.jpg)
Web InterfacesJupyter
Notebooks
TOMs
API
![Page 24: ALeRCE broker: current status and demo · Javier Arredondo, Ernesto Castillo, Francisco Forster, Diego Rodríguez, Daniela Ruz, Ignacio Reyes, Camilo Valenzuela. ALeRCE infrastructure](https://reader033.fdocuments.net/reader033/viewer/2022042409/5f24d90a0ac77c2e420ef8e6/html5/thumbnails/24.jpg)
Web Interfaces
ZTF Explorerhttp://alerce.online
SN Hunterhttp://snhunter.alerce.online
![Page 25: ALeRCE broker: current status and demo · Javier Arredondo, Ernesto Castillo, Francisco Forster, Diego Rodríguez, Daniela Ruz, Ignacio Reyes, Camilo Valenzuela. ALeRCE infrastructure](https://reader033.fdocuments.net/reader033/viewer/2022042409/5f24d90a0ac77c2e420ef8e6/html5/thumbnails/25.jpg)
Mobile Phones
![Page 26: ALeRCE broker: current status and demo · Javier Arredondo, Ernesto Castillo, Francisco Forster, Diego Rodríguez, Daniela Ruz, Ignacio Reyes, Camilo Valenzuela. ALeRCE infrastructure](https://reader033.fdocuments.net/reader033/viewer/2022042409/5f24d90a0ac77c2e420ef8e6/html5/thumbnails/26.jpg)
SNe reported by ALeRCE
![Page 27: ALeRCE broker: current status and demo · Javier Arredondo, Ernesto Castillo, Francisco Forster, Diego Rodríguez, Daniela Ruz, Ignacio Reyes, Camilo Valenzuela. ALeRCE infrastructure](https://reader033.fdocuments.net/reader033/viewer/2022042409/5f24d90a0ac77c2e420ef8e6/html5/thumbnails/27.jpg)
API
● ZTF Database: http://ztf.alerce.online
● Avro/Stamps: http://avro.alerce.online
● catsHTM Cone Search & Xmatch: http://catshtm.alerce.online *
* Soumagnac & Ofek (2018), (Ofek 2014; ascl.soft 07005)
![Page 28: ALeRCE broker: current status and demo · Javier Arredondo, Ernesto Castillo, Francisco Forster, Diego Rodríguez, Daniela Ruz, Ignacio Reyes, Camilo Valenzuela. ALeRCE infrastructure](https://reader033.fdocuments.net/reader033/viewer/2022042409/5f24d90a0ac77c2e420ef8e6/html5/thumbnails/28.jpg)
Jupyter Notebooks
● API
● Transients
● Variable Stars
● Active Galactic Nuclei
https://github.com/alercebroker/usecases/tree/master/notebooks