Research

I am the director of the SPARK (Sensing, Perception, Autonomy, and Robot Kinetics) group. The mission of my group is to develop theoretical understanding and practical algorithms for safe, robust, and efficient robot perception. For an autonomous vehicle (e.g., micro aerial vehicles, self-driving cars, planetary rovers) operating in an unknown environment, perception is the problem of creating an internal model of the surroundings using sensor data and prior knowledge. As such, perception includes a broad set of robotics and computer vision problems, including object detection and pose estimation, semantic understanding, robot localization and mapping, among others. My group is currently working on the key problems discussed below.

 

Sponsors

Certifiable Perception

Perception algorithms have been increasingly used in safety-critical applications, including intelligent transportation and military endeavors, where algorithmic failures may put human lives at risk. SPARK develops perception algorithms that work in extreme conditions (e.g., in the presence of extreme amounts of outliers due to false detections or sensor malfunction) and provide formal performance guarantees. In particular, we are currently developing a new generation of algorithms, certifiable perception algorithms, that are able to compute a robust world model and assess its correctness in real-time time. Contrary to the brittleness of existing perception methods, these algorithms are “hard to break” and have the potential to attain super-human performance, paving the way to safe and trustworthy autonomy.

Selected publications:

  • [PDF] J. Shi, H. Yang, and L. Carlone. Optimal Pose and Shape Estimation for Category-level 3D Object Perception. In Robotics: Science and Systems (RSS), 2021.
    [Bibtex] 
    @InProceedings{Shi21rss-pace,
title={Optimal Pose and Shape Estimation for Category-level {3D} Object Perception},
author={J. Shi and H. Yang and L. Carlone},
booktitle=rss,
nonote = {arXiv preprint: 2104.08383, \linkToPdf{https://arxiv.org/pdf/2104.08383.pdf}, \linkToVideo{https://youtu.be/kiNBS0IF2-g}\award{, finalist for Best Paper Award}},
pdf={https://arxiv.org/pdf/2104.08383.pdf},
year={2021},
}
  • [PDF] J. Shi, H. Yang, and L. Carlone. ROBIN: a Graph-Theoretic Approach to Reject Outliers in Robust Estimation using Invariants. In IEEE Intl. Conf. on Robotics and Automation (ICRA), 2021.
    [Bibtex] 
    @InProceedings{Shi21icra-robin,
title={{ROBIN:} a Graph-Theoretic Approach to Reject Outliers in Robust Estimation using Invariants},
author={J. Shi and H. Yang and L. Carlone},
booktitle=icra,
nonote = {arXiv preprint: 2011.03659, \linkToPdf{https://arxiv.org/pdf/2011.03659.pdf}},
pdf="https://arxiv.org/pdf/2011.03659.pdf",
year={2021}
}
  • [PDF] H. Yang and L. Carlone. One Ring to Rule Them All: Certifiably Robust Geometric Perception with Outliers. In Conf. on Neural Information Processing Systems (NeurIPS), volume 33, page 18846–18859, 2020.
    [Bibtex] 
    @InProceedings{Yang20neurips-certifiablePerception,
fullauthor = {Yang, Heng and Carlone, Luca},
author = {H. Yang and L. Carlone},
booktitle = neurips,
pages = {18846--18859},
title = {One Ring to Rule Them All: Certifiably Robust Geometric Perception with Outliers},
url = {https://proceedings.neurips.cc/paper/2020/file/da6ea77475918a3d83c7e49223d453cc-Paper.pdf},
nonote = {\linkToPdf{https://arxiv.org/pdf/2006.06769.pdf}},
pdf={https://arxiv.org/pdf/2006.06769.pdf},
volume = {33},
year = {2020}
}
  • [PDF] H. Yang, J. Shi, and L. Carlone. TEASER: Fast and Certifiable Point Cloud Registration. IEEE Trans. Robotics, 37(2):314–333, 2020.
    [Bibtex] 
    @article{Yang20tro-teaser,
title={{TEASER: Fast and Certifiable Point Cloud Registration}},
author={H. Yang and J. Shi and L. Carlone},
journal=tro,
volume = 37,
number = 2,
pages = {314--333},
nonote = {extended arXiv version 2001.07715 \linkToPdf{https://arxiv.org/pdf/2001.07715.pdf}},
pdf={https://arxiv.org/pdf/2001.07715.pdf},
Year = {2020}
}
  • [PDF] H. Yang and L. Carlone. In Perfect Shape: Certifiably Optimal 3D Shape Reconstruction from 2D Landmarks. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020.
    [Bibtex] 
    @inproceedings{Yang20cvpr-shapeStar,
title={In Perfect Shape: Certifiably Optimal {3D} Shape Reconstruction from {2D} Landmarks},
author={H. Yang and L. Carlone},
booktitle=cvpr,
nonote = {Arxiv version: 1911.11924, \linkToPdf{https://arxiv.org/pdf/1911.11924.pdf}},
pdf="https://arxiv.org/pdf/1911.11924.pdf",
year={2020}
}
  • [PDF] H. Yang, P. Antonante, V. Tzoumas, and L. Carlone. Graduated Non-Convexity for Robust Spatial Perception: From Non-Minimal Solvers to Global Outlier Rejection. IEEE Robotics and Automation Letters (RA-L), 5(2):1127–1134, 2020.
    [Bibtex] 
    @article{Yang20ral-GNC,
Author = {H. Yang and P. Antonante and V. Tzoumas and L. Carlone},
FullAuthor = {Heng Yang, Pasquale Antonante, Vasileios Tzoumas, Luca Carlone},
Title = {Graduated Non-Convexity for Robust Spatial Perception: From Non-Minimal Solvers to Global Outlier Rejection},
Volume = 5,
Number = 2,
Pages = {1127--1134},
nonote = {arXiv preprint:1909.08605 (with supplemental material), \linkToPdf{https://arxiv.org/pdf/1909.08605.pdf}\award{, ICRA Best paper award in Robot Vision}},
pdf = "https://arxiv.org/pdf/1909.08605.pdf",
journal=ral,
year={2020}
}
  • [PDF] V. Tzoumas, P. Antonante, and L. Carlone. Outlier-Robust Spatial Perception: Hardness, General-Purpose Algorithms, and Guarantees. In IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), 2019.
    [Bibtex] 
    @inproceedings{Tzoumas19iros-outliers,
Author = {V. Tzoumas and P. Antonante and L. Carlone},
Title = {Outlier-Robust Spatial Perception: Hardness, General-Purpose Algorithms, and Guarantees},
nonote = {Extended arxiv version: 1903.11683, \linkToPdf{https://arxiv.org/pdf/1903.11683.pdf}},
pdf = "https://arxiv.org/pdf/1903.11683.pdf",
booktitle= iros,
Year = 2019}
  • [PDF] H. Yang and L. Carlone. A Quaternion-based Certifiably Optimal Solution to the Wahba Problem with Outliers. In Intl. Conf. on Computer Vision (ICCV), 2019.
    [Bibtex] 
    @inproceedings{Yang19iccv-QUASAR,
title={A Quaternion-based Certifiably Optimal Solution to the {Wahba} Problem with Outliers},
author={H. Yang and L. Carlone},
fullauthor={Yang, Heng and Carlone, Luca},
booktitle=iccv,
nonote = {(Oral Presentation, accept rate: 4\%), Arxiv version: 1905.12536, \linkToPdf{https://arxiv.org/pdf/1905.12536.pdf}},
pdf = "https://arxiv.org/pdf/1905.12536.pdf",
year={2019}
}
  • [PDF] P. Lajoie, S. Hu, G. Beltrame, and L. Carlone. Modeling Perceptual Aliasing in SLAM via Discrete-Continuous Graphical Models. IEEE Robotics and Automation Letters (RA-L), 2019.
    [Bibtex] 
    @article{Lajoie19ral-DCGM,
Author = {P. Lajoie and S. Hu and G. Beltrame and L. Carlone},
Title = {Modeling Perceptual Aliasing in {SLAM} via Discrete-Continuous Graphical Models},
Journal = ral,
nonote = {extended ArXiv version:
\linkToPdf{https://arxiv.org/pdf/1810.11692.pdf},
Supplemental Material:
\linkToPdf{https://www.dropbox.com/s/vupak65wi75yzbl/2018j-RAL-DCGM-supplemental.pdf?dl=0}},
pdf = "https://arxiv.org/pdf/1810.11692.pdf",
Year = 2019}
  • [PDF] D. M. Rosen, L. Carlone, A. S. Bandeira, and J. J. Leonard. SE-Sync: A Certifiably Correct Algorithm for Synchronization over the Special Euclidean Group. In Intl. Workshop on the Algorithmic Foundations of Robotics (WAFR), San Francisco, CA, December 2016.
    [Bibtex] 
    @inproceedings{Rosen16wafr-sesync,
Author = {D.M. Rosen and L. Carlone and A.S. Bandeira and J.J. Leonard},
Booktitle = WAFR,
Title = {{SE-Sync}: A Certifiably Correct Algorithm for Synchronization over the {Special Euclidean} Group},
month = {December},
address = {San Francisco, CA},
nonote = {
extended arxiv preprint: 1611.00128,
\linkToPdf{http://arxiv.org/abs/1611.00128}
\linkToPdf{http://wafr2016.berkeley.edu/papers/WAFR_2016_paper_138.pdf}
\linkToCode{https://github.com/david-m-rosen/SE-Sync}\award{, Best paper award}},
pdf = "http://arxiv.org/abs/1611.00128",
Year = 2016}
  • [PDF] L. Carlone, G. Calafiore, C. Tommolillo, and F. Dellaert. Planar Pose Graph Optimization: Duality, Optimal Solutions, and Verification. IEEE Trans. Robotics, 32(3):545–565, 2016.
    [Bibtex] 
    @article{Carlone16tro-duality2D,
Author = {L. Carlone and G. Calafiore and C. Tommolillo and F. Dellaert},
Journal = tro,
Title = {Planar Pose Graph Optimization: Duality, Optimal Solutions, and Verification},
Volume = 32,
Number = 3,
Pages = {545--565},
nonote = {\linkToPdf{https://www.dropbox.com/s/peoktkct0cw42av/2015j-TRO-dualityPGO2D.pdf?dl=0}
\linkToCode{https://www.bitbucket.org/lucacarlone/pgo2d-duality-opencode}},
pdf = "https://www.dropbox.com/s/peoktkct0cw42av/2015j-TRO-dualityPGO2D.pdf?dl=0",
Year = 2016}

High-level Perception and Spatial AI

Situational awareness in complex, unstructured, and dynamical environments requires autonomous agents to build and maintain a multifaceted model of the environment, including both a 3D geometric model (useful for navigation and coordination) and semantic labels (useful to characterize areas of interest and to provide more succinct information to human operators). SPARK is currently investigating metric-semantic perception, with the goal of developing practical algorithms for joint-metric semantic understanding. Joint metric-semantic understanding has the potential to increase robustness of perception by leveraging the redundancy between semantics and geometry, provide opportunity for data compression, and enhance high-level human-robot spatial interaction.

Selected publications:

  • [PDF] R. Talak, S. Hu, L. Peng, and L. Carlone. Neural Trees for Learning on Graphs. In Conf. on Neural Information Processing Systems (NeurIPS), 2021.
    [Bibtex] 
    @InProceedings{Talak21neurips-neuralTree,
title={Neural Trees for Learning on Graphs},
author={R. Talak and S. Hu and L. Peng and L. Carlone},
booktitle=neurips,
nonote = {\linkToPdf{https://arxiv.org/pdf/2105.07264.pdf}},
pdf={https://arxiv.org/pdf/2105.07264.pdf},
year={2021}
}
  • [PDF] A. Rosinol, A. Violette, M. Abate, N. Hughes, Y. Chang, J. Shi, A. Gupta, and L. Carlone. Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphs. Intl. J. of Robotics Research, 40(12–14):1510–1546, 2021.
    [Bibtex] 
    @Article{Rosinol21ijrr-Kimera,
title={Kimera: from {SLAM} to Spatial Perception with {3D} Dynamic Scene Graphs},
author={A. Rosinol and A. Violette and M. Abate and N. Hughes and Y. Chang
and J. Shi and A. Gupta and L. Carlone},
journal=ijrr,
volume = {40},
number = {12--14},
pages = {1510--1546},
nonote = {arXiv preprint: 2101.06894, \linkToPdf{https://arxiv.org/pdf/2101.06894.pdf}},
pdf={https://arxiv.org/pdf/2101.06894.pdf},
year={2021},
}
  • [PDF] Y. Chang, Y. Tian, J. P. How, and L. Carlone. Kimera-Multi: a System for Distributed Multi-Robot Metric-Semantic Simultaneous Localization and Mapping. In IEEE Intl. Conf. on Robotics and Automation (ICRA), 2021.
    [Bibtex] 
    @InProceedings{Chang21icra-KimeraMulti,
title={{Kimera-Multi}: a System for Distributed Multi-Robot Metric-Semantic Simultaneous Localization and Mapping},
author={Y. Chang and Y. Tian and J.P. How and L. Carlone},
booktitle=icra,
nonote = {arXiv preprint: 2011.04087, \linkToPdf{https://arxiv.org/pdf/2011.04087.pdf}},
pdf="https://arxiv.org/pdf/2011.04087.pdf",
year={2021}
}
  • [PDF] [DOI] F. Milano, A. Loquercio, A. Rosinol, D. Scaramuzza, and L. Carlone. Primal-Dual Mesh Convolutional Neural Networks. In Conf. on Neural Information Processing Systems (NeurIPS), 2020.
    [Bibtex] 
    @InProceedings{Milano20neurips-PDMeshNet,
title = {Primal-Dual Mesh Convolutional Neural Networks},
author = {F. Milano and A. Loquercio and A. Rosinol and D. Scaramuzza and L. Carlone},
booktitle = neurips,
year = 2020,
code = {https://github.com/MIT-SPARK/PD-MeshNet},
pdf={https://proceedings.neurips.cc/paper/2020/file/0a656cc19f3f5b41530182a9e03982a4-Paper.pdf},
doi={arxiv-2010.12455},
}
  • [PDF] [DOI] A. Rosinol, A. Gupta, M. Abate, J. Shi, and L. Carlone. 3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans. In Robotics: Science and Systems (RSS), 2020.
    [Bibtex] 
    @InProceedings{Rosinol20rss-dynamicSceneGraphs,
title={{3D} Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans},
author={A. Rosinol and A. Gupta and M. Abate and J. Shi and L. Carlone},
booktitle=rss,
year=2020,
pdf={https://arxiv.org/pdf/2002.06289.pdf},
url={http://news.mit.edu/2020/robots-spatial-perception-0715},
video={https://www.youtube.com/watch?v=SWbofjhyPzI},
doi={10.15607/RSS.2020.XVI.079},
nonote = {\linkToPdf{https://arxiv.org/pdf/2002.06289.pdf},
\linkToMedia{http://news.mit.edu/2020/robots-spatial-perception-0715},
\linkToVideo{https://www.youtube.com/watch?v=SWbofjhyPzI&feature=youtu.be}},
}
  • [PDF] [DOI] A. Rosinol, M. Abate, Y. Chang, and L. Carlone. Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mapping. In IEEE Intl. Conf. on Robotics and Automation (ICRA), 2020.
    [Bibtex] 
    @InProceedings{Rosinol20icra-Kimera,
title={Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mapping},
author={A. Rosinol and M. Abate and Y. Chang and L. Carlone},
booktitle=icra,
year={2020},
pdf={https://arxiv.org/pdf/1910.02490.pdf},
video={https://www.youtube.com/watch?v=-5XxXRABXJs},
code={https://github.com/MIT-SPARK/Kimera},
doi={10.1109/ICRA40945.2020.9196885},
nonote = {arXiv preprint: 1910.02490,
\linkToVideo{https://www.youtube.com/watch?v=-5XxXRABXJs},
\linkToCode{https://github.com/MIT-SPARK/Kimera},
\linkToPdf{https://arxiv.org/pdf/1910.02490.pdf}},
}
  • [PDF] [DOI] A. Rosinol, T. Sattler, M. Pollefeys, and L. Carlone. Incremental Visual-Inertial 3D Mesh Generation with Structural Regularities. In IEEE Intl. Conf. on Robotics and Automation (ICRA), 2019.
    [Bibtex] 
    @InProceedings{Rosinol19icra-mesh,
author = {A. Rosinol and T. Sattler and M. Pollefeys and L. Carlone},
title = {Incremental {V}isual-{I}nertial {3D} {M}esh {G}eneration with {S}tructural {R}egularities},
booktitle = icra,
year = 2019,
pdf = {https://arxiv.org/pdf/1903.01067.pdf},
video = {https://www.youtube.com/watch?v=C5fFDEJ9cFQ},
url = {https://www.mit.edu/\%7Earosinol/research/struct3dmesh.html},
doi={10.1109/ICRA.2019.8794456},
nonote = {\linkToPdf{https://arxiv.org/pdf/1903.01067.pdf},
\linkToWeb{https://www.mit.edu/~arosinol/research/struct3dmesh.html}},
}

Efficient and Task-driven Perception

Pervasive applications of computational perception in virtual and augmented reality (AR/VR) and miniaturized aerial platforms require perception algorithms to run in real-time on computationally constrained platforms. This issue is exacerbated by the fact that perception has to process high-rate high-dimensional sensor data (e.g., camera images). Research in SPARK has focused on developing a framework for task-driven perception, that selects the most relevant subset of sensor data to complete a control task under given constraints on performance and computation/power budget. Moreover, past research includes algorithm and hardware co-design of the first chip for visual-inertial navigation, that enables a palm-size drone to see and understand the external world and autonomously navigate among obstacles.

Selected publications:

  • V. Tripathi, L. Ballotta, L. Carlone, and E. Modiano. Computation and Communication Co-Design for Real-Time Monitoring and Control in Multi-Agent Systems. In Intl. Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks (WiOpt), 2021.
    [Bibtex] 
    @InProceedings{Tripathi21wiopt-compCommCodesign,
title={Computation and Communication Co-Design for Real-Time Monitoring and Control in Multi-Agent Systems},
author={V. Tripathi and L. Ballotta and L. Carlone and E. Modiano},
booktitle={Intl. Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks (WiOpt)},
year={2021}
}
  • [PDF] L. Ballotta, L. Schenato, and L. Carlone. From Sensor to Processing Networks: Optimal Estimation with Computation and Communication Latency. In IFAC World Congress, 2020.
    [Bibtex] 
    @InProceedings{Ballotta20ifac-processingNetworks,
title={From Sensor to Processing Networks: Optimal Estimation with Computation and Communication Latency},
author={L. Ballotta and L. Schenato and L. Carlone},
nonote = {arXiv preprint: 2003.08301, \linkToPdf{https://arxiv.org/pdf/2003.08301.pdf}\award{, winner of the Young Author Prize}},
year=2020,
booktitle = {IFAC World Congress},
pdf={https://arxiv.org/pdf/2003.08301.pdf}
}
  • [PDF] L. Carlone and S. Karaman. Attention and Anticipation in Fast Visual-Inertial Navigation. IEEE Trans. Robotics, 2018.
    [Bibtex] 
    @article{Carlone18tro-attentionVIN,
Author = {L. Carlone and S. Karaman},
Title = {Attention and Anticipation in Fast Visual-Inertial Navigation},
Journal = tro,
nonote = {arxiv preprint: 1610.03344,
\linkToPdf{https://www.dropbox.com/s/c19kyrikroypahw/2017j-visualAttention.pdf?dl=0}},
pdf = "https://www.dropbox.com/s/c19kyrikroypahw/2017j-visualAttention.pdf?dl=0",
Year = 2018}
  • [PDF] L. Carlone and C. Pinciroli. Robot Co-design: Beyond the Monotone Case. In IEEE Intl. Conf. on Robotics and Automation (ICRA), 2019.
    [Bibtex] 
    @inproceedings{Carlone19icra-codesign,
Author = {L. Carlone and C. Pinciroli},
Title = {Robot Co-design: Beyond the Monotone Case},
nonote = {Extended arxiv version: 1902.05880, \linkToPdf{https://arxiv.org/pdf/1902.05880.pdf}},
pdf = "https://arxiv.org/pdf/1902.05880.pdf",
booktitle= icra,
Year = 2019}
  • [PDF] F. Ma, L. Carlone, U. Ayaz, and S. Karaman. Sparse Depth Sensing for Resource-Constrained Robot Perception. Intl. J. of Robotics Research, 2018.
    [Bibtex] 
    @article{Ma18ijrr-sparseSensing,
Author = {F. Ma and L. Carlone and U. Ayaz and S. Karaman},
Title = {Sparse Depth Sensing for Resource-Constrained Robot Perception},
Journal = ijrr,
nonote = {arxiv preprint: 1703.01398,
\linkToPdf{https://arxiv.org/pdf/1703.01398.pdf}},
pdf = "https://arxiv.org/pdf/1703.01398.pdf",
Year = 2018}
  • [PDF] Z. Zhang, A. Suleiman, L. Carlone, V. Sze, and S. Karaman. Visual-Inertial Odometry on Chip: An Algorithm-and-Hardware Co-design Approach. In Robotics: Science and Systems (RSS), 2017.
    [Bibtex] 
    @inproceedings{Zhang17rss-vioChip,
Author = {Z. Zhang and A. Suleiman and L. Carlone and V. Sze and S. Karaman},
title={Visual-Inertial Odometry on Chip: An Algorithm-and-Hardware Co-design Approach},
booktitle= rss,
nonote = {\linkToPdf{http://rss2017.lids.mit.edu/static/papers/74.pdf}
\linkToWeb{http://navion.mit.edu/}, highlighted in the MIT News:
\linkToWeb{http://news.mit.edu/2017/miniaturizing-brain-smart-drones-0712}},
pdf = "http://www.roboticsconference.org/static/papers/74.pdf",
year={2017}}