I am the director of the SPARK (Sensing, Perception, Autonomy, and Robot Kinetics) group. The mission of my group is to develop theoretical understanding and practical algorithms for safe, robust, and efficient robot perception. For an autonomous vehicle (e.g., micro aerial vehicles, self-driving cars, planetary rovers) operating in an unknown environment, perception is the problem of creating an internal model of the surroundings using sensor data and prior knowledge. As such, perception includes a broad set of robotics and computer vision problems, including object detection and pose estimation, semantic understanding, robot localization and mapping, among others. My group is currently working on the following key problems:

 

Certifiable Perception

Perception algorithms have been increasingly used in safety-critical applications, including intelligent transportation and military endeavors, where algorithmic failures may put human lives at risk. SPARK develops perception algorithms that work in extreme conditions (e.g., in the presence of extreme amounts of outliers due to false detections or sensor malfunction) and provide formal performance guarantees. In particular, we are currently developing a new generation of algorithms, certifiable perception algorithms, that are able to compute a robust world model and assess its correctness in real-time time. Contrary to the brittleness of existing perception methods, these algorithms are “hard to break” and have the potential to attain super-human performance, paving the way to safe and trustworthy autonomy.

Selected publications:

  • Heng Yang and Luca Carlone. One Ring to Rule Them All: Certifiably Robust Geometric Perception with Outliers. arXiv preprint arXiv:2006.06769, 2020.
    [Bibtex]
    @article{Yang20arXiv-certifiablePerception,
    title={One Ring to Rule Them All: Certifiably Robust Geometric Perception with Outliers},
    author={Heng Yang and Luca Carlone},
    journal={arXiv preprint arXiv:2006.06769},
    year={2020}
    }
  • [PDF] H. Yang, J. Shi, and L. Carlone. TEASER: Fast and Certifiable Point Cloud Registration. arXiv preprint arXiv: 2001.07715, 2020.
    [Bibtex]
    @article{Yang20arxiv-teaser,
    title={{TEASER: Fast and Certifiable Point Cloud Registration}},
    author={H. Yang and J. Shi and L. Carlone},
    journal={arXiv preprint arXiv: 2001.07715},
    nonote = {\linkToPdf{https://arxiv.org/pdf/2001.07715.pdf}},
    pdf={https://arxiv.org/pdf/2001.07715.pdf},
    Year = {2020}
    }
  • [PDF] H. Yang and L. Carlone. In Perfect Shape: Certifiably Optimal 3D Shape Reconstruction from 2D Landmarks. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020.
    [Bibtex]
    @inproceedings{Yang20cvpr-shapeStar,
    title={In Perfect Shape: Certifiably Optimal {3D} Shape Reconstruction from {2D} Landmarks},
    author={H. Yang and L. Carlone},
    booktitle=cvpr,
    nonote = {Arxiv version: 1911.11924, \linkToPdf{https://arxiv.org/pdf/1911.11924.pdf}},
    pdf="https://arxiv.org/pdf/1911.11924.pdf",
    year={2020}
    }
  • [PDF] H. Yang, P. Antonante, V. Tzoumas, and L. Carlone. Graduated Non-Convexity for Robust Spatial Perception: From Non-Minimal Solvers to Global Outlier Rejection. IEEE Robotics and Automation Letters (RA-L), 5(2):1127–1134, 2020.
    [Bibtex]
    @article{Yang20ral-GNC,
    Author = {H. Yang and P. Antonante and V. Tzoumas and L. Carlone},
    FullAuthor = {Heng Yang, Pasquale Antonante, Vasileios Tzoumas, Luca Carlone},
    Title = {Graduated Non-Convexity for Robust Spatial Perception: From Non-Minimal Solvers to Global Outlier Rejection},
    Volume = 5,
    Number = 2,
    Pages = {1127--1134},
    nonote = {arXiv preprint arXiv:1909.08605 (with supplemental material), \linkToPdf{https://arxiv.org/pdf/1909.08605.pdf}, \award{ICRA Best paper award in Robot Vision}},
    pdf = "https://arxiv.org/pdf/1909.08605.pdf",
    journal=ral,
    year={2020}
    }
  • [PDF] V. Tzoumas, P. Antonante, and L. Carlone. Outlier-Robust Spatial Perception: Hardness, General-Purpose Algorithms, and Guarantees. In IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), 2019.
    [Bibtex]
    @inproceedings{Tzoumas19iros-outliers,
    Author = {V. Tzoumas and P. Antonante and L. Carlone},
    Title = {Outlier-Robust Spatial Perception: Hardness, General-Purpose Algorithms, and Guarantees},
    nonote = {Extended arxiv version: 1903.11683, \linkToPdf{https://arxiv.org/pdf/1903.11683.pdf}},
    pdf = "https://arxiv.org/pdf/1903.11683.pdf",
    booktitle= iros,
    Year = 2019}
  • [PDF] H. Yang and L. Carlone. A Quaternion-based Certifiably Optimal Solution to the Wahba Problem with Outliers. In Intl. Conf. on Computer Vision (ICCV), 2019.
    [Bibtex]
    @inproceedings{Yang19iccv-QUASAR,
    title={A Quaternion-based Certifiably Optimal Solution to the {Wahba} Problem with Outliers},
    author={H. Yang and L. Carlone},
    fullauthor={Yang, Heng and Carlone, Luca},
    booktitle=iccv,
    nonote = {(Oral Presentation, accept rate: 4\%), Arxiv version: 1905.12536, \linkToPdf{https://arxiv.org/pdf/1905.12536.pdf}},
    pdf = "https://arxiv.org/pdf/1905.12536.pdf",
    year={2019}
    }
  • [PDF] P. Lajoie, S. Hu, G. Beltrame, and L. Carlone. Modeling Perceptual Aliasing in SLAM via Discrete-Continuous Graphical Models. IEEE Robotics and Automation Letters (RA-L), 2019.
    [Bibtex]
    @article{Lajoie19ral-DCGM,
    Author = {P. Lajoie and S. Hu and G. Beltrame and L. Carlone},
    Title = {Modeling Perceptual Aliasing in {SLAM} via Discrete-Continuous Graphical Models},
    Journal = ral,
    nonote = {extended ArXiv version:
    \linkToPdf{https://arxiv.org/pdf/1810.11692.pdf},
    Supplemental Material:
    \linkToPdf{https://www.dropbox.com/s/vupak65wi75yzbl/2018j-RAL-DCGM-supplemental.pdf?dl=0}},
    pdf = "https://arxiv.org/pdf/1810.11692.pdf",
    Year = 2019}
  • [PDF] D. M. Rosen, L. Carlone, A. S. Bandeira, and J. J. Leonard. SE-Sync: A Certifiably Correct Algorithm for Synchronization over the Special Euclidean Group. In Intl. Workshop on the Algorithmic Foundations of Robotics (WAFR), San Francisco, CA, December 2016.
    [Bibtex]
    @inproceedings{Rosen16wafr-sesync,
    Author = {D.M. Rosen and L. Carlone and A.S. Bandeira and J.J. Leonard},
    Booktitle = WAFR,
    Title = {{SE-Sync}: A Certifiably Correct Algorithm for Synchronization over the {Special Euclidean} Group},
    month = {December},
    address = {San Francisco, CA},
    nonote = {
    extended arxiv preprint: 1611.00128,
    \linkToPdf{http://arxiv.org/abs/1611.00128}
    \linkToPdf{http://wafr2016.berkeley.edu/papers/WAFR_2016_paper_138.pdf}
    \linkToCode{https://github.com/david-m-rosen/SE-Sync}\award{, Best paper award}},
    pdf = "http://arxiv.org/abs/1611.00128",
    Year = 2016}
  • [PDF] L. Carlone, G. Calafiore, C. Tommolillo, and F. Dellaert. Planar Pose Graph Optimization: Duality, Optimal Solutions, and Verification. IEEE Trans. Robotics, 32(3):545–565, 2016.
    [Bibtex]
    @article{Carlone16tro-duality2D,
    Author = {L. Carlone and G. Calafiore and C. Tommolillo and F. Dellaert},
    Journal = tro,
    Title = {Planar Pose Graph Optimization: Duality, Optimal Solutions, and Verification},
    Volume = 32,
    Number = 3,
    Pages = {545--565},
    nonote = {\linkToPdf{https://www.dropbox.com/s/peoktkct0cw42av/2015j-TRO-dualityPGO2D.pdf?dl=0}
    \linkToCode{https://www.bitbucket.org/lucacarlone/pgo2d-duality-opencode}},
    pdf = "https://www.dropbox.com/s/peoktkct0cw42av/2015j-TRO-dualityPGO2D.pdf?dl=0",
    Year = 2016}

High-level Perception and Spatial AI

Situational awareness in complex, unstructured, and dynamical environments requires autonomous agents to build and maintain a multifaceted model of the environment, including both a 3D geometric model (useful for navigation and coordination) and semantic labels (useful to characterize areas of interest and to provide more succinct information to human operators). SPARK is currently investigating metric-semantic perception, with the goal of developing practical algorithms for joint-metric semantic understanding. Joint metric-semantic understanding has the potential to increase robustness of perception by leveraging the redundancy between semantics and geometry, provide opportunity for data compression, and enhance high-level human-robot spatial interaction.

Selected publications:

  • [PDF] A. Rosinol, A. Gupta, M. Abate, J. Shi, and L. Carlone. 3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans. In Robotics: Science and Systems (RSS), 2020.
    [Bibtex]
    @InProceedings{Rosinol20rss-dynamicSceneGraphs,
    title={{3D} Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans},
    author={A. Rosinol and A. Gupta and M. Abate and J. Shi and L. Carlone},
    booktitle=rss,
    nonote = {\linkToPdf{https://arxiv.org/pdf/2002.06289.pdf},
    \linkToMedia{http://news.mit.edu/2020/robots-spatial-perception-0715},
    \linkToVideo{https://www.youtube.com/watch?v=SWbofjhyPzI&feature=youtu.be}},
    pdf={https://arxiv.org/pdf/2002.06289.pdf},
    year={2020}
    }
  • [PDF] A. Rosinol, M. Abate, Y. Chang, and L. Carlone. Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mapping. In IEEE Intl. Conf. on Robotics and Automation (ICRA), 2020.
    [Bibtex]
    @InProceedings{Rosinol20icra-Kimera,
    title={Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mapping},
    author={A. Rosinol and M. Abate and Y. Chang and L. Carlone},
    year={2020},
    booktitle = icra,
    nonote = {arXiv preprint arXiv: 1910.02490, \linkToVideo{https://www.youtube.com/watch?v=-5XxXRABXJs&feature=youtu.be},
    \linkToCode{https://github.com/MIT-SPARK/Kimera},
    \linkToPdf{https://arxiv.org/pdf/1910.02490.pdf}},
    pdf={https://arxiv.org/pdf/1910.02490.pdf}
    }
  • [PDF] [DOI] A. Rosinol, T. Sattler, M. Pollefeys, and L. Carlone. Incremental Visual-Inertial 3D Mesh Generation with Structural Regularities. In IEEE Intl. Conf. on Robotics and Automation (ICRA), 2019.
    [Bibtex]
    @InProceedings{Rosinol19icra-mesh,
    author = {A. Rosinol and T. Sattler and M. Pollefeys and L. Carlone},
    title = {Incremental {V}isual-{I}nertial {3D} {M}esh {G}eneration with {S}tructural {R}egularities},
    nonote = {\linkToPdf{https://arxiv.org/pdf/1903.01067.pdf},
    \linkToWeb{https://www.mit.edu/~arosinol/research/struct3dmesh.html}},
    pdf = "https://arxiv.org/pdf/1903.01067.pdf",
    url = {https://www.mit.edu/\%7Earosinol/research/struct3dmesh.html},
    doi={10.1109/ICRA.2019.8794456},
    booktitle = icra,
    year = 2019
    }

Efficient and Task-driven Perception

Pervasive applications of computational perception in virtual and augmented reality (AR/VR) and miniaturized aerial platforms require perception algorithms to run in real-time on computationally constrained platforms. This issue is exacerbated by the fact that perception has to process high-rate high-dimensional sensor data (e.g., camera images). Research in SPARK has focused on developing a framework for task-driven perception, that selects the most relevant subset of sensor data to complete a control task under given constraints on performance and computation/power budget. Moreover, past research includes algorithm and hardware co-design of the first chip for visual-inertial navigation, that enables a palm-size drone to see and understand the external world and autonomously navigate among obstacles.

Selected publications:

  • [PDF] L. Ballotta, L. Schenato, and L. Carlone. From Sensor to Processing Networks: Optimal Estimation with Computation and Communication Latency. In IFAC World Congress, 2020.
    [Bibtex]
    @InProceedings{Ballotta20ifac-processingNetworks,
    title={From Sensor to Processing Networks: Optimal Estimation with Computation and Communication Latency},
    author={L. Ballotta and L. Schenato and L. Carlone},
    nonote = {arXiv preprint arXiv: 2003.08301, \linkToPdf{https://arxiv.org/pdf/2003.08301.pdf}, \award{winner of the Young Author Prize}},
    year={2020},
    booktitle = {IFAC World Congress},
    pdf={https://arxiv.org/pdf/2003.08301.pdf}
    }
  • [PDF] L. Carlone and S. Karaman. Attention and Anticipation in Fast Visual-Inertial Navigation. IEEE Trans. Robotics, 2018.
    [Bibtex]
    @article{Carlone18tro-attentionVIN,
    Author = {L. Carlone and S. Karaman},
    Title = {Attention and Anticipation in Fast Visual-Inertial Navigation},
    Journal = tro,
    nonote = {arxiv preprint: 1610.03344,
    \linkToPdf{https://www.dropbox.com/s/c19kyrikroypahw/2017j-visualAttention.pdf?dl=0}},
    pdf = "https://www.dropbox.com/s/c19kyrikroypahw/2017j-visualAttention.pdf?dl=0",
    Year = 2018}
  • [PDF] L. Carlone and C. Pinciroli. Robot Co-design: Beyond the Monotone Case. In IEEE Intl. Conf. on Robotics and Automation (ICRA), 2019.
    [Bibtex]
    @inproceedings{Carlone19icra-codesign,
    Author = {L. Carlone and C. Pinciroli},
    Title = {Robot Co-design: Beyond the Monotone Case},
    nonote = {Extended arxiv version: 1902.05880, \linkToPdf{https://arxiv.org/pdf/1902.05880.pdf}},
    pdf = "https://arxiv.org/pdf/1902.05880.pdf",
    booktitle= icra,
    Year = 2019}
  • [PDF] F. Ma, L. Carlone, U. Ayaz, and S. Karaman. Sparse Depth Sensing for Resource-Constrained Robot Perception. Intl. J. of Robotics Research, 2018.
    [Bibtex]
    @article{Ma18ijrr-sparseSensing,
    Author = {F. Ma and L. Carlone and U. Ayaz and S. Karaman},
    Title = {Sparse Depth Sensing for Resource-Constrained Robot Perception},
    Journal = ijrr,
    nonote = {arxiv preprint: 1703.01398,
    \linkToPdf{https://arxiv.org/pdf/1703.01398.pdf}},
    pdf = "https://arxiv.org/pdf/1703.01398.pdf",
    Year = 2018}
  • [PDF] Z. Zhang, A. Suleiman, L. Carlone, V. Sze, and S. Karaman. Visual-Inertial Odometry on Chip: An Algorithm-and-Hardware Co-design Approach. In Robotics: Science and Systems (RSS), 2017.
    [Bibtex]
    @inproceedings{Zhang17rss-vioChip,
    Author = {Z. Zhang and A. Suleiman and L. Carlone and V. Sze and S. Karaman},
    title={Visual-Inertial Odometry on Chip: An Algorithm-and-Hardware Co-design Approach},
    booktitle= rss,
    nonote = {\linkToPdf{http://rss2017.lids.mit.edu/static/papers/74.pdf}
    \linkToWeb{http://navion.mit.edu/}, highlighted in the MIT News:
    \linkToWeb{http://news.mit.edu/2017/miniaturizing-brain-smart-drones-0712}},
    pdf = "http://www.roboticsconference.org/static/papers/74.pdf",
    year={2017}}