I am the director of the SPARK (Sensing, Perception, Autonomy, and Robot Kinetics) group. The mission of my group is to develop theoretical understanding and practical algorithms for safe, robust, and efficient robot perception. For an autonomous vehicle (e.g., micro aerial vehicles, self-driving cars, planetary rovers) operating in an unknown environment, perception is the problem of creating an internal model of the surroundings using sensor data and prior knowledge. As such, perception includes a broad set of robotics and computer vision problems, including object detection and pose estimation, semantic understanding, robot localization and mapping, among others. My group is currently working on the following key problems:

 

Verifiable and Provably Correct Perception

Perception algorithms have been increasingly used in safety-critical applications, including intelligent transportation and military endeavors, where algorithmic failures may put human lives at risk. SPARK develops perception algorithms that work in extreme conditions (e.g., in the presence of extreme amounts of outliers due to false detections or sensor malfunction) and provide formal performance guarantees. In particular, we are currently developing a new generation of algorithms, certifiably robust algorithms, that are able to compute a robust world model and assess its correctness in real-time time. Contrary to the brittleness of existing perception methods, these algorithms are “hard to break” and have the potential to attain super-human performance, paving the way to safe and trustworthy autonomy.

Selected publications:

  • [PDF] H. Yang and L. Carlone. A Polynomial-time Solution for Robust Registration with Extreme Outlier Rates. In Robotics: Science and Systems (RSS), 2019.
    [Bibtex]
    @inproceedings{Yang19rss-teaser,
    Author = {H. Yang and L. Carlone},
    Title = {A Polynomial-time Solution for Robust Registration with Extreme Outlier Rates},
    nonote = {\linkToPdf{http://rss2019.informatik.uni-freiburg.de/papers/0013_FI.pdf}, \linkToVideo{http://rss2019.informatik.uni-freiburg.de/videos/0013_VI_fi.mp4},
    \linkToMedia{http://news.mit.edu/2019/spotting-objects-cars-robots-0620},
    \linkToMedia{https://www.sciencedaily.com/releases/2019/06/190620121444.htm},
    \linkToMedia{http://www.ansa.it/canale_scienza_tecnica/notizie/tecnologie/2019/06/21/i-robot-imparano-a-vedere-nella-nebbia-_9e59485c-ff17-4d62-8224-1d42f44111b9.html}},
    booktitle= rss,
    pdf = "http://rss2019.informatik.uni-freiburg.de/papers/0013_FI.pdf",
    Year = 2019}
  • [PDF] Heng Yang and Luca Carlone. A Quaternion-based Certifiably Optimal Solution to the Wahba Problem with Outliers. In Intl. Conf. on Computer Vision (ICCV), 2019.
    [Bibtex]
    @inproceedings{Yang19iccv-QUASAR,
    title={A Quaternion-based Certifiably Optimal Solution to the {Wahba} Problem with Outliers},
    author={Yang, Heng and Carlone, Luca},
    booktitle=iccv,
    nonote = {Arxiv version: 1905.12536, \linkToPdf{https://arxiv.org/pdf/1905.12536.pdf}},
    pdf = "https://arxiv.org/pdf/1905.12536.pdf",
    year={2019}
    }
  • [PDF] P. Lajoie, S. Hu, G. Beltrame, and L. Carlone. Modeling Perceptual Aliasing in SLAM via Discrete-Continuous Graphical Models. IEEE Robotics and Automation Letters (RA-L), 2019.
    [Bibtex]
    @article{Lajoie19ral-DCGM,
    Author = {P. Lajoie and S. Hu and G. Beltrame and L. Carlone},
    Title = {Modeling Perceptual Aliasing in {SLAM} via Discrete-Continuous Graphical Models},
    Journal = ral,
    nonote = {extended ArXiv version:
    \linkToPdf{https://arxiv.org/pdf/1810.11692.pdf},
    Supplemental Material:
    \linkToPdf{https://www.dropbox.com/s/vupak65wi75yzbl/2018j-RAL-DCGM-supplemental.pdf?dl=0}},
    pdf = "https://arxiv.org/pdf/1810.11692.pdf",
    Year = 2019}
  • [PDF] D. M. Rosen, L. Carlone, A. S. Bandeira, and J. J. Leonard. SE-Sync: A Certifiably Correct Algorithm for Synchronization over the Special Euclidean Group. In Intl. Workshop on the Algorithmic Foundations of Robotics (WAFR), San Francisco, CA, December 2016.
    [Bibtex]
    @inproceedings{Rosen16wafr-sesync,
    Author = {D.M. Rosen and L. Carlone and A.S. Bandeira and J.J. Leonard},
    Booktitle = WAFR,
    Title = {{SE-Sync}: A Certifiably Correct Algorithm for Synchronization over the {Special Euclidean} Group},
    month = {December},
    address = {San Francisco, CA},
    nonote = {
    extended arxiv preprint: 1611.00128,
    \linkToPdf{http://arxiv.org/abs/1611.00128}
    \linkToPdf{http://wafr2016.berkeley.edu/papers/WAFR_2016_paper_138.pdf}
    \linkToCode{https://github.com/david-m-rosen/SE-Sync}\award{, best paper award}},
    pdf = "http://arxiv.org/abs/1611.00128",
    Year = 2016}
  • [PDF] L. Carlone, G. Calafiore, C. Tommolillo, and F. Dellaert. Planar Pose Graph Optimization: Duality, Optimal Solutions, and Verification. IEEE Trans. Robotics, 32(3):545–565, 2016.
    [Bibtex]
    @article{Carlone16tro-duality2D,
    Author = {L. Carlone and G. Calafiore and C. Tommolillo and F. Dellaert},
    Journal = tro,
    Title = {Planar Pose Graph Optimization: Duality, Optimal Solutions, and Verification},
    Volume = 32,
    Number = 3,
    Pages = {545--565},
    nonote = {\linkToPdf{https://www.dropbox.com/s/peoktkct0cw42av/2015j-TRO-dualityPGO2D.pdf?dl=0}
    \linkToCode{https://www.bitbucket.org/lucacarlone/pgo2d-duality-opencode}},
    pdf = "https://www.dropbox.com/s/peoktkct0cw42av/2015j-TRO-dualityPGO2D.pdf?dl=0",
    Year = 2016}

Efficient and Task-driven Perception

Pervasive applications of computational perception in virtual and augmented reality (AR/VR) and miniaturized aerial platforms require perception algorithms to run in real-time on computationally constrained platforms. This issue is exacerbated by the fact that perception has to process high-rate high-dimensional sensor data (e.g., camera images). Research in SPARK has focused on developing a framework for task-driven perception, that selects the most relevant subset of sensor data to complete a control task under given constraints on performance and computation/power budget. Moreover, past research includes algorithm and hardware co-design of the first chip for visual-inertial navigation, that enables a palm-size drone to see and understand the external world and autonomously navigate among obstacles.

Selected publications:

  • [PDF] V. Tzoumas, L. Carlone, G. J. Pappas, and A. Jadbabaie. Sensing-Constrained LQG Control. Technical Report, 2019.
    [Bibtex]
    @techreport{Tzoumas17tr-sLQG,
    Author = {V. Tzoumas and L. Carlone and G.J. Pappas and A. Jadbabaie},
    FullAuthor = {Vasileios Tzoumas and Luca Carlone and George J. Pappas and Ali Jadbabaie},
    Title = {Sensing-Constrained {LQG} Control},
    nonote = {arxiv preprint: 1709.08826,
    \linkToPdf{https://arxiv.org/pdf/1709.08826.pdf}},
    pdf = "https://arxiv.org/pdf/1709.08826.pdf",
    Year = 2019}
  • [PDF] Z. Zhang, A. Suleiman, L. Carlone, V. Sze, and S. Karaman. Visual-Inertial Odometry on Chip: An Algorithm-and-Hardware Co-design Approach. In Robotics: Science and Systems (RSS), 2017.
    [Bibtex]
    @inproceedings{Zhang17rss-vioChip,
    Author = {Z. Zhang and A. Suleiman and L. Carlone and V. Sze and S. Karaman},
    title={Visual-Inertial Odometry on Chip: An Algorithm-and-Hardware Co-design Approach},
    booktitle= rss,
    nonote = {\linkToPdf{http://rss2017.lids.mit.edu/static/papers/74.pdf}
    \linkToWeb{http://navion.mit.edu/}, highlighted in the MIT News:
    \linkToWeb{http://news.mit.edu/2017/miniaturizing-brain-smart-drones-0712}},
    pdf = "http://www.roboticsconference.org/static/papers/74.pdf",
    year={2017}}

Metric-semantic Perception for Spatial AI

Situational awareness in complex, unstructured, and dynamical environments requires autonomous agents to build and maintain a multifaceted model of the environment, including both a 3D geometric model (useful for navigation and coordination) and semantic labels (useful to characterize areas of interest and to provide more succinct information to human operators). SPARK is currently investigating metric-semantic perception, with the goal of developing practical algorithms for joint-metric semantic understanding. Joint metric-semantic understanding has the potential to increase robustness of perception by leveraging the redundancy between semantics and geometry, provide opportunity for data compression, and enhance high-level human-robot spatial interaction.

Selected publications:

  • [PDF] [DOI] A. Rosinol, T. Sattler, M. Pollefeys, and L. Carlone. Incremental Visual-Inertial 3D Mesh Generation with Structural Regularities. In IEEE Intl. Conf. on Robotics and Automation (ICRA), 2019.
    [Bibtex]
    @InProceedings{Rosinol19icra-mesh,
    author = {A. Rosinol and T. Sattler and M. Pollefeys and L. Carlone},
    title = {Incremental {V}isual-{I}nertial {3D} {M}esh {G}eneration with {S}tructural {R}egularities},
    nonote = {\linkToPdf{https://arxiv.org/pdf/1903.01067.pdf},
    \linkToWeb{https://www.mit.edu/~arosinol/research/struct3dmesh.html}},
    pdf = "https://arxiv.org/pdf/1903.01067.pdf",
    doi={10.1109/ICRA.2019.8794456},
    booktitle = icra,
    year = 2019
    }