Researchdirections

Overview We present a novel approach to tackle the ObjectNav task for non-stationary and potentially occluded targets in an indoor environment. We refer to this task Portable ObjectNav (or P-ObjectNav), and in this work, present its formulation, feasibility, and a navigation benchmark using a novel memory-enhanced LLM-based policy. In contrast to ObjNav where target object locations are fixed for each episode, P-ObjectNav tackles the challenging case where the target objects move during the episode.

Paper Code Dataset Tech Report RoadTrack GitHub Repository TRAF/MOT/KITTI Tech Report The first video is a short summary of our work. The second video demonstrates RVO. The final video demonstrates the SimCAI algorithm.

We present a realtime tracking algorithm, RoadTrack, to track heterogeneous road-agents in dense traffic videos. Our approach is designed for traffic scenarios that consist of different road-agents such as pedestrians, two-wheelers, cars, buses, etc.

Paper Code Dataset RobustTP, ACM CSCS 2019 GitHub Code TRAF We present RobustTP, an end-to-end algorithm for predicting future trajectories of road-agents in dense traffic with noisy sensorinput trajectories obtained from RGB cameras (either static or moving) through a tracking algorithm. In this case, we consider noise as the deviation from the ground truth trajectory. The amount of noise depends on the accuracy of the tracking algorithm.

– – Shivam Akhauri, Laura Zheng, Tom Goldstein, Ming Lin – Description Autonomous driving models are prone to different types of perturbations at all levels of development. Elements such as blur, noise, distortion, and coloration can vary from dataset to dataset. On this project page, we list several projects related to improving robustness for autonomous driving.
Papers Gradient-Free Adversarial Training Against Image Corruption for Learning-based Steering.

Paper Code FAR GitHub Code Abstract: We introduce SOAR, a novel Self-supervised pre- training algorithm for aerial footage captured by Unmanned Aerial Vehicles (UAVs). We incorporate human object knowl- edge throughout the pretraining process to enhance UAV video pretraining efficiency and downstream action recognition performance. This is in contrast to prior works that primarily incorporate object information during the fine-tuning stage.

Overview of SS-SFDA: In stage 1, our model is pre-trained on a clear weather source dataset. In stage 2, our model is initialized with the pre-trained model from stage 1 and trained using our self-supervised algorithm, SS-SFDA, on the unlabeled adverse weather dataset. For heterogeneous weather datasets, we perform additional refinement steps based on model distillation (stage 3) Paper: Link GitHub Code: Link

We present a novel approach for unsupervised road segmentation in adverse weather conditions such as rain or fog.

Abstract We present a novel classifier network called STEP, to classify perceived human emotion from gaits, based on a Spatial Temporal Graph Convolutional Network (ST-GCN) architecture. Given an RGB video of an individual walking, our formulation implicitly exploits the gait features to classify the emotional state of the human into one of four emotions: happy, sad, angry, or neutral. We use hundreds of annotated real-world gait videos and augment them with thousands of annotated synthetic gaits generated using a novel generative network called STEP-Gen, built on an ST-GCN based Conditional Variational Autoencoder (CVAE).

– Paper Code Under Review Github (TBD)
Authors: Ming Lin Abstract: Data for training learning-enabled self-driving cars in the physical world are typically collected in a safe, normal environment. Such data distribution often engenders a strong bias towards safe driving, making self-driving cars unprepared when encountering adversarial scenarios like unexpected accidents. Due to a dearth of such adverse data that is practically unsafe for drivers to collect, autonomous vehicles can perform poorly when experiencing such rare events.

Abstract Differentiable physics is a powerful approach to learning and control problems that involve physical objects and environments. While notable progress has been made, the capabilities of differentiable physics solvers remain limited. We develop a scalable framework for differentiable physics that can support a large number of objects and their interactions. To accommodate objects with arbitrary geometry and topology, we adopt meshes as our representation and leverage the sparsity of contacts for scalable differentiable collision handling.