How Lyft’s Library for Self-driving Simulation works

Autonomous Vehicles(AV) are changing the future of transportation. Lyft, a ride sharing company, has taken self-driving to another level. They believe that self-driving cars can make transportation safer, eco-friendly and easily available to everyone. The framework proposed by lyft is for developing learning-based solutions to prediction, planning and simulation problems in self-driving. The goal of this framework is to create data-driven approaches and simulations of real world driving data to allow and contribute to state-of-the-art solutions.

This software is developed by Lyft Level 5 self-driving division and is open to external contributors.

Overview of Lyft’s Framework

The framework consists of three modules:

1. Datasets – data available for training ML models.

To use the framework, download the Lyft Level 5 Prediction dataset from this available link. The dataset contains the following components:

• 1000 hours of traffic scenes that capture the motions of traffic participants around 20 self-driving vehicles. This data is stored in 30 second chunks using the zarr format.
• A hand-annotated, HD semantic map. This data is stored using protobuf format.
• NearMap provides a high-resolution aerial image of the area.

To know more about the dataset, check dataset whitepaper.

2. L5Kit – The library used for reading the data allows you to convert the planning and simulation problems into Machine Learning(ML) problems. This kit allows you to :

• Load driving scenes from zarr files
• Read semantic maps
• Read aerial maps
• Create birds-eye-view (BEV) images that represent a scene around an AV or another vehicle
• Sample data
• Train neural networks
• Visualize results

3. Examples – This framework contains many examples from visualizing the data to its training. Some of them are detailed out in the next sections.

Installation

Install the required framework from PyPI.

pip install l5kit

Demo – Visualization of Dataset via L5kit

This demo shows some visualization with the help of L5kit.  L5Kit contains two main components :

1. Rasterization: It contains classes for getting visual data as multi-channel tensors and turning them into interpretable RGB images.

2. Visualization: It contains utilities to draw additional information (e.g. trajectories) onto RGB images.

The following are the steps for the given demo:

1. Git clone the repository and change the working directory:

 !git clone https://github.com/lyft/l5kit.git
 %cd l5kit 

2. Run the .sh file to download the data and install all the dependencies:

!sh examples/setup_notebook_colab.sh

3. After step 2, a file named “dataset_dir.txt” will be formed in your current working directory. It contains the paths to the downloaded dataset. Open the file and copy the path. Add this path to your environment variable L5KIT_DATA_FOLDER. In our case, the path was /tmp/tmp.VsJInxjELG

 import os 
 os.environ["L5KIT_DATA_FOLDER"] = "/tmp/tmp.VsJInxjELG" 

4. Now, you can start with the actual code. Import all the required modules and packages. The code is available here.
5. Configure the path to your data files :

 # Dataset is assumed to be on the folder specified
 # in the L5KIT_DATA_FOLDER environment variable
 # get config
 cfg = load_config_data("./visualisation_config.yaml")
 print(cfg) 

6. You can check all the parameters mentioned in above configuration file by :

 print(f'current raster_param:\n')
 for k,v in cfg["raster_params"].items():
     print(f"{k}:{v}") 

7. Load the dataset.

 dm = LocalDataManager()
 dataset_path = dm.require(cfg["val_data_loader"]["key"])
 zarr_dataset = ChunkedDataset(dataset_path)
 zarr_dataset.open()
 print(zarr_dataset) 

8. Working with raw data : .zarr files support most of the traditional numpy array operations. In the following cell we iterate over the frames to get a scatter plot of the AV locations:

 frames = zarr_dataset.frames
 coords = np.zeros((len(frames), 2))
 for idx_coord, idx_data in enumerate(tqdm(range(len(frames)), desc="getting centroid to plot trajectory")):
     frame = zarr_dataset.frames[idx_data]
     coords[idx_coord] = frame["ego_translation"][:2]
 plt.scatter(coords[:, 0], coords[:, 1], marker='.')
 axes = plt.gca()
 axes.set_xlim([-2500, 1600])
 axes.set_ylim([-2500, 1600]) 

The output will be : 

Since .zarr files support numpy operations fully, you can identify the distribution of agent with the help of label_probabilities :

 agents = zarr_dataset.agents
 probabilities = agents["label_probabilities"]
 labels_indexes = np.argmax(probabilities, axis=1)
 counts = []
 for idx_label, label in enumerate(PERCEPTION_LABELS):
     counts.append(np.sum(labels_indexes == idx_label))
 table = PrettyTable(field_names=["label", "counts"])
 for count, label in zip(counts, PERCEPTION_LABELS):
     table.add_row([label, count])
 print(table) 

9. Working with data abstraction: If you do not wish to use raw data, L5kit contains some abstract classes to generate the input and the target. Here, buil_rasterizer converts the dataset into PyTorch ready datasets and EgoDataset iterates over AV annotations.

 rast = build_rasterizer(cfg, dm)
 dataset = EgoDataset(cfg, zarr_dataset, rast) 

10. Visualize the Autonomous Vehicle: Get a sample from the dataset and use `rasterizer` to get an RGB image that can be plotted.

 data = dataset[50]
 im = data["image"].transpose(1, 2, 0)
 im = dataset.rasterizer.to_rgb(im)
 target_positions_pixels = transform_points(data["target_positions"], data["raster_from_agent"])
 draw_trajectory(im, target_positions_pixels, TARGET_POINTS_COLOR, yaws=data["target_yaws"])
plt.imshow(im) 
plt.show()

11. If you want to change the rasterizer, for example to get an aerial view, you can do this by following code snippet : 

 cfg["raster_params"]["map_type"] = "py_satellite"
 rast = build_rasterizer(cfg, dm)
 dataset = EgoDataset(cfg, zarr_dataset, rast)
 data = dataset[50]
 im = data["image"].transpose(1, 2, 0)
 im = dataset.rasterizer.to_rgb(im)
 target_positions_pixels = transform_points(data["target_positions"], data["raster_from_agent"])
 draw_trajectory(im, target_positions_pixels, TARGET_POINTS_COLOR, yaws=data["target_yaws"])
 plt.imshow(im)
 plt.show()
 

12. If you want to visualize an agent, you can do it by changing the EgoDataset to AgentDataset.

 dataset = AgentDataset(cfg, zarr_dataset, rast)
 data = dataset[0]
 im = data["image"].transpose(1, 2, 0)
 im = dataset.rasterizer.to_rgb(im)
 target_positions_pixels = transform_points(data["target_positions"], data["raster_from_agent"])
 draw_trajectory(im, target_positions_pixels, TARGET_POINTS_COLOR, yaws=data["target_yaws"])
 plt.imshow(im)
 plt.show()
 

13. You can check this link, to show how the entire scene looks.

You can check the full tutorial in this Colab Notebook. Note: First three steps are not mentioned in this colab notebook. Add them, once you get started with this.

Demo – Agent Motion Prediction

This demo contains training and testing of baseline models to predict the future agents’ trajectories. The first two steps from the above demo are to be repeated here as well then follow the following procedure : 

1. Import all the required modules and packages. The code snippet is available here.
2. Prepare the data path and load the config. Replace the “PATH_TO_DATA” by the path mentioned in dataset_dir.txt present in the current working directory.

 # set env variable for data
 os.environ["L5KIT_DATA_FOLDER"] = "PATH_TO_DATA"
 dm = LocalDataManager(None)
 # get config
 cfg = load_config_data("./examples/agent_motion_prediction/agent_motion_config.yaml")
 print(cfg) 

3. Build the model : The baseline model is a simple resnet50 pre trained on imagenet. The model architecture is given here.
4. Load the train data : 

 # ===== INIT DATASET
 train_cfg = cfg["train_data_loader"]
 rasterizer = build_rasterizer(cfg, dm)
 train_zarr = ChunkedDataset(dm.require(train_cfg["key"])).open()
 train_dataset = AgentDataset(cfg, train_zarr, rasterizer)
 train_dataloader = DataLoader(train_dataset, shuffle=train_cfg["shuffle"], batch_size=train_cfg["batch_size"], 
                              num_workers=train_cfg["num_workers"])
 print(train_dataset)
 Initialize the model and specify the device, optimizer loss criteria :
 # ==== INIT MODEL
 device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
 model = build_model(cfg).to(device)
 optimizer = optim.Adam(model.parameters(), lr=1e-3)
 criterion = nn.MSELoss(reduction="none") 

5. Start the training loop  : 

 # ==== TRAIN LOOP
 tr_it = iter(train_dataloader)
 progress_bar = tqdm(range(cfg["train_params"]["max_num_steps"]))
 losses_train = []
 for _ in progress_bar:
     try:
         data = next(tr_it)
     except StopIteration:
         tr_it = iter(train_dataloader)
         data = next(tr_it)
     model.train()
     torch.set_grad_enabled(True)
     loss, _ = forward(data, model, device, criterion)
     # Backward pass
     optimizer.zero_grad()
     loss.backward()
     optimizer.step()
     losses_train.append(loss.item())
     progress_bar.set_description(f"loss: {loss.item()} loss(avg): {np.mean(losses_train)}") 

6. Once the training is done, plot the loss curve :

 plt.plot(np.arange(len(losses_train)), losses_train, label="train loss")
 plt.legend()
 plt.show()
 

7. For evaluation, load the chopped dataset, the link for its description is available here.

 # ===== GENERATE AND LOAD CHOPPED DATASET
 num_frames_to_chop = 100
 eval_cfg = cfg["val_data_loader"]
 eval_base_path = create_chopped_dataset(dm.require(eval_cfg["key"]), cfg["raster_params"]["filter_agents_threshold"], 
                               num_frames_to_chop, cfg["model_params"]["future_num_frames"], MIN_FUTURE_STEPS) 

    The full code for the evaluation step is available here.

8. Store the predictions for the above generated dataset. The code snippet is given here.
9. After saving the results,  calculate the performance evaluation.

 metrics = compute_metrics_csv(eval_gt_path, pred_path, [neg_multi_log_likelihood, time_displace])
 for metric_name, metric_mean in metrics.items():
     print(metric_name, metric_mean) 

10. Now, the last step is to visualize the results. The code is available here and it prints every 100^th scene.

Demo – Planning for Self-Driving Vehicles

Usage of Lyft’s Framework

• Convert predictions, planning & simulation problems into ML problems.
• Use of Neural Network (NN) to model important components of Autonomous Vehicle stack(user experience, distribution, software, cloud, etc).
• Use previously observed data points to make inference for future motion prediction.
• Enhance the performance of the model as the amount of data increases.
• Plan behavior of an AV to imitate human driving.

Conclusion

In this blog, we have discussed Lyft’s framework for ml-prediction, planning & simulation for self-driving. 

Official Links are as follows : 

• Github
• Documentation
• Research Paper