Facebook AI has announced the release of ‘SaLinA,’ a lightweight library for implementing sequential decision models, including reinforcement learning (RL) algorithms. 

​​According to Ludovic Denoyer, a research scientist at Facebook, “SaLinA is a Pytorch modification that allows users to combine agents instead of modules, giving the computation a time dimension. Classic RL algorithms may be constructed in a few lines using this abstraction, and they are not dependent on policy designs.”

SaLinA is a lightweight library for developing sequential decision models that extends PyTorch components. It can be used for RL, as well as in supervised/unsupervised learning scenarios.

• It enables the rapid development of extremely complicated sequential models (or policies) in a few lines.
• It is compatible with multiple CPUs and GPUs.

Source: SaLinA

What is SaLinA

• A sandbox for developing large-scale sequential models.
• A simple (300 hundred lines) ‘core’ programme that defines all of the components necessary to construct agents in sequential decision learning systems.
• It is simple to comprehend and use because it adheres to Pytorch’s fundamental ideas, just extending nn.Module to an agent to handle the temporal dimension.
• A collection of agents that may be combined in various ways (similar to Pytorch modules) to produce complex behaviours.
• A collection of implementations and samples from many fields. There are several types of learning: RL, imitation learning, and computer vision, with many more to come.

Why SaLinA

SaLina’s goal is to make the implementation of sequential decision processes, particularly those including RL, natural and easy for practitioners who have a working knowledge of how neural networks can be implemented. SaLina aims to handle any sequential decision problem by employing simple ‘agents’ that progressively process data. The intended audience includes researchers in natural language processing or computer vision and experts in natural language processing who are looking for a more natural way to model conversations in their models, trying to make them more straightforward and easily understood than previous methods.

Advantages of SaLinA

• Simplicity: A working knowledge of the Agent and Workspace APIs is sufficient to comprehend SaLinA and create complicated sequential decision models. There are no hidden mechanics, and the two classes are extremely straightforward and intuitive to anyone who has used PyTorch.
• Modularity: SaLinA enables the construction of sophisticated agents through the use of predefined container agents.
• Flexibility: SaLinA’s flexibility is enhanced by the addition of tools that aid in the implementation of complex models. SaLinA includes wrappers for capturing openAI Gym settings, DataLoader environments, and Brax environments as agents, enabling rapid development of a diverse set of models. 
• Scalability: SaLinA includes an NRemoteAgent wrapper that allows any agent to be executed over several processes, significantly speeding up the computation of any individual agent. When combined with the ability to run agents on either CPU or GPU, the library can scale to very big problems with only a few changes to the code.

Additional features of SaLinA

• Speed: SaLinA is a complete Python library that incurs low overheads and performs on a par with existing alternatives.
• From policies to recurring policies: The workspace concept enables the easy implementation of complicated policies without modifying any other code.
• Replay Buffer: There is no need to design a sophisticated replay buffer class in SaLinA, as a collection of workspaces can naturally be utilised as a replay buffer due to the agents’ playback capability.
• Batch RL: It is simple to compute complex losses across defined trajectories utilising SaLinA’s replay feature. 
• Model-based RL: Because environments in SaLinA are agents, it is feasible to replace any environment agent at any time with an agent that models the world. 
• Multi-agent RL: SaLinA naturally supports multi-agent settings by integrating several agents into a single one.
• The SaLinA RL benchmark: SaLinA’s RL benchmark currently contains implementations of Double DQN, Reinforce, and Behavioral Cloning. 

Conclusion

By integrating agents, SaLinA enables the implementation of sequential decision-making algorithms in a novel method. It is a small library, extremely flexible, and scalable. It enables the creation of new algorithms and the rapid evaluation of novel ideas without losing training or testing speed. 

Future directions include the following: 

a) enabling the execution of agents on remote computers; 

b) developing new tools for agent implementation; and 

c) establishing algorithms in various disciplines.