How often have you found some important files archived in some weird folder of your computer? Why do we humans tend to forget things? What good is it? Unlike the human brain, a computer is going to remember the folders and documents you feed it—unless hit by a virus. Regardless of its forgetfulness, the human brain functions quite well. This is because our brain tends to remember what it deems essential. Maybe forgetfulness is a feature and not a bug. Now, what if machines began to work that way too?
Facebook has a plan
Facebook’s AI team has announced, through a blog post, a new tool called Expire-Span, which would allow neural networks to forget at scale.
Why would Facebook do this? Computers will keep storing your information for a long time—even if it is something as irrelevant as notes for a class test you gave five years ago. However, a user wanting to feed further information to a machine would then mean that AI systems would need to expend more computing resources to handle the increase in data, which is unsustainable in terms of cost and lowers the efficiency of these machines. Previous attempts at solving this problem involved compressing information to make it smaller in size, which tackles the need to reduce data but distorts the data at hand(think: image pixelation). This is where Facebook’s Expire-Span comes into play. According to the team, Expire-Span would help build an environment that would enable the alleviation of these problems in machines and AI without compromising the quality of important information.
About Expire-Span
Expire-Span works by first predicting information that it deems most relevant to the task at hand. It does this based on contextual information from data and by memories surrounding this information. From this, it places expiration dates on the piece of information based on its relevance. Thus, more helpful information will stay on longer than information that is not as vital. Once a piece of information reaches past its expiry date, it disappears from the AI system. This opens up memory space which can then be used to process the retained important information more efficiently (i.e. faster and at a much larger scale). An example provided by Facebook was of a model learning to perform a word prediction task. In such a scenario, the AI could be taught to remember rare words such as names but forget filler and trivial words like ‘as’ and ‘of’. Expire-Span would then use context to predict what words belong to the trivial filler category and what terms will be remembered.
The model has also been shown to be efficient. Facebook tested it compared to previous models, Adaptive-Span and Compressive Transformer, on context-based tasks such as language modelling and moving objects and found Expire-Span to be faster and more efficient (based on the load on GPU memory) than the other models.
One very brainy AI
This whole body of work on Expire-Span was inspired by how the human brain functions. When AI is presented with too much data, it undergoes the risk of being overwhelmed by its sheer volume. Expire-Span is designed to operate more like a brain and only remember what is important. Of course, the human brain is much more complicated than a machine learning algorithm. For one, the human brain has different types of information that it feels is important. For example, a handmade sweater by your grandmother is going to remain a special memory regardless of how often you wear it. Expire-Span may not possess this sentimentality that comes with humans and may declare something important to the user irrelevant because it has not been used in a while.
Another issue with Expire-Span is how Facebook did not mention whether the tool will notify users that it will delete their information permanently. Some people believe that Expiry-Span will not delete one’s data itself but will instead ask the user whether a file can be deleted on the expiration date. Another speculation is whether these files will be deleted or sent to a form of recycling bin instead? As of now, Facebook has not answered these questions.
In light of the above concerns, it becomes imperative to note that Facebook is still in the research phase of Expire-Span, and may continue to research and fix any bugs it encounters. Additionally, Facebook’s AI team is also looking into making Expire-Span turn AI more like a brain by incorporating different types of memories into neural networks. This makes for an exciting and innovative venture into the realm of artificial intelligence. Who knows, kids could one day use ‘my computer lost my homework’ as an excuse to get out of homework!
