Building customised builds that can match the complexity exhibited by the real-world or imagined scenes such as pictures, sketches, or physical objects was a challenge. A group of researchers from MIT have come up with a solution. The paper titled ‘Image2Lego: Customised LEGO Set Generation from Images’ unveils Image2Lego. 

LEGO is a Danish consumer product company engaged in manufacturing toys for children, video and online games. It also has products for children to help them improve their creativity and innovative playing and learning activities.

The 3D model generation from images and LEGO generation from 3D models has always remained an uphill task. The model presented in the paper has bridged this gap and come out with a complete Image2LEGO pipeline. Hence, it opens the gate for those who wish to create custom LEGO models from 2D images. Simply put, Image2Lego is an AI to create buildable 3D LEGO from single 2D images. 

The work includes three sequential components:

• converts a 2D image to a latent representation,
• decodes the latent representation to a 3D voxel model, and
• applies an algorithm to transform the voxelised model to 3D LEGO bricks.

Methodology

The presented model is the combination of:

• Octree Generating Network, consisting of a convolutional block in an ‘octree’ structure that sequentially up- (or down-) samples the resolution of a 3D model by a factor of two in each spatial dimension.
• TL-Embedding network accomplishes image-to-3D reconstruction by simultaneously training an autoencoder to compress the 3D model representation into a latent vector and an image encoder to predict the latent representation of a model from a single 2D image.

“We begin by training an autoencoder on the voxelised 3D models. The decoder of this network is adapted from the Octree Generating Network. It consists of multiple blocks, each of which upsamples the resolution by a factor of two in each dimension (i.e. an octree structure) and changes the number of channels, beginning with the latent representation, parameterised as a 256-dimensional vector (i.e. a single voxel with 256 channels),” says the paper.

In the next step, they train a separate encoder network to predict the 256-dimension latent representations learned from the 3D autoencoder using the rendered single 2D images of the objects. Additionally, researchers use AlexNet – a seminal image classification, to design their encoder. Now, in the final step, voxelised 3D models are converted to LEGO builds. This is executed with the help of an algorithm adapted from the ColouredVoxels2Lego project. The algorithm converts each voxel to a 1-by-1 LEGO brick and then iterates through each layer in the z-coordinate of the model to find optimal groups of bricks to be combined into larger cuboid bricks.

It is crucial to consider that colour is a critical factor in the LEGO model of faces. Since the TL-Octree network cannot add colours, LEGO reconstruction of faces was done via Volumetric Regression Network (VRN). However, it produces some patchiness to the colour. As a result, the k-means clustering algorithm got involved in giving uniformity to the colours. “We believe this work will greatly improve the accessibility of creative, personalised LEGO creations for builders of all skills and all ages,” concluded the paper.

Way Forward

Bringing AI can further help LEGO drive its business growth. Lego was ranked as one of the major toy companies globally in 2019, based on revenue, ahead of companies such as Mattel and Hasbro. During the last decade (2010 – 2020), the company has seen a rise in revenue from 2.2 billion euros to 5.87 billion euros, thereby a jump of over 166 per cent, as per the industry report. The newly introduced approach equips the company to come out with several innovative AI-based gaming experiences to further fuel its revenue growth.