Guide To Twitter’s Image Crop Analysis

The developers of this system focused on an automated image cropping system that automatically crops images that users submit on Twitter to show image previews with respect to different aspect ratios across multiple devices such as a tablet, mobile phones, and desktop.

Automated image cropping is a task that constructs a viewport based on the dimension or aspect ratio of a given image that crops the image. It fits the aspect ratio by preserving the most relevant or interesting part of the image within the viewport. It has a wide range of applications, e.g. in the cinematography and broadcasting industry; the film footage’s aspect ratio is changed to match the display aspect ratio. Many modern platforms operate on multiple types of devices that require varying aspect ratios, increasing the number of crops needed even on a single image.

There is not any ideal system. This automated image cropping system is built on machine learning technology designed to ensure that average error is low but do not consider the disparate impact of cases where the error is not uniformly distributed across demographic groups.  

The developers of this system focused on an automated image cropping system that automatically crops images that users submit on Twitter to show image previews with respect to different aspect ratios across multiple devices such as a tablet, mobile phones, and desktop. The system employees supervised machine learning model on existing saliency maps to predict saliency score over any given input image. In computer vision, a saliency map is an image that shows each pixel’s unique quality. A saliency map aims to simplify and change the representation of an image into something more meaningful and easier to analyze.

AIM Daily XO

Join our editors every weekday evening as they steer you through the most significant news of the day, introduce you to fresh perspectives, and provide unexpected moments of joy
Your newsletter subscriptions are subject to AIM Privacy Policy and Terms and Conditions.

The saliency score is meant to capture the importance of each region of the image. After having a saliency score, the model selects the crop by trying to centre it around the most salient point, shifting to stay within the original image.

Source: Official Release  

Twitter’s image cropping algorithm relies on a machine learning model trained to predict saliency. The Twitter algorithm finds the most salient point, and then a set of heuristics is used to create a suitable center crop around that point for a given aspect ratio. 

Download our Mobile App

Cropping is conducted as follow;

  1. For a given image, the image is discretized into a grid of points, and each grid point is associated with a saliency score predicted by the model. 
  1. The image and the coordinates of the most salient point and the desired aspect ratio are passed as an input to a cropping algorithm. This is repeated for each aspect ratio to show the image on multiple devices.
  1. If the saliency map is almost symmetric horizontally, then a center crop is performed irrespective of the aspect ratio.
  1. Otherwise, the cropping algorithm tries to ensure that the most salient point is within the crop with the desired aspect ratio. This is done by cropping only one dimension.

The model is trained by using WikiData Query Service consisting of images and labels of celebrities in wikidata. WikiCelb consists of images of individual Wikipedia pages obtained through the wikidata query service. It contains 4073 images of individual labels along with gender identity and ethnic group as recorded on wikidata.

More details regarding this research can be found here.

Code Implementation of Image Crop Analysis:

Load all dependencies

 import logging
 import shlex
 import subprocess
 import sys
 from collections import namedtuple
 from pathlib import Path
 import matplotlib.image as mpimg
 import matplotlib.pyplot as plt
 import numpy as np
 from matplotlib.collections import PatchCollection
 from matplotlib.patches import Rectangle 

Load repository and create a virtual environment for google colab, for local machine follow the instruction mentioned in the repository.

 import platform
 BIN_MAPS = {"Darwin": "mac", "Linux": "linux"}
 HOME_DIR = Path("../").expanduser()
     import google.colab
     ! pip install pandas scikit-learn scikit-image statsmodels requests dash
     ! [[ -d image-crop-analysis ]] || git clone
     HOME_DIR = Path("./image-crop-analysis").expanduser()
     IN_COLAB = True
     IN_COLAB = False
 sys.path.append(str(HOME_DIR / "src"))
 bin_dir = HOME_DIR / Path("./bin")
 bin_path = bin_dir / BIN_MAPS[platform.system()] / "candidate_crops"
 model_path = bin_dir / "fastgaze.vxm"
 data_dir = HOME_DIR / Path("./data/")

Put the image at ‘/content/image-crop-analysis/data’; by default, there will dummy.jpeg image.

Mind the extension carefully; extension of your uploaded image and extension defined in the directory should be the same as below.

 img_path = next(data_dir.glob("./*.jpg"))



See the uploaded image.

 img = mpimg.imread(img_path)
     Rectangle((0, 0), 200, 112, linewidth=1, edgecolor="b", facecolor="none")) 

Load and run the model 

 from crop_api import ImageSaliencyModel
 model = ImageSaliencyModel(crop_binary_path=bin_path, crop_model_path=model_path)
 for img_path in data_dir.glob("*.jpg"):


From the above output, the automated image crops a yellow rectangle on the image. The leftmost image is a heat map of the saliency score produced by the Twitter Image Cropping model on the image. The location with high predicted saliency could be humans, objects, texts and high contrast background.

The other is original images with their maximum saliency point as a yellow dot, and the automatic cop yellow rectangle is produced for different aspect ratio. 


This article discussed mainly Twitter’s saliency-based image cropping algorithm, which automatically crops images to different aspect ratios by centering crops around the most salient area. The different aspect ratio is produced based on saliency score as shown in the above output. However, this research addresses some major issues faced by previous algorithms, such as cropping light-skinned over dark-skinned individuals and favouring cropping women’s body overhead. One who is interested in detailed research can refer to the paper from the reference below.


Sign up for The Deep Learning Podcast

by Vijayalakshmi Anandan

The Deep Learning Curve is a technology-based podcast hosted by Vijayalakshmi Anandan - Video Presenter and Podcaster at Analytics India Magazine. This podcast is the narrator's journey of curiosity and discovery in the world of technology.

Vijaysinh Lendave
Vijaysinh is an enthusiast in machine learning and deep learning. He is skilled in ML algorithms, data manipulation, handling and visualization, model building.

Our Upcoming Events

24th Mar, 2023 | Webinar
Women-in-Tech: Are you ready for the Techade

27-28th Apr, 2023 I Bangalore
Data Engineering Summit (DES) 2023

23 Jun, 2023 | Bangalore
MachineCon India 2023 [AI100 Awards]

21 Jul, 2023 | New York
MachineCon USA 2023 [AI100 Awards]

3 Ways to Join our Community

Telegram group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our Daily newsletter

Get our daily awesome stories & videos in your inbox