Article complete

Google Adds Fast Wordpiece Tokenization To Tensorflow

Google’s LinMaxMatch approach improves performance, makes computation faster and reduces complexity

Google Adds Fast Wordpiece Tokenization To Tensorflow

Key takeaways

  • Google's Fast WordPiece Tokenization enhances the tokenisation process, significantly saving computing resources.
  • The new LinMaxMatch approach improves computation speed, achieving up to 8x faster performance compared to traditional methods.
  • This tokenisation method reduces complexity, making it more efficient for natural language processing tasks.
  • The technique adopts a linear tokenisation time, optimising the process and minimising repetitive iterations.
  • Google has integrated this advancement into TensorFlow Text, broadening its application across various systems.

Takeaways drafted with AI from the article text.

Using the greedy longest-match-first strategy to tokenise a single word, WordPiece iteratively picks the longest prefix of the remaining text that matches a word in the model’s vocabulary. This approach is called MaxMatch and has been used in Chinese word segmentation since the 1980s. Despite its wide use in NLP, it is still computation intensive. Google has proposed an alternative to this – the LinMaxMatch, which has a tokenisation time that is strictly linear with respect to n.

In other terms, if tire-matching cannot match an input character for a given node, the standard algorithm backtracks to the last character where a token was matched and restarts the trie matching procedure from there. This results in repetitive and wasteful iterations. Instead of backtracking, Google’s method triggers a ‘failure transition’ in two steps. It first collects the precomputed tokens stored at that node, known as ‘failure pops’ and then follows the precomputed ‘failure link’ to a new node from where the trie matching process continues. As n operations are required to read the entire input, LinMaxMatch is asymptotically optimal for the MaxMatch problem.

As the existing systems pre-tokenise the input text by splitting it into words by whitespace, punctuation and characters, and then call WordPiece tokenisation on each resulting word, Google has proposed an end-to-end WordPiece tokeniser. It combines pre-tokenisation and WordPiece into a single, linear-time pass. It uses the LinMaxMatch trie-matching and failure transitions and only checks for whitespace and punctuation characters among the few input characters that are not handled by the loop. This makes it more efficient as it traverses the input only once, performs fewer whitespace/punctuation checks, and skips the creation of intermediate words.

Published

Corrections: AIM corrects errors of fact and notes the change on the story. Spotted one? Tell the editors. Read our editorial standards.

More in this story

Google · 5 stories
  1. Beginners Guide To Text Generation With RNNs
  2. Difference Between PyTorch And PySyft
  3. You are here: Google Adds Fast Wordpiece Tokenization To Tensorflow
  4. Soon After $1 Bn OpenAI Deal, Disney Accuses Google of AI Copyright Infringement
  5. Google’s Gemma 4 and Medical Data Toolkit Power NHA's Aarogya Setu 2.0
About the author

Contributor

All stories
Belamy · Daily

The day’s AI news, before your first meeting.