Article complete

A New Language Model Trained on Dark Web Emerges

DarkBERT finds its use cases in researching dark web, alongside identifying cyber security threats like Ransomware and leak site detection and others.

A hooded figure sits in front of a laptop, shrouded in darkness, suggesting a clandestine online activity.

Key takeaways

  • DarkBERT is a language model specifically designed for the dark web, based on the RoBERTa architecture.
  • The model excels in tasks related to cyber security, including ransomware detection and leak site identification.
  • DarkBERT was trained on a large-scale dark web corpus from the Tor network, ensuring relevance to dark web language.
  • Researchers implemented data filtering and pre-processing to mitigate ethical concerns associated with sensitive dark web content.
  • Previous studies by the same researchers have contributed to understanding the language and activities on the dark web.

Takeaways drafted with AI from the article text.

Researchers from South Korea recently released DarkBERT, a dark web domain-specific language model based on the RoBERTa architecture. 

This new model is said to show promising applicability in future research in the dark web domain and in the cyber security industry. It also outperformed existing language models with evaluations on dark web domain tasks and datasets. 

But, how did they do it? To allow DarkBERT to adapt well to the language used in the Dark Web, the researchers have pre-trained the model on a large-scale dark web corpus collected by crawling the Tor network. In addition to this, they also polished the pre-training corpus through data filtering and deduplication, alongside data pre-processing to address the potential ethical concerns in Dark Web texts related to sensitive information. 

Showcasing DarkBERT pretraining process and the various use case scenarios for evaluation. (Source: arXiv) 


The same group of researchers last year worked on ‘Shedding New Light on the Langauge of the Dark Web,’ where they introduced CoDA, a dark web text corpus collected from various onion services divided into topical categories. Another notable study includes – ‘The Language of Legal and Illegal Activity on the Darknet,’ done by Israeli researchers, where they identified several distinguishing factors between legal and illegal texts, taking a variety of approaches. This includes predictive (text classification), and application-based (named entity Wikification), along with an approach based on raw statistics. 

All of this research work and more has inspired the researchers to develop DarkBERT. 

What next? 

In the coming months, the researchers said that they plan to improve the performance of dark web domain-specific pre-trained language models using more latest architectures and crawl additional data to allow the construction of multilingual language models. 

Published

Spotted an error? Tell our editors. Read our editorial standards and corrections.

More in this story

AI (Artificial Intelligence) · 5 stories
  1. Facebook To Set Up Institute For AI Ethics With Initial Investment Of $7.5 Million
  2. A Step-by-Step Guide To Build ML Models For Research
  3. The New Windows Store To Support Third-party App Stores
  4. Alethea.AI Introduces CharacterGPT, First Multimodal Text-to-Character Generator
  5. You are here: A New Language Model Trained on Dark Web Emerges
About the author

Journalist

Mohit writes about AI in simple, explainable, and often funny words. He's especially passionate about chatting with those building AI for Bharat, with the occasional detour into AGI.

All stories
Belamy · Daily

The day’s AI news, before your first meeting.