Article complete

Hugging Face Releases A State-of-the-Art LLM For Code

The Hugging Face model is an improved version of the StarCoderBase model trained on 35 billion Python tokens. 

A smiling emoji with open hands on a blue background, surrounded by quotation marks.

Key takeaways

  • Hugging Face and ServiceNow have collaborated to create StarCoder, an advanced open-source language model for code development.
  • StarCoder is an enhanced version of the StarCoderBase model, trained on 35 billion Python tokens and has demonstrated superior performance on several benchmarks compared to larger models.
  • The model has been trained on a diverse dataset exceeding 1 trillion tokens, including over 80 programming languages, ensuring legal compliance in code generation.
  • StarCoder serves primarily as a technical assistant for generating realistic code rather than providing explicit instructions for programming tasks.
  • Despite its strengths, StarCoder has limitations and is not intended for issuing direct programming commands.

Takeaways drafted with AI from the article text.

Researchers stated, StarCoder's capabilities have been tested on a range of benchmarks, including HumanEval benchmark for Python. The model has outperformed larger models like PaLM, LaMDA, and LLaMA, and has proven to be on par with or even better than closed models like OpenAI's code-Cushman-001 (the original Codex model that powered early versions of GitHub Copilot).

Trained on over 1 trillion tokens and with a context window of 8192 tokens, the model boasts an impressive 15.5 billion parameters. It was created using data from GitHub, including 80+ programming languages, Git commits, GitHub issues, and Jupyter notebooks. This vast dataset was preprocessed to include only content with permissive licenses, ensuring that the resulting model can generate source code while adhering to legal criteria.

StarCoder's primary function is as a technical assistant, generating realistic code and supporting 80 programming languages. However, it is not designed for issuing instructions or directives like "write a function that computes the square root." Instead, users can follow on-screen prompts to transform StarCoder into a helpful programming tool.

Despite its impressive capabilities, StarCoder has its set of limitations. Like other LLMs, it can produce incorrect and offensive information. To address these concerns, the researchers have released the StarCoder models under an Open Responsible AI Model (RAIL) license and have open-sourced all code repositories for creating the model on GitHub. To ensure that the model adheres to responsible AI principles, the model license includes usage restrictions, and a set of attribution tools is available to end-users to identify potentially plagiarized model generations.

Notably, StarCoder is not the first mover in the domain. LLM-based coding platforms are continuing to improve themselves, with Google researchers demonstrating earlier this month that they can be used to self-debug. Apart from Microsoft’s GitHub Copilot and Amazon CodeWhisperer, the India-based Replit has also joined the LLM race. 

Read: Replit Knows India Better than Amazon, Microsoft

Published

Spotted an error? Tell our editors. Read our editorial standards and corrections.

More in this story

Hugging Face · 5 stories
  1. The dazzling ascent of Hugging Face
  2. Hugging Face raises USD 100 Mn in Series C
  3. ChatGPT Clone for Just $300
  4. You are here: Hugging Face Releases A State-of-the-Art LLM For Code
  5. What Does Open Source Mean for the Lucrative RL Environment Business?
About the author

Contributor

All stories
Belamy · Daily

The day’s AI news, before your first meeting.