|
Listen to this story
|
Hugging Face and ServiceNow, two major players in AI, have partnered to develop a new open-source language model for codes called StarCoder. The model created as a part of the BigCode initiative is an improved version of the StarCoderBase model trained on 35 billion Python tokens.
Researchers stated, StarCoder’s capabilities have been tested on a range of benchmarks, including HumanEval benchmark for Python. The model has outperformed larger models like PaLM, LaMDA, and LLaMA, and has proven to be on par with or even better than closed models like OpenAI’s code-Cushman-001 (the original Codex model that powered early versions of GitHub Copilot).
Trained on over 1 trillion tokens and with a context window of 8192 tokens, the model boasts an impressive 15.5 billion parameters. It was created using data from GitHub, including 80+ programming languages, Git commits, GitHub issues, and Jupyter notebooks. This vast dataset was preprocessed to include only content with permissive licenses, ensuring that the resulting model can generate source code while adhering to legal criteria.
StarCoder’s primary function is as a technical assistant, generating realistic code and supporting 80 programming languages. However, it is not designed for issuing instructions or directives like “write a function that computes the square root.” Instead, users can follow on-screen prompts to transform StarCoder into a helpful programming tool.
Despite its impressive capabilities, StarCoder has its set of limitations. Like other LLMs, it can produce incorrect and offensive information. To address these concerns, the researchers have released the StarCoder models under an Open Responsible AI Model (RAIL) license and have open-sourced all code repositories for creating the model on GitHub. To ensure that the model adheres to responsible AI principles, the model license includes usage restrictions, and a set of attribution tools is available to end-users to identify potentially plagiarized model generations.
Notably, StarCoder is not the first mover in the domain. LLM-based coding platforms are continuing to improve themselves, with Google researchers demonstrating earlier this month that they can be used to self-debug. Apart from Microsoft’s GitHub Copilot and Amazon CodeWhisperer, the India-based Replit has also joined the LLM race.
