Stability AI has introduced Stable Code 3B, a pre-trained 2.7 billion parameter decoder-only language model on a staggering 1.3 trillion tokens, encompassing a rich tapestry of textual and code datasets.
One of the standout features of Stable Code 3B is its Fill in the Middle (FIM) capability. Unlike traditional code completion models that suggest single lines, Stable Code 3B can now seamlessly complete larger missing sections of code. This breakthrough opens up new possibilities for developers, allowing them to bridge gaps in their codebase and enhance productivity effortlessly.
Check out code and datasets here.
Drawing inspiration from the 2023 StackOverflow Developer Survey, the model has been trained in 18 programming languages, including popular choices like Python, Java, JavaScript, and C++. This breadth of language coverage positions Stable Code 3B as a versatile solution catering to the needs of a vast developer community.
With a colossal dataset comprising 1.3 trillion tokens sourced from GitHub repositories, books, and websites, Stable Code 3B underwent a rigorous training regimen. Masked language modelling, a technique where the model predicts missing words within code snippets, played a crucial role in shaping the decoder-only language model.
More about the model
The model showcases its performance, especially compared to similar models. For instance, it distinguishes itself among coding language models through its performance, achieving comparable accuracy to larger models like CodeLLaMA on MultiPL-E metrics across various programming languages.
Stable Code 3B incorporates RoPE (Recurrent Processing Elements), expanding its context size to 100,000 tokens. This enhancement ensures a deeper understanding of lengthy code sequences, enabling the model to grasp intricate relationships within extensive codebases. The result is a code completion tool that doesn’t just understand lines of code but comprehensively interprets the nuances of entire programs.
Despite its smaller size, Stable Code 3B leverages the foundation of Stable LM 3B, inheriting a wealth of general language understanding. Additionally, it excels at code translation, seamlessly bridging gaps between different programming languages. The model’s prowess extends to answering code-related questions, providing valuable insights into functionality and bug-fixing.
Validation comes from MultiPL-E metrics, and the model has proven its mettle across various programming languages, as tested using BigCode’s Evaluation Harness. Its applications include code generation, which can craft anything from functions to entire programs.
The model’s open-source availability on platforms like Hugging Face promotes transparency and community collaboration, fostering ongoing improvements and innovations. Subscribers gain access to a suite of AI tools, including SDXL, StableLM Zephyr, Stable Audio, and Stable Video. Its notable efficiency allows it to run effectively on standard laptops without GPUs, enhancing accessibility for a broader user base.
