Facebook parent company Meta has introduced an AI-based tool for coding, called Code Llama.
A large language model (LLM) that can use text prompts to generate code, Code Llama is a code-specialized version of Llama 2. It was built by further training on code-specific datasets, sampling more data from the same dataset for a longer period. Code Llama can generate code and natural language about code from both code and natural language prompts, such as “Write me a function that outputs the fibonacci sequence.” The tool also can be used for code completion and debugging. Languages supported include Python, C++, Java, PHP, TypeScript, JavaScript, C#, and Bash.
Developers can request access to Code Llama from the Meta AI webpage. Free for research and commercial usage, Code Llama is being released in three sizes, with 7B, 13B, and 34B parameters respectively. Each model is trained with 500B tokens of code and code-related data. The 7B and 13B base and instruct models have been trained with fill-in-the-middle (FIM) capability, enabling insertion of code into existing code. This supports tasks such as code completion out of the box.
The three models address different serving and latency requirements, with the 7B model, for example, served on a single GPU while the 34B model returns the best results and allows for better coding assistance. Meta has fine-tuned two additional variations of the tool: Code Llama – Python was further fine-tuned on 100B tokens of Python code, and Code Llama – Instruct was fine-tuned to understand natural language instructions.