Photos are matrices of pixels. Each pixel can be a tuple of RGB values (the most common) or other color spaces such as Lab or HSV. RGB is great for its additive nature. HSV, however, considered hue as an angle in some arbitrary order. It is better than RGB if...
[more]
Peng et al (2023) YaRN: Efficient Context Window Extension of Large Language Models
This paper studies the positional encoding in large language models, since it is the factor that needs to change if we want to support a longer context length than it was pretrained. The original transformer model’s position encoding scheme was called the absolute sinusoidal position encoding. Then, there are models...
[more]
Reading the LLaMA code
The LLaMA and LLaMA 2 model released by Meta/Facebook is available on GitHub and there’s a guide to help you using it. Of course, from Meta, this model is using PyTorch. But surprisingly, the repo on GitHub is very short that you can read and understand it in a day...
[more]
Naveed et al (2023) A Comprehensive Overview of LLMs
This is a survey paper of the current status of large language models. It is a
nice read because it is short (37 pages only) and covered some key components
needed to understand what these models are about.
[more]
Self-hosted Copilot for Your VSCode
GitHub has its Copilot service that we can pay a subscription for. It is a coding assistant in your IDE, which requires a plugin on your editor and performs auto-complete for the code you typed. There are off-the-shelf language models that can generate code like GitHub’s Copilot. But the model...
[more]