∫ntegrabℓε ∂iﬀerentiαℓs · unorganised notes, code, and writings of random topics

Kirillov et al (2023) Segment Anything

April 4, 2025 paper code

The goal of the paper is to produce a promptable segmentation model. It is a model that supports flexible prompting and can output segmentation masks in real time. For any given segmentation prompt (e.g., a point in the image, and may be ambiguous), the model is expected to return a... [more]

Dosovitskiy et al (2021) An Image is Worth 16x16 Words

March 26, 2025 paper code math

This is the paper that introduced the Vision Transformer (ViT), which proposed that transformers can be used for image classification replacing CNNs. Inspired by the success of transformer models in NLP, this paper explored the technique of using transformers to process 2D image data. The goal is to create a... [more]

An Intuitive Analogy to Attention Operation

February 20, 2025 blog math

I tried to explain what is attention, but there should be a simpler explanation of what the attention operation is actually doing. Let’s focus on the core operation of attention, namely: [more]

Kwon et al (2023) PagedAttention

February 5, 2025 paper

This is the paper that proposed PagedAttention and crafted the design of vLLM. The authors pointed out that LLMs are autoregressive. One token is generated from the prompt concatenated with the previously generated sequence. The KV cache (described as “incremental multi-head attention” in the GQA paper) is to share the... [more]

Ainslie et al (2023) GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

February 1, 2025 paper

This is the paper that proposed the Grouped-Query Attention (GQA). While MQA, by sharing the $K,V$ tensors in attention, speeds up the decoder inference, it is found to degrade in quality. GQA is a generalization of MQA. Just like group norm is a generalization between instance norm and layer norm,... [more]