Paged Attention and vLLM
Published:
Paged Attention is a memory optimization on which the vLLM Inference Engine is based. Here is a summary of the paper on paged attention and the key features of vLLM that make it so powerful.
Posts by Tags
Attention
Autoencoders
Are Autoencoders Fundamentally Denoisers?
Published:
The core idea behind Autoencoders is to bottleneck information flow so that the DNN is forced to prioritize what information to propagate to the next layer (by restricting the number of dimensions in the latent space). In this project, I explore how this can be a useful denoising tool.
Autoregressive Training
Implementing GPT from Scratch
Published:
This article contains a conceptual explanation, necessary for building a language model from scratch, using the decoder-only transformer architecture. It is based on Andrej’s Karpathys GPT from scratch. The code for this conceptual guide can be found here.
Circuits
Review: Interpretability in the Wild: A Circuit for Indirect Object Detection in GPT2-Small
Published:
A paper review highlighting the key discoveries with respect to attention heads and the algorithms used.
Review: A Mathematical Framework for Transformer Circuits
Published:
This paper provides a mental model for reasoning about the internal workings of transformers and attention heads in deep neural networks. The insights here help understand and analyze the behaviors of large models.
Deep Learning
Are Autoencoders Fundamentally Denoisers?
Published:
The core idea behind Autoencoders is to bottleneck information flow so that the DNN is forced to prioritize what information to propagate to the next layer (by restricting the number of dimensions in the latent space). In this project, I explore how this can be a useful denoising tool.
Einops and Einsum Summarized
Published:
A brief summary on einops and einsum, usage documentation and an implementation of Average Pooling in CNNs using einops (inspired from the max pooling layer implemented in the original library documentation).
Review: Interpretability in the Wild: A Circuit for Indirect Object Detection in GPT2-Small
Published:
A paper review highlighting the key discoveries with respect to attention heads and the algorithms used.
Review: A Mathematical Framework for Transformer Circuits
Published:
This paper provides a mental model for reasoning about the internal workings of transformers and attention heads in deep neural networks. The insights here help understand and analyze the behaviors of large models.
Denoising
Are Autoencoders Fundamentally Denoisers?
Published:
The core idea behind Autoencoders is to bottleneck information flow so that the DNN is forced to prioritize what information to propagate to the next layer (by restricting the number of dimensions in the latent space). In this project, I explore how this can be a useful denoising tool.
Einops and Einsum
Einops and Einsum Summarized
Published:
A brief summary on einops and einsum, usage documentation and an implementation of Average Pooling in CNNs using einops (inspired from the max pooling layer implemented in the original library documentation).
KV Cache
Paged Attention and vLLM
Published:
Paged Attention is a memory optimization on which the vLLM Inference Engine is based. Here is a summary of the paper on paged attention and the key features of vLLM that make it so powerful.
LLM Inference
Paged Attention and vLLM
Published:
Paged Attention is a memory optimization on which the vLLM Inference Engine is based. Here is a summary of the paper on paged attention and the key features of vLLM that make it so powerful.
Language Models
Implementing GPT from Scratch
Published:
This article contains a conceptual explanation, necessary for building a language model from scratch, using the decoder-only transformer architecture. It is based on Andrej’s Karpathys GPT from scratch. The code for this conceptual guide can be found here.
Review: Interpretability in the Wild: A Circuit for Indirect Object Detection in GPT2-Small
Published:
A paper review highlighting the key discoveries with respect to attention heads and the algorithms used.
Review: A Mathematical Framework for Transformer Circuits
Published:
This paper provides a mental model for reasoning about the internal workings of transformers and attention heads in deep neural networks. The insights here help understand and analyze the behaviors of large models.
Linear Algebra
Einops and Einsum Summarized
Published:
A brief summary on einops and einsum, usage documentation and an implementation of Average Pooling in CNNs using einops (inspired from the max pooling layer implemented in the original library documentation).
Mechanistic Interpretability
Review: Interpretability in the Wild: A Circuit for Indirect Object Detection in GPT2-Small
Published:
A paper review highlighting the key discoveries with respect to attention heads and the algorithms used.
Review: A Mathematical Framework for Transformer Circuits
Published:
This paper provides a mental model for reasoning about the internal workings of transformers and attention heads in deep neural networks. The insights here help understand and analyze the behaviors of large models.
PyTorch
Are Autoencoders Fundamentally Denoisers?
Published:
The core idea behind Autoencoders is to bottleneck information flow so that the DNN is forced to prioritize what information to propagate to the next layer (by restricting the number of dimensions in the latent space). In this project, I explore how this can be a useful denoising tool.
Einops and Einsum Summarized
Published:
A brief summary on einops and einsum, usage documentation and an implementation of Average Pooling in CNNs using einops (inspired from the max pooling layer implemented in the original library documentation).
Implementing GPT from Scratch
Published:
This article contains a conceptual explanation, necessary for building a language model from scratch, using the decoder-only transformer architecture. It is based on Andrej’s Karpathys GPT from scratch. The code for this conceptual guide can be found here.
Signal Processing
Are Autoencoders Fundamentally Denoisers?
Published:
The core idea behind Autoencoders is to bottleneck information flow so that the DNN is forced to prioritize what information to propagate to the next layer (by restricting the number of dimensions in the latent space). In this project, I explore how this can be a useful denoising tool.
Tensor Operations
Einops and Einsum Summarized
Published:
A brief summary on einops and einsum, usage documentation and an implementation of Average Pooling in CNNs using einops (inspired from the max pooling layer implemented in the original library documentation).
Transformers
Implementing GPT from Scratch
Published:
This article contains a conceptual explanation, necessary for building a language model from scratch, using the decoder-only transformer architecture. It is based on Andrej’s Karpathys GPT from scratch. The code for this conceptual guide can be found here.
Review: Interpretability in the Wild: A Circuit for Indirect Object Detection in GPT2-Small
Published:
A paper review highlighting the key discoveries with respect to attention heads and the algorithms used.
Review: A Mathematical Framework for Transformer Circuits
Published:
This paper provides a mental model for reasoning about the internal workings of transformers and attention heads in deep neural networks. The insights here help understand and analyze the behaviors of large models.
vLLM
Paged Attention and vLLM
Published:
Paged Attention is a memory optimization on which the vLLM Inference Engine is based. Here is a summary of the paper on paged attention and the key features of vLLM that make it so powerful.