Our story

Grey Matter
Colorful Ideas

Welcome to Dead Neuron! We are sharing key insights, tips, and tricks for making the most of your machine learning models.

Faster Models

Optimizing network architecture design to improve training and inference times.

Smarter Models

Developing training and tuning techniques to improve generalization and accuracy.

Smaller Models

Pruning, quantizing, and distilling large models into smaller sizes to reduce computational cost.

Our Story

I am Dr. Tim Whitaker and I am a research scientist who is deeply engaged in exploring the evolving paradigms of artificial intelligence and machine learning. I recently completed my PhD in Computer Science at Colorado State University where I conducted novel research on a variety of topics related to neural network optimization. Deep learning has absolutely exploded over the last several years and I wanted to create a place to share interesting and important insights that I've found throughout my journey. I hope that Dead Neuron can serve as a valuable resource for aspiring machine learning practitioners to explore some of the lesser known implementation details that help to maximize performance.

use burn::{
    module::Module,
    nn::{
        conv::{Conv2d, Conv2dConfig},
        pool::{AdaptiveAvgPool2d, AdaptiveAvgPool2dConfig},
        Linear, LinearConfig, ReLU,
    },
    tensor::{backend::Backend, Tensor},
};

#[derive(Module)]
pub struct Net<B: Backend> {
    conv1: Conv2d<B>,
    conv2: Conv2d<B>,
    pool: AdaptiveAvgPool2d,
    fc1: Linear<B>,
    fc2: Linear<B>,
    activation: ReLU,
}

impl<B: Backend> Net<B> {
    pub fn new(num_outputs: usize, device: &B::Device) -> Self {
        Self {
            conv1: Conv2dConfig::new([3, 8], [3, 3]).init(device),
            conv2: Conv2dConfig::new([8, 16], [3, 3]).init(device),
            pool: AdaptiveAvgPool2dConfig::new([8, 8]).init(),
            fc1: LinearConfig::new(16*8*8, 256).init(device),
            fc2: LinearConfig::new(256, num_outputs).init(device),
            activation: ReLU::new(),
        }
    }

    pub fn forward(&self, x: Tensor<B, 4>) -> Tensor<B, 2> {
        let x = self.conv1.forward(x);
        let x = self.activation.forward(x);
        let x = self.conv2.forward(x);
        let x = self.activation.forward(x);
        let x = self.pool.forward(x);
        let x = x.reshape([0, -1]);
        let x = self.fc1.forward(x);
        let x = self.activation.forward(x);
        self.fc2.forward(x)
    }
}

Creating For Next Generation Deep Learning Frameworks

I am a polyglot at heart who loves to explore new programming languages and paradigms. While Python has long dominated the machine learning space, there is a lot of potential for new frameworks to enable better and more interesting ways to build performant models. Burn is one new promising framework in Rust that offers an elegant type safe approach that emphasizes performance, flexibility, and portability for high performance tensor computing.

This framework is a pleasure to develop in and I couldn't be more excited about the opportunities for it overcome some of the pain points of deploying models with established Python frameworks. I am excited to build and contribute to this community so I am starting to work on a model zoo that includes important, impactful, and exotic neural network architectures through history. The goal is to create clean and extensible code that is easy to plug into your own projects. Check out our repo on github to get a feel for this new project.

Model Zoo