DNA Language Model

About

What is this project?

I think this technology is fascinating. I want to explore modeling biological sequences with the recent novel architectures. I felt like I need to start small here. I put this together in a tutorial format to help me easily retrace some concepts when I need to. I'd be glad to get some feedback or help on some helpful resources to keep exploring. This is a hands-on tutorial for building a DNA language model — the same architecture behind DNABERT, Nucleotide Transformer, and Evo — using nothing but a plain CPU (or Apple Silicon GPU) and open-access genome data.

Real Genome Data

Downloads human mitochondrial DNA and a C. elegans fragment directly from public FASTA files — no login required.

Transformer from Scratch

~50 readable lines of PyTorch. Causal attention, masked self-attention, blocks — every part built and explained step by step.

No GPU Required

Trains in ~2–3 min on CPU. Automatically uses CUDA, Apple MPS, or CPU — whichever is available.

Evaluate & Generate

Tracks perplexity against meaningful baselines. Generates novel DNA and verifies it reproduces real 3-mer frequency patterns.

Download & explore genomes

Fetch real FASTA files, measure GC content and base composition.

Tokenize DNA

Character-level tokenization: map A C G T to integers.

Build a Transformer

Multi-head causal self-attention, feed-forward blocks, LayerNorm.

Train & monitor

AdamW optimizer, cosine LR schedule, live loss/perplexity curve.

Generate & evaluate

Sample new sequences; compare 3-mer frequencies to real genomes.

Scale up

Pointers to DNABERT, Nucleotide Transformer, HyenaDNA, and Evo.

Project Log

Progress

Updated as the project evolves. Newest entries at the top.

2026-07-12

v0.1 — Initial tutorial notebook

First complete end-to-end pass: data download → tokenization → Transformer training → evaluation → generation.

Character-level tokenizer over A C G T
2-layer Transformer, 4 heads, 64-dim embeddings
Trained on human MT genome + C. elegans fragment
Validation perplexity: ~3.7 (vs uniform baseline 4.0)
Fixed macOS/Anaconda OpenMP kernel-crash bug (KMP_DUPLICATE_LIB_OK)
Auto device selection: CUDA → Apple MPS → CPU

transformer character-level pytorch bugfix

Current Results

What the model achieves

Trained on ~20 k bases of real genome sequence, ~2 min on CPU.

3.7

Validation perplexity
(lower = better)

4.0

Uniform baseline
(4 equi-probable bases)

~3.87

Base-composition baseline
(uses letter frequencies)

~0.75

3-mer frequency correlation
vs real genome

Beating both baselines confirms the model learned real sequential structure in DNA — not just letter frequencies.

Figures

Architecture & Results

Transformer Architecture

The full DNAGPT stack: token embedding + positional encoding → N× transformer blocks (pre-LayerNorm, causal multi-head self-attention, feed-forward with residual connections) → final LayerNorm → language model head.

Training — Loss & Perplexity Curves

Left: cross-entropy loss dropping below the random baseline (ln 4 ≈ 1.39). Right: validation perplexity falling well below both the uniform (4.0) and base-composition (~3.87) baselines across 3 000 iterations.

Evaluation — Real vs Generated 3-mer Frequencies

Side-by-side comparison of every 3-mer frequency in real genome sequence (blue) vs. 6 000 bases generated by the model (orange). High correlation (~0.96) shows the model internalized real DNA sequence structure.

3-mer frequency comparison: real vs generated

Quick Start

Run it yourself

Local Google Colab

# 1. Clone
git clone https://github.com/EnuhBlaise/DNA_LargeLanguageModel.git
cd DNA_LargeLanguageModel

# 2. Install dependencies
pip install torch matplotlib requests jupyter

# 3. Launch
jupyter notebook dna_language_model_tutorial.ipynb

macOS / Anaconda users: if the kernel crashes on import torch, the notebook already applies the fix automatically — see the note cell inside the notebook.
Apple Silicon: MPS GPU acceleration is detected and used automatically.

Google Colab: upload the .ipynb → Runtime → Run all. For GPU: Runtime → Change runtime type → T4 GPU.