First published: 2026-07-14

Introduction

I've built game AIs before — an Othello AI, then a Mahjong AI. While working on the Mahjong AI I came across PPO (a reinforcement learning technique), and from there the idea of building a language model itself, from scratch, started to take shape naturally.

The direct trigger was the news that Anthropic's frontier models (Claude Mythos, etc.) were temporarily suspended due to U.S. export controls. Seeing that access to frontier AI abroad could be cut off by a single government decision made me realize how little domestic infrastructure Japan has for developing large language models on its own. Beyond the international-competitiveness angle, it also bothered me that Japan can't handle its own data with its own technology.

On top of that concern, I was simply curious how far one person, with limited resources, could get in building a language model. Looking further ahead, I also have some "just for fun" use cases in mind — embedding it in a game AI, or turning it into a Discord bot.

This article is both a personal record of that process and, hopefully, a useful reference for anyone else attempting something at the same scale.

The short version: building a ~109.53M-parameter language model from pretraining through SFT and a RAG-based inference pipeline is entirely achievable on a single consumer GPU (8GB VRAM). What you shouldn't expect, though, is much real-world usefulness from its language ability — it picks up Japanese-like grammar and style, but falls well short of practical coherence or factual reliability. "Can it be built?" and "Is it useful?" turned out to be two very different questions.

Table of Contents

  1. Hardware
  2. Software & Environment
  3. Dataset
  4. Tokenizer
  5. Model Architecture
  6. Training Config
  7. Training Results
  8. Training Time
  9. Troubleshooting
  10. Generated Samples
  11. SFT
  12. Applying RAG
  13. What's Next
  14. Notes for Anyone Building This Themselves
  15. Acknowledgements
  16. Code
  17. References

1. Hardware

Item Value
OS Windows
RAM 32GB
GPU NVIDIA GeForce RTX 5060
VRAM 8GB

2. Software & Environment

Item Value
Virtual environment Docker (docker-compose)
Language / libraries Python 3.10 / PyTorch (CUDA)
Dtype bfloat16

3. Dataset

Item Value
Total tokens 2,000,000,127 tokens
Storage format Sharded binary files (.bin)
Loading method Random access via np.memmap, no full in-memory expansion

Corpus mix

Corpus Share
matsuo-lab/JP-LLM-Corpus-PII-Filtered-10B 50%
wikimedia/wikipedia (20231101.ja) 30%
globis-university/aozorabunko-clean 20%

4. Tokenizer

Item Value
Algorithm ByteLevel BPE
Normalization NFKC
Vocab size 32,000
Config limit_alphabet=6000

5. Model Architecture

Item Value
Total parameters 109.53M (~100M)
Layers (n_layers) 12
Attention heads (n_heads) 12
Hidden size (d_model) 768
Max context length (max_seq_len) 2048

The design is a from-scratch implementation following the Llama-style architecture rather than the classic GPT-2 recipe (absolute position embeddings, LayerNorm, GELU). It uses the following instead:

Component Choice
Positional encoding RoPE (Rotary Positional Embeddings)
Normalization RMSNorm (Pre-Norm placement)
Feed-forward SwiGLU (hidden dim = 8/3 × d_model, rounded up to a multiple of 256)
Attention PyTorch scaled_dot_product_attention (Flash Attention–style kernels selected automatically)
Weight sharing Weight tying between the input embedding and output (lm_head) layers
Weight init Normal distribution (std=0.02)

The attention module itself is written to support GQA (Grouped Query Attention, i.e. n_kv_heads set lower than n_heads), but for this run n_kv_heads was left unset and defaults to the same value as n_heads, so in practice it behaves as ordinary Multi-Head Attention.

Each TransformerBlock is Pre-Norm (x + Attention(RMSNorm(x)), h + SwiGLU(RMSNorm(h))), stacking Attention → RMSNorm → residual → SwiGLU → RMSNorm → residual across 12 layers.

5.1 Implementation (excerpts)

The normalization layer is RMSNorm.

class RMSNorm(nn.Module):
    def __init__(self, dim: int, eps: float = 1e-6):
        super().__init__()
        self.eps = eps
        self.weight = nn.Parameter(torch.ones(dim))

    def _norm(self, x):
        return x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)

    def forward(self, x):
        output = self._norm(x.float()).type_as(x)
        return output * self.weight

The feed-forward layer is SwiGLU. The hidden dimension is 8/3 × d_model, rounded up to the nearest multiple of 256.

class SwiGLU(nn.Module):
    def __init__(self, d_model: int, hidden_dim: int):
        super().__init__()
        self.w1 = nn.Linear(d_model, hidden_dim, bias=False)
        self.w2 = nn.Linear(hidden_dim, d_model, bias=False)
        self.w3 = nn.Linear(d_model, hidden_dim, bias=False)

    def forward(self, x):
        return self.w2(torch.nn.functional.silu(self.w1(x)) * self.w3(x))

TransformerBlock is Pre-Norm: RMSNorm precedes both Attention and SwiGLU, with residual connections added back afterward.

class TransformerBlock(nn.Module):
    def forward(self, x: torch.Tensor, freqs_cis: torch.Tensor):
        h = x + self.attention(self.attention_norm(x), freqs_cis)
        out = h + self.feed_forward(self.ffn_norm(h))
        return out

Attention uses RoPE for positional encoding, and the actual computation is delegated to PyTorch's built-in scaled_dot_product_attention (is_causal=True). This means Flash Attention–style optimized kernels get selected automatically depending on the runtime environment and tensor shapes.

def forward(self, x: torch.Tensor, freqs_cis: torch.Tensor):
    bsz, seqlen, _ = x.shape
    xq, xk, xv = self.wq(x), self.wk(x), self.wv(x)

    xq = xq.view(bsz, seqlen, self.n_local_heads, self.head_dim)
    xk = xk.view(bsz, seqlen, self.n_local_kv_heads, self.head_dim)
    xv = xv.view(bsz, seqlen, self.n_local_kv_heads, self.head_dim)

    xq, xk = apply_rotary_emb(xq, xk, freqs_cis=freqs_cis)

    xq = xq.transpose(1, 2)
    xk = xk.transpose(1, 2)
    xv = xv.transpose(1, 2)

    output = torch.nn.functional.scaled_dot_product_attention(
        xq, xk, xv, attn_mask=None, dropout_p=0.0, is_causal=True
    )

    output = output.transpose(1, 2).contiguous().view(bsz, seqlen, -1)
    return self.wo(output)

6. Training Config

Item Value
Total steps 16,000
Physical batch size 2
Gradient accumulation 32
Effective batch size 64
Max LR 6.0e-4
Min LR 6.0e-5
Warmup steps 40
LR schedule Cosine decay with warmup
Optimizer AdamW (betas=(0.9, 0.95), eps=1e-8)
Weight decay 0.1
Gradient clipping (max norm) 1.0
Data sampling PyTorch DataLoader (shuffle=True), random batch sampling
Train/val split 90% / 10%
Early stopping Stop if val loss fails to improve for 5 consecutive evals
Eval / checkpoint interval Every 500 steps

Learning rate schedule (cosine decay with warmup)

You can see the short warmup, the peak at 6.0e-4, and the cosine decay down to 6.0e-5 over the 16,000 steps.

7. Training Results

Train and validation loss

Loss drops sharply right after the start, converging to roughly 3.0 by around step 6,000. From there it declines more gradually, settling to about 2.6–2.7 on both train and validation loss by step 16,000. Train loss is noticeably noisier than validation loss, which trends more smoothly.

8. Training Time

Item Value
Start 2026-07-11 05:52:56 (UTC)
End 2026-07-12 04:28:04 (UTC)
Total training time 81,287.8 sec (~22h 35m)
Total steps 16,000
Avg. step time ~5.08 sec/step
Throughput ~25,800 tokens/sec
Total tokens processed ~2.1B tokens

For a 109.53M-parameter model, ~2.1B tokens of training lines up well with the Chinchilla rule of thumb (roughly 20 tokens per parameter is considered compute-optimal): 109.53M × 20 ≒ 2.19B, almost exactly matching what was actually used — a reasonable training budget for this model size.

Even on a single consumer-grade 8GB VRAM GPU, this scale of pretraining can be completed in roughly a day — useful as a reference point.

I'd also set up an automatic shutdown after training completed (in case it finished overnight), but training didn't finish at the time I expected, so it never actually triggered.

9. Troubleshooting

Running training at this scale on limited hardware (system RAM and 8GB VRAM) meant running into several out-of-memory situations. Here's what caused each one and how it was addressed.

9.1 OOM during tokenizer training (system RAM)

trainer = BpeTrainer(
    vocab_size=32000,
    min_frequency=2,
    special_tokens=["<unk>", "<s>", "</s>", "<pad>"],
    limit_alphabet=6000,  # keeps Japanese's huge kanji inventory from blowing up pair counts
    show_progress=True
)

9.2 OOM during dataset extraction (system RAM)

ds_matsuo = load_dataset(
    'matsuo-lab/JP-LLM-Corpus-PII-Filtered-10B', split='train', streaming=True
)
iter_matsuo = iter(ds_matsuo)
matsuo_buffer = []

# Each JSONL line packs 1,400 articles, so expand it into a buffer
# once, then pop one article at a time instead of loading it all.
if not matsuo_buffer:
    row = next(iter_matsuo)
    jsonl_data = row['jsonl']
    if isinstance(jsonl_data, bytes):
        jsonl_data = jsonl_data.decode('utf-8')
    for line in jsonl_data.strip().split('\n'):
        if not line:
            continue
        data = json.loads(line)
        t = data.get("text", "").strip()
        if t:
            matsuo_buffer.append(t)

if matsuo_buffer:
    text = matsuo_buffer.pop()

9.3 OOM during training data loading (system RAM)

class BinaryTokenDataset(Dataset):
    def __init__(self, data_dir, seq_len):
        self.seq_len = seq_len
        self.files = sorted(glob.glob(os.path.join(data_dir, "*.bin")))

        # Memory-map every chunk so nothing is fully expanded into RAM
        self.cum_tokens = [0]
        self.mmap_files = []
        for f in self.files:
            m = np.memmap(f, dtype=np.uint16, mode='r')
            self.mmap_files.append(m)
            self.cum_tokens.append(self.cum_tokens[-1] + len(m))

        self.total_tokens = self.cum_tokens[-1]
        self.num_samples = self.total_tokens // (seq_len + 1)

    def __getitem__(self, idx):
        start = idx * self.seq_len
        chunk_idx = bisect.bisect_right(self.cum_tokens, start) - 1
        local_start = start - self.cum_tokens[chunk_idx]
        chunk = self.mmap_files[chunk_idx]

        data = chunk[local_start:local_start + self.seq_len + 1].astype(np.int64)
        x = torch.tensor(data[:-1], dtype=torch.long)
        y = torch.tensor(data[1:], dtype=torch.long)
        return x, y

9.4 OOM during model training (GPU VRAM)

BATCH_SIZE = 2          # physical batch size, kept minimal to fit 8GB VRAM
GRAD_ACCUM_STEPS = 32   # effective batch size = 2 * 32 = 64

optimizer.zero_grad(set_to_none=True)

for _ in range(GRAD_ACCUM_STEPS):
    x, y = next(train_iter)
    x, y = x.to(DEVICE), y.to(DEVICE)

    with torch.autocast(device_type=DEVICE, dtype=DTYPE):  # DTYPE = torch.bfloat16
        _, loss = model(x, y)
        loss = loss / GRAD_ACCUM_STEPS

    scaler.scale(loss).backward()

scaler.unscale_(optimizer)
torch.nn.utils.clip_grad_norm_(model.parameters(), MAX_GRAD_NORM)
scaler.step(optimizer)
scaler.update()

10. Generated Samples

Sample outputs from the base pretrained model. Generation parameters:

Parameter Value
Repetition penalty 1.2
Temperature 0.8
Top-k 40
Max new tokens 150

Prompt: 東京とは (roughly "What is Tokyo")

違う。 「マメ」は「女人（女装者）」。「男同士の男性が男として振る舞い、女装する」という意味がある。 人名ではなく、「女」という表現を用いていることが多いことからしばしば混同されるが、日本以外の国の人を指す場合はこの項でまとめて参照した。

男女の区別 男性の場合は、男性と女性の二重婚である。

男女の混血などの場合は、必ずしも二人目のときの2分の1以上の性的不一致を区別しないことに注意する必要がある。また男性の場合は、複数の男性の女性と結婚した場合、両者の一連の性別は両性の関係となるため厳密には同一ではない。

不倫の場合 結婚後の3 - 4年後の年月

Rough gist: Opens with "No, that's wrong," then veers into a garbled explanation of a supposed term for "cross-dressing men," followed by nonsensical claims about marriage and gender ("men are a dual marriage of men and women"), and trails off mid-thought about "years after marriage." What's wrong: None of this has anything to do with Tokyo. The model never engages with the prompt at all — it appears to be reproducing fragments from an unrelated Wikipedia-style article about gender/marriage terminology that happened to be nearby in its training distribution.

Prompt: 日本とは (roughly "What is Japan")

、中国の伝統的な伝統文化を守っていくことが目的です。それぞれの民族の文化はそれぞれの文化に特徴があり、自分達と違う価値を持つことがあります。 日本とアメリカは違い、文化の違いを共有しています。日本の文化は他の社会と大きく異なる点で異なります。例えば、日本人の多くは、彼らの文化を理解することで世界や世界を理解する傾向がありますが、他の人々がこれらの問題を解決しようとする時、それらの影響を受ける場合があります。【ご予約・お問合せ】 アカネフォトスタジオ ◆1020歳未満の方（高校生アサイン含む）の皆様へのお願い! 10月21日（金） 11月中旬まで休館日とさせていただきます。（営業時間外は電話でのご連絡となります。） ※最終受付時間は変更となる場合がございます

Rough gist: Starts mid-sentence about "preserving China's traditional culture," rambles about cultural differences between nations in vague, circular language, then abruptly cuts to what looks like a photo studio's business notice — reservation info, an age-restriction notice, and holiday closure dates. What's wrong: The response starts by talking about China, not Japan, and the cultural-difference paragraph is largely content-free filler. Then, without any transition, it drops entirely out of "essay mode" into what's clearly a scraped advertisement for a photo studio (complete with a business name, "アカネフォトスタジオ"). This is a good example of the training corpus leaking through directly — ad copy and boilerplate business notices are common noise in raw web text.

Prompt: アメリカとは (roughly "What is America")

無関係である。

概要 18世紀中頃から19世紀末にかけて、フランスで盛んに行われた料理法の一つに『シャトレーゼ』がある（日本では1840年（天保元年）頃に「イタリア・ピエモンテ」、以降現在の「ピエートロ」と呼ぶ）。これは中世フランスで考案されたとされるが、これは単に「牛肉から野菜を垂らすもの」（preville populaire）、つまり牛の肉を食べさせる肉、あるいは煮て食べさせる肉を指す。鶏肉は牛肉より柔らかく美味く、煮た牛肉は肉よりも柔らかいので、食肉として利用されることも多い。鶏肉も同様に鍋やステーキなどの料理に用いられるが、鶏肉

Rough gist: Opens with "unrelated," then launches into a fabricated food-history article about an invented French cooking method ("Chateraise"), complete with made-up 19th-century dates, invented etymology, and a tangent about chicken vs. beef being more tender. What's wrong: Everything here is confabulated — dates, terminology, and the supposed culinary history don't correspond to anything real (this reads like the model pattern-matching to "encyclopedia article about a historical topic" and filling in plausible-sounding but fabricated specifics). It also has nothing to do with America.

Grammatically the model rarely breaks down, and it has clearly picked up Japanese-like local statistical patterns and register — but coherence collapses almost immediately. Asking "what is Tokyo" gets a tangent about cross-dressing terminology; "what is Japan" drifts mid-response into a photo studio's business notice. This is most likely training-data leakage — miscellaneous Wikipedia articles and fragments of ad copy from the web corpus bleeding straight through.

At ~109.53M parameters and ~2B tokens of pretraining, this is roughly the expected outcome: you get a glimmer of fluency, but nowhere near the coherence needed for practical use. The goal here was to build and understand the full pretraining pipeline end to end — tokenizer construction, data processing, training on a single 8GB VRAM setup with no distributed setup — so this result was well within expectations.

11. SFT

The pretrained model was fine-tuned (Supervised Fine-Tuning) on instruction-response data.

11.1 Dataset

The base dataset was kunishou/databricks-dolly-15k-ja, filtered as follows:

Filter Value
Task categories open_qa, closed_qa, information_extraction only
User input (question + context) length ≤ 60 characters
AI output (answer) length ≤ 80 characters
Exclusions Answers containing list formatting (e.g. 1., *) removed
Sample size 2,000 randomly sampled from the filtered set

The task categories were narrowed to three, and both input and output were restricted to short text, because a 109.53M-parameter model with a 2048-token context wasn't judged capable of handling long context or complex instructions (summarization, creative writing, classification, etc.) yet. Sticking to short, self-contained Q&A was meant to make the "shape" of a good response easier to learn given the model's limited capacity.

11.2 Training Setup

Item Value
Training samples 2,000
Epochs 3

Unlike pretraining (Section 1), SFT was run on a MacBook Air (M5, mps device).

Average loss per epoch

Epoch Average Loss
1 1.5017
2 1.1061
3 0.7799

11.3 Prompt Template

Training data was formatted using a ChatML-style template.

<|im_start|>system
あなたは親切なAIアシスタントです。<|im_end|>
<|im_start|>user
こんにちは<|im_end|>
<|im_start|>assistant

(system: "You are a helpful AI assistant." / user: "Hello.")

11.4 Generated Samples After SFT

Same 3 prompts and same generation parameters as Section 10 (repetition penalty 1.2, temperature 0.8, top-k 40, max_new_tokens 150), run against the SFT model.

User: 東京とは

東京は、主に日本の歴史的な場所・都市の1つとして描かれる領域ですが、地方自治体や民間企業によって様々な地域があるため、「東京」という地名は存在します。

Rough gist: "Tokyo is often depicted as one of Japan's historic places/cities, and because there are various regions run by local governments and private companies, the place name 'Tokyo' exists." What's wrong: It's a short, grammatical, on-topic-sounding answer — a real improvement in form over the base model — but the actual content is close to meaningless. "The place name 'Tokyo' exists because there are various regions run by local governments and companies" isn't a coherent claim; it's plausible-sounding word salad dressed up as an answer.

User: 日本とは

日本は、現在、中国とアメリカの一部であり、中国は世界の領土である。

Rough gist: "Japan is currently part of China and America, and China is the territory of the world." What's wrong: Factually and geopolitically nonsensical — Japan is not "part of" China or America, and "China is the territory of the world" isn't a coherent statement at all. It reads confidently and grammatically, which arguably makes the error more noticeable, not less.

User: アメリカとは

アメリカは、文明の発展とともに発展したヨーロッパの一部である

Rough gist: "America is a part of Europe that developed alongside the development of civilization." What's wrong: Same category of error as above — geographically and factually wrong, but delivered with the same confident, well-formed sentence structure.

Compared to the base model (Section 10), the most obvious change is output length. The base model rambled on for as long as context allowed and would drift into unrelated tangents (like the photo studio ad); after SFT, responses wrap up in one or two sentences. That directly reflects training on answers capped at 80 characters — the model clearly learned the shape of "answer briefly and stop."

What it did not improve is factual accuracy. Errors like "Japan is part of China and America" or "America is part of Europe" carry over the same factual and geographic confusion seen in the base model. SFT on 2,000 examples for 3 epochs, at this scale, helps with response shape and behavior — it doesn't meaningfully add knowledge. That's roughly the expected result.

12. Applying RAG

12.1 Prompt-based RAG

Rather than further fine-tuning, this approach fed retrieved context directly into the prompt of the already-SFT'd model — and it worked reasonably well. The model itself is unchanged; relevant context is simply retrieved and inserted into the prompt before generation. The retrieval corpus was small — only 50 documents — and generation used temperature 0.1 (everything else matching Sections 10–11: repetition penalty 1.2, top-k 40, max_new_tokens 150).

User: 東京とは

Top 2 retrieved results (with cosine similarity scores):

東京は、東京市（とうきょうと）とも呼ばれ、日本の首都であり、日本を代表する観光地である。東京ディズニーリゾートの最寄駅は、東京駅から約2.5kmの距離にある「東京ベイサイドライン」である。東京ディズニーランドや東京ディズニーシーなどの大都市圏へのアクセスも良く、東京ディズニーリゾートの観光名所としては最も歴史のある場所である。

Rough gist: "Tokyo, also called Tokyo City, is Japan's capital and a leading tourist destination. The nearest station to Tokyo Disney Resort is 'Tokyo Bayside Line,' about 2.5km from Tokyo Station. It has good access to major areas like Tokyo Disneyland and Tokyo DisneySea, and is the oldest sightseeing spot in the Tokyo Disney Resort area." What's wrong: Mostly coherent and on-topic — noticeably better than either the base or SFT-only outputs — though it drifts into odd, overly specific claims about Tokyo Disney Resort that don't really follow from the retrieved context, and "the oldest sightseeing spot in the Tokyo Disney Resort area" is an odd, fabricated-sounding detail.

User: 日本とは

Top 2 retrieved results:

日本の領土は、日本列島（大日本帝国）から構成されています。日本の領土は、日本列島（日本国民連合軍総司令部）から構成されます。日本の領土は、日本列島（日本国民連合軍総司令部）から構成され、日本の国土を統治する政府直轄領です。（以下同様の文が反復）

Rough gist: "Japan's territory consists of the Japanese archipelago (Great Empire of Japan). Japan's territory consists of the Japanese archipelago (Japanese National Allied Forces General Headquarters)..." — the same sentence repeats with minor substitutions. What's wrong: Degenerates into repetitive, near-identical sentences, mixing in garbled historical/political terms ("Great Empire of Japan," a garbled version of GHQ) that don't cohere into any real claim. Classic repetition-loop failure mode.

User: アメリカとは

Top 2 retrieved results:

イギリスとフランスは、この2国間経済連携協定により、ロンドンやニューヨークの「ロンドン・ビジネス・センター」を共同設立した。ロンドン・ビジネス・センターはロンドン証券取引所とロンドン証券取引所プライム市場に上場しており、ロンドン証券取引所はロンドン証券取引所の子会社である。

Rough gist: "The UK and France jointly established the 'London Business Center' in London and New York under this bilateral economic partnership agreement. The London Business Center is listed on the London Stock Exchange and the London Stock Exchange's Prime Market, and the London Stock Exchange is a subsidiary of the London Stock Exchange." What's wrong: Completely unrelated to America, and the retrieved context (both chunks about Japan) offered nothing useful about America either. The final sentence is also circular/nonsensical ("the London Stock Exchange is a subsidiary of the London Stock Exchange"). This is a clean example of retrieval failure cascading into generation failure.

Compared to the no-context generations in Sections 10–11, the "東京とは" answer clearly benefits from grounding in retrieved context — prompt-based RAG is visibly doing its job there. But "日本とは" collapses into repetition, and "アメリカとは" is the worst case: neither retrieved chunk was actually about America, so the model produced an unrelated tangent about the UK and France instead. The retrieval corpus was only 50 documents, so it's likely no article about America existed in it at all. This is a clear illustration that corpus size caps retrieval recall, which in turn caps generation quality.

There's also an implementation issue worth flagging: no similarity-score cutoff was applied. In the "アメリカとは" case, the retrieved score was still ~0.81 — which looks superficially relevant — but since the retrieval corpus is entirely Japanese-language text, almost any query will land in that same score range as a baseline. Without a threshold below which retrieval results get dropped ("no relevant document found"), the system ends up forcing an answer out of irrelevant context regardless. Both the small corpus size and the missing cutoff logic contributed to this failure.

12.2 Additional RAG-oriented Fine-tuning, and Overfitting

Separately, fine-tuning the SFT model further with RAG use in mind led to overfitting.

The symptom: no matter what question was asked, the model would respond only with some form of "I don't know." The cause wasn't data imbalance — it was a bug in the evaluation (loss) function itself. There was a loophole where answering "I don't know" incurred lower loss than attempting an actual (possibly wrong) answer, and the model found and exploited it precisely. Responding "I don't know" even to things it presumably knew wasn't really the model expressing uncertainty — it was choosing refusal as the loss-minimizing optimal output.

Strictly speaking, "reward hacking" is a term more associated with RL/RLHF-style reward optimization, and this was supervised fine-tuning directly minimizing a loss function — so "loss minimization artifact" or "shortcut learning" might be the more precise term. That said, in the broader sense of a model finding a loophole in an evaluation signal and optimizing toward unintended behavior, it's close enough to reward hacking to call it that. Either way, the root cause wasn't a capability gap in the model — it was a design flaw in the evaluation function. The penalty weighting between "answer wrong" and "refuse to answer" was unbalanced, so the model settled on refusal as the consistently safest (lowest-loss) output.

13. What's Next

Next up: pretraining at roughly 350M parameters on 15B tokens.

Chinchilla scaling (roughly 20 tokens per parameter as compute-optimal) would suggest about 350M × 20 ≒ 7B tokens. This time the plan is to deliberately go more than double that, to 15B tokens. Recent flagship models like Llama, Qwen, and Gemma tend to "overtrain" well past the Chinchilla-optimal point, prioritizing inference cost and real-world performance — this plan follows that trend.

14. Notes for Anyone Building This Themselves

A few takeaways for anyone attempting a similar solo LLM pretraining project.

15. Acknowledgements

This project made use of the following public datasets. Thanks to their respective creators and maintainers.

Dataset Used for License
matsuo-lab/JP-LLM-Corpus-PII-Filtered-10B Pretraining corpus (50%) Inherits the CC license of its CommonCrawl-derived source data (see the original source for details)
wikimedia/wikipedia (20231101.ja) Pretraining corpus (30%) CC BY-SA 3.0 / GFDL *
globis-university/aozorabunko-clean Pretraining corpus (20%) Public domain (distributed under terms equivalent to CC BY 4.0)
kunishou/databricks-dolly-15k-ja SFT dataset CC BY-SA 3.0 *

* CC BY-SA (share-alike) licenses carry conditions such as requiring derivative distributions to use the same license. Check the primary source for each dataset before reuse. This is not legal advice.

16. Code

The full implementation covered in this article is available at:

github.com/onemunanoyo/109MParameterLLM

17. References

Original papers for the techniques used in this article:

  1. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems 30 (NeurIPS 2017).
  2. Sennrich, R., Haddow, B., & Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units. Proceedings of ACL 2016. arXiv:1508.07909.
  3. Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., & Liu, Y. (2024). RoFormer: Enhanced Transformer with Rotary Position Embedding. Neurocomputing, 568, 127063. arXiv:2104.09864.
  4. Zhang, B., & Sennrich, R. (2019). Root Mean Square Layer Normalization. Advances in Neural Information Processing Systems 32 (NeurIPS 2019). arXiv:1910.07467.
  5. Shazeer, N. (2020). GLU Variants Improve Transformer. arXiv:2002.05202.
  6. Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. Advances in Neural Information Processing Systems 35 (NeurIPS 2022). arXiv:2205.14135.
  7. Loshchilov, I., & Hutter, F. (2019). Decoupled Weight Decay Regularization. International Conference on Learning Representations (ICLR 2019). arXiv:1711.05101.
  8. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., & Lample, G. (2023). LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971.
  9. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vinyals, O., & Sifre, L. (2022). Training Compute-Optimal Large Language Models. Advances in Neural Information Processing Systems 35 (NeurIPS 2022). arXiv:2203.15556.
  10. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347.