Repository navigation
Conversation
Companion to deepspeedai/DeepSpeed#9131: continues training a causal LM (dense or AutoEP MoE) on WikiText-103 under BF16 and several quantization recipes, and reports validation loss, step time and peak memory. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: Zhipeng Wang <7927074+PKUWZP@users.noreply.github.com>
…example from_scratch.py trains a Llama-architecture model from random initialization on all of WikiText-103 with a warmup + cosine schedule and reports final validation and test loss, so FP8 recipes can be compared with BF16 at convergence. The README gains the launch commands and two-seed reference results. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: Zhipeng Wang <7927074+PKUWZP@users.noreply.github.com>
--moe builds a Qwen2-MoE-architecture model and trains it with AutoEP, so quantized grouped experts can be compared with BF16 at convergence. The README gains the launch command and two-seed MoE reference results. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: Zhipeng Wang <7927074+PKUWZP@users.noreply.github.com>
from_scratch.py gains the int8 and int8_h128 recipes, and the README gains their two-seed MoE reference results. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: Zhipeng Wang <7927074+PKUWZP@users.noreply.github.com>
PKUWZP
marked this pull request as ready for review
November 22, 2026 16:49
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
training/subchannel_quantization/, the companion example for deepspeedai/DeepSpeed#9131 (proposal: deepspeedai/DeepSpeed#9132). It should not merge before that PR does.What it adds
train.py: continues training a pretrained causal LM on WikiText-103 with DeepSpeed, either as a dense model (data parallel) or as an MoE model with AutoEP.--recipeselects thequantization.subchannelsettings:bf16(baseline),e4m3_row,e4m3,int8,int8_h128. Every recipe sees the same data in the same order, and the script reports validation loss, median step time and peak memory.from_scratch.py: trains a Llama-architecture model (default 1.82B parameters) from random initialization on all of WikiText-103 with a warmup + cosine schedule, under any of the five recipes, and reports final validation and test loss. With--moeit trains a Qwen2-MoE-architecture model with AutoEP instead. This is the script behind the full-training results in Add subchannel FP8/INT8 quantized training for nn.Linear and AutoEP experts DeepSpeed#9131.README.md: prerequisites, the recipes, launch commands for both scripts, and reference results.requirements.txt: installs DeepSpeed from the tip of Add subchannel FP8/INT8 quantized training for nn.Linear and AutoEP experts DeepSpeed#9131.Validation
Hardware: 4x NVIDIA H200, driver 595.112.07, torch 2.8.0+cu129, Triton 3.4.0, transformers 5.23.0.
e4m3_rowon Qwen1.5-MoE-A2.7B (AutoEP) and on Qwen2.5-14B (--dense --grad_checkpointing --bf16_states); both completed and wrote their result files.--denserather than a separate flag, and memory statistics came fromtorch.cudainstead of the accelerator interface.from_scratch.pybefore it was moved here. The committed version differs in a license header, a--data_diroption replacing a fixed cache path, creating the output directory, and an argument check. It was rerun after those changes for 100 steps withbf16ande4m3_rowon 2 GPUs with a small model (--layers 2 --hidden 1024 --intermediate 2048); both completed. The full runs were not repeated, and--prepwas not rerun with the new--data_diroption.autoep_size=2) were produced withfrom_scratch.pyas committed apart from line wrapping and the docstring, launched with the--moecommand in the README plus--data_dir. Before those runs, a 2-layer model was inspected to confirm that AutoEP had replaced the MoE blocks and that the grouped experts carried the quantized GEMM hook.int8andint8_h128, two seeds, 28,788 steps each) were produced withfrom_scratch.pyexactly as committed. INT8 was not trained to convergence on the dense model: 150-step runs of both recipes completed there, and the full runs were skipped.requirements.txtwas not installed into a clean environment.AI assistance
This example was written with AI assistance (Claude Code) and needs human review of every line before it merges.