LoongForge

A modular, scalable, and highly efficient training framework for language, multimodal, and embodied models. Built upon Megatron-LM with significant enhancements.

—

Get Started

• Installation
• Support Model
• Parallelism Strategies & Optimization Guide

LLM Training

• Quick Start: LLM Pre-training
  • 0. Resource Preparation
  • 1. Prepare the data
  • 2. Prepare the checkpoint
  • 3. Launch pre-training
  • 4. Monitoring
• Quick Start: LLM SFT
  • 0. Resource Preparation
  • 1. Prepare the data
  • 2. Prepare the checkpoint
  • 3. Launch SFT training
  • 4. Monitoring
• Checkpoint Conversion for LLM
  • 1. Common Parameters
  • 2. Example Scripts
• Advanced Features
  • FP8 Training
  • Adaptive FP8 Training (Selective FP8)
  • MoE All-to-All Overlap
  • Optimizer Support
  • Fused Linear Cross Entropy
  • Mcore-Bridge: Online HF Checkpoint Loading & Saving
  • LoRA Feature Usage Guide

VLM Training

• Quick Start: VLM Pre-training
  • 0. Resource Preparation
  • 1. Data Preparation
  • 2. Model Weight Preparation
  • 3. Start Pre-training
• Quick Start: VLM SFT
  • 0. Resource Preparation
  • 1. Data Preparation
  • 2. Model Weight Preparation
  • 3. Start SFT Training
• VLM Dataset Conversion
  • 1. Dataset Format and Processing
  • 2. Supported Data Types (--sample_type)
  • 3. Conversion Script Usage
  • 4. Input JSON Conventions (Common Fields)
  • 5. Offline Packing Data Processing
• Model Checkpoint Conversion for VLM
  • 1. Parameter Passing Methods
  • 2. Common Parameters
  • 3. Script Examples and Parameter Explanation
  • Custom Model Construction
• Advanced Features
  • Offline Packing
  • DP Load-Balancing
  • Model Combination
  • FP8 Training for VLM
  • Heterogeneous Parallel
  • ViT Encoder DP Load-Balancing

VLA Training

• Quick Start: Pi0.5 Training

Diffusion Training

• Quick Start: Wan Model Training
• Wan2.2 Packing Training

KunLun Training

• Kunlunxin P800 README
• Installation on Kunlunxin P800
• Quick Start: LLM Model Pretrain Training on Kunlunxin P800
• Quick Start: LLM Model SFT Training on Kunlunxin P800
• Quick Start: VLM Model SFT Training on Kunlunxin P800
• Quick Start: VLA Model SFT Training on Kunlunxin P800

Developer Guide

• Support New Model

More

• Contribute
• License and File Header Guidelines
• FAQs