Practical overview
When it fits
Use it when you can modify a PyTorch Transformer workflow and want compiler-level optimization for inference or training, with CUDA-based setup documented by the project.
Who it is for
- Developers optimizing Transformer-like model inference.
- Teams compiling large-model training workloads with FSDP-aware layer-wise compilation.
Documented uses
- Capture a Transformer inference workload across block boundaries for whole-graph compilation.
- Compile training workloads layer by layer with FSDP-aware handling.
- Manage memory and data movement through selective asynchronous offloading and heuristic activation recomputation.
- Inspect compilation timelines, decompiled bytecode flows, split subgraphs, and backend artifacts using the included magi_depyf introspection toolkit.
Requirements & limits
The repository is public and licensed under Apache-2.0. The README documents local installation and a prebuilt Docker image.
- Requires Python 3.12 or later, PyTorch 2.9.x or 2.12.x, and CUDA Toolkit.
- CUTLASS is optional and required only for the documented matmul-plus-epilogue fusion pass on sm_90 or sm_120.
- The quick-start example uses CUDA; the documentation does not specify a complete hardware support matrix.
- The documented target is Transformer-like architectures, not a general-purpose compiler for arbitrary models.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Install MagiCompiler from source
Set up the package locally using the documented source-install route.
Before you start
- Python 3.12 or later
- PyTorch 2.9.x or 2.12.x
- CUDA Toolkit
- Graphviz is optional and listed for FX graph visualization.
Steps
Clone the MagiCompiler repository and change into its directory.
Install the Python dependencies with pip install -r requirements.txt.
Install the package for end users with pip install .
Optionally install CUTLASS and configure CUDACXX and its build for the listed GPU architecture if the CUTLASS matmul-plus-epilogue fusion pass is needed.
Expected outcome
The local package is installed; CUTLASS enables the documented fusion pass, while the compiler can still work without that optional optimization.
Guide limitations
- The documented requirements do not provide a full supported hardware matrix.
- CUTLASS is required only for its specific fusion pass; the README says that optimization is skipped without it.
Apply the compiler decorator to a Transformer block
Use the documented quick-start pattern to compile a core Transformer module.
Before you start
- An installed MagiCompiler environment
- A PyTorch Transformer block and CUDA device, as used in the README example
Steps
Import torch, nn, and magi_compile.
Decorate the Transformer block class with @magi_compile.
Instantiate the model on CUDA and call it with inputs, as in the example.
Expected outcome
The example routes ordinary model calls through whole-graph compilation and demonstrates calls with different batch sizes.
Guide limitations
- The example documents a Transformer block and CUDA execution; it does not establish compatibility with other architectures or hardware.
Compare options for Model inference and serving
These projects document a common task and share a project role. This is a source-based comparison, not an execution benchmark or a guarantee of interchangeability.
| Project | Documented purpose | Platform | Access and requirements |
|---|---|---|---|
| Magicompiler | Capture a Transformer inference workload across block boundaries for whole-graph compilation. | Linux | The repository is public and licensed under Apache-2.0. The README documents local installation and a prebuilt Docker image. |
| Nano Vllm V1 | Generate text offline from prompts using a local model. | Linux, Self-hosted | The repository is public and identifies the project as MIT-licensed. It is designed for Linux and self-hosted use. |
| PyTorch C++ Inference | Build a C++ executable that classifies an image using an exported TorchScript model. | Linux, Docker, Server | The repository is publicly accessible and identified as open source. No license is specified in the supplied repository details. |
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 23 October 2026
- Original website title
- SandAI-org/MagiCompiler · Official homepage title · retained in its source language