2. Introduction to AXera NPU (Neutron)#
AXera NPU (Neutron) is the neural-network compute engine used in AXera chip platforms. Neutron versions differ in system architecture, the number of compute cores, virtual NPU (vNPU) partitioning, the offline compilation npu_mode, and AXEngine runtime modes.
A vNPU is a logical partition of the underlying NPU compute resources. During model compilation, use the --npu_mode option of pulsar2 build to specify the amount of compute resources required by the model. When the model is loaded on the chip, AXEngine schedules it to matching NPU Core or vNPU resources according to the current NPU runtime mode.
This section first introduces the basic Neutron terminology and the responsibilities of each EU type. The subsections then describe how NPU compute resources are organized in each Neutron version. Finally, the chip platform table links each platform and its available npu_mode values to the corresponding hardware resources and AXEngine runtime modes.
2.1. Hardware terminology#
NPU CoreThe basic NPU execution unit. After compilation, a model is divided into executable tasks that are mapped to one or more NPU Cores. Each NPU Core generally contains EUs, Local OCM, and synchronization management capabilities.
EUExecution Unit, a hardware unit that performs computation or data movement. Different EUs handle different types of work, such as convolution, tensor or vector computation, data movement, and data rearrangement.
Local OCMOn-Chip Memory within an NPU Core. Compared with DDR, OCM has lower access latency and higher bandwidth. It is generally used for input tiles, weight tiles, intermediate results, and temporary data that the current task accesses frequently.
Shared OCMOn-chip memory shared by multiple NPU Cores or vNPUs. It is used to exchange intermediate data across tasks and reduce the additional transfers and memory accesses caused by staging data through DDR.
SyncManagerA synchronization management module that describes and manages execution dependencies among NPU Cores, EUs, and subtasks, ensuring the correct order of data reads and writes during parallel execution.
npu_modeAn offline compilation option of
pulsar2 buildthat specifies the target amount of NPU resources. Values such asNPU1,NPU2,NPU3, andNPU4represent compilation targets of different sizes; they do not identify a particular NPU Core or runtime vNPU.AXEngine runtime modeThe hardware resource partitioning mode configured through
AX_ENGINE_NPU_ATTR_T.eHardModewhenAX_ENGINE_Initinitializes AXEngine. It determines the vNPU or NPU resource layout exposed to models at runtime and must satisfy thenpu_moderesource requirements of the model being loaded.
2.2. EU types#
EUs are the hardware resources that perform work in Neutron. They can be divided into compute EUs and data-movement EUs. Based on operator types, tensor shapes, data layouts, OCM pressure, and target hardware capabilities, the compiler automatically maps model tasks to suitable EUs. Users normally do not need to select EU types manually.
CONVA convolution compute EU for highly reusable, compute-intensive deep-learning workloads. Typical tasks include convolution, deconvolution, convolution post-processing, and some matrix computations that can be mapped to a convolution dataflow.
TENGTensor Engine, a tensor or vector compute EU suitable for multi-channel parallel data processing and general tensor computation. Typical tasks include element-wise operations, activations, lookups, reductions, coordinate or address calculation, and some data replication and format-assistance operations.
SDMAA data-movement EU responsible for transferring data among DDR, OCM, and on-chip memory regions. Typical tasks include loading model inputs and weights, reading and writing intermediate results, writing outputs, channel cropping or padding, and data-format encoding or decoding.
DAUA compute EU used for some data rearrangement, filtering, and sorting tasks. Typical tasks include Padding, Transpose, TopK/Sort, and other data organization operations.
Note
The number of EUs, supported sub-functions, and available runtime modes may differ among Neutron versions and chip platforms. Refer to the chip specification and toolchain support for the actual capabilities.
2.3. Offline compilation and runtime modes#
npu_mode and the AXEngine runtime mode are settings at two different stages and must be used together:
npu_modetakes effect during offline compilation. It determines the resource scale, task partitioning, and scheduling constraints generated by the compiler. The compiledaxmodelcarries these resource requirements.The AXEngine runtime mode takes effect when the application initializes the NPU. It determines whether the underlying hardware is exposed as one complete NPU resource, multiple physical or logical vNPUs, or scheduling resources across multiple NPU Cores.
A runtime mode cannot automatically turn a model compiled for fewer resources into a larger-resource model, nor can a model compiled for more resources run in a runtime mode that provides insufficient resources. During deployment, determine the model's
npu_modefirst, and then select an AXEngine mode that provides matching resources.
2.4. Neutron architectures#
See the following subsections for the resource organization of each Neutron version:
2.4.5. Chip platforms and available npu_mode values#
The following table lists the Neutron version and available npu_mode options for each chip platform. npu_mode describes the amount of NPU resources used to compile a model; it does not identify a particular NPU Core or vNPU.
For the mapping among AXEngine runtime modes, offline npu_mode values, and hardware compute resources, see the corresponding Neutron architecture document.
Chip platform |
Neutron version |
Available |
Details |
|---|---|---|---|
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|