Cannot run the double-die version of the demo model Deepseek-7B

[chripell]

I followed https://github.com/DC-DeepComputing/Framework/blob/main/FML13V03/DC-ROMA%20RISC-V%20AI%20PC%2C%20RISC-V%20Mainboard%20II%20NPU%20Memory%20Adjustment%20Instructions.md for the 32G RAM version (which I have) to expand the reserved memory. Here the dmesg output:

[    0.000000] Linux version 6.6.92-eic7x-2025.07 (root@b67a2314ece6) (riscv64-unknown-linux-gnu-gcc () 13.2.0, GNU ld (GNU Binutils) 2.42) #2025.09.26.03.45+ SMP Fri Sep 26 03:53:01 UTC 2025
[    0.000000] Machine model: DeepComputing FML13V03
[    0.000000] SBI specification v1.0 detected
[    0.000000] SBI implementation ID=0x1 Version=0x10003
[    0.000000] SBI TIME extension detected
[    0.000000] SBI IPI extension detected
[    0.000000] SBI RFENCE extension detected
[    0.000000] SBI SRST extension detected
[    0.000000] earlycon: sbi0 at I/O port 0x0 (options '')
[    0.000000] printk: bootconsole [sbi0] enabled
[    0.000000] efi: EFI v2.10 by Das U-Boot
[    0.000000] efi: RTPROP=0xe8cc8040 SMBIOS=0xe8cf5000 INITRD=0xe33bc040 MEMRESERVE=0xe33bb040 
[    0.000000] OF: reserved mem: OVERLAP DETECTED!
               mmz_nid_0_part_0@1,c0000000 (0x00000001c0000000--0x0000000480000000) overlaps with g2d_8GB_boundary_reserved_4k (0x00000001fe000000--0x0000000200000000)
[    0.000000] OF: reserved mem: OVERLAP DETECTED!
               mmz_nid_1_part_0@21,40000000 (0x0000002140000000--0x0000002400000000) overlaps with d1_g2d_8GB_boundary_reserved_4k (0x00000021fe000000--0x0000002200000000)
[    0.000000] Reserved memory: created CMA memory pool at 0x0000002120000000, size 512 MiB
[    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[    0.000000] OF: reserved mem: 0x0000002120000000..0x000000213fffffff (524288 KiB) map reusable linux,cma
[    0.000000] OF: reserved mem: 0x0000000059000000..0x00000000593fffff (4096 KiB) nomap non-reusable sprammemory@59000000
[    0.000000] OF: reserved mem: 0x0000000079000000..0x00000000793fffff (4096 KiB) nomap non-reusable sprammemory@79000000
[    0.000000] OF: reserved mem: 0x0000000080000000..0x000000008007ffff (512 KiB) nomap non-reusable mmode_resv0@80000000
[    0.000000] OF: reserved mem: 0x00000000dffe0000..0x00000000dfffffff (128 KiB) nomap non-reusable lpcpures@dffe0000
[    0.000000] OF: reserved mem: 0x00000000e0000000..0x00000000e1ffffff (32768 KiB) nomap non-reusable region@e0000000
[    0.000000] OF: reserved mem: 0x00000000fff00000..0x00000000ffffffff (1024 KiB) nomap non-reusable ramoops@fff00000
[    0.000000] Reserved memory: created mmz_nid_0_part_0@1,c0000000 eswin reserve memory at 0x00000001c0000000, size 11264 MiB
[    0.000000] OF: reserved mem: initialized node mmz_nid_0_part_0@1,c0000000, compatible id eswin-reserve-memory
[    0.000000] OF: reserved mem: 0x00000001c0000000..0x000000047fffffff (11534336 KiB) nomap non-reusable mmz_nid_0_part_0@1,c0000000
[    0.000000] OF: reserved mem: 0x00000001fe000000..0x00000001ffffffff (32768 KiB) nomap non-reusable g2d_8GB_boundary_reserved_4k
[    0.000000] OF: reserved mem: 0x00000002fffff000..0x00000002ffffffff (4 KiB) nomap non-reusable g2d_12GB_boundary_reserved_4k
[    0.000000] OF: reserved mem: 0x0000002040000000..0x00000020403fffff (4096 KiB) nomap non-reusable nid_1_zero_device_simu@2040000000
[    0.000000] OF: reserved mem: 0x00000020e0000000..0x00000020e1ffffff (32768 KiB) nomap non-reusable region@20,e0000000
[    0.000000] OF: reserved mem: 0x00000020fffff000..0x00000020ffffffff (4 KiB) nomap non-reusable d1_g2d_4GB_boundary_reserved_4k
[    0.000000] Reserved memory: created mmz_nid_1_part_0@21,40000000 eswin reserve memory at 0x0000002140000000, size 11264 MiB
[    0.000000] OF: reserved mem: initialized node mmz_nid_1_part_0@21,40000000, compatible id eswin-reserve-memory
[    0.000000] OF: reserved mem: 0x0000002140000000..0x00000023ffffffff (11534336 KiB) nomap non-reusable mmz_nid_1_part_0@21,40000000
[    0.000000] OF: reserved mem: 0x00000021fe000000..0x00000021ffffffff (32768 KiB) nomap non-reusable d1_g2d_8GB_boundary_reserved_4k
[    0.000000] OF: NUMA: parsing numa-distance-map-v1
[    0.000000] NUMA: NODE_DATA [mem 0x1bfffe1c0-0x1bfffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x211fcb71c0-0x211fcb8fff]

then I followed https://github.com/DC-DeepComputing/Framework/blob/main/FML13V03/DC-ROMA%20RISC-V%20AI%20PC%20Install%20AI%20Models(Deepseek-7B)%20Guide.md

The single-die model works. Unfortunately, the 2 die does NOT. It just hangs after I enter the question:

root@roma:/home/roma# /opt/eswin/sample-code/npu_sample/qwen_sample/bin/es_qwen2 /opt/eswin/sample-code/npu_sample/qwen_sample/src/deepseek_7b_1k_int8_peer/config.json
Loading models: [==================================================] 100.00% ( 70.834682 seconds )
----------------------------------------------------------------------------------
0: Role setting: 你是一个智能助理.
----------------------------------------------------------------------------------
1: 介绍一下大语言模型
2: The quantum computers
3: Humans and robots coexist
4: Customized prompts
----------------------------------------------------------------------------------
[YOU]: 4
[YOU]: Who are you?

On the serial console I get messages that point toward a bug in some driver blocking the system:

[ 1344.354910] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[ 1344.360906] rcu:     0-...0: (5 ticks this GP) idle=f28c/1/0x4000000000000002 softirq=26168/26170 fqs=7031
[ 1344.370318] rcu:              hardirqs   softirqs   csw/system
[ 1344.375895] rcu:      number: 45651993          0            0
[ 1344.381473] rcu:     cputime:        0          0            0   ==> 30028(ms)
[ 1344.388441] rcu:     (detected by 6, t=15010 jiffies, g=39517, q=632 ncpus=8)
[ 1451.314391] INFO: task kworker/u21:2:332 blocked for more than 120 seconds.
[ 1451.321379]       Not tainted 6.6.92-eic7x-2025.07 #2025.09.26.03.45+
[ 1451.327832] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1451.335841] INFO: task login:834 blocked for more than 120 seconds.
[ 1451.342117]       Not tainted 6.6.92-eic7x-2025.07 #2025.09.26.03.45+
[ 1451.348571] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

If I press C-c I get:

[YOU]: Who are you?
^C^Cterminate called after throwing an instance of 'std::system_error'
  what():  Resource deadlock avoided
^C^C^C^C

and actually the system is deadlocked for good. I am just reading the documentation linked from another issue so basically I don't know what I am doing, but I would like to see the demo working before trying anything more complex.

Thanks!

[DC-DeepComputing]

I followed https://github.com/DC-DeepComputing/Framework/blob/main/FML13V03/DC-ROMA%20RISC-V%20AI%20PC%2C%20RISC-V%20Mainboard%20II%20NPU%20Memory%20Adjustment%20Instructions.md for the 32G RAM version (which I have) to expand the reserved memory. Here the dmesg output:

[    0.000000] Linux version 6.6.92-eic7x-2025.07 (root@b67a2314ece6) (riscv64-unknown-linux-gnu-gcc () 13.2.0, GNU ld (GNU Binutils) 2.42) #2025.09.26.03.45+ SMP Fri Sep 26 03:53:01 UTC 2025
[    0.000000] Machine model: DeepComputing FML13V03
[    0.000000] SBI specification v1.0 detected
[    0.000000] SBI implementation ID=0x1 Version=0x10003
[    0.000000] SBI TIME extension detected
[    0.000000] SBI IPI extension detected
[    0.000000] SBI RFENCE extension detected
[    0.000000] SBI SRST extension detected
[    0.000000] earlycon: sbi0 at I/O port 0x0 (options '')
[    0.000000] printk: bootconsole [sbi0] enabled
[    0.000000] efi: EFI v2.10 by Das U-Boot
[    0.000000] efi: RTPROP=0xe8cc8040 SMBIOS=0xe8cf5000 INITRD=0xe33bc040 MEMRESERVE=0xe33bb040 
[    0.000000] OF: reserved mem: OVERLAP DETECTED!
               mmz_nid_0_part_0@1,c0000000 (0x00000001c0000000--0x0000000480000000) overlaps with g2d_8GB_boundary_reserved_4k (0x00000001fe000000--0x0000000200000000)
[    0.000000] OF: reserved mem: OVERLAP DETECTED!
               mmz_nid_1_part_0@21,40000000 (0x0000002140000000--0x0000002400000000) overlaps with d1_g2d_8GB_boundary_reserved_4k (0x00000021fe000000--0x0000002200000000)
[    0.000000] Reserved memory: created CMA memory pool at 0x0000002120000000, size 512 MiB
[    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[    0.000000] OF: reserved mem: 0x0000002120000000..0x000000213fffffff (524288 KiB) map reusable linux,cma
[    0.000000] OF: reserved mem: 0x0000000059000000..0x00000000593fffff (4096 KiB) nomap non-reusable sprammemory@59000000
[    0.000000] OF: reserved mem: 0x0000000079000000..0x00000000793fffff (4096 KiB) nomap non-reusable sprammemory@79000000
[    0.000000] OF: reserved mem: 0x0000000080000000..0x000000008007ffff (512 KiB) nomap non-reusable mmode_resv0@80000000
[    0.000000] OF: reserved mem: 0x00000000dffe0000..0x00000000dfffffff (128 KiB) nomap non-reusable lpcpures@dffe0000
[    0.000000] OF: reserved mem: 0x00000000e0000000..0x00000000e1ffffff (32768 KiB) nomap non-reusable region@e0000000
[    0.000000] OF: reserved mem: 0x00000000fff00000..0x00000000ffffffff (1024 KiB) nomap non-reusable ramoops@fff00000
[    0.000000] Reserved memory: created mmz_nid_0_part_0@1,c0000000 eswin reserve memory at 0x00000001c0000000, size 11264 MiB
[    0.000000] OF: reserved mem: initialized node mmz_nid_0_part_0@1,c0000000, compatible id eswin-reserve-memory
[    0.000000] OF: reserved mem: 0x00000001c0000000..0x000000047fffffff (11534336 KiB) nomap non-reusable mmz_nid_0_part_0@1,c0000000
[    0.000000] OF: reserved mem: 0x00000001fe000000..0x00000001ffffffff (32768 KiB) nomap non-reusable g2d_8GB_boundary_reserved_4k
[    0.000000] OF: reserved mem: 0x00000002fffff000..0x00000002ffffffff (4 KiB) nomap non-reusable g2d_12GB_boundary_reserved_4k
[    0.000000] OF: reserved mem: 0x0000002040000000..0x00000020403fffff (4096 KiB) nomap non-reusable nid_1_zero_device_simu@2040000000
[    0.000000] OF: reserved mem: 0x00000020e0000000..0x00000020e1ffffff (32768 KiB) nomap non-reusable region@20,e0000000
[    0.000000] OF: reserved mem: 0x00000020fffff000..0x00000020ffffffff (4 KiB) nomap non-reusable d1_g2d_4GB_boundary_reserved_4k
[    0.000000] Reserved memory: created mmz_nid_1_part_0@21,40000000 eswin reserve memory at 0x0000002140000000, size 11264 MiB
[    0.000000] OF: reserved mem: initialized node mmz_nid_1_part_0@21,40000000, compatible id eswin-reserve-memory
[    0.000000] OF: reserved mem: 0x0000002140000000..0x00000023ffffffff (11534336 KiB) nomap non-reusable mmz_nid_1_part_0@21,40000000
[    0.000000] OF: reserved mem: 0x00000021fe000000..0x00000021ffffffff (32768 KiB) nomap non-reusable d1_g2d_8GB_boundary_reserved_4k
[    0.000000] OF: NUMA: parsing numa-distance-map-v1
[    0.000000] NUMA: NODE_DATA [mem 0x1bfffe1c0-0x1bfffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x211fcb71c0-0x211fcb8fff]

then I followed https://github.com/DC-DeepComputing/Framework/blob/main/FML13V03/DC-ROMA%20RISC-V%20AI%20PC%20Install%20AI%20Models(Deepseek-7B)%20Guide.md

The single-die model works. Unfortunately, the 2 die does NOT. It just hangs after I enter the question:

root@roma:/home/roma# /opt/eswin/sample-code/npu_sample/qwen_sample/bin/es_qwen2 /opt/eswin/sample-code/npu_sample/qwen_sample/src/deepseek_7b_1k_int8_peer/config.json
Loading models: [==================================================] 100.00% ( 70.834682 seconds )
----------------------------------------------------------------------------------
0: Role setting: 你是一个智能助理.
----------------------------------------------------------------------------------
1: 介绍一下大语言模型
2: The quantum computers
3: Humans and robots coexist
4: Customized prompts
----------------------------------------------------------------------------------
[YOU]: 4
[YOU]: Who are you?

On the serial console I get messages that point toward a bug in some driver blocking the system:

[ 1344.354910] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[ 1344.360906] rcu:     0-...0: (5 ticks this GP) idle=f28c/1/0x4000000000000002 softirq=26168/26170 fqs=7031
[ 1344.370318] rcu:              hardirqs   softirqs   csw/system
[ 1344.375895] rcu:      number: 45651993          0            0
[ 1344.381473] rcu:     cputime:        0          0            0   ==> 30028(ms)
[ 1344.388441] rcu:     (detected by 6, t=15010 jiffies, g=39517, q=632 ncpus=8)
[ 1451.314391] INFO: task kworker/u21:2:332 blocked for more than 120 seconds.
[ 1451.321379]       Not tainted 6.6.92-eic7x-2025.07 #2025.09.26.03.45+
[ 1451.327832] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1451.335841] INFO: task login:834 blocked for more than 120 seconds.
[ 1451.342117]       Not tainted 6.6.92-eic7x-2025.07 #2025.09.26.03.45+
[ 1451.348571] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

If I press C-c I get:

[YOU]: Who are you?
^C^Cterminate called after throwing an instance of 'std::system_error'
  what():  Resource deadlock avoided
^C^C^C^C

and actually the system is deadlocked for good. I am just reading the documentation linked from another issue so basically I don't know what I am doing, but I would like to see the demo working before trying anything more complex.

Thanks!

I've pinpointed the issue: your system is experiencing an RCU Scheduler Stall. Since we haven't encountered this scenario before, it may be related to your kernel version or kernel drivers. Have you installed any custom drivers or third-party modules?
Could you please provide your kernel version information? I will attempt to reproduce your issue.

[chripell]

I think that the RCU error message is just one of the many effects (see also workqueues/tasks that don't get scheduled) of something more destructive, like some code that disabled interrupts or memory corruption. I didn't update anything on the board (it is the same version of everything as arrived and it runs via ssh/serial port (I am saying this because I see, in dmesg, a memory conflict of the NPU memory with something that sounds related to the graphic accelerator). I also checked that content of https://www.eswincomputing.com/community/api/uploads/2025/10/21/17610330745f0343f1535bc4ab.gz is the same that arrived on the disk. BTW thanks for finally providing enough material (in another thread) to get started with installation of additional models, as soon I validate that the HW works I will definitely try it.

Today I just turned on the system and I noticed another symptom. Even if I see no output after running the command, the fans spin up, as if something is actually computing. My uneducated guest is that something is wrong in the memory mapping for the 32G version perhaps and it causes memory corruption in addition to not working. Unfortunately, there is ton of documentation to go through.

Anyway, let me attach full dmesg (you can get kernel version and all there) and output of some commands, before I send a prompt that kills the system. Let me know if you need some more specific information.

Thanks!

procinterrupts.txt
extracted-dts.txt
journal.txt
free.txt
psfax.txt
lsmod.txt
dmesg.txt

[DC-DeepComputing]

I think that the RCU error message is just one of the many effects (see also workqueues/tasks that don't get scheduled) of something more destructive, like some code that disabled interrupts or memory corruption. I didn't update anything on the board (it is the same version of everything as arrived and it runs via ssh/serial port (I am saying this because I see, in dmesg, a memory conflict of the NPU memory with something that sounds related to the graphic accelerator). I also checked that content of https://www.eswincomputing.com/community/api/uploads/2025/10/21/17610330745f0343f1535bc4ab.gz is the same that arrived on the disk. BTW thanks for finally providing enough material (in another thread) to get started with installation of additional models, as soon I validate that the HW works I will definitely try it.

Today I just turned on the system and I noticed another symptom. Even if I see no output after running the command, the fans spin up, as if something is actually computing. My uneducated guest is that something is wrong in the memory mapping for the 32G version perhaps and it causes memory corruption in addition to not working. Unfortunately, there is ton of documentation to go through.

Anyway, let me attach full dmesg (you can get kernel version and all there) and output of some commands, before I send a prompt that kills the system. Let me know if you need some more specific information.

Thanks!

procinterrupts.txt extracted-dts.txt journal.txt free.txt psfax.txt lsmod.txt dmesg.txt

I checked your kernel logs, and they show no hardware failures. I ran tests on my 32GB setup, and the results are completely opposite to yours, as follows:

roma@roma ~> sudo /opt/eswin/sample-code/npu_sample/qwen_sample/bin/es_qwen2 /opt/eswin/sample-code/npu_sample/qwen_sample/src/deepseek_7b_1k_int8_peer/config.json
 [==================================================] 100.00% ( 68.458277 seconds )
----------------------------------------------------------------------------------
0: Role setting: 你是一个智能助理.
----------------------------------------------------------------------------------
1: 介绍一下大语言模型
2: The quantum computers
3: Humans and robots coexist
4: Customized prompts
----------------------------------------------------------------------------------
[YOU]: 4
[YOU]: who are you?
[Qwen2]: <think>

</think>

I'm DeepSeek-R1, an AI assistant created exclusively by the Chinese Company DeepSeek. I'm excited to assist you! If you have any questions or need any help, feel free to ask.
--------------------------------------------------------
Throughput: 9.10048tokens/s
--------------------------------------------------------

I'm not quite sure what exactly is causing the runtime errors on your device. Could it be that the model files are corrupted? Would you try reinstalling it?

[chripell]

Still no luck, I reinstalled everything from the tar.gz :-( Just to double check I am doing the right thing: I need to use all the components that I find in https://www.eswincomputing.com/community/api/uploads/2025/10/21/17610330745f0343f1535bc4ab.gz , right? I mean also the files like /opt/eswin/sample-code/npu_sample/qwen_sample/bin/es_qwen2 is different from what I see preinstalled. Now I reliability get a kernel SEGFAULT at least, that I attach below.

Might I ask if you can do a .tar.gz of /opt/eswin/sample-code/npu_sample/ that is working for you for a 32G board and share it? And share your correct dmesg output? Thanks!

Now I get:

[19672.862289] Unable to handle kernel paging request at virtual address ffffffffffffffea
[19672.862289] Unable to handle kernel paging request at virtual address ffffffffffffffea
[19672.870243] Oops [#1]
[19672.870249] Modules linked in: tls rfcomm algif_hash algif_skcipher af_alg qrtr ax88796b bnep btusb btmtk btrtl btbcm binfmt_misc btintel apple_mfi_fastcharge asix usbnet cros_usbpd_charger cros_sbs_battery cros_usbpd_notify cros_ec_sysfs bluetooth cr4
[19672.930350] CPU: 3 PID: 5226 Comm: es_qwen2 Not tainted 6.6.92-eic7x-2025.07 #2025.09.26.03.45+
[19672.939052] Hardware name: DeepComputing FML13V03 (DT)
[19672.944189] epc : io_tensor_to_io_addr+0x2a6/0x4ea [eic7700_npu]
[19672.951640]  ra : io_tensor_to_io_addr+0x294/0x4ea [eic7700_npu]
[19672.958028] epc : ffffffff02d31358 ra : ffffffff02d31346 sp : ffff8f8035ca3d10
[19672.965250]  gp : ffffffff825e2ae8 tp : ffffaf9fd9439d00 t0 : 0000000000000001
[19672.972475]  t1 : 0000000000000004 t2 : 0000000000000000 s0 : ffff8f8035ca3dc0
[19672.979698]  s1 : ffffaf80cefe9020 a0 : ffffffffffffffea a1 : ffffaf9fd0f21800
[19672.986920]  a2 : 0000000000000001 a3 : ffffffff82028080 a4 : 0000000000000004
[19672.994142]  a5 : ffffffffffffffff a6 : 0000000000000000 a7 : ffffaf80d884d430
[19673.001365]  s2 : 0000000000000004 s3 : ffffffff02d3c0f8 s4 : 0000000000000000
[19673.008588]  s5 : 0000000000000004 s6 : ffffaf80cefe93f0 s7 : 0000000000000001
[19673.015810]  s8 : ffffaf80cefe9310 s9 : 0000000000000000 s10: 0000000000000004
[19673.023033]  s11: ffffaf80c71ac000 t3 : 0000000000000000 t4 : 0000000000000008
[19673.030255]  t5 : 0000000000000820 t6 : ffff8f8035ca38e0
[19673.035568] status: 0000000200000120 badaddr: ffffffffffffffea cause: 000000000000000f
[19673.043486] [<ffffffff02d31358>] io_tensor_to_io_addr+0x2a6/0x4ea [eic7700_npu]
[19673.051098] [<ffffffff02d2e8ca>] npu_dev_ioctl+0x7fe/0x108a [eic7700_npu]
[19673.058094] [<ffffffff802a7e5a>] __riscv_sys_ioctl+0x86/0xae
[19673.063763] [<ffffffff8109958e>] do_trap_ecall_u+0x116/0x128
[19673.069427] [<ffffffff810a43ec>] ret_from_exception+0x0/0x64
[19673.075093] Code: b0c0 3783 f784 3703 f604 94be b023 2ca4 c155 57fd (c11c) be03 
[19673.082496] Oops [#2]
[19673.082566] ---[ end trace 0000000000000000 ]---
[19673.084798] Modules linked in: tls rfcomm algif_hash algif_skcipher af_alg qrtr ax88796b bnep btusb btmtk btrtl btbcm binfmt_misc btintel apple_mfi_fastcharge asix usbnet cros_usbpd_charger cros_sbs_battery cros_usbpd_notify cros_ec_sysfs bluetooth cr4
[19673.139396] CPU: 4 PID: 5229 Comm: es_qwen2 Tainted: G      D            6.6.92-eic7x-2025.07 #2025.09.26.03.45+
[19673.149586] Hardware name: DeepComputing FML13V03 (DT)
[19673.154730] epc : io_tensor_to_io_addr+0x2a6/0x4ea [eic7700_npu]
[19673.160917]  ra : io_tensor_to_io_addr+0x294/0x4ea [eic7700_npu]
[19673.167030] epc : ffffffff02d31358 ra : ffffffff02d31346 sp : ffff8f8038a7bd10
[19673.174265]  gp : ffffffff825e2ae8 tp : ffffaf9fd7f68000 t0 : 0000000000000001
[19673.181494]  t1 : 0000000000000004 t2 : 0000000000000000 s0 : ffff8f8038a7bdc0
[19673.188721]  s1 : ffffaf9fd0fd7020 a0 : ffffffffffffffea a1 : ffffaf9fd0f21800
[19673.195950]  a2 : 0000000000000001 a3 : ffffffff82028080 a4 : 0000000000000004
[19673.203176]  a5 : ffffffffffffffff a6 : 0000000000000000 a7 : ffffaf80d884d430
[19673.210404]  s2 : 0000000000000004 s3 : ffffffff02d3c0f8 s4 : 0000000000000000
[19673.217633]  s5 : 0000000000000004 s6 : ffffaf9fd0fd73f0 s7 : 0000000000000001
[19673.224860]  s8 : ffffaf9fd0fd7310 s9 : 0000000000000000 s10: 0000000000000004
[19673.232125]  s11: ffffaf9fd1bac000 t3 : 0000000000000000 t4 : 0000000000000008
[19673.232137]  t5 : 0000000000000820 t6 : ffff8f8038a7b8e0
[19673.232143] status: 0000000200000120 badaddr: ffffffffffffffea cause: 000000000000000f
[19673.232155] [<ffffffff02d31358>] io_tensor_to_io_addr+0x2a6/0x4ea [eic7700_npu]
[19673.259955] [<ffffffff02d2e8ca>] npu_dev_ioctl+0x7fe/0x108a [eic7700_npu]
[19673.266846] [<ffffffff802a7e5a>] __riscv_sys_ioctl+0x86/0xae
[19673.272544] [<ffffffff8109958e>] do_trap_ecall_u+0x116/0x128
[19673.278222] [<ffffffff810a43ec>] ret_from_exception+0x0/0x64
[19673.283903] Code: b0c0 3783 f784 3703 f604 94be b023 2ca4 c155 57fd (c11c) be03 
[19673.292058] ---[ end trace 0000000000000000 ]---

[chripell]

So, with pre-installed version of es_qwen2 I get the lockup with no dmesg output, with the one from the archive I get the kernle ooops, so I hope it is a software misalignment.

[DC-DeepComputing]

Still no luck, I reinstalled everything from the tar.gz :-( Just to double check I am doing the right thing: I need to use all the components that I find in https://www.eswincomputing.com/community/api/uploads/2025/10/21/17610330745f0343f1535bc4ab.gz , right? I mean also the files like /opt/eswin/sample-code/npu_sample/qwen_sample/bin/es_qwen2 is different from what I see preinstalled. Now I reliability get a kernel SEGFAULT at least, that I attach below.

Might I ask if you can do a .tar.gz of /opt/eswin/sample-code/npu_sample/ that is working for you for a 32G board and share it? And share your correct dmesg output? Thanks!

Now I get:

[19672.862289] Unable to handle kernel paging request at virtual address ffffffffffffffea
[19672.862289] Unable to handle kernel paging request at virtual address ffffffffffffffea
[19672.870243] Oops [#1]
[19672.870249] Modules linked in: tls rfcomm algif_hash algif_skcipher af_alg qrtr ax88796b bnep btusb btmtk btrtl btbcm binfmt_misc btintel apple_mfi_fastcharge asix usbnet cros_usbpd_charger cros_sbs_battery cros_usbpd_notify cros_ec_sysfs bluetooth cr4
[19672.930350] CPU: 3 PID: 5226 Comm: es_qwen2 Not tainted 6.6.92-eic7x-2025.07 #2025.09.26.03.45+
[19672.939052] Hardware name: DeepComputing FML13V03 (DT)
[19672.944189] epc : io_tensor_to_io_addr+0x2a6/0x4ea [eic7700_npu]
[19672.951640]  ra : io_tensor_to_io_addr+0x294/0x4ea [eic7700_npu]
[19672.958028] epc : ffffffff02d31358 ra : ffffffff02d31346 sp : ffff8f8035ca3d10
[19672.965250]  gp : ffffffff825e2ae8 tp : ffffaf9fd9439d00 t0 : 0000000000000001
[19672.972475]  t1 : 0000000000000004 t2 : 0000000000000000 s0 : ffff8f8035ca3dc0
[19672.979698]  s1 : ffffaf80cefe9020 a0 : ffffffffffffffea a1 : ffffaf9fd0f21800
[19672.986920]  a2 : 0000000000000001 a3 : ffffffff82028080 a4 : 0000000000000004
[19672.994142]  a5 : ffffffffffffffff a6 : 0000000000000000 a7 : ffffaf80d884d430
[19673.001365]  s2 : 0000000000000004 s3 : ffffffff02d3c0f8 s4 : 0000000000000000
[19673.008588]  s5 : 0000000000000004 s6 : ffffaf80cefe93f0 s7 : 0000000000000001
[19673.015810]  s8 : ffffaf80cefe9310 s9 : 0000000000000000 s10: 0000000000000004
[19673.023033]  s11: ffffaf80c71ac000 t3 : 0000000000000000 t4 : 0000000000000008
[19673.030255]  t5 : 0000000000000820 t6 : ffff8f8035ca38e0
[19673.035568] status: 0000000200000120 badaddr: ffffffffffffffea cause: 000000000000000f
[19673.043486] [<ffffffff02d31358>] io_tensor_to_io_addr+0x2a6/0x4ea [eic7700_npu]
[19673.051098] [<ffffffff02d2e8ca>] npu_dev_ioctl+0x7fe/0x108a [eic7700_npu]
[19673.058094] [<ffffffff802a7e5a>] __riscv_sys_ioctl+0x86/0xae
[19673.063763] [<ffffffff8109958e>] do_trap_ecall_u+0x116/0x128
[19673.069427] [<ffffffff810a43ec>] ret_from_exception+0x0/0x64
[19673.075093] Code: b0c0 3783 f784 3703 f604 94be b023 2ca4 c155 57fd (c11c) be03 
[19673.082496] Oops [#2]
[19673.082566] ---[ end trace 0000000000000000 ]---
[19673.084798] Modules linked in: tls rfcomm algif_hash algif_skcipher af_alg qrtr ax88796b bnep btusb btmtk btrtl btbcm binfmt_misc btintel apple_mfi_fastcharge asix usbnet cros_usbpd_charger cros_sbs_battery cros_usbpd_notify cros_ec_sysfs bluetooth cr4
[19673.139396] CPU: 4 PID: 5229 Comm: es_qwen2 Tainted: G      D            6.6.92-eic7x-2025.07 #2025.09.26.03.45+
[19673.149586] Hardware name: DeepComputing FML13V03 (DT)
[19673.154730] epc : io_tensor_to_io_addr+0x2a6/0x4ea [eic7700_npu]
[19673.160917]  ra : io_tensor_to_io_addr+0x294/0x4ea [eic7700_npu]
[19673.167030] epc : ffffffff02d31358 ra : ffffffff02d31346 sp : ffff8f8038a7bd10
[19673.174265]  gp : ffffffff825e2ae8 tp : ffffaf9fd7f68000 t0 : 0000000000000001
[19673.181494]  t1 : 0000000000000004 t2 : 0000000000000000 s0 : ffff8f8038a7bdc0
[19673.188721]  s1 : ffffaf9fd0fd7020 a0 : ffffffffffffffea a1 : ffffaf9fd0f21800
[19673.195950]  a2 : 0000000000000001 a3 : ffffffff82028080 a4 : 0000000000000004
[19673.203176]  a5 : ffffffffffffffff a6 : 0000000000000000 a7 : ffffaf80d884d430
[19673.210404]  s2 : 0000000000000004 s3 : ffffffff02d3c0f8 s4 : 0000000000000000
[19673.217633]  s5 : 0000000000000004 s6 : ffffaf9fd0fd73f0 s7 : 0000000000000001
[19673.224860]  s8 : ffffaf9fd0fd7310 s9 : 0000000000000000 s10: 0000000000000004
[19673.232125]  s11: ffffaf9fd1bac000 t3 : 0000000000000000 t4 : 0000000000000008
[19673.232137]  t5 : 0000000000000820 t6 : ffff8f8038a7b8e0
[19673.232143] status: 0000000200000120 badaddr: ffffffffffffffea cause: 000000000000000f
[19673.232155] [<ffffffff02d31358>] io_tensor_to_io_addr+0x2a6/0x4ea [eic7700_npu]
[19673.259955] [<ffffffff02d2e8ca>] npu_dev_ioctl+0x7fe/0x108a [eic7700_npu]
[19673.266846] [<ffffffff802a7e5a>] __riscv_sys_ioctl+0x86/0xae
[19673.272544] [<ffffffff8109958e>] do_trap_ecall_u+0x116/0x128
[19673.278222] [<ffffffff810a43ec>] ret_from_exception+0x0/0x64
[19673.283903] Code: b0c0 3783 f784 3703 f604 94be b023 2ca4 c155 57fd (c11c) be03 
[19673.292058] ---[ end trace 0000000000000000 ]---

It seems that the driver is faulty. I recommend you reinstall with a clean Ubuntu image: http://120.92.155.32:8082/artifactory/virtOS/fml13v03-eswin/15009-ubuntu-24.04-desktop-grub-sdcard.img.zip. Please give it another try after reinstalling the system.

I've already packaged my local model. How can I share it with you for analysis?

[chripell]

Thanks, I will reinstall it today and report. Feel free to use Google Drive (as in other issues) if it suits you.

[chripell]

Wait, just a question: I noticed that there are newer images on the site: http://120.92.155.32:8082/artifactory/virtOS/fml13v03-eswin/16499-ubuntu-24.04-desktop-grub-sdcard.img.zip

Do you really want me to try the older first?

[chripell]

Also, let me note something that I find a bit scaring. In the guide about re-flashing: https://github.com/DC-DeepComputing/Framework/blob/main/FML13V03/How%20to%20Install%20an%20Image%20System%20on%20an%20SSD%20(e.g.%20Ubuntu).pdf the first option is really badly described. It is not written anywhere that you need to boot from something different than the standard image, otherwise you are going to write a file system in use, which might lead to all kinds of disasters:

root@roma:/home/roma# mount | grep nvme0n1
/dev/nvme0n1p3 on / type ext4 (rw,relatime)
/dev/nvme0n1p1 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)

IMHO, you should consider having a "recovery partition" or provide instructions about how to boot a rootfs from USB.

[DC-DeepComputing]

Wait, just a question: I noticed that there are newer images on the site: http://120.92.155.32:8082/artifactory/virtOS/fml13v03-eswin/16499-ubuntu-24.04-desktop-grub-sdcard.img.zip

Do you really want me to try the older first?

No no no, just use the latest image. The link hasn't been updated in time yet, hahaha!

[DC-DeepComputing]

Also, let me note something that I find a bit scaring. In the guide about re-flashing: https://github.com/DC-DeepComputing/Framework/blob/main/FML13V03/How%20to%20Install%20an%20Image%20System%20on%20an%20SSD%20(e.g.%20Ubuntu).pdf the first option is really badly described. It is not written anywhere that you need to boot from something different than the standard image, otherwise you are going to write a file system in use, which might lead to all kinds of disasters:

root@roma:/home/roma# mount | grep nvme0n1
/dev/nvme0n1p3 on / type ext4 (rw,relatime)
/dev/nvme0n1p1 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)

IMHO, you should consider having a "recovery partition" or provide instructions about how to boot a rootfs from USB.

Also, let me note something that I find a bit scaring. In the guide about re-flashing: https://github.com/DC-DeepComputing/Framework/blob/main/FML13V03/How%20to%20Install%20an%20Image%20System%20on%20an%20SSD%20(e.g.%20Ubuntu).pdf the first option is really badly described. It is not written anywhere that you need to boot from something different than the standard image, otherwise you are going to write a file system in use, which might lead to all kinds of disasters:

root@roma:/home/roma# mount | grep nvme0n1
/dev/nvme0n1p3 on / type ext4 (rw,relatime)
/dev/nvme0n1p1 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)

IMHO, you should consider having a "recovery partition" or provide instructions about how to boot a rootfs from USB.

I understand your point. Therefore, the first method is not recommended, and the second method is more recommended. The first method is suitable for USB boot.

However, the machine's default boot order is NVMe priority, so if you need to boot from USB, you need to change the boot order in uboot.

[chripell]

I reinstalled everything with the latest image and 17610330745f0343f1535bc4ab.gz version and still nothing. Now I get this in dmesg.

[12459.619656] arm-smmu-v3 50c00000.iommu: event 0x10 received:
[12459.619671] arm-smmu-v3 50c00000.iommu:      0x0000001200000010
[12459.619677] arm-smmu-v3 50c00000.iommu:      0x0000020800000000
[12459.619682] arm-smmu-v3 50c00000.iommu:      0x00000000fdc00000
[12459.619687] arm-smmu-v3 50c00000.iommu:      0x0000000000000000
[12459.619693] arm-smmu-v3 50c00000.iommu: event 0x10 received:
[12459.619698] arm-smmu-v3 50c00000.iommu:      0x0000001200000010
[12459.619703] arm-smmu-v3 50c00000.iommu:      0x0000020800000000
[12459.619708] arm-smmu-v3 50c00000.iommu:      0x00000000fdc00000
[12459.619713] arm-smmu-v3 50c00000.iommu:      0x0000000000000000
[12459.622540] arm-smmu-v3 70c00000.iommu: event 0x10 received:
[12459.622590] arm-smmu-v3 70c00000.iommu:      0x0000001200000010
[12459.622599] arm-smmu-v3 70c00000.iommu:      0x0000020800000000
[12459.622605] arm-smmu-v3 70c00000.iommu:      0x00000000fdc00000
[12459.622611] arm-smmu-v3 70c00000.iommu:      0x0000000000000000
[12459.622619] arm-smmu-v3 70c00000.iommu: event 0x10 received:
[12459.622779] arm-smmu-v3 70c00000.iommu:      0x0000001200000010
[12459.622790] arm-smmu-v3 70c00000.iommu:      0x0000020800000000
[12459.622797] arm-smmu-v3 70c00000.iommu:      0x00000000fdc00000
[12459.622803] arm-smmu-v3 70c00000.iommu:      0x0000000000000000
[12490.006364] eswin_npu 51c00000.eswin-npu: npu tik frame timeout.
[12490.006385] eswin_npu 71c00000.eswin-npu: npu tik frame timeout.
[12490.006364] eswin_npu 51c00000.eswin-npu: npu tik frame timeout.
[12490.006385] eswin_npu 71c00000.eswin-npu: npu tik frame timeout.

I start suspecting bad hardware. I will contact you via email about next steps.

[chripell]

Interestingly, now also the single TPU model doesn't work, but I see half of messages in dmesg:

[ 1496.823878] arm-smmu-v3 50c00000.iommu: event 0x10 received:
[ 1496.823896] arm-smmu-v3 50c00000.iommu:      0x0000001200000010
[ 1496.823902] arm-smmu-v3 50c00000.iommu:      0x0000020800000000
[ 1496.823907] arm-smmu-v3 50c00000.iommu:      0x00000000fdc00000
[ 1496.823912] arm-smmu-v3 50c00000.iommu:      0x0000000000000000
[ 1496.823919] arm-smmu-v3 50c00000.iommu: event 0x10 received:
[ 1496.823924] arm-smmu-v3 50c00000.iommu:      0x0000001200000010
[ 1496.823929] arm-smmu-v3 50c00000.iommu:      0x0000020800000000
[ 1496.823934] arm-smmu-v3 50c00000.iommu:      0x00000000fdc00000
[ 1496.823939] arm-smmu-v3 50c00000.iommu:      0x0000000000000000
[ 1527.130903] eswin_npu 51c00000.eswin-npu: npu tik frame timeout.
[ 1527.130903] eswin_npu 51c00000.eswin-npu: npu tik frame timeout.

Let me add the full dmesg for single and double TPU cases. I also add ps fax for the double case, so you see what is running on the system.

dmesg.double.txt
dmesg.single.txt
psfax.double.txt

[chripell]

The explanation of the error by Gemini is interesting, it still gives me the hope that there is a memory map issue. May I ask again for the dmesg on a working system?

These messages indicate a Translation Fault occurring in the Input/Output Memory Management Unit (IOMMU).

Even though you are running on a RISC-V architecture (the EIC7x SoC), the chip designer (likely ESWIN) utilized ARM's SMMUv3 Intellectual Property (IP) block for memory management. The driver is reporting that a hardware peripheral tried to access a memory address that was not mapped or allowed.

Here is the breakdown of what the hexadecimal dump means.

1. The Core Error: Event 0x10

The key indicator is this line: arm-smmu-v3 70c00000.iommu: event 0x10 received:

According to the ARM SMMUv3 Architecture Specification, Event ID 0x10 corresponds to:

F_TL_PF (Translation Fault) This means a page fault occurred during the translation process. A device requested an address, but the SMMU could not find a valid page table entry for it, or the entry was marked invalid.

2. Decoding the Hex Data

The SMMU outputs a 256-bit (32-byte) event record, printed as four 64-bit hexadecimal words. Here is the decoding of the specific values from your log:

Word | Log Value | Interpretation -- | -- | -- 0 | 0x0000001200000010 | Event Metadata• Lower 8 bits (0x10): The Event ID (F_TL_PF).• Upper 32 bits (0x00000012): The StreamID (SID). This is the culprit. StreamID 0x12 (decimal 18) identifies the specific hardware device causing the fault. 1 | 0x0000020800000000 | Flags & AttributesContains permission flags (Read/Write) and privilege levels associated with the request. The bit 0x200 generally indicates the type of transaction (e.g., Read vs. Write). 2 | 0x00000000fdc00000 | Input Address (IOVA)The device tried to access virtual address 0xfdc00000. 3 | 0x0000000000000000 | Physical AddressZero, because the translation failed, so there is no physical address to report.

Export to Sheets

3. Summary of the Situation

• The Problem: A hardware component (StreamID 0x12) tried to access memory address 0xfdc00000.

• The Blocker: The IOMMU blocked this access because the operating system (Linux) had not allocated that memory for the device, or the driver setup for that device is incorrect.

• The Platform: You are seeing this on RISC-V because modern SoCs are "mix-and-match." The EIC7700 CPU cores are RISC-V, but the memory controller for peripherals is the standard ARM SMMUv3.

4. Common Causes on Embedded Linux

1. Driver Bug: A kernel driver allocated a buffer, passed the address to the hardware, but failed to "map" it through the DMA/IOMMU API.

2. U-Boot/Firmware Residue: Sometimes the bootloader sets up a device (like a splash screen on a display controller) to read from an address. When Linux boots, it resets page tables. If the device is still active and trying to read that old address (0xfdc00000), it triggers this fault.

3. Device Tree Error: The dma-ranges or iommus property in the device tree might be misconfigured for the device with StreamID 0x12.

---

How to Identify the Culprit (StreamID 0x12)

To fix this, you first need to know which device owns StreamID 0x12 (18).

Step 1: Check IOMMU Groups Run the following command to see how devices are grouped. Look for Group 18 (or similar, grouping logic can vary):

Bash

ls -l /sys/kernel/iommu_groups/*/devices/

Step 2: Check DebugFS (If enabled) If you have debugfs mounted, you might find the mapping here:

Bash

cat /sys/kernel/debug/iommu/arm-smmu-v3/masters

Step 3: Check Device Tree If you have access to the source code or dtc, look at the .dts (Device Tree) file. Search for iommus = <&smmu ...> entries. The second number in that tuple is often the StreamID.

• Example: iommus = <&apps_smmu 0x12 0x0>;

Given the address 0xfdc00000 (which looks like a typical MMIO or high-memory buffer), this is likely a Video Decoder (VDEC), GPU, or Display Controller (DC).

Next Step

Would you like me to help you command-line instructions to dump the Device Tree from your running system so we can grep for StreamID 0x12 together?

[chripell]

After chatting with Gemini https://gemini.google.com/share/143ec923f5f7 I got this suggestion (I need to tweak uboot to try it):

You have provided the crucial evidence needed to solve this. Comparing your dmesg logs with the expected behavior of the EIC7700 SoC confirms that the Device Tree is indeed missing a critical memory reservation for the NPU.

Here is the analysis of the files and the logs you provided, followed by the specific fix.

1. The Diagnosis: Missing "Mailbox" Memory

Your dmesg confirms that both NPU cores (on Die 0 and Die 1) crashed because they tried to access physical address 0xfdc00000, but the OS had not reserved it.

Evidence A (The Crash): At timestamp 149.284, both SMMUs blocked access to 0xfdc00000. At timestamp 179.547 (30s later), eswin_npu timed out waiting for a response.

Evidence B (The Missing Node): In your dmesg (lines 372-381), the kernel prints the reserved memory map.

    It lists ramoops@fff00000 and region@e0000000.

    It does NOT list 0xfdc00000. This confirms system-faulty.txt is missing this node.

2. The Fix: Porting the Node from the EVB DTS

You need to copy the missing reserved-memory node from the "good" file (eic7702-evb-a1.txt) into your running Device Tree.

Step 1: Locate the missing node Open eic7702-evb-a1.txt and search for fdc00000. You will likely find a block that looks exactly like this:
DTS

/* FROM eic7702-evb-a1.txt (The "Good" File) /
reserved-memory {
/ ... other nodes ... */

npu_share_reserved: npu-share-memory@fdc00000 {
    compatible = "eswin,npu-share-memory"; /* or shared-dma-pool */
    reg = <0x0 0xfdc00000 0x0 0x200000>;    /* Size might be 2MB (0x200000) or 4MB */
    no-map;
};

};

Step 2: Add it to your system-faulty.dts

Open your system-faulty.dts (converted to source).

Find the reserved-memory section (search for reserved-memory {).

Paste the npu-share-memory@fdc00000 node inside that block.

Step 3: Link the NPU to this memory The NPU device node needs to know this memory exists. Search system-faulty.dts for the NPU nodes (likely npu@51c00000 and npu@71c00000). Ensure they have a memory-region property pointing to the node you just added.

It should look like this:
DTS

/* NPU Node 0 /
&npu0 {
compatible = "eswin,eic7700-npu";
/ ... /
memory-region = <&npu_share_reserved>; / <--- This line is key */
};

/* NPU Node 1 /
&npu1 {
/ ... */
memory-region = <&npu_share_reserved>;
};

(Note: If you use a phandle like <0x2b>, ensure you define the label npu_share_reserved: correctly or use the raw phandle number if editing binary decompilation.)
3. Why This Happened (The "Double Die" Issue)

Your logs show a fascinating behavior of the Dual-Die EIC7700:

Die 0 SMMU (50c...) blocked 0xfdc00000.

Die 1 SMMU (70c...) also blocked 0xfdc00000.

This means the firmware uses a Global Shared Buffer at 0xfdc00000 for both NPUs to communicate or load microcode. Because it was missing from the Device Tree, Linux treated that address as "regular RAM" and likely overwrote it or blocked the NPU from touching it to prevent corruption.
Next Step

Edit the DTS: Add the fdc00000 reservation.

Compile: dtc -I dts -O dtb system-faulty.dts -o fixed.dtb

Boot: Update your bootloader to use fixed.dtb.