Is there a way to set the number of threads in the session/executor?

[pfeatherstone]

It looks like run_main_graph() runs on 1 thread.

[AlexandreEichenberger]

Absolutely. onnx-mlir -O3 -march=xxx -parallel will generate OpenMP parallel code for many operations. I recommend the -march option as it enables SIMD code as well. We support SIMD for arm64, z14 and later, and x86-64 for SSE2 (AVX2 can be further specified with a -march=x86-64 -mcpu=avx2). Nothing happens unless -O3 is present. Note that you have to compile LLVM with OpenMP enabled before building ONNX-MLIR to enable a multi-threading onnx-mlir. See our pages on how to build onnx-mlir.

You may then run your graph with OMP_NUM_THREADS or OMP_THREAD_LIMIT set as environment variable to control the number of OpenMP threads.

[pfeatherstone]

Cool thank you. Do you plan on introducing a custom thread pool? I've seen a lot of benchmarks where modern lock-free task-stealing work queues outperform openmp quite dramatically.

[pfeatherstone]

Strange i compiled with -O3 -march=x86-64 -mcpu=avx2 and i get error:

'avx2' is not a recognized processor for this target (ignoring processor)

The output of lscpu on my machine is:

Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          39 bits physical, 48 bits virtual
  Byte Order:             Little Endian
CPU(s):                   12
  On-line CPU(s) list:    0-11
Vendor ID:                GenuineIntel
  Model name:             Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
    CPU family:           6
    Model:                158
    Thread(s) per core:   2
    Core(s) per socket:   6
    Socket(s):            1
    Stepping:             10
    CPU max MHz:          4500.0000
    CPU min MHz:          800.0000
    BogoMIPS:             5199.98
    Flags:                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdts
                          cp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est t
                          m2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch 
                          cpuid_fault epb pti ssbd ibrs ibpb stibp tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdsee
                          d adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp vnmi md_clear flush
                          _l1d arch_capabilities
Virtualisation features:  
  Virtualisation:         VT-x
Caches (sum of all):      
  L1d:                    192 KiB (6 instances)
  L1i:                    192 KiB (6 instances)
  L2:                     1.5 MiB (6 instances)
  L3:                     12 MiB (1 instance)
NUMA:                     
  NUMA node(s):           1
  NUMA node0 CPU(s):      0-11
Vulnerabilities:          
  Gather data sampling:   Mitigation; Microcode
  Itlb multihit:          KVM: Mitigation: VMX disabled
  L1tf:                   Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
  Mds:                    Mitigation; Clear CPU buffers; SMT vulnerable
  Meltdown:               Mitigation; PTI
  Mmio stale data:        Mitigation; Clear CPU buffers; SMT vulnerable
  Reg file data sampling: Not affected
  Retbleed:               Mitigation; IBRS
  Spec rstack overflow:   Not affected
  Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:             Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
  Srbds:                  Mitigation; Microcode
  Tsx async abort:        Not affected

So this should work right?
I'm using the onnx-mlir.py script

[pfeatherstone]

Also, when I compile with -parallel i get the error:

cannot find -lompruntime

Shouldn't it link with -fopenmp ?

[AlexandreEichenberger]

You need to build a custom LLVM with OpenMP enabled. And then rebuild onnx-mlir from scratch. We are phasing in OpenMP support for our Linux build mainly.

See https://github.com/onnx/onnx-mlir/blob/main/docs/BuildOnLinuxOSX.md for details. What is needed is these lines

   -DLLVM_ENABLE_PROJECTS="mlir;clang;openmp" \

   -DENABLE_LIBOMPTARGET=OFF \

If you are using Windows, probably the same change would work (I have no access to windows machine).

For the avx2, try skylake. I think the issue is that the backend does not like that option. My understanding is that Skylake has AVX2. If you have another CPU target that also use AVX2 and you would like me to add it too, happy to do so.

[AlexandreEichenberger]

Is it supported by MLIR / LLVM? If so, we might be interested in looking into such a scheme too.

[pfeatherstone]

I don't believe so no. I guess you would have to embed a work queue when using --emitLib for example. I guess it wouldn't apply to the other outputs.