version 2.4.27 query takes too long to run

[leaves-bin]

The previously deployed version was 2.4.13. Due to the inability to load data after restarting, we upgraded to version 2.4.17. After the upgrade, the query was very slowly. There were only a few vectors in database , but the query took 7-10 seconds. Don't understand why！！！

this is deployment file:

apiVersion: milvus.io/v1beta1
kind: Milvus
metadata:
name: milvus-cluster
labels:
app: milvus
spec:
mode: cluster
components:
#image: milvusdb/milvusv:2.4.13-hotfix
image: milvusdb/milvus:v2.4.17
mixCoord:
replicas: 1
resources:
limits:
cpu: 2
memory: 4Gi
indexNode:
replicas: 3
resources:
limits:
cpu: 2
memory: 4Gi
proxy:
replicas: 1
resources:
limits:
cpu: 2
memory: 4Gi
config:
queryNode:
enableDisk: true
dataCoord:
segment:
maxSize: 2048
diskSegmentMaxSize: 2048
gc:
interval: 7200
missingTolerance: 43200
log:
level: warn
file:
maxAge: 1
maxBackups: 1
dependencies:
etcd:
inCluster:
values:
autoCompactionMode: revision
autoCompactionRetention: "1000"
extraEnvVars:
- name: ETCD_QUOTA_BACKEND_BYTES
value: "4294967296"
- name: ETCD_HEARTBEAT_INTERVAL
value: "500"
- name: ETCD_ELECTION_TIMEOUT
value: "25000"
- name: ETCD_SNAPSHOT_COUNT
value: "10000"
- name: ETCD_ENABLE_PPROF
value: "true"
persistence:
accessMode: ReadWriteOnce
enabled: true
size: 30Gi #SSD Required
storageClass: nfs
replicaCount: 2
resources:
limits:
cpu: 2
memory: 4Gi
requests:
cpu: 1
memory: 4Gi

pulsar:
  inCluster:
    values:
      proxy:
        replicaCount: 2
        configData:
          PULSAR_MEM: >
            -Xms4096M
            -Xmx4096M
            -XX:MaxDirectMemorySize=4096M
          httpNumThreads: "100"
      zookeeper:
        volumes:
          persistence: true
          data:
            name: data
            size: 20Gi   #SSD Required
            storageClassName: nfs
        resources:
          requests:
            memory: 2Gi
            cpu: 1
        configData:
          PULSAR_MEM: >
            -Xms2048M
            -Xmx2048M
      bookkeeper:
        volumes:
          journal:
            name: journal
            size: 200Gi
            storageClassName: nfs
          ledgers:
            name: ledgers
            size: 500Gi   #SSD Required
            storageClassName: nfs
        resources:
          requests:
            memory: 4Gi
            cpu: 2
        configData:
          PULSAR_MEM: >
            -Xms4096M
            -Xmx4096M
            -XX:MaxDirectMemorySize=4096M
      broker:
        component: broker
        podMonitor:
          enabled: false
        replicaCount: 2
        resources:
          requests:
            memory: 4Gi
            cpu: 2
        configData:
          PULSAR_MEM: >
            -Xms4096M
            -Xmx4096M
            -XX:MaxDirectMemorySize=4096M

storage:
  inCluster:
    values:
      mode: distributed
      resources:
        limits: 
          cpu: 2
          memory: 4Gi
      persistence:
        storageClass: nfs 
        accessMode: ReadWriteOnce
        size: 500Gi

[yhmo]

Try search with consistency_level="Eventually".
If it is fast with "Eventually", the possible reason is the internal time-tick is delay.
Follow this doc to fetch the full log for further investigation:
https://github.com/milvus-io/milvus/tree/master/deployments/export-log

[xiaofan-luan]

If you are using standalone, usually it slow because rocksdb consume slow.
please offer detailed logs so we can investigate on it

[leaves-bin]

milvus-log.tar.gz

This is the exported log. I have not analyzed any obvious errors. Please ask the professionals to find out what the problem is. thks!!!

[xiaofan-luan]

[xiaofan-luan]

From your log, the pod seems to be very slow.
Init a threadpool takes almost 10 minutes.
You probably need to check the host machine cpu usage.
The reason why milvus can not serve is due to the init is not done.

[leaves-bin]

I see ,and where is the initialization threadpool configured? Maybe I can try making the threadpool smaller.

[xiaofan-luan]

since you only have 1-2 core I think adding threadpool won't help too much.
And it also seems under your case the pod is slow, maybe on your K8s there are some offline task consumes too much cpu that affect the performance

[xiaofan-luan]

probably you need to check with your devops team see

1. if the pod throttled by K8s
2. is there any heavy work load happens on your host machine

[leaves-bin]

thks, Maybe this needs to be solved by the devops team