Model is not supported: llama3_2_3b_instruct_q40

[Znbne]

Hi,
I want to work with the newly added model llama3_2_3b_instruct_q40, but it shows me error when downloading the model in docker container. I checked the launch.py and the issue is caused by this line:
modelName = sys.argv[1].replace('-', '_')
if modelName not in MODELS:
print(f'Model is not supported: {modelName}')
exit(1)

I appreciate your guidance on this problem.
Thanks

[b4rtaz]

Could you provide what error you have? And what command in the CLI? The correct command: python launch.py llama3_2_3b_instruct_q40

[Znbne]

Sure, I want to run it in a docker container, here is the commands inside the docker file:
RUN apt-get install -y git
RUN git clone https://github.com/b4rtaz/distributed-llama.git
WORKDIR /distributed-llama
RUN make dllama
RUN make dllama-api
RUN apt-get install python3 python3-pip -y
RUN pip3 install requests --break-system-packages
RUN echo "y" | python3 launch.py llama3-2-3b-instruct-q40

, and this is the output of build command:

16 [13/13] RUN echo "y" | python3 launch.py llama3-2-3b-instruct-q40
16 0.359 Model is not supported: llama3_2_3b_instruct_q40
16 ERROR: process "/bin/sh -c echo "y" | python3 launch.py llama3-2-3b-instruct-q40" did not complete successfully: exit code: 1

[13/13] RUN echo "y" | python3 launch.py llama3-2-3b-instruct-q40:
0.359 Model is not supported: llama3_2_3b_instruct_q40

Dockerfile:17

15 | RUN apt-get install python3 python3-pip -y
16 | RUN pip3 install requests --break-system-packages
17 | >>> RUN echo "y" | python3 launch.py llama3-2-3b-instruct-q40

---

ERROR: failed to solve: process "/bin/sh -c echo "y" | python3 launch.py llama3-2-3b-instruct-q40" did not complete successfully: exit code: 1

[b4rtaz]

I don't see any error, also echo "y" | python3 launch.py llama3_2_3b_instruct_q40 seems to work on my machine.

But I added a new option --run to the launch.py script. Now you don't need the echo "y" part. The CLI won't ask a question. Maybe that will help. For example:

python launch.py llama3_2_3b_instruct_q40 --run

BTW: the model name uses underscores, not dashesh.

[Znbne]

Thank you for your response and the modification.
Unfortunately, the issue persists, and I’m not sure why. I’m running the code on a Windows environment, and when I build the Docker image (docker build -t aaa .), the error still occurs. However, when I dockerize the model for the ARM platform (docker buildx build --platform linux/arm64 -t a/aaa:arm64 --push .), it works perfectly without any issues. I write the model name exactly same as what exists in launch.py, i mean with underscores.
The issue is related to the cached layers of docker image. Thanks again for your prompt response.

[b4rtaz]

Hmm... maybe by some reason you are using the old version of the repository. Could you try the below commands?

RUN git clone https://github.com/b4rtaz/distributed-llama.git
RUN git checkout d10699fec2e4b9303065807555e3898e69f6cf18

[Znbne]

Thank you for your guidance.
I built the image with --no-cache option and it works :).