-2

I'm not trying to use the oobabooga webui, just a cmd line. I keep reading I should be able to use llama.cpp and so I cloned the github repo but I can't make heads or tails of the instructions. GPT4All is pretty straightforward and I got that working, Alpaca.cpp was super simple, I just use the .exe in the cmd-line and boom. gpt-x-alpaca-13b-native-4bit-128g-cuda.pt is suppose to be the latest model but I don't know how to run it with anything I have so far. I'm on a windows 10 i9 rtx 3060 and I can't download any large files right now as i'm stuck with throttled cell data until I can get to the library on Thursday. I have a cuda terminal if that helps.

TL;DR windows 10 i9 rtx 3060 gpt-x-alpaca-13b-native-4bit-128g-cuda.pt how to run in cmd no webui

I've tried running it with GPT4All but it says something about not being a JSON. I'm completely lost on this and could really use help.

Nick ODell
  • 15,465
  • 3
  • 32
  • 66
Rhellic
  • 11
  • 1

1 Answers1

1

I have tested it using llama.cpp

1- download the latest release of llama.cpp from github extract the zip

2- download the ggml-model-q4_1.bin and put it in the same folder

3- create a run.bat script with this content :

title llama.cpp
:start
main -i --interactive-first -r "### Human:" --temp 0 -c 2048 -n -1 --ignore-eos --repeat_penalty 1.2 --instruct -m ggml-model-q4_1.bin
pause
goto start

4- run the script! Done.

Mourad GHERSA
  • 155
  • 6