Getting connection refused error using openllm library of python

Question

I am trying to utilise this github repo, particularly the below python code:

import openllm client = openllm.client.HTTPClient('http://localhost:3000') client.query('Explain to me the difference between "further" and "farther"')

But this is throwing following error:connection refuse error

How do I run the server for above code to work ?

score 0 · Answer 1 · answered Jul 29 '23 at 15:03

You should refer to the "Starting an LLM Server " section of the github project you linked to.

For instance, to start a server with the OPT model, you would do as follows:

openllm start opt

Once you do that, you can open http://localhost:3000 on a web browser to check if it's working and to experiment with the endpoints and sample input prompts.

Getting connection refused error using openllm library of python

1 Answers1