JupyterAI Local LLM Integration

JupyterHub
, , , ,
1

Hi all,
I have a locally hosted Llama 3.1 70b model running on our server at the address: 10.1xx.1xx.50:8084/generate. Since there is no API key required for this setup, I use the following code to get responses from the model:

response = requests.request(method=“POST”, url=“10.1xx.1xx.50:8084/generate”, headers=headers, data=payload)

How can I integrate this endpoint into Jupyter-AI?
Any help would be really appreciated!

2

Please do not cross post across every single platform/medium possible (I saw this on 5 different channels), it only spreads thin the already sparse resources of volunteers trying to help out. Answered in jupyter notebook - JupyterAI Local LLM Integration - Stack Overflow

2 Likes