OpenAI proxy does not get called

Support
1

Description

my OpenAI BaseURL is not working - I do have moduleConfig.text2vec-openai.baseURL set to the url of my OpenAI proxy when I query the schema and I also have the header set up in the client:

        params = ConnectionParams.from_url(f"http://{WEAVIATE_IP}:8080", grpc_port=50051)
        self.client = weaviate.WeaviateClient(
            params,
            additional_headers={
                "X-OpenAI-Api-Key": OPENAI_API_KEY,
                "X-OpenAI-BaseURL": OPENAI_BASE_URL,
            },
        )
        self.client.connect()

However, when I try to insert objects, it tries to embed them through the actual OpenAI link, not my proxy:

{"action":"requests_total","api":"rest","class_name":"Product","error":"update vector: connection to: OpenAI API failed with status: 403","level":"error","msg":"unexpected error","query_type":"objects","time":"2024-03-18T19:08:08Z"}

How could I debug this?

Server Setup Information

  • Weaviate Server Version: 1.24.3
  • Deployment Method: docker on a cloud vm
  • Multi Node? Number of Running Nodes:
  • Client Language and Version: python client 4.5.2

Any additional Information

2

Hi @dorekhov1 !

Welcome to our community :hugs:

I could not reproduce this.

This is what I did. created two instances of the https://webhook.site/ service (you can use an incognito browser to create the second one)

import weaviate
from weaviate import classes as wvc

endpoint = "https://webhook.site/abb03671-6ada-4d74-8045-22cfc8ef0829"

client = weaviate.connect_to_local()
client.collections.delete("Collection")
collection = client.collections.create(
    name="Collection",
    vectorizer_config=wvc.config.Configure.Vectorizer.text2vec_openai(
        base_url=endpoint
    )
)
collection.data.insert({"text": "this is a test"})

and finally changing it by query time.
Note here: remember to get the collection again, as any collection you got previously had the client without the X-OpenAI-BaseURL in its headers :wink:

client = weaviate.connect_to_local(
    headers={
        "X-OpenAI-BaseURL": "https://webhook.site/af604250-63f0-4a68-aa0f-6bb03f3f3aa1"
    }
)
collection = client.collections.get("Collection")
collection.data.insert({"text": "this is a test"})

In each example, I was able to receive the payload in it’s respective instance in webhook.site

Let me know if this helps!

3

Hi @DudaNogueira ! Thank you for the help

Unfortunately, I still run into the same issue with this code:

import weaviate
import weaviate.classes as wvc

from .config import OPENAI_API_KEY, OPENAI_BASE_URL, WCS_CLUSTER_URL

params = ConnectionParams.from_url(WCS_CLUSTER_URL, grpc_port=50051)
client = weaviate.WeaviateClient(
    params,
    additional_headers={
        "X-OpenAI-Api-Key": OPENAI_API_KEY,
        "X-OpenAI-BaseURL": OPENAI_BASE_URL,
    },
)
client.connect()

client.collections.delete("Collection")
collection = client.collections.create(
    name="Collection",
    vectorizer_config=wvc.config.Configure.Vectorizer.text2vec_openai(base_url=OPENAI_BASE_URL),
)
collection.data.insert({"text": "this is a test"})

The only difference here seems to be the way the client is initialized. Could that somehow affect this?

4

I don’t think.

Can you upgrade the client to latest version (4.5.4)?

This worked for me, for instance:

OPENAI_BASE_URL = "https://webhook.site/2d013b43-c0f1-4a60-aff1-a33af675ed20"
import os

import weaviate
import weaviate.classes as wvc
from weaviate.connect import ConnectionParams

#from .config import OPENAI_API_KEY, OPENAI_BASE_URL, WCS_CLUSTER_URL

params = ConnectionParams.from_url("http://localhost:8080", grpc_port=50051)
client = weaviate.WeaviateClient(
    params,
    additional_headers={
        "X-OpenAI-Api-Key": os.environ.get("OPENAI_APIKEY"),
        "X-OpenAI-BaseURL": OPENAI_BASE_URL,
    },
)
client.connect()

client.collections.delete("Collection")
collection = client.collections.create(
    name="Collection",
    vectorizer_config=wvc.config.Configure.Vectorizer.text2vec_openai(base_url=OPENAI_BASE_URL),
)
collection.data.insert({"text": "this is a test"})
5

Actually, trying with the webhook site as well, I was able to see the payload…

So for some reason it seems like the Weaviate server does not want to actually make use of my proxy. The base url I am providing is a serverless container which just takes a request it received and resends it to the OpenAI API using a SOCKS5 proxy. So, for example, this works fine for me:

curl https://myproxyurl.net/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant."
      },
      {
        "role": "user",
        "content": "Hello!"
      }
    ]
  }'

When running this, I see lots of informative logs on the cloud. But when the Weaviate server is expected to make calls to OpenAI, there are no logs for the proxy container and the only logs I see are the Weaviate server’s OpenAI API failed with status: 403.

6

Actually this is likely what Weaviate is sending to your proxy:

curl https://myproxyurl.net/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer SOME-KEY-HERE" \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant."
      },
      {
        "role": "user",
        "content": "Hello!"
      }
    ]
  }'

**Note the ** -H "Authorization: Bearer SOME-KEY-HERE" \

And when this happens, Your proxy is returning 403.

How are you running this proxy? And what is it doing? Maybe there is an option to accept this header and ignore it?

Thanks!

7

Thanks for clarifying the ability to pass base_url into the vectorizer config, which is not in the documentation of Weaviate client and burried.

Using liteLLM, with relevant authorisations and own proxy bearer token, it strangely does not work.

Calling to create embeddings using openai works:
python3 /mnt/d/Data/litellm/embeddings.py
CreateEmbeddingResponse(data=[Embedding(embedding=[0.019413867965340614, -0.03536645695567131, 0…

The endpoint is running on localhost:4000 but Weaviate returns:

raise UnexpectedStatusCodeError(error_msg, response=res) weaviate.exceptions.UnexpectedStatusCodeError: Object was not added! Unexpected status code: 500, with response body: {‘error’: [{‘message’: ‘update vector: send POST request: Post “http://localhost:4000/v1/embeddings”: dial tcp 127.0.0.1:4000: connect: connection refused’}]}.

odd is that there is not log entry in litellm, but the endpoint is definitely reachable. any thoughts?

8

Resolved. Weaviate container needed to talk to litellm outside its network, and not the localhost. Adjusted endpoint to correct IP:

Last30dTopEndUsersSpend Exists!
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:4000 (Press CTRL+C to quit)
INFO: 172.23.32.1:57270 - “GET / HTTP/1.1” 200 OK
INFO: 172.23.32.1:57270 - “GET /openapi.json HTTP/1.1” 200 OK
INFO: 172.23.32.1:57307 - “POST /v1/embeddings HTTP/1.1” 200 OK
INFO: 172.23.32.1:57307 - “POST /v1/embeddings HTTP/1.1” 200 OK
INFO: 172.23.32.1:57309 - “POST /v1/embeddings HTTP/1.1” 200 OK

Works perfectly. Tq

1 Like