Local Python app connecting to Weaviate client 4.0 within Kubernetes

Support
1

Good morning, I hope you can maybe assist me witht he following issue.

I have a kubernetes cluster with weaviate running on it as setup as per the documentation on the site using minikube.

I can see that it’s running when I minikube tunnel as I’m able to view it in the browser on 127.0.0.1:80

I’m also able to curl that location that returns a successful status of 200.

My issue is when I try and connect from our python application on my local machine to the local kubernetes cluster it’s not able to connect to weaviate even though above mentioned it worked.

My client is as follows (Note i’ve tried on both port 8080 and 80)

client = weaviate.connect_to_custom(
            http_host="localhost",
            http_port="8080", or 80
            http_secure=False,
            grpc_host="localhost",
            grpc_port="50051",
            grpc_secure=False,
        )

So when I try and run it locally the following error occurs

{"errorMessage": "An error occurred while processing the file: Failed to initialize Weaviate client: Meta endpoint! Unexpected status code: 404, with response body: None.", "errorType": "RuntimeError", "requestId": "82da9ab1-06c3-4e19-beb9-62efb0e72205"

Any insight into this matter would be greatly appreciated

2

hi @Jurie_Oberholzer !!

Welcome to our community! :hugs:

Are you using our official helm chart for that?

If that’s the case, can you share your values.yml?

Also, can you paste the entire stack trace?

Thanks!

3

Thank you very much.
Yes I’m using the official helm chart for that using the values.yml that was generated.

The following is my yaml file

And my stacktrace is

START RequestId: d5fe8460-82c9-482e-a2cf-fc59a8838c60 Version: $LATEST
2024-10-02 07:06:11.079 | INFO     | shared.service.process_file:process_files:14 - Starting process_files for PDF: Expendo Android QRG_SBSA_All Modes_20220318 2.pdf in bucket: mervin-pdf-bucket.
2024-10-02 07:06:11.079 | INFO     | shared.helper.data.extract_text:extract_text_from_pdfs:9 - Attempting to parse file from S3.
/var/lang/lib/python3.12/site-packages/botocore/auth.py:419: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
  datetime_now = datetime.datetime.utcnow()
2024-10-02 07:06:12.893 | INFO     | shared.helper.data.extract_text:extract_text_from_pdfs:15 - File loaded successfully.
/var/lang/lib/python3.12/site-packages/tika/tika.py:667: ResourceWarning: unclosed file <_io.TextIOWrapper name='/dev/null' mode='w' encoding='utf-8'>
  _ = Popen(java_path, stdout=open(os.devnull, "w"), stderr=open(os.devnull, "w"))
ResourceWarning: Enable tracemalloc to get the object allocation traceback
2024-10-02 07:06:12,998 [MainThread  ] [WARNI]  Failed to see startup log message; retrying...
[WARNING]       2024-10-02T07:06:12.998Z        6af15f31-d000-459b-b556-5f94d1f6530b    Failed to see startup log message; retrying...
/var/lang/lib/python3.12/site-packages/tika/tika.py:599: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/tika-server.log' mode='w' encoding='utf-8'>
  status = startServer(jarPath, TikaJava, TikaJavaArgs, serverHost, port, classpath, config_path)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/var/lang/lib/python3.12/subprocess.py:1127: ResourceWarning: subprocess 18 is still running
  _warn("subprocess %s is still running" % self.pid,
ResourceWarning: Enable tracemalloc to get the object allocation traceback
2024-10-02 07:06:18.746 | INFO     | shared.helper.data.extract_text:extract_text_from_pdfs:19 - File parsed successfully.
2024-10-02 07:06:18.775 | ERROR    | shared.helper.database.get_client:get_client:21 - Failed to initialize Weaviate client: Meta endpoint! Unexpected status code: 404, with response body: None.
2024-10-02 07:06:18.775 | ERROR    | shared.service.process_file:process_files:41 - An error occurred during processing of Expendo Android QRG_SBSA_All Modes_20220318 2.pdf: Failed to initialize Weaviate client: Meta endpoint! Unexpected status code: 404, with response body: None.
2024-10-02 07:06:18.776 | ERROR    | create_db_handler:handler:41 - Error: An error occurred while processing the file: Failed to initialize Weaviate client: Meta endpoint! Unexpected status code: 404, with response body: None.
LAMBDA_WARNING: Unhandled exception. The most likely cause is an issue in the function code. However, in rare cases, a Lambda runtime update can cause unexpected function behavior. For functions using managed runtimes, runtime updates can be triggered by a function change, or can be applied automatically. To determine if the runtime has been updated, check the runtime version in the INIT_START log entry. If this error correlates with a change in the runtime version, you may be able to mitigate this error by temporarily rolling back to the previous runtime version. For more information, see https://docs.aws.amazon.com/lambda/latest/dg/runtimes-update.html
[ERROR] RuntimeError: An error occurred while processing the file: Failed to initialize Weaviate client: Meta endpoint! Unexpected status code: 404, with response body: None.
Traceback (most recent call last):
  File "/var/task/create_db_handler.py", line 42, in handler
    raise e
  File "/var/task/create_db_handler.py", line 30, in handler
    process_files(s3_pdf_bucket, s3_pdf_key)  # Updated call)
  File "/var/task/shared/service/process_file.py", line 42, in process_files
    raise RuntimeError(f"An error occurred while processing the file: {e}")
END RequestId: 6af15f31-d000-459b-b556-5f94d1f6530b
REPORT RequestId: 6af15f31-d000-459b-b556-5f94d1f6530b  Init Duration: 0.29 ms  Duration: 8487.79 ms    Billed Duration: 8488 ms        Memory Size: 2048 MB  Max Memory Used: 2048 MB
{"errorMessage": "An error occurred while processing the file: Failed to initialize Weaviate client: Meta endpoint! Unexpected status code: 404, with response body: None.", "errorType": "RuntimeError", "requestId": "6af15f31-d000-459b-b556-5f94d1f6530b", "stackTrace": ["  File \"/var/task/create_db_handler.py\", line 42, in handler\n    raise e\n", "  File \"/var/task/create_db_handler.py\", line 30, in handler\n    process_files(s3_pdf_bucket, s3_pdf_key)  # Updated call)\n", "  File \"/var/task/shared/service/process_file.py\", line 42, in process_files\n    raise RuntimeError(f\"An error occurred while processing the file: {e}\")\n"]}
[ERROR] [1727852778818] LAMBDA_RUNTIME Failed to get next invocation. No Response from endpoint

Thanks for taking the time to respond

4

Hi @DudaNogueira ,
I would just like to follow up with the above issue.

Kind regards,

Jurie

5

@DudaNogueira any updates?

6

Hi!

This issue comes with not exposing your Weaviate server correctly.

Here is culprit:

create_db_handler:handler:41 - Error: An error occurred while processing the file: Failed to initialize Weaviate client: Meta endpoint! Unexpected status code: 404, with response body: None.

You need to make sure that you expose the REST port AND GRPC port correctly.

Those same ports/hosts need to be passed to Weaviate Client on it’s initialization.

Can you try forwarding the ports to your local machine and connecting using those ports?

7

I’ve done the port forwarding and minikube tunnel.

I’m able to view it when I go into my browser (http://127.0.0.1/v1) as well as I’m able to curl it and I receice a status of 200.

Might the issue be trying to run it locally using sam build with the lambda.

We’ve used local lambda testing Chroma / Pinecone and it seemed to work with it.

8

Once you are able t connect to the rest and GRPC - you can use GRPCurl to test - the issue should be a network one.