Very frustrated, mostly at myself because I am BRAND NEW to Weaviate and just don’t know what I’m doing.-
Weaviate Server Version: 1.31.7
- Deployment Method: No Docker–full deployment on Ubuntu
-Single Server running Weaviate - Client Language and Version: Python client v4.16.4
The issue is so simple and that is why I am so frustrated. I have a postgres table and one of the fields in it is pgvector, 4096 dimensions. This table has only 40 records in it. I am creating a collection in weaviate and simply trying to create 40 records with metadata and the embedding associated with it. Below is the code I’m using. The code runs, tells me 40 records were successfully added. When I check the records–everything is there EXCEPT the 4096 dimension embedding. The vectors are empty. I’ve looed at it 100 different ways and tried so many different things. I’ve talked to others about it. I just don’t get it. I don’t see what I’m doing wrong.
import os
import requests
import json
import ast
from dotenv import load_dotenv
import psycopg2
Load .env file
load_dotenv(‘/home/mintdude/Github/sparky/TestScripts/.env’)
Read all config from .env
POSTGRES_URL = os.getenv(“POSTGRES_URL”)
WEAVIATE_URL = os.getenv(“WEAVIATE_URL”)
WEAVIATE_PRIORITY_CLASS = os.getenv(“WEAVIATE_PRIORITY_CLASS”, “PriorityPrototype”)
Connect to Postgres
conn = psycopg2.connect(POSTGRES_URL)
cur = conn.cursor()
Fetch all records (assuming ‘embedding’ is a pgvector field)
cur.execute(“SELECT id, text, embedding FROM priority_prototypes”)
rows = cur.fetchall()
count = 0
for row in rows:
proto_id, text, embedding = row
# Convert pgvector to Python list if needed
if isinstance(embedding, memoryview):
embedding = list(embedding.tobytes()) # handle memoryview case (may need adjustment)
elif hasattr(embedding, ‘tolist’):
embedding = embedding.tolist()
elif isinstance(embedding, str):
try:
embedding = ast.literal_eval(embedding)
except Exception as e:
print(f"Failed to parse embedding for proto_id {proto_id}: {e}“)
continue
if not isinstance(embedding, list):
print(f"Skipping proto_id {proto_id}: embedding not a list!”)
continue
# Compose Weaviate REST API payload
payload = {
"class": WEAVIATE_PRIORITY_CLASS,
"properties": {
"proto_id": str(proto_id),
"text": text
},
"vector": embedding
}
resp = requests.post(WEAVIATE_URL, json=payload)
if resp.status_code not in (200, 201):
print(f"Error uploading proto_id={proto_id}: {resp.status_code}, {resp.text}")
else:
count += 1
print(f"Uploaded proto_id={proto_id} OK")
cur.close()
conn.close()
print(f"\nUpload complete. {count} objects sent to Weaviate.")
