After some experimentation, I found what could be wrong. I batch insert a json which has a key (‘embedding’) that is list of float. The issue is that I am creating the collection with properties that include this key! (should have pop and assigned that to the vector). I am surprised that despite defining it explicitly (as Property) in my collection, my data violated the “schema” but weaviate is totally silent about it. I can imagine 768 numbers are being inverted indexed like mad and could explain my very slow batch inserts and extreme memory consumption.
I tried a barebone # of properties and it is behaving a lot better. Will add back to whole thing and hope all are still well.