Consider the following psuedo schemas for a Document and its corresponding chunks.
Document - title - link DocChunk - name - document_id
The idea being that a document is chunked and indexed, and we also store a vector for the document itself. The DocChunks therefore have a reference back to
document_id, basically a foreign key.
My question: is it possible to do this transactionally? The steps are:
- Insert a vector do Document table; store the ID.
- Chunk the document.
- Insert chunks into DocChunk, with the ID back to Document.
What would happen tho if the chunk insert failed? Is relational DB, could just wrap this all in a transactional block. Is anything like this possible in Weaviate? If not, do you have any examples of patterns around this?
PS - I see this is similar to Verba but didn’t see how Verba addressed this: https://github.com/weaviate/Verba/blob/main/goldenverba/ingestion/init_schema.py