Async BATCH request for real-time ANN search

General
,
1

Hi everyone!

I am running weaviate instance locally and referring to this stackoverflow post.

So, let’s say that all the logic with respect to weaviate API incapsulated into IWeaviateDocStore class. I’m receiving requests in async manner from the client and I’d like to perform ANN search on them using batches instead of manual one by one iteration. Is that possible now with weavaite? The pseudo-code is below:


import json
import aiohttp
import asyncio
from sir.storing.weaviate import IWeaviateDocStore

async def search(query_string: str, url:str):   

    query = '''
        {
            Get{
                Article(
                    limit: 20
                    hybrid: {
                        query: "<QUERY_PLACEHOLDER>"
                        alpha: 0.75
                        fusionType: relativeScoreFusion
                    }){
                        texte
                        title
                    }
                    _additional {
                        score
                    }
                }
            }
        }
    '''.replace("<QUERY_PLACEHOLDER>", query_string)

    async with aiohttp.ClientSession() as session:
        async with session.post(url, headers=headers, data=json.dumps({'query': query})) as response:
            return await response.json()


def main():
   BATCH_SIZE = 10
   INCOMING_REQS: List[str] = ["What are the rules of the hunger games by S.Collins?", "What is lumos maxima?", ... ,]
   # HELP. Do I need to manually perform `batching` or I can simply send it to weavaite and it'll handle it automatically?

   session = aiohttp.ClientSession()
   URL = "http://localhost:8080"
   
   batch_reqs = await asyncio.gather(*[search(session=session, url=URL, query_string=query) for query in INCOMING_REQS]
   #
   await session.close()

      
   

if __name__ == "__main__":
   asyncio.run(main())

ps. Thx in advance and really sorry if this functionality is already in the docs, haven’t found it.

2

Hi!

if you consume the endpoints directly, I believe this would work :thinking:

There are some work in progress for allowing this from the client, specially with the new GRPC connection that improves performance.