Stream generated query results in python


I was wondering whether it is possible to ‘stream’ the generated answer as one can do with OpenAI API.

My goal is to use the weaviate generate feature in a FastAPI endpoint as such:

async def get_topic_summary(topic: str):
    return StreamingResponse(sample_summary(), media_type='text/event-stream')

In OpenAI, one can do this like (from their docs):

stream =
    messages=[{"role": "user", "content": "Say this is a test"}],
for chunk in stream:
    if chunk.choices[0].delta.content is not None:
        print(chunk.choices[0].delta.content, end="")

I could not found documentation about this somewhere. Thanks in advance!

Hi @bram!

This is not possible :frowning:

For example, in Verba, our team used open ai directly to be able to stream the generation to the frontend:

Let me know if this helps.


Hi @DudaNogueira ,

Thanks, I will have I look!