Running weaviate with chainlit and openai

Support
1

Description

Server Setup Information

  • Weaviate Server Version: 1.25.1
  • Deployment Method: docker
  • Multi Node? Number of Running Nodes: No
  • Client Language and Version: weaviate-client 4.6.3 A python native Weaviate client
  • Multitenancy?: NO

Any additional Information

Chainlit version: chainlit 1.1.202
Openai version: openai 1.30.5

Does anyone here have experience with running the Weaviate - Chainlit - OpenAI combination for RAG? If so please chime in on this thread, thanks

I have been banging my head on this today, only to find out that there is something in the internals of weaviate that, when run within chainlit (1.1.202) completely hangs the openai client.

I am running this code:

import asyncio
import os
import chainlit as cl
import weaviate
from openai import AsyncOpenAIimport logginglogging.basicConfig(level=logging.DEBUG)api_key = os.environ.get("OPENAI_API_KEY")settings = {
"model": "gpt-4",
"temperature": 0.7,
"max_tokens": 500,
"top_p": 1,
"frequency_penalty": 0,
"presence_penalty": 0,
}BOT_NAME = "ChatBot"def search_weaviate(query):
with weaviate.connect_to_local() as wva:
documents = wva.collections.get("Document")
results = documents.query.hybrid(query=query, limit=5)
result = results.objects
response_text = ""
for r in result:
doc = r.properties
response_text += (
f"Title: {doc['title']}\n"
f"URL: {doc['url']}\n"
f"Source Type: {doc['source_type']}\n"
f"Chunk: {doc['textChunk']}\n\n"
)
wva.close()
return response_text@cl.on_chat_start
async def start_chat():
cl.user_session.set(
"message_history",
[
{
"role": "system",
"content": "You are a helpful assistant that can answer any question about <this> a company.",
}
],
)
await cl.Avatar(
name="ChatBot",
url="/public/ai.jpg",
).send() msg = cl.Message(
author=BOT_NAME,
content="Hello! I am a helpful assistant that can answer any question about <this> a company.",
)
await msg.send()

@cl.on_message
async def on_message(message):
message_history = cl.user_session.get("message_history")
try:
response_text = search_weaviate(message.content)
except Exception as e:
response_text = f"Error: {str(e)}"
logging.error(response_text)
[logging.info](http://logging.info/)(f"response_text: {response_text}")
message_history.append({"role": "user", "content": message.content})
message_history.append({"role": "assistant", "content": response_text}) msg = cl.Message(author=BOT_NAME, content="")
await msg.send()
[logging.info](http://logging.info/)(f"Before OpenAI request with message_history: {message_history}")
async with AsyncOpenAI(api_key=api_key) as openai_client:
try:
response = await openai_client.chat.completions.create(
messages=message_history, stream=True, **settings
)
[logging.info](http://logging.info/)("OpenAI request successful")
except Exception as e:
logging.error(f"OpenAI request failed: {str(e)}")
full_response = ""
async for chunk in response:
content = chunk.choices[0].delta.content
if content is not None:
full_response += content
await msg.stream_token(content)
openai_client.close()
await msg.send()
message_history.append({"role": "assistant", "content": full_response})

And what happens is that the await openai_client.chat.completions.create never completes, because the openai client is never sending the HTTP request to the OpenAI API

if I remove the call to search_weaviate no problem. But if I run that beforehand… no cigar

If I do calculate the embeddings outside of Weaviate client, and pass the vector to the weaviate client then the problem goes away

But. If I run the same code in ipython, then the problem doesn’t exist

so it’s not just weaviate that interferes with openai, but it’s weaviate + openai within chainlit

If anyone ran this successfully please ping me and tell me how you managed If not and you have ideas how I can debug (I tried everything in my books) please lmk.Thanks

2

Hi @Stefano_Mosconi !!

Welcome to our community :hugs:

Never played with chainlit, it seems really cool.

I will do Chainlit Get Started docs, and will try using Weavaite with it today :slight_smile:

Stay tuned :slight_smile:

3

I think it’s pretty nice, if someone is building a chat app (and there’s many) and they’re not planning to build their own front end, chainlit and streamlit are two very strong options, with the former in my opinion having a nicer UI and a very active community behind.

Thanks for the effort, let me know how it plays out in the end (it’s really simple to get started with it anyway) :wink::pray:

4

Hi @Stefano_Mosconi !!

Did some initial work with this:

import chainlit as cl
import weaviate
import requests
import json
import os

from weaviate.classes.config import Configure


def initialize_or_search(query):
    client = weaviate.connect_to_local(
        headers={
            "X-OpenAI-Api-Key": os.environ.get("OPENAI_APIKEY")
        })
    if not client.collections.exists("Question"):
        collections = client.collections.create(
            name="Question",
            # If set to "none" you must always provide vectors yourself. Could be any other "text2vec-*" also.
            vectorizer_config=Configure.Vectorizer.text2vec_openai(),
            # Ensure the `generative-openai` module is used for generative queries
            generative_config=Configure.Generative.openai(),
        )
        resp = requests.get(
            "https://raw.githubusercontent.com/weaviate-tutorials/quickstart/main/data/jeopardy_tiny.json"
        )
        data = json.loads(resp.text)  # Load data

        question_objs = list()
        for i, d in enumerate(data):
            question_objs.append(
                {
                    "answer": d["Answer"],
                    "question": d["Question"],
                    "category": d["Category"],
                }
            )

        collections.data.insert_many(question_objs)
    else:
        collections = client.collections.get("Question")
    query = collections.generate.hybrid(
        query=query,
        grouped_task=f"What all those have in common?",
        limit=3
    )
    client.close()
    return query


@cl.on_message
async def main(message: cl.Message):
    # Your custom logic goes here...
    content = initialize_or_search(message.content)
    print(content)
    # Send a response back to the user
    await cl.Message(
        content=content.generated,
    ).send()

This was the result:

image
image1636Ă—672 30.1 KB

Your code example was a little bit malformated :frowning:

Can you extend this one so we can follow up on this?

Thanks!