This question is not specific to Weaviate, however I am using Weaviate in an enterprise environment and the answer might inform future deployment decisions.
I’ve noticed that DSPy has recently added support for images, like this:
I’ve semi-successfully run the MIPROv2 optimizer on a simple program with a small training set of images. I can get it to work with gemini because of the long context window but it fails with openai, etc. - I suspect it’s stuffing many images base64 encoded into the text prompt during the prompt-writing stage.
My question is - are you aware of a way to send the image as an image instead of stuffing it in the text prompt, thereby overcoming context length limitations during the DSPy optimization process?
Ultimately this will become part of a multimodal RAG pipeline using Weaviate.
Thanks very much.