Spark connector with Weaviate

Support
1

Description

I am integrating Glue Streaming with Weaviate using Spark Connector. I have uploaded Spark connector jar in S3, and added S3 path in Glue job. After adding, I have below code in my Glue Job.

@DudaNogueira

try:
    spark = SparkSession.builder.config(
        "spark.jars",
        "spark-connector-assembly-1.3.2.jar", 
    ).appName("KafkaToS3") \
        .getOrCreate()
except Exception as e:
    logger.error(f"Failed to initialize SparkSession: {str(e)}")
    traceback.print_exc()
    sys.exit(1)

@hsm207 - Iā€™m referring your post for the above.

Server Setup Information

  • Weaviate Server Version: 1.26
  • Deployment Method: Docker
  • Multi Node? Number of Running Nodes: 1
  • Client Language and Version: Python, 3
  • Multitenancy?: Yes
2

hi @saurbhhsharrma !!

I have not used the Spark Connector yet :grimacing:

Its on my lists of things to learn for quite a while now, hehehe

Sorry, not sure if I got it, are you facing any issues here or just sharing the code?

Thanks!