Description
I am integrating Glue Streaming with Weaviate using Spark Connector. I have uploaded Spark connector jar in S3, and added S3 path in Glue job. After adding, I have below code in my Glue Job.
try:
spark = SparkSession.builder.config(
"spark.jars",
"spark-connector-assembly-1.3.2.jar",
).appName("KafkaToS3") \
.getOrCreate()
except Exception as e:
logger.error(f"Failed to initialize SparkSession: {str(e)}")
traceback.print_exc()
sys.exit(1)
@hsm207 - Iām referring your post for the above.
Server Setup Information
- Weaviate Server Version: 1.26
- Deployment Method: Docker
- Multi Node? Number of Running Nodes: 1
- Client Language and Version: Python, 3
- Multitenancy?: Yes