Hi everyone,
I hope you’re doing well. I’m facing a challenge while integrating JSON data into Weaviate using batch operations. Everything was smooth sailing until I encountered a specific JSON structure that’s causing an error.
{
"SKU": "93d411b7-1350-5700-98bd-fd10113d9c802",
"SUPPLIER_SKU": "JKL0HZM/A",
"NAME": "JKL0HZM/A",
"TITLE": "Apple JKL0HZM/A mobile device charger Smartwatch White USB Wireless charging Fast charging Indoor",
"DESCRIPTION": "Apple Watch Magnetic Fast Charger to USB-C Cable (1 m)",
"BRAND": "Apple",
"PRICE_GBP": 21.27,
"COST_PRICE_GBP": 21.27,
"SUPPLIER_NAME": "Select",
"SUPPLIER_COUNTRY": "GB",
"CATEGORIES": "AB106-IT>A01B201-Hardware>A0101B314-Other",
"PRODUCT_TYPE": "Physical",
"PRODUCT_DESCRIPTION": "The Apple JKL0HZM/A mobile device charger is a convenient and efficient way to charge your Apple Watch. This charger features a magnetic design and comes with a USB-C cable that is 1 meter long. It supports wireless charging and fast charging, making it easy to power up your smartwatch quickly and effortlessly. The sleek white design adds a touch of style to your charging setup. Available at a price of 21.27 GBP from the supplier Select.",
"KEY_FEATURES": "Magnetic design, USB-C cable (1m) included, Wireless charging, Fast charging, White color",
"KEYWORDS": "Apple JKL0HZM/A, mobile device charger, Apple Watch charger, USB-C cable, wireless charging, fast charging, white, Select"
}
The error message I’m receiving is: “Out of range float values are not JSON compliant.”
I’ve been troubleshooting this for a while now but haven’t been able to pinpoint the exact issue. Any insights or suggestions on what might be causing this problem and how to resolve it would be greatly appreciated.
Thank you in advance for your help!
1 Like
The error message “Out of range float values are not JSON compliant” indicates that there’s a floating-point value (a number with decimals) in your JSON that’s outside the range that JSON can represent. Here’s how to troubleshoot and fix the issue:
Identifying the culprit:
Manual inspection: Look closely at the values in your JSON, particularly focusing on floating-point numbers. Are any of them extremely large or small? You can use a JSON validator tool online to format your JSON and make it easier to read.
Code-based approach: If you’re working with the JSON programmatically, use a loop to iterate through the values and check if any are infinite (positive or negative) or Not a Number (NaN). Here’s an example in Python using the math library:
Python
import math
def check_for_invalid_floats(data):
for key, value in data.items():
if isinstance(value, float) and (not math.isfinite(value)):
return key, value
return None
# Example usage
invalid_key, invalid_value = check_for_invalid_floats(your_json_data)
if invalid_key:
print(f"Found invalid float value: {invalid_value} for key: {invalid_key}")
Fixing the issue:
Once you’ve identified the problematic value, there are a few ways to fix it:
Replace with a valid number: If the value represents a real number but is just outside the JSON range, you can replace it with a valid number close to it. For example, you could replace positive infinity with a very large number and negative infinity with a very small number.
Replace with null: If the value is truly invalid or doesn’t make sense in the context of your data, you can replace it with null. JSON supports null values.
Modify data source: If the invalid values are coming from your data source, you might need to modify the data generation process or data pipeline to ensure it outputs valid floating-point numbers.
@warner Thanks for your prompt reply.
It’s odd, I followed your suggested solution and tested it with a dataframe comprising 250 rows, but no invalid entries were detected. However, upon attempting to push the data back into Weaviate, I encountered the same error once more.
It’s possible there are invalid values hidden within nested structures of your data.
- Deep Inspection: If your JSON has nested dictionaries or lists, make sure to iterate through them recursively to check for invalid floats at all levels.
The only json I have is
{
"SKU": "93d411b7-1350-5700-98bd-fd10113d9c802",
"SUPPLIER_SKU": "JKL0HZM/A",
"NAME": "JKL0HZM/A",
"TITLE": "Apple JKL0HZM/A mobile device charger Smartwatch White USB Wireless charging Fast charging Indoor",
"DESCRIPTION": "Apple Watch Magnetic Fast Charger to USB-C Cable (1 m)",
"BRAND": "Apple",
"PRICE_GBP": 21.27,
"COST_PRICE_GBP": 21.27,
"SUPPLIER_NAME": "Select",
"SUPPLIER_COUNTRY": "GB",
"CATEGORIES": "AB106-IT>A01B201-Hardware>A0101B314-Other",
"PRODUCT_TYPE": "Physical",
"PRODUCT_DESCRIPTION": "The Apple JKL0HZM/A mobile device charger is a convenient and efficient way to charge your Apple Watch. This charger features a magnetic design and comes with a USB-C cable that is 1 meter long. It supports wireless charging and fast charging, making it easy to power up your smartwatch quickly and effortlessly. The sleek white design adds a touch of style to your charging setup. Available at a price of 21.27 GBP from the supplier Select.",
"KEY_FEATURES": "Magnetic design, USB-C cable (1m) included, Wireless charging, Fast charging, White color",
"KEYWORDS": "Apple JKL0HZM/A, mobile device charger, Apple Watch charger, USB-C cable, wireless charging, fast charging, white, Select"
}
And I was successful in pushing other data ( More than 300 rows ) into weaviate .
I’m getting error when it is hitting this line
batch.add_data_object(properties, class_name)
Ensure that the property names in your JSON exactly match the property names defined in your Weaviate schema for the class_name you’re using. Weaviate is case-sensitive, so capitalization matters.
Include error handling in your code to catch any exceptions raised during batch.add_data_object . This will help you identify the specific error message and pinpoint the cause more easily.
Python
try:
batch.add_data_object(properties, class_name)
except Exception as e:
print(f"Error adding data object: {e}")
I have tried everything, including deleting and recreating the schema. However, each time I attempt to push data into Weaviate, I encounter errors with different JSON files. The errors are not consistent; they occur with a new JSON file each time. 
Solved: 
The issue was with a String column “Description” that had a missing value in one row. After deleting this row, everything works fine.
I still don’t understand why it was returning random JSON errors. I tracked all the JSON files that triggered errors, and none of them had any missing values.
