Valid ISO8601 and RFC3339 date string discarded as invalid

Support
1

Description

I have declared a property (via the python V4 client) as of type date.
When I try to insert a date string such as ‘2024-05-14’ I get an error as follows:
Weaviate error: Object was not added! Unexpected status code: 422, with response body: {'error': [{'message': "invalid object: invalid date property 'isoEditionDate' on class 'Articles_intfloat_multilingual_e5_large': requires a string with a RFC3339 formatted date, but the given value is '2024-05-14'"}]}

In my opinion this is wrong since a YYYY-MM-DD is valid for both ISO8601 and RFC3339. You can see it visually at the following website

Server Setup Information

-Weaviate server 1.24.11 deployed with docker compose (tried 1.25.1 is the same)

  • Python client 4.5.6
2

Hi @rjalex !

Great question, as always :slight_smile:

I believe the line here is that, as stated in doc, Weaviate will receive RFC3339 timestamps.

so not all formats. I believe the error message is misleading here in not enforcing “RFC3339 formatted timestamps”

WDYT?

1 Like
3

Hey Duda happy to hear from you.

Well there are several use cases that need to be described as happening on a given day irrespective of a time.

For example think of my case in which I am recording the date of a newspaper issue. It has no meaning to say the newspaper issue is that of 2024-05-30T12:45:00 or even a conventional 2024-05-30T00:00:00.

That TIME would be an artificial and possibly misleading information.

ISO8601 and its profile RFC3339 do qualify “2024-05-30” as compliant.

So I guess if it’s internally necessary for the indexing/filtering engine to store them always as YYYY-MM-DDTHH:MM:SS the documentation could say so and not refer to the standard.

PS In my case I resorted to defining that property as TEXT.

4

The python client accepts datetime objects for time-metadata and date-properties and then sends it in the correct format to weaviate (Note, please make sure to set a timezone)

1 Like
5

Ok thanks very clear. maybe this should be conveyed more in the doc and not just refer to RFC3339 which does allow YYYY-MM-DD ?

As a suggestion do I understand correctly that a simple date with no time attribute (just as an example the date of the annoucement of a product) should therefore modeled with a TEXT (in the ISO8601 sortable format) property?

Take care.

6

I’ll ping our docs team :slight_smile:

As a suggestion do I understand correctly that a simple date with no time attribute (just as an example the date of the annoucement of a product) should therefore modeled with a TEXT (in the ISO8601 sortable format) property?

You can do that, but than larger/smaller (before/after) filtering does not work. Equal would still work.

I’d recommend to use a datetime object doing something like this

from datetime import date
date_string = "2015-01-30"
date_object = date.fromisoformat(date_string)

you can pass date_object as the date property and the python client will create the correct string automatically

2 Likes
7

Thank a lot. That is a good solution.

Just to be sure I understand, the <,>,>=,<= would not work on a sortable TEXT property like a YYYY-MM-DD string?

1 Like
8

Hi @rjalex ! I hope you are well.

Sorry about the inconvenience. We have some examples like this one on Metadata filters, and this may be useful for working with datetime at import.

We’ll work on adding more examples for datetimes though.

2 Likes
9

Hi there @jphwang nice to hear from you!

Now all works well.

As so many other things about Weaviate the way you define the server may influence the way you declare a collection and this will definitely impact on the way you insert and query your object’s properties.

In my case now isoEditionDate is declared as DATE,

        wvcc.Property(
            name="isoEditionDate",
            data_type=wvcc.DataType.DATE,  # DATE for RFC3339 ISO8601 date
            skip_vectorization=True,
        ),

when I insert the data I make sure that whatever is passed is converted to an ISO8601/RFC3339 fulll format datetime value including the timezone info:

document["isoEditionDate"] = rome_tz.localize(
            datetime.strptime(mdb_document["isoEditionDate"], "%Y-%m-%d")
        )

and lastly the filters take this into account (and by virtue of complex but poweful pydantic declarations I check that the input dates are valid and that fromDate is always <= toDate.

Thanks to all of you guys, you rock!

1 Like