Custom filter with weaviate typescrit v3

Support
1

I am using weawiate typescript version3, I have below code :

const fileNameArrayFromPostgres = [ 'The-Unhoneymooners-PDF-Book-pages-3.pdf', 'The-Unhoneymooners-PDF-Book-pages-4.pdf' ] 
const startUrlArrayFromPostgres = [ '[https://www.example.com](https://www.example.com/)', '[https://www.cibc.ca](https://www.cibc.ca/)' ]

deleteResult = await myCollection.data.deleteMany(combinedFilter, { dryRun: true, });

I want to you help me write a logic of "combinedFilter "to delete items meet below requirements:

there are property named “file_name” and “start_url” inside weaviate collection.

i want to delete items that’s “file_name” not inside fileNameArrayFromPostgres, and “start_url” not inside startUrlArrayFromPostgres

2

Hi @alisha_liu,
There is no filter for doesn't contain or a not operator in Weaviate.

There is a GH issue with a similar request if you would like to add you comment or vote in there.

Here is an example of how you could delete objects where the values match. But that is the opposite of what you asked for :sweat:

import weaviate, { Filters } from 'weaviate-client';

myCollection = client.collection.get("MyCollection")

await myCollection.data.deleteMany(
  Filters.and(
    myCollection.filter.byProperty('file_name').containsAny(fileNameArrayFromPostgres),
    myCollection.filter.byProperty('start_url').containsAny(startUrlArrayFromPostgres)
  ),
  { dryRun: true, }
)
3

Thanks for your quick response, is their an alternative way to achieve my goal?

4

Yes, you could construct array of notEqual filters, like this:

import weaviate, { Filters } from 'weaviate-client';

const myCollection = client.collection.get("MyCollection")

const fileNameFilters = fileNameArrayFromPostgres.map(
  val => animals.filter.byProperty("file_name").notEqual(val)
)
const startUrlFilters = startUrlArrayFromPostgres.map(
  val => animals.filter.byProperty("start_url").notEqual(val)
)

Then, you could use a spread operator with Filters.and to construct your filter, and test it with a fetch query, like this:

const searchResults = await myCollection.query.fetchObjects({
    filters: Filters.and(...kindFilters, ...nameFilters)
})

for (const item of searchResults.objects) {
    console.log("\n Search Results \n");
    console.log(JSON.stringify(item.properties, null, 2))        
}

And if you are happy with that, you can call delete with the same filter:

await myCollection.data.deleteMany(
  Filters.and(...fileNameFilters, ...startUrlFilters),
  { dryRun: true, }
)

Side note – tokenization for filter matching

This might be a topic for later, but if your filters don’t give the results you expect. It might be due to how Weaviate builds keyword indexes.
Where the equal and notEqual filters are keyword-based.

You may need to set the tokenization on your queried properties to something more strict like field tokenization.

You can learn more about different types of tokenization in the docs.

And here is how you can set tokenization type on a property in your collection.

1 Like