Visualize Database Contents

Support
1

Hello guys,

I am uncertain if this question has already been asked (I didn’t find it).
I want to display my database contents is a 3D grid which means I need 3 points.

I know how to do that for a query, but I would like to display the whole database and their respective positions to be able to see the clusters and not the relative position of the query.

(For a query, I can add: _additional { id distance featureProjection(dimensions: 3) { vector } })

Is there a way to do that?

3

hi @micartey !

we had a podcast recently with Adriy Mulyar from Nomic AI: https://www.youtube.com/watch?v=qb2nLeRpMWQ

I believe this might be usefull for your usecase. However, I have not tried it yet myself :frowning: yet!

1 Like
4

Thank you for your reply.

I didn’t watch the podcast except the first 2 minutes and looked up the GitHub repository, which gave me a good Idea.

My TypeScript code looks like this:

const response = await client.graphql
  .get()
  .withClassName('Test')
  .withFields('id _additional { featureProjection(dimensions: 3) { vector } }')
  .do();

This returns all elements with a vector of RxRxR which is what I wanted.
Hope this is like it is correct.

Sad that such information is not really in the docs but on some “podcasts”.
The docs require some more attention…

5

Thanks for the input, we really appreciate it!

New features and docs are constantly being built. For now we have a focus on new core features, client libraries, etc.

We should have some more news regarding visualization, but those are usually done with third party softwares.

There is also a nice tool called Phoenix, from Arise, check it out:

6

HI @micartey

The feature projection operator is covered here: GraphQL - Additional properties | Weaviate - vector database

FYI your code snippet will work, but only to the maximum of the get query, which for many is lower than the size of the whole database.

To get all objects, I suggest you use the cursor API to loop through the entire DB :slight_smile: Read all objects | Weaviate - vector database

In relation to the comment about the docs & Nomic, they are not a part of our product - which is why it’s not captured in our docs :slight_smile:

1 Like
7

Hi @jphwang

Thanks for your input.

I understand that Nomic is not documented as it is not part of the product.
I, personally, don’t like how the documentation is structured. It feels clunky, and it’s hard to know what is documented and where without clicking through all the pages.

Let me give you an example:

  • I want my search results to be projected into 3 dimensions. What do I do? I go to How-to: Search. There are a lot of different search types (which is good if you know what they are) and when you find the fitting search method and also look at the base page of the how-to guide you still don’t know if and how it is possible to project the response into 3 dimensions. Where is it documented? 📚 References: API > GraphQL | Search > GraphQL - Additional properties How should I know?
  • I want to understand what algorithm is used to get the nearest neighbor. Where do I look? :eyes: Configuration? Concepts? If I look at concepts, is it in Indexing, Vector Indexing or Filtered Vector Search?

I don’t want to talk bad about the documentation.
It just seems like I am to stupid to find things…

8

Hi @micartey - thanks for the feedback. And I’m sorry you’ve had a frustrating experience.

The first point is fair. We haven’t yet created a page for aspects like feature projection.

The second is an interesting one. I think for very specific questions like that, maybe the site search would be useful. Have you tried using it? How was your experience with it?

9

Hi @jphwang

Yes, I’ve used the seach feature and it is quite good at finding results.
The main issue with searching is the following: How do I know what to search for if I don’t know it exists?

From the perspective of a DAU (dumbest assumed user) it is impractical.
There should be a clear structure I can follow (or start at any point) without having to first buy a book and read everything. This tool is clearly aimed at developers that already worked with vector databases or have minimal knowledge about the theories behind vector search.

I spent 2 days setting up everything, storing and reading data and visualizing it and trying to understand some concepts. This could’ve taken just an hour or two. I think there is room for improvement in that regard. I suggest you try an experiment with some experienced and inexperienced developers (with and without knowledge) and observe them trying to solve some common use cases like I did. This would greatly improve your product.

10

Hi @micartey .

Again, sorry that you had a frustrating experience. Do you mind if we follow up with you separately for further feedback?

Cheers,
JP