Simple keyword search not working

Support
1

Hello Weaviate team,
We are using Weaviate database to store vector embeddings of user generated data. We are using OpenAI model ‘text-embedding-ada-002’ to generate the vector embeddings. We have an API to search the saved data. This API generates the vector embeddings of search query which queries Weaviate database. But we found out that the search result are not up to the mark.
Example:
As per above process, we generated the vector embeddings of below data and stored in Weaviate database. But when user is trying to search for ‘laptop’ or ‘laptop offer’, Weaviate couldn’t return the actual matching result instead returned non-matching ones. This is failing when Weaviate database contains around 2k-2.5k records. It is searchable if same object is pushed where database contains less than 40-60 records.

Data for which we generated the embeddings and saved in Weaviate database:
“EMAIL
From: John Doe
Title: New Laptop Offer
Body: Hello Brooklyn, I have found few laptop offers. Lenovo - Amazon Link HP - Flipkart Link Dell - Chroma Link Let me know if you found it interesting.
Scribe type: THREAD
Scribe Open/Closed status: Open
Number of attachments: 0
Created at: 15 June 2023 06:32:24 IST
Last Updated at: 15 June 2023 06:32:24 IST
Replies:
Brooklyn Simmons: this scribe not showing in search sometime”
I request you to please look into this on priority and let me know if you need more details on it.

@antas-marcin is looking into it, but thought to create issue here as well to track and resolve this immediately as it’s Severity-1 for us and as per ‘Standard’ plan it has to be resolved within one hour.

2

Hi @ayraa-qa!

I understand why this might feel like a Severity-1 (Sev-1) issue for you. However, I’d like to clarify that in our classification system, a Sev-1 issue indicates a complete system outage where your cluster is inaccessible and all functions cease to operate. Also, I would like to address any potential misunderstanding concerning the one-hour timeframe. This duration isn’t indicative of the time within which the issue will be resolved, but rather, it’s our commitment to you that we will respond and begin addressing your concern within that time.

We will look into this with priority. However, we do need a bit more information from your end to reproduce and diagnose the problem accurately:

  • The schema that was being used
  • The full GraphQL query or code that you utilized to query Weaviate

I hope we can get to a quick resolution of your issue!

3

Hi @byronvoorbach,
Thanks for clarifying what Sev-1 issue means for Weaviate but for us I think if this basic query does not work then entire search feature is down which is core of our product. So, for us its a very high priority issue and hence it needs to be resolved asap.

I have shared all the details with @antas-marcin on slack. He is looking into it actively I believe.
But, let me share it here as well.

Schema definition:
Class name: Post_219

Class definition:

{
  "class": "Post_219",
  "invertedIndexConfig": {
    "bm25": {
      "b": 0.75,
      "k1": 1.2
    },
    "cleanupIntervalSeconds": 60,
    "stopwords": {
      "additions": null,
      "preset": "en",
      "removals": null
    }
  },
  "properties": [
    {
      "dataType": [
        "string"
      ],
      "name": "postId",
      "tokenization": "word"
    },
    {
      "dataType": [
        "string"
      ],
      "name": "lastUpdatedAt",
      "tokenization": "word"
    },
    {
      "dataType": [
        "string"
      ],
      "name": "commentId",
      "tokenization": "word"
    }
  ],
  "replicationConfig": {
    "factor": 1
  },
  "shardingConfig": {
    "virtualPerPhysical": 128,
    "desiredCount": 1,
    "actualCount": 1,
    "desiredVirtualCount": 128,
    "actualVirtualCount": 128,
    "key": "_id",
    "strategy": "hash",
    "function": "murmur3"
  },
  "vectorIndexConfig": {
    "skip": false,
    "cleanupIntervalSeconds": 300,
    "maxConnections": 64,
    "efConstruction": 128,
    "ef": -1,
    "dynamicEfMin": 100,
    "dynamicEfMax": 500,
    "dynamicEfFactor": 8,
    "vectorCacheMaxObjects": 1000000000000,
    "flatSearchCutoff": 40000,
    "distance": "cosine",
    "pq": {
      "enabled": false,
      "bitCompression": false,
      "segments": 0,
      "centroids": 256,
      "encoder": {
        "type": "kmeans",
        "distribution": "log-normal"
      }
    }
  },
  "vectorIndexType": "hnsw",
  "vectorizer": "none"
}

GraphQL query:

{
  Get {
    Post_219(
      nearVector: {
        vector: [-0.005462623, -0.018235134, 0.0043476676, -0.023394277, -0.018578196, 0.0069668237, -0.034464665, -0.027550621, -0.0137357265, -0.02097964, 0.03335631, 0.03435911, -0.032432675, 0.017535811, 0.00058964, 0.009922446, 0.025663773, -0.009632162, 0.037631404, -0.018828897, -0.0316146, -0.000011100849, 0.024594998, -0.01791846, -0.018485833, -0.01663857, 0.009698135, -0.019778918, 0.004888652, -0.005033794, 0.037367508, -0.009473825, -0.03256462, -0.01786568, -0.022906072, 0.0017153164, -0.0014134867, -0.0065808776, -0.011670749, -0.009717927, 0.020900471, 0.013260716, -0.005274598, 0.0135905845, -0.024397077, 0.00094589806, -0.006102568, -0.0034966066, -0.041774552, 0.014369074, -0.001819225, 0.026336705, -0.0120270075, -0.006732617, -0.03578414, 0.0041266554, -0.010694338, 0.0072109266, 0.012851679, -0.0005521175, -0.006940434, 0.011248518, -0.028421475, 0.028500643, 0.002060029, 0.005413143, -0.0066369553, 0.014342684, 0.0004321278, 0.008939438, 0.038607813, 0.030321516, 0.013465234, 0.016018417, -0.0063598654, 0.009546395, -0.017483033, 0.0035493856, -0.0137621155, 0.017905265, 0.015873274, -0.01947544, -0.0145538, -0.0017433553, 0.011512413, -0.016269116, -0.0033102308, 0.00009318787, -0.012422849, -0.018472638, 0.0063070864, 0.012640563, 0.0017895369, 0.031482656, 0.011037402, 0.023644978, 0.0016188299, 0.023922067, -0.003367958, -0.01859139, -0.004360862, 0.026745742, -0.016559402, -0.010601975, -0.02863259, 0.0040112017, 0.0020468342, 0.011123167, 0.039135605, -0.0027840906, -0.020781718, 0.032459065, 0.027339505, 0.0069272392, 0.0022282621, -0.029872896, 0.0053867535, -0.036206372, 0.003214569, -0.022497036, 0.012383265, 0.010028004, 0.047870524, -0.008075182, 0.012356875, -0.0024426766, -0.01611078, 0.0062114247, 0.002579572, 0.012436044, 0.014791305, 0.0010786701, 0.003971617, -0.01663857, -0.020029617, -0.006227918, -0.00918354, 0.019369882, -0.020794913, -0.036391098, 0.0060464903, 0.02451583, -0.015464237, -0.014527411, -0.01023912, 0.030770138, -0.0048787557, 0.014910058, 0.001934679, 0.008536998, -0.008728322, -0.011380465, 0.02401443, -0.020715745, -0.011512413, 0.0020352888, -0.012528407, -0.0044103423, -0.0022975344, -0.0003059531, 0.009816888, 0.018340692, 0.027181169, -0.002426183, 0.00293583, 0.027867295, 0.012323889, 0.016348286, 0.0035032039, 0.009025203, -0.011980825, 0.016994828, -0.00409037, 0.013056197, 0.0048589637, 0.0016633621, -0.010397457, -0.003041388, -0.030215958, -0.006752409, -0.0145538, 0.025940862, 0.014712137, 0.020267123, -0.0032211663, -0.0060794773, -0.01611078, 0.0067095263, 0.006310385, -0.018459443, 0.014632968, 0.015530211, 0.007916845, -0.009012009, -0.6882377, -0.016730933, 0.0048193797, -0.01061517, -0.0021045613, 0.018406665, 0.017746927, 0.0025861694, -0.010423846, 0.010258912, 0.0043707583, -0.0024212352, 0.007963027, 0.0020517823, -0.009348474, -0.028527033, 0.008273103, 0.0068546683, 0.0068876552, -0.007844274, -0.007916845, 0.0032920882, -0.019792113, 0.004875457, -0.005482415, 0.012323889, 0.009823485, -0.009671746, -0.014817695, 0.028447863, -0.018287912, 0.013788505, -0.001217215, -0.027022831, 0.052409515, -0.026745742, -0.011136362, 0.053834546, 0.016427454, 0.03475495, -0.011439841, -0.022589399, -0.022998435, 0.00069932133, -0.029952064, -0.004096967, 0.011796099, -0.012792302, -0.015239927, -0.01914557, 0.0076199635, -0.007078979, 0.007844274, -0.010212731, 0.0036219566, 0.017786512, 0.03127154, -0.018683754, 0.015596185, -0.0027807918, -0.0010621768, 0.006722721, -0.025716553, -0.033013243, -0.027603399, -0.0020072502, -0.027999243, -0.00574631, 0.0035724763, -0.018155964, 0.003753904, 0.0136301685, 0.008174143, -0.0068216817, -0.0040936684, 0.022694957, 0.029872896, -0.009678343, -0.000352547, 0.008695335, 0.034121603, 0.0045884713, -0.013003418, -0.009625564, 0.05227757, -0.026402678, -0.0026818314, -0.010311691, -0.0037044238, 0.008893256, 0.021744935, 0.024383882, -0.01611078, -0.025347099, -0.012666952, 0.020306708, -0.02240467, -0.010918649, 0.0003859462, 0.0053966492, -0.010126964, 0.018353885, 0.013108976, 0.020992834, 0.013551, 0.015675353, 0.005673739, 0.015451043, 0.013999621, -0.040164795, 0.012990223, -0.03069097, -0.00981029, -0.016480232, 0.01339926, -0.031113202, 0.008543596, -0.001116605, 0.025386684, -0.029213158, 0.022536619, -0.007422042, 0.016651765, -0.012442642, -0.03325075, 0.0065379944, -0.008114766, -0.030268738, -0.0052515073, -0.0049645216, -0.013999621, 0.007923443, 0.00715155, -0.022351893, 0.01262077, -0.0010811442, 0.008035597, -0.0135905845, 0.020135175, -0.023605393, -0.03818558, -0.0028814017, -0.013333286, -0.001633674, 0.00010138303, -0.0343855, 0.008002611, -0.007085576, -0.013933647, 0.0137753105, -0.0027659477, -0.0048193797, 0.00511956, 0.027761737, 0.019158766, -0.021124782, 0.0008052916, 0.0008424018, -0.012356875, 0.00016019242, 0.002963869, 0.012238123, -0.008866866, 0.015411458, -0.0052317153, -0.021718545, -0.005841972, 0.04647188, -0.0136697525, -0.038634203, 0.012330486, 0.013894063, 0.0022595995, 0.01892126, -0.003349815, -0.009975225, -0.0066402536, -0.00011751567, 0.015424653, -0.004677536, -0.0039452277, -0.017126774, 0.002920986, -0.0036615408, 0.017654564, 0.019831697, 0.008015806, 0.027009636, -0.024555415, 0.008411648, -0.007989416, 0.014263516, -0.005327377, -0.014474632, -0.001997354, -0.007560587, 0.0136169735, 0.018353885, 0.02682491, 0.000003665922, 0.03731473, -0.015622574, 0.032749347, -0.019607386, 0.0136433635, -0.05325398, 0.017232332, -0.019858086, 0.024502635, 0.020056007, 0.0009013658, -0.03525635, -0.003522996, -0.01889487, 0.0044598226, 0.042988468, -0.004393849, 0.009605772, -0.023341497, -0.017812902, -0.00040862465, -0.0025449358, 0.008761308, -0.015094785, 0.008774503, 0.018551808, -0.005014002, 0.0263499, 0.020438654, -0.033989657, -0.014461437, -0.0056143627, 0.009018606, 0.030374296, -0.0036318528, 0.008259908, 0.0055088047, -0.029213158, 0.0255978, -0.004044188, -0.010978025, 0.02396165, 0.022180362, -0.02301163, -0.003224465, 0.012594381, 0.039716173, -0.0053438703, 0.0026636885, -0.0034537236, -0.02170535, 0.00459177, -0.0069998107, 0.005221819, 0.024911672, -0.03570497, 0.0061916327, 0.01719275, 0.033804927, 0.028183969, -0.005452727, 0.005680336, 0.013069392, -0.012310694, -0.008985619, -0.0024030923, -0.003691229, 0.0040507857, -0.011255114, 0.024555415, 0.021467844, -0.009335279, 0.01844625, -0.0077453135, 0.01508159, -0.0031683873, 0.006472021, 0.030427074, 0.03121876, 0.018433055, -0.011004414, -0.0326174, 0.013788505, -0.004835873, 0.0077980924, -0.0106613515, -0.039161995, 0.00084941153, 0.031429876, 0.01967336, 0.0019132376, 0.018736534, 0.011162751, -0.0033184777, -0.0140524, -0.013221132, 0.043621816, -0.0059343353, 0.005680336, 0.0150552, -0.0063532684, -0.0016806802, -0.034596615, -0.01964697, 0.030532632, -0.00040305813, -0.012792302, -0.011637762, 0.020636575, 0.00958598, -0.013960036, -0.01917196, 0.0061421525, -0.018630976, -0.021692155, 0.0021919764, -0.00071375305, 0.012983626, 0.022945656, 0.01786568, -0.010265509, -0.002160639, -0.025518632, 0.012053397, 0.025201958, 0.009684941, -0.013933647, 0.022351893, -0.008220324, -0.02562419, -0.026244342, -0.0033564125, 0.012323889, 0.011281504, 0.007837676, 0.011657555, -0.0033448671, -0.0073032896, 0.022008829, -0.00065850007, 0.016058002, -0.0076595475, 0.0091637485, -0.030559022, -0.00011462932, -0.0052811955, 0.028658979, -0.004746808, 0.02501723, -0.01096483, 0.016744127, -0.005152547, 0.015688548, 0.00026327634, -0.013465234, 0.0009351773, -0.0022876384, -0.0045950687, 0.0061916327, 0.018842092, 0.019515023, 0.02123034, 0.0030149985, 0.011327686, 0.012732926, 0.028342305, 0.014949642, 0.010318288, 0.01716636, -0.014250321, 0.009368267, 0.026983246, -0.0007092174, -0.018063601, 0.03475495, 0.008576582, -0.0054989085, 0.0074418345, -0.00223321, 0.011697139, 0.0016633621, -0.01636148, -0.00504369, -0.006669942, -0.017746927, -0.011631165, 0.016704543, 0.0049018464, -0.007540795, -0.012178747, -0.009981822, -0.0168101, -0.01736428, 0.0015726483, 0.000056953864, -0.030611802, -0.032063223, -0.006221321, 0.010813091, -0.005139352, 0.016480232, -0.0020847693, -0.011954436, 0.04127315, -0.013115574, -0.022299113, -0.010166548, -0.01427671, 0.010140159, -0.004654445, 0.0028467656, -0.011862073, -0.0033366203, -0.011090181, 0.026943663, -0.009618967, 0.014501021, -0.00073107117, -0.011683944, 0.011149556, 0.01560938, 0.0022381581, -0.00471712, -0.024225546, 0.019607386, -0.020412266, -0.003325075, 0.0011438192, -0.0130496, 0.018472638, 0.009704732, 0.01939627, -0.0048490674, -0.0021853792, 0.025386684, -0.014263516, -0.008853672, 0.006208126, -0.003971617, 0.031324316, -0.006610566, -0.0055088047, 0.005591272, -0.012475628, -0.020069202, -0.023354692, 0.019739334, 0.0053702598, 0.01628231, -0.0009376513, 0.03016318, -0.012086383, -0.008259908, 0.025413074, 0.013880868, 0.0050403913, -0.01427671, -0.01909279, -0.029002042, -0.0054362337, 0.008259908, 0.01917196, 0.0051459493, -0.01719275, -0.019633776, 0.0091505535, 0.00715155, 0.010041199, -0.031772938, -0.020636575, -0.0008444635, 0.00097228755, -0.001648518, 0.020465044, -0.003744008, 0.03222156, -0.02281371, -0.012990223, -0.016796906, -0.022074804, -0.040507857, -0.03995368, 0.013465234, 0.036628604, 0.028711759, -0.0031535432, 0.0060794773, -0.006468722, -0.007435237, 0.012059994, -0.020913666, -0.008299492, -0.017047606, 0.008906451, 0.035045233, -0.0013862726, 0.0040145004, -0.0050008073, -0.022972045, 0.02712839, 0.006663345, 0.0041893306, -0.030031232, -0.018155964, -0.000379555, -0.022299113, -0.015701743, 0.009579383, -0.0150552, -0.009797096, 0.014144763, -0.0003890387, 0.0250964, 0.009341877, 0.014738526, -0.026020031, 0.016823296, 0.007936637, 0.00549561, -0.008622764, -0.014844084, -0.0031782833, 0.024106793, 0.010252315, -0.0019132376, 0.019066403, -0.004219019, 0.0058683613, 0.0058683613, 0.021467844, 0.008998814, -0.026943663, 0.0026735845, -0.014382268, -0.0070723817, -0.036206372, -0.013254118, -0.01967336, 0.013267313, 0.0070525897, -0.014421853, 0.023644978, -0.03478134, 0.001959419, -0.006755708, 0.0105557935, 0.021032419, -0.0016188299, 0.023301914, 0.019383077, -0.0072703026, -0.026310315, -0.0045027058, 0.006227918, -0.017826097, 0.009909251, 0.036364708, 0.0030166479, -0.011512413, 0.023117188, -0.00085930753, 0.01663857, -0.026692962, -0.00015266729, 0.009618967, 0.0038033843, -0.026943663, -0.019726139, -0.028896485, 0.0027263635, 0.0032277638, 0.021547014, -0.0023651575, 0.00015792457, 0.007045992, 0.016876075, -0.017430253, 0.0026405978, -0.0013780259, -0.010846077, 0.010773507, -0.02529432, 0.009038398, -0.010628365, -0.0072307186, 0.027049221, -0.005241611, -0.0026158576, -0.007963027, 0.015398264, -0.011914852, 0.0038000857, 0.005818881, 0.02170535, -0.017417058, -0.026943663, -0.0025333904, 0.0120270075, 0.026429068, -0.007382458, -0.00024905076, -0.0074814186, -0.008543596, -0.023156771, -0.01892126, -0.008477622, -0.015411458, -0.014012815, -0.023077603, -0.022919267, -0.0073890556, -0.0003038914, -0.00421572, 0.0022101193, -0.01214576, -0.0020286916, -0.0064456314, 0.0064225406, 0.0059871143, -0.014738526, 0.014527411, 0.029872896, -0.02012198, 0.035203572, 0.0015775963, 0.017179554, 0.000105403305, 0.03575775, -0.005235014, -0.01149262, 0.007949832, -0.008787698, -0.011703736, -0.020319903, -0.003991409, 0.0060431915, -0.0005677862, 0.0067062275, 0.03464939, -0.0014827591, -0.0068018893, -0.035968866, -0.0268513, -0.0015841937, -0.029054822, -0.000051593503, 0.02276093, 0.009473825, -0.01658579, 0.0065676826, 0.01783929, 0.013273911, -0.032934077, -0.01739067, -0.0056341547, 0.021916466, 0.006452229, 0.0006861266, -0.014171152, -0.0017169658, -0.01658579, -0.007850871, -0.0073230816, 0.013920452, 0.004268499, 0.007751911, -0.0033432178, 0.015015616, -0.012911055, -0.00040883082, -0.0029292328, -0.010008211, -0.019198349, -0.02562419, -0.009440837, 0.024476247, 0.0061718407, -0.0055615837, -0.028131189, -0.004308083, -0.03831753, -0.011881865, 0.0077255215, -0.001011047, -0.0007558113, -0.007349471, -0.0052811955, 0.025399879, -0.011941241, 0.022338698, 0.010819688, -0.011446439, 0.018024018, -0.009223125, 0.018987233, -0.0042223176, -0.028315917, -0.01427671, 0.013056197, -0.0063070864, -0.0015495574, -0.0023107291, -0.008141155, -0.0145538, -0.007593574, 0.005297689, -0.0066864355, 0.0068348763, 0.014250321, -0.021533819, 0.00013967871, -0.009256111, -0.010978025, -0.009170346, -0.001766446, 0.008345674, -0.011749918, -0.002848415, -0.014936448, -0.01839347, 0.012845081, -0.01911918, 0.004971119, -0.0064654234, -0.0039122407, 0.021810908, -0.0010151705, -0.0255978, 0.012482226, 0.006376359, -0.023803314, -0.010654754, 0.039003655, -0.051776167, -0.0120797865, -0.00068241556, 0.026086004, -0.048187196, -0.0022991837, -0.015912859, -0.019554608, -0.011024207, 0.010140159, 0.025492242, 0.014580189, 0.0009607421, -0.007296692, 0.0023140279, -0.017707344, -0.012178747, 0.02346025, -0.038449477, -0.0026587404, 0.011453036, -0.010450236, -0.01967336, -0.007738716, 0.016255923, 0.009077982, 0.009605772, 0.17987072, 0.0020468342, -0.0022546514, 0.035018846, -0.0130496, 0.03631193, 0.018433055, -0.0053504677, 0.0057924916, 0.017351085, -0.031086812, 0.019712944, -0.012231526, -0.004604965, 0.0035724763, 0.009355072, -0.02712839, -0.023829704, -0.02526793, 0.013313495, 0.02301163, -0.018749729, 0.0005438708, -0.026534626, 0.034121603, -0.007864066, -0.016216338, -0.0017944849, 0.032459065, 0.015319095, -0.0136697525, -0.007890455, 0.022655372, 0.0064126444, -0.009018606, 0.011268309, -0.0005129456, -0.009599174, 0.012488823, 0.03367298, -0.0037044238, 0.015965637, 0.0060266983, -0.028500643, -0.00868214, 0.015860079, -0.0070393947, -0.024146378, 0.018762924, 0.009447435, -0.00920993, -0.0077783004, 0.0281048, 0.019040013, -0.030321516, -0.008807491, -0.019211544, -0.0061586457, -0.0034834119, -0.009077982, -0.008484219, 0.027418673, -0.0090449955, 0.009038398, 0.007758508, 0.0073560686, -0.003951825, 0.0042124214, 0.007751911, -0.031482656, 0.009302293, -0.002426183, -0.021810908, -0.008352271, -0.0042421096, -0.03821197, 0.027365895, 0.01764137, 0.00574631, 0.022668567, 0.0046313545, -0.018855287, -0.004133253, -0.018208744, -0.014237126, -0.023658171, -0.0073560686, -0.015094785, 0.008820685, 0.0121985385, 0.009942238, -0.018802507, -0.00632358, -0.0030446867, 0.016506622, 0.009645356, -0.015767716, 0.028447863, -0.012422849, 0.019528218, -0.0404023, -0.0010333132, 0.0053735585, -0.017944848, 0.005182235, -0.0007801391, 0.013894063, 0.012403057, -0.0005385104, -0.00025255562, -0.027233947, -0.035177182, 0.009473825, -0.0029589208, -0.0056572454, 0.016255923, -0.0004129542, -0.023117188, 0.006260905, -0.0049678204, 0.017034411, -0.02293246, 0.013880868, -0.018433055, -0.0026768832, -0.002097964, -0.0042520054, 0.018565001, 0.0063730604, -0.017773317, 0.01099122, -0.015015616, 0.0122249285, -0.021441456, -0.013867673, -0.00016967613, 0.009803693, -0.013135365, -0.007243913, -0.0011998968, 0.008121364, -0.008800893, 0.015860079, -0.010463431, 0.010153354, -0.008418245, 0.026468651, 0.017522616, -0.007685937, -0.019000428, -0.010582183, -0.0061520482, 0.0005030495, -0.02295885, 0.027550621, 0.0065248, -0.04143149, -0.014936448, 0.013973231, 0.0014423502, -0.015688548, -0.018960845, 0.028790927, -0.0038132805, -0.0076727425, 0.014870474, -0.17100385, 0.015912859, 0.00009241473, -0.016876075, 0.009038398, 0.030479854, 0.0069536287, 0.011288102, 0.0012732926, 0.011472828, 0.014949642, 0.006168542, -0.03433272, -0.016902464, 0.0032327117, 0.013603779, -0.010390859, 0.021784518, 0.02117756, -0.0028616097, 0.028395085, -0.033435475, 0.0019627179, 0.009889459, 0.010311691, -0.0035098013, -0.016717738, 0.023354692, -0.009031801, -0.010852675, -0.009731122, -0.022470646, 0.037446678, 0.022628982, 0.026838105, -0.0073428736, -0.024660973, -0.008926243, 0.0051756375, 0.018723339, 0.0024624686, 0.044202384, 0.01608439, 0.012277707, 0.014395463, 0.011881865, 0.019290712, -0.014870474, 0.010001615, -0.01942266, 0.03578414, -0.029186768, -0.0138544785, -0.012825289, 0.028922874, 0.0013656558, 0.001174332, -0.0021210548, -0.014659358, 0.009124164, -0.018881675, -0.020623382, 0.011156154, -0.0030710762, -0.007468224, -0.023156771, -0.0013590584, 0.0021309508, 0.012442642, 0.0019478737, -0.024383882, -0.020596992, 0.028183969, -0.005159144, -0.000113392314, 0.006386255, -0.0061190617, 0.034860507, -0.013320092, -0.027207557, -0.0032607506, 0.030981254, 0.005254806, -0.0026092601, -0.01264716, 0.0021441455, 0.004783094, -0.00306118, -0.000757873, -0.008233519, 0.042988468, -0.02576933, 0.00036079373, -0.026033226, 0.019303907, 0.010529404, -0.007369263, 0.004571978, -0.02173174, -0.0067491103, -0.008569985, -0.0045455885, -0.025848499, -0.00093435263, 0.034913287, 0.0017169658, 0.01772054, 0.01666496, 0.042671796, -0.0023404174, -0.010997817, 0.016321896, 0.03472856, 0.020319903, 0.006326879, -0.014646163, -0.007956429, -0.013273911, 0.022774125, 0.0068216817, 0.022747735, -0.009869667, 0.00818074, 0.0139072575, -0.0044433293, -0.011255114, -0.07985458, -0.032010444, 0.008042195, 0.0057199206, -0.007916845, 0.023882482, -0.007362666, 0.010437041, -0.017905265, 0.041721772, 0.0003686281, -0.027207557, -0.030506244, -0.020069202, 0.00039336825, -0.0008931191, -0.007666145, -0.007976222, -0.018987233, 0.013973231, -0.006241113, -0.0058122836, 0.014171152, -0.0051228586, -0.022312308, 0.000933528, -0.017324695, -0.0034471264, 0.02757701, 0.011070388, 0.026204757, -0.012323889, 0.01942266, -0.008781101, -0.0039683185, -0.0062642037, -0.018789312, -0.0062510087, 0.039557837, -0.02963539, -0.0010827935, -0.017456643, -0.007890455, -0.03267018, 0.0024624686, 0.0038528645, -0.01121553, 0.014870474, 0.013564195, -0.031324316, 0.002612559, -0.009381462, -0.03937311, -0.0051954295, 0.023948457, -0.027867295, 0.008609569, 0.018472638, -0.014910058, -0.016678154, -0.020887276, 0.006973421, -0.018063601, 0.016480232, -0.005891452, -0.008477622, -0.0067392145, -0.021626182, 0.020583797, -0.006973421, 0.0021837298, -0.019026818, -0.016255923, 0.01783929, -0.017905265, 0.013946842, -0.013894063, -0.0058122836, 0.009427643, -0.011202335, -0.010707533, -0.026587404, 0.016071195, 0.007943234, 0.017654564, 0.0070591867, 0.014527411, 0.014355879, -0.009645356, -0.01164436, 0.015147564, 0.04129954, 0.03259101, -0.0063202814, -0.018076796, -0.009816888, -0.009896057, -0.0010737221, 0.010984623, -0.007184537, 0.016018417, -0.012693342, -0.06417923, 0.016955243, -0.011030804, -0.017443448, 0.022140777, 0.014791305, -0.00235856, -0.032881297, -0.00080487924, -0.0025333904, -0.022694957, 0.0065313973, -0.019567803, -0.0027642983, -0.025228346, -0.005884855, 0.03478134, -0.0047204187, 0.013920452, 0.011222128, 0.014184347, 0.0062872944, -0.0020286916, 0.012765912, -0.02454222, -0.00052201696, -0.006666643, 0.008451232, -0.009935641, -0.039161995, -0.0056671416, -0.0091637485, -0.0016287259, 0.024964452, -0.012238123, -0.0043377713, 0.004707224, 0.0030760241, -0.0040705777, 0.031456266, -0.016796906, -0.024436662, 0.024555415, -0.022523424, -0.02323594, -0.0014786357, -0.010773507, -0.012792302, 0.013280507, 0.011683944, 0.022180362, 0.01058878, 0.012944042, -0.008207129, -0.014065594, -0.0281048, 0.021124782, 0.00074550294, 0.008391856, -0.01816916, 0.020781718, 0.0034240354, 0.008695335, 0.008108169, 0.016704543, -0.008424843, -0.0013516364, 0.0018274718, 0.01616356, -0.014844084, 0.006663345, 0.00043171545, 0.02117756, -0.00338775, 0.02331511, 0.0020418863, 0.0015800704, 0.0012015462, -0.018380275, 0.013141963, -0.0060860747, -0.0050535863, -0.019937254, -0.004740211, 0.045126017, 0.017060801, -0.0137753105, -0.022127582, 0.007085576, 0.014619774, -0.019818502, 0.0047369124, 0.0021507428, 0.00858318, -0.01136727, 0.007949832, -0.0071119657, -0.021824103, 0.006587475, 0.006458826, 0.0073758606, 0.0015792457, -0.015728133, -0.020689355, -0.013933647, 0.013214534, -0.0038891502, -0.019528218, -0.0130496, 0.023803314, 0.025637383, 0.01262077, 0.013161755, 0.026719352, -0.01861778, -0.00020266298, -0.015701743, -0.008127961, -0.03778974, 0.01302321, 0.03530913, 0.0034075421, 0.022193557, -0.012290902, 0.01995045, 0.007204329, 0.02223314, -0.026165172, -0.0015949145, -0.0009821836, 0.019963644, -0.011433244, -0.014580189, -0.015187148, -0.010285301, 0.0049513266, -0.005874959, 0.034860507, -0.013445442, 0.07742675, -0.004374057, 0.0025663772, 0.0021771323, 0.0032310623, 0.027207557, 0.014091984, -0.003691229, -0.026481846, -0.026495041, 0.0015973884, 0.005188832, -0.00041831454, 0.0030892189, -0.0286062, 0.021428261, -0.008952633, 0.011235323, -0.025281126, -0.020781718, 0.040798143, 0.013973231, 0.00071911345, 0.0077980924, -0.00064901635, 0.012746121, 0.016453844, -0.0150552, -0.015226732, -0.029477052, 0.004816081, 0.0075539895, -0.029186768, -0.008853672, 0.026363093, -0.021639377, -0.013128768, -0.002055081, -0.0047863927, 0.007969624, -0.00028574863, 0.032432675, -0.021269923, -0.023592198, 0.026495041, -0.0066930326, -0.015846886, 0.00983668, -0.017509423]
      },
      limit: 3000
    ) {
      postId
      _additional {
        id
        certainty
        distance
      }
    }
  }
}

Please keep me updated and also share the ETA on the same

4

@byronvoorbach @antas-marcin
Could you please help with the current status and the ETA of the fix ?

5

HNSW is an approximate nearest neighbour algorithm. This means there is always a chance a vector will not be found in a search even though it would be found via brute force (using where filter).

The solution to this problem is to re-index your data with higher efConstruction parameter (for example efConstruction=512) that way your newly built graph will have higher number of node candidates that will be considered during vector search.