# Using a local reranker-transformers reduces performance by 100x

**URL:** <https://forum.weaviate.io/t/using-a-local-reranker-transformers-reduces-performance-by-100x/2502>\
**Category:** Support\
**Tags:** python\
**Created:** [May 27, 2024, 6:15pm UTC](https://forum.weaviate.io/t/using-a-local-reranker-transformers-reduces-performance-by-100x/2502 "2024-05-27T18:15:16Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Benjamin\_Lush](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/benjamin_lush/32/1433_2.png) [@Benjamin\_Lush](https://forum.weaviate.io/u/Benjamin_Lush)\
**Post date:** [May 27, 2024, 6:15pm UTC](https://forum.weaviate.io/t/using-a-local-reranker-transformers-reduces-performance-by-100x/2502/1 "2024-05-27T18:15:16Z")

</div>

### Description

Using Weaviate running from docker-compose  
Image: [cr.weaviate.io/semitechnologies/weaviate:1.25.1](http://cr.weaviate.io/semitechnologies/weaviate:1.25.1)  
Reranker: [cr.weaviate.io/semitechnologies/reranker-transformers:cross-encoder-ms-marco-MiniLM-L-6-v2](http://cr.weaviate.io/semitechnologies/reranker-transformers:cross-encoder-ms-marco-MiniLM-L-6-v2)

### Server Setup Information

- Weaviate Server Version: 1.25.1
- Deployment Method docker-compose
- Multi Node? No
- Client Language and Version: Python 3.12.3

### Any additional Information

When running

```auto
collection = clientv4.collections.get(collection_name)
hybrid_documentsv4 = collection.query.hybrid(
    query=user_input,
    limit=4,
    query_properties=["text", "key"],
    rerank=Rerank(prop="text", query=user_input),
    return_metadata=MetadataQuery(score=True)
)

```

responses take 50000+ ms

When I disable reranking:

```auto
collection = clientv4.collections.get(collection_name)
hybrid_documentsv4 = collection.query.hybrid(
    query=user_input,
    limit=4,
    query_properties=["text", "key"],
    return_metadata=MetadataQuery(score=True)
)

```

responses take 500 ms (100x faster)

My reranker-transformers docker container uses a max of about 109% of 1 of 10 CPU core and 1.6GB RAM when executing for the 50 seconds. Even if I run parallel python threads there is no improvement in speed of reranking. And I cannot get the CPU and RAM usage to take more from my host.

I have confirmed my Docker machine can access up to 10 cores and 14 GB of RAM and there is no resoruce contention while doing the hybrid-search. This is purely and issue with the container or how weaviate client does the reranking it seems. I am unsure of how to improve my performance. 50 seconds is dreadfully slow!

---

<div class="post-metadata">

**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\
**Post date:** [May 31, 2024, 6:14pm UTC](https://forum.weaviate.io/t/using-a-local-reranker-transformers-reduces-performance-by-100x/2502/2 "2024-05-31T18:14:37Z")

</div>

Hi!

I believe that for improving performance you will need a CUDA enabled hardware and make sure it will run with it.

Let me know if that helps.

Thanks!

---

<div class="post-metadata">

**Author:** ![Nicholas\_Miller](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/nicholas_miller/32/1930_2.png) [@Nicholas\_Miller](https://forum.weaviate.io/u/Nicholas_Miller)\
**Post date:** [July 11, 2024, 10:31pm UTC](https://forum.weaviate.io/t/using-a-local-reranker-transformers-reduces-performance-by-100x/2502/3 "2024-07-11T22:31:59Z")

</div>

I had a similar issue. Same docker container. In my case, the memory usage of the container climbed a bit with each rerank call. Eventually the host machine needed to use swap, which reduced performance. Then the host machine crashed. Took some time to diagnose as I couldn’t even ssh to the machine when out of memory!  
So there must be a memory leak in the reranker container somewhere. I note from the source on Github that the CrossEncoder class uses the threadpool. There are some bugs filed with sentence-transformers to do with memory leaks and the threadpool.

---

<div class="post-metadata">

**Author:** ![DudaNogueira](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.weaviate.io/dudanogueira/32/7846_2.png) [@DudaNogueira](https://forum.weaviate.io/u/DudaNogueira)\
**Post date:** [July 15, 2024, 4:45pm UTC](https://forum.weaviate.io/t/using-a-local-reranker-transformers-reduces-performance-by-100x/2502/4 "2024-07-15T16:45:32Z")

</div>

hi @Nicholas_Miller !!

Welcome to our community!!

Do you mean this code, right?

> **[Issues · weaviate/reranker-transformers](https://github.com/weaviate/reranker-transformers/issues)**
>
> Contribute to weaviate/reranker-transformers development by creating an account on GitHub.

So in your findings, this could be something from upstream?

Could you open a issue so we can keep track of that? Also, mentioning this thread.

Thanks for helping us on that!