Crashing 1.24.10

Dan_Gordon · May 6, 2024, 5:38pm

Description

I have been trying to batch import around 400,000 new entries today. I was using 4.4.4 and 1.23.9 and my python script kept throwing RPC error after x number of entries. So updated my docker to 1.24.10 and my python client to 4.5.7 and tried again - after I refactored my code to drop wcs.dataObject I was no longer getting the python error, but after every few hundred entries, the client would pause for 10-15 seconds. Looking at the logs, it looks like the docker instance is crashing and getting reset

Do I need to add some more resources to the docker container? how can I check that?

I’m running this on a DO droplet with 2 vCPU, 4 GB RAM and 80 GB HD

Server Setup Information

Weaviate Server Version: 1.24.10
Deployment Method: docker
Multi Node? 1
Client Language and Version: Python 4.5.7

Logs

weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"Federal","index":"federal","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/federal/code_federal_regulations/lsm/property_section_searchable/segment-1715016532273976675.wal","shard":"code_federal_regulations","time":"2024-05-06T17:31:58Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal_success","class":"Federal","index":"federal","level":"info","msg":"successfully recovered from write-ahead-log","path":"/var/lib/weav/federal/code_federal_regulations/lsm/property_section_searchable/segment-1715016532273976675.wal","shard":"code_federal_regulations","time":"2024-05-06T17:31:58Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"Federal","index":"federal","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/federal/code_federal_regulations/lsm/property_section_title/segment-1715016532276025275.wal","shard":"code_federal_regulations","time":"2024-05-06T17:31:58Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal_success","class":"Federal","index":"federal","level":"info","msg":"successfully recovered from write-ahead-log","path":"/var/lib/weav/federal/code_federal_regulations/lsm/property_section_title/segment-1715016532276025275.wal","shard":"code_federal_regulations","time":"2024-05-06T17:31:58Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"Federal","index":"federal","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/federal/code_federal_regulations/lsm/property_section_title_searchable/segment-1715016532277888914.wal","shard":"code_federal_regulations","time":"2024-05-06T17:31:58Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal_success","class":"Federal","index":"federal","level":"info","msg":"successfully recovered from write-ahead-log","path":"/var/lib/weav/federal/code_federal_regulations/lsm/property_section_title_searchable/segment-1715016532277888914.wal","shard":"code_federal_regulations","time":"2024-05-06T17:31:58Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"Federal","index":"federal","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/federal/code_federal_regulations/lsm/property_body/segment-1715016532288822664.wal","shard":"code_federal_regulations","time":"2024-05-06T17:31:58Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal_success","class":"Federal","index":"federal","level":"info","msg":"successfully recovered from write-ahead-log","path":"/var/lib/weav/federal/code_federal_regulations/lsm/property_body/segment-1715016532288822664.wal","shard":"code_federal_regulations","time":"2024-05-06T17:31:58Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"Federal","index":"federal","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/federal/code_federal_regulations/lsm/property_body_searchable/segment-1715016532291176413.wal","shard":"code_federal_regulations","time":"2024-05-06T17:31:58Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal_success","class":"Federal","index":"federal","level":"info","msg":"successfully recovered from write-ahead-log","path":"/var/lib/weav/federal/code_federal_regulations/lsm/property_body_searchable/segment-1715016532291176413.wal","shard":"code_federal_regulations","time":"2024-05-06T17:31:58Z"}
weaviate_1       | {"level":"info","msg":"Completed loading shard federal_code_federal_regulations in 23.080164ms","time":"2024-05-06T17:31:58Z"}
weaviate_1       | {"action":"hnsw_vector_cache_prefill","count":3000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-05-06T17:31:58Z","took":117874}
weaviate_1       | {"action":"hnsw_vector_cache_prefill","count":321691,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-05-06T17:32:24Z","took":25760808930}

Docker setup

services:
  weaviate:
    command:
      - --host
      - 0.0.0.0
      - --port
      - '8060'
      - --scheme
      - http
    image: semitechnologies/weaviate:1.24.10
    ports:
      - 8060:8060
      - 50051:50051
    volumes:
      - weaviate_data:/var/lib/weav
    restart: on-failure:0
    environment:
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: '/var/lib/weav'
      DEFAULT_VECTORIZER_MODULE: 'none'
      ASYNC_INDEXING: 'true'
      CLUSTER_HOSTNAME: 'node1'

DudaNogueira · May 6, 2024, 5:45pm

hi @Dan_Gordon !!

Can you get the logs prior to the ones you pasted? Those logs will usually appear when starting Weavaite after a crash.

Not sure about the dimensions of your vectors, but it may be lack of memory.

Check here the docs on how to proper plan the resources usage for your objects: Resource Planning | Weaviate - Vector Database

Notice that as a good rule of thumb, we double the memory calculated based on the objects you store, so Weaviate has enough room to process ingestions.

Let me know if this helps.

Thanks!

Dan_Gordon · May 6, 2024, 5:57pm

weaviate_1       | {"action":"startup","default_vectorizer_module":"none","level":"info","msg":"the default vectorizer modules is set to \"none\", as a result all new schema classes without an explicit vectorizer setting, will use this vectorizer","time":"2024-05-06T17:56:15Z"}
weaviate_1       | {"action":"startup","auto_schema_enabled":true,"level":"info","msg":"auto schema enabled setting is set to \"true\"","time":"2024-05-06T17:56:15Z"}
weaviate_1       | {"level":"info","msg":"No resource limits set, weaviate will use all available memory and CPU. To limit resources, set LIMIT_RESOURCES=true","time":"2024-05-06T17:56:15Z"}
weaviate_1       | {"level":"info","msg":"async indexing enabled","time":"2024-05-06T17:56:15Z"}
weaviate_1       | {"action":"grpc_startup","level":"info","msg":"grpc server listening at [::]:50051","time":"2024-05-06T17:56:15Z"}
weaviate_1       | {"action":"restapi_management","level":"info","msg":"Serving weaviate at http://[::]:8060","time":"2024-05-06T17:56:15Z"}
weaviate_1       | {"action":"telemetry_push","level":"info","msg":"telemetry started","payload":"\u0026{MachineID:6333bcf3-eaec-45f0-811d-3cd21ab7d1a9 Type:INIT Version:1.24.10 Modules: NumObjects:0 OS:linux Arch:amd64}","time":"2024-05-06T17:56:15Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"Federal","index":"federal","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/federal/code_federal_regulations/lsm/objects/segment-1715017905405488676","shard":"code_federal_regulations","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"County","index":"county","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/county/whatcom_county_code/lsm/objects/segment-1715017905393954074","shard":"whatcom_county_code","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"County","index":"county","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/county/whatcom_county_code/lsm/property__id/segment-1715017905444387042","shard":"whatcom_county_code","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"City","index":"city","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/city/bellingham_municipal_code/lsm/objects/segment-1715017905461352510","shard":"bellingham_municipal_code","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"City","index":"city","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/city/bellingham_municipal_code/lsm/property__id/segment-1715017905481906937","shard":"bellingham_municipal_code","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"State","index":"state","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/state/revised_code_washington/lsm/objects/segment-1715017905487950053","shard":"revised_code_washington","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"Federal","index":"federal","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/federal/code_federal_regulations/lsm/property_section/segment-1715017905441673092","shard":"code_federal_regulations","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"Federal","index":"federal","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/federal/code_federal_regulations/lsm/property_section_searchable/segment-1715017905441869214","shard":"code_federal_regulations","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"Admin","index":"admin","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/admin/washington_admin_code/lsm/objects/segment-1715017905421204103","shard":"washington_admin_code","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"Federal","index":"federal","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/federal/code_federal_regulations/lsm/property_section_title/segment-1715017905426135621","shard":"code_federal_regulations","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"Federal","index":"federal","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/federal/code_federal_regulations/lsm/property_section_title_searchable/segment-1715017905426226983","shard":"code_federal_regulations","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"Federal","index":"federal","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/federal/code_federal_regulations/lsm/property_body/segment-1715017905426316001","shard":"code_federal_regulations","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"Federal","index":"federal","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/federal/code_federal_regulations/lsm/property_body_searchable/segment-1715017905442058762","shard":"code_federal_regulations","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"Federal","index":"federal","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/federal/code_federal_regulations/lsm/property__id/segment-1715017905420448030","shard":"code_federal_regulations","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"Admin","index":"admin","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/admin/washington_admin_code/lsm/property__id/segment-1715017905463321517","shard":"washington_admin_code","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"County","index":"county","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/county/whatcom_county_code/lsm/property_section/segment-1715017905416946111","shard":"whatcom_county_code","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"level":"info","msg":"Completed loading shard federal_code_federal_regulations in 28.338847ms","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"County","index":"county","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/county/whatcom_county_code/lsm/property_section_searchable/segment-1715017905439946568","shard":"whatcom_county_code","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"hnsw_vector_cache_prefill","count":3000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-05-06T17:56:16Z","took":589299}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"County","index":"county","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/county/whatcom_county_code/lsm/property_section_title/segment-1715017905427976367","shard":"whatcom_county_code","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"County","index":"county","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/county/whatcom_county_code/lsm/property_section_title_searchable/segment-1715017905453303208","shard":"whatcom_county_code","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"County","index":"county","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/county/whatcom_county_code/lsm/property_body/segment-1715017905422205272","shard":"whatcom_county_code","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"State","index":"state","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/state/revised_code_washington/lsm/property__id/segment-1715017905570939267","shard":"revised_code_washington","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"County","index":"county","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/county/whatcom_county_code/lsm/property_body_searchable/segment-1715017905423921849","shard":"whatcom_county_code","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"State","index":"state","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/state/revised_code_washington/lsm/property_section/segment-1715017905541697023","shard":"revised_code_washington","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"State","index":"state","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/state/revised_code_washington/lsm/property_section_searchable/segment-1715017905548684159","shard":"revised_code_washington","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"State","index":"state","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/state/revised_code_washington/lsm/property_section_title/segment-1715017905565964543","shard":"revised_code_washington","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"State","index":"state","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/state/revised_code_washington/lsm/property_section_title_searchable/segment-1715017905601727656","shard":"revised_code_washington","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"State","index":"state","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/state/revised_code_washington/lsm/property_body/segment-1715017905565214527","shard":"revised_code_washington","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"State","index":"state","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/state/revised_code_washington/lsm/property_body_searchable/segment-1715017905661056278","shard":"revised_code_washington","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"level":"info","msg":"Completed loading shard county_whatcom_county_code in 55.30637ms","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"City","index":"city","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/city/bellingham_municipal_code/lsm/property_section/segment-1715017905518536629","shard":"bellingham_municipal_code","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"City","index":"city","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/city/bellingham_municipal_code/lsm/property_section_searchable/segment-1715017905540551917","shard":"bellingham_municipal_code","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"City","index":"city","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/city/bellingham_municipal_code/lsm/property_section_title/segment-1715017905527423170","shard":"bellingham_municipal_code","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"City","index":"city","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/city/bellingham_municipal_code/lsm/property_section_title_searchable/segment-1715017905536979715","shard":"bellingham_municipal_code","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"City","index":"city","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/city/bellingham_municipal_code/lsm/property_body/segment-1715017905524606138","shard":"bellingham_municipal_code","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"City","index":"city","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/city/bellingham_municipal_code/lsm/property_body_searchable/segment-1715017905548867183","shard":"bellingham_municipal_code","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"Admin","index":"admin","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/admin/washington_admin_code/lsm/property_section/segment-1715017905448371857","shard":"washington_admin_code","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"Admin","index":"admin","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/admin/washington_admin_code/lsm/property_section_searchable/segment-1715017905462753588","shard":"washington_admin_code","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"level":"info","msg":"Completed loading shard city_bellingham_municipal_code in 105.943763ms","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"Admin","index":"admin","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/admin/washington_admin_code/lsm/property_body/segment-1715017905444697867","shard":"washington_admin_code","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"Admin","index":"admin","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/admin/washington_admin_code/lsm/property_body_searchable/segment-1715017905516257950","shard":"washington_admin_code","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"Admin","index":"admin","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/admin/washington_admin_code/lsm/property_section_title/segment-1715017905432062944","shard":"washington_admin_code","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"Admin","index":"admin","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/admin/washington_admin_code/lsm/property_section_title_searchable/segment-1715017905443959688","shard":"washington_admin_code","time":"2024-05-06T17:56:16Z"}
weaviate_1       | {"action":"hnsw_vector_cache_prefill","count":5000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-05-06T17:56:16Z","took":252279688}
weaviate_1       | {"action":"hnsw_vector_cache_prefill","count":7000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-05-06T17:56:16Z","took":379431959}
weaviate_1       | {"level":"info","msg":"Completed loading shard admin_washington_admin_code in 659.591846ms","time":"2024-05-06T17:56:17Z"}
weaviate_1       | {"level":"info","msg":"Completed loading shard state_revised_code_washington in 709.244983ms","time":"2024-05-06T17:56:17Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"Federal","index":"federal","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/federal/national_institute_health/lsm/objects/segment-1715018017407364498","shard":"national_institute_health","time":"2024-05-06T17:56:17Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"Federal","index":"federal","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/federal/national_institute_health/lsm/property__id/segment-1715017906674002479","shard":"national_institute_health","time":"2024-05-06T17:56:17Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"Federal","index":"federal","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/federal/national_institute_health/lsm/property_section/segment-1715017906682470097","shard":"national_institute_health","time":"2024-05-06T17:56:17Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"Federal","index":"federal","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/federal/national_institute_health/lsm/property_section_title/segment-1715017906728008520","shard":"national_institute_health","time":"2024-05-06T17:56:17Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal_success","class":"Federal","index":"federal","level":"info","msg":"successfully recovered from write-ahead-log","path":"/var/lib/weav/federal/national_institute_health/lsm/property__id/segment-1715017906674002479.wal","shard":"national_institute_health","time":"2024-05-06T17:56:17Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"Federal","index":"federal","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/federal/national_institute_health/lsm/property_body/segment-1715017906870925119","shard":"national_institute_health","time":"2024-05-06T17:56:17Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal_success","class":"Federal","index":"federal","level":"info","msg":"successfully recovered from write-ahead-log","path":"/var/lib/weav/federal/national_institute_health/lsm/property_section_title/segment-1715017906728008520.wal","shard":"national_institute_health","time":"2024-05-06T17:56:17Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"Federal","index":"federal","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/federal/national_institute_health/lsm/property_section_title_searchable/segment-1715017906738396250","shard":"national_institute_health","time":"2024-05-06T17:56:17Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal_success","class":"Federal","index":"federal","level":"info","msg":"successfully recovered from write-ahead-log","path":"/var/lib/weav/federal/national_institute_health/lsm/property_section_title_searchable/segment-1715017906738396250.wal","shard":"national_institute_health","time":"2024-05-06T17:56:17Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal_success","class":"Federal","index":"federal","level":"info","msg":"successfully recovered from write-ahead-log","path":"/var/lib/weav/federal/national_institute_health/lsm/property_section/segment-1715017906682470097.wal","shard":"national_institute_health","time":"2024-05-06T17:56:17Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"Federal","index":"federal","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/federal/national_institute_health/lsm/property_section_searchable/segment-1715017906722679155","shard":"national_institute_health","time":"2024-05-06T17:56:17Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal_success","class":"Federal","index":"federal","level":"info","msg":"successfully recovered from write-ahead-log","path":"/var/lib/weav/federal/national_institute_health/lsm/property_section_searchable/segment-1715017906722679155.wal","shard":"national_institute_health","time":"2024-05-06T17:56:17Z"}
weaviate_1       | {"action":"lsm_compaction","class":"Federal","compaction":5263,"index":"federal","level":"info","msg":"start lsm compaction","shard":"national_institute_health","time":"2024-05-06T17:56:18Z"}
weaviate_1       | {"action":"lsm_compaction_finished","class":"Federal","compaction":5263,"index":"federal","level":"info","msg":"start lsm compaction","shard":"national_institute_health","time":"2024-05-06T17:56:18Z"}
weaviate_1       | {"action":"lsm_compaction","class":"Federal","compaction":9628,"index":"federal","level":"info","msg":"start lsm compaction","shard":"national_institute_health","time":"2024-05-06T17:56:18Z"}
weaviate_1       | {"action":"lsm_compaction_finished","class":"Federal","compaction":9628,"index":"federal","level":"info","msg":"start lsm compaction","shard":"national_institute_health","time":"2024-05-06T17:56:18Z"}
weaviate_1       | {"action":"lsm_compaction","class":"Federal","compaction":839,"index":"federal","level":"info","msg":"start lsm compaction","shard":"national_institute_health","time":"2024-05-06T17:56:18Z"}
weaviate_1       | {"action":"lsm_compaction_finished","class":"Federal","compaction":839,"index":"federal","level":"info","msg":"start lsm compaction","shard":"national_institute_health","time":"2024-05-06T17:56:18Z"}
weaviate_1       | {"action":"hnsw_vector_cache_prefill","count":69046,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-05-06T17:56:21Z","took":3970112420}
weaviate_1       | {"action":"hnsw_vector_cache_prefill","count":69046,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-05-06T17:56:21Z","took":4254443789}
weaviate_1       | {"action":"lsm_compaction","class":"Federal","compaction":8902,"index":"federal","level":"info","msg":"start lsm compaction","shard":"national_institute_health","time":"2024-05-06T17:56:21Z"}
weaviate_1       | {"action":"lsm_compaction_finished","class":"Federal","compaction":8902,"index":"federal","level":"info","msg":"start lsm compaction","shard":"national_institute_health","time":"2024-05-06T17:56:21Z"}
weaviate_1       | {"action":"lsm_compaction","class":"Federal","compaction":4239,"index":"federal","level":"info","msg":"start lsm compaction","shard":"national_institute_health","time":"2024-05-06T17:56:21Z"}
weaviate_1       | {"action":"lsm_compaction_finished","class":"Federal","compaction":4239,"index":"federal","level":"info","msg":"start lsm compaction","shard":"national_institute_health","time":"2024-05-06T17:56:21Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal_success","class":"Federal","index":"federal","level":"info","msg":"successfully recovered from write-ahead-log","path":"/var/lib/weav/federal/national_institute_health/lsm/property_body/segment-1715017906870925119.wal","shard":"national_institute_health","time":"2024-05-06T17:56:22Z"}
weaviate_1       | {"action":"lsm_segment_init","class":"Federal","index":"federal","level":"info","msg":"Discarded (partially written) LSM segment, because an active WAL for the same segment was found. A recovery from the WAL will follow.","path":"/var/lib/weav/federal/national_institute_health/lsm/property_body_searchable/segment-1715017906966999687.db","shard":"national_institute_health","time":"2024-05-06T17:56:22Z","wal_path":"segment-1715017906966999687.wal"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"Federal","index":"federal","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/federal/national_institute_health/lsm/property_body_searchable/segment-1715017906966999687","shard":"national_institute_health","time":"2024-05-06T17:56:22Z"}
weaviate_1       | {"action":"lsm_compaction","class":"Federal","compaction":952,"index":"federal","level":"info","msg":"start lsm compaction","shard":"national_institute_health","time":"2024-05-06T17:56:24Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal_success","class":"Federal","index":"federal","level":"info","msg":"successfully recovered from write-ahead-log","path":"/var/lib/weav/federal/national_institute_health/lsm/property_body_searchable/segment-1715017906966999687.wal","shard":"national_institute_health","time":"2024-05-06T17:56:24Z"}
weaviate_1       | {"action":"lsm_recover_from_active_wal","class":"Federal","index":"federal","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weav/federal/national_institute_health/lsm/property_body_searchable/segment-1715018017588486282","shard":"national_institute_health","time":"2024-05-06T17:56:24Z"}
weaviate_1       | {"action":"lsm_compaction_finished","class":"Federal","compaction":952,"index":"federal","level":"info","msg":"start lsm compaction","shard":"national_institute_health","time":"2024-05-06T17:56:25Z"}
weaviate_1       | {"action":"lsm_compaction","class":"Federal","compaction":3857,"index":"federal","level":"info","msg":"start lsm compaction","shard":"national_institute_health","time":"2024-05-06T17:56:25Z"}
weaviate_1       | {"action":"lsm_compaction_finished","class":"Federal","compaction":3857,"index":"federal","level":"info","msg":"start lsm compaction","shard":"national_institute_health","time":"2024-05-06T17:56:25Z"}
weaviate_1       | {"level":"info","msg":"Completed loading shard federal_national_institute_health in 10.213221636s","time":"2024-05-06T17:56:27Z"}
weaviate_1       | {"action":"hnsw_vector_cache_prefill","count":321691,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-05-06T17:56:47Z","took":20395413911}

DudaNogueira · May 6, 2024, 6:27pm

That doesn’t help either, those are all after the crash and while restarting and recovering from it.

how is the memory and disk consumption?

Dan_Gordon · May 7, 2024, 12:11pm

Duda,

As always, thank you for your help. I doubled up my droplet to 8GB and 4 vCPUs and it’s working fine.

DudaNogueira · May 7, 2024, 9:20pm

hi @Dan_Gordon ! Glad to hear that, my friend!

We are here to help you succeed

Topic		Replies	Views
Weaviate Batch Errors during Batch Insertion with v4 client Support bug , developer-experience , wcs , python , documentation	11	938	May 15, 2024
gRPC is not available. Please make sure that gRPC is configured correctly in the client and on the server: Weaviate v1.23.10 Support python	3	1435	March 7, 2024
Weaviate Startup Issue - WCS gRPC Connection error Support bug , wcs , typescript , javascript	2	376	July 31, 2024
An error occurred: The 'objects' creation was cancelled because it took longer than the configured timeout of 60s. Try reducing the batch size (currently 1) to a lower value. Aim to on average complete batch request within less than 10s Support bug	1	50	October 15, 2024
Exception: Query call with protocol GRPC batch failed with message recvmsg:Connection reset by peer Support	7	368	October 17, 2024

Crashing 1.24.10

Description

Server Setup Information

Logs

Docker setup

Related topics