I did some further research on this as we encountered the same problem (Some objects not readable after batch import / flush and switch failed - Support - Weaviate Community Forum).
The only code path I was able to find that could lead to this exact error output (other calls to os.remove() / os.removeAll should have more output in the log) is:
weaviate/adapters/repos/db/lsmkv/bucket.go at main · weaviate/weaviate (github.com)
weaviate/adapters/repos/db/lsmkv/memtable_flush.go at main · weaviate/weaviate (github.com)
weaviate/adapters/repos/db/lsmkv/segmentindex/indexes.go at main · weaviate/weaviate (github.com)
So in the end os.removeAll() is called on the scratch directory, which is failing with “directory not empty”.
Doing a search on when that would happen I came up with
os: “os.RemoveAll” sometimes returns error “remove files: directory not empty” · Issue #23452 · golang/go (github.com), which provokes the question: are there any goroutines potentially still accessing the scratch directory while trying to remove it?
Hope this helps a bit in figuring out the root cause of this.