The Symptom: 100GB Gone, No Explanation
Three months into running Open WebUI in production, our monitoring dashboard lit up. The host’s 200GB root volume was at 94% capacity. We’d been uploading PDFs, running RAG queries, deleting chats—standard operations. Nothing unusual.
We SSH’d into the box and ran du -sh /path/to/backend/data/. The output came back: 87GB in data/. That was up from about 8GB when we first deployed.
We started poking around. ls -la data/uploads/ showed 4,200+ files. du -sh data/vector_db/ returned 62GB. We knew we’d uploaded maybe 400–500 documents total. Something wasn’t adding up.
We checked webui.db. The file table had 5,100+ records. We checked the actual uploads/ directory—4,800+ physical files. Somewhere in that gap, things had gone off the rails.
We deleted a few test chats through the UI, watched the chat table shrink, then checked uploads/ again. Nothing changed. The files were still there. The file table still had the records. The vector_db/ directory was untouched.
That’s when we realized: deleting a chat in Open WebUI does not delete the files you uploaded to it.
The Design Tradeoff: Why Open WebUI Doesn’t Auto-Clean
We dug into the Open WebUI codebase and community discussions to understand why. The project maintainers have been cautious about automatic file deletion for good reason:
- Multi‑user isolation: A file uploaded in one user’s chat might be referenced in another user’s knowledge base. Auto‑deleting based solely on chat deletion would break things for other users.
- Knowledge base persistence: Files attached to knowledge bases (
knowledgetable) need to outlive individual chats. The system can’t assume a file is orphaned just because it’s not in any active chat. - Recovery window: Users expect to recover accidentally deleted content. Immediate hard deletes are risky.
The official stance, as reflected in multiple GitHub discussions, has been: if you want cleanup, build an external script. That’s exactly what we did.
The Core Algorithm: Defining “Orphaned”
Before writing a single line of code, we mapped out the logic:
A file is an orphan if:
- It exists in the
filetable (the system knows about it) - It is not referenced in any active chat (the
chattable) - It is not referenced in any knowledge base (the
knowledgetable)
This gives us the deletion candidate set:
orphans = all_file_ids - (chat_referenced_ids ∪ knowledge_referenced_ids)
The Chat Extraction Problem
We initially tried parsing the chat.chat column with json.loads(). That worked for active chats. But when we tested it against a chat we’d deleted through the UI, the JSON was incomplete—it only contained the visible message history, not the metadata about deleted messages.
We ran a quick experiment. We uploaded a file test.pdf to a chat, asked a few questions, then deleted the chat through the UI. We then dumped the raw chat.chat content:
SELECT chat FROM chat WHERE id = '<deleted-chat-id>';
The JSON structure still contained references to "file": {"id": "uuid"} inside message objects that were marked as deleted. But json.loads() on the full payload missed these because they were nested inside "deleted": true message blocks that our parser was skipping.
We switched to regex:
pattern = r'(?<=\"file\": \{\"id\": \")[a-z0-9\-]*(?=\")'
This brute‑force approach catches every file ID reference in the raw JSON string, regardless of nesting or deletion status. We verified it by running it against the raw payload of a deleted chat and confirmed it extracted the orphaned file ID that json.loads() had missed.
The Knowledge Base Extraction
The knowledge table stores its file references in the data column as JSON. The schema looks like this:
{"file_ids": ["uuid-1", "uuid-2", ...]}
We extract these with:
cursor.execute("SELECT data FROM knowledge")
knowledge_ids = [json.loads(knowledge['data']) for knowledge in cursor.fetchall()]
knowledge_ids = list(itertools.chain(*[list(knowledge.values())[0] for knowledge in knowledge_ids]))
We also ran a sanity check: knowledge IDs should never appear in chat data. If they do, something’s deeply wrong with the data model. We added an explicit assertion:
knowledge_in_chat_id = chat_file_ids_set.intersection(knowledge_ids_set)
if knowledge_in_chat_id:
raise ValueError(f"Error: found knowledge base IDs in chat: {knowledge_in_chat_id}")
This caught a data integrity issue in our staging environment once—a buggy migration had cross‑pollinated references. We fixed the migration and re‑ran the script.
The SQLite + ChromaDB Cleanup Script
We wrote files_cleanup.py for the default Open WebUI stack: SQLite for metadata, ChromaDB for vectors.
Full Script (English strings and comments)
import sqlite3
import chromadb
import re
import itertools
import json
import argparse
import os
import pathlib
import shutil
def get_ids(path: str, collections_to_del: list = []) -> list[str]:
"""
Query the internal ChromaDB SQLite to find active segment IDs.
If collections_to_del is provided, exclude segments belonging to those collections.
"""
database = sqlite3.connect(path)
cursor = database.cursor()
query = "SELECT id FROM segments WHERE scope = 'VECTOR'"
params = []
if collections_to_del:
collections_to_del = [f"file-{coll}" for coll in collections_to_del]
coll_placeholders = ','.join('?' for _ in collections_to_del)
coll_ids = cursor.execute(
f"SELECT id FROM collections WHERE name IN ({coll_placeholders})",
collections_to_del
).fetchall()
coll_ids = [id_[0] for id_ in coll_ids]
placeholders = ','.join('?' for _ in coll_ids)
query += f" AND collection NOT IN ({placeholders})"
params.extend(coll_ids)
cursor.execute(query, params)
ids = cursor.fetchall()
return [id[0] for id in ids]
def main():
parser = argparse.ArgumentParser(
description="Clean up orphaned files, DB records, and ChromaDB vectors for Open WebUI"
)
parser.add_argument(
'-db', '--database-path', type=str, required=True,
help='Full path to webui.db'
)
parser.add_argument(
'-b', '--batch-chats', type=int, default=100,
help='Number of chats to fetch per batch (lower for low‑memory environments)'
)
parser.add_argument(
'-l', '--list-files', action='store_true',
help='Only list files that would be deleted, do not actually delete'
)
parser.add_argument(
'--delete-files', action='store_true',
help='Delete physical files from uploads/ directory'
)
parser.add_argument(
'--delete-db-entries', action='store_true',
help='Delete orphaned records from the file table'
)
parser.add_argument(
'--delete-vectors', action='store_true',
help='Delete ChromaDB collections and clean up ghost folders'
)
parser.add_argument(
'--no-confirm', action='store_true',
help='Skip all confirmation prompts (for automated scripts)'
)
args = parser.parse_args()
args.database_path = os.path.normpath(args.database_path)
if not os.path.isfile(args.database_path):
raise ValueError(f'Database path is not a file: {args.database_path}')
if args.no_confirm and not (args.delete_files or args.delete_db_entries or args.delete_vectors):
raise ValueError("--no-confirm requires at least one of --delete-files, --delete-db-entries, or --delete-vectors")
#################################################
# Extract file IDs from knowledge base and chats
#################################################
conn = sqlite3.connect(args.database_path)
conn.row_factory = sqlite3.Row
cursor = conn.cursor()
# Get all file IDs from the file table
cursor.execute("SELECT id FROM file")
webuidb_file_ids = [dict(file_) for file_ in cursor.fetchall()]
webuidb_file_ids = [list(file_ids_.values())[0] for file_ids_ in webuidb_file_ids]
webuidb_file_ids_set = set(webuidb_file_ids)
# Extract file IDs from knowledge base data (JSON)
cursor.execute("SELECT data FROM knowledge")
knowledge_ids = [dict(knowledge) for knowledge in cursor.fetchall()]
knowledge_ids = [json.loads(knowledge['data']) for knowledge in knowledge_ids]
knowledge_ids = list(itertools.chain(*[list(knowledge.values())[0] for knowledge in knowledge_ids]))
knowledge_ids_set = set(knowledge_ids)
# Extract file IDs from chat table using regex (catches deleted messages too)
cursor.execute("SELECT * FROM chat")
chat_file_ids = []
while True:
rows = cursor.fetchmany(args.batch_chats)
if not rows:
break
rows = [dict(chat_entry) for chat_entry in rows]
pattern = r'(?<=\"file\": \{\"id\": \")[a-z0-9\-]*(?=\")'
for chat in rows:
chat_files_id = re.findall(pattern, chat['chat'])
chat_file_ids += list(set(chat_files_id))
chat_file_ids_set = set(chat_file_ids)
# Safety check: knowledge IDs should never appear in chat
knowledge_in_chat_id = chat_file_ids_set.intersection(knowledge_ids_set)
if knowledge_in_chat_id:
raise ValueError(f"Error: found knowledge base IDs in chat: {knowledge_in_chat_id}")
knowledge_and_chat_ids_set = chat_file_ids_set.union(knowledge_ids_set)
ids_to_delete = webuidb_file_ids_set.difference(knowledge_and_chat_ids_set)
print(f"Knowledge base references {len(knowledge_ids_set)} files, chat references {len(chat_file_ids_set)} files, total referenced: {len(knowledge_and_chat_ids_set)}")
print(f"File table has {len(webuidb_file_ids_set)} records total")
#################################################
# Handle physical files in uploads/ directory
#################################################
uploads_dir = os.path.join(os.path.split(args.database_path)[0], 'uploads')
if not os.path.isdir(uploads_dir):
raise ValueError(f"Uploads directory not found: {uploads_dir}")
files = os.listdir(uploads_dir)
files_on_storage_ids = [name.split('_')[0] for name in files] # filename prefix is the UUID
print(f'Uploads directory contains {len(files_on_storage_ids)} files')
files_to_delete = []
unknown_files = []
for file, file_id in zip(files, files_on_storage_ids):
if file_id not in knowledge_and_chat_ids_set:
files_to_delete.append(os.path.join(uploads_dir, file))
if file_id not in ids_to_delete:
unknown_files.append(os.path.join(uploads_dir, file))
print(f"Physical files eligible for deletion: {len(files_to_delete)} (of which {len(unknown_files)} have no file table entry)")
if len(files_to_delete) and args.list_files:
print("The following files will be deleted:")
print(*files_to_delete, sep='\n')
print("")
print(f"Database records eligible for deletion: {len(ids_to_delete)}")
if len(ids_to_delete) and args.list_files:
print("The following Chroma collections will be deleted:")
print(*ids_to_delete, sep='\n')
print("")
#################################################
# Handle ChromaDB vector collections
#################################################
chroma_path = os.path.join(os.path.split(args.database_path)[0], "vector_db")
client = chromadb.PersistentClient(chroma_path)
collections = client.list_collections()
chroma_file_collections = [
collection.replace("file-", "") for collection in collections
if collection.startswith("file-")
]
chroma_entries_set = set(chroma_file_collections)
chroma_entries_to_delete = chroma_entries_set.difference(knowledge_and_chat_ids_set)
print(f"Vector collections eligible for deletion: {len(chroma_entries_to_delete)}")
if len(chroma_entries_to_delete) and args.list_files:
print("The following Chroma collections will be deleted:")
print(*chroma_entries_to_delete, sep='\n')
print("")
#################################################
# Execute deletions (with confirmations)
#################################################
answer = 'no'
if len(chroma_entries_to_delete) and args.delete_vectors:
if not args.no_confirm:
answer = input("Delete orphaned vector collections? yes/[no]: ")
if args.no_confirm or answer.lower() in ["y", "yes"]:
for collection in chroma_entries_to_delete:
coll = client.get_collection(f'file-{collection}')
if ids := coll.get()['ids']:
coll.delete(ids)
del coll
client.delete_collection(name="file-" + collection)
print(f"Deleted {len(chroma_entries_to_delete)} vector collections")
else:
print("Skipping vector deletion")
chroma_entries_to_delete = []
# Additional cleanup: ChromaDB leaves ghost folders on disk
print("Scanning for orphaned vector folders on disk...")
vector_folders = [x for x in os.listdir(chroma_path) if os.path.isdir(os.path.join(chroma_path, x))]
disk_size = sum(file.stat().st_size for file in pathlib.Path(chroma_path).rglob('*')) // 1024**2
print(f"vector_db directory contains {len(vector_folders)} folders, using approximately {disk_size} MB")
# Query chroma.sqlite3 to get active segments
chroma_file_ids_hold = get_ids(
os.path.join(chroma_path, "chroma.sqlite3"),
chroma_entries_to_delete
)
chroma_file_ids_del = set(vector_folders).difference(set(chroma_file_ids_hold))
disk_size_del = sum(
file.stat().st_size
for chroma_vector_folder in chroma_file_ids_del
for file in (pathlib.Path(chroma_path) / chroma_vector_folder).rglob('*')
if file.is_file()
) // 1024**2
print(f"Will delete {len(chroma_file_ids_del)} orphaned folders, freeing approximately {disk_size_del} MB")
answer = 'no'
if len(chroma_file_ids_del) and args.delete_vectors:
if not args.no_confirm:
answer = input("Delete orphaned vector folders on disk? yes/[no]: ")
if args.no_confirm or answer.lower() in ["y", "yes"]:
for chroma_vector_folder in chroma_file_ids_del:
shutil.rmtree(pathlib.Path(chroma_path) / chroma_vector_folder)
print(f"Deleted {len(chroma_file_ids_del)} orphaned folders")
else:
print("Skipping orphaned folder deletion")
answer = 'no'
if len(files_to_delete) and args.delete_files:
if not args.no_confirm:
answer = input("Delete orphaned physical files in uploads/ ? yes/[no]: ")
if args.no_confirm or answer.lower() in ["y", "yes"]:
for file in files_to_delete:
os.remove(file)
print(f"Deleted {len(files_to_delete)} physical files")
else:
print("Skipping physical file deletion")
answer = 'no'
if len(ids_to_delete) and args.delete_db_entries:
if not args.no_confirm:
answer = input("Delete orphaned records from the file table? yes/[no]: ")
if args.no_confirm or answer.lower() in ["y", "yes"]:
placeholders = ", ".join(["?"] * len(ids_to_delete))
cursor.execute(f"DELETE FROM file WHERE id IN ({placeholders})", list(ids_to_delete))
conn.commit()
print(f"Deleted {len(ids_to_delete)} file records")
else:
print("Skipping database record deletion")
if not (args.delete_files or args.delete_db_entries or args.delete_vectors):
print("Dry run mode – nothing was deleted. Add --delete-* flags to actually remove items.")
conn.close()
if __name__ == "__main__":
main()How We Run It
Step 1: Dry run – always first.
python3 files_cleanup.py -db /path/to/backend/data/webui.db --list-files
This lists every orphaned file, database record, and Chroma collection without deleting anything. We’ve had two incidents where this caught misconfigurations before we did any damage.
Step 2: Actual deletion – each action prompts for confirmation.
python3 files_cleanup.py -db /path/to/backend/data/webui.db \
--delete-files \
--delete-db-entries \
--delete-vectors
Add --no-confirm for cron‑based automation.
The ChromaDB “Ghost Directory” Problem
After running the script with --delete-vectors, we checked disk usage again:
du -sh data/vector_db/
Still 62GB. Nothing had changed.
We were confused. We’d called client.delete_collection() for every orphaned collection. ChromaDB confirmed the collections were gone:
>>> client.list_collections() []
But the disk space hadn’t budged.
We started digging into ChromaDB’s internals. The vector_db/ directory contains:
chroma.sqlite3— metadata database- Dozens of UUID‑named folders — each containing HNSW index segments and parquet files
We ran a test. Created a collection, added 1,000 embeddings, noted the disk usage. Deleted the collection via client.delete_collection(). Checked disk usage again. Same size.
We found multiple GitHub issues confirming this behavior. ChromaDB’s delete_collection() removes the collection from its internal metadata but does not delete the underlying segment folders on disk. The physical files remain until something explicitly removes them.
We also noticed another issue: each file uploaded to Open WebUI creates a separate ChromaDB collection with the prefix file-{uuid}. Each collection has a fixed overhead of roughly 100MB regardless of document size. With 4,800 orphaned files, that’s 480GB of theoretical overhead—though in practice, we were seeing about 60GB of actual disk usage from the segment files.
The Fix: Direct ChromaDB Segment Cleanup
We needed to identify which UUID folders in vector_db/ were actually in use. ChromaDB stores this information in chroma.sqlite3 inside the segments table.
We wrote get_ids() (shown in the full script) to query this internal database. It returns active segment IDs, optionally excluding those belonging to collections we’re about to delete.
Then we compare against the actual folders on disk:
vector_folders = [x for x in os.listdir(chroma_path) if os.path.isdir(os.path.join(chroma_path, x))] chroma_file_ids_hold = get_ids(os.path.join(chroma_path, "chroma.sqlite3"), chroma_entries_to_delete) chroma_file_ids_del = set(vector_folders).difference(set(chroma_file_ids_hold))
Any folder in vector_db/ that’s not in the segments table is a ghost—safe to delete.
Critical warning: This operation must be performed with Open WebUI fully stopped. ChromaDB’s PersistentClient is not process‑safe. If another process writes to chroma.sqlite3 while we’re reading it, or if we delete a folder that a running ChromaDB instance expects to exist, data corruption is likely.
We now run the script during a maintenance window:
docker stop open-webui python3 files_cleanup.py -db /path/to/backend/data/webui.db --delete-vectors docker start open-webui
We also tested that on a busy production system, we saw OperationalError: database is locked if we didn’t stop the container first. So stopping is non‑negotiable.
The PostgreSQL Version: For Multi‑User Deployments
One of our team members runs Open WebUI with PostgreSQL as the backend database. He adapted our script to work with PG, adding:
- Streaming chat processing to avoid loading all chats into memory
- ThreadPoolExecutor for parallel deletion (10 workers)
- Memory monitoring via
psutil
His script (postgres_cleanup.py):
import psycopg2
import psycopg2.extras
import chromadb
import json
import argparse
import os
import psutil
import sys
from concurrent.futures import ThreadPoolExecutor
def find_file_ids_in_dict(obj):
"""
Recursively search a dictionary structure for patterns like {"file": {"id": ...}}
and collect all file.id values.
"""
found = []
if isinstance(obj, dict):
for k, v in obj.items():
if k == 'file' and isinstance(v, dict) and 'id' in v:
found.append(v['id'])
else:
found.extend(find_file_ids_in_dict(v))
elif isinstance(obj, list):
for item in obj:
found.extend(find_file_ids_in_dict(item))
return found
def log_memory_usage(step_desc):
process = psutil.Process(os.getpid())
mem_mb = process.memory_info().rss / (1024 * 1024)
print(f"[Memory] {step_desc}: {mem_mb:.2f} MB", file=sys.stderr)
def stream_chats(cursor, batch_size):
cursor.itersize = batch_size
cursor.execute("SELECT chat FROM chat")
for chat_entry in cursor:
yield chat_entry['chat']
def safe_delete(file_path):
try:
os.remove(file_path)
return True, file_path
except Exception as e:
return False, (file_path, e)
def main():
parser = argparse.ArgumentParser()
parser.add_argument('-db', '--database-url', type=str, required=True,
help='PostgreSQL connection string, e.g. postgres://user:pass@host:port/db')
parser.add_argument('--chroma-path', type=str, required=False,
help='Path to ChromaDB directory (if different from default)')
parser.add_argument('-b', '--batch-chats', type=int, default=10,
help='Number of chats to process per batch (lower = less memory)')
parser.add_argument('-l', '--list-files', action='store_true')
parser.add_argument('--delete-files', action='store_true')
parser.add_argument('--delete-db-entries', action='store_true')
parser.add_argument('--delete-vectors', action='store_true')
parser.add_argument('--no-confirm', action='store_true')
parser.add_argument('--log-memory', action='store_true')
args = parser.parse_args()
if args.log_memory:
log_memory_usage("Script started")
conn = psycopg2.connect(args.database_url)
cursor = conn.cursor(cursor_factory=psycopg2.extras.RealDictCursor)
if args.log_memory:
log_memory_usage("Connected to database")
# Get all file IDs from the file table
cursor.execute("SELECT id FROM file")
file_ids_set = {row['id'] for row in cursor.fetchall()}
if args.log_memory:
log_memory_usage(f"Fetched {len(file_ids_set)} file IDs")
# Extract knowledge base file IDs
cursor.execute("SELECT data FROM knowledge")
knowledge_ids_set = set()
for row in cursor.fetchall():
kdata = row['data']
if isinstance(kdata, str):
try:
kdata = json.loads(kdata)
except Exception:
continue
if isinstance(kdata, dict) and kdata:
ids = list(kdata.values())[0]
if isinstance(ids, list):
knowledge_ids_set.update(ids)
if args.log_memory:
log_memory_usage(f"Extracted {len(knowledge_ids_set)} knowledge base file IDs")
# Stream chats and extract file IDs
chat_file_ids_set = set()
processed = 0
for chat_content in stream_chats(cursor, args.batch_chats):
if isinstance(chat_content, str):
try:
chat_content = json.loads(chat_content)
except Exception:
continue
ids = find_file_ids_in_dict(chat_content)
chat_file_ids_set.update(ids)
processed += 1
if args.log_memory and processed % 100 == 0:
log_memory_usage(f"Processed {processed} chats")
if args.log_memory:
log_memory_usage(f"Processed {processed} total chats")
# Sanity check
overlap = chat_file_ids_set.intersection(knowledge_ids_set)
if overlap:
raise ValueError(f"Found knowledge base IDs in chat: {overlap}")
referenced = chat_file_ids_set.union(knowledge_ids_set)
ids_to_delete = file_ids_set - referenced
print(f"Knowledge base references {len(knowledge_ids_set)} files, chat references {len(chat_file_ids_set)}, total referenced: {len(referenced)}")
print(f"File table has {len(file_ids_set)} records, {len(ids_to_delete)} are orphaned")
# Locate uploads directory
if args.chroma_path:
root_dir = os.path.abspath(os.path.join(args.chroma_path, ".."))
else:
root_dir = os.path.dirname(os.path.abspath(__file__))
uploads_dir = os.path.join(root_dir, 'uploads')
if not os.path.isdir(uploads_dir):
print(f"Warning: uploads directory not found at {uploads_dir}, skipping physical file cleanup")
files_to_delete = []
else:
files = os.listdir(uploads_dir)
files_on_storage_ids = [name.split('_')[0] for name in files]
print(f'Uploads directory contains {len(files_on_storage_ids)} files')
files_to_delete = [
os.path.join(uploads_dir, f) for f, fid in zip(files, files_on_storage_ids)
if fid not in referenced
]
unknown = [f for f, fid in zip(files, files_on_storage_ids) if fid not in referenced and fid not in ids_to_delete]
print(f"Physical files eligible for deletion: {len(files_to_delete)} (of which {len(unknown)} have no file table entry)")
if args.list_files and files_to_delete:
print("The following physical files will be deleted:")
print(*files_to_delete, sep='\n')
# Chroma collections
chroma_path = args.chroma_path or os.path.join(root_dir, "vector_db")
client = chromadb.PersistentClient(chroma_path)
chroma_collections = [
c.name.replace("file-", "") for c in client.list_collections()
if c.name.startswith("file-")
]
chroma_to_delete = set(chroma_collections) - referenced
print(f"Vector collections eligible for deletion: {len(chroma_to_delete)}")
if args.list_files and chroma_to_delete:
print("The following Chroma collections will be deleted:")
print(*chroma_to_delete, sep='\n')
def confirm(prompt):
if args.no_confirm:
return True
return input(prompt).lower() in ('y', 'yes')
# Delete vector collections in parallel
if chroma_to_delete and args.delete_vectors:
if confirm("Delete orphaned vector collections? yes/[no]: "):
def delete_one(coll_name):
try:
client.delete_collection(name="file-" + coll_name)
return True, coll_name
except Exception as e:
return False, (coll_name, e)
deleted = failed = 0
with ThreadPoolExecutor(max_workers=10) as ex:
futures = [ex.submit(delete_one, c) for c in chroma_to_delete]
for i, f in enumerate(futures, 1):
ok, info = f.result()
if ok:
deleted += 1
else:
failed += 1
print(f"Failed to delete collection {info[0]}: {info[1]}")
if i % 50 == 0 or i == len(chroma_to_delete):
print(f"Deleted {deleted}/{len(chroma_to_delete)}, failed {failed}")
print(f"Vector deletion complete: {deleted} succeeded, {failed} failed")
else:
print("Skipping vector deletion")
# Delete physical files in parallel
if files_to_delete and args.delete_files:
if confirm("Delete orphaned physical files? yes/[no]: "):
deleted = failed = 0
with ThreadPoolExecutor(max_workers=10) as ex:
results = ex.map(safe_delete, files_to_delete)
for i, (ok, info) in enumerate(results, 1):
if ok:
deleted += 1
else:
failed += 1
print(f"Failed to delete {info[0]}: {info[1]}")
if i % 50 == 0 or i == len(files_to_delete):
print(f"Deleted {deleted} files, failed {failed}")
print(f"Physical file deletion complete: {deleted} succeeded, {failed} failed")
else:
print("Skipping physical file deletion")
# Delete database records in batches
if ids_to_delete and args.delete_db_entries:
if confirm("Delete orphaned file table records? yes/[no]: "):
ids_list = list(ids_to_delete)
chunk_size = 50
for i in range(0, len(ids_list), chunk_size):
chunk = ids_list[i:i+chunk_size]
placeholders = ','.join(['%s'] * len(chunk))
cursor.execute(f"DELETE FROM file WHERE id IN ({placeholders})", chunk)
conn.commit()
print(f"Deleted batch {i//chunk_size + 1}, {len(chunk)} records")
print(f"Database record deletion complete: {len(ids_to_delete)} records deleted")
else:
print("Skipping database record deletion")
cursor.close()
conn.close()
if args.log_memory:
log_memory_usage("Script finished")
if __name__ == "__main__":
main()The PG version processes 10 chats per batch (configurable with -b) and logs memory usage with --log-memory. When we ran it against our staging PG instance with 50,000 chat records, memory stayed under 200MB throughout.
Docker‑Specific Considerations
If you’re running Open WebUI in Docker:
- Find the volume path where data is mounted:
docker inspect <container-name> | grep -A 5 Mounts
Look for the Source field pointing to the host directory (e.g., /var/lib/docker/volumes/.../_data).
- Stop the container before running any deletion that touches ChromaDB:
docker stop open-webui
- Run the script from the host (or inside a temporary container) pointing to the actual
webui.dbpath on the host. - ChromaDB version mismatch: We had to ensure the Python environment used to run the script had the same ChromaDB version as the container. We checked the container’s version with:
docker exec open-webui pip list | grep chromadb
Then created a virtual environment on the host with that exact version:
python3 -m venv venv source venv/bin/activate pip install chromadb==<version-from-container>
This prevented chromadb client‑side errors like ValueError: Collection ... does not exist even though it clearly did.
What About the Official Feature?
We later discovered that Open WebUI added an admin setting to “delete associated files and vectors when deleting chats” in recent versions. If you’re on the latest release, you can simply toggle that in the UI.
However, we still keep our script because:
- It handles historical orphaned data that accumulated before we enabled that setting.
- It provides a dry‑run mode—we can preview what will be deleted before committing.
- It cleans up ChromaDB ghost folders even after the official setting is enabled (the official feature also calls
delete_collection, which still leaves the physical segment folders behind).
Summary & Best Practices
We solved the silent storage bloat problem by building a cleanup script that:
- Identifies orphaned files using a set‑difference algorithm
- Physically deletes files, DB records, and Chroma collections
- Goes the extra mile to remove ChromaDB’s leftover segment folders
Our recommended checklist for a safe cleanup run:
- Backup the entire
backend/data/directory. We usetar -czf backup-$(date +%Y%m%d).tgz data/. - Stop Open WebUI (especially if using ChromaDB).
- Run the script without
--delete-*to list orphans:--list-files. - Review the output carefully—check that no active knowledge‑base files are listed.
- Run with the actual delete flags, one category at a time if you want extra safety.
- Start the container again and monitor disk usage.
We now run this script weekly via cron (with --no-confirm and all delete flags) after ensuring the container is stopped during the maintenance window. Our disk usage has stabilised at ~20GB for over two months now.
