Files
Store Any File. Find It by What's Inside.
HexDB is more than a document database. It stores whole binary files, from JPEGs and WAVs to MP4s, PSDs and PDFs, chunked and named with BLAKE3, encrypted and spread across the vertices. Then it reads the metadata out of each file by its type, so a camera model, a GPS fix, an artist, a duration or a document author becomes a field you can query in GraphQL and SQL.
What Happens to a File
- Stream It In
PUT a file to a folder and name, or send a large one in resumable 8 MB parts. Uploads stream to disk, never held whole in memory, and one sent to a replica streams on to the Overseer. The type comes from the file's first bytes, not its extension, and the file gets a SHA-256.
- Chunk With BLAKE3
Content-defined chunking (FastCDC) cuts the file into pieces of 512 KB to 8 MB, 2 MB on average, so an edit changes only the chunks around it. Each chunk is named by a keyed BLAKE3 hash of its content: identical content is stored once across every file, version and tessellation, and with the key secret, a chunk's name reveals nothing.
- Encrypt and Spread
Each chunk is sealed with AES-256-GCM, then split with the same Reed-Solomon 4+2 code as documents: six fragments, one per vertex, each with its own BLAKE3 hash. Lose or corrupt any two and the chunk is rebuilt and repaired. Put the six vertices on six disks and a file survives losing two of them.
- Extract by File Type
Once the content is fsynced and the document committed, the Overseer reads the file in the background with the extractors its type calls for: EXIF and XMP for a photo, ID3 for an MP3, QuickTime atoms for a MOV, document properties for a PDF. Images, PSDs and H.264 videos get a thumbnail: for a video, a keyframe about a third of the way in, decoded on the server. A file nothing can read is still found by name, folder, type, size, hash and dates.
Put Files In
Over REST, from the hexdb command, from the Node.js, Python and .NET drivers, or by dragging them onto the admin UI's Files page. It browses by folder or by type, previews images, audio and video, and searches as you type, forgiving typos: "tatoo" still finds "tattooed".
# A files tessellation, then a photo into a folder
curl -X POST http://localhost:7700/tessellations \
-H "Content-Type: application/json" -d '{ "name": "media", "kind": "files" }'
curl -X PUT --data-binary @glacier-lagoon.jpg \
"http://localhost:7700/media/_files/2024/Iceland/glacier-lagoon.jpg"
# Or a whole folder from the command line, keeping its structure
hexdb put media ./Pictures -r --to /photos
hexdb ls media --search "glacier lagoon"
hexdb get media 2024/Iceland/glacier-lagoon.jpg -o ~/DownloadsSearch It in GraphQL and SQL
Every file is a document in a files tessellation, so GraphQL, SQL and the REST filters see its metadata like any other fields, with indexes, full-text search, aggregations, roles and row filters applied unchanged. A files tessellation starts with a text index over names, folders, tags and metadata; index a field such as meta.artist to make its filters fast. Queries only touch the documents, so searching a terabyte of video costs what searching its catalog costs.
# Photos taken in Iceland with a GPS position, newest first.
# Variables: { "filter": {
# "content_type": { "$startsWith": "image/" },
# "meta.location.country": "Iceland",
# "meta.gps": { "$exists": true } } }
query Photos($filter: JSON) {
documents(
tessellation: "media"
filter: $filter
sort: [{ field: "meta.taken_at", descending: true }]
limit: 50
) {
total
documents {
id
camera: field(path: "meta.camera.model")
gps: field(path: "meta.gps")
}
}
}-- The ten longest Miles Davis tracks
SELECT name, meta.title, meta.duration_ms FROM media
WHERE meta.artist = 'Miles Davis'
ORDER BY meta.duration_ms DESC LIMIT 10Further reading: FastCDC paper (USENIX ATC '16) · fastcdc for Rust · BLAKE3
What It Reads
- ImagesJPEG, PNG, WebP, HEIC, AVIF, TIFF, camera RAW (CR3, NEF, ARW, DNG, ...), PSD
- EXIF with GPS, IPTC, XMP, PNG text, Photoshop size, layers and color mode, camera, lens and exposure, dimensions
- AudioMP3, AAC and M4A, FLAC, Ogg, Opus, WAV and Broadcast WAV, AIFF, WMA, APE
- ID3v1 and ID3v2, Vorbis comments, APE and MP4 tags, RIFF INFO, bext and iXML from field recorders, duration, bitrate, sample rate, channels
- VideoMP4, MOV, 3GP, MKV, WebM, WMV
- QuickTime metadata (camera, date, location), Matroska tags, MPEG-7 descriptions, duration, size and codec, and a thumbnail from an H.264 keyframe, upright even for portrait phone video
- DocumentsPDF, Illustrator, EPS, Word, Excel, PowerPoint, OpenDocument, EPUB, SVG, fonts
- Document properties and XMP, page and word counts. The start of the text of Office, OpenDocument, Markdown, SVG and plain-text files goes into full-text search
- Everything elseDICOM scans, ZIP, JAR, APK and TAR archives, any other binary
- DICOM tags (never patient names or IDs), archive listings, C2PA content credentials. Anything else is stored and found by name, folder, type, size, hash and tags
- Byte-exact
Images, audio and video, lossless or not, and already-compressed formats (ZIP, Office files, EPUB) are stored exactly as they arrived. Everything else gets Zstandard when it saves at least 5%, and that's lossless too.
- Replicated
Every hex copies the chunks it's missing from the Overseer, sealed in transit, and hexes repair each other: a chunk one hex can't rebuild is fetched from another.
- Versions
Uploading to the same path makes a new version and keeps the last ten. Versions share unchanged chunks, and restoring one makes a new version, so a rollback can itself be undone.
- Trash
Deleting moves a file to the trash for 30 days, where it can be restored, renamed if its old name is taken, or deleted for good.
- Downloads
Range requests, ETags and HEAD work as browsers and media players expect, so audio and video can seek. Stored HTML and SVG can't run scripts in a browser.
- Scrubbed
Every fragment is verified once a day and bad ones are rewritten. /status reports chunks, stored bytes, degraded chunks and repairs per vertex.
- Where it lives
On the hexes' own disks, encrypted and split into six coded fragments, with a full copy on every hex. A hex that's missing a file fetches it from another, and nothing leaves your servers unless you turn on AI tagging.
OCR
Scanned Documents You Can Search by What They Say.
A scanned contract, a photographed receipt or a faxed form is just a picture until something reads it. HexDB reads the text in images and scanned PDFs with Tesseract, the open-source OCR engine, and indexes it with the file, so the words in your scans are as searchable as your documents. It runs next to your data: nothing is sent anywhere.
Listen to the podcast - 5:39 min
OCR: Scans You Can Search by What They Say
How HexDB spots a scan, reads pictures and scanned PDFs with Tesseract, keeps only text that reads like words, and indexes it with the file, so a full-text search finds a scanned letter or invoice by its words. Your file is stored as always, and nothing leaves your lattice.
What Happens to a Scan
- Uploaded
A picture (JPEG, PNG, TIFF, WebP...) or a PDF. Its metadata is read first, as for every file, PDF body text included.
- Scans Found
A PDF with less than 200 characters of text of its own is a scan: HexDB renders its pages, up to 20, at about 200 dpi.
- Read
Tesseract, the open-source OCR engine, reads each picture or page. Text that doesn't read like words, a photo's stray marks, isn't kept.
- Searchable
The text joins the file's metadata, so a full-text search finds a scanned invoice or letter by the words in it, next to everything else you store.
Set It Up
Nothing to do with the Docker image or HexDB Enterprise. Elsewhere, install Tesseract (and ffmpeg, for more thumbnails) and HexDB finds them at startup.
# hexdb.toml: the defaults (the Docker image has both programs)
[files]
tesseract = "auto" # find it on this machine, a path, or "off"
ocr_languages = "eng" # e.g. "eng+deu": each needs its language data
ocr_pages = 20 # pages of a scanned PDF read
ffmpeg = "auto" # HEIC, AVIF, HEVC, VP9 and AV1 thumbnailsOCR and outside tools in the manual · Try it on the playground (media, ocr-samples)
What You Get
- Tesseract
The open-source OCR engine (Apache 2.0), run when it's installed. The Docker image and HexDB Enterprise include it with English; elsewhere HexDB finds it on its own.
- Your Languages
Read English, German, Japanese or any of Tesseract's 100-plus languages, or several at once (eng+deu), with their language data installed.
- Your File, Stored as Always
The upload is kept like any other: encrypted, versioned and replicated. OCR only reads it, from a pipe, so no unencrypted copy is written to disk, and Tesseract runs in HexDB's sandbox with no network.
- Says How It Read
Each file records the engine, its version, the languages and the pages read, and existing files are read again in the background after an upgrade.
- Thumbnails of Almost Anything
PDF first pages, SVG drawings, RAW camera photos and H.264 video out of the box; with ffmpeg (in the Docker image), HEIC and AVIF photos and HEVC, VP9 and AV1 video too.
- Typed and Printed Text
Clean scans of print and typescript read well, faded or low-resolution ones less so. Handwriting isn't read: Tesseract is made for printed text.
Malware scanning
Every Upload Scanned With ClamAV and YARA Before It's Stored.
A database that stores files can spread what's in them. HexDB scans each upload for viruses and malware with two free, open-source engines before it becomes a file, so nothing infected enters your lattice or reaches another hex. No accounts, no keys, nothing sent anywhere: the scanning runs next to your data.
What Happens to an Upload
- Uploaded
The file streams to disk in chunks, as every upload does. Nothing about it is a file yet: no document, nothing on another hex.
- Scanned
ClamAV and HexDB's YARA rules read it, after its content is on disk and before its document commits.
- Clean: Stored
The file becomes a document with its verdict, the engines and when it was scanned, and replicates to every hex.
- A Match: Quarantined
The upload gets 422 naming what matched. The file is kept on the Overseer only, and never reaches another hex.
Set It Up
Without a ClamAV, HexDB scans with its YARA rules alone and says so on the dashboard. Point it at one and both engines check every file.
# hexdb.toml: the defaults, with a ClamAV of your own
[files.scan]
enabled = true # scan every upload
clamd = "tcp://clamav:3310" # "auto" looks on this machine
yara_rules = "/var/lib/hexdb-yara" # more rules, beside HexDB's own
rescan_hours = 24 # rescans when signatures changeWhat Keeps It Safe
- ClamAV
The open-source antivirus engine, with signatures updated through the day. HexDB talks to its daemon over a socket and never links it, so its GPL stays separate. Docker Compose and the Helm chart run one for you.
- YARA Rules, Built In
YARA-X, the Rust rewrite of the YARA pattern-matching engine, is part of HexDB with HexDB's own rules. Add your own, or a community set such as YARA Forge, from a folder.
- Quarantine
Administrators see each flagged file and what matched on the Quarantine page, then release a false positive or delete the file for good. Every action is in the audit trail.
- Rescans as Signatures Change
A file that was clean when it was stored may match a signature published later. When the signatures or rules change, stored files are scanned again in the background.
- Fails Closed
While ClamAV can't answer, uploads wait (503 with Retry-After) instead of going through unscanned. A file too large to scan fully is recorded as not scanned.
- On by Default, Yours to Switch
Scanning is on from the first start, with no accounts or keys to set up. An administrator can turn it off and on for the whole lattice in Settings, without a restart.
