Forem: Mukhtar

Audit Your File System with Surveilr: A Practical Guide

Mukhtar — Thu, 14 May 2026 20:57:43 +0000

How to scan your file system and query files using SQL

TL;DR: In this guide, you'll learn how to use surveilr to scan directories, capture file metadata in a SQLite database, and query for files using SQL—find PDFs, track document changes, and audit file system activity.

Prerequisites: surveilr installed

The Problem: Finding Files is Hard

If you've ever needed to track files across your system for compliance audits or project management, you know the pain:

Scattered files — Documents, spreadsheets, and PDFs across multiple directories
Manual searches — Using find and Finder/Explorer is tedious and limited
No audit trail — No record of what existed when
Point-in-time only — No way to track file changes over time

What if you could turn your entire file system into a queryable SQL database where you can find any file with SQL queries?

That's exactly what surveilr does.

What is surveilr?

surveilr is an edge-based, SQL-first surveillance platform that turns your file system into a Resource Surveillance State Database (RSSD)—a SQLite database containing all your file metadata.

Think of it as "Git for your entire file system's metadata" that you can query with SQL.

Key features:

📂 Scans directories and captures file metadata (size, timestamps, hashes)
🔍 Query files with SQL (no custom DSL to learn)
🔄 Watch mode for continuous monitoring
🔒 Works offline (edge-based, no cloud required)
📊 Perfect for compliance audits (HIPAA, GDPR, SOC 2)

Let's see it in action.

Step 1: Create Your First RSSD

Initialize a new surveillance database:

surveilr admin init -d my-audit.db

What just happened?
surveilr created my-audit.db—a SQLite database that will store all your file evidence. This is your Resource Surveillance State Database (RSSD).

💡 Pro tip: The RSSD is a standard SQLite database. Once created, it's independent of surveilr and can be queried by any tool that supports SQLite.

Step 2: Ingest Files from a Directory

Scan your Documents folder (or any directory):

# Scan your Documents folder
surveilr ingest files -r ~/Documents -d my-audit.db

What's happening behind the scenes:

surveilr walks the directory tree recursively
Computes file hashes (SHA-256) for content tracking
Captures metadata: size, timestamps, permissions, extensions
Stores everything in SQLite tables

Step 3: Find Sensitive Files with SQL

Now comes the fun part. Open the interactive SQL shell:

surveilr shell -d my-audit.db

📌 Note: File metadata lives in two joined tables:

uniform_resource — Core resource metadata (hash, size, timestamps)

ur_ingest_session_fs_path_entry — File-specific metadata (path, name, extension)

Query 1: Find PDFs Modified Recently

Find all PDF files modified in the last 30 days:

SELECT
    e.file_path_abs,
    e.file_basename,
    u.size_bytes / 1024 AS size_kb,
    u.last_modified_at
FROM uniform_resource u
JOIN ur_ingest_session_fs_path_entry e
  ON u.uniform_resource_id = e.uniform_resource_id
WHERE e.file_extn = 'pdf'
  AND u.last_modified_at > datetime('now', '-30 days')
ORDER BY u.last_modified_at DESC;

Example output:

/Users/you/Documents/report-2024.pdf    | report-2024.pdf    | 2048 | 2024-05-12
/Users/you/Projects/proposal.pdf        | proposal.pdf       | 3248 | 2024-05-10
/Users/you/Desktop/invoice-may.pdf      | invoice-may.pdf    | 512  | 2024-05-08

Query 2: Find All Excel Files

Find all Excel spreadsheets across your system:

SELECT
    e.file_path_abs,
    e.file_basename,
    u.size_bytes / 1024 AS size_kb,
    u.last_modified_at
FROM uniform_resource u
JOIN ur_ingest_session_fs_path_entry e
  ON u.uniform_resource_id = e.uniform_resource_id
WHERE e.file_extn IN ('xlsx', 'xls', 'csv')
ORDER BY u.last_modified_at DESC
LIMIT 20;

Query 3: Find Word Documents by Name

Search for documents by filename:

SELECT
    e.file_path_abs,
    e.file_basename,
    u.last_modified_at
FROM uniform_resource u
JOIN ur_ingest_session_fs_path_entry e
  ON u.uniform_resource_id = e.uniform_resource_id
WHERE e.file_extn IN ('docx', 'doc')
  AND e.file_basename LIKE '%report%'
ORDER BY u.last_modified_at DESC;

Step 4: Export Your Findings

Export results as JSON for reports:

surveilr shell -d my-audit.db --cmd "
SELECT e.file_path_abs, e.file_basename, u.size_bytes, u.last_modified_at
FROM uniform_resource u
JOIN ur_ingest_session_fs_path_entry e
  ON u.uniform_resource_id = e.uniform_resource_id
WHERE e.file_extn = 'pdf'
" --output json > pdf-files-report.json

Or save as a SQL file for reuse:

cat > find-pdfs.sql <<'EOF'
SELECT e.file_path_abs, e.file_basename, u.size_bytes
FROM uniform_resource u
JOIN ur_ingest_session_fs_path_entry e
  ON u.uniform_resource_id = e.uniform_resource_id
WHERE e.file_extn = 'pdf'
EOF

surveilr shell -d my-audit.db find-pdfs.sql --output json > pdfs.json

Step 5: Automate with Watch Mode

Set up continuous monitoring so surveilr automatically re-scans when files change:

surveilr ingest files -r ~/Documents -d my-audit.db --watch

What happens:

surveilr monitors the directory for changes
New or modified files are automatically ingested
Press Ctrl+C to stop

Real-World Use Cases

🏥 HIPAA Compliance

Track PHI locations — Document where patient health information files are stored for GDPR/HIPAA audits.

🌍 GDPR Data Governance

Map PII locations — Track where personally identifiable information is stored across your organization.

🚨 Incident Response

Detect tampering — Establish file baselines and detect unauthorized modifications during security incidents.

Troubleshooting

Problem: "Permission denied" errors when scanning
Solution: Run with appropriate permissions or scan a different directory

Problem: RSSD file getting too large
Solution: Use --dry-run first to check file counts:

surveilr ingest files -r ~/Documents --dry-run

What's Next?

You've just created your first compliance evidence database! Here's what to explore next:

Email Compliance Tracking — Track email communications via IMAP
GitHub/GitLab/Jira Tracking — Project data with Singer taps
surveilr.com — Full documentation and advanced features

Learn More

surveilr.com — Official website and documentation
Installation Guide — Download and install surveilr

Key Takeaways

✅ surveilr turns your file system into a queryable SQL database
✅ Standard SQLite = no vendor lock-in, works with any SQLite tool
✅ Perfect for compliance audits (HIPAA, GDPR, SOC 2)
✅ Continuous monitoring with watch mode
✅ All data stays on your machine (edge-based, no cloud required)

Result: A complete audit trail of your file system that you can query anytime

Found this helpful? Visit surveilr.com to learn more! Have questions? Open an issue.

Query Everything with SQL: The Ultimate Compliance and Security Toolkit

Mukhtar — Wed, 13 May 2026 16:47:24 +0000

Three practical guides to turn your files, emails, and project data into queryable SQLite databases

If you've ever struggled with compliance audits (HIPAA, SOX, GDPR, SOC 2), security investigations, or e-discovery requests, you know the pain:

Data is everywhere — Files on disks, emails in inboxes, issues in GitHub/Jira
No unified view — Each system has its own limited search interface
Manual exports — Hours spent copying data into spreadsheets for auditors
No audit trail — Can't prove what data existed at a specific time
Point-in-time only — No historical tracking of changes

What if you could turn all of this into queryable SQL databases and ask questions like:

-- Find all PDFs modified in the last 30 days
SELECT * FROM files WHERE extension = 'pdf' AND modified > date('now', '-30 days');

-- Show emails from vendors about invoices
SELECT * FROM emails WHERE sender LIKE '%vendor%' AND subject LIKE '%invoice%';

-- Track all code changes for SOC 2 audit
SELECT * FROM commits WHERE date > '2024-01-01' ORDER BY date DESC;

That's what surveilr enables.

What is surveilr?

surveilr is an edge-based, SQL-first surveillance platform that turns your files, emails, and external APIs into Resource Surveillance State Databases (RSSDs)—SQLite databases you can query with standard SQL.

Key Features

📂 File system scanning — Turn directories into queryable metadata
📧 Email ingestion — Connect to Gmail/Outlook via IMAP
🔌 Singer tap integration — Extract data from 600+ sources (GitHub, Jira, Salesforce, etc.)
🔍 Standard SQL — No custom query language to learn
🔒 Edge-based — All data stays on your machine (no cloud required)
⚖️ Compliance-ready — Perfect for HIPAA, SOX, GDPR, SOC 2

Why SQLite?

surveilr uses SQLite—the world's most deployed database:

✅ Zero dependencies — No servers, no installation, no configuration
✅ One file — Your entire database is a single .db file
✅ Portable — Works everywhere (macOS, Linux, Windows, mobile)
✅ No vendor lock-in — Standard SQLite works with 1000s of tools

Three Practical Guides

Guide 1: Audit Your File System

Read the full guide →

Learn how to scan your file system and query files with SQL.

Perfect for: Compliance audits, file tracking, document management

Guide 2: Email Compliance Tracking

Read the full guide →

Learn how to ingest Gmail/Outlook emails via IMAP and query them for compliance reporting.

Perfect for: HIPAA compliance, SOX compliance, e-discovery, communication audits

Guide 3: GitHub/GitLab/Jira Tracking

Read the full guide →

Learn how to use Singer taps to extract project data and query it for SOC 2 and change management audits.

Perfect for: SOC 2 audits, DevOps metrics, security tracking, audit trails

Which Guide Should You Start With?

Security Professionals → Start with Guide 1 - File Auditing
Compliance Officers → Start with Guide 2 - Email Tracking
DevOps Teams → Start with Guide 3 - Project Tracking

Why surveilr vs. Other Tools?

vs. Commercial Compliance Platforms (Vanta, Drata, Secureframe)

Feature	surveilr	Commercial Tools
Cost	Available	$10,000-$50,000/year
Data location	Your machine (edge)	Their cloud
Query language	Standard SQL	Proprietary UIs
Extensibility	Unlimited (SQLite + Singer)	Limited integrations
Vendor lock-in	None (standard SQLite)	Complete lock-in

vs. Custom Scripts

Feature	surveilr	Custom Scripts
Setup	Minutes	Days/weeks
Maintenance	Automatic	Constant fixes
Audit trail	Built-in	You build it
Query language	SQL	grep/awk/jq
Historical tracking	Automatic	You build it

Installation

macOS / Linux

brew tap surveilr/tap && brew install surveilr

Verify Installation

surveilr --version

Get Started

Pick one of the three guides above and start querying your data with SQL.

Each guide is self-contained and works independently—start with whichever solves your biggest problem.

Community & Support

surveilr.com — Official website and documentation
Installation Guide — Download and install surveilr
Report Issues — Bug reports and feature requests

Key Takeaways

✅ Turn files, emails, and APIs into queryable SQL databases
✅ Perfect for HIPAA, SOX, GDPR, SOC 2 compliance
✅ Standard SQLite = no vendor lock-in
✅ Edge-based = your data never leaves your machine

Found this helpful? Visit surveilr.com to learn more and share with your team!

Implementing OpenDAL with Filesystem (FS) In Rust

Mukhtar — Sat, 19 Jul 2025 18:55:47 +0000

Introduction to OpenDAL with SQLite Virtual Tables

OpenDAL is a powerful and unified data access layer that provides an abstraction for different storage backends such as local filesystems, cloud storage, and object stores. It simplifies file and metadata operations by offering a unified API, allowing seamless interaction with different storage solutions.

This guide explains the concepts behind integrating OpenDAL with SQLite virtual tables, allowing you to query filesystem metadata using SQL. The code examples demonstrate key concepts rather than complete implementations.

Core Concepts

1. OpenDAL Operator

The foundation of any OpenDAL integration is the Operator - your interface to the storage backend.

Concept: Create a configured operator for your storage type

// Conceptual example - actual implementation needs error handling
let fs_builder = Fs::default().root("/");
let operator = Operator::new(fs_builder)?.finish();

Key Ideas:

The operator abstracts away storage-specific details
Different services (Fs, S3, Azure, etc.) use the same Operator interface
Configuration happens through service-specific builders

2. SQLite Virtual Table Architecture

Virtual tables in SQLite allow you to present non-SQL data as queryable tables.

Concept: Bridge between OpenDAL and SQLite

// Conceptual structure - real implementation much more complex
struct FileSystemTable {
    base: sqlite3_vtab,           // Required SQLite base
    operator: Rc<Operator>,       // Your OpenDAL operator
}

Key Ideas:

Virtual tables implement a specific SQLite interface
They translate SQL queries into storage operations
The table schema defines what data is queryable

3. Schema Design

Design your virtual table schema to expose useful file metadata.

Concept: Map file attributes to SQL columns

CREATE TABLE filesystem_view(
    name TEXT,                    -- File name
    path TEXT,                    -- Full path
    last_modified TEXT,           -- When file was changed
    content BLOB,                 -- File contents (lazy-loaded)
    size INTEGER,                 -- File size in bytes
    content_type TEXT,            -- MIME type
    digest TEXT,                  -- File hash/checksum
    arg_path HIDDEN              -- Query parameter (hidden column)
);

Key Ideas:

Hidden columns can accept query parameters
BLOB columns can hold binary data
Schema should balance utility with performance

4. Query Translation Flow

The magic happens in translating SQL queries into OpenDAL operations.

Concept: SQL WHERE clause becomes OpenDAL list operations

// Conceptual flow - real implementation involves cursor management
fn handle_query_with_path_filter(path: &str) -> Vec<FileMetadata> {
    // 1. Use OpenDAL to list files in the specified path
    let lister = operator.lister_with(path)
        .metakey(Metakey::ContentLength | Metakey::LastModified)
        .call()?;

    // 2. Convert OpenDAL entries to your table format
    let mut results = Vec::new();
    for entry in lister {
        if entry.metadata().mode() == EntryMode::FILE {
            results.push(convert_to_table_row(entry));
        }
    }

    results
}

Key Ideas:

SQL filters become OpenDAL query parameters
Metadata is fetched on-demand to avoid performance issues
Results are converted to SQLite-compatible format

5. Lazy Loading Strategy

For performance, expensive operations (like reading file contents) should be lazy.

Concept: Only load data when specifically requested

// Conceptual approach to lazy loading
fn get_column_value(&self, column: i32) -> SqliteValue {
    match column {
        0 => SqliteValue::Text(self.file_name.clone()),
        1 => SqliteValue::Text(self.file_path.clone()),
        2 => SqliteValue::Integer(self.file_size),
        3 => {
            // Expensive operation - only do when column is requested
            let content = self.operator.read(&self.file_path)?;
            SqliteValue::Blob(content)
        }
        _ => SqliteValue::Null,
    }
}

Key Ideas:

Separate metadata operations from content operations
Use OpenDAL's stat() for metadata, read() for content
Only perform expensive operations when columns are actually selected

6. Error Handling Patterns

Bridge OpenDAL errors to SQLite errors appropriately.

Concept: Convert storage errors to SQL errors

// Conceptual error mapping
fn map_opendal_error(err: OpenDalError) -> SqliteError {
    match err.kind() {
        ErrorKind::NotFound => SqliteError::NotFound,
        ErrorKind::PermissionDenied => SqliteError::Auth,
        ErrorKind::Unexpected => SqliteError::Internal,
        _ => SqliteError::Internal,
    }
}

Key Ideas:

OpenDAL errors need translation to SQLite error codes
Some errors should be handled gracefully (missing files)
Others should propagate up to the SQL layer

Usage Patterns

Once implemented, your virtual table enables powerful queries:

-- List all files in a directory
SELECT name, size FROM filesystem_view WHERE arg_path = '/documents';

-- Find large files
SELECT path, size FROM filesystem_view 
WHERE arg_path = '/media' AND size > 1000000;

-- Search by file type (if you implement content_type detection)
SELECT name, path FROM filesystem_view 
WHERE arg_path = '/projects' AND content_type LIKE 'text/%';

Benefits of This Approach

Why combine OpenDAL with SQLite virtual tables?

Unified Query Interface: Use SQL to query any storage backend
Flexibility: Same queries work across local files, S3, Azure, etc.
Integration: Easy to embed in existing SQLite-based applications
Performance: SQLite's query optimization benefits your file operations
Ecosystem: Leverage SQLite's rich ecosystem of tools and extensions

Next Steps

To implement this approach:

Study SQLite's virtual table interface documentation
Explore OpenDAL's service implementations for your storage needs
Design your schema based on your specific use cases
Implement the virtual table interface with proper error handling
Add performance optimizations like caching and pagination

This pattern opens up powerful possibilities for unified data access across different storage systems while maintaining the familiar SQL interface.

Conclusion

This guide demonstrated how to use OpenDAL for filesystem operations in Rust and integrate it with SQLite virtual tables. By leveraging OpenDAL, you can seamlessly interact with local storage while maintaining a structured query interface via SQLite.

For more details, check out OpenDAL's documentation.

Getting Started with Rust in 2025: Why Now Is a Great Time to Learn Rust 🦀

Mukhtar — Sun, 13 Jul 2025 20:48:12 +0000

Getting Started with Rust in 2025 🦀

Rust has been growing steadily over the years, and in 2025, it's more relevant than ever. If you've been on the fence about learning it, now's the perfect time to jump in.

🧠 Why Learn Rust?

Rust is a modern systems programming language focused on:

Performance (like C/C++)
Safety (no nulls, no data races)
Modern developer experience (great tooling, compiler messages)

Companies like Microsoft, Amazon, Cloudflare, and Dropbox use Rust in production. It's built to help you write fast, reliable, and maintainable software.

🌟 What Makes Rust Special?

Memory safety without garbage collection
Fearless concurrency – build multi-threaded apps without race conditions
Helpful compiler – the compiler actually teaches you!
Powerful tooling – cargo, clippy, rustfmt, and more
Zero-cost abstractions – safety and performance, no tradeoffs

If you've dealt with C/C++ bugs or hit performance walls in Python or JavaScript, Rust is the best of both worlds.

🆚 Rust vs Other Languages

Rust sits in a unique position compared to other programming languages:

Performance – As fast as C/C++ but much safer
Memory Management – Manual control without the crashes
Type Safety – Catches bugs at compile time, not runtime
Concurrency – Built-in support for safe multi-threading
Learning Curve – Steeper initially, but pays off long-term

Unlike garbage-collected languages, Rust gives you control. Unlike manual memory management languages, Rust prevents common bugs.

🚀 Code Examples

Hello World

Every Rust program starts with a main() function. This is your entry point!

// This is the main function - where your program starts
fn main() {
    // println! is a macro that prints to the console
    println!("Hello, world!");
}

Variables and Types

Rust is statically typed but can often infer types. Variables are immutable by default - you need mut to change them.

fn main() {
    // Variables are immutable by default
    let name = "Alice";           // string slice (&str)
    let age: u32 = 30;           // unsigned 32-bit integer
    let mut score = 100;         // 'mut' makes it mutable (changeable)

    // We can change mutable variables
    score += 50;

    // {} are placeholders for variables in println!
    println!("{} is {} years old with a score of {}", name, age, score);
}

Functions

Functions in Rust are defined with fn. The last expression (without semicolon) is the return value.

// Function that takes two i32 parameters and returns an i32
fn add(a: i32, b: i32) -> i32 {
    a + b  // no semicolon = this is the return value
}

fn main() {
    // Call the function and store the result
    let result = add(5, 3);
    println!("5 + 3 = {}", result);
}

Ownership (Rust's Superpower)

Ownership is Rust's way of managing memory safely. Each value has one owner, and when the owner goes out of scope, the value is dropped.

fn main() {
    // Create a String on the heap
    let s1 = String::from("hello");

    // This MOVES s1 to s2. s1 is no longer valid!
    let s2 = s1;

    // This would cause a compile error because s1 was moved
    // println!("{}", s1); 

    // Only s2 is valid now
    println!("{}", s2);
}

Borrowing

Borrowing lets you use a value without taking ownership. Think of it like borrowing a book - you can read it, but you have to give it back!

// This function borrows a String reference (&String)
// It doesn't take ownership, just looks at the data
fn print_length(s: &String) {
    println!("Length: {}", s.len());
}

fn main() {
    let my_string = String::from("hello");

    // Pass a reference (&) to the function
    print_length(&my_string);

    // We can still use my_string because we only borrowed it
    println!("{}", my_string);
}

Structs & Methods

Structs are like classes in other languages - they group related data together. You can add methods to them using impl blocks.

// Define a struct (like a class in other languages)
struct Person {
    name: String,
    age: u32,
}

// Implementation block - where we define methods
impl Person {
    // Associated function (like a constructor)
    fn new(name: String, age: u32) -> Person {
        Person { name, age }  // shorthand for name: name, age: age
    }

    // Method that takes &self (borrows the instance)
    fn greet(&self) {
        println!("Hi, I'm {}", self.name);
    }
}

fn main() {
    // Create a new Person instance
    let person = Person::new("Bob".to_string(), 25);

    // Call the method
    person.greet();
}

Error Handling

Rust doesn't have exceptions. Instead, it uses Result to handle errors explicitly. This forces you to deal with potential failures.

// Function returns Result<T, E> - either Ok(value) or Err(error)
fn divide(a: f64, b: f64) -> Result<f64, String> {
    if b == 0.0 {
        // Return an error
        Err("Cannot divide by zero".to_string())
    } else {
        // Return the successful result
        Ok(a / b)
    }
}

fn main() {
    // Use match to handle both success and error cases
    match divide(10.0, 2.0) {
        Ok(result) => println!("Result: {}", result),
        Err(error) => println!("Error: {}", error),
    }
}

🛠️ Getting Started

Install Rust: curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
Create new project: cargo new my_project
Build and run: cargo run
Read "The Rust Book" online for free

🔥 What You Can Build

Web servers (Actix, Rocket frameworks)
Command-line tools (ripgrep, bat)
Game engines (Bevy)
Blockchain projects (Solana, Polkadot)
WebAssembly applications
Operating systems (Redox OS)

💡 Why 2025 is Perfect

Mature ecosystem – tons of quality crates
Industry adoption – major companies using it in production
Great learning resources – excellent documentation and community
Job market – high demand, good salaries
Future-proof – language designed for next-generation computing

Rust's combination of performance, safety, and growing ecosystem makes it an excellent choice for modern development in 2025!