Forem: CØDE N!NJΔ

DataFuse is a declarative .NET framework that aggregates data from SQL databases, REST APIs, MongoDB, and Entity Framework into unified, strongly-typed objects — replacing hundreds of lines of manual orchestration code with a clean, schema-driven configuration.

Real-World Use Cases
Why DataFuse?
Quick Start (5 Minutes)
Core Concepts
Packages
Query Implementation Guide
Transformer Guide
Advanced Features
Architecture
Best Practices
Migrating from Schemio to DataFuse
Getting Support

Real-World Use Cases

1. E-Commerce Product Page

Your product page needs data from three different systems: inventory from a SQL Server database, live pricing from a pricing microservice REST API, and customer reviews from MongoDB.

Without DataFuse — you write this every time:

// 50+ lines of manual orchestration, repeated across your codebase
public async Task<ProductPage> GetProductPage(int productId)
{
    var product = await _db.QueryFirstAsync<ProductRecord>(
        "SELECT * FROM Products WHERE Id = @Id", new { Id = productId });

    var pricing = await _httpClient.GetFromJsonAsync<PricingResponse>(
        $"https://pricing-api/products/{productId}");

    var reviews = await _mongoDb.GetCollection<ReviewRecord>("reviews")
        .Find(r => r.ProductId == productId).ToListAsync();

    // Manual assembly — error-prone, no caching, no parallel execution
    return new ProductPage
    {
        ProductId = product.Id,
        Name = product.Name,
        Description = product.Description,
        CurrentPrice = pricing.Price,
        DiscountPercent = pricing.Discount,
        Reviews = reviews.Select(r => new Review
        {
            Rating = r.Stars,
            Comment = r.Text,
            Author = r.AuthorName
        }).ToArray(),
        AverageRating = reviews.Average(r => r.Stars)
    };
}

With DataFuse — declare your schema once, use everywhere:

public class ProductPageConfiguration : EntityConfiguration<ProductPage>
{
    public override IEnumerable<Mapping<ProductPage, IQueryResult>> GetSchema()
    {
        return CreateSchema.For<ProductPage>()
            .Map<ProductQuery, ProductTransform>(For.Paths("product"),
                product => product.Dependents
                    .Map<PricingApiQuery, PricingTransform>(For.Paths("product/pricing"))
                    .Map<ReviewsMongoQuery, ReviewsTransform>(For.Paths("product/reviews")))
            .End();
    }
}

// Usage — one line to get fully hydrated data
var productPage = dataProvider.GetData(new ProductRequest { ProductId = 42 });

DataFuse automatically:

Executes the SQL product query first
Runs pricing and reviews API calls in parallel (they're siblings)
Passes the product result to dependent queries so they know which product to fetch
Assembles the final ProductPage object via type-safe transformers

2. Customer 360 Dashboard

Build a unified customer view pulling from a CRM database, a billing REST API, and a support ticket system:

public class Customer360Configuration : EntityConfiguration<Customer360>
{
    public override IEnumerable<Mapping<Customer360, IQueryResult>> GetSchema()
    {
        return CreateSchema.For<Customer360>()
            .Map<CustomerQuery, CustomerTransform>(For.Paths("customer"),
                customer => customer.Dependents
                    .Map<BillingApiQuery, BillingTransform>(For.Paths("customer/billing"))
                    .Map<TicketsApiQuery, TicketsTransform>(For.Paths("customer/tickets"))
                    .Map<OrdersQuery, OrdersTransform>(For.Paths("customer/orders"),
                        orders => orders.Dependents
                            .Map<OrderItemsQuery, OrderItemsTransform>(
                                For.Paths("customer/orders/order/items"))))
            .End();
    }
}

Need just the basic profile without orders? Use selective loading:

var basicProfile = dataProvider.GetData(new CustomerRequest
{
    CustomerId = 123,
    SchemaPaths = new[] { "customer", "customer/billing" }
});

3. Multi-Service Reporting

Aggregate metrics from multiple microservices into a single report object, mixing SQL databases with REST APIs:

public class SalesReportConfiguration : EntityConfiguration<SalesReport>
{
    public override IEnumerable<Mapping<SalesReport, IQueryResult>> GetSchema()
    {
        return CreateSchema.For<SalesReport>()
            .Map<SalesSummaryQuery, SalesSummaryTransform>(For.Paths("report"),
                report => report.Dependents
                    .Map<InventoryApiQuery, InventoryTransform>(For.Paths("report/inventory"))
                    .Map<ShippingApiQuery, ShippingTransform>(For.Paths("report/shipping"))
                    .Map<RevenueQuery, RevenueTransform>(For.Paths("report/revenue")))
            .End();
    }
}

Why DataFuse?

The Problem

In modern architectures, a single page or API response often needs data from multiple backend systems — SQL databases, REST APIs, MongoDB, third-party services, caches. The standard approach is to write manual orchestration code for each scenario. This leads to:

Boilerplate explosion: Every new data combination means another 50+ lines of fetch-assemble-transform code
No parallelism by default: Developers write sequential calls unless they manually add Task.WhenAll
Tight coupling: Changing a data source means rewriting orchestration code
No selective loading: You fetch everything even when the consumer only needs a subset
Inconsistent patterns: Every developer solves the same problem differently

How DataFuse Solves This

DataFuse provides a declarative, schema-driven approach where you:

Define your entity (the shape of the data you want)
Configure a schema mapping queries to entity paths
Let DataFuse handle execution order, parallelism, dependency passing, and assembly

DataFuse vs Alternatives

Capability	Manual Code	GraphQL	MediatR	DataFuse
Multi-source aggregation	Manual wiring	Resolver-based	Manual wiring	Declarative schema
Parallel execution	Manual `Task.WhenAll`	Per-resolver	Manual	Automatic
Selective loading	Manual if/else	Built-in	Manual	Schema path filtering
Dependency management	Manual ordering	Implicit	Manual	Parent-child hierarchy
Type safety	Varies	Schema-based	Yes	Strongly typed
New data source support	Rewrite orchestration	New resolver	New handler	Add adapter (SQL, API, MongoDB, EF, custom)
Learning curve	None	High (new language)	Low	Low (C# only)
Backend-only (no client changes)	Yes	No (client queries)	Yes	Yes

When to Use DataFuse

Use DataFuse when:

Your data lives across different systems (SQL + APIs + external services)
You need to compose complex object graphs from multiple sources
You want automatic parallelism without managing it yourself
You need selective loading — different consumers need different subsets of data
You're building API gateways, BFF patterns, or composite APIs

Use plain Dapper/EF Core when:

All your data is in a single database
Simple queries with SQL joins suffice

Use GraphQL when:

You need client-controlled query shapes
You're building a public API where consumers define what they need

Quick Start

Get up and running in 5 minutes.

1. Install Packages

dotnet add package DataFuse.Integration
dotnet add package DataFuse.Adapters.SQL               # For SQL with Dapper
dotnet add package DataFuse.Adapters.WebAPI             # For REST APIs
dotnet add package DataFuse.Adapters.MongoDB            # For MongoDB
dotnet add package DataFuse.Adapters.EntityFramework    # For EF Core

2. Define Your Entity

public class Product : IEntity
{
    public int ProductId { get; set; }
    public string Name { get; set; }
    public decimal Price { get; set; }
    public Category Category { get; set; }
    public Review[] Reviews { get; set; }
}

3. Create Queries

SQL query (fetches product from database):

public class ProductQuery : SQLQuery<ProductResult>
{
    protected override Func<IDbConnection, Task<ProductResult>> GetQuery(
        IDataContext context, IQueryResult parentQueryResult)
    {
        var request = (ProductRequest)context.Request;
        return connection => connection.QueryFirstOrDefaultAsync<ProductResult>(
            "SELECT ProductId as Id, Name, Price, CategoryId FROM Products WHERE ProductId = @Id",
            new { Id = request.ProductId });
    }
}

MongoDB query (fetches reviews from MongoDB):

public class ReviewsMongoQuery : MongoQuery<CollectionResult<ReviewResult>>
{
    protected override Func<IMongoDatabase, Task<CollectionResult<ReviewResult>>> GetQuery(
        IDataContext context, IQueryResult? parentQueryResult)
    {
        var product = (ProductResult)parentQueryResult; // Parent result is available
        return async database =>
        {
            var collection = database.GetCollection<ReviewResult>("reviews");
            var reviews = await collection.Find(r => r.ProductId == product.Id).ToListAsync();
            return new CollectionResult<ReviewResult>(reviews);
        };
    }
}

4. Create Transformers

public class ProductTransform : BaseTransformer<ProductResult, Product>
{
    public override void Transform(ProductResult queryResult, Product entity)
    {
        entity.ProductId = queryResult.Id;
        entity.Name = queryResult.Name;
        entity.Price = queryResult.Price;
    }
}

public class ReviewsTransform : BaseTransformer<CollectionResult<ReviewResult>, Product>
{
    public override void Transform(CollectionResult<ReviewResult> queryResult, Product entity)
    {
        entity.Reviews = queryResult?.Select(r => new Review
        {
            ReviewId = r.Id, Comment = r.Comment, Rating = r.Rating
        }).ToArray() ?? Array.Empty<Review>();
    }
}

5. Configure the Schema

public class ProductConfiguration : EntityConfiguration<Product>
{
    public override IEnumerable<Mapping<Product, IQueryResult>> GetSchema()
    {
        return CreateSchema.For<Product>()
            .Map<ProductQuery, ProductTransform>(For.Paths("product"),
                product => product.Dependents
                    .Map<CategoryQuery, CategoryTransform>(For.Paths("product/category"))
                    .Map<ReviewsMongoQuery, ReviewsTransform>(For.Paths("product/reviews")))
            .End();
    }
}

6. Register with DI

services.UseDataFuse()
    .WithEngine(c => new QueryEngine(sqlConfiguration))                      // SQL adapter
    .WithEngine<DataFuse.Adapters.WebAPI.QueryEngine>()                      // Web API adapter
    .WithEngine(c => new DataFuse.Adapters.MongoDB.QueryEngine(mongoDatabase)) // MongoDB adapter
    .WithPathMatcher(c => new XPathMatcher())
    .WithEntityConfiguration<Product>(c => new ProductConfiguration());

services.AddHttpClient();

7. Use It

public class ProductService
{
    private readonly IDataProvider<Product> _dataProvider;

    public ProductService(IDataProvider<Product> dataProvider)
    {
        _dataProvider = dataProvider;
    }

    public Product GetProduct(int productId)
    {
        return _dataProvider.GetData(new ProductRequest { ProductId = productId });
    }

    // Selective loading — only fetch product + reviews, skip category
    public Product GetProductWithReviews(int productId)
    {
        return _dataProvider.GetData(new ProductRequest
        {
            ProductId = productId,
            SchemaPaths = new[] { "product", "product/reviews" }
        });
    }
}

Core Concepts

Entities

Entities represent the final aggregated data structure. They implement IEntity and define the complete object graph that DataFuse will hydrate from multiple sources:

public class Customer : IEntity
{
    public int CustomerId { get; set; }
    public string Name { get; set; }

    // Each of these can come from a different data source
    public Communication Communication { get; set; }  // From REST API
    public Address Address { get; set; }                // From SQL database
    public Order[] Orders { get; set; }                 // From EF Core
}

Queries

Queries fetch data from a specific source. DataFuse provides base classes for common sources:

Base Class	Source	Adapter Package
`SQLQuery<TResult>`	SQL databases via Dapper	`DataFuse.Adapters.SQL`
`SQLQuery<TResult>` (EF)	Entity Framework Core	`DataFuse.Adapters.EntityFramework`
`MongoQuery<TResult>`	MongoDB collections	`DataFuse.Adapters.MongoDB`
`WebQuery<TResult>`	REST APIs via HttpClient	`DataFuse.Adapters.WebAPI`

Parent-Child Query Dependencies

Child queries receive their parent's result, enabling dependent data fetching:

// Root query — fetches customer from database
public class CustomerQuery : SQLQuery<CustomerResult>
{
    protected override Func<IDbConnection, Task<CustomerResult>> GetQuery(
        IDataContext context, IQueryResult parentQueryResult)
    {
        var request = (CustomerRequest)context.Request;
        return connection => connection.QueryFirstOrDefaultAsync<CustomerResult>(
            "SELECT CustomerId as Id, Name, Code FROM Customers WHERE CustomerId = @Id",
            new { Id = request.CustomerId });
    }
}

// Child query — uses parent's CustomerId to fetch orders
public class OrdersQuery : SQLQuery<CollectionResult<OrderResult>>
{
    protected override Func<IDbConnection, Task<CollectionResult<OrderResult>>> GetQuery(
        IDataContext context, IQueryResult parentQueryResult)
    {
        var customer = (CustomerResult)parentQueryResult;
        return async connection =>
        {
            var orders = await connection.QueryAsync<OrderResult>(
                "SELECT OrderId, OrderNumber, OrderDate FROM Orders WHERE CustomerId = @Id",
                new { Id = customer.Id });
            return new CollectionResult<OrderResult>(orders);
        };
    }
}

Execution Order

1. Root Query (CustomerQuery) executes first
   |
   v  CustomerResult passed to children

2. Child Queries execute IN PARALLEL:
   - OrdersQuery (uses CustomerId)
   - CommunicationQuery (uses CustomerId)
   - AddressQuery (uses CustomerId)
   |
   v  OrderResult passed to grandchildren

3. Grandchild Queries execute:
   - OrderItemsQuery (uses OrderId from OrderResult)

Queries at the same level run in parallel automatically. DataFuse manages the dependency tree.

Transformers

Transformers map query results onto the entity. Each query has a paired transformer:

public class CustomerTransform : BaseTransformer<CustomerResult, Customer>
{
    public override void Transform(CustomerResult queryResult, Customer entity)
    {
        entity.CustomerId = queryResult.Id;
        entity.Name = queryResult.Name;
    }
}

public class OrdersTransform : BaseTransformer<CollectionResult<OrderResult>, Customer>
{
    public override void Transform(CollectionResult<OrderResult> queryResult, Customer entity)
    {
        entity.Orders = queryResult?.Select(o => new Order
        {
            OrderId = o.OrderId,
            OrderNumber = o.OrderNumber,
            OrderDate = o.OrderDate
        }).ToArray() ?? Array.Empty<Order>();
    }
}

Schema Configuration

The schema defines the full query-transform-path hierarchy:

public class CustomerConfiguration : EntityConfiguration<Customer>
{
    public override IEnumerable<Mapping<Customer, IQueryResult>> GetSchema()
    {
        return CreateSchema.For<Customer>()
            .Map<CustomerQuery, CustomerTransform>(For.Paths("customer"),
                customer => customer.Dependents
                    .Map<CommunicationQuery, CommunicationTransform>(
                        For.Paths("customer/communication"))
                    .Map<OrdersQuery, OrdersTransform>(For.Paths("customer/orders"),
                        orders => orders.Dependents
                            .Map<OrderItemsQuery, OrderItemsTransform>(
                                For.Paths("customer/orders/order/items"))))
            .End();
    }
}

Schema paths correspond to the entity's object graph:

customer — root properties
customer/communication — Customer.Communication
customer/orders — Customer.Orders collection
customer/orders/order/items — Order.Items for each order

Packages

DataFuse.Integration

Core package with orchestration, DI extensions, helpers, and path matchers.

dotnet add package DataFuse.Integration

Key components: DataProvider<T>, QueryBuilder<T>, EntityBuilder<T>, DataFuseOptionsBuilder, XPath/JSONPath matchers, DI registration via UseDataFuse().

DataFuse.Adapters.Abstraction

Interfaces and base classes: IEntity, IQuery, IQueryResult, ITransformer, IQueryEngine, BaseQuery, BaseTransformer, EntityConfiguration<T>, CreateSchema, CollectionResult<T>, CacheResultAttribute.

DataFuse.Adapters.SQL

SQL database support using Dapper.

dotnet add package DataFuse.Adapters.SQL

Supports SQL Server, SQLite, MySQL, PostgreSQL, Oracle.

DataFuse.Adapters.EntityFramework

Entity Framework Core integration.

dotnet add package DataFuse.Adapters.EntityFramework

Full LINQ support, DbContext factory integration.

DataFuse.Adapters.MongoDB

MongoDB support using the official MongoDB driver.

dotnet add package DataFuse.Adapters.MongoDB

Full IMongoDatabase access, LINQ and filter builder support.

DataFuse.Adapters.WebAPI

REST API support using HttpClient.

dotnet add package DataFuse.Adapters.WebAPI

Request/response header management, JSON deserialization.

Compatibility

Package	Target Frameworks
DataFuse.Adapters.Abstraction	netstandard2.1, net8.0, net9.0, net10.0
DataFuse.Integration	netstandard2.1, net8.0, net9.0, net10.0
DataFuse.Adapters.SQL	netstandard2.1, net8.0, net9.0, net10.0
DataFuse.Adapters.EntityFramework	net10.0
DataFuse.Adapters.MongoDB	netstandard2.1, net8.0, net9.0, net10.0
DataFuse.Adapters.WebAPI	netstandard2.1, net8.0, net9.0, net10.0

Query Implementation Guide

SQL Queries (Dapper)

// Single result
public class CustomerQuery : SQLQuery<CustomerResult>
{
    protected override Func<IDbConnection, Task<CustomerResult>> GetQuery(
        IDataContext context, IQueryResult parentQueryResult)
    {
        var request = (CustomerRequest)context.Request;
        return connection => connection.QueryFirstOrDefaultAsync<CustomerResult>(
            @"SELECT CustomerId as Id, CustomerName as Name, CustomerCode as Code
              FROM Customers WHERE CustomerId = @CustomerId",
            new { CustomerId = request.CustomerId });
    }
}

// Collection result
public class OrdersQuery : SQLQuery<CollectionResult<OrderResult>>
{
    protected override Func<IDbConnection, Task<CollectionResult<OrderResult>>> GetQuery(
        IDataContext context, IQueryResult parentQueryResult)
    {
        var customer = (CustomerResult)parentQueryResult;
        return async connection =>
        {
            var orders = await connection.QueryAsync<OrderResult>(
                "SELECT OrderId, OrderNumber, OrderDate FROM Orders WHERE CustomerId = @Id",
                new { Id = customer.Id });
            return new CollectionResult<OrderResult>(orders);
        };
    }
}

Entity Framework Queries

public class CustomerQuery : SQLQuery<CustomerResult>
{
    protected override Func<DbContext, Task<CustomerResult>> GetQuery(
        IDataContext context, IQueryResult parentQueryResult)
    {
        var request = (CustomerRequest)context.Request;
        return async dbContext =>
        {
            return await dbContext.Set<CustomerEntity>()
                .Where(c => c.CustomerId == request.CustomerId)
                .Select(c => new CustomerResult
                {
                    Id = c.CustomerId,
                    Name = c.Name,
                    Code = c.Code
                })
                .FirstOrDefaultAsync();
        };
    }
}

MongoDB Queries

// Single document
public class CustomerQuery : MongoQuery<CustomerResult>
{
    protected override Func<IMongoDatabase, Task<CustomerResult>> GetQuery(
        IDataContext context, IQueryResult? parentQueryResult)
    {
        var request = (CustomerRequest)context.Request;
        return async database =>
        {
            var collection = database.GetCollection<CustomerResult>("customers");
            return await collection.Find(c => c.Id == request.CustomerId).FirstOrDefaultAsync();
        };
    }
}

// Collection query with filter
public class OrdersQuery : MongoQuery<CollectionResult<OrderResult>>
{
    protected override Func<IMongoDatabase, Task<CollectionResult<OrderResult>>> GetQuery(
        IDataContext context, IQueryResult? parentQueryResult)
    {
        var customer = (CustomerResult)parentQueryResult;
        return async database =>
        {
            var collection = database.GetCollection<OrderResult>("orders");
            var filter = Builders<OrderResult>.Filter.Eq(o => o.CustomerId, customer.Id);
            var orders = await collection.Find(filter).SortByDescending(o => o.OrderDate).ToListAsync();
            return new CollectionResult<OrderResult>(orders);
        };
    }
}

// Aggregation pipeline
public class SalesSummaryQuery : MongoQuery<SalesSummaryResult>
{
    protected override Func<IMongoDatabase, Task<SalesSummaryResult>> GetQuery(
        IDataContext context, IQueryResult? parentQueryResult)
    {
        var customer = (CustomerResult)parentQueryResult;
        return async database =>
        {
            var collection = database.GetCollection<BsonDocument>("orders");
            var result = await collection.Aggregate()
                .Match(Builders<BsonDocument>.Filter.Eq("customerId", customer.Id))
                .Group(new BsonDocument
                {
                    { "_id", "$customerId" },
                    { "totalSpent", new BsonDocument("$sum", "$amount") },
                    { "orderCount", new BsonDocument("$sum", 1) }
                })
                .FirstOrDefaultAsync();

            return new SalesSummaryResult
            {
                TotalSpent = result?["totalSpent"].ToDecimal() ?? 0,
                OrderCount = result?["orderCount"].ToInt32() ?? 0
            };
        };
    }
}

Web API Queries

// Basic API query
public class PricingApiQuery : WebQuery<PricingResult>
{
    public PricingApiQuery() : base("https://api.pricing.com/") { }

    protected override Func<Uri> GetQuery(IDataContext context, IQueryResult parentQueryResult)
    {
        var product = (ProductResult)parentQueryResult;
        return () => new Uri($"products/{product.Id}/pricing", UriKind.Relative);
    }
}

// API query with custom headers
public class AuthenticatedApiQuery : WebQuery<UserDataResult>
{
    public AuthenticatedApiQuery() : base("https://api.secure.com/") { }

    protected override Func<Uri> GetQuery(IDataContext context, IQueryResult parentQueryResult)
    {
        var request = (SecureRequest)context.Request;
        return () => new Uri($"secure/data/{request.Id}", UriKind.Relative);
    }

    protected override IDictionary<string, string> GetRequestHeaders()
    {
        return new Dictionary<string, string>
        {
            { "Authorization", "Bearer " + GetAccessToken() },
            { "Accept", "application/json" }
        };
    }

    protected override IEnumerable<string> GetResponseHeaders()
    {
        return new[] { "X-Rate-Limit-Remaining", "X-Request-Id" };
    }
}

Query Result Types

// Simple result
public class CustomerResult : IQueryResult
{
    public int Id { get; set; }
    public string Name { get; set; }
    public string Code { get; set; }
}

// Collection result
public class CollectionResult<T> : List<T>, IQueryResult
{
    public CollectionResult(IEnumerable<T> items) : base(items) { }
    public CollectionResult() { }
}

// Cached result — cached automatically across queries
[CacheResult]
public class CategoryResult : IQueryResult
{
    public int Id { get; set; }
    public string Name { get; set; }
}

// Web result with response headers
public class UserApiResult : WebHeaderResult
{
    public int Id { get; set; }
    public string Name { get; set; }
}

Transformer Guide

Basic Transformers

public class CustomerTransform : BaseTransformer<CustomerResult, Customer>
{
    public override void Transform(CustomerResult queryResult, Customer entity)
    {
        if (queryResult == null) return;

        entity.CustomerId = queryResult.Id;
        entity.CustomerName = queryResult.Name ?? string.Empty;
        entity.CustomerCode = queryResult.Code;
    }
}

Collection Transformers

public class OrdersTransform : BaseTransformer<CollectionResult<OrderResult>, Customer>
{
    public override void Transform(CollectionResult<OrderResult> queryResult, Customer entity)
    {
        entity.Orders = queryResult?.Select(o => new Order
        {
            OrderId = o.OrderId,
            OrderNumber = o.OrderNumber,
            OrderDate = o.OrderDate,
            TotalAmount = o.TotalAmount
        }).ToArray() ?? Array.Empty<Order>();
    }
}

Context-Aware Transformers

Transformers can access the request context for conditional logic:

public class ProductTransform : BaseTransformer<ProductResult, Product>
{
    public override void Transform(ProductResult queryResult, Product entity)
    {
        entity.ProductId = queryResult.Id;
        entity.Name = queryResult.Name;
        entity.Price = queryResult.Price;

        var request = Context.Request as ProductRequest;
        if (request?.IncludeMetadata == true)
        {
            entity.CreatedDate = queryResult.CreatedDate;
            entity.LastModified = queryResult.LastModified;
        }
    }
}

Grandchild Transformers

For deeply nested data, transformers map grandchild results back to the correct parent:

public class OrderItemsTransform : BaseTransformer<CollectionResult<OrderItemResult>, Customer>
{
    public override void Transform(CollectionResult<OrderItemResult> queryResult, Customer entity)
    {
        if (queryResult == null || !queryResult.Any()) return;

        var itemsByOrder = queryResult.GroupBy(item => item.OrderId);
        foreach (var group in itemsByOrder)
        {
            var order = entity.Orders.FirstOrDefault(o => o.OrderId == group.Key);
            if (order != null)
            {
                order.Items = group.Select(item => new OrderItem
                {
                    ItemId = item.ItemId,
                    ProductName = item.ProductName,
                    Quantity = item.Quantity,
                    Price = item.Price
                }).ToArray();
            }
        }
    }
}

Advanced Features

Caching

Mark query results with [CacheResult] to cache them for use in other queries and transformers:

[CacheResult]
public class CategoryResult : IQueryResult
{
    public int Id { get; set; }
    public string Name { get; set; }
}

// Access cached results in a transformer
public class ProductTransform : BaseTransformer<ProductResult, Product>
{
    public override void Transform(ProductResult queryResult, Product entity)
    {
        entity.ProductId = queryResult.Id;

        if (Context.Cache.TryGetValue("CategoryResult", out var cached))
        {
            entity.CategoryName = ((CategoryResult)cached).Name;
        }
    }
}

Caching is useful when a grandchild query needs data from a grandparent query that isn't its direct parent.

Selective Loading

Control which parts of the object graph to load via SchemaPaths:

// Load everything
var fullCustomer = dataProvider.GetData(new CustomerRequest
{
    CustomerId = 123
    // SchemaPaths = null loads all configured paths
});

// Load only customer + orders (skip communication, address, etc.)
var withOrders = dataProvider.GetData(new CustomerRequest
{
    CustomerId = 123,
    SchemaPaths = new[] { "customer", "customer/orders" }
});

// Load just the basic profile
var basicProfile = dataProvider.GetData(new CustomerRequest
{
    CustomerId = 123,
    SchemaPaths = new[] { "customer" }
});

This avoids unnecessary queries — if a consumer only needs the customer name, DataFuse won't execute order or communication queries.

Parallel Execution

Sibling queries at the same level execute in parallel automatically:

return CreateSchema.For<Customer>()
    .Map<CustomerQuery, CustomerTransform>(For.Paths("customer"),
        customer => customer.Dependents
            .Map<ContactQuery, ContactTransform>(For.Paths("customer/contact"))           // Parallel
            .Map<PreferencesQuery, PreferencesTransform>(For.Paths("customer/preferences")) // Parallel
            .Map<AddressQuery, AddressTransform>(For.Paths("customer/address")))           // Parallel
    .End();

No Task.WhenAll boilerplate needed. DataFuse manages execution order based on the dependency tree.

Conditional Query Execution

Child queries can skip execution based on parent data:

public class PremiumServicesQuery : WebQuery<PremiumServicesResult>
{
    protected override Func<Uri> GetQuery(IDataContext context, IQueryResult parentQueryResult)
    {
        var customer = (CustomerResult)parentQueryResult;

        if (customer.CustomerType != "Premium")
            return null; // Returning null skips this query

        return () => new Uri($"premium-services/{customer.Id}", UriKind.Relative);
    }
}

Cross-Source Dependencies

Mix data sources within the same entity. A root query can be SQL while its children pull from MongoDB and REST APIs:

return CreateSchema.For<Customer>()
    .Map<CustomerQuery, CustomerTransform>(For.Paths("customer"),          // SQL
        customer => customer.Dependents
            .Map<BillingApiQuery, BillingTransform>(For.Paths("customer/billing"))     // REST API
            .Map<ReviewsMongoQuery, ReviewsTransform>(For.Paths("customer/reviews"))   // MongoDB
            .Map<AnalyticsQuery, AnalyticsTransform>(For.Paths("customer/analytics"))) // EF Core
    .End();

Custom Query Engines

Add support for any data source by implementing IQueryEngine:

public class RedisQueryEngine : IQueryEngine
{
    private readonly IConnectionMultiplexer _redis;

    public RedisQueryEngine(IConnectionMultiplexer redis)
    {
        _redis = redis;
    }

    public bool CanExecute(IQuery query) => query is IRedisQuery;

    public async Task<IQueryResult> Execute(IQuery query)
    {
        var redisQuery = (IRedisQuery)query;
        return await redisQuery.Run(_redis.GetDatabase());
    }
}

// Register alongside built-in adapters
services.UseDataFuse()
    .WithEngine(c => new RedisQueryEngine(redisConnection))
    .WithEngine(c => new QueryEngine(sqlConfiguration))
    .WithEngine(c => new DataFuse.Adapters.MongoDB.QueryEngine(mongoDatabase))
    // ...

Path Matching

DataFuse supports XPath and JSONPath patterns for schema paths:

XPath patterns:

customer — exact match
customer/orders — nested path
customer/orders/order/items — deep nesting

JSONPath patterns:

$.customer — root level
$.customer.orders — nested property
$.customer.orders[*].items — array elements

services.UseDataFuse()
    .WithPathMatcher(c => new XPathMatcher())   // or JPathMatcher

Architecture

Execution Flow

Client Request
    |
    v
DataProvider<T>
    |
    v
QueryBuilder — filters by SchemaPaths, resolves dependencies, builds execution plan
    |
    v
QueryExecutor — executes level-by-level, parallelizes siblings, propagates results
    |
    v
EntityBuilder — applies transformers in dependency order, assembles final entity
    |
    v
Fully Hydrated Entity

Component Responsibilities

Component	Responsibility
`DataProvider<T>`	Main orchestrator — coordinates the full pipeline
`QueryBuilder<T>`	Filters queries by requested schema paths, resolves parent-child dependencies
`QueryExecutor`	Executes queries level-by-level with parallel processing of siblings
`EntityBuilder<T>`	Matches results to transformers and builds the final entity
`IQueryEngine`	Adapter interface — each data source implements this
`EntityConfiguration<T>`	Declares the schema: query-transform-path mappings

Configuration Patterns

Linear — flat entity:

return CreateSchema.For<Customer>()
    .Map<CustomerQuery, CustomerTransform>(For.Paths("customer"))
    .Map<AddressQuery, AddressTransform>(For.Paths("customer/address"))
    .End();

Branching — multiple children:

return CreateSchema.For<Customer>()
    .Map<CustomerQuery, CustomerTransform>(For.Paths("customer"),
        c => c.Dependents
            .Map<ContactQuery, ContactTransform>(For.Paths("customer/contact"))
            .Map<PreferencesQuery, PrefsTransform>(For.Paths("customer/preferences")))
    .End();

Deep nesting — up to 5 levels:

return CreateSchema.For<Customer>()
    .Map<CustomerQuery, CustomerTransform>(For.Paths("customer"),
        c => c.Dependents
            .Map<OrdersQuery, OrdersTransform>(For.Paths("customer/orders"),
                o => o.Dependents
                    .Map<ItemsQuery, ItemsTransform>(For.Paths("customer/orders/order/items"))))
    .End();

Best Practices

Query Design

Keep queries focused — one query per data concern
Use CollectionResult<T> for queries returning multiple records
Return null from GetQuery to conditionally skip execution
Use [CacheResult] for expensive, reusable query results

Transformer Design

Always handle null query results gracefully
Initialize collections to prevent null reference exceptions
Keep transformers focused on mapping — avoid business logic
Use Context.Request for conditional transformation

Schema Design

Use meaningful, hierarchical path names that mirror your entity structure
Keep nesting depth reasonable (3-4 levels is typical)
Use selective loading for performance-sensitive endpoints

DI Registration

services.UseDataFuse()
    .WithEngine(c => new QueryEngine(sqlConfig))                               // SQL
    .WithEngine<DataFuse.Adapters.WebAPI.QueryEngine>()                        // REST API
    .WithEngine(c => new DataFuse.Adapters.MongoDB.QueryEngine(mongoDatabase)) // MongoDB
    .WithPathMatcher(c => new XPathMatcher())
    .WithEntityConfiguration<Customer>(c => new CustomerConfiguration())
    .WithEntityConfiguration<Product>(c => new ProductConfiguration());

services.AddLogging();
services.AddHttpClient();

Migrating from Schemio (v2.x) to DataFuse (v3.0.0)

DataFuse v3.0.0 is a complete rebrand of the Schemio framework. All package names, namespaces, and registration APIs have changed. Follow this guide to upgrade your project.

Step 1: Update NuGet Packages

Remove the old Schemio packages and install the new DataFuse equivalents:

Old Package (Remove)	New Package (Install)
`Schemio.Core`	`DataFuse.Adapters.Abstraction` + `DataFuse.Integration`
`Schemio.SQL`	`DataFuse.Adapters.SQL`
`Schemio.EntityFramework`	`DataFuse.Adapters.EntityFramework`
`Schemio.API`	`DataFuse.Adapters.WebAPI`

# Remove old packages
dotnet remove package Schemio.Core
dotnet remove package Schemio.SQL
dotnet remove package Schemio.EntityFramework
dotnet remove package Schemio.API

# Install new packages
dotnet add package DataFuse.Integration
dotnet add package DataFuse.Adapters.SQL
dotnet add package DataFuse.Adapters.EntityFramework
dotnet add package DataFuse.Adapters.WebAPI
dotnet add package DataFuse.Adapters.MongoDB  # New in v3.0.0

Step 2: Update Namespaces

Find and replace using statements across your codebase:

Old Namespace	New Namespace
`using Schemio;`	`using DataFuse.Adapters.Abstraction;`
`using Schemio.SQL;`	`using DataFuse.Adapters.SQL;`
`using Schemio.EntityFramework;`	`using DataFuse.Adapters.EntityFramework;`
`using Schemio.API;`	`using DataFuse.Adapters.WebAPI;`

Step 3: Update DI Registration

The service registration method and options builder have been renamed:

// Before (Schemio v2.x)
services.UseSchemio(new SchemioOptionsBuilder()
    .WithEngine(c => new QueryEngine(sqlConfig))
    .WithPathMatcher(c => new XPathMatcher())
    .WithEntityConfiguration<Customer>(c => new CustomerConfiguration()));

// After (DataFuse v3.0.0)
services.UseDataFuse(new DataFuseOptionsBuilder()
    .WithEngine(c => new QueryEngine(sqlConfig))
    .WithPathMatcher(c => new XPathMatcher())
    .WithEntityConfiguration<Customer>(c => new CustomerConfiguration()));

Changed APIs:

Old (Schemio)	New (DataFuse)
`UseSchemio()`	`UseDataFuse()`
`SchemioOptionsBuilder`	`DataFuseOptionsBuilder`
`ISchemioOptions`	`IDataFuseOptions`

Step 4: Verify

No changes are required to your queries, transformers, entity configurations, or schema definitions — only the package references, namespaces, and DI registration need updating. After making these changes:

Build the solution and resolve any remaining namespace errors
Run your existing tests to confirm behavior is unchanged
Verify DI registration at startup

New in v3.0.0

After migrating, you can take advantage of new features:

MongoDB Adapter — aggregate data from MongoDB collections with DataFuse.Adapters.MongoDB
Transform Hooks — use ITransformerHooks with PreTransformContext and PostTransformContext for pipeline control
Query Result Caching — mark results with [CacheResult] for automatic caching
Multi-Target Framework Support — packages now support netstandard2.1, net8.0, net9.0, and net10.0

See the Release Notes for full details.

Getting Support

GitHub: github.com/CodeShayk/DataFuse.Net
Issues: Report bugs and feature requests
Wiki: Developer guide and documentation
Samples: Check example projects in the repository for real-world usage patterns

[Boost]

CØDE N!NJΔ — Sun, 22 Mar 2026 11:36:21 +0000

SourceFlow.Net - Event Sourcing Framework. - DEV Community

A modern, lightweight, and extensible framework for building event-sourced applications using...

dev.to

ActiveForge - Active Record based ORM

CØDE N!NJΔ — Fri, 13 Mar 2026 16:52:55 +0000

The Active Record Pattern
Architecture Overview
Entities — Record and IdentityRecord
Field Types (TField System)
DataConnection — The Gateway
CRUD Operations
Query Predicates (QueryTerm API)
Sorting and Pagination
LINQ Query Support
Transactions (Manual API)
Unit of Work & Transactions
Action Queue (Batch Operations)
Field Subsets (Partial Fetch / Update)
Field Encryption
Custom Field Mappers
Polymorphic Mapping (BaseFactory)
Optimistic Locking
Lazy Streaming
Raw SQL and Stored Procedures
Lookup / Cached Reference Tables
Architecture Deep Dive
Quick Reference Cheat Sheet
MongoDB Provider
SQLite Provider
Attributes Reference
Joins and Relationships
Complete Examples

1. The Active Record Pattern

1.1 What is Active Record?

Active Record is a design pattern first named by Martin Fowler in Patterns of Enterprise Application Architecture (2003). Its defining idea is simple: an object that represents a database row also knows how to persist itself. The class carries both the data (fields mapping to columns) and the behaviour that reads, writes, and deletes that data from the database. There is no separate layer sitting between the object and the database — the object is the persistence unit.

In ActiveForge every entity class inherits from Record. A Record instance holds typed field objects (TString, TDecimal, TBool, …) for each column, and exposes methods like Insert(), Update(), Delete(), and Read() that execute the corresponding SQL:

// The object carries data AND knows how to save itself.
var product = new Product(conn);
product.Name.SetValue("Widget");
product.Price.SetValue(9.99m);
product.Insert();   // ← persistence behaviour on the object itself

product.Price.SetValue(14.99m);
product.Update();   // ← object tells the DB to update its own row

1.2 Why Use Active Record?

Simplicity for CRUD-heavy applications. When the primary job of your code is creating, reading, updating and deleting records — as it is in the overwhelming majority of business applications — Active Record keeps the pattern count low. You work with one object that represents the row; you do not need to maintain a separate repository class, a separate mapping class, and a separate domain model class for each table.

Reduced boilerplate. With Data Mapper (see §1.3) you write a ProductRepository, a ProductMapper, and a Product domain object for every entity. With Active Record you write one Product class and you are done. For applications with dozens of tables this is a meaningful reduction in total code.

Immediate navigability. Because persistence methods live on the entity, any developer reading the code can see at a glance what a class can do. product.Insert() is self-evident in a way that _unitOfWork.ProductRepository.Add(product) is not.

Low cognitive overhead for queries. The QueryTerm predicate tree and the LINQ layer translate directly into typed field references on the entity itself:

conn.Query<Product>().Where(p => p.Price > 50m && p.InStock == true).ToList();

The field references (p.Price, p.InStock) are real TField objects on the same class you use everywhere else — not magic strings, not a separate query model.

1.3 How Active Record Differs from Other Patterns

Data Mapper

Data Mapper separates domain objects completely from persistence logic. A plain Product POCO knows nothing about databases; a separate ProductMapper (or ORM configuration file) handles the translation. Entity Framework Core is the canonical .NET example.

Concern	Active Record (ActiveForge)	Data Mapper (EF Core)
Where does persistence logic live?	On the entity class (`Record`)	In the ORM mapping layer, separate from the domain object
Domain object awareness	Entity knows its own columns and how to save itself	Entity is a plain class with no ORM dependency
Boilerplate per table	One class	Entity + mapping configuration (fluent or attributes) + optional repository
Ideal for	CRUD-heavy applications, direct DB → object mapping	Complex domain models where business logic must be insulated from persistence concerns
Testability	Test via a stub `DataConnection`	Test domain objects with no DB dependency at all

When to prefer Data Mapper: If your domain objects need to be completely ignorant of the database — for example in a DDD (Domain-Driven Design) project with a rich domain model, complex invariants, and a desire to keep the persistence mechanism fully swappable — Data Mapper gives a cleaner separation. The cost is more infrastructure code.

When Active Record wins: When your application's entities largely mirror your database schema, the business logic is not so complex that it demands strict separation, and development speed matters. This covers the majority of line-of-business applications.

Repository Pattern

The Repository pattern is often used on top of a Data Mapper ORM to add a collection-like interface over the persistence layer:

// Repository pattern (not ActiveForge):
IProductRepository repo = new SqlProductRepository(dbContext);
Product p = await repo.GetByIdAsync(42);
p.Price = 14.99m;
await repo.SaveAsync(p);

With Active Record the entity already is its own repository in a sense. You can still add a separate service or repository class around ActiveForge entities if you want to centralise query logic or enforce business rules — the ORM does not prevent it — but the pattern is not forced on you.

Table Gateway

Table Gateway assigns one object per table (not per row), and that object contains all the methods for querying that table. This is coarser-grained than Active Record: you call ProductGateway.FindAll() and receive raw data rows, rather than calling product.Read() and getting a fully hydrated typed object back.

Record / DTO (plain classes)

Some teams use plain data transfer objects with a separate command/query handler (CQRS style). This decouples read and write models completely but requires hand-written SQL or a micro-ORM (Dapper) and significant ceremony to wire up. Active Record is a better fit when the read model and the write model are the same thing, which is again the common case in business software.

1.4 Active Record in ActiveForge — Key Decisions

Fields, not properties. ActiveForge represents columns as public TField instance fields rather than auto-properties. This lets the ORM discover them by reflection without attributes on every getter, lets them carry null/loaded state independently of their value, and enables the predicate system (a TField reference in a QueryTerm constructor tells the ORM exactly which column to filter on).

Shared connection, not embedded connection. The entity does not open its own database connection. Instead, one DataConnection is passed in at construction time and shared across all objects in a unit of work. This keeps connection management explicit and testable, while still letting the object call Insert() / Update() without the caller needing to think about the connection.

Delegation, not inheritance from the connection. Record.Insert() delegates to conn.Insert(this). The SQL generation, parameter binding, and result hydration live in DataConnection (and its SQL Server implementation), not in each entity. Entities therefore stay lean — they contain field declarations and business logic, nothing else.

Optional Unit of Work on top. Pure Active Record is sometimes criticised for making it hard to batch multiple saves into a single transaction in a clean way. ActiveForge addresses this with the IUnitOfWork / With.Transaction layer (§11), which can wrap any number of Insert() / Update() / Delete() calls in a managed transaction without changing the entity code at all.

2. Architecture Overview

ActiveForge is a lightweight Active Record ORM for .NET 8. It is split across provider assemblies to keep database-specific concerns separate from the core abstractions.

2.1 Assembly Split

Assembly	Contents
`ActiveForge`	Core — entities, TField types, QueryTerm predicates, LINQ layer, transactions (abstract), adapters (abstract)
`ActiveForge.SqlServer`	SQL Server provider — `SqlServerConnection`, SQL adapters, `SqlServerUnitOfWork`
`ActiveForge.PostgreSQL`	PostgreSQL provider — `PostgreSQLConnection`, Npgsql adapters, `PostgreSQLUnitOfWork`
`ActiveForge.MongoDB`	MongoDB provider — `MongoDataConnection`, BSON mapping, `MongoUnitOfWork`
`ActiveForge.SQLite`	SQLite provider — `SQLiteConnection`, Microsoft.Data.Sqlite adapters, `SQLiteUnitOfWork`

Entity classes only reference ActiveForge. Applications add the appropriate provider package alongside it.

┌──────────────────────────────────────────────────────────────────────────────────────────┐
│  Your Application                                                                        │
│                                                                                          │
│  Record subclass ──── CRUD ────► DataConnection (abstract, core)                         │
│  (fields, logic)                         │                                               │
│  QueryTerm / LINQ ─── query calls ───────┤                                               │
│                                          │ implemented by (choose one)                   │
│           ┌──────────────────────────────┼──────────────────────────────┐                │
│  SqlServerConnection  PostgreSQLConnection  MongoDataConnection  SQLiteConnection        │
│  (SqlServer provider) (PostgreSQL provider) (MongoDB provider)  (SQLite provider)        │
│           │                   │                    │                    │                │
│      SQL Server           PostgreSQL            MongoDB              SQLite              │
└──────────────────────────────────────────────────────────────────────────────────────────┘

2.2 Core Principles

Active Record — Record instances know how to persist themselves via a shared DataConnection.
Type-safe fields — every column is represented by a TField subclass, not a bare property. This tracks null/loaded state and enables predicate construction.
Composable predicates — QueryTerm objects compose with C# &, |, ! operators to build arbitrary WHERE clauses.
Connection-centric — DataConnection is the single point of query execution; entities delegate to it.
Provider-agnostic core — ActiveForge has no dependency on Microsoft.Data.SqlClient or Npgsql. Only the provider packages do.

3. Entities — Record and IdentityRecord

3.1 Base Classes

Class	When to use
`Record`	Tables without a single integer auto-identity primary key
`IdentityRecord`	Tables with an `INT IDENTITY(1,1)` primary key (exposed as `ID: TPrimaryKey`)

3.2 Defining an Entity

using ActiveForge;
using ActiveForge.Attributes;

[Table("Products")]
public class Product : IdentityRecord
{
    // Each public field maps to a column.
    // The [Column] attribute sets the DB column name.
    [Column("Name")]      public TString  Name      = new TString();
    [Column("Price")]     public TDecimal Price     = new TDecimal();
    [Column("InStock")]   public TBool    InStock   = new TBool();
    [Column("CreatedAt")] public TDateTime CreatedAt = new TDateTime();

    // Required: parameterless constructor for ORM hydration.
    public Product() { }

    // Constructor accepting a connection (used in application code).
    public Product(DataConnection conn) : base(conn) { }
}

Conventions:

[Table("TableName")] — maps the class to a DB table.
[Column("ColumnName")] — maps a field to a DB column (required).
[Identity] — marks a field as auto-generated (implicit on IdentityRecord.ID).
Fields are public instance fields, not properties. The ORM finds them via reflection.
A no-arg constructor is mandatory; the ORM calls it when hydrating query results.

3.3 IdentityRecord.ID

IdentityRecord adds:

[Column("ID")]
[Identity]
public TPrimaryKey ID = new TPrimaryKey();

After Insert(), ID.GetValue() returns the new auto-generated integer.

3.4 Embedded / Joined Objects

Embed another Record as a field to express a JOIN:

[Table("OrderLines")]
public class OrderLine : IdentityRecord
{
    [Column("OrderID")]  public TForeignKey OrderID  = new TForeignKey();
    [Column("Qty")]      public TInt        Qty      = new TInt();
    [Column("Price")]    public TDecimal    UnitPrice = new TDecimal();

    // Embedded — triggers a JOIN in SELECT queries.
    public Order Order = new Order();

    public OrderLine() { }
    public OrderLine(DataConnection conn) : base(conn) { }
}

4. Field Types (TField System)

4.1 Common API

Every TField subtype shares this interface:

// Read value
object  GetValue();           // returns the typed value, or null if IsNull()
object  GetRawValue();        // raw DB value (may differ for encrypted fields)

// Write value
void    SetValue(object value);
void    SetNull();

// State
bool    IsNull();             // true if value is SQL NULL
bool    IsLoaded();           // true if value was loaded from DB (or set)

// Copy
void    CopyFrom(TField other);

Many subtypes also expose implicit conversions so you can write:

string name = product.Name;          // TString → string implicit cast
product.Name.SetValue("Widget");     // always works (explicit)

Note: TField.Value is protected. Never assign .Value = directly from outside the class — use SetValue().

4.2 Numeric Types

Type	CLR type	DB type
`TInt`	`int`	INT
`TInt16`	`short`	SMALLINT
`TInt64` / `TLong`	`long`	BIGINT
`TByte`	`byte`	TINYINT
`TSByte`	`sbyte`	TINYINT (signed)
`TUInt`	`uint`	INT (unsigned)
`TUInt16`	`ushort`	SMALLINT (unsigned)
`TUInt64`	`ulong`	BIGINT (unsigned)
`TFloat`	`float`	REAL
`TDouble`	`double`	FLOAT
`TDecimal`	`decimal`	DECIMAL / NUMERIC

product.Price.SetValue(19.99m);
decimal price = (decimal)product.Price.GetValue();
// or via implicit conversion where available:
decimal price2 = product.Price;

4.3 String Types

Type	Notes
`TString`	`NVARCHAR` / `VARCHAR`
`TChar`	Single character
`THtmlString`	Same as TString, marks HTML content
`TIpAddress`	Stored as VARCHAR, parsed as IP

product.Name.SetValue("Widget");
string name = (string)product.Name.GetValue();

// Null check — use (TString)null to avoid ambiguity with == overloads:
var term = new EqualTerm(template, template.Name, (TString)null); // IS NULL

4.4 Key Types

Type	Notes
`TPrimaryKey`	Auto-identity integer PK; read-only after insert
`TForeignKey`	Integer FK referencing another table

int id = (int)order.ID.GetValue();          // primary key after insert
orderLine.OrderID.SetValue(id);              // set foreign key

4.5 Date / Time Types

Type	CLR type	Notes
`TDateTime`	`DateTime`	Local or unspecified
`TUtcDateTime`	`DateTime`	Stored/read as UTC
`TLocalDateTime`	`DateTime`	Converted to local on read
`TDate`	`DateTime` (date only)	Time portion zeroed
`TUtcDate`	`DateTime`	Date in UTC
`TLocalDate`	`DateTime`	Date in local TZ
`TTime`	`TimeSpan`	Time-of-day

product.CreatedAt.SetValue(DateTime.UtcNow);
DateTime created = (DateTime)product.CreatedAt.GetValue();

4.6 Boolean and Binary

// TBool — maps to SQL BIT
product.InStock.SetValue(true);
bool inStock = (bool)product.InStock.GetValue();

// TByteArray — maps to VARBINARY / IMAGE
attachment.Data.SetValue(File.ReadAllBytes("file.pdf"));
byte[] bytes = (byte[])attachment.Data.GetValue();

// TGuid — maps to UNIQUEIDENTIFIER
record.ExternalId.SetValue(Guid.NewGuid());
Guid id = (Guid)record.ExternalId.GetValue();

4.7 Null Handling

product.Name.SetNull();              // explicitly set to NULL
if (product.Name.IsNull()) { ... }   // check for NULL
if (!product.Name.IsLoaded()) { ... } // never set at all

5. DataConnection — The Gateway

DataConnection is the abstract base; SqlServerConnection is the concrete SQL Server implementation.

5.1 Creating a Connection

SqlServerConnection, PostgreSQLConnection, and MongoDataConnection all live in the ActiveForge namespace (in their respective provider assemblies), so no extra using directive is needed once the provider assembly is referenced.

// SQL Server (reference ActiveForge + ActiveForge.SqlServer)
using ActiveForge;

var conn = new SqlServerConnection(
    "Server=.;Database=MyDB;Integrated Security=True;TrustServerCertificate=True;",
    new BaseFactory());
conn.Connect();

// PostgreSQL (reference ActiveForge + ActiveForge.PostgreSQL)
using ActiveForge;

var conn = new PostgreSQLConnection(
    "Host=localhost;Database=mydb;Username=app;Password=secret;",
    new BaseFactory());
conn.Connect();

// MongoDB (reference ActiveForge + ActiveForge.MongoDB)
using ActiveForge;

var conn = new MongoDataConnection(
    "mongodb://localhost:27017",
    "myDatabase");
conn.Connect();

5.2 Lifecycle

conn.Connect();     // opens the ADO.NET connection
conn.Disconnect();  // closes it

5.3 Factory Pattern

Pass a BaseFactory to control how the ORM instantiates objects. The default BaseFactory uses Activator.CreateInstance. Override Create(Type) for polymorphic mapping (see §16).

5.4 Provider Dialect Comparison

Feature	SQL Server	PostgreSQL	MongoDB
Parameter mark	`@name`	`@name`	N/A
Identifier quoting	`[Name]`	`"name"`	N/A
Row limit syntax	`SELECT TOP n …`	`SELECT … LIMIT n`	`.Limit(n)`
String concatenation	`+`	`\	\
Identity retrieval	{% raw %}`SELECT SCOPE_IDENTITY()`	`SELECT LASTVAL()`	Counter collection
Row lock hint	`WITH (UPDLOCK)`	`FOR UPDATE` (end)	N/A
Identity insert control	`SET IDENTITY_INSERT … ON/OFF`	Not needed	Not needed
Schema introspection	`SYSOBJECTS` / `SYSCOLUMNS`	`information_schema.columns`	`[Table]` attribute
Identity storage	Auto-generated int column	Auto-generated int (`serial`)	`_id` field (int)
Identifier case	Case-insensitive	Case-sensitive (lower-case)	Exact match
Unit of Work class	`SqlServerUnitOfWork`	`PostgreSQLUnitOfWork`	`MongoUnitOfWork`
Driver	`Microsoft.Data.SqlClient`	`Npgsql`	`MongoDB.Driver`
SQL operations	Full	Full	`NotSupportedException`
Transaction scope	RDBMS transaction	RDBMS transaction	Requires replica set

PostgreSQL naming note: PostgreSQL folds unquoted identifiers to lower-case at parse time. All [Table] and [Column] attribute values should be lower-case unless you created the table with quoted identifiers.

MongoDB naming note: [Table("collectionName")] maps to the MongoDB collection name. [Column("fieldName")] maps to the BSON field name. The [Identity] field maps to the special _id BSON field and is stored as an integer using a counter document for auto-increment simulation.

6. CRUD Operations

6.1 Insert

var product = new Product(conn);
product.Name.SetValue("Widget");
product.Price.SetValue(9.99m);
product.InStock.SetValue(true);
product.CreatedAt.SetValue(DateTime.UtcNow);

product.Insert();                    // executes INSERT; product.ID is populated

Console.WriteLine(product.ID.GetValue()); // e.g. 42

Alternatively, use conn.Insert(product).

6.2 Read (by Primary Key)

var product = new Product(conn);
product.ID.SetValue(42);
product.Read();

Console.WriteLine(product.Name.GetValue()); // "Widget"

Alternatively, use conn.Read(product).

6.3 Update

// Update all mapped columns:
product.Price.SetValue(14.99m);
product.Update();

// Update only changed columns (requires initial snapshot):
product.Price.SetValue(19.99m);
product.UpdateChanged();

// With locking options:
product.Update(RecordLock.UpdateOption.IgnoreLock);

6.4 Delete

product.Delete();                    // deletes by PK

// Delete by predicate:
var template = new Product(conn);
var term = new EqualTerm(template, template.InStock, false);
template.Delete(term);               // DELETE WHERE InStock = 0

6.5 ReadForUpdate (Advisory Lock)

product.ID.SetValue(1);
product.ReadForUpdate();             // SELECT ... WITH (UPDLOCK)
product.Price.SetValue(29.99m);
product.Update();

7. Query Predicates (QueryTerm API)

7.1 Equality

var template = new Product(conn);
var term = new EqualTerm(template, template.Name, "Widget");
// → WHERE Name = @Name

7.2 Comparisons

new GreaterThanTerm(template, template.Price, 50m)     // >
new GreaterOrEqualTerm(template, template.Price, 50m)  // >=
new LessThanTerm(template, template.Price, 10m)        // <
new LessOrEqualTerm(template, template.Price, 10m)     // <=

7.3 String Matching

new LikeTerm(template, template.Name, "%widget%")       // LIKE
new ContainsTerm(template, template.Name, "widget")     // LIKE '%widget%' (built-in)

7.4 Null Checks

new IsNullTerm(template, template.Name)     // IS NULL
!new IsNullTerm(template, template.Name)   // IS NOT NULL (via NotTerm)

7.5 IN Clause

var ids = new List<int> { 1, 2, 3 };
new InTerm(template, template.ID, ids)
// → WHERE ID IN (@IN_ID0, @IN_ID1, @IN_ID2)

7.6 Full-Text Search

new FullTextTerm(template, template.Name, "widget NEAR gadget")
// → WHERE CONTAINS(Name, 'widget NEAR gadget')

7.7 EXISTS Sub-query

// "Products that have at least one OrderLine"
var orderLine = new OrderLine(conn);
var subQuery = conn.Query<OrderLine>()
    .Where(ol => ol.OrderID == order.ID);  // LINQ style

// Or using the classic Query<T> builder:
var exists = new ExistsTerm(
    template,
    conn.Query<OrderLine>(orderLine)
        .Where(new EqualTerm(orderLine, orderLine.OrderID, template.ID)));

7.8 Raw SQL Predicate

new RawSqlTerm("Price BETWEEN 10 AND 50")
// Injected verbatim — use sparingly; prefer parameterized terms.

7.9 Logical Composition

var inStock = new EqualTerm(template, template.InStock, true);
var cheap   = new LessThanTerm(template, template.Price, 20m);
var named   = new ContainsTerm(template, template.Name, "widget");

// AND
QueryTerm both   = inStock & cheap;

// OR
QueryTerm either = inStock | cheap;

// NOT
QueryTerm notIn  = !inStock;

// Complex
QueryTerm complex = (inStock & cheap) | (named & !inStock);

7.10 Executing Queries

var template = new Product(conn);

// All matching rows (loads into memory):
RecordCollection results = conn.QueryAll(template, term, sortOrder, 0, null);

// First match only — populates `template` in-place; returns true if a row was found:
bool found = conn.QueryFirst(template, term, sortOrder, null);
if (found)
    Console.WriteLine(template.Name.GetValue());

// Count:
int count = conn.QueryCount(template, term);

// Page:
var page = conn.QueryPage(template, term, sortOrder, startRecord: 0, pageSize: 20);
// page.StartRecord, page.PageSize, page.IsMoreData, page.TotalRowCount

// Lazy stream (no memory buffer):
IEnumerable<Product> stream = conn.LazyQueryAll<Product>(template, term, sortOrder);

8. Sorting and Pagination

8.1 Sort Orders

var template = new Product(conn);

SortOrder byName      = new OrderAscending(template, template.Name);
SortOrder byPriceDesc = new OrderDescending(template, template.Price);

// Compose multiple sorts:
// Use CombinedSortOrder (Linq namespace) or pass the first to QueryAll
// and rely on secondary ORDER BY via SQL.

8.2 QueryPage

QueryPage page = conn.QueryPage(
    template,
    new EqualTerm(template, template.InStock, true),
    new OrderAscending(template, template.Name),
    startRecord: 0,
    pageSize: 20);

Console.WriteLine($"Page 1: {page.Count} items, more={page.IsMoreData}, total={page.TotalRowCount}");

// Next page:
page = conn.QueryPage(template, term, sort, startRecord: 20, pageSize: 20);

9. LINQ Query Support

(Requires using ActiveForge.Linq;)

9.1 Entry Point

IQueryable<Product> q = conn.Query<Product>();

Internally creates a template instance and wraps it in OrmQueryable<T>. The database is NOT queried until you enumerate.

9.2 Where (Predicates)

// Equality / inequality
conn.Query<Product>().Where(p => p.Name == "Widget").ToList();
conn.Query<Product>().Where(p => p.Name != "Widget").ToList();

// Comparisons
conn.Query<Product>().Where(p => p.Price >= 50m).ToList();
conn.Query<Product>().Where(p => p.Price <  10m).ToList();

// Null checks (cast null explicitly to avoid ambiguity)
conn.Query<Product>().Where(p => p.Name == (TString)null).ToList();  // IS NULL
conn.Query<Product>().Where(p => p.Name != (TString)null).ToList();  // IS NOT NULL

// Logical AND / OR / NOT
conn.Query<Product>().Where(p => p.InStock == true && p.Price > 20m).ToList();
conn.Query<Product>().Where(p => p.Price < 5m  || p.Price > 100m).ToList();
conn.Query<Product>().Where(p => !(p.InStock == true)).ToList();

// IN clause — use List<T>, NOT arrays (array causes MemoryExtensions ambiguity)
var names = new List<string> { "Widget", "Gadget", "Gizmo" };
conn.Query<Product>().Where(p => names.Contains(p.Name)).ToList();


// String Wildcards
conn.Query<Product>().Where(p => p.Name.StartsWith("App")).ToList();
// → WHERE Name LIKE 'App%'

conn.Query<Product>().Where(p => p.Name.EndsWith("inc")).ToList();
// → WHERE Name LIKE '%inc'

// Implicit Booleans
// You can natively pass `TBool` fields directly into the lambda constraint:

conn.Query<Product>().Where(p => p.IsActive).ToList();
// → WHERE IsActive = 1

// Captured local variables
string target  = "Widget";
decimal minP   = 10m;
conn.Query<Product>().Where(p => p.Name == target && p.Price >= minP).ToList();

9.3 Chained Where (auto-ANDed)

conn.Query<Product>()
    .Where(p => p.InStock == true)
    .Where(p => p.Price   >  10m)
    .Where(p => p.Name    != "Discontinued")
    .ToList();
// Equivalent to: WHERE InStock = 1 AND Price > 10 AND Name <> 'Discontinued'

9.4 Sorting

conn.Query<Product>().OrderBy(p => p.Name).ToList();
conn.Query<Product>().OrderByDescending(p => p.Price).ToList();

// Multi-column:
conn.Query<Product>()
    .OrderBy(p => p.Name)
    .ThenByDescending(p => p.Price)
    .ToList();

9.5 Pagination

// Page 2, 20 items per page:
conn.Query<Product>()
    .Where(p => p.InStock == true)
    .OrderBy(p => p.Name)
    .Skip(20)
    .Take(20)
    .ToList();

When Take or Skip is set, QueryPage is used internally.
Without Skip/Take, LazyQueryAll is used (memory-efficient streaming).

9.6 Full Chain

using ActiveForge.Linq;

var featured = new List<string> { "Widget", "Gadget" };
decimal min  = 5m;

List<Product> results = conn.Query<Product>()
    .Where(p => featured.Contains(p.Name) || p.Price > min)
    .Where(p => p.InStock == true)
    .OrderBy(p => p.Price)
    .ThenBy(p => p.Name)
    .Skip(0)
    .Take(10)
    .ToList();

9.7 Supported Operators

LINQ	Generated Term	Notes
`x.F == v`	`EqualTerm`
`x.F != v`	`!EqualTerm`
`x.F > v`	`GreaterThanTerm`
`x.F >= v`	`GreaterOrEqualTerm`
`x.F < v`	`LessThanTerm`
`x.F <= v`	`LessOrEqualTerm`
`x.F == null`	`IsNullTerm`	Cast null: `(TString)null`
`x.F != null`	`!IsNullTerm`
`a && b`	`AndTerm`
`a \	\	b`
`!a`	`NotTerm`
`list.Contains(x.F)`	`InTerm`	Use `List<T>`, not arrays
`x.F.StartsWith(v)`	`LikeTerm` (`v%`)
`x.F.EndsWith(v)`	`LikeTerm` (`%v`)
`x.BoolField`	`EqualTerm` with `true`	Evaluates raw boolean fields natively
`OrderBy`	`OrderAscending`
`OrderByDescending`	`OrderDescending`
`ThenBy`	Appended ascending
`ThenByDescending`	Appended descending
`Take(n)`	`pageSize`
`Skip(n)`	`startRecord`
`Count()`, `LongCount()`	`DataConnection.QueryCount`	Executes scalar COUNT immediately
`First()`, `FirstOrDefault()`	`DataConnection.QueryFirst`	Executes scalar SELECT TOP 1 immediately
`Single()`, `SingleOrDefault()`	`DataConnection.QueryFirst`	Throws if multiple results
`Any()`	`DataConnection.QueryFirst`	Evaluates existence query directly
`Select(x => new { ... })`	FieldSubset projection	Constructs partial SELECTs dynamically

9.8 Scalar & Terminal Methods

ActiveForge supports invoking terminal scalar executors directly on the query, compiling immediately and sending a constrained scalar demand to the DB.

// Returns scalar INT directly
int count = conn.Query<Product>().Where(p => p.IsActive).Count();

// Returns scalar Bool (Exists check)
bool hasAny = conn.Query<Product>().Where(p => p.Price > 100).Any();

// Retrieves the TOP 1 entity
var topItem = conn.Query<Product>().OrderBy(p => p.Price).FirstOrDefault();

9.9 Projections (Select)

Anonymous type projection parses requested properties to prune the retrieved columns securely at the database level by evaluating a tailored FieldSubset.

// The SQL executed will ONLY 'SELECT p.Id, p.Name FROM Products p'
var lightweightList = conn.Query<Product>()
    .Where(p => p.IsActive)
    .Select(p => new {
        p.ID,
        p.Name
    })
    .ToList();

9.10 Limitations

Limitation	Workaround
No `GroupBy`	Use raw SQL (`ExecSQL`)
No `Join`	Use embedded `Record` fields
No async	Async planned for a future release

9.11 Mixing LINQ with QueryTerm

OrmQueryable<Product> orm = (OrmQueryable<Product>)conn.Query<Product>()
    .Where(p => p.InStock == true);

// Combine accumulated term with additional classic terms:
var template = orm.Template;
QueryTerm combined = orm.WhereTerm & new ContainsTerm(template, template.Name, "premium");

// Execute via classic API:
var results = conn.QueryAll(template, combined, null, 0, null);

10. Transactions (Manual API)

10.1 Explicit Transaction

conn.BeginTransaction();
try
{
    product.Insert();
    orderLine.Insert();
    conn.CommitTransaction();
}
catch
{
    conn.RollbackTransaction();
    throw;
}

10.2 Isolation Levels

using System.Data;

conn.BeginTransaction(IsolationLevel.Serializable);
// ... work ...
conn.CommitTransaction();

Available: ReadUncommitted, ReadCommitted (default), RepeatableRead, Serializable, Snapshot.

10.3 Nested Transactions

ActiveForge uses a depth counter internally. Outer BeginTransaction starts the real ADO.NET transaction; inner calls increment depth only:

conn.BeginTransaction();            // depth: 0→1, real tx starts
    conn.BeginTransaction();        // depth: 1→2, reuses same tx
    conn.CommitTransaction();       // depth: 2→1, no DB commit yet
conn.CommitTransaction();           // depth: 1→0, real COMMIT

If any inner scope calls RollbackTransaction(), the entire outer transaction will roll back when it unwinds.

10.4 TransactionState

TransactionState state = conn.TransactionState();
// returns: None, Active, Committed, Aborted

11. Unit of Work & Transactions

(Namespace: ActiveForge.Transactions; requires Castle.Core NuGet for interceptors)

11.1 Overview

IUnitOfWork manages connection lifetime and transaction nesting behind a clean abstraction.
BaseUnitOfWork opens the database connection when the first CreateTransaction() call is
made (depth 0 → 1) and closes it when the outermost Commit() or Rollback() completes —
but only if the UoW was the one that opened it. This means no manual Connect() or
Disconnect() calls are required around transactional code.

Castle DynamicProxy's TransactionInterceptor is the sole interceptor; it handles both
connection and transaction lifecycle automatically.

11.2 IUnitOfWork Interface

public interface IUnitOfWork : IDisposable
{
    bool InTransaction { get; }
    BaseTransaction CreateTransaction(IsolationLevel level = IsolationLevel.ReadCommitted);
    void Commit();
    void Rollback();
}

11.3 Provider-specific implementations

Provider	Class	Package
SQL Server	`SqlServerUnitOfWork`	`ActiveForge.SqlServer`
PostgreSQL	`PostgreSQLUnitOfWork`	`ActiveForge.PostgreSQL`
MongoDB	`MongoUnitOfWork`	`ActiveForge.MongoDB`
SQLite	`SQLiteUnitOfWork`	`ActiveForge.SQLite`

// No Connect() needed — BaseUnitOfWork opens the connection on first CreateTransaction()
var conn = new SqlServerConnection("Server=...;Database=...;...");
using IUnitOfWork uow = new SqlServerUnitOfWork(conn);

11.4 With.Transaction

// Action overload — commits on success, rolls back on exception:
With.Transaction(uow, () =>
{
    product.Insert();
    orderLine.Insert();
});

// Func<T> overload — returns a value:
int newId = With.Transaction(uow, () =>
{
    product.Insert();
    return (int)product.ID.GetValue();
});

// Specify isolation level:
With.Transaction(uow, () => { ... }, IsolationLevel.Serializable);

// Shorthands:
With.SerializableTransaction(uow, () => { ... });
With.RepeatableReadTransaction(uow, () => { ... });
With.SnapshotTransaction(uow, () => { ... });

11.5 Nested Transactions

The depth counter is managed by BaseUnitOfWork. Inner With.Transaction calls reuse the existing ADO.NET transaction:

With.Transaction(uow, () =>         // depth 0→1, opens connection + real tx begins
{
    product.Insert();

    With.Transaction(uow, () =>     // depth 1→2, reuses tx
    {
        orderLine.Insert();
    });                             // depth 2→1

});                                 // depth 1→0, COMMIT + connection closed

Rollback semantics: If an inner scope rolls back (exception), _rollbackOnly is set. When the outermost scope tries to commit, it rolls back instead.

11.6 [Transaction] Attribute

Decorate methods with [Transaction] to have them automatically wrapped in a transaction when proxied.
The TransactionInterceptor also opens the connection on the first call (via BaseUnitOfWork)
and closes it on completion — no separate connection-scope attribute is required.

public class ProductService : IProductService, IService
{
    private readonly DataConnection _conn;
    public ProductService(DataConnection conn) { _conn = conn; }

    [Transaction(IsolationLevel.ReadCommitted)]
    public int CreateProduct(string name, decimal price)
    {
        var p = new Product(_conn);
        p.Name.SetValue(name);
        p.Price.SetValue(price);
        p.InStock.SetValue(true);
        _conn.Insert(p);
        return (int)p.ID.GetValue();
        // connection opened before first DB call; commits here; connection closed
    }

    // No [Transaction] — no connection or transaction management
    public int CountProducts()
        => _conn.QueryCount(new Product(_conn));
}

[Transaction] can be placed at method level or class level (applies to all methods).
Methods without the attribute pass through unchanged.
When using an interface proxy (the default for IService services), place [Transaction] on the implementation class — the interceptor reads IInvocation.MethodInvocationTarget.

11.7 Setting Up Castle DynamicProxy Interception

For Arbitrary Service Classes

using Castle.DynamicProxy;

var conn        = new SqlServerConnection("...");
using IUnitOfWork uow  = new SqlServerUnitOfWork(conn);
var interceptor = new TransactionInterceptor(uow);
var generator   = new ProxyGenerator();

ProductService real  = new ProductService(conn);
ProductService proxy = (ProductService)generator.CreateClassProxyWithTarget(
    typeof(ProductService), real, interceptor);

int id = proxy.CreateProduct("Widget", 9.99m);  // connection opened, transaction committed automatically

IService auto-registration and DI

Services that implement IService are registered by AddServices() as Castle interface
proxies — no virtual keyword required on any method.

// Program.cs
builder.Services
    .AddActiveForgeSqlServer("Server=...;...")
    .AddServices(typeof(Program).Assembly);

// Service definition
public interface IOrderService
{
    Order GetById(int id);
    void  Ship(int orderId);
}

public class OrderService : IOrderService, IService
{
    private readonly DataConnection _conn;
    public OrderService(DataConnection conn) { _conn = conn; }

    // No [Transaction] — no UoW involved, connection auto-managed per read call
    public Order GetById(int id) { ... }

    [Transaction]   // opens connection, begins tx, commits on return, closes connection
    public void Ship(int orderId) { ... }
}

// Controller — injects by interface; proxy is transparent
public class CheckoutController : ControllerBase
{
    public CheckoutController(IOrderService orders) { _orders = orders; }
}

11.8 ActiveForgeServiceLocator (Ambient DI)

// Register a factory:
ActiveForgeServiceLocator.SetUnitOfWorkFactory(() => new SqlServerUnitOfWork(conn));

// Or register a full IServiceProvider:
ActiveForgeServiceLocator.SetProvider(serviceProvider);

// Use without explicit UoW argument:
With.Transaction(() =>
{
    product.Insert();
});

// Reset (e.g., in tests):
ActiveForgeServiceLocator.Reset();

11.9 Error Handling

Scenario	Behaviour
Exception inside `With.Transaction`	`Rollback()` called; exception re-thrown
Inner scope rolls back	`_rollbackOnly` set; outer scope will roll back even if it tries to commit
`Dispose()` with open transaction	Transaction is rolled back; connection closed if UoW owns it
`Commit()` when `_rollbackOnly`	Silently converts to `Rollback()`
`CreateTransaction()` when already active	Depth incremented; no new ADO.NET transaction

11.10 Standalone (no DI)

var conn = new SqlServerConnection("Server=...;...");
var uow  = new SqlServerUnitOfWork(conn);
var svc  = ActiveForgeServiceFactory.Create<IOrderService>(new OrderService(conn), conn, uow);

svc.Ship(42);   // proxy begins transaction (opening connection), executes, commits, closes

11.11 Connection-level lifecycle without a proxy

For code that doesn't go through a service proxy, assign UnitOfWork on the connection once. Every
write operation (Insert, Update, Delete, ProcessActionQueue, ExecStoredProcedure)
automatically opens the connection, begins a transaction, commits, and closes:

var conn = new SqlServerConnection("...");
var uow  = new SqlServerUnitOfWork(conn);
conn.UnitOfWork = uow;   // wire once

var product = new Product(conn);
product.Name.SetValue("Widget");
product.Price.SetValue(9.99m);
product.Insert();   // opens connection → begins transaction → inserts → commits → closes

Read operations (Read, QueryAll, QueryPage, …) auto-connect and disconnect but do not start
a transaction regardless of whether UnitOfWork is set.

Proxy strategies summary

Scenario	Proxy type	Requirements
`IService` + `AddServices()` (auto-scan)	`CreateInterfaceProxyWithTarget`	None — no virtual required
`AddService<TInterface, TImpl>()`	`CreateInterfaceProxyWithTarget`	None
`AddService<TClass>()` (class proxy)	`CreateClassProxyWithTarget`	Non-sealed; intercepted methods `virtual`
`ActiveForgeServiceFactory.Create<T>()`	Interface or class proxy (auto-detected)	See above

12. Action Queue (Batch Operations)

The action queue batches DML without executing immediately:

// Queue operations:
product1.QueueForInsert();
product2.QueueForUpdate();
product3.QueueForDelete();

// Execute all queued operations in a single round-trip batch:
conn.ProcessActionQueue();

// Discard without executing:
conn.ClearActionQueue();

Use case: High-throughput imports, bulk updates, or deferred persistence.

13. Field Subsets (Partial Fetch / Update)

FieldSubset specifies which columns are included in a SELECT or UPDATE.

13.1 Creating Subsets

var template = new Product(conn);

FieldSubset all     = conn.FieldSubset(template, FieldSubsetInitialState.IncludeAll);
FieldSubset none    = conn.FieldSubset(template, FieldSubsetInitialState.ExcludeAll);
FieldSubset dflt    = conn.DefaultFieldSubset(template);

// Single-field subset:
FieldSubset nameOnly = conn.FieldSubset(template, FieldSubsetInitialState.ExcludeAll);
nameOnly += template.Name;  // add Name column only

13.2 Composing Subsets

FieldSubset base1 = conn.FieldSubset(template, FieldSubsetInitialState.IncludeAll);
FieldSubset base2 = conn.FieldSubset(template, FieldSubsetInitialState.ExcludeAll);
base2 += template.Price;

FieldSubset union        = base1 | base2;  // union
FieldSubset intersection = base1 & base2;  // intersection
FieldSubset removed      = base1 - base2;  // difference

13.3 Partial Fetch (SELECT)

FieldSubset subset = conn.FieldSubset(template, FieldSubsetInitialState.ExcludeAll);
subset += template.Name;
subset += template.Price;

// QueryAll with subset — only Name and Price columns are fetched:
var results = conn.QueryAll(template, null, null, 0, subset);

13.4 Partial Update

// UpdateChanged() only updates fields that changed since the initial snapshot:
product.Price.SetValue(14.99m);   // only Price changed
product.UpdateChanged();           // generates: UPDATE Products SET Price=@Price WHERE ID=@ID

// Or queue for partial update with explicit subset:
product.QueueForUpdate(priceSubset);
conn.ProcessActionQueue();

13.5 InitialState Values

Value	Meaning
`IncludeAll`	All columns included
`ExcludeAll`	No columns included
`Default`	ORM-defined default (usually all non-identity columns)
`IncludeAllJoins`	Include all joined/embedded object columns
`ExcludeAllJoins`	Exclude all joined/embedded object columns

14. Field Encryption

14.1 Marking a Field for Encryption

[Table("Customers")]
public class Customer : IdentityRecord
{
    [Column("SSN")]
    [Encrypted]                          // transparent encrypt/decrypt
    public TString SSN = new TString();

    public Customer() { }
    public Customer(DataConnection conn) : base(conn) { }
}

14.2 Providing an Encryption Algorithm

Implement IEncryptionAlgorithm (or EncryptionAlgorithm) and register it with the connection:

public class AesEncryption : EncryptionAlgorithm
{
    private readonly byte[] _key;
    public AesEncryption(byte[] key) { _key = key; }

    public override byte[] Encrypt(byte[] plaintext) { /* AES encrypt */ }
    public override byte[] Decrypt(byte[] ciphertext) { /* AES decrypt */ }
}

// Register:
conn.SetEncryptionAlgorithm(new AesEncryption(myKey));

Once registered, all [Encrypted] fields are transparently encrypted on write and decrypted on read.

15. Custom Field Mappers

Implement IDBFieldMapper to handle non-standard CLR ↔ DB type conversions:

public class MoneyMapper : IDBFieldMapper
{
    // Called when reading from DB:
    public object MapFromDB(object dbValue, TField field)
    {
        if (dbValue is long cents)
            return cents / 100m;
        return dbValue;
    }

    // Called when writing to DB:
    public object MapToDB(object clrValue, TField field)
    {
        if (clrValue is decimal d)
            return (long)(d * 100);
        return clrValue;
    }
}

16. Polymorphic Mapping (BaseFactory)

BaseFactory maps abstract base types to concrete subtypes. The ORM uses the map when it needs to instantiate an object during query hydration.

Static type substitution — the common case:

[Table("shapes")]
public abstract class Shape : IdentityRecord
{
    [Column("kind")]   public TString  Kind   = new TString();
    [Column("colour")] public TString  Colour = new TString();
    protected Shape() { }
    protected Shape(DataConnection conn) : base(conn) { }
}

[Table("shapes")]
public class Circle : Shape
{
    [Column("radius")] public TDecimal Radius = new TDecimal();
    public Circle() { }
    public Circle(DataConnection conn) : base(conn) { }
}

[Table("shapes")]
public class Rectangle : Shape
{
    [Column("width")]  public TDecimal Width  = new TDecimal();
    [Column("height")] public TDecimal Height = new TDecimal();
    public Rectangle() { }
    public Rectangle(DataConnection conn) : base(conn) { }
}

// Factory: always map Shape → Circle when Circle is the only concrete subtype
public class ShapeFactory : BaseFactory
{
    protected override void CreateTypeMap()
    {
        AddTypeMapping(typeof(Shape), typeof(Circle));
    }
}

var conn = new SqlServerConnection(connectionString, new ShapeFactory());

Query using the abstract type — the factory substitutes Circle transparently:

var template = new Shape(conn);          // Shape is the query template
var circles  = conn.QueryAll(template, null, null, 0, null);
// Each result is a Circle instance cast to Shape
foreach (Shape s in circles)
    Console.WriteLine($"Radius: {((Circle)s).Radius.GetValue()}");

Multiple concrete types — use expectedTypes parameter to hydrate different subtypes in one query:

// Pass both concrete types; the ORM routes each row to the correct type via AddTypeMapping
var results = conn.QueryAll(
    new Shape(conn),
    null, null, 0,
    new[] { typeof(Circle), typeof(Rectangle) },
    null);

foreach (Shape s in results)
{
    if (s is Circle c)         Console.WriteLine($"Circle   radius={c.Radius.GetValue()}");
    else if (s is Rectangle r) Console.WriteLine($"Rectangle {r.Width.GetValue()}×{r.Height.GetValue()}");
}

17.1 Optimistic Locking

RecordLock.UpdateOption controls what happens when another process has modified the row:

// Default — throws ObjectLockException if row was modified elsewhere:
product.Update();

// Ignore lock — always overwrite:
product.Update(RecordLock.UpdateOption.IgnoreLock);

// Release lock after update:
product.Update(RecordLock.UpdateOption.ReleaseLock);

// Retain lock after update:
product.Update(RecordLock.UpdateOption.RetainLock);

17.2 Pessimistic Locking (ReadForUpdate)

Acquires a row-level update lock (SQL Server UPDLOCK, PostgreSQL FOR UPDATE) within a transaction to block other writers until commit.

using var tx = conn.BeginTransaction();

var product = new Product(conn);
product.ID.SetValue(42);
conn.ReadForUpdate(product, null); // Blocks other sessions

product.Price.SetValue(20.00m);
product.Update();

conn.CommitTransaction(tx); // Lock released

Handle lock conflicts:

try
{
    product.Update();
}
catch (ObjectLockException ex)
{
    Console.WriteLine($"Conflict: {ex.Message}");
    // re-read and retry...
}

18. Lazy Streaming

LazyQueryAll<T> returns an IEnumerable<T> that streams rows one at a time — no full in-memory buffer:

var template = new Product(conn);
var term = new EqualTerm(template, template.InStock, true);

foreach (Product p in conn.LazyQueryAll<Product>(template, term, null))
{
    // Processed one row at a time — ideal for large result sets.
    Console.WriteLine(p.Name.GetValue());
}

Equivalent via LINQ (without Take/Skip):

foreach (Product p in conn.Query<Product>().Where(p => p.InStock == true))
{
    Console.WriteLine(p.Name.GetValue());
}

19. Raw SQL and Stored Procedures

19.1 ExecSQL (Direct Results)

Executes raw SQL and returns a BaseReader for manual iteration.

using var reader = conn.ExecSQL("SELECT COUNT(*) FROM Products");
if (reader.Read())
{
    int count = (int)reader.ColumnValue(0);
}

19.2 ExecSQL (Typed Mapping)

Maps raw SQL results directly to Record instances using a template.

var template = new Product(conn);
var results  = conn.ExecSQL(template, "SELECT * FROM Products WHERE Price > 100");

foreach (Product p in results)
{
    Console.WriteLine(p.Name.GetValue());
}

19.3 Stored Procedures

Executes a command set to CommandType.StoredProcedure.

var pCategoryId = new Record.SPParameter { Name = "CategoryId", Value = 5 };
var pMaxPrice   = new Record.SPParameter { Name = "MaxPrice",   Value = 100m };

var template = new Product(conn);
var results  = conn.ExecStoredProcedure(template, "GetProductsByCategory", 0, 0, pCategoryId, pMaxPrice);

20. Lookup / Cached Reference Tables

LookupRecord caches its rows after the first load, suitable for small reference tables:

[Table("Categories")]
public class Category : LookupRecord
{
    [Column("Name")] public TString Name = new TString();
    public Category() { }
    public Category(DataConnection conn) : base(conn) { }
}

// First call loads all rows; subsequent calls return cached data:
var categories = conn.QueryAll(new Category(conn), null, null, 0, null);

21. Architecture Deep Dive

21.0 Assembly Boundaries

ActiveForge (core — no provider dependency)
├── Record / IdentityRecord / LookupRecord
├── DataConnection (abstract) / DBDataConnection (abstract)
├── TField subtypes (25+)
├── QueryTerm tree (EqualTerm, AndTerm, InTerm, …)
├── LINQ layer (OrmQueryable, ExpressionToQueryTermVisitor, …)
├── Adapter abstractions (BaseConnection, BaseCommand, BaseReader, BaseTransaction)
└── Transactions (IUnitOfWork, BaseUnitOfWork, With, TransactionInterceptor, …)

ActiveForge.SqlServer (depends on ActiveForge + Microsoft.Data.SqlClient)
├── SqlServerConnection : DBDataConnection
├── Adapters/SqlAdapterConnection     (wraps SqlConnection)
├── Adapters/SqlAdapterCommand        (wraps SqlCommand)
├── Adapters/SqlAdapterReader         (wraps SqlDataReader)
├── Adapters/SqlAdapterTransaction    (wraps SqlTransaction)
└── Transactions/SqlServerUnitOfWork  : BaseUnitOfWork

ActiveForge.PostgreSQL (depends on ActiveForge + Npgsql)
├── PostgreSQLConnection : DBDataConnection
├── Adapters/NpgsqlAdapterConnection  (wraps NpgsqlConnection)
├── Adapters/NpgsqlAdapterCommand     (wraps NpgsqlCommand)
├── Adapters/NpgsqlAdapterReader      (wraps NpgsqlDataReader)
├── Adapters/NpgsqlAdapterTransaction (wraps NpgsqlTransaction)
└── Transactions/PostgreSQLUnitOfWork : BaseUnitOfWork

ActiveForge.MongoDB (depends on ActiveForge + MongoDB.Driver)
├── MongoDataConnection : DataConnection   ← extends DataConnection directly (not DBDataConnection)
├── Internal/MongoFieldDescriptor          (per-field BSON name cache)
├── Internal/MongoTypeCache                (per-type collection name + field descriptors)
├── Internal/MongoMapper                   (Record ↔ BsonDocument serialization)
├── Internal/MongoQueryTranslator          (QueryTerm → FilterDefinition, SortOrder → SortDefinition)
└── Transactions/MongoUnitOfWork : BaseUnitOfWork

All provider types use the ActiveForge namespace — the same namespace as the core types they extend. This means consuming code only needs using ActiveForge;.

MongoDB vs SQL architecture: SQL providers extend DBDataConnection, which inherits DataConnection and adds SQL generation, ADO.NET adapter management, and the ObjectBinding cache. MongoDataConnection extends DataConnection directly because there is no SQL to generate. It builds its own minimal ObjectBinding from reflection for QueryTerm lookup, and performs all CRUD via MongoDB.Driver API calls.

21.1 ObjectBinding — Reflection Cache

ObjectBinding is the ORM's per-type reflection cache. It holds:

The list of TField FieldInfo objects decorated with [Column]
The [Table] name
Identity field info

RecordCollection : List<Record> is returned by bulk query methods.

21.2 OrmQueryable State Machine

The LINQ pipeline accumulates state immutably in OrmQueryable<T>:

OrmQueryable<T>
├── Connection    : DataConnection
├── Template      : T              (template Record instance)
├── WhereTerm     : QueryTerm?     (accumulated AND tree)
├── SortOrder     : SortOrder?     (primary; CombinedSortOrder for multi-column)
├── PageSize      : int?           (Take)
└── SkipCount     : int?           (Skip)

Each LINQ operator creates a new OrmQueryable<T> with updated state and sets its Expression property to the incoming MethodCallExpression — enabling correct recursive chain rebuilding.

21.3 ExpressionToQueryTermVisitor

Translates a LambdaExpression predicate into a QueryTerm tree:

BinaryExpression (AndAlso) → AndTerm
BinaryExpression (OrElse) → OrTerm
UnaryExpression (Not) → NotTerm
BinaryExpression (Equal, null) → IsNullTerm
BinaryExpression (Equal, value) → EqualTerm
BinaryExpression (NotEqual, null) → !IsNullTerm
BinaryExpression (NotEqual, value) → !EqualTerm
BinaryExpression (GreaterThan) → GreaterThanTerm
MethodCallExpression (Contains on List<T>) → InTerm

Local variable capture is handled by compiling and invoking the captured sub-expression: Expression.Lambda(expr).Compile().DynamicInvoke().

21.4 BaseUnitOfWork Depth Counter and Connection Ownership

State: _depth = 0, _rollbackOnly = false, _currentTransaction = null,
       _ownedConnection = false

CreateTransaction() called:
  if _depth == 0:
    _ownedConnection = !_connection.IsOpen
    if _ownedConnection: _connection.Connect()
    BeginTransactionCore(level) → _currentTransaction
  _depth++

Commit() called:
  _depth--
  if _depth == 0:
    if _rollbackOnly: CommitOrRollback(commit: false)
    else:             CommitOrRollback(commit: true)

Rollback() called:
  _rollbackOnly = true
  _depth--
  if _depth == 0: CommitOrRollback(commit: false)

CommitOrRollback(commit):
  try:
    if commit: _currentTransaction.Commit()
    else:      _currentTransaction.Rollback()
  finally:
    _currentTransaction.Dispose(); _currentTransaction = null
    _rollbackOnly = false
    notify connection (NotifyTransactionCommitted / NotifyTransactionRolledBack)
    if _ownedConnection: _connection.Disconnect(); _ownedConnection = false

Dispose():
  if _depth > 0: _currentTransaction.Rollback() [swallow exceptions]
  _currentTransaction?.Dispose()
  if _ownedConnection: _connection.Disconnect() [swallow exceptions]

21.5 CombinedSortOrder

When multiple ThenBy/ThenByDescending calls are chained, CombinedSortOrder wraps primary and secondary sorts:

public class CombinedSortOrder : SortOrder
{
    public override string GetSQL(ObjectBinding binding)
        => _primary.GetSQL(binding) + ", " + _secondary.GetSQL(binding);
}

22. Quick Reference Cheat Sheet

Entity Definition

[Table("TableName")]
public class MyEntity : IdentityRecord
{
    [Column("ColA")] public TString  ColA = new TString();
    [Column("ColB")] public TDecimal ColB = new TDecimal();
    [Column("ColC")] public TBool    ColC = new TBool();

    public MyEntity() { }
    public MyEntity(DataConnection conn) : base(conn) { }
}

CRUD

var e = new MyEntity(conn);
e.ColA.SetValue("hello");
e.Insert();                    // INSERT

e.ID.SetValue(1);
e.Read();                      // SELECT by PK

e.ColB.SetValue(9.99m);
e.Update();                    // UPDATE all columns
e.UpdateChanged();             // UPDATE changed columns only

e.Delete();                    // DELETE by PK

QueryTerm

var t = new MyEntity(conn);
QueryTerm q = new EqualTerm(t, t.ColA, "hello")
            & new GreaterThanTerm(t, t.ColB, 5m);
var rows = conn.QueryAll(t, q, new OrderAscending(t, t.ColA), 0, null);

LINQ

using ActiveForge.Linq;

var rows = conn.Query<MyEntity>()
    .Where(e => e.ColA == "hello" && e.ColB > 5m)
    .OrderBy(e => e.ColA)
    .Skip(0).Take(20)
    .ToList();

Unit of Work

using IUnitOfWork uow = new SqlServerUnitOfWork(conn);
With.Transaction(uow, () =>
{
    e.Insert();
    e2.Insert();
});

[Transaction] Attribute

[Transaction]
public virtual void DoWork() { ... }

// Wire up with Castle:
var proxy = generator.CreateClassProxyWithTarget(typeof(MyService), real, interceptor);
proxy.DoWork();  // auto-commits

Pagination

var page = conn.QueryPage(template, term, sort, startRecord: 0, pageSize: 20);
// page.IsMoreData, page.TotalRowCount

Field Subset

FieldSubset subset = conn.FieldSubset(template, FieldSubsetInitialState.ExcludeAll);
subset += template.ColA;
var rows = conn.QueryAll(template, null, null, 0, subset);

Lazy Stream

foreach (MyEntity e in conn.LazyQueryAll<MyEntity>(template, term, null))
    Console.WriteLine(e.ColA.GetValue());

23. MongoDB Provider

23.1 Overview

MongoDataConnection brings the standard ActiveForge CRUD and query API to MongoDB. It extends DataConnection directly (not DBDataConnection, which is SQL-specific) and uses the MongoDB.Driver 2.28.0 library.

Key differences from SQL providers:

Aspect	SQL Providers	MongoDB
Base class	`DBDataConnection` (SQL generation)	`DataConnection` (no SQL)
Storage	Relational tables	BSON documents in collections
`[Table]` attribute	SQL table name	Collection name
`[Column]` attribute	Column name	BSON field name
`[Identity]` field	Auto-increment int (DB-generated)	`_id` (int via counter collection)
Query translation	SQL `WHERE` clause	`FilterDefinition<BsonDocument>`
Sort translation	SQL `ORDER BY`	`SortDefinition<BsonDocument>`
Transactions	RDBMS transactions	`IClientSessionHandle` (requires replica set)
`ExecSQL` / `ExecStoredProcedure`	Supported	`NotSupportedException`

23.2 Connecting

using ActiveForge;

var conn = new MongoDataConnection(
    connectionString: "mongodb://localhost:27017",
    databaseName:     "myDatabase");
conn.Connect();

23.3 Defining Entities

Entity definitions are provider-agnostic — use the same [Table] / [Column] / [Identity] attributes:

using ActiveForge;
using ActiveForge.Attributes;

[Table("products")]
public class Product : IdentityRecord
{
    [Column("name")]     public TString  Name    = new TString();
    [Column("price")]    public TDecimal Price   = new TDecimal();
    [Column("in_stock")] public TBool    InStock = new TBool();

    public Product() { }
    public Product(DataConnection conn) : base(conn) { }
}

The [Identity] field (added by IdentityRecord) maps to MongoDB's _id field stored as int32. Auto-increment is simulated via an __activeforge_counters collection.

23.4 CRUD

// INSERT (auto-generates _id)
var p = new Product(conn);
p.Name.SetValue("Widget");
p.Price.SetValue(9.99m);
p.InStock.SetValue(true);
p.Insert();
// p.ID now has the generated _id value

// READ by primary key
var p2 = new Product(conn);
p2.ID.SetValue(1);
bool found = p2.Read();

// UPDATE
p.Price.SetValue(14.99m);
p.Update(RecordLock.UpdateOption.IgnoreLock);

// DELETE by primary key
p.Delete();

// DELETE by query
var template = new Product(conn);
var term = new EqualTerm(template, template.InStock, false);
template.Delete(term);

23.5 Querying

The full QueryAll, QueryFirst, QueryCount, QueryPage, and LazyQueryAll methods are supported. QueryTerms are translated to FilterDefinition<BsonDocument> internally:

var template = new Product(conn);

// All in-stock products
var inStock  = new EqualTerm(template, template.InStock, true);
var results  = conn.QueryAll(template, inStock, null, 0, null);

// Price range
var lowPrice  = new GreaterThanTerm(template, template.Price, 5.00m);
var highPrice = new LessThanTerm(template, template.Price, 50.00m);
var range     = conn.QueryAll(template, lowPrice & highPrice, null, 0, null);

// Count
int count = conn.QueryCount(template, inStock);

// Paged results (skip 20, take 10)
var page = conn.QueryPage(template, inStock, null, 20, 10, null);

// Streaming
foreach (Product item in conn.LazyQueryAll<Product>(template, inStock, null, 0, null))
    Console.WriteLine(item.Name.GetValue());

23.6 Supported QueryTerms

QueryTerm	MongoDB translation
`EqualTerm`	`Filter.Eq(field, value)`
`GreaterThanTerm`	`Filter.Gt(field, value)`
`LessThanTerm`	`Filter.Lt(field, value)`
`IsNullTerm`	`Filter.Eq(field, BsonNull.Value)`
`InTerm`	`Filter.In(field, values)`
`ContainsTerm`	`Filter.Regex(field, /value/i)`
`AndTerm` (`&`)	`Filter.And(left, right)`
`OrTerm` (`\	`)
`NotTerm` (`!`)	`Filter.Not(inner)`

23.7 Transactions

MongoDB multi-document transactions require a replica set or sharded cluster. On a standalone mongod they are not supported.

using IUnitOfWork uow = new MongoUnitOfWork(conn);

With.Transaction(uow, () =>
{
    product.Status.SetValue("Shipped");
    product.Update(RecordLock.UpdateOption.IgnoreLock);
    shipment.Insert();
});

For single-document operations no transaction is needed — MongoDB's document model guarantees atomicity per document.

23.8 Unsupported Features

The following operations throw NotSupportedException on MongoDataConnection:

ExecSQL(...) — no SQL dialect
ExecStoredProcedure(...) — MongoDB has no stored procedures
GetDynamicObjectBinding(...) — SQL-specific reader-based binding
GetTargetFieldInfo(string, string, string) — SQL-specific schema introspection
ExistsTerm / GenerateExistsSQLQuery — SQL sub-query syntax
LINQ ExistsTerm (translated via SQL correlated subquery)

The LINQ query interface (conn.Query<T>()) is not supported because its translation layer generates SQL expressions.

24. SQLite Provider

24.1 Overview

SQLiteConnection extends DBDataConnection and adds the SQLite dialect on top of the standard SQL generation pipeline provided by the core. It uses Microsoft.Data.Sqlite 8.0.0.

SQLite specifics compared to the other SQL providers:

Aspect	SQL Server / PostgreSQL	SQLite
Name quoting	`[…]` / `"…"`	`"…"` (double quotes)
Row limiting	`SELECT TOP N …` / `LIMIT N`	`SELECT … LIMIT N` (appended)
Identity after INSERT	`SCOPE_IDENTITY()` / `LASTVAL()`	`last_insert_rowid()`
Schema introspection	`SYSOBJECTS` / `information_schema`	`PRAGMA table_info(table)`
IDENTITY_INSERT toggle	`SET IDENTITY_INSERT … ON/OFF`	Not required (empty string)
Update lock hint	`WITH (UPDLOCK)` / `FOR UPDATE`	Not applicable (empty string)
String concatenation	`+` / `\	\
Stored procedures	Supported	{% raw %}`NotSupportedException`
In-memory databases	Not supported	`Data Source=:memory:`

24.2 Connecting

using ActiveForge;

// File-based database
var conn = new SQLiteConnection("Data Source=app.db");
conn.Connect();

// In-memory database (connection must stay open; destroyed when closed)
var conn = new SQLiteConnection("Data Source=:memory:");
conn.Connect();

// Named shared-cache in-memory (can be reopened)
var conn = new SQLiteConnection("Data Source=mydb;Mode=Memory;Cache=Shared");
conn.Connect();

24.3 Schema Setup

SQLite does not auto-create tables. Create the schema before inserting data:

conn.ExecSQL(
    "CREATE TABLE IF NOT EXISTS products (" +
    "  id       INTEGER PRIMARY KEY AUTOINCREMENT," +
    "  name     TEXT    NOT NULL," +
    "  price    NUMERIC NOT NULL DEFAULT 0," +
    "  in_stock INTEGER NOT NULL DEFAULT 1)");

INTEGER PRIMARY KEY in SQLite is an alias for the internal rowid — it auto-increments without AUTOINCREMENT. INTEGER PRIMARY KEY AUTOINCREMENT prevents rowid reuse after deletes.

24.4 Defining Entities

Entity definitions are provider-agnostic — the same class works with any provider:

using ActiveForge;
using ActiveForge.Attributes;

[Table("products")]
public class Product : IdentityRecord
{
    [Column("name")]     public TString  Name    = new TString();
    [Column("price")]    public TDecimal Price   = new TDecimal();
    [Column("in_stock")] public TBool    InStock = new TBool();

    public Product() { }
    public Product(DataConnection conn) : base(conn) { }
}

Naming convention: SQLite is case-insensitive for identifiers by default, but it
stores names as given. Using lower-case [Table] and [Column] values is safest
for portability.

24.5 CRUD

// INSERT — ID populated via last_insert_rowid()
var p = new Product(conn);
p.Name.SetValue("Widget");
p.Price.SetValue(9.99m);
p.InStock.SetValue(true);
p.Insert();
// p.ID now holds the generated rowid

// READ by primary key
var p2 = new Product(conn);
p2.ID.SetValue(1);
bool found = p2.Read();

// UPDATE
p.Price.SetValue(14.99m);
p.Update(RecordLock.UpdateOption.IgnoreLock);

// DELETE by primary key
p.Delete();

// DELETE by query
var template = new Product(conn);
var term = new EqualTerm(template, template.InStock, false);
template.Delete(term);

24.6 Transactions

SQLite supports ReadCommitted and Serializable isolation levels natively. Other levels are mapped to the nearest supported equivalent by SQLiteAdapterConnection:

Requested level	Mapped to
`ReadUncommitted`	`ReadCommitted`
`RepeatableRead`	`Serializable`
`Snapshot`	`Serializable`
`ReadCommitted`	`ReadCommitted` (no mapping)
`Serializable`	`Serializable` (no mapping)

// Manual transaction
conn.BeginTransaction();
try
{
    order.Status.SetValue("Shipped");
    order.Update(RecordLock.UpdateOption.IgnoreLock);
    shipment.Insert();
    conn.CommitTransaction();
}
catch
{
    conn.RollbackTransaction();
    throw;
}

// With.Transaction helper
var uow = new SQLiteUnitOfWork(conn);
With.Transaction(uow, () =>
{
    order.Status.SetValue("Shipped");
    order.Update(RecordLock.UpdateOption.IgnoreLock);
    shipment.Insert();
});

// Automatic via conn.UnitOfWork
conn.UnitOfWork = new SQLiteUnitOfWork(conn);
product.Insert();  // auto: connect → begin tx → insert → commit → disconnect

24.7 DI Registration

// Program.cs
builder.Services
    .AddActiveForgeSQLite("Data Source=app.db")
    .AddServices(typeof(Program).Assembly);

Or with an in-memory database for testing:

builder.Services
    .AddActiveForgeSQLite("Data Source=testdb;Mode=Memory;Cache=Shared");

24.8 Type Affinity Mapping

SQLite uses type affinity rather than strict types. SQLiteConnection.MapNativeType applies the following mapping rules (first match wins):

Declared type contains	Mapped CLR type
`INT`	`long`
`REAL`, `FLOA`, `DOUB`	`double`
`NUM`, `DEC`, `MONEY`	`decimal`
`BOOL`	`bool`
`DATE`, `TIME`	`DateTime`
`GUID`, `UUID`	`Guid`
`BLOB` or empty	`byte[]`
Anything else (TEXT, VARCHAR, …)	`string`

24.9 Limitations

Stored procedures — SQLite has no stored procedure support. Calling ExecStoredProcedure throws NotSupportedException.
GetUpdateLock() — SQLite uses file-level locking, not row-level locks. The method returns an empty string (no hint appended to SELECT).
IDENTITY_INSERT toggle — not needed; PreInsertIdentityCommand/PostInsertIdentityCommand return empty strings.
Isolation level mapping — levels not supported by SQLite are silently promoted (see §24.6).
In-memory lifetime — a Data Source=:memory: connection is destroyed when it closes. Use a named shared-cache string (Mode=Memory;Cache=Shared) when the connection must be reopened or shared.

25. Attributes Reference

Every attribute lives in the ActiveForge.Attributes namespace (except [Transaction] which is in ActiveForge.Transactions). Add using ActiveForge.Attributes; to any file that uses them.

25.1 Entity / Class Attributes

`[Table]`

Maps a class to a database table, view, or MongoDB collection.

[Table("products")]
public class Product : IdentityRecord { ... }

Parameter	Type	Description
`sourceName`	`string`	Table / collection / view name as it appears in the database

PostgreSQL: fold to lower-case. MongoDB: used verbatim as collection name.

`[BaseTable]`

Used in multi-table inheritance hierarchies. Marks the root table that the concrete class's rows ultimately belong to. Applied to the abstract base class alongside [Table].

// Abstract base stores shared columns in "employees"
[Table("employees")]
[BaseTable("employees")]
public abstract class Employee : IdentityRecord
{
    [Column("name")]       public TString  Name     = new TString();
    [Column("hire_date")]  public TDateTime HireDate = new TDateTime();

    protected Employee() { }
    protected Employee(DataConnection conn) : base(conn) { }
}

// Concrete type extends with columns in the same "employees" table
[Table("employees")]
public class Manager : Employee
{
    [Column("department")] public TString Department = new TString();
    public Manager() { }
    public Manager(DataConnection conn) : base(conn) { }
}

`[Computed]`

Marks a class as representing a derived table (a computed view or joined projection) in a multi-table inheritance hierarchy. The ORM skips DDL generation for it and treats it as read-only.

[Table("v_product_summary")]
[Computed]
public class ProductSummary : Record
{
    [Column("name")]        public TString  Name      = new TString();
    [Column("total_sold")]  public TInt     TotalSold = new TInt();
    [Column("revenue")]     public TDecimal Revenue   = new TDecimal();

    public ProductSummary() { }
    public ProductSummary(DataConnection conn) : base(conn) { }
}

`[Function]`

Marks a class as mapping to a table-valued function rather than a table or view. The ORM passes parameters to the function call instead of issuing a plain SELECT.

[Table("fn_products_by_category")]
[Function]
public class ProductByCategory : IdentityRecord
{
    [Column("name")]  public TString  Name  = new TString();
    [Column("price")] public TDecimal Price = new TDecimal();

    [ParameterPosition(0)] public TInt CategoryId = new TInt();

    public ProductByCategory() { }
    public ProductByCategory(DataConnection conn) : base(conn) { }
}

25.2 Field Mapping Attributes

`[Column]`

Maps a TField instance field to a database column (or BSON field for MongoDB).

[Column("product_name")]
public TString Name = new TString();

Parameter	Type	Description
`columnName`	`string`	Column name as it appears in the database

Every TField that should be persisted must have [Column]. Fields without it are ignored by the ORM.

`[Identity]`

Marks a field as an auto-generated primary key. IdentityRecord applies it implicitly to its ID: TPrimaryKey field. Use explicitly only when defining a custom PK field on a Record subclass.

// Explicit usage on Record (not IdentityRecord):
[Table("orders")]
public class Order : Record
{
    [Column("order_id")]
    [Identity]
    public TPrimaryKey OrderId = new TPrimaryKey();

    [Column("customer_id")] public TForeignKey CustomerId = new TForeignKey();

    public Order() { }
    public Order(DataConnection conn) : base(conn) { }
}

The ORM never writes this field in INSERT statements; the database generates the value. After Insert() it is populated with the generated key.

`[ReadOnly]`

Marks a column as read-only: included in SELECT queries but never written in INSERT or UPDATE. Suitable for database-computed columns or columns managed by triggers.

[Column("created_at")]
[ReadOnly]
public TDateTime CreatedAt = new TDateTime();

[Column("row_hash")]
[ReadOnly]
public TString RowHash = new TString();  // computed by DB trigger

`[DefaultValue]`

Provides a default value that the ORM assigns when a new entity is constructed (before any SetValue call). Useful to avoid null state on fields that always carry an initial value.

[Column("status")]
[DefaultValue("pending")]
public TString Status = new TString();

[Column("quantity")]
[DefaultValue(1)]
public TInt Quantity = new TInt();

[Column("created_at")]
[DefaultValue(typeof(DateTime))]   // convention: pass Type for "DateTime.UtcNow"
public TDateTime CreatedAt = new TDateTime();

`[NoPreload]`

Prevents a field from being included in the default SELECT. The field is still writable. Use for very large columns (e.g. TEXT blobs) that should only be fetched when explicitly requested via a FieldSubset.

[Column("description")]
[NoPreload]
public TString Description = new TString();   // not fetched in QueryAll by default

[Column("thumbnail")]
[NoPreload]
public TByteArray Thumbnail = new TByteArray();

To fetch a [NoPreload] field, include it in a FieldSubset:

var template = new Product(conn);
FieldSubset full = conn.DefaultFieldSubset(template);
full += template.Description;   // explicitly include the no-preload field
var results = conn.QueryAll(template, null, null, 0, full);

`[NoTrim]`

Prevents the ORM from trimming trailing whitespace when reading a CHAR or VARCHAR column. By default, TString trims trailing spaces on read. Apply this attribute to preserve exact stored values.

[Column("fixed_code")]
[NoTrim]
public TString FixedCode = new TString();   // "ABC   " stored and returned as-is

`[Optional]`

Marks a column as optional: the column may not exist in the target database schema. If the column is absent from the schema introspection result, the ORM skips it silently instead of throwing. Useful when the same entity is used against multiple database versions.

[Column("discount_pct")]
[Optional]
public TDecimal DiscountPct = new TDecimal();   // column may not exist in v1 schema

`[Generator]`

Specifies the name of a database sequence (PostgreSQL, Oracle) or generator that should supply the value for this field on INSERT. Used instead of [Identity] when the PK is fed by a named sequence rather than an IDENTITY / SERIAL column.

[Column("invoice_no")]
[Generator("seq_invoice_no")]
public TInt InvoiceNo = new TInt();

`[ParameterPosition]`

Used together with [Function] on table-valued function entities. Assigns the positional index of a TField when it is passed as a parameter to the function call.

[Table("fn_orders_by_date")]
[Function]
public class OrdersByDate : IdentityRecord
{
    [Column("total")]       public TDecimal Total = new TDecimal();
    [Column("order_date")]  public TDate    OrderDate = new TDate();

    [ParameterPosition(0)]  public TDate FromDate  = new TDate();
    [ParameterPosition(1)]  public TDate ToDate    = new TDate();

    public OrdersByDate() { }
    public OrdersByDate(DataConnection conn) : base(conn) { }
}

`[Encrypted]`

Marks a TString or TByteArray field for transparent encryption at the ORM layer. The raw database value is ciphertext; GetValue() always returns plaintext.

using ActiveForge.Attributes;

[Table("customers")]
public class Customer : IdentityRecord
{
    [Column("name")]   public TString Name = new TString();

    [Column("ssn")]
    [Encrypted(typeof(AesFieldEncryption))]
    public TString Ssn = new TString();

    [Column("card_number")]
    [Encrypted(typeof(AesFieldEncryption), EncryptedAttribute.EncryptionMethodType.PartialEncryption)]
    public TString CardNumber = new TString();   // last 4 digits remain in plain text

    public Customer() { }
    public Customer(DataConnection conn) : base(conn) { }
}

Parameter	Type	Description
`algorithmType`	`Type`	`IEncryptionAlgorithm` implementation to use
`method`	`EncryptionMethodType`	`AllDataEncrypted` (default) or `PartialEncryption`

Implement the encryption algorithm:

public class AesFieldEncryption : EncryptionAlgorithm
{
    private static readonly byte[] Key = /* load from secure config */;

    public override byte[] Encrypt(byte[] plaintext)
    {
        using var aes = Aes.Create();
        aes.Key = Key;
        aes.GenerateIV();
        using var enc = aes.CreateEncryptor();
        var ct = enc.TransformFinalBlock(plaintext, 0, plaintext.Length);
        return aes.IV.Concat(ct).ToArray();          // prepend IV
    }

    public override byte[] Decrypt(byte[] ciphertext)
    {
        using var aes = Aes.Create();
        aes.Key = Key;
        aes.IV = ciphertext[..16];
        using var dec = aes.CreateDecryptor();
        return dec.TransformFinalBlock(ciphertext, 16, ciphertext.Length - 16);
    }
}

`[Compressible]`

Instructs the ORM to compress the column value before writing to the database and decompress on read. Useful for large TEXT or BLOB columns.

[Column("xml_payload")]
[Compressible]
public TString XmlPayload = new TString();

`[Sensitive]`

Marks a field as sensitive (passwords, API keys, PII). The ORM masks the value in diagnostic output, logging, and serialised diagnostic reports. Does not affect the actual stored or returned value.

[Column("api_key")]
[Sensitive]
public TString ApiKey = new TString();

[Column("password_hash")]
[Sensitive]
public TString PasswordHash = new TString();

`[Description]`

Attaches a human-readable description to a field or class. Used by the ORM for validation error messages and UI-hint generation. Inherited by subclasses.

[Table("products")]
[Description("A catalogue item offered for sale")]
public class Product : IdentityRecord
{
    [Column("name")]
    [Description("Display name shown on product listings")]
    public TString Name = new TString();

    [Column("price")]
    [Description("Retail price excluding tax, in GBP")]
    public TDecimal Price = new TDecimal();

    public Product() { }
    public Product(DataConnection conn) : base(conn) { }
}

`[FieldMapping]`

Associates a custom IDBFieldMapper with a field, controlling how its value is transformed when reading from and writing to the database. Overrides the default type mapping.

// Custom mapper: stores money as integer cents
public class CentsMapper : IDBFieldMapper
{
    public object MapFromDB(object dbValue, TField field)
        => dbValue is long cents ? cents / 100m : dbValue;

    public object MapToDB(object clrValue, TField field)
        => clrValue is decimal d ? (long)(d * 100) : clrValue;
}

[Table("invoices")]
public class Invoice : IdentityRecord
{
    [Column("amount_cents")]
    [FieldMapping(typeof(CentsMapper))]
    public TDecimal Amount = new TDecimal();   // app sees decimal; DB stores long

    public Invoice() { }
    public Invoice(DataConnection conn) : base(conn) { }
}

`[EagerLoad]`

Controls whether an embedded Record field (join target) is fetched eagerly (default true) or excluded from the default FieldSubset and loaded only when explicitly included.

[Table("order_lines")]
public class OrderLine : IdentityRecord
{
    [Column("order_id")] public TForeignKey OrderId = new TForeignKey();
    [Column("qty")]      public TInt        Qty     = new TInt();

    // Loaded by default in every QueryAll
    public Order Order;

    // Heavy join — only load when requested
    [EagerLoad(false)]
    public Product Product;

    public OrderLine()                          { Order = new Order(); Product = new Product(); }
    public OrderLine(DataConnection conn) : base(conn)
    {
        Order   = new Order(conn);
        Product = new Product(conn);
    }
}

// To include the lazy join:
FieldSubset fs = conn.DefaultFieldSubset(template);
fs |= conn.FieldSubset(template.Product, FieldSubsetInitialState.IncludeAll);
var rows = conn.QueryAll(template, null, null, 0, fs);

25.3 Join Attributes

`[JoinSpec]`

Declares an explicit SQL JOIN on the entity class. Applied at class level, it overrides or supplements the automatic FK convention join. AllowMultiple = true — stack as many as needed.

[Table("order_lines")]
[JoinSpec("OrderId",    "Order",    "ID", JoinSpecAttribute.JoinTypeEnum.InnerJoin)]
[JoinSpec("ProductId",  "Product",  "ID", JoinSpecAttribute.JoinTypeEnum.LeftOuterJoin)]
public class OrderLine : IdentityRecord
{
    [Column("order_id")]   public TForeignKey OrderId   = new TForeignKey();
    [Column("product_id")] public TForeignKey ProductId = new TForeignKey();
    [Column("qty")]        public TInt        Qty       = new TInt();

    public Order   Order;
    public Product Product;

    public OrderLine()
    {
        Order   = new Order();
        Product = new Product();
    }
    public OrderLine(DataConnection conn) : base(conn)
    {
        Order   = new Order(conn);
        Product = new Product(conn);
    }
}

Parameter	Type	Description
`foreignKeyField`	`string`	Name of the FK field on this entity (e.g. `"OrderId"`)
`targetField`	`string`	Name of the embedded `Record` field on this entity (e.g. `"Order"`)
`targetPrimaryKeyField`	`string`	PK field name on the target entity (default `"ID"`)
`joinType`	`JoinTypeEnum`	`InnerJoin` (default) or `LeftOuterJoin`

`[Join]`

Applied to an embedded Record field (rather than the class). Overrides the automatic FK-naming convention for that specific join. Used primarily with the MongoDB provider or when the FK field name does not follow the <TargetType>ID convention.

[Table("shipments")]
public class Shipment : IdentityRecord
{
    [Column("carrier_ref")] public TForeignKey CarrierRef = new TForeignKey();

    // Convention would look for "CarrierId"; use [Join] to point to "carrier_ref"
    [Join(ForeignKey = "carrier_ref", TargetField = "ID",
          JoinType = JoinAttribute.JoinTypeEnum.LeftOuterJoin)]
    public Carrier Carrier;

    public Shipment()                          { Carrier = new Carrier(); }
    public Shipment(DataConnection conn) : base(conn) { Carrier = new Carrier(conn); }
}

Property	Type	Description
`ForeignKey`	`string`	FK column name on this table
`TargetField`	`string`	PK column name on the joined table
`JoinType`	`JoinTypeEnum`	`InnerJoin` (default) or `LeftOuterJoin`

25.4 Service / Proxy Attributes

`[Transaction]`

Marks a method or class so that TransactionInterceptor wraps the call in an IUnitOfWork
transaction. Commits on successful return; rolls back on exception. BaseUnitOfWork also opens
the connection before the first CreateTransaction() call and closes it when the outermost
commit or rollback completes — no separate connection-scope attribute is needed.

using ActiveForge.Transactions;

public class OrderService : IOrderService, IService
{
    private readonly DataConnection _conn;

    public OrderService(DataConnection conn) { _conn = conn; }

    [Transaction]                                      // ReadCommitted by default
    public int PlaceOrder(int customerId)
    {
        var order = new Order(_conn);
        order.CustomerId.SetValue(customerId);
        order.Status.SetValue("new");
        order.Insert();
        return (int)order.ID.GetValue();
        // connection opened before first DB access; committed + closed here
    }

    [Transaction(IsolationLevel.Serializable)]         // explicit isolation level
    public void CancelOrder(int orderId)
    {
        var order = new Order(_conn);
        order.ID.SetValue(orderId);
        order.Read();
        order.Status.SetValue("cancelled");
        order.Update(RecordLock.UpdateOption.IgnoreLock);
    }
}

Parameter	Type	Description
`isolationLevel`	`IsolationLevel`	Default: `ReadCommitted`. Any `System.Data.IsolationLevel` value

26. Joins and Relationships

ActiveForge expresses table joins through embedded Record fields. When the ORM builds the SELECT query it inspects each public field that is itself a Record and adds the appropriate JOIN clause automatically.

26.1 Convention-based INNER JOIN

The simplest join requires no attributes beyond the FK field declaration. If the embedded field is called Category and there is a column field named CategoryID (or CategoryId), the ORM wires the join automatically:

[Table("categories")]
public class Category : IdentityRecord
{
    [Column("name")] public TString Name = new TString();

    public Category() { }
    public Category(DataConnection conn) : base(conn) { }
}

[Table("products")]
public class Product : IdentityRecord
{
    [Column("name")]        public TString     Name       = new TString();
    [Column("price")]       public TDecimal    Price      = new TDecimal();
    [Column("CategoryID")]  public TForeignKey CategoryID = new TForeignKey();

    // Embedded Record — convention: field name "Category" + "ID" suffix = FK column "CategoryID"
    public Category Category;

    public Product()                          { Category = new Category(); }
    public Product(DataConnection conn) : base(conn) { Category = new Category(conn); }
}

Query — the generated SQL is an INNER JOIN:

var template = new Product(conn);
var results  = conn.QueryAll(template, null, null, 0, null);

foreach (Product p in results)
{
    string productName  = (string)p.Name.GetValue();
    string categoryName = (string)p.Category.Name.GetValue();
    Console.WriteLine($"{productName} ({categoryName})");
}
// Products without a matching Category are excluded (INNER JOIN semantics)

26.2 Explicit JOIN with `[JoinSpec]`

Use [JoinSpec] on the class when the FK column name does not follow the <EmbeddedFieldName>ID convention, or when you want to specify LeftOuterJoin (or RightOuterJoin) at definition time.

[Table("products")]
[JoinSpec("CategoryID", "Category", "ID", JoinSpecAttribute.JoinTypeEnum.InnerJoin)]
public class ProductWithExplicitJoin : IdentityRecord
{
    [Column("name")]        public TString     Name       = new TString();
    [Column("CategoryID")]  public TForeignKey CategoryID = new TForeignKey();

    public Category Category;

    public ProductWithExplicitJoin()
    { Category = new Category(); }
    public ProductWithExplicitJoin(DataConnection conn) : base(conn)
    { Category = new Category(conn); }
}

Multiple joins on one class:

[Table("order_lines")]
[JoinSpec("OrderId",   "Order",   "ID", JoinSpecAttribute.JoinTypeEnum.InnerJoin)]
[JoinSpec("ProductId", "Product", "ID", JoinSpecAttribute.JoinTypeEnum.LeftOuterJoin)]
public class OrderLineWithJoins : IdentityRecord
{
    [Column("order_id")]   public TForeignKey OrderId   = new TForeignKey();
    [Column("product_id")] public TForeignKey ProductId = new TForeignKey();
    [Column("qty")]        public TInt        Qty       = new TInt();
    [Column("unit_price")] public TDecimal    UnitPrice = new TDecimal();

    public Order   Order;
    public Product Product;

    public OrderLineWithJoins()
    {
        Order   = new Order();
        Product = new Product();
    }
    public OrderLineWithJoins(DataConnection conn) : base(conn)
    {
        Order   = new Order(conn);
        Product = new Product(conn);
    }
}

26.3 LEFT OUTER JOIN

To include rows where the joined record is absent (e.g. products without a category), set JoinTypeEnum.LeftOuterJoin:

[Table("products")]
[JoinSpec("CategoryID", "Category", "ID", JoinSpecAttribute.JoinTypeEnum.LeftOuterJoin)]
public class ProductOuter : IdentityRecord
{
    [Column("name")]        public TString     Name       = new TString();
    [Column("CategoryID")]  public TForeignKey CategoryID = new TForeignKey();

    public Category Category;

    public ProductOuter()                          { Category = new Category(); }
    public ProductOuter(DataConnection conn) : base(conn) { Category = new Category(conn); }
}

var template = new ProductOuter(conn);
var results  = conn.QueryAll(template, null, null, 0, null);

foreach (ProductOuter p in results)
{
    // Products with no category will have Category.Name.IsNull() == true
    string cat = p.Category.Name.IsNull() ? "(none)" : (string)p.Category.Name.GetValue();
    Console.WriteLine($"{p.Name.GetValue()} — {cat}");
}

26.4 Query-time Join Type Override (`JoinOverride`)

Override the join type at query time without changing the entity definition. Useful when the same entity needs INNER joins in most queries but LEFT OUTER in specific ones.

using ActiveForge.Query;   // JoinOverride, JoinSpecification

// Force LEFT OUTER on Category join for this query only
var overrides = new List<JoinOverride>
{
    new JoinOverride(typeof(Category), JoinSpecification.JoinTypeEnum.LeftOuterJoin)
};

var template = new Product(conn);   // Product has convention INNER JOIN by default
var results  = conn.QueryAll(template, null, null, 0, null, overrides);
// → returns products WITH and WITHOUT a matching category

// Override back to INNER on an entity that has LEFT OUTER by default
var overrides = new List<JoinOverride>
{
    new JoinOverride(typeof(Category), JoinSpecification.JoinTypeEnum.InnerJoin)
};

var template = new ProductOuter(conn);
var results  = conn.QueryAll(template, null, null, 0, null, overrides);
// → excludes products with no category

The override is not sticky — it applies only to that single QueryAll / QueryPage / LazyQueryAll call and does not affect subsequent queries using the same template.

26.5 Filtering on Joined Columns (QueryTerm API)

Pass the embedded Record (not the root template) as the first argument to QueryTerm constructors to filter on a joined column:

var template = new Product(conn);

// Filter on the joined Category's name
var term    = new EqualTerm(template.Category, template.Category.Name, "Electronics");
var results = conn.QueryAll(template, term, null, 0, null);
// → SELECT ... FROM products INNER JOIN categories ... WHERE categories.name = @name

// Combine root and joined predicates
var inStock     = new EqualTerm(template, template.InStock, true);
var electronics = new EqualTerm(template.Category, template.Category.Name, "Electronics");
var combined    = inStock & electronics;

var results = conn.QueryAll(template, combined, null, 0, null);

26.6 Sorting on Joined Columns

Pass the embedded Record as the first argument to OrderAscending / OrderDescending:

var template = new Product(conn);

// Sort by joined category name, then by product name
var sortByCat  = new OrderAscending(template.Category, template.Category.Name);
var sortByName = new OrderAscending(template, template.Name);

// Single-column sort:
var results = conn.QueryAll(template, null, sortByCat, 0, null);

// Multi-column sort via LINQ:
var results2 = conn.Query(new Product(conn))
    .OrderBy(p => p.Category.Name)
    .ThenBy(p => p.Name)
    .ToList();

26.7 LINQ with Joins

The LINQ query layer fully supports cross-join predicates and sort selectors:

using ActiveForge.Linq;

// Filter on joined field
var electronics = conn.Query(new Product(conn))
    .Where(p => p.Category.Name == "Electronics")
    .ToList();

// Filter for products with no category (LEFT OUTER join required)
var noCategory = conn.Query(new ProductOuter(conn))
    .Where(p => p.Category.Name == (TString)null)
    .ToList();

// Sort by joined field, then by own field
var sorted = conn.Query(new Product(conn))
    .OrderBy(p => p.Category.Name)
    .ThenBy(p => p.Price)
    .ToList();

// Join-type override, filter, sort, and paginate in one chain
var results = conn.Query(new Product(conn))
    .LeftOuterJoin<Category>()               // override INNER → LEFT OUTER
    .Where(p => p.Name != (TString)null)
    .OrderBy(p => p.Category.Name)
    .ThenBy(p => p.Name)
    .Skip(0)
    .Take(20)
    .ToList();

LINQ join-type extension methods:

Method	Effect
`.InnerJoin<TTarget>()`	Forces INNER JOIN for `TTarget` in this query
`.LeftOuterJoin<TTarget>()`	Forces LEFT OUTER JOIN for `TTarget` in this query

26.8 `[Join]` Attribute (field-level, non-conventional FK names)

When the FK column does not follow <EmbeddedFieldName>ID naming, annotate the embedded field directly with [Join]:

[Table("shipments")]
public class Shipment : IdentityRecord
{
    [Column("tracking_no")]    public TString     TrackingNo  = new TString();
    [Column("carrier_ref")]    public TForeignKey CarrierRef  = new TForeignKey();

    // FK column is "carrier_ref", not "CarrierId" — override with [Join]
    [Join(ForeignKey = "carrier_ref", TargetField = "ID",
          JoinType = JoinAttribute.JoinTypeEnum.LeftOuterJoin)]
    public Carrier Carrier;

    public Shipment()                          { Carrier = new Carrier(); }
    public Shipment(DataConnection conn) : base(conn) { Carrier = new Carrier(conn); }
}

26.9 MongoDB Joins

MongoDB uses $lookup + $unwind aggregation pipeline stages for joins. The same convention and [JoinSpec] / [Join] attributes are honoured:

[Table("orders")]
public class MongoOrder : IdentityRecord
{
    [Column("customer_id")] public TForeignKey CustomerId = new TForeignKey();
    [Column("total")]       public TDecimal    Total      = new TDecimal();

    // Convention-based join: field "Customer" + "Id" suffix = "customer_id" FK
    public MongoCustomer Customer;

    public MongoOrder()                          { Customer = new MongoCustomer(); }
    public MongoOrder(DataConnection conn) : base(conn) { Customer = new MongoCustomer(conn); }
}

MongoDB LEFT OUTER JOIN: $unwind is generated with preserveNullAndEmptyArrays: true for LeftOuterJoin.
MongoDB INNER JOIN: $unwind without that flag excludes documents with no match.
All LINQ join overrides (.LeftOuterJoin<T>() / .InnerJoin<T>()) work identically on MongoDataConnection.

27. Complete Examples

The examples below demonstrate realistic end-to-end usage covering all major features.

27.1 Complete CRUD Lifecycle

using ActiveForge;
using ActiveForge.Attributes;
using ActiveForge.Query;
using ActiveForge.Linq;

// ── Entity ───────────────────────────────────────────────────────────────────

[Table("products")]
[Description("A product available in the catalogue")]
public class Product : IdentityRecord
{
    [Column("name")]
    [Description("Display name shown on product listings")]
    public TString Name = new TString();

    [Column("sku")]
    [NoTrim]                               // preserve exact SKU padding
    public TString Sku = new TString();

    [Column("price")]
    [Description("Retail price excluding tax")]
    public TDecimal Price = new TDecimal();

    [Column("in_stock")]
    [DefaultValue(true)]
    public TBool InStock = new TBool();

    [Column("created_at")]
    [ReadOnly]                             // set by DB default / trigger
    public TDateTime CreatedAt = new TDateTime();

    [Column("notes")]
    [NoPreload]                            // loaded only when explicitly requested
    public TString Notes = new TString();

    [Column("api_key")]
    [Sensitive]                            // masked in diagnostic output
    public TString ApiKey = new TString();

    public Product() { }
    public Product(DataConnection conn) : base(conn) { }
}

// ── Connection ────────────────────────────────────────────────────────────────

var conn = new SqlServerConnection(
    "Server=.;Database=Demo;Integrated Security=True;TrustServerCertificate=True;");
conn.Connect();

// ── INSERT ────────────────────────────────────────────────────────────────────

var p = new Product(conn);
p.Name.SetValue("Widget Pro");
p.Sku.SetValue("WGT-001  ");             // trailing spaces preserved by [NoTrim]
p.Price.SetValue(29.99m);
p.InStock.SetValue(true);
p.Insert();
int id = (int)p.ID.GetValue();           // auto-generated identity

// ── READ by primary key ───────────────────────────────────────────────────────

var found = new Product(conn);
found.ID.SetValue(id);
found.Read();
Console.WriteLine((string)found.Name.GetValue());  // "Widget Pro"

// Fetch the [NoPreload] Notes field explicitly:
var template = new Product(conn);
FieldSubset withNotes = conn.DefaultFieldSubset(template);
withNotes += template.Notes;
var results = conn.QueryAll(template,
    new EqualTerm(template, template.ID, id),
    null, 0, withNotes);

// ── UPDATE ────────────────────────────────────────────────────────────────────

found.Price.SetValue(24.99m);
found.Update(RecordLock.UpdateOption.IgnoreLock);   // update all columns

// Or update only what changed:
found.Notes.SetValue("Now with 10% discount");
found.UpdateChanged();                              // generates UPDATE ... SET Notes=@Notes

// ── QUERY ─────────────────────────────────────────────────────────────────────

// Classic QueryTerm API:
var inStock  = new EqualTerm(template, template.InStock, true);
var cheap    = new LessThanTerm(template, template.Price, 30m);
var byPrice  = new OrderAscending(template, template.Price);
RecordCollection all = conn.QueryAll(template, inStock & cheap, byPrice, 0, null);

// LINQ API:
List<Product> linq = conn.Query(new Product(conn))
    .Where(x => x.InStock == true && x.Price < 30m)
    .OrderBy(x => x.Price)
    .ToList();

// Pagination (QueryTerm):
QueryPage page = conn.QueryPage(template, inStock, byPrice, startRecord: 0, pageSize: 10);
Console.WriteLine($"Total: {page.TotalRowCount}, more: {page.IsMoreData}");

// Pagination (LINQ):
var linqPage = conn.Query(new Product(conn))
    .Where(x => x.InStock == true)
    .OrderBy(x => x.Name)
    .Skip(0).Take(10)
    .ToList();

// Count:
int count = conn.QueryCount(template, inStock);

// Lazy stream (no full buffer):
foreach (Product item in conn.LazyQueryAll<Product>(template, inStock, byPrice))
    Console.WriteLine(item.Name.GetValue());

// ── DELETE ────────────────────────────────────────────────────────────────────

found.Delete();                                     // delete by PK

// Delete by predicate:
var discontinued = new EqualTerm(template, template.InStock, false);
template.Delete(discontinued);

conn.Disconnect();

27.2 Joins — Full Example

using ActiveForge;
using ActiveForge.Attributes;
using ActiveForge.Query;
using ActiveForge.Linq;

// ── Entities ──────────────────────────────────────────────────────────────────

[Table("categories")]
public class Category : IdentityRecord
{
    [Column("name")] public TString Name = new TString();

    public Category() { }
    public Category(DataConnection conn) : base(conn) { }
}

// Convention INNER JOIN: "CategoryID" field + embedded "Category" object
[Table("products")]
public class ProductWithCategory : IdentityRecord
{
    [Column("name")]        public TString     Name       = new TString();
    [Column("price")]       public TDecimal    Price      = new TDecimal();
    [Column("CategoryID")]  public TForeignKey CategoryID = new TForeignKey();

    public Category Category;

    public ProductWithCategory()
    { Category = new Category(); }
    public ProductWithCategory(DataConnection conn) : base(conn)
    { Category = new Category(conn); }
}

// LEFT OUTER JOIN variant via [JoinSpec]
[Table("products")]
[JoinSpec("CategoryID", "Category", "ID", JoinSpecAttribute.JoinTypeEnum.LeftOuterJoin)]
public class ProductOuter : IdentityRecord
{
    [Column("name")]        public TString     Name       = new TString();
    [Column("price")]       public TDecimal    Price      = new TDecimal();
    [Column("CategoryID")]  public TForeignKey CategoryID = new TForeignKey();

    public Category Category;

    public ProductOuter()
    { Category = new Category(); }
    public ProductOuter(DataConnection conn) : base(conn)
    { Category = new Category(conn); }
}

// ── INNER JOIN queries ────────────────────────────────────────────────────────

var conn = new SqlServerConnection("...");
conn.Connect();

// All products (excludes those with no category)
var t1      = new ProductWithCategory(conn);
var results = conn.QueryAll(t1, null, null, 0, null);

// Filter on joined column
var term = new EqualTerm(t1.Category, t1.Category.Name, "Electronics");
var elec = conn.QueryAll(t1, term, null, 0, null);

// ── LEFT OUTER JOIN queries ───────────────────────────────────────────────────

var t2  = new ProductOuter(conn);
var all = conn.QueryAll(t2, null, null, 0, null);   // includes uncategorised products

foreach (ProductOuter p in all)
{
    string cat = p.Category.Name.IsNull() ? "(none)" : (string)p.Category.Name.GetValue();
    Console.WriteLine($"{p.Name.GetValue()} → {cat}");
}

// ── Query-time override ───────────────────────────────────────────────────────

// Promote INNER → LEFT OUTER for one query
var overrides = new List<JoinOverride>
{
    new JoinOverride(typeof(Category), JoinSpecification.JoinTypeEnum.LeftOuterJoin)
};
var withOrphans = conn.QueryAll(new ProductWithCategory(conn), null, null, 0, null, overrides);

// ── LINQ cross-join predicates and sorts ──────────────────────────────────────

// Filter by joined column
var byCategory = conn.Query(new ProductWithCategory(conn))
    .Where(p => p.Category.Name == "Electronics")
    .ToList();

// NULL check on joined column (LEFT OUTER)
var orphans = conn.Query(new ProductOuter(conn))
    .Where(p => p.Category.Name == (TString)null)
    .ToList();

// Sort by joined column
var sorted = conn.Query(new ProductWithCategory(conn))
    .OrderBy(p => p.Category.Name)
    .ThenBy(p => p.Price)
    .ToList();

// Full chain: override + filter + sort + pagination
var page = conn.Query(new ProductWithCategory(conn))
    .LeftOuterJoin<Category>()
    .Where(p => p.Price < 100m)
    .OrderBy(p => p.Category.Name)
    .ThenBy(p => p.Name)
    .Skip(0)
    .Take(20)
    .ToList();

conn.Disconnect();

27.3 Polymorphic Records — Full Example

BaseFactory maps abstract base types to concrete subtypes. Register mappings in CreateTypeMap(). Pass the factory instance to the connection constructor. When the ORM hydrates a query result it calls BaseFactory.MapType(typeof(BaseType)) to determine which concrete class to instantiate.

using ActiveForge;
using ActiveForge.Attributes;
using ActiveForge.Query;

// ── Entities — all stored in the same table ───────────────────────────────────

[Table("notifications")]
public abstract class Notification : IdentityRecord
{
    [Column("recipient")]  public TString   Recipient = new TString();
    [Column("channel")]    public TString   Channel   = new TString();   // "email"|"sms"|"push"
    [Column("sent_at")]    public TDateTime SentAt    = new TDateTime();
    [ReadOnly]
    [Column("created_at")] public TDateTime CreatedAt = new TDateTime();

    protected Notification() { }
    protected Notification(DataConnection conn) : base(conn) { }
}

[Table("notifications")]
public class EmailNotification : Notification
{
    [Column("subject")] public TString Subject = new TString();
    [Column("body")]    public TString Body    = new TString();

    public EmailNotification() { }
    public EmailNotification(DataConnection conn) : base(conn) { }
}

[Table("notifications")]
public class SmsNotification : Notification
{
    [Column("phone")]   public TString Phone   = new TString();
    [Column("message")] public TString Message = new TString();

    public SmsNotification() { }
    public SmsNotification(DataConnection conn) : base(conn) { }
}

// ── Factory — static type substitution ───────────────────────────────────────
//
// BaseFactory.AddTypeMapping registers a static base→concrete substitution.
// All rows queried as Notification will be instantiated as the mapped type.
// Use the expectedTypes overload when multiple concrete types are in the result.

public class NotificationFactory : BaseFactory
{
    protected override void CreateTypeMap()
    {
        // When querying Notification rows, default to EmailNotification
        AddTypeMapping(typeof(Notification), typeof(EmailNotification));
    }
}

// ── Usage ─────────────────────────────────────────────────────────────────────

var conn = new SqlServerConnection(
    "Server=.;Database=Demo;Integrated Security=True;TrustServerCertificate=True;",
    new NotificationFactory());
conn.Connect();

// INSERT — use the concrete types directly
var email = new EmailNotification(conn);
email.Recipient.SetValue("alice@example.com");
email.Channel.SetValue("email");
email.SentAt.SetValue(DateTime.UtcNow);
email.Subject.SetValue("Your order has shipped");
email.Body.SetValue("Track it at...");
email.Insert();

var sms = new SmsNotification(conn);
sms.Recipient.SetValue("alice@example.com");
sms.Channel.SetValue("sms");
sms.SentAt.SetValue(DateTime.UtcNow);
sms.Phone.SetValue("+447700900123");
sms.Message.SetValue("Your order has shipped");
sms.Insert();

// QUERY all rows as the abstract type — factory substitutes EmailNotification
var template = new Notification(conn);
var all      = conn.QueryAll(template, null, null, 0, null);

foreach (Notification n in all)
{
    Console.WriteLine($"Channel: {n.Channel.GetValue()}, Recipient: {n.Recipient.GetValue()}");

    if (n is EmailNotification e)
        Console.WriteLine($"  Subject: {e.Subject.GetValue()}");
    else if (n is SmsNotification s)
        Console.WriteLine($"  Phone: {s.Phone.GetValue()}");
}

// QUERY both concrete types in one call via expectedTypes
var mixed = conn.QueryAll(
    new Notification(conn),
    null, null, 0,
    new[] { typeof(EmailNotification), typeof(SmsNotification) },
    null);

// Filter using abstract-type fields
var emailOnly = conn.QueryAll(template,
    new EqualTerm(template, template.Channel, "email"), null, 0, null);

// Update via concrete type
email.Subject.SetValue("Updated: your order has shipped");
email.Update(RecordLock.UpdateOption.IgnoreLock);

// UpdateChanged — only writes modified fields
sms.Message.SetValue("Update: order dispatched");
sms.UpdateChanged();

// Delete
sms.Delete();

conn.Disconnect();

ActiveForge — .NET / Active Record / SQL Server / PostgreSQL / MongoDB / SQLite

SourceFlow.Net - Event Sourcing Framework.

CØDE N!NJΔ — Sat, 29 Nov 2025 11:07:48 +0000

A modern, lightweight, and extensible framework for building event-sourced applications using Domain-Driven Design (DDD) principles and Command Query Responsibility Segregation (CQRS) patterns.

Introduction
Core Concepts
Architecture Overview
Getting Started
Framework Components
Persistence with Entity Framework
EntityFramework Usage Examples
Implementation Guide
Advanced Features
Performance and Observability
Best Practices
FAQ

Introduction

SourceFlow.Net is a modern, lightweight, and extensible .NET framework designed for building scalable event-sourced applications using Domain-Driven Design (DDD) principles and Command Query Responsibility Segregation (CQRS) patterns. Built for .NET 8+ with performance and developer experience as core priorities.

What Makes SourceFlow.Net Special?

SourceFlow.Net provides a complete toolkit for event sourcing, domain modeling, and command/query separation, enabling developers to build maintainable, scalable applications with a strong foundation in proven architectural patterns.

Key Features

🏗️ Domain-Driven Design Support - First-class support for aggregates, entities, value objects
⚡ CQRS Implementation - Complete command/query separation with optimized read models
📊 Event Sourcing Foundation - Event-first design with full audit trail
🧱 Clean Architecture - Clear separation of concerns and dependency management
💾 Flexible Persistence - Multiple storage options including Entity Framework Core
🔄 Event Replay - Built-in command replay for debugging and state reconstruction
🎯 Type Safety - Strongly-typed commands, events, and projections
📦 Dependency Injection - Seamless integration with .NET DI container
📈 OpenTelemetry Integration - Built-in distributed tracing and metrics for operations at scale
⚡ Memory Optimization - ArrayPool-based optimization for extreme throughput scenarios
🛡️ Resilience Patterns - Polly integration for fault tolerance with retry policies and circuit breakers

Core Concepts

Event Sourcing

Event Sourcing is an architectural pattern where the state of an application is determined by a sequence of events. Instead of storing the current state directly, the system stores all the events that have occurred, allowing for complete state reconstruction at any point in time.

Key Benefits:

Complete Audit Trail: Every change is recorded as an immutable event
Time Travel: Reconstruct system state at any point in history
Debugging: Full visibility into how the system reached its current state
Scalability: Events can be replayed to build multiple read models

Example in SourceFlow.Net:

// Events are immutable records of what happened
public class AccountCreated : Event<BankAccount>
{
    public AccountCreated(BankAccount payload) : base(payload) { }
}

public class MoneyDeposited : Event<BankAccount>
{
    public MoneyDeposited(BankAccount payload) : base(payload) { }
}

Domain-Driven Design (DDD)

Domain-Driven Design is a software design approach that focuses on modeling software to match the business domain. It emphasizes collaboration between technical and domain experts to create a shared understanding of the problem space.

Core DDD Elements in SourceFlow.Net:

Entities: Objects with unique identity

public class BankAccount : IEntity
{
    public int Id { get; set; }
    public string AccountName { get; set; }
    public decimal Balance { get; set; }
    public bool IsClosed { get; set; }
    public DateTime CreatedOn { get; set; }
}

Aggregates: Coordinate business logic and ensure consistency

public class AccountAggregate : Aggregate<BankAccount>
{
    public void CreateAccount(int accountId, string holder, decimal amount)
    {
        // Business logic validation
        Send(new CreateAccount(new CreateAccountPayload
        {
            Id = accountId,
            AccountName = holder,
            InitialAmount = amount
        }));
    }
}

Sagas: Orchestrate long-running business processes

public class AccountSaga : Saga<BankAccount>,
    IHandles<CreateAccount>,
    IHandlesWithEvent<CreateAccount, AccountCreated>
{
    public async Task<BankAccount> Handle(BankAccount entity, CreateAccount command)
    {
        // Validate and update entity
        entity.AccountName = command.Payload.AccountName;
        entity.Balance = command.Payload.InitialAmount;
        entity.CreatedOn = DateTime.UtcNow;

        // Return entity - saga framework automatically persists and raises AccountCreated event
        return entity;
    }
}

Command Query Responsibility Segregation (CQRS)

CQRS separates read and write operations, allowing for optimized data models for different purposes.

Commands: Represent intent to change state

public class CreateAccount : Command<CreateAccountPayload>
{
    // Parameterless constructor required for deserialization
    public CreateAccount() : base() { }

    public CreateAccount(CreateAccountPayload payload) : base(payload) { }
}

Queries: Handled through optimized view models

public class AccountViewModel : IViewModel
{
    public int Id { get; set; }
    public string AccountName { get; set; }
    public decimal CurrentBalance { get; set; }
    public DateTime LastUpdated { get; set; }
    public int TransactionCount { get; set; }
}

Projections: Update read models based on events

public class AccountView : View<AccountViewModel>,
    IProjectOn<AccountCreated>,
    IProjectOn<MoneyDeposited>
{
    public AccountView(
        IViewModelStoreAdapter viewModelStore,
        ILogger<IView> logger)
        : base(viewModelStore, logger)
    {
    }

    public async Task<AccountViewModel> On(AccountCreated @event)
    {
        var viewModel = new AccountViewModel
        {
            Id = @event.Payload.Id,
            AccountName = @event.Payload.AccountName,
            CurrentBalance = @event.Payload.Balance,
            CreatedDate = @event.Payload.CreatedOn
        };

        // Return view model - framework automatically persists it
        return viewModel;
    }

    public async Task<AccountViewModel> On(MoneyDeposited @event)
    {
        var viewModel = await Find<AccountViewModel>(@event.Payload.Id);
        viewModel.CurrentBalance = @event.Payload.Balance;
        viewModel.TransactionCount++;
        viewModel.LastUpdated = DateTime.UtcNow;

        return viewModel;
    }
}

Architecture Overview

High-Level Architecture

Component Interactions

Aggregates encapsulate business logic and send commands
Command Bus routes commands to appropriate saga handlers
Sagas handle commands and maintain consistency across aggregates
Sagas persist entities to the Entity Store
Sagas raise events to the Event Queue
Event Queue dispatches events to subscribers
Views are projections that update read models (ViewModels) based on events
Command Store persists commands for replay capability
Entity Store persists root aggregates (entities) within bounded context
ViewModel Store persists transformed view models from events

Entity Framework Stores provide persistence using EF Core with support for multiple databases

Getting Started

Installation

# Install the core package
dotnet add package SourceFlow.Net

# Install Entity Framework persistence
dotnet add package SourceFlow.Stores.EntityFramework

Basic Setup

// Program.cs
using SourceFlow;
using SourceFlow.Stores.EntityFramework;
using SourceFlow.Stores.EntityFramework.Extensions;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;

var services = new ServiceCollection();

// Add logging
services.AddLogging(builder =>
{
    builder.AddConsole();
    builder.SetMinimumLevel(LogLevel.Information);
});

// Register entity and view model types BEFORE building service provider
EntityDbContext.RegisterAssembly(typeof(Program).Assembly);
ViewModelDbContext.RegisterAssembly(typeof(Program).Assembly);

// Configure SourceFlow with automatic discovery
services.UseSourceFlow(typeof(Program).Assembly);

// Add Entity Framework stores with SQL Server (default)
services.AddSourceFlowEfStores(
    "Server=localhost;Database=SourceFlow;Integrated Security=true;TrustServerCertificate=true;");

var serviceProvider = services.BuildServiceProvider();

// Initialize databases
var commandContext = serviceProvider.GetRequiredService<CommandDbContext>();
await commandContext.Database.EnsureCreatedAsync();

var entityContext = serviceProvider.GetRequiredService<EntityDbContext>();
await entityContext.Database.EnsureCreatedAsync();
entityContext.ApplyMigrations();

var viewModelContext = serviceProvider.GetRequiredService<ViewModelDbContext>();
await viewModelContext.Database.EnsureCreatedAsync();
viewModelContext.ApplyMigrations();

// Start using SourceFlow
var aggregateFactory = serviceProvider.GetRequiredService<IAggregateFactory>();
var accountAggregate = await aggregateFactory.Create<IAccountAggregate>();

accountAggregate.CreateAccount(1, "John Doe", 1000m);

For other database providers (PostgreSQL, MySQL, SQLite), see EntityFramework Usage Examples.

Framework Components

1. Aggregates

Aggregates are the primary building blocks that encapsulate business logic and coordinate with the command bus.

public abstract class Aggregate<TEntity> : IAggregate
    where TEntity : class, IEntity
{
    protected ICommandPublisher commandPublisher;
    protected ILogger logger;

    // Send commands to command bus
    protected async Task Send(ICommand command);

    // Subscribe to external events
    public virtual Task On(IEvent @event);
}

Key Features:

Command publishing
Event subscription for external changes
Logger integration
Generic entity support

2. Sagas

Sagas handle commands and coordinate business processes, maintaining consistency across aggregate boundaries.

public abstract class Saga<TAggregate> : ISaga<TAggregate>
    where TAggregate : class, IEntity
{
    protected Lazy<ICommandPublisher> commandPublisher;
    protected IEventQueue eventQueue;
    protected IEntityStoreAdapter entityStore;
    protected ILogger<ISaga> logger;

    public Saga(
        Lazy<ICommandPublisher> commandPublisher,
        IEventQueue eventQueue,
        IEntityStoreAdapter entityStore,
        ILogger<ISaga> logger);

    // Publish commands
    protected Task Publish<TCommand>(TCommand command) where TCommand : ICommand;

    // Raise events
    protected Task Raise<TEvent>(TEvent @event) where TEvent : IEvent;
}

Key Features:

Command handling via IHandles<TCommand> with signature: Task<TAggregate> Handle(TAggregate entity, TCommand command)
Automatic event raising via IHandlesWithEvent<TCommand, TEvent> - events are automatically raised after successful persistence
Entity store access for persistence (renamed from repository to entityStore)
Built-in logging
Lazy-loaded command publisher for preventing circular dependencies
Entities are automatically persisted and returned after Handle completes
Return null from Handle to skip persistence

3. Command Bus

The command bus routes commands to appropriate saga handlers and manages command persistence.

public interface ICommandBus
{
    // Publish commands to sagas
    Task Publish<TCommand>(TCommand command) where TCommand : ICommand;

    // Event dispatchers for command lifecycle
    event EventHandler<ICommand> Dispatchers;
}

4. Event Queue

The event queue manages event flow and dispatches events to subscribers.

public interface IEventQueue
{
    // Enqueue events for processing
    Task Enqueue<TEvent>(TEvent @event) where TEvent : IEvent;

    // Event dispatchers
    event EventHandler<IEvent> Dispatchers;
}

5. Stores (Persistence Layer)

SourceFlow.Net defines three core store interfaces:

ICommandStore

Persists commands for event sourcing and replay

public interface ICommandStore
{
    Task Save(ICommand command);
    Task<IEnumerable<ICommand>> Load(int entityId);
}

IEntityStore

Persists domain entities

public interface IEntityStore
{
    Task<TEntity> Persist<TEntity>(TEntity entity) where TEntity : class, IEntity;
    Task<TEntity> Get<TEntity>(int id) where TEntity : class, IEntity;
    Task Delete<TEntity>(TEntity entity) where TEntity : class, IEntity;
}

IViewModelStore

Persists read models (projections)

public interface IViewModelStore
{
    Task<TViewModel> Persist<TViewModel>(TViewModel model) where TViewModel : class, IViewModel;
    Task<TViewModel> Get<TViewModel>(int id) where TViewModel : class, IViewModel;
    Task Delete<TViewModel>(TViewModel model) where TViewModel : class, IViewModel;
}

Persistence with Entity Framework

SourceFlow.Stores.EntityFramework provides production-ready persistence using Entity Framework Core with support for multiple database providers.

Features

✅ Multiple Database Support: SQL Server, PostgreSQL, SQLite, and more
✅ Flexible Configuration: Single or separate connection strings per store
✅ Dynamic Type Registration: Runtime registration of entities and view models
✅ Migration Support: Manual table creation bypassing EF Core model caching
✅ Thread-Safe: Designed for concurrent access
✅ Optimized Tracking: Proper EF Core change tracking management
✅ Production-Ready Enhancements: Resilience, observability, and memory optimization

Installation

dotnet add package SourceFlow.Stores.EntityFramework

Configuration Options

1. Single Connection String (All Stores)

Use the same database for all stores:

services.AddSourceFlowEfStores("Server=localhost;Database=SourceFlow;Integrated Security=true;");

2. Separate Connection Strings

Use different databases for each store:

services.AddSourceFlowEfStores(
    commandConnectionString: "Server=localhost;Database=SourceFlow_Commands;...",
    entityConnectionString: "Server=localhost;Database=SourceFlow_Entities;...",
    viewModelConnectionString: "Server=localhost;Database=SourceFlow_Views;..."
);

3. Configuration-Based

Read from appsettings.json:

{
  "ConnectionStrings": {
    "SourceFlow.Default": "Server=localhost;Database=SourceFlow;Integrated Security=true;",
    "SourceFlow.Command": "Server=localhost;Database=Commands;...",
    "SourceFlow.Entity": "Server=localhost;Database=Entities;...",
    "SourceFlow.ViewModel": "Server=localhost;Database=Views;..."
  }
}

services.AddSourceFlowEfStores(configuration);

4. Options Pattern

Configure using options:

services.AddSourceFlowEfStores(options =>
{
    options.DefaultConnectionString = "Server=localhost;Database=SourceFlow;...";
    // Or specify individual connection strings
    options.CommandConnectionString = "...";
    options.EntityConnectionString = "...";
    options.ViewModelConnectionString = "...";
});

5. Custom Database Provider

Use PostgreSQL, SQLite, or other providers:

// PostgreSQL
services.AddSourceFlowEfStoresWithCustomProvider(options =>
    options.UseNpgsql("Host=localhost;Database=sourceflow;Username=postgres;Password=..."));

// SQLite
services.AddSourceFlowEfStoresWithCustomProvider(options =>
    options.UseSqlite("Data Source=sourceflow.db"));

// In-Memory (for testing)
services.AddSourceFlowEfStoresWithCustomProvider(options =>
    options.UseInMemoryDatabase("SourceFlowTest"));

6. Different Providers Per Store

Use different database types for each store:

services.AddSourceFlowEfStoresWithCustomProviders(
    commandContextConfig: options => options.UseSqlServer("..."),
    entityContextConfig: options => options.UseNpgsql("..."),
    viewModelContextConfig: options => options.UseSqlite("...")
);

Dynamic Type Registration

Entity Framework requires types to be registered before creating the database schema. SourceFlow.Stores.EntityFramework provides multiple registration strategies:

1. Explicit Type Registration

// In your startup or test setup
EntityDbContext.RegisterEntityType<BankAccount>();
EntityDbContext.RegisterEntityType<Customer>();

ViewModelDbContext.RegisterViewModelType<AccountViewModel>();
ViewModelDbContext.RegisterViewModelType<CustomerViewModel>();

// Then build service provider and ensure databases are created
var serviceProvider = services.BuildServiceProvider();

var entityContext = serviceProvider.GetRequiredService<EntityDbContext>();
entityContext.Database.EnsureCreated();
entityContext.ApplyMigrations(); // Creates tables for registered types

var viewModelContext = serviceProvider.GetRequiredService<ViewModelDbContext>();
viewModelContext.Database.EnsureCreated();
viewModelContext.ApplyMigrations(); // Creates tables for registered view models

2. Assembly Scanning

// Register the test or application assembly
EntityDbContext.RegisterAssembly(typeof(BankAccount).Assembly);
ViewModelDbContext.RegisterAssembly(typeof(AccountViewModel).Assembly);

var serviceProvider = services.BuildServiceProvider();

// Apply migrations to create tables
var entityContext = serviceProvider.GetRequiredService<EntityDbContext>();
entityContext.Database.EnsureCreated();
entityContext.ApplyMigrations();

var viewModelContext = serviceProvider.GetRequiredService<ViewModelDbContext>();
viewModelContext.Database.EnsureCreated();
viewModelContext.ApplyMigrations();

3. Auto-Discovery (Fallback)

The DbContexts automatically discover types from loaded assemblies (fallback mechanism):

// Just ensure databases are created
var entityContext = serviceProvider.GetRequiredService<EntityDbContext>();
entityContext.Database.EnsureCreated();

var viewModelContext = serviceProvider.GetRequiredService<ViewModelDbContext>();
viewModelContext.Database.EnsureCreated();

// Note: This may not catch all types reliably; explicit registration is recommended

Table Naming Convention

All dynamically created tables use the T prefix:

BankAccount entity → TBankAccount table
AccountViewModel → TAccountViewModel table
Customer entity → TCustomer table

This convention helps distinguish dynamically created tables from EF Core's built-in tables.

Migration Helper

The DbContextMigrationHelper manually creates database schemas, bypassing EF Core's model caching:

// Called automatically by ApplyMigrations()
public static void CreateEntityTables(
    EntityDbContext context,
    IEnumerable<Type> entityTypes)
{
    // Creates tables with proper columns and primary keys
    // Supports int, long, string, bool, DateTime, decimal, double, float, byte[], enums
}

public static void CreateViewModelTables(
    ViewModelDbContext context,
    IEnumerable<Type> viewModelTypes)
{
    // Creates tables for view models
}

Complete Setup Example

using SourceFlow;
using SourceFlow.Stores.EntityFramework;
using SourceFlow.Stores.EntityFramework.Extensions;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Microsoft.EntityFrameworkCore;

var services = new ServiceCollection();

// Add logging
services.AddLogging(builder =>
{
    builder.AddConsole();
    builder.SetMinimumLevel(LogLevel.Information);
});

// Register types BEFORE building service provider
EntityDbContext.RegisterAssembly(typeof(BankAccount).Assembly);
ViewModelDbContext.RegisterAssembly(typeof(AccountViewModel).Assembly);

// Configure SourceFlow
services.UseSourceFlow(typeof(Program).Assembly);

// Add Entity Framework stores (SQL Server by default)
services.AddSourceFlowEfStores(
    "Server=localhost;Database=SourceFlow;Integrated Security=true;TrustServerCertificate=true;");

// Or use custom provider for other databases:
// services.AddSourceFlowEfStoresWithCustomProvider(options =>
//     options.UseNpgsql("Host=localhost;Database=sourceflow;Username=postgres;Password=..."));

var serviceProvider = services.BuildServiceProvider();

// Ensure all databases are created and migrated
var commandContext = serviceProvider.GetRequiredService<CommandDbContext>();
await commandContext.Database.EnsureCreatedAsync();

var entityContext = serviceProvider.GetRequiredService<EntityDbContext>();
await entityContext.Database.EnsureCreatedAsync();
entityContext.ApplyMigrations(); // Create tables for registered entity types

var viewModelContext = serviceProvider.GetRequiredService<ViewModelDbContext>();
await viewModelContext.Database.EnsureCreatedAsync();
viewModelContext.ApplyMigrations(); // Create tables for registered view model types

// Start using SourceFlow
var aggregateFactory = serviceProvider.GetRequiredService<IAggregateFactory>();
var accountAggregate = await aggregateFactory.Create<IAccountAggregate>();

accountAggregate.CreateAccount(1, "John Doe", 1000m);

Testing with In-Memory Database

For unit and integration tests, use SQLite in-memory databases with proper setup:

using NUnit.Framework;
using Microsoft.Data.Sqlite;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.DependencyInjection;
using SourceFlow.Stores.EntityFramework;
using SourceFlow.Stores.EntityFramework.Extensions;
using SourceFlow.Stores.EntityFramework.Options;
using SourceFlow.Stores.EntityFramework.Services;
using SourceFlow.Stores.EntityFramework.Stores;

[TestFixture]
public class BankAccountIntegrationTests
{
    private ServiceProvider? _serviceProvider;
    private SqliteConnection? _connection;

    [SetUp]
    public void Setup()
    {
        // Clear previous registrations
        EntityDbContext.ClearRegistrations();
        ViewModelDbContext.ClearRegistrations();

        // Register test types
        EntityDbContext.RegisterEntityType<BankAccount>();
        ViewModelDbContext.RegisterViewModelType<AccountViewModel>();

        // Create shared in-memory SQLite connection for all contexts
        _connection = new SqliteConnection("DataSource=:memory:");
        _connection.Open();

        var services = new ServiceCollection();

        // Add logging for better test diagnostics
        services.AddLogging(builder =>
        {
            builder.AddConsole();
            builder.SetMinimumLevel(LogLevel.Debug);
        });

        // Configure SQLite with shared connection
        // Use EnableServiceProviderCaching(false) to avoid EF Core 9.0 multiple provider conflicts
        services.AddDbContext<CommandDbContext>(options =>
            options.UseSqlite(_connection)
                .EnableServiceProviderCaching(false));
        services.AddDbContext<EntityDbContext>(options =>
            options.UseSqlite(_connection)
                .EnableServiceProviderCaching(false));
        services.AddDbContext<ViewModelDbContext>(options =>
            options.UseSqlite(_connection)
                .EnableServiceProviderCaching(false));

        // Register SourceFlowEfOptions with default settings
        var efOptions = new SourceFlowEfOptions();
        services.AddSingleton(efOptions);

        // Register common services manually (avoids provider conflicts)
        services.AddScoped<IDatabaseResiliencePolicy, DatabaseResiliencePolicy>();
        services.AddScoped<IDatabaseTelemetryService, DatabaseTelemetryService>();
        services.AddScoped<ICommandStore, EfCommandStore>();
        services.AddScoped<IEntityStore, EfEntityStore>();
        services.AddScoped<IViewModelStore, EfViewModelStore>();

        // Register SourceFlow
        services.UseSourceFlow(Assembly.GetExecutingAssembly());

        _serviceProvider = services.BuildServiceProvider();

        // Create all database schemas
        var commandContext = _serviceProvider.GetRequiredService<CommandDbContext>();
        commandContext.Database.EnsureCreated();

        var entityContext = _serviceProvider.GetRequiredService<EntityDbContext>();
        entityContext.Database.EnsureCreated();
        entityContext.ApplyMigrations();  // Create tables for registered entity types

        var viewModelContext = _serviceProvider.GetRequiredService<ViewModelDbContext>();
        viewModelContext.Database.EnsureCreated();
        viewModelContext.ApplyMigrations();  // Create tables for registered view model types
    }

    [TearDown]
    public void TearDown()
    {
        // Clean up resources
        _connection?.Close();
        _connection?.Dispose();
        _serviceProvider?.Dispose();
    }

    [Test]
    public async Task CreateAccount_StoresInDatabase()
    {
        // Arrange
        var aggregateFactory = _serviceProvider.GetRequiredService<IAggregateFactory>();
        var accountAggregate = await aggregateFactory.Create<IAccountAggregate>();

        // Act
        accountAggregate.CreateAccount(1, "John Doe", 1000m);

        // Wait for async processing
        await Task.Delay(100);

        // Assert
        var entityStore = _serviceProvider.GetRequiredService<IEntityStoreAdapter>();
        var account = await entityStore.Get<BankAccount>(1);

        Assert.That(account, Is.Not.Null);
        Assert.That(account.AccountName, Is.EqualTo("John Doe"));
        Assert.That(account.Balance, Is.EqualTo(1000m));
    }
}

EntityFramework Usage Examples

This section provides practical examples for common scenarios using SourceFlow.Stores.EntityFramework.

Example 1: Simple Console Application with SQL Server

Complete working example for a console application:

using System;
using System.Threading.Tasks;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using SourceFlow;
using SourceFlow.Stores.EntityFramework;
using SourceFlow.Stores.EntityFramework.Extensions;

class Program
{
    static async Task Main(string[] args)
    {
        // Setup service collection
        var services = new ServiceCollection();

        // Add logging
        services.AddLogging(builder =>
        {
            builder.AddConsole();
            builder.SetMinimumLevel(LogLevel.Information);
        });

        // Register entity and view model types BEFORE building service provider
        EntityDbContext.RegisterEntityType<BankAccount>();
        ViewModelDbContext.RegisterViewModelType<AccountViewModel>();

        // Configure SourceFlow
        services.UseSourceFlow(typeof(Program).Assembly);

        // Add Entity Framework stores with SQL Server
        services.AddSourceFlowEfStores(
            "Server=localhost;Database=SourceFlowDemo;Integrated Security=true;TrustServerCertificate=true;");

        var serviceProvider = services.BuildServiceProvider();

        // Ensure databases are created
        var commandContext = serviceProvider.GetRequiredService<CommandDbContext>();
        await commandContext.Database.EnsureCreatedAsync();

        var entityContext = serviceProvider.GetRequiredService<EntityDbContext>();
        await entityContext.Database.EnsureCreatedAsync();
        entityContext.ApplyMigrations();

        var viewModelContext = serviceProvider.GetRequiredService<ViewModelDbContext>();
        await viewModelContext.Database.EnsureCreatedAsync();
        viewModelContext.ApplyMigrations();

        // Use the aggregate
        var aggregateFactory = serviceProvider.GetRequiredService<IAggregateFactory>();
        var accountAggregate = await aggregateFactory.Create<IAccountAggregate>();

        // Execute business operations
        accountAggregate.CreateAccount(1, "Alice Smith", 5000m);
        accountAggregate.Deposit(1, 1500m);
        accountAggregate.Withdraw(1, 500m);

        // Give async processing time to complete
        await Task.Delay(500);

        // Query the read model
        var viewModelStore = serviceProvider.GetRequiredService<IViewModelStoreAdapter>();
        var accountView = await viewModelStore.Find<AccountViewModel>(1);

        Console.WriteLine($"Account: {accountView.AccountName}");
        Console.WriteLine($"Balance: {accountView.CurrentBalance:C}");
        Console.WriteLine($"Transactions: {accountView.TransactionCount}");
        Console.WriteLine($"Created: {accountView.CreatedDate:yyyy-MM-dd}");
    }
}

Example 2: ASP.NET Core Web API with PostgreSQL

Complete setup for a web API using PostgreSQL:

// Program.cs
using Microsoft.EntityFrameworkCore;
using SourceFlow;
using SourceFlow.Stores.EntityFramework;
using SourceFlow.Stores.EntityFramework.Extensions;

var builder = WebApplication.CreateBuilder(args);

// Add services to the container
builder.Services.AddControllers();
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();

// Register entity and view model types
EntityDbContext.RegisterAssembly(typeof(Program).Assembly);
ViewModelDbContext.RegisterAssembly(typeof(Program).Assembly);

// Configure SourceFlow with PostgreSQL
builder.Services.UseSourceFlow(typeof(Program).Assembly);

builder.Services.AddSourceFlowEfStoresWithCustomProvider(options =>
    options.UseNpgsql(builder.Configuration.GetConnectionString("SourceFlow")));

var app = builder.Build();

// Initialize databases on startup
using (var scope = app.Services.CreateScope())
{
    var commandContext = scope.ServiceProvider.GetRequiredService<CommandDbContext>();
    await commandContext.Database.EnsureCreatedAsync();

    var entityContext = scope.ServiceProvider.GetRequiredService<EntityDbContext>();
    await entityContext.Database.EnsureCreatedAsync();
    entityContext.ApplyMigrations();

    var viewModelContext = scope.ServiceProvider.GetRequiredService<ViewModelDbContext>();
    await viewModelContext.Database.EnsureCreatedAsync();
    viewModelContext.ApplyMigrations();
}

if (app.Environment.IsDevelopment())
{
    app.UseSwagger();
    app.UseSwaggerUI();
}

app.UseHttpsRedirection();
app.UseAuthorization();
app.MapControllers();
app.Run();

// appsettings.json
{
  "ConnectionStrings": {
    "SourceFlow": "Host=localhost;Database=sourceflow;Username=postgres;Password=yourpassword"
  },
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.EntityFrameworkCore": "Warning"
    }
  }
}

// Controllers/AccountController.cs
using Microsoft.AspNetCore.Mvc;
using SourceFlow;

[ApiController]
[Route("api/[controller]")]
public class AccountController : ControllerBase
{
    private readonly IAggregateFactory _aggregateFactory;
    private readonly IViewModelStoreAdapter _viewModelStore;
    private readonly ILogger<AccountController> _logger;

    public AccountController(
        IAggregateFactory aggregateFactory,
        IViewModelStoreAdapter viewModelStore,
        ILogger<AccountController> logger)
    {
        _aggregateFactory = aggregateFactory;
        _viewModelStore = viewModelStore;
        _logger = logger;
    }

    [HttpPost]
    public async Task<IActionResult> CreateAccount(CreateAccountRequest request)
    {
        var aggregate = await _aggregateFactory.Create<IAccountAggregate>();
        aggregate.CreateAccount(request.Id, request.AccountName, request.InitialBalance);

        _logger.LogInformation("Account created: {AccountId}", request.Id);
        return CreatedAtAction(nameof(GetAccount), new { id = request.Id }, request);
    }

    [HttpGet("{id}")]
    public async Task<ActionResult<AccountViewModel>> GetAccount(int id)
    {
        try
        {
            var account = await _viewModelStore.Find<AccountViewModel>(id);
            return Ok(account);
        }
        catch (InvalidOperationException)
        {
            return NotFound();
        }
    }

    [HttpPost("{id}/deposit")]
    public async Task<IActionResult> Deposit(int id, [FromBody] TransactionRequest request)
    {
        var aggregate = await _aggregateFactory.Create<IAccountAggregate>();
        aggregate.Deposit(id, request.Amount);

        _logger.LogInformation("Deposited {Amount} to account {AccountId}", request.Amount, id);
        return NoContent();
    }

    [HttpPost("{id}/withdraw")]
    public async Task<IActionResult> Withdraw(int id, [FromBody] TransactionRequest request)
    {
        try
        {
            var aggregate = await _aggregateFactory.Create<IAccountAggregate>();
            aggregate.Withdraw(id, request.Amount);

            _logger.LogInformation("Withdrew {Amount} from account {AccountId}", request.Amount, id);
            return NoContent();
        }
        catch (InvalidOperationException ex)
        {
            return BadRequest(new { error = ex.Message });
        }
    }
}

public record CreateAccountRequest(int Id, string AccountName, decimal InitialBalance);
public record TransactionRequest(decimal Amount);

Example 3: Microservices with Separate Databases

Using different databases for different stores in a microservices architecture:

// Program.cs for Banking Microservice
var builder = WebApplication.CreateBuilder(args);

// Register types
EntityDbContext.RegisterAssembly(typeof(Program).Assembly);
ViewModelDbContext.RegisterAssembly(typeof(Program).Assembly);

// Configure SourceFlow
builder.Services.UseSourceFlow(typeof(Program).Assembly);

// Each store uses a different database optimized for its purpose
builder.Services.AddSourceFlowEfStoresWithCustomProviders(
    // Commands: PostgreSQL with JSONB support for efficient command storage
    commandContextConfig: opt => opt.UseNpgsql(
        builder.Configuration.GetConnectionString("CommandStore")),

    // Entities: SQL Server with optimized indexes for transactional workload
    entityContextConfig: opt => opt.UseSqlServer(
        builder.Configuration.GetConnectionString("EntityStore")),

    // ViewModels: SQLite for fast read queries in read-heavy scenarios
    viewModelContextConfig: opt => opt.UseSqlite(
        builder.Configuration.GetConnectionString("ViewStore"))
);

var app = builder.Build();

// Initialize all databases
using (var scope = app.Services.CreateScope())
{
    var commandContext = scope.ServiceProvider.GetRequiredService<CommandDbContext>();
    await commandContext.Database.MigrateAsync();

    var entityContext = scope.ServiceProvider.GetRequiredService<EntityDbContext>();
    await entityContext.Database.MigrateAsync();
    entityContext.ApplyMigrations();

    var viewModelContext = scope.ServiceProvider.GetRequiredService<ViewModelDbContext>();
    await viewModelContext.Database.MigrateAsync();
    viewModelContext.ApplyMigrations();
}

app.Run();

// appsettings.json
{
  "ConnectionStrings": {
    "CommandStore": "Host=postgres-commands.internal;Database=banking_commands;Username=app;Password=...",
    "EntityStore": "Server=sqlserver-entities.internal;Database=banking_entities;User Id=app;Password=...;",
    "ViewStore": "Data Source=/data/banking_views.db"
  }
}

Example 4: Production Configuration with Resilience and Observability

Complete production setup with all enterprise features:

// Program.cs
using SourceFlow;
using SourceFlow.Observability;
using SourceFlow.Stores.EntityFramework;
using SourceFlow.Stores.EntityFramework.Extensions;
using OpenTelemetry;
using Microsoft.EntityFrameworkCore;

var builder = WebApplication.CreateBuilder(args);

// Register domain types
EntityDbContext.RegisterAssembly(typeof(Program).Assembly);
ViewModelDbContext.RegisterAssembly(typeof(Program).Assembly);

// Configure SourceFlow with observability
builder.Services.AddSourceFlowTelemetry(options =>
{
    options.Enabled = true;
    options.ServiceName = "BankingService";
    options.ServiceVersion = builder.Configuration["AppVersion"] ?? "1.0.0";
});

// Configure OpenTelemetry exporters
builder.Services.AddOpenTelemetry()
    .AddSourceFlowOtlpExporter(builder.Configuration["Observability:OtlpEndpoint"])
    .AddSourceFlowResourceAttributes(
        ("environment", builder.Environment.EnvironmentName),
        ("deployment.region", builder.Configuration["Deployment:Region"]),
        ("service.instance.id", Environment.MachineName)
    )
    .ConfigureSourceFlowBatchProcessing(
        maxQueueSize: 2048,
        maxExportBatchSize: 512,
        scheduledDelayMilliseconds: 5000
    );

// Register SourceFlow
builder.Services.UseSourceFlow(typeof(Program).Assembly);

// Configure Entity Framework stores with resilience and observability
builder.Services.AddSourceFlowEfStores(options =>
{
    options.DefaultConnectionString = builder.Configuration.GetConnectionString("SourceFlow");

    // Resilience configuration
    options.Resilience.Enabled = true;
    options.Resilience.Retry.MaxRetryAttempts = 3;
    options.Resilience.Retry.BaseDelayMs = 1000;
    options.Resilience.Retry.UseExponentialBackoff = true;
    options.Resilience.Retry.UseJitter = true;
    options.Resilience.CircuitBreaker.Enabled = true;
    options.Resilience.CircuitBreaker.FailureThreshold = 10;
    options.Resilience.CircuitBreaker.BreakDurationMs = 60000;
    options.Resilience.Timeout.Enabled = true;
    options.Resilience.Timeout.TimeoutMs = 30000;

    // Observability configuration
    options.Observability.Enabled = true;
    options.Observability.ServiceName = "BankingService.EntityFramework";
    options.Observability.Tracing.Enabled = true;
    options.Observability.Tracing.TraceDatabaseOperations = true;
    options.Observability.Tracing.IncludeSqlInTraces = false;  // Don't log SQL in production
    options.Observability.Tracing.SamplingRatio = 0.1;         // Sample 10%
    options.Observability.Metrics.Enabled = true;
    options.Observability.Metrics.CollectDatabaseMetrics = true;

    // Table naming conventions
    options.EntityTableNaming.Casing = TableNameCasing.SnakeCase;
    options.EntityTableNaming.Pluralize = true;
    options.ViewModelTableNaming.Casing = TableNameCasing.SnakeCase;
    options.ViewModelTableNaming.Suffix = "_view";
});

// Add health checks
builder.Services.AddHealthChecks()
    .AddDbContextCheck<CommandDbContext>("command-store")
    .AddDbContextCheck<EntityDbContext>("entity-store")
    .AddDbContextCheck<ViewModelDbContext>("viewmodel-store");

var app = builder.Build();

// Initialize databases
using (var scope = app.Services.CreateScope())
{
    var logger = scope.ServiceProvider.GetRequiredService<ILogger<Program>>();

    try
    {
        var commandContext = scope.ServiceProvider.GetRequiredService<CommandDbContext>();
        await commandContext.Database.EnsureCreatedAsync();
        logger.LogInformation("Command store initialized");

        var entityContext = scope.ServiceProvider.GetRequiredService<EntityDbContext>();
        await entityContext.Database.EnsureCreatedAsync();
        entityContext.ApplyMigrations();
        logger.LogInformation("Entity store initialized");

        var viewModelContext = scope.ServiceProvider.GetRequiredService<ViewModelDbContext>();
        await viewModelContext.Database.EnsureCreatedAsync();
        viewModelContext.ApplyMigrations();
        logger.LogInformation("ViewModel store initialized");
    }
    catch (Exception ex)
    {
        logger.LogError(ex, "Failed to initialize databases");
        throw;
    }
}

app.MapHealthChecks("/health");
app.Run();

// appsettings.Production.json
{
  "ConnectionStrings": {
    "SourceFlow": "Server=prod-db.internal;Database=BankingService;User Id=app_user;Password=...;Max Pool Size=100;Min Pool Size=10;"
  },
  "Observability": {
    "OtlpEndpoint": "http://otel-collector.internal:4317"
  },
  "Deployment": {
    "Region": "us-east-1"
  },
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.EntityFrameworkCore": "Warning",
      "SourceFlow": "Information"
    }
  }
}

Example 5: Custom Table Naming with Schema Organization

Organizing tables with custom naming conventions and schemas:

services.AddSourceFlowEfStores(options =>
{
    options.DefaultConnectionString = connectionString;

    // Command store: Audit schema with snake_case
    options.CommandTableNaming.UseSchema = true;
    options.CommandTableNaming.SchemaName = "audit";
    options.CommandTableNaming.Casing = TableNameCasing.SnakeCase;
    // Results in: audit.command_record

    // Entity store: Domain schema with pluralized tables
    options.EntityTableNaming.UseSchema = true;
    options.EntityTableNaming.SchemaName = "domain";
    options.EntityTableNaming.Casing = TableNameCasing.SnakeCase;
    options.EntityTableNaming.Pluralize = true;
    // BankAccount -> domain.bank_accounts

    // ViewModel store: Reporting schema with view suffix
    options.ViewModelTableNaming.UseSchema = true;
    options.ViewModelTableNaming.SchemaName = "reporting";
    options.ViewModelTableNaming.Casing = TableNameCasing.SnakeCase;
    options.ViewModelTableNaming.Suffix = "_view";
    options.ViewModelTableNaming.Pluralize = true;
    // AccountViewModel -> reporting.account_views
});

// Create schemas before initializing
var entityContext = serviceProvider.GetRequiredService<EntityDbContext>();
await entityContext.Database.ExecuteSqlRawAsync("CREATE SCHEMA IF NOT EXISTS audit");
await entityContext.Database.ExecuteSqlRawAsync("CREATE SCHEMA IF NOT EXISTS domain");
await entityContext.Database.ExecuteSqlRawAsync("CREATE SCHEMA IF NOT EXISTS reporting");
await entityContext.Database.EnsureCreatedAsync();
entityContext.ApplyMigrations();

Example 6: Background Service Processing Commands

Processing commands in a background service:

public class CommandProcessorBackgroundService : BackgroundService
{
    private readonly IServiceProvider _serviceProvider;
    private readonly ILogger<CommandProcessorBackgroundService> _logger;

    public CommandProcessorBackgroundService(
        IServiceProvider serviceProvider,
        ILogger<CommandProcessorBackgroundService> logger)
    {
        _serviceProvider = serviceProvider;
        _logger = logger;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        _logger.LogInformation("Command Processor Background Service starting");

        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                using var scope = _serviceProvider.CreateScope();

                var commandStore = scope.ServiceProvider.GetRequiredService<ICommandStoreAdapter>();
                var aggregateFactory = scope.ServiceProvider.GetRequiredService<IAggregateFactory>();

                // Process any pending commands (example: replay for specific entities)
                var entityIds = await GetPendingEntityIds();

                foreach (var entityId in entityIds)
                {
                    var commands = await commandStore.Retrieve(entityId);

                    if (commands.Any())
                    {
                        _logger.LogInformation(
                            "Processing {Count} commands for entity {EntityId}",
                            commands.Count(), entityId);

                        // Replay commands to rebuild state
                        var aggregate = await aggregateFactory.Create<IAccountAggregate>();
                        // Process commands...
                    }
                }

                await Task.Delay(TimeSpan.FromMinutes(5), stoppingToken);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error in command processor");
                await Task.Delay(TimeSpan.FromMinutes(1), stoppingToken);
            }
        }

        _logger.LogInformation("Command Processor Background Service stopping");
    }

    private async Task<List<int>> GetPendingEntityIds()
    {
        // Implementation to identify entities needing processing
        return new List<int>();
    }
}

// Register in Program.cs
builder.Services.AddHostedService<CommandProcessorBackgroundService>();

Example 7: Multi-Tenant Setup with Database Per Tenant

Implementing multi-tenancy with separate databases:

// ITenantProvider.cs
public interface ITenantProvider
{
    string GetCurrentTenantId();
    string GetConnectionString(string tenantId);
}

// TenantDbContextFactory.cs
public class TenantDbContextFactory<TContext> where TContext : DbContext
{
    private readonly ITenantProvider _tenantProvider;
    private readonly IServiceProvider _serviceProvider;

    public TenantDbContextFactory(ITenantProvider tenantProvider, IServiceProvider serviceProvider)
    {
        _tenantProvider = tenantProvider;
        _serviceProvider = serviceProvider;
    }

    public TContext CreateDbContext()
    {
        var tenantId = _tenantProvider.GetCurrentTenantId();
        var connectionString = _tenantProvider.GetConnectionString(tenantId);

        var optionsBuilder = new DbContextOptionsBuilder<TContext>();
        optionsBuilder.UseSqlServer(connectionString);

        return (TContext)Activator.CreateInstance(typeof(TContext), optionsBuilder.Options);
    }
}

// Program.cs
builder.Services.AddScoped<ITenantProvider, HttpContextTenantProvider>();

// Register SourceFlow with multi-tenant support
builder.Services.UseSourceFlow(typeof(Program).Assembly);

// Custom multi-tenant store registration
builder.Services.AddScoped(sp =>
{
    var tenantProvider = sp.GetRequiredService<ITenantProvider>();
    var tenantId = tenantProvider.GetCurrentTenantId();
    var connectionString = tenantProvider.GetConnectionString(tenantId);

    var optionsBuilder = new DbContextOptionsBuilder<EntityDbContext>();
    optionsBuilder.UseSqlServer(connectionString);

    return new EntityDbContext(optionsBuilder.Options);
});

// Similar for CommandDbContext and ViewModelDbContext

Implementation Guide

Creating a Complete Feature

Let's implement a complete banking feature using SourceFlow.Net with Entity Framework persistence:

1. Define Domain Objects

// Entity
public class BankAccount : IEntity
{
    public int Id { get; set; }
    public string AccountName { get; set; }
    public decimal Balance { get; set; }
    public bool IsClosed { get; set; }
    public DateTime CreatedOn { get; set; }
    public DateTime ActiveOn { get; set; }
    public string ClosureReason { get; set; }
}

// Command Payloads
public class CreateAccountPayload : IPayload
{
    public int Id { get; set; }
    public string AccountName { get; set; }
    public decimal InitialAmount { get; set; }
}

public class TransactionPayload : IPayload
{
    public int Id { get; set; }
    public decimal Amount { get; set; }
}

2. Create Commands

public class CreateAccount : Command<CreateAccountPayload>
{
    // Parameterless constructor required for command deserialization from store
    public CreateAccount() : base() { }

    public CreateAccount(CreateAccountPayload payload) : base(payload) { }
}

public class DepositMoney : Command<TransactionPayload>
{
    // Parameterless constructor required for command deserialization from store
    public DepositMoney() : base() { }

    public DepositMoney(TransactionPayload payload) : base(payload) { }
}

public class WithdrawMoney : Command<TransactionPayload>
{
    // Parameterless constructor required for command deserialization from store
    public WithdrawMoney() : base() { }

    public WithdrawMoney(TransactionPayload payload) : base(payload) { }
}

3. Define Events

public class AccountCreated : Event<BankAccount>
{
    public AccountCreated(BankAccount payload) : base(payload) { }
}

public class MoneyDeposited : Event<BankAccount>
{
    public MoneyDeposited(BankAccount payload) : base(payload) { }
}

public class MoneyWithdrawn : Event<BankAccount>
{
    public MoneyWithdrawn(BankAccount payload) : base(payload) { }
}

4. Implement Saga

public class AccountSaga : Saga<BankAccount>,
    IHandles<CreateAccount>,
    IHandlesWithEvent<CreateAccount, AccountCreated>,
    IHandles<DepositMoney>,
    IHandlesWithEvent<DepositMoney, MoneyDeposited>,
    IHandles<WithdrawMoney>,
    IHandlesWithEvent<WithdrawMoney, MoneyWithdrawn>
{
    public AccountSaga(
        Lazy<ICommandPublisher> commandPublisher,
        IEventQueue eventQueue,
        IEntityStoreAdapter entityStore,
        ILogger<ISaga> logger)
        : base(commandPublisher, eventQueue, entityStore, logger)
    {
    }

    public async Task<BankAccount> Handle(BankAccount entity, CreateAccount command)
    {
        // Validation
        if (string.IsNullOrEmpty(command.Payload.AccountName))
            throw new ArgumentException("Account name is required");

        if (command.Payload.InitialAmount <= 0)
            throw new ArgumentException("Initial amount must be positive");

        // Update entity
        entity.AccountName = command.Payload.AccountName;
        entity.Balance = command.Payload.InitialAmount;
        entity.CreatedOn = DateTime.UtcNow;
        entity.ActiveOn = DateTime.UtcNow;

        logger.LogInformation("Account created: {AccountId} for {Holder} with balance {Balance}",
            entity.Id, entity.AccountName, entity.Balance);

        // Return entity - framework automatically persists and raises AccountCreated event
        return entity;
    }

    public async Task<BankAccount> Handle(BankAccount entity, DepositMoney command)
    {
        if (entity.IsClosed)
            throw new InvalidOperationException("Cannot deposit to closed account");

        entity.Balance += command.Payload.Amount;

        logger.LogInformation("Deposited {Amount} to account {AccountId}. New balance: {Balance}",
            command.Payload.Amount, entity.Id, entity.Balance);

        // Return entity - framework automatically persists and raises MoneyDeposited event
        return entity;
    }

    public async Task<BankAccount> Handle(BankAccount entity, WithdrawMoney command)
    {
        if (entity.IsClosed)
            throw new InvalidOperationException("Cannot withdraw from closed account");

        if (entity.Balance < command.Payload.Amount)
            throw new InvalidOperationException("Insufficient funds");

        entity.Balance -= command.Payload.Amount;

        logger.LogInformation("Withdrew {Amount} from account {AccountId}. New balance: {Balance}",
            command.Payload.Amount, entity.Id, entity.Balance);

        // Return entity - framework automatically persists and raises MoneyWithdrawn event
        return entity;
    }
}

5. Create Aggregate

public interface IAccountAggregate : IAggregate
{
    void CreateAccount(int accountId, string holder, decimal amount);
    void Deposit(int accountId, decimal amount);
    void Withdraw(int accountId, decimal amount);
}

public class AccountAggregate : Aggregate<BankAccount>, IAccountAggregate
{
    public void CreateAccount(int accountId, string holder, decimal amount)
    {
        Send(new CreateAccount(new CreateAccountPayload
        {
            Id = accountId,
            AccountName = holder,
            InitialAmount = amount
        }));
    }

    public void Deposit(int accountId, decimal amount)
    {
        Send(new DepositMoney(new TransactionPayload
        {
            Id = accountId,
            Amount = amount
        }));
    }

    public void Withdraw(int accountId, decimal amount)
    {
        Send(new WithdrawMoney(new TransactionPayload
        {
            Id = accountId,
            Amount = amount
        }));
    }
}

6. Build Read Models

public class AccountViewModel : IViewModel
{
    public int Id { get; set; }
    public string AccountName { get; set; }
    public decimal CurrentBalance { get; set; }
    public DateTime CreatedDate { get; set; }
    public DateTime LastUpdated { get; set; }
    public int TransactionCount { get; set; }
    public bool IsClosed { get; set; }
    public string ClosureReason { get; set; }
    public int Version { get; set; }
    public DateTime ActiveOn { get; set; }
}

public class AccountView : View<AccountViewModel>,
    IProjectOn<AccountCreated>,
    IProjectOn<MoneyDeposited>,
    IProjectOn<MoneyWithdrawn>
{
    public AccountView(
        IViewModelStoreAdapter viewModelStore,
        ILogger<IView> logger)
        : base(viewModelStore, logger)
    {
    }

    public async Task<AccountViewModel> On(AccountCreated @event)
    {
        var viewModel = new AccountViewModel
        {
            Id = @event.Payload.Id,
            AccountName = @event.Payload.AccountName,
            CurrentBalance = @event.Payload.Balance,
            CreatedDate = @event.Payload.CreatedOn,
            LastUpdated = DateTime.UtcNow,
            TransactionCount = 0,
            IsClosed = false,
            ActiveOn = @event.Payload.ActiveOn
        };

        logger.LogInformation("Created view model for account {AccountId}", viewModel.Id);

        // Return view model - framework automatically persists it
        return viewModel;
    }

    public async Task<AccountViewModel> On(MoneyDeposited @event)
    {
        var viewModel = await Find<AccountViewModel>(@event.Payload.Id);
        viewModel.CurrentBalance = @event.Payload.Balance;
        viewModel.TransactionCount++;
        viewModel.LastUpdated = DateTime.UtcNow;

        logger.LogInformation("Updated view model for account {AccountId} after deposit", viewModel.Id);

        // Return view model - framework automatically persists it
        return viewModel;
    }

    public async Task<AccountViewModel> On(MoneyWithdrawn @event)
    {
        var viewModel = await Find<AccountViewModel>(@event.Payload.Id);
        viewModel.CurrentBalance = @event.Payload.Balance;
        viewModel.TransactionCount++;
        viewModel.LastUpdated = DateTime.UtcNow;

        logger.LogInformation("Updated view model for account {AccountId} after withdrawal", viewModel.Id);

        // Return view model - framework automatically persists it
        return viewModel;
    }
}

7. Application Setup

// Program.cs
using SourceFlow;
using SourceFlow.Stores.EntityFramework;
using SourceFlow.Stores.EntityFramework.Extensions;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;

var services = new ServiceCollection();

// Add logging
services.AddLogging(builder =>
{
    builder.AddConsole();
    builder.SetMinimumLevel(LogLevel.Information);
});

// Register entity and view model types BEFORE building service provider
EntityDbContext.RegisterEntityType<BankAccount>();
ViewModelDbContext.RegisterViewModelType<AccountViewModel>();

// Configure SourceFlow
services.UseSourceFlow(typeof(Program).Assembly);

// Add Entity Framework stores
services.AddSourceFlowEfStores(
    "Server=localhost;Database=SourceFlow;Integrated Security=true;TrustServerCertificate=true;");

var serviceProvider = services.BuildServiceProvider();

// Ensure databases are created and migrated
var commandContext = serviceProvider.GetRequiredService<CommandDbContext>();
await commandContext.Database.EnsureCreatedAsync();

var entityContext = serviceProvider.GetRequiredService<EntityDbContext>();
await entityContext.Database.EnsureCreatedAsync();
entityContext.ApplyMigrations();

var viewModelContext = serviceProvider.GetRequiredService<ViewModelDbContext>();
await viewModelContext.Database.EnsureCreatedAsync();
viewModelContext.ApplyMigrations();

// Use the aggregate
var aggregateFactory = serviceProvider.GetRequiredService<IAggregateFactory>();
var accountAggregate = await aggregateFactory.Create<IAccountAggregate>();

accountAggregate.CreateAccount(999, "John Doe", 1000m);
accountAggregate.Deposit(999, 500m);
accountAggregate.Withdraw(999, 200m);

// Give async processing time to complete
await Task.Delay(500);

// Query the read model
var viewModelStore = serviceProvider.GetRequiredService<IViewModelStoreAdapter>();
var accountView = await viewModelStore.Find<AccountViewModel>(999);

Console.WriteLine($"Account: {accountView.AccountName}");
Console.WriteLine($"Balance: {accountView.CurrentBalance:C}");
Console.WriteLine($"Transactions: {accountView.TransactionCount}");

For complete examples including ASP.NET Core, PostgreSQL, and production configurations, see EntityFramework Usage Examples.

Advanced Features

Event Replay

SourceFlow.Net provides built-in command replay functionality for debugging and state reconstruction:

var accountAggregate = serviceProvider.GetRequiredService<IAccountAggregate>();

// Replay all commands for an aggregate
await accountAggregate.ReplayHistory(accountId);

// The framework automatically handles:
// 1. Loading commands from store
// 2. Marking commands as replay
// 3. Re-executing command handlers
// 4. Updating projections

Metadata and Auditing

Every command and event includes rich metadata to add producer and consumer centric custom properties.

public interface IMetadata
{
    Guid EventId { get; set; }
    bool IsReplay { get; set; }
    DateTime OccurredOn { get; set; }
    int SequenceNo { get; set; }
    IDictionary<string, object> Properties { get; set; }
}

Store Adapters

SourceFlow provides high-level adapters for common operations:

// Entity Store Adapter
public interface IEntityStoreAdapter
{
    Task<TEntity> Persist<TEntity>(TEntity entity) where TEntity : class, IEntity;
    Task<TEntity> Get<TEntity>(int id) where TEntity : class, IEntity;
}

// ViewModel Store Adapter
public interface IViewModelStoreAdapter
{
    Task<TViewModel> Persist<TViewModel>(TViewModel model) where TViewModel : class, IViewModel;
    Task<TViewModel> Find<TViewModel>(int id) where TViewModel : class, IViewModel;
}

// Command Store Adapter
public interface ICommandStoreAdapter
{
    Task Commit(ICommand command);
    Task<IEnumerable<ICommand>> Retrieve(int entityId);
}

View Base Class

SourceFlow provides an abstract View<TViewModel> base class for building projections:

public abstract class View<TViewModel> : IView
    where TViewModel : class, IViewModel
{
    protected IViewModelStoreAdapter viewModelStore;
    protected ILogger<IView> logger;

    protected View(
        IViewModelStoreAdapter viewModelStore,
        ILogger<IView> logger);

    // Find existing view model
    protected Task<TViewModel> Find<TViewModel>(int id) where TViewModel : class, IViewModel;

    // Persist view model (returns persisted instance)
    protected Task<TViewModel> Persist<TViewModel>(TViewModel viewModel) where TViewModel : class, IViewModel;
}

Key Features:

Projection handling via IProjectOn<TEvent> with signature: Task<TViewModel> On(TEvent @event)
View models are automatically persisted after On completes
Return null from On to skip persistence
Access to viewModelStore for finding existing view models
Built-in logging
Type-safe view model management

Performance and Observability

SourceFlow.Net includes comprehensive production-ready features for monitoring, fault tolerance, and high-performance scenarios.

OpenTelemetry Integration

Built-in support for distributed tracing and metrics collection at scale.

Features

Distributed Tracing: Automatically track command execution, event dispatching, and store operations
Metrics Collection: Monitor command rates, saga executions, entity creations, and operation durations
Multiple Exporters: Support for Console, OTLP (Jaeger, Zipkin), and custom exporters
Minimal Overhead: <1ms latency impact, <2% CPU overhead

Quick Setup

Development (Console Exporter):

using SourceFlow.Observability;
using OpenTelemetry;

var services = new ServiceCollection();

// Enable observability with console output
services.AddSourceFlowTelemetry(
    serviceName: "MyEventSourcedApp",
    serviceVersion: "1.0.0");

services.AddOpenTelemetry()
    .AddSourceFlowConsoleExporter();

services.UseSourceFlow();

Production (OTLP Exporter):

services.AddSourceFlowTelemetry(options =>
{
    options.Enabled = true;
    options.ServiceName = "ProductionApp";
    options.ServiceVersion = "1.0.0";
});

services.AddOpenTelemetry()
    .AddSourceFlowOtlpExporter("http://localhost:4317")
    .AddSourceFlowResourceAttributes(
        ("environment", "production"),
        ("region", "us-east-1")
    );

Instrumented Operations

All core operations are automatically traced:

Command Operations:

sourceflow.commandbus.dispatch - Command dispatch and persistence
sourceflow.commanddispatcher.send - Command distribution to sagas
sourceflow.domain.command.append - Command persistence
sourceflow.domain.command.load - Command loading

Event Operations:

sourceflow.eventqueue.enqueue - Event queuing
sourceflow.eventdispatcher.dispatch - Event distribution

Store Operations:

sourceflow.entitystore.persist / get / delete - Entity operations
sourceflow.viewmodelstore.persist / find / delete - ViewModel operations

Metrics Collected

sourceflow.domain.commands.executed - Counter of executed commands
sourceflow.domain.sagas.executed - Counter of saga executions
sourceflow.domain.entities.created - Counter of entity creations
sourceflow.domain.operation.duration - Histogram of operation durations (ms)
sourceflow.domain.serialization.duration - Histogram of serialization performance

Integration with Existing Telemetry

services.AddOpenTelemetry()
    .WithTracing(builder => builder
        .AddSource("SourceFlow.Domain")  // Add SourceFlow
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddOtlpExporter())
    .WithMetrics(builder => builder
        .AddMeter("SourceFlow.Domain")   // Add SourceFlow
        .AddAspNetCoreInstrumentation()
        .AddPrometheusExporter());

ArrayPool Memory Optimization

Dramatically reduce memory allocations in high-throughput scenarios using ArrayPool<T>.

Performance Benefits

Memory Allocation Reduction:

Before: ~40MB allocations for 10,000 commands
After: <1MB allocations for 10,000 commands
Result: ~40x reduction in allocations

GC Pressure Reduction:

Gen 0 Collections: ↓70%
Gen 1 Collections: ↓50%
Gen 2 Collections: ↓30%

Throughput Improvements:

Command Throughput: +25-40%
Event Dispatching: +30-50%
Serialization: +20-35%

Features

Task Buffer Pooling: Reduces allocations in parallel task execution
JSON Serialization Pooling: Reuses byte buffers for JSON operations
Zero Configuration: Works automatically, no code changes required
Production Tested: Optimized for extreme throughput scenarios

Optimized Components

TaskBufferPool:

Pools task arrays for parallel execution
Used in CommandDispatcher and EventDispatcher
Automatic buffer rental and return

ByteArrayPool:

Pools byte arrays for JSON serialization
Used in CommandStoreAdapter
Custom IBufferWriter<byte> implementation

ArrayPool optimizations are automatically applied to:

Command serialization/deserialization
Event dispatching (parallel task execution)
Store adapter operations

Resilience with Polly (Entity Framework)

The Entity Framework integration includes Polly-based resilience patterns for fault tolerance.

Features

Retry Policy: Automatic retry with exponential backoff and jitter
Circuit Breaker: Prevents cascading failures
Timeout Policy: Enforces maximum execution time

Configuration

services.AddSourceFlowEfStores(options =>
{
    options.DefaultConnectionString = connectionString;

    // Enable resilience
    options.Resilience.Enabled = true;

    // Retry configuration
    options.Resilience.Retry.MaxRetryAttempts = 3;
    options.Resilience.Retry.BaseDelayMs = 1000;
    options.Resilience.Retry.UseExponentialBackoff = true;
    options.Resilience.Retry.UseJitter = true;

    // Circuit breaker configuration
    options.Resilience.CircuitBreaker.Enabled = true;
    options.Resilience.CircuitBreaker.FailureThreshold = 5;
    options.Resilience.CircuitBreaker.BreakDurationMs = 30000;

    // Timeout configuration
    options.Resilience.Timeout.Enabled = true;
    options.Resilience.Timeout.TimeoutMs = 30000;
});

Benefits

Transient Failure Handling: Automatically recovers from temporary issues
Cascading Failure Prevention: Circuit breaker stops calling failing services
Resource Protection: Timeouts prevent hanging operations
Self-Healing: System automatically recovers when service becomes available

Entity Framework Observability

Additional observability features specific to Entity Framework stores.

Configuration

services.AddSourceFlowEfStores(options =>
{
    options.DefaultConnectionString = connectionString;

    // Configure observability
    options.Observability.Enabled = true;
    options.Observability.ServiceName = "MyApplication";
    options.Observability.ServiceVersion = "1.0.0";

    // Tracing configuration
    options.Observability.Tracing.Enabled = true;
    options.Observability.Tracing.TraceDatabaseOperations = true;
    options.Observability.Tracing.IncludeSqlInTraces = false;  // Enable for debugging
    options.Observability.Tracing.SamplingRatio = 0.1;         // Sample 10% in production

    // Metrics configuration
    options.Observability.Metrics.Enabled = true;
    options.Observability.Metrics.CollectDatabaseMetrics = true;
});

// Configure exporters
services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .AddSource("SourceFlow.EntityFramework")
        .AddEntityFrameworkCoreInstrumentation()
        .AddJaegerExporter())
    .WithMetrics(metrics => metrics
        .AddMeter("SourceFlow.EntityFramework")
        .AddPrometheusExporter());

Additional Traces

sourceflow.ef.command.append - EF command storage
sourceflow.ef.command.load - EF command loading
sourceflow.ef.entity.persist - EF entity persistence
sourceflow.ef.viewmodel.persist - EF view model persistence

Additional Metrics

sourceflow.commands.appended - EF command append counter
sourceflow.commands.loaded - EF command load counter
sourceflow.entities.persisted - EF entity persistence counter
sourceflow.viewmodels.persisted - EF view model persistence counter
sourceflow.database.connections - Active connection gauge

Production Configuration Examples

Development:

services.AddSourceFlowTelemetry("DevApp", "1.0.0");
services.AddOpenTelemetry()
    .AddSourceFlowConsoleExporter();

services.AddSourceFlowEfStores(options =>
{
    options.DefaultConnectionString = "Data Source=dev.db";
    options.Resilience.Enabled = false;  // Easier debugging
    options.Observability.Enabled = true;
    options.Observability.Tracing.IncludeSqlInTraces = true;
    options.Observability.Tracing.SamplingRatio = 1.0;  // Trace everything
});

Production:

services.AddSourceFlowTelemetry(options =>
{
    options.Enabled = true;
    options.ServiceName = "ProductionApp";
    options.ServiceVersion = "1.0.0";
});

services.AddOpenTelemetry()
    .AddSourceFlowOtlpExporter(otlpEndpoint)
    .AddSourceFlowResourceAttributes(
        ("environment", "production"),
        ("deployment.region", region)
    );

services.AddSourceFlowEfStores(options =>
{
    options.DefaultConnectionString = connectionString;

    // Production resilience settings
    options.Resilience.Enabled = true;
    options.Resilience.Retry.MaxRetryAttempts = 3;
    options.Resilience.CircuitBreaker.Enabled = true;
    options.Resilience.CircuitBreaker.FailureThreshold = 10;

    // Production observability settings
    options.Observability.Enabled = true;
    options.Observability.Tracing.IncludeSqlInTraces = false;
    options.Observability.Tracing.SamplingRatio = 0.1;  // Sample 10%
});

High-Throughput:

services.AddSourceFlowTelemetry(options =>
{
    options.Enabled = true;
    options.ServiceName = "HighThroughputApp";
});

services.AddOpenTelemetry()
    .AddSourceFlowOtlpExporter(otlpEndpoint)
    .ConfigureSourceFlowBatchProcessing(
        maxQueueSize: 2048,
        maxExportBatchSize: 512,
        scheduledDelayMilliseconds: 5000
    );

services.AddSourceFlowEfStores(options =>
{
    options.DefaultConnectionString = connectionString;

    // Optimized for throughput
    options.Resilience.Enabled = true;
    options.Resilience.Retry.MaxRetryAttempts = 2;
    options.Resilience.Retry.BaseDelayMs = 500;

    // Reduced overhead
    options.Observability.Enabled = true;
    options.Observability.Tracing.SamplingRatio = 0.01;  // Sample 1%
});

// ArrayPool optimizations are automatically applied

Monitoring Dashboard Queries

Use these queries in Grafana/Prometheus:

Average Command Processing Time:

rate(sourceflow_domain_operation_duration_sum{operation="sourceflow.commandbus.dispatch"}[5m])
/ rate(sourceflow_domain_operation_duration_count{operation="sourceflow.commandbus.dispatch"}[5m])

Command Throughput:

rate(sourceflow_domain_commands_executed[5m])

Serialization Performance (P95):

histogram_quantile(0.95,
  rate(sourceflow_domain_serialization_duration_bucket[5m])
)

Package Dependencies

Core SourceFlow:

OpenTelemetry (1.14.0)
OpenTelemetry.Api (1.14.0)
OpenTelemetry.Exporter.Console (1.14.0)
OpenTelemetry.Exporter.OpenTelemetryProtocol (1.14.0)
OpenTelemetry.Extensions.Hosting (1.14.0)
Microsoft.Extensions.DependencyInjection.Abstractions (10.0.0)
Microsoft.Extensions.Logging.Abstractions (10.0.0)

Entity Framework Stores:

Microsoft.EntityFrameworkCore (9.0.0)
Polly (8.4.2) - For resilience patterns
OpenTelemetry.Instrumentation.EntityFrameworkCore (1.0.0-beta.12)

All packages are free from known vulnerabilities (as of November 2025).

Additional Resources

OpenTelemetry Documentation: https://opentelemetry.io/docs/
ArrayPool Documentation: https://docs.microsoft.com/en-us/dotnet/api/system.buffers.arraypool-1
Polly Documentation: https://github.com/App-vNext/Polly
SourceFlow.Net OBSERVABILITY_AND_PERFORMANCE.md: Detailed performance documentation
SourceFlow.Stores.EntityFramework ENHANCEMENTS.md: EF-specific enhancements guide

Best Practices

1. Command Design

Always include a parameterless constructor for serialization support:

// ✅ Good: Specific, intention-revealing commands with proper constructors
public class WithdrawMoney : Command<WithdrawPayload>
{
    // Required for deserialization from command store
    public WithdrawMoney() : base() { }

    public WithdrawMoney(WithdrawPayload payload) : base(payload) { }
}

public class DepositMoney : Command<DepositPayload>
{
    // Required for deserialization from command store
    public DepositMoney() : base() { }

    public DepositMoney(DepositPayload payload) : base(payload) { }
}

// ❌ Bad: Generic, unclear commands
public class UpdateAccount : Command<AccountPayload> { }

// ❌ Bad: Missing parameterless constructor
public class TransferMoney : Command<TransferPayload>
{
    public TransferMoney(TransferPayload payload) : base(payload) { }
    // This command cannot be deserialized from the command store!
}

Key Requirements:

Use specific, intention-revealing names
Always include a public parameterless constructor
Include a constructor that accepts the payload
Keep commands immutable after creation

2. Event Granularity

// ✅ Good: Fine-grained, specific events
public class AccountCreated : Event<BankAccount> { }
public class AccountCredited : Event<BankAccount> { }
public class AccountDebited : Event<BankAccount> { }

// ❌ Bad: Coarse-grained, generic events
public class AccountChanged : Event<BankAccount> { }

3. Saga Responsibility and Patterns

// ✅ Good: Single responsibility with automatic event raising
public class AccountSaga : Saga<BankAccount>,
    IHandles<CreateAccount>,
    IHandlesWithEvent<CreateAccount, AccountCreated>,
    IHandles<CloseAccount>,
    IHandlesWithEvent<CloseAccount, AccountClosed>
{
    public AccountSaga(
        Lazy<ICommandPublisher> commandPublisher,
        IEventQueue eventQueue,
        IEntityStoreAdapter entityStore,
        ILogger<ISaga> logger)
        : base(commandPublisher, eventQueue, entityStore, logger)
    {
    }

    public async Task<BankAccount> Handle(BankAccount entity, CreateAccount command)
    {
        // Update and return entity - framework handles persistence and event raising
        entity.AccountName = command.Payload.AccountName;
        entity.Balance = command.Payload.InitialAmount;
        return entity;
    }
}

// ❌ Bad: Missing IHandlesWithEvent (events won't be raised automatically)
public class AccountSaga : Saga<BankAccount>,
    IHandles<CreateAccount>  // Missing IHandlesWithEvent!
{
    public async Task<BankAccount> Handle(BankAccount entity, CreateAccount command)
    {
        entity.AccountName = command.Payload.AccountName;
        return entity;
        // No AccountCreated event will be raised!
    }
}

// ❌ Bad: Multiple responsibilities
public class MegaSaga : Saga<BankAccount>,
    IHandles<CreateAccount>,
    IHandles<ProcessLoan>,
    IHandles<SendEmail>
{
    // Too many responsibilities
}

4. Type Registration

// ✅ Good: Register types early, before building service provider
EntityDbContext.RegisterAssembly(typeof(BankAccount).Assembly);
ViewModelDbContext.RegisterAssembly(typeof(AccountViewModel).Assembly);

var serviceProvider = services.BuildServiceProvider();

// Apply migrations after database creation
entityContext.Database.EnsureCreated();
entityContext.ApplyMigrations();

// ❌ Bad: Relying solely on auto-discovery
var serviceProvider = services.BuildServiceProvider();
// Types may not be discovered reliably

5. View/Projection Patterns

// ✅ Good: Inherit from View<TViewModel> and return view model
public class AccountView : View<AccountViewModel>,
    IProjectOn<AccountCreated>,
    IProjectOn<MoneyDeposited>
{
    public AccountView(
        IViewModelStoreAdapter viewModelStore,
        ILogger<IView> logger)
        : base(viewModelStore, logger)
    {
    }

    public async Task<AccountViewModel> On(AccountCreated @event)
    {
        var viewModel = new AccountViewModel
        {
            Id = @event.Payload.Id,
            AccountName = @event.Payload.AccountName,
            CurrentBalance = @event.Payload.Balance
        };
        // Return view model - framework automatically persists it
        return viewModel;
    }

    public async Task<AccountViewModel> On(MoneyDeposited @event)
    {
        var viewModel = await Find<AccountViewModel>(@event.Payload.Id);
        viewModel.CurrentBalance += @event.Payload.Amount;
        return viewModel;
    }
}

// ❌ Bad: Old pattern - manually calling persistence
public class AccountView : IProjectOn<AccountCreated>
{
    private IViewModelStoreAdapter provider;

    public async Task Apply(AccountCreated @event)
    {
        var viewModel = new AccountViewModel { /* ... */ };
        await provider.Push(viewModel);  // Manual persistence - not needed!
    }
}

6. Error Handling

public class AccountSaga : Saga<BankAccount>,
    IHandles<WithdrawMoney>,
    IHandlesWithEvent<WithdrawMoney, MoneyWithdrawn>
{
    public async Task<BankAccount> Handle(BankAccount entity, WithdrawMoney command)
    {
        try
        {
            // Validate business rules
            if (entity.IsClosed)
                throw new AccountClosedException($"Account {entity.Id} is closed");

            if (entity.Balance < command.Payload.Amount)
                throw new InsufficientFundsException($"Insufficient funds in account {entity.Id}");

            // Process transaction
            entity.Balance -= command.Payload.Amount;

            logger.LogInformation("Withdrew {Amount} from account {AccountId}",
                command.Payload.Amount, entity.Id);

            // Return entity - framework persists and raises MoneyWithdrawn event
            return entity;
        }
        catch (Exception ex)
        {
            logger.LogError(ex, "Failed to process withdrawal for account {AccountId}",
                command.Payload.Id);

            // Optionally publish failure event
            await Raise(new WithdrawalFailed(new WithdrawalFailureDetails
            {
                AccountId = command.Payload.Id,
                Amount = command.Payload.Amount,
                Reason = ex.Message
            }));

            throw;
        }
    }
}

7. Database Migrations

// ✅ Good: Use ApplyMigrations for dynamic types
entityContext.Database.EnsureCreated();
entityContext.ApplyMigrations(); // Creates tables for registered types

viewModelContext.Database.EnsureCreated();
viewModelContext.ApplyMigrations(); // Creates tables for view models

// For production: Use EF Core migrations for static schema
// dotnet ef migrations add InitialCreate
// dotnet ef database update

8. Production Monitoring

// ✅ Good: Enable observability in production
services.AddSourceFlowTelemetry(options =>
{
    options.Enabled = true;
    options.ServiceName = "ProductionApp";
    options.ServiceVersion = Assembly.GetExecutingAssembly()
        .GetName().Version.ToString();
});

// Configure appropriate sampling rate
services.AddOpenTelemetry()
    .AddSourceFlowOtlpExporter(otlpEndpoint)
    .AddSourceFlowResourceAttributes(
        ("environment", Environment.GetEnvironmentVariable("ENVIRONMENT"))
    );

// ❌ Bad: No observability in production
services.UseSourceFlow();
// Can't diagnose performance issues or failures

9. Resilience Configuration

// ✅ Good: Enable resilience for production databases
services.AddSourceFlowEfStores(options =>
{
    options.DefaultConnectionString = connectionString;
    options.Resilience.Enabled = true;
    options.Resilience.Retry.MaxRetryAttempts = 3;
    options.Resilience.CircuitBreaker.Enabled = true;
});

// ❌ Bad: No resilience (will fail on transient errors)
services.AddSourceFlowEfStores(connectionString);

10. Command Serialization Requirements

// ✅ Good: Command with parameterless constructor
public class ProcessPayment : Command<PaymentPayload>
{
    // REQUIRED: Parameterless constructor for deserialization
    public ProcessPayment() : base() { }

    public ProcessPayment(PaymentPayload payload) : base(payload) { }
}

// ❌ Bad: Missing parameterless constructor
public class ProcessPayment : Command<PaymentPayload>
{
    public ProcessPayment(PaymentPayload payload) : base(payload) { }
    // Will throw MissingMethodException during command replay!
}

// ✅ Good: Payload classes don't need parameterless constructors
public class PaymentPayload : IPayload
{
    public int Id { get; set; }
    public decimal Amount { get; set; }
    public string Currency { get; set; }
}

Important Notes:

Commands MUST have a public parameterless constructor for deserialization from the command store
Payload classes use property setters for deserialization (no parameterless constructor required)
Without a parameterless constructor, command replay and aggregate reconstruction will fail
The Entity Framework CommandStoreAdapter uses reflection to recreate command instances

FAQ

Q: How does SourceFlow.Net handle persistence?

A: SourceFlow.Net uses a store abstraction pattern with multiple implementation options:

In-Memory Stores: Built-in for testing and prototyping
Entity Framework Stores: Production-ready with support for SQL Server, PostgreSQL, SQLite, etc.
Custom Stores: Implement ICommandStore, IEntityStore, and IViewModelStore for your own persistence

Q: Can I use different databases for commands, entities, and view models?

A: Yes! The Entity Framework integration supports separate databases:

services.AddSourceFlowEfStoresWithCustomProviders(
    commandContextConfig: options => options.UseSqlServer("..."),
    entityContextConfig: options => options.UsePostgreSql("..."),
    viewModelContextConfig: options => options.UseSqlite("...")
);

Q: How do I handle dynamic entity and view model types?

A: Use the type registration and migration system:

Register types before building the service provider
Call EnsureCreated() to create the base schema
Call ApplyMigrations() to create tables for registered types

EntityDbContext.RegisterEntityType<MyEntity>();
ViewModelDbContext.RegisterViewModelType<MyViewModel>();

// Build service provider...

entityContext.Database.EnsureCreated();
entityContext.ApplyMigrations();

Q: Why do my commands need a parameterless constructor?

A: The CommandStoreAdapter uses reflection to deserialize commands from the database. When replaying commands, it needs to create instances without knowing the payload in advance.

Required pattern:

public class CreateAccount : Command<CreateAccountPayload>
{
    // Required for deserialization
    public CreateAccount() : base() { }

    // Used for creating new commands
    public CreateAccount(CreateAccountPayload payload) : base(payload) { }
}

Without the parameterless constructor, command replay will fail with a MissingMethodException.

Q: What database providers are supported?

A: SourceFlow.Stores.EntityFramework supports any EF Core provider:

SQL Server (default)
PostgreSQL (via Npgsql.EntityFrameworkCore.PostgreSQL)
SQLite (via Microsoft.EntityFrameworkCore.Sqlite)
MySQL (via Pomelo.EntityFrameworkCore.MySql)
In-Memory (via Microsoft.EntityFrameworkCore.InMemoryDatabase)
And more...

Q: How do I test with SourceFlow.Net?

A: Use in-memory databases for fast, isolated tests:

[SetUp]
public void Setup()
{
    EntityDbContext.RegisterEntityType<TestEntity>();
    ViewModelDbContext.RegisterViewModelType<TestViewModel>();

    var connection = new SqliteConnection("DataSource=:memory:");
    connection.Open();

    services.AddSourceFlowEfStoresWithCustomProvider(options =>
        options.UseSqlite(connection));

    // Build and setup...
}

Q: Why use the "T" prefix for table names?

A: The "T" prefix distinguishes dynamically created tables from EF Core's built-in tables, making it clear which tables are part of your domain model versus infrastructure.

Q: Should I enable observability in production?

A: Yes! Observability has minimal overhead (<1ms latency, <2% CPU) and provides invaluable insights:

Distributed tracing helps debug issues across services
Metrics help identify performance bottlenecks
Sampling (10%) provides good coverage with minimal cost

services.AddSourceFlowTelemetry(options =>
{
    options.Enabled = true;
    options.ServiceName = "ProductionApp";
});
services.AddOpenTelemetry()
    .AddSourceFlowOtlpExporter(otlpEndpoint);

Q: When should I use resilience patterns?

A: Always enable resilience in production for database operations:

Retry policies handle transient network failures
Circuit breakers prevent cascading failures
Timeouts prevent hanging operations

services.AddSourceFlowEfStores(options =>
{
    options.Resilience.Enabled = true;
});

Q: How much does ArrayPool improve performance?

A: In high-throughput scenarios (>1000 commands/second):

Memory: 40x reduction in allocations (40MB → <1MB for 10K commands)
GC: 70% reduction in Gen0 collections
Throughput: 25-40% improvement
Zero configuration: Works automatically once enabled

ArrayPool optimizations are built-in and automatically applied to command serialization and event dispatching.

Q: How do I handle schema changes?

A: For production applications:

Use EF Core migrations for base schema
Use ApplyMigrations() for dynamic types
Version your entities and view models
Implement upcasting for old events

For development/testing:

Use Database.EnsureDeleted() and EnsureCreated()
Use in-memory databases that reset on each test

Q: Can I use SourceFlow.EntityFramework with MySQL or other databases?

A: Yes! Use AddSourceFlowEfStoresWithCustomProvider for any EF Core supported database:

// MySQL
var serverVersion = new MySqlServerVersion(new Version(8, 0, 21));
services.AddSourceFlowEfStoresWithCustomProvider(options =>
    options.UseMySql(connectionString, serverVersion));

// SQLite
services.AddSourceFlowEfStoresWithCustomProvider(options =>
    options.UseSqlite("Data Source=sourceflow.db"));

// PostgreSQL
services.AddSourceFlowEfStoresWithCustomProvider(options =>
    options.UseNpgsql(connectionString));

The AddSourceFlowEfStores methods without "CustomProvider" use SQL Server by default.

Q: What's the difference between EnsureCreated() and ApplyMigrations()?

EnsureCreated(): Creates the database and base schema (Commands, fixed tables)
ApplyMigrations(): Creates tables for dynamically registered entities and view models

Always call both in the correct order:

await entityContext.Database.EnsureCreatedAsync();  // Create database
entityContext.ApplyMigrations();                     // Create dynamic tables

Q: How do I configure EF Core 9.0 for testing to avoid provider conflicts?

A: When testing with multiple DbContext providers (e.g., SQLite for tests, SQL Server for production), use EnableServiceProviderCaching(false):

services.AddDbContext<EntityDbContext>(options =>
    options.UseSqlite(connection)
        .EnableServiceProviderCaching(false));  // Required for EF Core 9.0

This prevents the "multiple provider" error when using different providers in the same service collection.

Q: Should I register stores manually or use AddSourceFlowEfStores?

A: Use AddSourceFlowEfStores for production. Only register manually for special cases like:

Testing scenarios requiring specific service configuration
Custom implementations of resilience or telemetry services
Avoiding provider conflicts in test setups

Example manual registration:

var efOptions = new SourceFlowEfOptions();
services.AddSingleton(efOptions);
services.AddScoped<IDatabaseResiliencePolicy, DatabaseResiliencePolicy>();
services.AddScoped<IDatabaseTelemetryService, DatabaseTelemetryService>();
services.AddScoped<ICommandStore, EfCommandStore>();
services.AddScoped<IEntityStore, EfEntityStore>();
services.AddScoped<IViewModelStore, EfViewModelStore>();

Q: How do table naming conventions affect my database schema?

A: Table naming conventions transform entity type names into table names:

// Default (PascalCase, no prefix/suffix)
BankAccount → BankAccount

// Snake case with pluralization
options.EntityTableNaming.Casing = TableNameCasing.SnakeCase;
options.EntityTableNaming.Pluralize = true;
BankAccount → bank_accounts

// With schema
options.EntityTableNaming.UseSchema = true;
options.EntityTableNaming.SchemaName = "domain";
BankAccount → domain.BankAccount

// Combined
BankAccount → domain.bank_accounts

Set naming conventions BEFORE calling ApplyMigrations() to ensure tables are created with the correct names.

Production Considerations

Performance Optimization

Use Separate Databases: Split command, entity, and view model stores across different databases
Enable Connection Pooling: Configure appropriate connection pool sizes
Optimize Queries: Use AsNoTracking() for read-only queries
Batch Operations: Use bulk insert/update operations where applicable
Enable ArrayPool: Automatically enabled for high-throughput scenarios (40x reduction in allocations)
Configure Observability: Use appropriate sampling rates for production (1-10%)
Enable Resilience: Use Polly policies for fault tolerance in production

Monitoring

// Health checks
services.AddHealthChecks()
    .AddDbContextCheck<CommandDbContext>("commandstore")
    .AddDbContextCheck<EntityDbContext>("entitystore")
    .AddDbContextCheck<ViewModelDbContext>("viewmodelstore");

// OpenTelemetry metrics and tracing
services.AddSourceFlowTelemetry("ProductionApp", "1.0.0");
services.AddOpenTelemetry()
    .AddSourceFlowOtlpExporter("http://localhost:4317");

// Monitor key metrics:
// - Command throughput: sourceflow.domain.commands.executed
// - Operation latency: sourceflow.domain.operation.duration (P50/P95/P99)
// - Circuit breaker state: polly.circuit_breaker.state
// - GC pressure: dotnet.gc.collections (reduced with ArrayPool)

Deployment

Migrations: Apply EF Core migrations during deployment
Connection Strings: Use environment-specific configuration
Logging: Configure appropriate logging levels
Error Handling: Implement global exception handling

Common Issues and Solutions

Command Deserialization Failures:

Symptom: MissingMethodException or InvalidOperationException during command replay
Cause: Command class missing parameterless constructor
Solution: Add public parameterless constructor to all command classes

// Fix: Add parameterless constructor
public class MyCommand : Command<MyPayload>
{
    public MyCommand() : base() { }  // Required!
    public MyCommand(MyPayload payload) : base(payload) { }
}

Entity Tracking Conflicts:

Symptom: "Instance already being tracked" errors
Cause: Multiple entity instances with same ID in EF change tracker
Solution: Use AsNoTracking() for read operations or detach entities after save

EF Core 9.0 Provider Conflicts (Testing):

Symptom: "An error occurred accessing the database provider factory"
Cause: Multiple providers registered in same service collection
Solution: Use EnableServiceProviderCaching(false) in test configurations

Migration Failures:

Symptom: Tables not created for entities or view models
Cause: Types not registered before calling ApplyMigrations()
Solution: Register types using EntityDbContext.RegisterAssembly() before building service provider

Community and Support

Resources

GitHub Repository: https://github.com/CodeShayk/SourceFlow.Net
Documentation: https://github.com/CodeShayk/SourceFlow.Net/wiki
Issues: https://github.com/CodeShayk/SourceFlow.Net/issues
Discussions: https://github.com/CodeShayk/SourceFlow.Net/discussions

License

SourceFlow.Net is released under the MIT License, making it free for both commercial and open-source use.

Conclusion

SourceFlow.Net provides a robust, scalable foundation for building event-sourced applications with .NET. By combining Event Sourcing, Domain-Driven Design, and CQRS patterns with flexible Entity Framework persistence, it enables developers to create maintainable, auditable, and performant systems.

Start your journey with SourceFlow.Net today and build better software with events as your foundation!

TurboMapper

CØDE N!NJΔ — Thu, 13 Nov 2025 19:46:50 +0000

Introduction
Getting Started
Basic Usage
Advanced Features
Mapping Modules
Dependency Injection
Performance Considerations
API Reference
Best Practices
Troubleshooting

Introduction

TurboMapper is a lightweight, high-performance object mapper for .NET that provides both shallow and deep mapping capabilities. It serves as a free alternative to AutoMapper with a simple, intuitive API.

Key Features

Automatic Property Mapping: Maps properties by name automatically
Custom Property Mapping: Define explicit mappings between different property names
Nested Object Support: Handles deep object hierarchies seamlessly
Type Conversion: Automatic conversion between compatible types
Mapping Modules: Organize mappings in reusable modules
Dependency Injection: First-class support for Microsoft.Extensions.DependencyInjection
Thread-Safe: Safe for concurrent operations
Multi-Framework Support: Compatible with .NET Framework 4.6.2, .NET Standard 2.0/2.1, and .NET 9.0

Installation

dotnet add package TurboMapper

Getting Started

Quick Start Example

using TurboMapper;
using Microsoft.Extensions.DependencyInjection;

// Setup
var services = new ServiceCollection();
services.AddTurboMapper();
var serviceProvider = services.BuildServiceProvider();
var mapper = serviceProvider.GetService<IMapper>();

// Define models
public class Source
{
    public string Name { get; set; }
    public int Age { get; set; }
}

public class Target
{
    public string Name { get; set; }
    public int Age { get; set; }
}

// Map objects
var source = new Source { Name = "John Doe", Age = 30 };
var target = mapper.Map<Source, Target>(source);

Basic Usage

Default Name-Based Mapping

TurboMapper automatically maps properties with matching names:

public class UserDTO
{
    public string FirstName { get; set; }
    public string LastName { get; set; }
    public int Age { get; set; }
}

public class UserEntity
{
    public string FirstName { get; set; }
    public string LastName { get; set; }
    public int Age { get; set; }
}

var dto = new UserDTO 
{ 
    FirstName = "Jane", 
    LastName = "Smith", 
    Age = 28 
};

var entity = mapper.Map<UserDTO, UserEntity>(dto);
// entity.FirstName = "Jane"
// entity.LastName = "Smith"
// entity.Age = 28

Nested Object Mapping

TurboMapper automatically handles nested objects:

public class Address
{
    public string Street { get; set; }
    public string City { get; set; }
    public string ZipCode { get; set; }
}

public class Person
{
    public string Name { get; set; }
    public Address Address { get; set; }
}

var source = new Person
{
    Name = "John",
    Address = new Address
    {
        Street = "123 Main St",
        City = "Boston",
        ZipCode = "02101"
    }
};

var target = mapper.Map<Person, Person>(source);
// All nested properties are mapped automatically

Type Conversion

TurboMapper handles common type conversions:

public class Source
{
    public int Age { get; set; }
    public string Status { get; set; }
}

public class Target
{
    public string Age { get; set; }  // int to string
    public StatusEnum Status { get; set; }  // string to enum
}

public enum StatusEnum { Active, Inactive }

var source = new Source { Age = 25, Status = "Active" };
var target = mapper.Map<Source, Target>(source);
// target.Age = "25"
// target.Status = StatusEnum.Active

Null Handling

TurboMapper gracefully handles null values:

// Null source returns null
Source source = null;
var target = mapper.Map<Source, Target>(source); // returns null

// Null nested objects are preserved
var person = new Person { Name = "John", Address = null };
var mapped = mapper.Map<Person, Person>(person);
// mapped.Address is null

Advanced Features

Explicit Property Mapping

Configure custom mappings without modules:

using TurboMapper.Impl;

var mapper = new Mapper();
var mappings = new List<PropertyMapping>
{
    new PropertyMapping
    {
        SourcePropertyPath = "FirstName",
        TargetPropertyPath = "FullName"
    },
    new PropertyMapping
    {
        SourcePropertyPath = "Age",
        TargetPropertyPath = "Years"
    }
};

mapper.CreateMap<UserSource, UserTarget>(mappings);

var source = new UserSource { FirstName = "John", Age = 30 };
var target = mapper.Map<UserSource, UserTarget>(source);
// target.FullName = "John"
// target.Years = 30

Nested Property Flattening

Map nested properties to flat structure:

public class Source
{
    public string Name { get; set; }
    public Address Address { get; set; }
}

public class FlatTarget
{
    public string Name { get; set; }
    public string Street { get; set; }
    public string City { get; set; }
}

var mappings = new List<PropertyMapping>
{
    new PropertyMapping
    {
        SourcePropertyPath = "Address.Street",
        TargetPropertyPath = "Street"
    },
    new PropertyMapping
    {
        SourcePropertyPath = "Address.City",
        TargetPropertyPath = "City"
    }
};

mapper.CreateMap<Source, FlatTarget>(mappings);

Nested Property Mapping

Map flat structure to nested objects:

public class TargetWithNested
{
    public string Name { get; set; }
    public AddressConfig Address { get; set; }
}

public class AddressConfig
{
    public string StreetName { get; set; }
    public string Location { get; set; }
}

var mappings = new List<PropertyMapping>
{
    new PropertyMapping
    {
        SourcePropertyPath = "Address.Street",
        TargetPropertyPath = "Address.StreetName"
    },
    new PropertyMapping
    {
        SourcePropertyPath = "Address.City",
        TargetPropertyPath = "Address.Location"
    }
};

mapper.CreateMap<Source, TargetWithNested>(mappings);

Disabling Default Mapping

Control which properties get mapped:

var mappings = new List<PropertyMapping>
{
    new PropertyMapping
    {
        SourcePropertyPath = "FirstName",
        TargetPropertyPath = "Name"
    }
};

// Only map explicitly defined properties
mapper.CreateMap<Source, Target>(mappings, enableDefaultMapping: false);

Release 1.2.0: Collection Mapping Support

Map collections of objects using the updated Map method that now returns IEnumerable:

// Define your models
public class User
{
    public string Name { get; set; }
    public int Age { get; set; }
}

public class UserDto
{
    public string Name { get; set; }
    public int Age { get; set; }
}

// Create mapper instance
var mapper = serviceProvider.GetService<IMapper>();

// Map collections - the Map method now returns IEnumerable<TDestination>
var users = new List<User>
{
    new User { Name = "Alice", Age = 25 },
    new User { Name = "Bob", Age = 32 }
};

// Map to IEnumerable<T>
IEnumerable<UserDto> userDtos = mapper.Map<User, UserDto>(users);

// Convert to List if needed
List<UserDto> userList = userDtos.ToList();

// Works with any IEnumerable
IEnumerable<User> source = users;
IEnumerable<UserDto> result = mapper.Map<User, UserDto>(source);

// Convert to array if needed
UserDto[] userArray = result.ToArray();

Release 1.2.0: Ignored Properties

Skip specific properties during mapping:

// Using mapping modules
internal class UserModule : MappingModule<UserEntity, UserDto>
{
    public UserModule() : base(true) { }

    public override Action<IMappingExpression<UserEntity, UserDto>> CreateMappings()
    {
        return config =>
        {
            // Ignore sensitive properties
            config.ForMember(dest => dest.Name, src => src.Name)
                  .ForMember(dest => dest.Email, src => src.Email)
                  .Ignore(dest => dest.Id); // Skip ID mapping
        };
    }
}

// Using IMappingExpression directly
var expression = new MappingExpression<UserEntity, UserDto>();
expression.ForMember(dest => dest.Name, src => src.Name)
         .Ignore(dest => dest.Password) // Skip password mapping
         .Ignore(dest => dest.CreatedAt); // Skip timestamp

Release 1.2.0: Custom Type Converters

// Define a mapping module that registers converters
internal class OrderMappingModule : MappingModule<OrderEntity, OrderDto>
{
    public OrderMappingModule() : base(true)
    {
        // Register converters within the module
        RegisterConverter<string, Guid>(str => Guid.Parse(str));
        RegisterConverter<string, DateTime>(str => DateTime.Parse(str));
        RegisterConverter<DateTime, string>(dt => dt.ToString("yyyy-MM-dd"));
    }

    public override Action<IMappingExpression<OrderEntity, OrderDto>> CreateMappings()
    {
        return config =>
        {
            // Define any custom property mappings if needed
            // Otherwise, default mapping will apply with custom converters available
        };
    }
}

// The custom converters will be registered when the module is loaded
// Use the custom converters in mapping
public class OrderEntity
{
    public string OrderId { get; set; }
    public string OrderDate { get; set; }
}

public class OrderDto 
{
    public Guid OrderId { get; set; }
    public DateTime OrderDate { get; set; }
}

var entity = new OrderEntity 
{ 
    OrderId = "12345678-1234-1234-1234-123456789012", 
    OrderDate = "2024-01-15" 
};

var dto = mapper.Map<OrderEntity, OrderDto>(entity);
// Custom converters automatically convert string to Guid and string to DateTime

Release 1.2.0: Conditional Mapping

Map properties based on conditions:

// Using mapping modules
internal class ConditionalUserModule : MappingModule<UserEntity, UserDto>
{
    public ConditionalUserModule() : base(true) { }

    public override Action<IMappingExpression<UserEntity, UserDto>> CreateMappings()
    {
        return config =>
        {
            // Only map premium features if user is premium
            config.ForMember(dest => dest.Name, src => src.Name)
                  .When(dest => dest.PremiumFeatures, src => src.IsPremium)
                  .When(dest => dest.AccountLevel, src => src.Age > 18);
        };
    }
}

// Using IMappingExpression directly
var expression = new MappingExpression<UserEntity, UserDto>();
expression.ForMember(dest => dest.Name, src => src.Name)
         .When(dest => dest.Status, src => src.Age >= 18) // Map status only if adult
         .When(dest => dest.PremiumFeatures, src => src.IsPremium); // Map premium features conditionally

// Example models for conditional mapping
public class UserEntity
{
    public string Name { get; set; }
    public int Age { get; set; }
    public bool IsPremium { get; set; }
}

public class UserDto
{
    public string Name { get; set; }
    public string Status { get; set; }  // Only mapped if Age >= 18
    public List<string> PremiumFeatures { get; set; } = new List<string>(); // Only mapped if IsPremium
}

Release 1.2.0: Mapping Transformations

Transform values during mapping:

// Using mapping modules
internal class TransformModule : MappingModule<UserEntity, UserDto>
{
    public TransformModule() : base(true) { }

    public override Action<IMappingExpression<UserEntity, UserDto>> CreateMappings()
    {
        return config =>
        {
            config.ForMember(dest => dest.Name, src => src.Name)
                  .MapWith<UserEntity, string>(dest => dest.Name, src => $"Mr./Ms. {src.Name}")
                  .MapWith<UserEntity, string>(dest => dest.Email, src => src.Email.ToLower())
                  .MapWith<UserEntity, int>(dest => dest.Age, src => src.Age + 1); // Add 1 to age
        };
    }
}

// Using IMappingExpression directly
var expression = new MappingExpression<UserEntity, UserDto>();
expression.ForMember(dest => dest.Name, src => src.Name)
         .MapWith<UserEntity, string>(dest => dest.Greeting, src => $"Hello, {src.Name}")
         .MapWith<UserEntity, string>(dest => dest.Slug, src => src.Name.Replace(" ", "-").ToLower());

// Example: Transform phone numbers to masked format
public class ContactEntity
{
    public string Name { get; set; }
    public string Phone { get; set; }  // e.g., "123-456-7890"
}

public class ContactDto
{
    public string Name { get; set; }
    public string Phone { get; set; }  // e.g., "###-###-7890"
}

// Module to mask phone numbers
internal class ContactModule : MappingModule<ContactEntity, ContactDto>
{
    public override Action<IMappingExpression<ContactEntity, ContactDto>> CreateMappings()
    {
        return config =>
        {
            config.ForMember(dest => dest.Name, src => src.Name)
                  .MapWith<ContactEntity, string>(dest => dest.Phone, src => 
                      string.IsNullOrEmpty(src.Phone) ? src.Phone : 
                      $"###-###-{src.Phone.Substring(src.Phone.Length - 4)}");
        };
    }
}

Release 1.2.0: Configuration Validation

Validate mapping configurations at runtime:

// Get mapper instance
var mapper = serviceProvider.GetService<IMapper>();

// Create a mapping configuration
mapper.CreateMap<UserEntity, UserDto>(new List<PropertyMapping>
{
    new PropertyMapping
    {
        SourcePropertyPath = "Name",
        TargetPropertyPath = "FullName"
    }
});

// Validate the configuration before using it
var validationResult = mapper.ValidateMapping<UserEntity, UserDto>();
if (!validationResult.IsValid)
{
    foreach (var error in validationResult.Errors)
    {
        Console.WriteLine($"Mapping error: {error}");
    }
}

// Example: Validating in application startup
public void ConfigureServices(IServiceCollection services)
{
    services.AddTurboMapper();

    var mapper = services.BuildServiceProvider().GetService<IMapper>();

    // Validate critical mappings
    var validationResult = mapper.ValidateMapping<OrderEntity, OrderDto>();
    if (!validationResult.IsValid)
    {
        throw new InvalidOperationException($"Invalid order mapping: {string.Join(", ", validationResult.Errors)}");
    }
}

Release 1.2.0: Improved Nullable Type Handling

TurboMapper now handles nullable to non-nullable type conversions seamlessly:

public class Source
{
    public string Name { get; set; }
    public int? Age { get; set; }
    public DateTime? BirthDate { get; set; }
}

public class Target
{
    public string Name { get; set; }
    public int Age { get; set; }  // Non-nullable
    public DateTime BirthDate { get; set; }  // Non-nullable
}

var source = new Source 
{ 
    Name = "John", 
    Age = 30,  // Has value
    BirthDate = null  // Is null
};

var target = mapper.Map<Source, Target>(source);
// target.Name = "John"
// target.Age = 30
// target.BirthDate = DateTime.MinValue (default value for DateTime)

Mapping Modules

Mapping modules provide a clean, reusable way to organize mapping configurations.

Creating a Mapping Module

using TurboMapper;
using System;

internal class UserMappingModule : MappingModule<UserSource, UserTarget>
{
    public UserMappingModule() : base(enableDefaultMapping: true)
    {
    }

    public override Action<IMappingExpression<UserSource, UserTarget>> CreateMappings()
    {
        return config =>
        {
            config.ForMember(dest => dest.Name, src => src.FirstName)
                  .ForMember(dest => dest.Years, src => src.Age);
        };
    }
}

Module with Nested Mappings

internal class AddressMappingModule : MappingModule<AddressSource, AddressTarget>
{
    public AddressMappingModule() : base(enableDefaultMapping: true)
    {
    }

    public override Action<IMappingExpression<AddressSource, AddressTarget>> CreateMappings()
    {
        return config =>
        {
            config.ForMember(dest => dest.Address.Street, src => src.Address.StreetName)
                  .ForMember(dest => dest.Address.Location, src => src.Address.City);
        };
    }
}

Module Without Default Mapping

When you want complete control over what gets mapped:

internal class StrictMappingModule : MappingModule<Source, Target>
{
    public StrictMappingModule() : base(enableDefaultMapping: false)
    {
    }

    public override Action<IMappingExpression<Source, Target>> CreateMappings()
    {
        return config =>
        {
            config.ForMember(dest => dest.Name, src => src.FirstName)
                  .ForMember(dest => dest.Years, src => src.Age);
            // Only these properties will be mapped
        };
    }
}

Module Auto-Discovery

Mapping modules are automatically discovered and registered when using dependency injection:

services.AddTurboMapper();
// All mapping modules in loaded assemblies are automatically registered

Dependency Injection

Registration

using Microsoft.Extensions.DependencyInjection;
using TurboMapper;

var services = new ServiceCollection();
services.AddTurboMapper();
var serviceProvider = services.BuildServiceProvider();

Singleton Behavior

The mapper is registered as a singleton and thread-safe:

var mapper1 = serviceProvider.GetService<IMapper>();
var mapper2 = serviceProvider.GetService<IMapper>();
// mapper1 and mapper2 are the same instance

Using in ASP.NET Core

// Startup.cs or Program.cs
public void ConfigureServices(IServiceCollection services)
{
    services.AddTurboMapper();
    services.AddControllers();
}

// In a controller
public class UserController : ControllerBase
{
    private readonly IMapper _mapper;

    public UserController(IMapper mapper)
    {
        _mapper = mapper;
    }

    [HttpPost]
    public IActionResult CreateUser(UserDTO dto)
    {
        var entity = _mapper.Map<UserDTO, UserEntity>(dto);
        // Save entity
        return Ok();
    }
}

Scoped Services

The mapper works correctly across service scopes:

using (var scope1 = serviceProvider.CreateScope())
using (var scope2 = serviceProvider.CreateScope())
{
    var mapper1 = scope1.ServiceProvider.GetService<IMapper>();
    var mapper2 = scope2.ServiceProvider.GetService<IMapper>();
    // Both resolve to the same singleton instance
}

Performance Considerations

Thread Safety

TurboMapper is fully thread-safe for concurrent mapping operations:

var sources = GetLargeDataSet();
var results = new ConcurrentBag<Target>();

Parallel.ForEach(sources, source =>
{
    var target = mapper.Map<Source, Target>(source);
    results.Add(target);
});

Batch Processing

For large datasets, process in batches:

var sources = GetLargeDataSet(); // 10,000 items
var targets = new List<Target>(sources.Count);

foreach (var source in sources)
{
    targets.Add(mapper.Map<Source, Target>(source));
}

Reusing Mapper Instance

Always reuse the same mapper instance (singleton pattern):

// Good - Reuse singleton
var mapper = serviceProvider.GetService<IMapper>();
for (int i = 0; i < 1000; i++)
{
    var result = mapper.Map<Source, Target>(sources[i]);
}

// Bad - Creating new instances
for (int i = 0; i < 1000; i++)
{
    var mapper = new Mapper(); // Don't do this
    var result = mapper.Map<Source, Target>(sources[i]);
}

Configuration Caching

Mapping configurations are cached internally, so repeated mappings are efficient:

mapper.CreateMap<Source, Target>(mappings);

// Subsequent calls use cached configuration
for (int i = 0; i < 10000; i++)
{
    mapper.Map<Source, Target>(sources[i]); // Fast
}

API Reference

IMapper Interface

public interface IMapper
{
    TTarget Map<TSource, TTarget>(TSource source);
    IEnumerable<TDestination> Map<TSource, TDestination>(IEnumerable<TSource> source);
}

Map Method (Single Object)

Parameters:
- TSource source - The source object to map from
Returns: TTarget - The mapped target object
Returns null if: Source is null

Map Method (Collection)

Parameters:
- IEnumerable<TSource> source - The collection of source objects to map from
Returns: IEnumerable<TDestination> - The mapped collection of target objects
Returns null if: Source is null

IMappingExpression Interface

public interface IMappingExpression<TSource, TTarget>
{
    IMappingExpression<TSource, TTarget> ForMember<TValue>(
        Expression<Func<TTarget, TValue>> targetMember, 
        Expression<Func<TSource, TValue>> sourceMember);
}

ForMember Method

Configures custom mapping between properties
Supports nested property paths
Returns the expression for fluent chaining

MappingModule Abstract Class

public abstract class MappingModule<TSource, TTarget>
{
    protected MappingModule(bool enableDefaultMapping = true);

    public abstract Action<IMappingExpression<TSource, TTarget>> CreateMappings();
}

Constructor Parameters

enableDefaultMapping - When true, unmapped properties use name-based mapping

ServiceCollectionExtensions

public static class ServiceCollectionExtensions
{
    public static IServiceCollection AddTurboMapper(
        this IServiceCollection services);
}

AddTurboMapper Method

Registers IMapper as singleton
Auto-discovers and registers mapping modules
Returns IServiceCollection for chaining

PropertyMapping Class

internal class PropertyMapping
{
    public string SourcePropertyPath { get; set; }
    public string TargetPropertyPath { get; set; }
}

Best Practices

1. Use Dependency Injection

Always register TurboMapper with DI for automatic module discovery:

services.AddTurboMapper();

2. Organize with Mapping Modules

Group related mappings in modules:

// UserMappings.cs
internal class UserDTOMappingModule : MappingModule<UserDTO, UserEntity>
{
    // ...
}

internal class UserEntityMappingModule : MappingModule<UserEntity, UserDTO>
{
    // ...
}

3. Enable Default Mapping

Unless you need strict control, enable default mapping:

public MyMappingModule() : base(enableDefaultMapping: true)
{
    // Only specify differences from default
}

4. Handle Null Values

Always check for null in business logic when needed:

var entity = mapper.Map<DTO, Entity>(dto);
if (entity?.Address == null)
{
    entity.Address = new Address();
}

5. Use Type-Safe Expressions

Leverage lambda expressions for compile-time safety:

config.ForMember(dest => dest.FullName, src => src.FirstName)
      .ForMember(dest => dest.Years, src => src.Age);

6. Test Your Mappings

Always unit test custom mappings:

[Test]
public void Map_UserDTOToEntity_MapsCorrectly()
{
    var dto = new UserDTO { FirstName = "John", Age = 30 };
    var entity = mapper.Map<UserDTO, UserEntity>(dto);

    Assert.AreEqual("John", entity.Name);
    Assert.AreEqual(30, entity.Years);
}

7. Avoid Deep Nesting

Keep object hierarchies reasonable for performance:

// Good - 2-3 levels
User.Address.City

// Potentially slow - 5+ levels
User.Profile.Settings.Preferences.Display.Theme

8. Reuse Mapping Configurations

Don't create mappings repeatedly:

// Good - Create once
mapper.CreateMap<Source, Target>(mappings);

// Bad - Inside loop
for (int i = 0; i < 1000; i++)
{
    mapper.CreateMap<Source, Target>(mappings); // Don't do this
}

Troubleshooting

Common Issues

Properties Not Mapping

Problem: Properties with matching names aren't mapping.

Solutions:

Ensure properties have public getters and setters
Check property names match exactly (case-sensitive)
Verify target property has a setter (set accessor)

// Won't map - no setter
public class Target
{
    public string Name { get; } // Read-only
}

// Will map - has setter
public class Target
{
    public string Name { get; set; }
}

Nested Objects Are Null

Problem: Nested objects aren't being created.

Solutions:

Check if source nested object is null
Ensure nested class has parameterless constructor
Verify property types match or are compatible

public class Address
{
    // Must have parameterless constructor
    public Address() { }

    public string Street { get; set; }
}

Type Conversion Fails

Problem: Automatic type conversion not working.

Solutions:

Check if types are compatible for conversion
Implement explicit mapping for complex conversions
Ensure enum names match string values (case-insensitive)

// Works - Compatible types
int age = 25;
string ageStr = age.ToString(); // TurboMapper handles this

// May fail - Incompatible types
object obj = new MyClass();
int value = (int)obj; // Need explicit handling

Mapping Module Not Found

Problem: Mapping module not being discovered.

Solutions:

Ensure module class is not abstract
Make module class internal or public (not private)
Verify assembly is loaded in AppDomain
Check that AddTurboMapper() is called

// Correct - Will be discovered
internal class MyMappingModule : MappingModule<Source, Target>
{
    // ...
}

// Wrong - Won't be discovered
private class MyMappingModule : MappingModule<Source, Target>
{
    // ...
}

Performance Issues

Problem: Mapping is slow with large datasets.

Solutions:

Reuse mapper instance (singleton)
Configure mappings once, not repeatedly
Use Parallel processing for large collections
Profile to identify specific bottlenecks

// Efficient approach
var mapper = serviceProvider.GetService<IMapper>();
var results = sources
    .AsParallel()
    .Select(s => mapper.Map<Source, Target>(s))
    .ToList();

Debug Tips

Enable Detailed Exceptions

Wrap mapping calls to get better error messages:

try
{
    var result = mapper.Map<Source, Target>(source);
}
catch (Exception ex)
{
    Console.WriteLine($"Mapping failed: {ex.Message}");
    Console.WriteLine($"Stack trace: {ex.StackTrace}");
    throw;
}

Verify Configuration

Test that your configuration is registered:

var mapper = serviceProvider.GetService<IMapper>();
var source = new Source { Name = "Test" };

try
{
    var target = mapper.Map<Source, Target>(source);
    Console.WriteLine("Mapping successful");
}
catch
{
    Console.WriteLine("Mapping configuration missing");
}

Check Assembly Loading

Verify assemblies containing modules are loaded:

var assemblies = AppDomain.CurrentDomain.GetAssemblies();
foreach (var assembly in assemblies)
{
    Console.WriteLine($"Loaded: {assembly.FullName}");
}

Examples

Example 1: Simple DTO Mapping

public class CreateUserRequest
{
    public string FirstName { get; set; }
    public string LastName { get; set; }
    public string Email { get; set; }
}

public class User
{
    public string FirstName { get; set; }
    public string LastName { get; set; }
    public string Email { get; set; }
    public DateTime CreatedAt { get; set; }
}

// Mapping
var request = new CreateUserRequest
{
    FirstName = "John",
    LastName = "Doe",
    Email = "john@example.com"
};

var user = mapper.Map<CreateUserRequest, User>(request);
user.CreatedAt = DateTime.UtcNow;

Example 2: Complex Business Object

public class OrderDTO
{
    public string OrderNumber { get; set; }
    public CustomerDTO Customer { get; set; }
    public List<OrderItemDTO> Items { get; set; }
}

public class OrderEntity
{
    public string OrderNumber { get; set; }
    public CustomerEntity Customer { get; set; }
    public List<OrderItemEntity> Items { get; set; }
}

// Map with nested objects
var dto = GetOrderDTO();
var entity = mapper.Map<OrderDTO, OrderEntity>(dto);

// Map collection
entity.Items = dto.Items
    .Select(item => mapper.Map<OrderItemDTO, OrderItemEntity>(item))
    .ToList();

Example 3: Flattening Hierarchy

internal class OrderFlatteningModule : MappingModule<Order, OrderFlat>
{
    public OrderFlatteningModule() : base(true)
    {
    }

    public override Action<IMappingExpression<Order, OrderFlat>> CreateMappings()
    {
        return config =>
        {
            config.ForMember(dest => dest.CustomerName, src => src.Customer.Name)
                  .ForMember(dest => dest.CustomerEmail, src => src.Customer.Email)
                  .ForMember(dest => dest.ShippingStreet, src => src.ShippingAddress.Street)
                  .ForMember(dest => dest.ShippingCity, src => src.ShippingAddress.City);
        };
    }
}

Version History

v1.2.0 - Enhanced Core & Mapping Features (Latest)

Performance Improvements: Significant performance enhancements (2x+) through compiled expression trees and metadata caching
Collection Mapping Support: Added Map method overload to IMapper for collection mapping operations that returns IEnumerable
Ignored Properties Option: Added Ignore method to IMappingExpression to skip properties during mapping
Custom Type Converters Registration: Added RegisterConverter method to MappingModule for custom type conversion functions that can be registered with type mappings
Improved Nullable Type Handling: Enhanced ConvertValue method to handle nullable types properly
Conditional Mapping: Added When method to IMappingExpression for conditional property mapping
Mapping Transformations: Added MapWith method for transformation functions during mapping
Comprehensive Type Conversions: Enhanced ConvertValue with DateTime, TimeSpan, and other common type conversions
Configuration Validation: Added ValidateMapping method to IMapper that returns ValidationResult for early validation
Improved Error Messages: Better debugging information for conversion failures
Refactored Code: Consolidated duplicate GetMemberPath methods into shared internal implementation
Optimized Object Creation: Replaced Activator.CreateInstance with factory delegates for faster instantiation
Configuration Caching: Added caching for mapping configurations and lookups for faster performance
Backward Compatibility: Maintains full backward compatibility with v1.0.0

v1.0.0

Initial release
Default name-based mapping
Custom property mapping
Nested object support
Type conversion
Mapping modules
Dependency injection integration
Multi-framework support

License

TurboMapper is licensed under the terms specified in the LICENSE file.

Support

For issues, questions, or contributions:

GitHub: https://github.com/CodeShayk/TurboMapper
Wiki: https://github.com/CodeShayk/TurboMapper/wiki

Contributing

Contributions are welcome! Please submit pull requests or open issues on GitHub.

Parsley.Net

CØDE N!NJΔ — Thu, 13 Nov 2025 19:38:19 +0000

Parsley.Net is a .net utility to help parse Fixed Width or Delimiter Separated file using strongly typed objects.

It is a simple and easy to use library that can be used to parse delimeter separated files in .net applications. It is designed to be lightweight and fast, making it ideal for use in high-performance applications.

Introduction
Understanding Fixed Width and Delimiter Separated Files
Why Choose Parsley.Net?
Getting Started
Core Components
Advanced Usage
Performance Considerations
Error Handling
Testing Guide
API Reference
Examples
Troubleshooting
Contributing

Introduction

What is Parsley.Net?

Parsley.Net is a lightweight, high-performance .NET library designed to parse Fixed Width and Delimiter Separated text files into strongly-typed C# objects. It provides a simple, intuitive API that allows developers to transform structured text data into usable .NET objects with minimal configuration and maximum flexibility.

Key Features

✅ Strongly-typed parsing - Convert text data directly into C# objects
✅ Multiple input formats - Support for files, streams, byte arrays, and string arrays
✅ Async/Sync operations - Full async support for high-performance applications
✅ Custom type converters - Extensible parsing for complex data types
✅ Error handling - Comprehensive error reporting per line and field
✅ Dependency injection support - Easy integration with modern .NET applications
✅ Multi-framework support - Compatible with .NET 9.0, .NET Standard 2.0, .NET Framework 4.6.2
✅ Zero dependencies - Minimal footprint with only essential Microsoft extensions

Supported Frameworks

.NET 9.0
.NET Standard 2.0
.NET Standard 2.1
.NET Framework 4.6.2

Understanding Fixed Width and Delimiter Separated Files

What are Delimiter Separated Files?

Delimiter separated files are text files where data fields are separated by a specific character (delimiter). The most common examples are:

CSV (Comma Separated Values) - Fields separated by commas
TSV (Tab Separated Values) - Fields separated by tabs
PSV (Pipe Separated Values) - Fields separated by pipes (|)
Custom delimiters - Any character can serve as a delimiter

Example: Pipe Separated File

|Mr|Jack Marias|Male|London, UK|Active|||
|Dr|Bony Stringer|Male|New Jersey, US|Active||Paid|
|Mrs|Mary Ward|Female||Active|||
|Mr|Robert Webb|||Active|||

What are Fixed Width Files?

Fixed width files allocate a specific number of characters for each field, regardless of the actual data length. Fields are padded with spaces to maintain consistent positioning.

Example: Fixed Width File

Mr    Jack Marias        Male  London, UK       Active
Dr    Bony Stringer      Male  New Jersey, US   Active
Mrs   Mary Ward          Female                 Active
Mr    Robert Webb                               Active

Common Use Cases

Data Migration - Moving data between different systems
ETL Processes - Extract, Transform, Load operations
Report Processing - Parsing structured reports from legacy systems
Batch Processing - Processing large volumes of structured data
Integration - Connecting with systems that export structured text files

Why Choose Parsley.Net?

Advantages Over Manual Parsing

Manual String Parsing	Parsley.Net
Error-prone string manipulation	Type-safe object mapping
No built-in error handling	Comprehensive error reporting
Manual type conversion	Automatic type conversion
Difficult to maintain	Clean, declarative syntax
No async support	Full async/await support
Performance concerns with large files	Optimized parallel processing

Performance Benefits

Parallel Processing - Utilizes Parallel.ForEach for multi-threaded parsing
Memory Efficient - Streaming support for large files
Optimized Reflection - Cached property information for repeated parsing
Async Operations - Non-blocking I/O for better application responsiveness

Developer Experience

// Before: Manual parsing (error-prone)
var parts = line.Split('|');
var employee = new Employee 
{
    Title = parts[0],
    Name = parts[1],
    // ... manual parsing, type conversion, error handling
};

// After: Parsley.Net (clean and safe)
var employees = parser.Parse<Employee>(filePath);

Getting Started

Installation

Install Parsley.Net via NuGet Package Manager:

Package Manager Console

Install-Package Parsley.Net

.NET CLI

dotnet add package Parsley.Net

PackageReference (in .csproj)

<PackageReference Include="Parsley.Net" Version="1.1.5" />

Quick Start Example

Create a data model:

using parsley;

public class Employee : IFileLine
{
    [Column(0)]
    public string Title { get; set; }

    [Column(1)]  
    public string Name { get; set; }

    [Column(2)]
    public string Gender { get; set; }

    [Column(3, "London, UK")] // Default value
    public string Location { get; set; }

    // IFileLine implementation
    public int Index { get; set; }
    public IList<string> Errors { get; set; }
}

Parse your data:

using parsley;

// Create parser with pipe delimiter
var parser = new Parser('|');

// Parse from file
var employees = parser.Parse<Employee>("employees.txt");

// Parse from string array
var lines = new[] { "|Mr|John Doe|Male|New York|" };
var result = parser.Parse<Employee>(lines);

Core Components

IParser Interface

The IParser interface is the main entry point for all parsing operations:

public interface IParser
{
    // Synchronous methods
    T[] Parse<T>(string filepath) where T : IFileLine, new();
    T[] Parse<T>(string[] lines) where T : IFileLine, new();
    T[] Parse<T>(byte[] bytes, Encoding encoding = null) where T : IFileLine, new();
    T[] Parse<T>(Stream stream, Encoding encoding = null) where T : IFileLine, new();

    // Asynchronous methods
    Task<T[]> ParseAsync<T>(string filepath) where T : IFileLine, new();
    Task<T[]> ParseAsync<T>(string[] lines) where T : IFileLine, new();
    Task<T[]> ParseAsync<T>(byte[] bytes, Encoding encoding = null) where T : IFileLine, new();
    Task<T[]> ParseAsync<T>(Stream stream, Encoding encoding = null) where T : IFileLine, new();

    // TryParse methods with explicit Result<T[]> return type
    Result<T[]> TryParse<T>(string filepath) where T : IFileLine, new();
    Result<T[]> TryParse<T>(string filepath, ParseOptions options) where T : IFileLine, new();
    Result<T[]> TryParse<T>(string[] lines) where T : IFileLine, new();
    Result<T[]> TryParse<T>(string[] lines, ParseOptions options) where T : IFileLine, new();
    Result<T[]> TryParse<T>(byte[] bytes, Encoding encoding = null) where T : IFileLine, new();
    Result<T[]> TryParse<T>(byte[] bytes, Encoding encoding, ParseOptions options) where T : IFileLine, new();
    Result<T[]> TryParse<T>(Stream stream, Encoding encoding = null) where T : IFileLine, new();
    Result<T[]> TryParse<T>(Stream stream, Encoding encoding, ParseOptions options) where T : IFileLine, new();

    // Asynchronous TryParse methods with explicit Result<T[]> return type
    Task<Result<T[]>> TryParseAsync<T>(string filepath) where T : IFileLine, new();
    Task<Result<T[]>> TryParseAsync<T>(string filepath, ParseOptions options) where T : IFileLine, new();
    Task<Result<T[]>> TryParseAsync<T>(string[] lines) where T : IFileLine, new();
    Task<Result<T[]>> TryParseAsync<T>(string[] lines, ParseOptions options) where T : IFileLine, new();
    Task<Result<T[]>> TryParseAsync<T>(byte[] bytes, Encoding encoding = null) where T : IFileLine, new();
    Task<Result<T[]>> TryParseAsync<T>(byte[] bytes, Encoding encoding, ParseOptions options) where T : IFileLine, new();
    Task<Result<T[]>> TryParseAsync<T>(Stream stream, Encoding encoding = null) where T : IFileLine, new();
    Task<Result<T[]>> TryParseAsync<T>(Stream stream, Encoding encoding, ParseOptions options) where T : IFileLine, new();
}

Parser Class

The concrete implementation of IParser:

// Default constructor (comma delimiter)
var parser = new Parser();

// Custom delimiter
var parser = new Parser('|');
var parser = new Parser('\t'); // Tab separated
var parser = new Parser(';');  // Semicolon separated

IFileLine Interface

All data models must implement IFileLine:

public interface IFileLine
{
    int Index { get; set; }           // Line number in source
    IList<string> Errors { get; set; } // Parse errors for this line
}

ColumnAttribute

Defines the mapping between file columns and object properties:

public class ColumnAttribute : Attribute
{
    public ColumnAttribute(int index, object defaultvalue = null)
    {
        Index = index;           // Zero-based column index
        DefaultValue = defaultvalue; // Default value if field is empty
    }
}

Usage Examples:

[Column(0)]              // Required field at index 0
[Column(1, "Unknown")]   // Field at index 1, default to "Unknown"
[Column(2, 0)]          // Numeric field with default value 0

ParseOptions Class

The ParseOptions class provides flexible configuration for parsing operations:

public class ParseOptions
{
    public char Delimiter { get; set; } = ',';              // Delimiter character (default: comma)
    public bool SkipHeaderLine { get; set; } = false;       // Skip first line as header
    public bool TrimFieldValues { get; set; } = true;       // Trim whitespace from values
    public bool IncludeEmptyLines { get; set; } = true;     // Include empty lines in result
    public int MaxErrors { get; set; } = -1;                // Max errors before stopping (-1 = unlimited)
    public int BufferSize { get; set; } = 1024;             // Buffer size for streaming operations

    // Constructors
    public ParseOptions();                                  // Default options
    public ParseOptions(char delimiter);                    // Custom delimiter
}

ParseOptions Usage Examples:

// Using default options
var options = new ParseOptions();

// Using specific delimiter
var pipeOptions = new ParseOptions('|');

// Custom configuration
var customOptions = new ParseOptions()
{
    Delimiter = ';',
    SkipHeaderLine = true,
    TrimFieldValues = true,
    IncludeEmptyLines = false
};

Result Class

The Result<T> class provides explicit success/failure semantics for parsing operations:

public class Result<T>
{
    public bool IsSuccess { get; }            // True if operation succeeded
    public bool IsFailure { get; }            // True if operation failed (opposite of IsSuccess)
    public T Value { get; }                   // The result value (only valid when IsSuccess is true)
    public IList<string> Errors { get; }     // List of error messages (only populated when IsFailure is true)

    // Static factory methods
    public static Result<T> Success(T value);           // Create a successful result
    public static Result<T> Failure(string error);      // Create a failure result with single error
    public static Result<T> Failure(IList<string> errors); // Create a failure result with multiple errors
    public static Result<T> Failure(string error, IList<string> errors); // Create failure with error and additional errors
}

Result Usage Examples:

var parser = new Parser('|');

// Using TryParse for explicit error handling
var result = parser.TryParse<Employee>("employees.txt");

if (result.IsSuccess)
{
    var employees = result.Value;
    Console.WriteLine($"Successfully parsed {employees.Length} employees");
}
else
{
    Console.WriteLine($"Parsing failed with errors:");
    foreach (var error in result.Errors)
    {
        Console.WriteLine($"  - {error}");
    }
}

// Using TryParseAsync for asynchronous operations
var asyncResult = await parser.TryParseAsync<Employee>(fileStream, encoding);

if (asyncResult.IsSuccess)
{
    var employees = asyncResult.Value;
    // Process employees
}
else
{
    // Handle errors
    LogErrors(asyncResult.Errors);
}

Advanced Usage

Custom Type Converters

Parsley.Net supports complex data types through custom TypeConverter implementations.

Method 1: Traditional TypeConverter

public class NameConverter : TypeConverter
{
    public override bool CanConvertFrom(ITypeDescriptorContext context, Type sourceType)
    {
        return sourceType == typeof(string) || base.CanConvertFrom(context, sourceType);
    }

    public override object ConvertFrom(ITypeDescriptorContext context, CultureInfo culture, object value)
    {
        if (value is string stringValue && !string.IsNullOrEmpty(stringValue))
        {
            return NameType.Parse(stringValue);
        }
        return base.ConvertFrom(context, culture, value);
    }
}

[TypeConverter(typeof(NameConverter))]
public class NameType
{
    public string FirstName { get; set; }
    public string Surname { get; set; }

    public static NameType Parse(string input)
    {
        var parts = input.Split(' ', StringSplitOptions.RemoveEmptyEntries);
        return new NameType 
        { 
            FirstName = parts.FirstOrDefault(), 
            Surname = parts.Skip(1).FirstOrDefault() 
        };
    }
}

Using Custom Types in FileLine Models

Here's how to use the NameType in your FileLine model:

public class Employee : IFileLine
{
    [Column(0)]
    public string Title { get; set; }

    [Column(1)]  // This column contains full names like "John Doe"
    public NameType Name { get; set; }

    [Column(2)]
    public string Department { get; set; }

    [Column(3)]
    public decimal Salary { get; set; }

    // IFileLine implementation
    public int Index { get; set; }
    public IList<string> Errors { get; set; }
}

// Example usage
var parser = new Parser(',');
var employees = parser.Parse<Employee>("employees.csv");

// Assuming the CSV contains:
// Mr,John Smith,IT,50000
// Dr,Jane Doe,HR,55000

// The Name property will automatically be converted to NameType
foreach (var employee in employees)
{
    Console.WriteLine($"Title: {employee.Title}");
    Console.WriteLine($"First Name: {employee.Name.FirstName}");
    Console.WriteLine($"Last Name: {employee.Name.Surname}");
    Console.WriteLine($"Department: {employee.Department}");
    Console.WriteLine($"Salary: {employee.Salary}");
}

Method 2: Built-in CustomConverter

[TypeConverter(typeof(CustomConverter<CodeType>))]
public class CodeType : ICustomType
{
    public string Batch { get; set; }
    public int SerialNo { get; set; }

    public ICustomType Parse(string input)
    {
        var parts = input.Split('-');
        if (parts.Length == 2 && int.TryParse(parts[1], out int serial))
        {
            return new CodeType { Batch = parts[0], SerialNo = serial };
        }
        throw new FormatException($"Invalid code format: {input}");
    }
}

Dependency Injection

Parsley.Net integrates seamlessly with .NET's dependency injection container:

using Microsoft.Extensions.DependencyInjection;
using parsley;

// Manual registration
services.AddTransient<IParser>(provider => new Parser(','));

// Using extension method
services.UseParsley('|'); // Pipe delimiter
services.UseParsley();    // Default comma delimiter

// Usage in controller/service
public class DataService
{
    private readonly IParser _parser;

    public DataService(IParser parser)
    {
        _parser = parser;
    }

    public async Task<Employee[]> ProcessEmployeeFile(Stream fileStream)
    {
        return await _parser.ParseAsync<Employee>(fileStream);
    }
}

v2.0.0 New Features

Parsley.Net v2.0.0 introduces several major enhancements:

1. Configuration Options with ParseOptions

Configure parsing behavior with the ParseOptions class:

var options = new ParseOptions()
{
    Delimiter = '|',                    // Custom delimiter
    SkipHeaderLine = true,             // Skip first line as header
    TrimFieldValues = true,            // Trim whitespace from values
    IncludeEmptyLines = false,         // Exclude empty lines from result
    MaxErrors = 100                    // Stop after 100 errors (use -1 for unlimited)
};

// Use with Parse methods
var employees = parser.Parse<Employee>(filePath, options);

// Use with ParseAsync methods
var employees = await parser.ParseAsync<Employee>(filePath, options);

2. Result Pattern with TryParse Methods

Handle errors explicitly using the Result pattern:

// Synchronous TryParse methods
var result = parser.TryParse<Employee>("employees.csv");
if (result.IsSuccess)
{
    var employees = result.Value;
    // Process successfully parsed employees
}
else
{
    // Handle errors
    foreach (var error in result.Errors)
    {
        Console.WriteLine($"Error: {error}");
    }
}

// Asynchronous TryParse methods
var asyncResult = await parser.TryParseAsync<Employee>(fileStream);
if (asyncResult.IsSuccess)
{
    var employees = asyncResult.Value;
    // Process employees
}
else
{
    // Handle errors
    var errors = asyncResult.Errors;
}

3. Enhanced Error Reporting

Improved error messages now include line numbers and property names:

// Error messages in v2.0.0+ include line numbers
// "Line 5: Property 'Name' failed to parse with error - Input string was not in a correct format"

4. Improved Performance and Parallel Processing

Enhanced parallel processing implementation
Better thread safety with reduced locking bottlenecks
Improved resource utilization for better performance

Enum Support

Parsley.Net provides robust enum parsing:

public enum Status
{
    Unknown = 0,
    Active = 1,
    Inactive = 2,
    Suspended = 3
}

public class Employee : IFileLine
{
    [Column(0)]
    public string Name { get; set; }

    [Column(1)]
    public Status Status { get; set; } // Supports both string and numeric values

    public int Index { get; set; }
    public IList<string> Errors { get; set; }
}

// File content can be:
// John Doe,Active     <- String representation
// Jane Smith,1        <- Numeric representation

Stream Processing for Large Files

For memory-efficient processing of large files:

public async Task ProcessLargeFile(string filePath)
{
    using var fileStream = File.OpenRead(filePath);

    // Process in chunks or all at once
    var records = await parser.ParseAsync<Employee>(fileStream);

    // Process records in batches
    await ProcessInBatches(records, batchSize: 1000);
}

private async Task ProcessInBatches<T>(T[] records, int batchSize)
{
    for (int i = 0; i < records.Length; i += batchSize)
    {
        var batch = records.Skip(i).Take(batchSize);
        await ProcessBatch(batch);
    }
}

Performance Considerations

Parallel Processing

Parsley.Net automatically uses parallel processing for better performance:

// Internal implementation uses Parallel.ForEach
Parallel.ForEach(inputs, () => new List<T>(),
    (obj, loopstate, localStorage) =>
    {
        var parsed = ParseLine<T>(obj.Line);
        parsed.Index = obj.Index;
        localStorage.Add(parsed);
        return localStorage;
    },
    finalStorage =>
    {
        lock (objLock)
            finalStorage.ForEach(f => list[f.Index] = f);
    });

Performance Tips

Use async methods for I/O bound operations
Process streams instead of loading entire files into memory
Minimize custom converters complexity
Cache parser instances for repeated operations
Use appropriate data types (avoid overly complex objects)

Benchmarking Results

File Size	Records	Sync Time	Async Time	Memory Usage
1MB	10K	45ms	38ms	12MB
10MB	100K	420ms	365ms	45MB
100MB	1M	4.2s	3.8s	180MB

Error Handling

Per-Line Error Tracking

Each parsed object tracks its own errors:

var employees = parser.Parse<Employee>(lines);

foreach (var employee in employees)
{
    if (employee.Errors?.Any() == true)
    {
        Console.WriteLine($"Line {employee.Index} has errors:");
        foreach (var error in employee.Errors)
        {
            Console.WriteLine($"  - {error}");
        }
    }
}

Common Error Types

Invalid Line Format - Incorrect number of columns
Type Conversion Errors - Cannot convert string to target type
Enum Parse Errors - Invalid enum values
Custom Converter Errors - Exceptions from custom converters
Missing Column Attributes - No column mappings found

Error Messages

Parsley.Net provides descriptive error messages:

// Example error messages
"Invalid line format - number of column values do not match"
"Name failed to parse with error - Input string was not in a correct format"
"Status failed to parse - Invalid enum value"
"No column attributes found on Line - Employee"

Best Practices for Error Handling

public async Task<ProcessResult> ProcessFile(string filePath)
{
    try
    {
        var records = await parser.ParseAsync<Employee>(filePath);
        var validRecords = new List<Employee>();
        var errorReport = new List<string>();

        foreach (var record in records)
        {
            if (record.Errors?.Any() == true)
            {
                errorReport.Add($"Line {record.Index}: {string.Join(", ", record.Errors)}");
            }
            else
            {
                validRecords.Add(record);
            }
        }

        return new ProcessResult 
        { 
            ValidRecords = validRecords, 
            Errors = errorReport 
        };
    }
    catch (FileNotFoundException)
    {
        return new ProcessResult { Errors = new[] { "File not found" } };
    }
    catch (UnauthorizedAccessException)
    {
        return new ProcessResult { Errors = new[] { "Access denied" } };
    }
}

Testing Guide

Unit Testing with Parsley.Net

Based on the test suite, here are recommended testing patterns:

Basic Parsing Tests

[Test]
public void Should_Parse_Valid_Data_Correctly()
{
    // Arrange
    var parser = new Parser('|');
    var lines = new[]
    {
        "GB-01|Bob Marley|True|Free",
        "UH-02|John Walsh McKinsey|False|Paid"
    };

    // Act
    var result = parser.Parse<FileLine>(lines);

    // Assert
    Assert.That(result.Length, Is.EqualTo(2));
    Assert.That(result[0].Code.Batch, Is.EqualTo("GB"));
    Assert.That(result[0].Name.FirstName, Is.EqualTo("Bob"));
    Assert.That(result[0].Errors, Is.Empty);
}

Error Handling Tests

[TestCase("invalid_data")]
[TestCase("too|few|columns")]
[TestCase("too|many|columns|here|extra")]
public void Should_Handle_Invalid_Input_Gracefully(string invalidLine)
{
    // Arrange
    var parser = new Parser('|');

    // Act
    var result = parser.Parse<FileLine>(new[] { invalidLine });

    // Assert
    Assert.That(result[0].Errors, Is.Not.Empty);
}

Async Testing

[Test]
public async Task Should_Parse_Async_Successfully()
{
    // Arrange
    var parser = new Parser('|');
    var lines = new[] { "GB-01|Bob Marley|True|Free" };

    // Act
    var result = await parser.ParseAsync<FileLine>(lines);

    // Assert
    Assert.That(result.Length, Is.EqualTo(1));
    Assert.That(result[0].Errors, Is.Empty);
}

Integration Testing

[Test]
public void Should_Parse_Real_File()
{
    // Arrange
    var parser = new Parser();
    var testFile = Path.Combine(TestContext.CurrentContext.TestDirectory, "TestData.csv");

    // Act
    var result = parser.Parse<Employee>(testFile);

    // Assert
    Assert.That(result, Is.Not.Empty);
    Assert.That(result.All(r => r.Errors == null || !r.Errors.Any()), Is.True);
}

Mocking for Dependencies

[Test]
public void Should_Process_File_Through_Service()
{
    // Arrange
    var mockParser = new Mock<IParser>();
    var expectedData = new[] { new Employee { Name = "Test" } };
    mockParser.Setup(p => p.ParseAsync<Employee>(It.IsAny<string>()))
              .ReturnsAsync(expectedData);

    var service = new EmployeeService(mockParser.Object);

    // Act
    var result = await service.ProcessEmployeeFile("test.csv");

    // Assert
    Assert.That(result, Is.EqualTo(expectedData));
}

API Reference

Parser Class Methods

Synchronous Methods

Method	Description	Parameters
`Parse<T>(string filepath)`	Parse file by path	`filepath`: Path to file
`Parse<T>(string[] lines)`	Parse string array	`lines`: Array of delimited strings
`Parse<T>(Stream stream, Encoding encoding)`	Parse stream	`stream`: Data stream, `encoding`: Optional encoding
`Parse<T>(byte[] bytes, Encoding encoding)`	Parse byte array	`bytes`: Byte data, `encoding`: Optional encoding

Asynchronous Methods

Method	Description	Parameters
`ParseAsync<T>(string filepath)`	Parse file asynchronously	`filepath`: Path to file
`ParseAsync<T>(string[] lines)`	Parse string array asynchronously	`lines`: Array of delimited strings
`ParseAsync<T>(Stream stream, Encoding encoding)`	Parse stream asynchronously	`stream`: Data stream, `encoding`: Optional encoding
`ParseAsync<T>(byte[] bytes, Encoding encoding)`	Parse byte array asynchronously	`bytes`: Byte data, `encoding`: Optional encoding

Attributes

ColumnAttribute

[Column(index)]                    // Required column
[Column(index, defaultValue)]      // Column with default
[Column(0, "N/A")]                // String default
[Column(1, 0)]                    // Numeric default
[Column(2, MyEnum.Default)]       // Enum default

Extension Methods

// Dependency injection extension
services.UseParsley();           // Comma delimiter
services.UseParsley('|');        // Custom delimiter

Examples

Example 1: Employee Management System

// Data model
public class Employee : IFileLine
{
    [Column(0)]
    public string EmployeeId { get; set; }

    [Column(1)]
    public FullName Name { get; set; }

    [Column(2)]
    public DateTime HireDate { get; set; }

    [Column(3)]
    public decimal Salary { get; set; }

    [Column(4, Department.Unknown)]
    public Department Department { get; set; }

    [Column(5, true)]
    public bool IsActive { get; set; }

    public int Index { get; set; }
    public IList<string> Errors { get; set; }
}

// Custom type for full names
[TypeConverter(typeof(CustomConverter<FullName>))]
public class FullName : ICustomType
{
    public string First { get; set; }
    public string Last { get; set; }

    public ICustomType Parse(string input)
    {
        var parts = input.Split(' ', 2);
        return new FullName 
        { 
            First = parts[0], 
            Last = parts.Length > 1 ? parts[1] : "" 
        };
    }
}

public enum Department { Unknown, IT, HR, Finance, Marketing }

// Usage
var parser = new Parser(',');
var employees = await parser.ParseAsync<Employee>("employees.csv");

// Process results
var validEmployees = employees.Where(e => e.Errors?.Any() != true).ToList();
var errorCount = employees.Count(e => e.Errors?.Any() == true);

Console.WriteLine($"Processed {validEmployees.Count} valid employees");
Console.WriteLine($"Found {errorCount} records with errors");

Example 2: Financial Transaction Processing

public class Transaction : IFileLine
{
    [Column(0)]
    public string TransactionId { get; set; }

    [Column(1)]
    public DateTime Date { get; set; }

    [Column(2)]
    public TransactionType Type { get; set; }

    [Column(3)]
    public decimal Amount { get; set; }

    [Column(4)]
    public Account FromAccount { get; set; }

    [Column(5)]
    public Account ToAccount { get; set; }

    [Column(6, "")]
    public string Description { get; set; }

    public int Index { get; set; }
    public IList<string> Errors { get; set; }
}

[TypeConverter(typeof(CustomConverter<Account>))]
public class Account : ICustomType
{
    public string BankCode { get; set; }
    public string AccountNumber { get; set; }

    public ICustomType Parse(string input)
    {
        if (string.IsNullOrEmpty(input)) return null;

        var parts = input.Split(':');
        if (parts.Length != 2)
            throw new FormatException($"Invalid account format: {input}");

        return new Account 
        { 
            BankCode = parts[0], 
            AccountNumber = parts[1] 
        };
    }
}

// Processing service
public class TransactionProcessor
{
    private readonly IParser _parser;

    public TransactionProcessor(IParser parser)
    {
        _parser = parser;
    }

    public async Task<ProcessingReport> ProcessTransactionFile(Stream fileStream)
    {
        var transactions = await _parser.ParseAsync<Transaction>(fileStream);

        var report = new ProcessingReport();

        foreach (var transaction in transactions)
        {
            if (transaction.Errors?.Any() == true)
            {
                report.AddError($"Line {transaction.Index}: {string.Join(", ", transaction.Errors)}");
            }
            else
            {
                report.AddTransaction(transaction);
            }
        }

        return report;
    }
}

Example 3: Configuration File Parsing

public class ConfigurationEntry : IFileLine
{
    [Column(0)]
    public string Section { get; set; }

    [Column(1)]
    public string Key { get; set; }

    [Column(2)]
    public ConfigValue Value { get; set; }

    [Column(3, "")]
    public string Comment { get; set; }

    public int Index { get; set; }
    public IList<string> Errors { get; set; }
}

[TypeConverter(typeof(CustomConverter<ConfigValue>))]
public class ConfigValue : ICustomType
{
    public string StringValue { get; set; }
    public ConfigType Type { get; set; }

    public ICustomType Parse(string input)
    {
        if (string.IsNullOrEmpty(input))
            return new ConfigValue { StringValue = "", Type = ConfigType.String };

        // Determine type based on value
        if (bool.TryParse(input, out _))
            return new ConfigValue { StringValue = input, Type = ConfigType.Boolean };

        if (int.TryParse(input, out _))
            return new ConfigValue { StringValue = input, Type = ConfigType.Integer };

        return new ConfigValue { StringValue = input, Type = ConfigType.String };
    }
}

public enum ConfigType { String, Integer, Boolean, Array }

Troubleshooting

Common Issues and Solutions

Issue: "Invalid line format - number of column values do not match"

Cause: The number of columns in the data doesn't match the number of [Column] attributes.

Solution:

// Ensure column attributes match data structure
// Data: "A,B,C"
public class MyClass : IFileLine
{
    [Column(0)] public string Field1 { get; set; }  // A
    [Column(1)] public string Field2 { get; set; }  // B
    [Column(2)] public string Field3 { get; set; }  // C
    // Don't add [Column(3)] without corresponding data
}

Issue: Type conversion errors

Cause: Cannot convert string data to target property type.

Solution:

// Use nullable types for optional data
[Column(2)] public int? OptionalNumber { get; set; }

// Provide default values
[Column(2, 0)] public int NumberWithDefault { get; set; }

// Use custom converters for complex types
[Column(2)] public CustomType ComplexData { get; set; }

Issue: Performance problems with large files

Solutions:

Use async methods: ParseAsync instead of Parse
Process streams instead of loading entire files
Implement batch processing for very large datasets

// Good for large files
using var stream = File.OpenRead(largeFile);
var data = await parser.ParseAsync<MyClass>(stream);

// Better for huge files - process in chunks
var batches = data.Chunk(1000);
foreach (var batch in batches)
{
    await ProcessBatch(batch);
}

Issue: Memory consumption

Solutions:

Use streams instead of loading files into memory
Process data in batches
Dispose of large objects promptly

// Memory efficient approach
await using var fileStream = File.OpenRead(filePath);
var records = await parser.ParseAsync<Record>(fileStream);

// Process immediately, don't store all in memory
foreach (var record in records)
{
    await ProcessRecord(record);
}

Debugging Tips

Check the Index property to identify problematic lines
Examine the Errors collection for detailed error messages
Use a debugger to inspect parsed objects
Validate your data format matches your model
Test with small datasets before processing large files

Getting Help

GitHub Issues: Report bugs or request features
Documentation: Check this wiki for detailed guidance
Examples: Look at the test project for usage patterns

Contributing

We welcome contributions to Parsley.Net! Here's how you can help:

Development Setup

Clone the repository:

git clone https://github.com/CodeShayk/parsley.net.git
cd parsley.net

Build the solution:

dotnet build

Run tests:

dotnet test

Contribution Guidelines

Fork the repository and create a feature branch
Write tests for new functionality
Follow existing code patterns and conventions
Update documentation as needed
Submit a pull request with clear description

Project Structure

parsley.net/
├── src/
│   └── Parsley/                 # Main library code
│       ├── Parser.cs           # Core parser implementation
│       ├── IParser.cs          # Parser interface
│       ├── IFileLine.cs        # Line interface
│       ├── ColumnAttribute.cs  # Column mapping attribute
│       └── CustomConverter.cs  # Built-in custom converter
├── tests/
│   └── Parsley.Tests/          # Unit tests
│       ├── ParserFixture.cs    # Main test class
│       └── FileLines/          # Test data models
└── .github/
    └── workflows/              # CI/CD workflows

Code Standards

Use C# naming conventions
Follow SOLID principles
Write comprehensive unit tests
Document public APIs with XML comments
Keep backward compatibility when possible

Version History

Version	Release Date	Key Features
v1.0.0	2024-12-01	Initial release with basic parsing functionality
v1.1.0	2024-12-15	Added async support, stream processing, dependency injection
v1.1.5	2025-01-15	Performance improvements in async parsing, bug fixes
v2.0.0	2025-10-13	Major release: Configuration options (ParseOptions), Result pattern (TryParse methods), enhanced error reporting with line numbers, improved parallel processing, and comprehensive feature consolidation

Advanced Scenarios

Scenario 1: Multi-Format File Processing

Sometimes you need to handle files with different formats in the same application:

public class FileProcessor
{
    private readonly Dictionary<string, IParser> _parsers;

    public FileProcessor()
    {
        _parsers = new Dictionary<string, IParser>
        {
            [".csv"] = new Parser(','),
            [".tsv"] = new Parser('\t'),
            [".psv"] = new Parser('|'),
            [".txt"] = new Parser(';')
        };
    }

    public async Task<T[]> ProcessFile<T>(string filePath) where T : IFileLine, new()
    {
        var extension = Path.GetExtension(filePath).ToLowerInvariant();

        if (!_parsers.TryGetValue(extension, out var parser))
        {
            throw new NotSupportedException($"File format {extension} is not supported");
        }

        return await parser.ParseAsync<T>(filePath);
    }
}

Scenario 2: Real-time Data Processing

For applications that process data in real-time:

public class RealTimeProcessor<T> where T : IFileLine, new()
{
    private readonly IParser _parser;
    private readonly Queue<string> _lineBuffer;
    private readonly object _lockObject = new object();

    public event Action<T[]> BatchProcessed;
    public event Action<string> ProcessingError;

    public RealTimeProcessor(IParser parser)
    {
        _parser = parser;
        _lineBuffer = new Queue<string>();
    }

    public void AddLine(string line)
    {
        lock (_lockObject)
        {
            _lineBuffer.Enqueue(line);
        }
    }

    public async Task ProcessBatch(int batchSize = 100)
    {
        string[] batch;

        lock (_lockObject)
        {
            if (_lineBuffer.Count < batchSize) return;

            batch = new string[batchSize];
            for (int i = 0; i < batchSize; i++)
            {
                batch[i] = _lineBuffer.Dequeue();
            }
        }

        try
        {
            var results = await _parser.ParseAsync<T>(batch);
            BatchProcessed?.Invoke(results);
        }
        catch (Exception ex)
        {
            ProcessingError?.Invoke($"Batch processing failed: {ex.Message}");
        }
    }
}

// Usage
var processor = new RealTimeProcessor<Employee>(new Parser(','));
processor.BatchProcessed += OnBatchProcessed;
processor.ProcessingError += OnProcessingError;

// Add lines as they come in
processor.AddLine("1,John Doe,IT,50000");
processor.AddLine("2,Jane Smith,HR,55000");

// Process when ready
await processor.ProcessBatch();

Scenario 3: Data Validation Pipeline

Implementing a comprehensive validation pipeline:

public class ValidationPipeline<T> where T : IFileLine, new()
{
    private readonly IParser _parser;
    private readonly List<IValidator<T>> _validators;

    public ValidationPipeline(IParser parser)
    {
        _parser = parser;
        _validators = new List<IValidator<T>>();
    }

    public ValidationPipeline<T> AddValidator(IValidator<T> validator)
    {
        _validators.Add(validator);
        return this;
    }

    public async Task<ValidationResult<T>> ProcessAsync(string filePath)
    {
        var parsed = await _parser.ParseAsync<T>(filePath);
        var result = new ValidationResult<T>();

        foreach (var item in parsed)
        {
            // Check parsing errors first
            if (item.Errors?.Any() == true)
            {
                result.AddInvalid(item, item.Errors);
                continue;
            }

            // Run custom validators
            var validationErrors = new List<string>();
            foreach (var validator in _validators)
            {
                var validationResult = validator.Validate(item);
                if (!validationResult.IsValid)
                {
                    validationErrors.AddRange(validationResult.Errors);
                }
            }

            if (validationErrors.Any())
            {
                result.AddInvalid(item, validationErrors);
            }
            else
            {
                result.AddValid(item);
            }
        }

        return result;
    }
}

public interface IValidator<T>
{
    ValidationResult Validate(T item);
}

public class EmployeeValidator : IValidator<Employee>
{
    public ValidationResult Validate(Employee employee)
    {
        var result = new ValidationResult();

        if (string.IsNullOrWhiteSpace(employee.Name))
            result.AddError("Name is required");

        if (employee.Salary <= 0)
            result.AddError("Salary must be positive");

        if (employee.HireDate > DateTime.Now)
            result.AddError("Hire date cannot be in the future");

        return result;
    }
}

// Usage
var pipeline = new ValidationPipeline<Employee>(new Parser(','))
    .AddValidator(new EmployeeValidator())
    .AddValidator(new EmailValidator());

var result = await pipeline.ProcessAsync("employees.csv");

Console.WriteLine($"Valid records: {result.ValidItems.Count}");
Console.WriteLine($"Invalid records: {result.InvalidItems.Count}");

Scenario 4: Configuration-Driven Parsing

For applications that need flexible, configuration-driven parsing:

public class ConfigurableParser
{
    public class ParsingConfiguration
    {
        public char Delimiter { get; set; } = ',';
        public bool HasHeader { get; set; } = false;
        public Dictionary<string, int> ColumnMappings { get; set; } = new();
        public Dictionary<string, object> DefaultValues { get; set; } = new();
        public Encoding Encoding { get; set; } = Encoding.UTF8;
    }

    public async Task<T[]> ParseWithConfiguration<T>(string filePath, ParsingConfiguration config) 
        where T : IFileLine, new()
    {
        var parser = new Parser(config.Delimiter);
        var lines = await File.ReadAllLinesAsync(filePath, config.Encoding);

        // Skip header if present
        if (config.HasHeader)
        {
            lines = lines.Skip(1).ToArray();
        }

        // Apply configuration-based transformations here
        // This is a simplified example - you could extend this significantly

        return await parser.ParseAsync<T>(lines);
    }
}

// Configuration from appsettings.json
{
  "ParsingConfiguration": {
    "Delimiter": "|",
    "HasHeader": true,
    "ColumnMappings": {
      "EmployeeId": 0,
      "Name": 1,
      "Department": 2
    },
    "DefaultValues": {
      "Department": "Unknown",
      "IsActive": true
    }
  }
}

Integration Examples

ASP.NET Core Web API

[ApiController]
[Route("api/[controller]")]
public class DataController : ControllerBase
{
    private readonly IParser _parser;
    private readonly ILogger<DataController> _logger;

    public DataController(IParser parser, ILogger<DataController> logger)
    {
        _parser = parser;
        _logger = logger;
    }

    [HttpPost("upload-employees")]
    public async Task<IActionResult> UploadEmployees(IFormFile file)
    {
        if (file == null || file.Length == 0)
            return BadRequest("No file uploaded");

        try
        {
            using var stream = file.OpenReadStream();
            var employees = await _parser.ParseAsync<Employee>(stream);

            var validEmployees = employees.Where(e => e.Errors?.Any() != true).ToList();
            var errorCount = employees.Count(e => e.Errors?.Any() == true);

            // Process valid employees (save to database, etc.)
            await ProcessEmployees(validEmployees);

            return Ok(new
            {
                ProcessedCount = validEmployees.Count,
                ErrorCount = errorCount,
                Errors = employees
                    .Where(e => e.Errors?.Any() == true)
                    .Select(e => new { Line = e.Index, Errors = e.Errors })
            });
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Error processing employee file");
            return StatusCode(500, "Error processing file");
        }
    }

    private async Task ProcessEmployees(List<Employee> employees)
    {
        // Your business logic here
        foreach (var employee in employees)
        {
            // Save to database, send notifications, etc.
        }
    }
}

// Startup.cs or Program.cs
public void ConfigureServices(IServiceCollection services)
{
    services.UseParsley(','); // Configure Parsley.Net
    services.AddControllers();
    // Other services...
}

Background Service for File Processing

public class FileProcessingService : BackgroundService
{
    private readonly IParser _parser;
    private readonly ILogger<FileProcessingService> _logger;
    private readonly IServiceScopeFactory _serviceScopeFactory;
    private readonly string _watchFolder;

    public FileProcessingService(
        IParser parser,
        ILogger<FileProcessingService> logger,
        IServiceScopeFactory serviceScopeFactory,
        IConfiguration configuration)
    {
        _parser = parser;
        _logger = logger;
        _serviceScopeFactory = serviceScopeFactory;
        _watchFolder = configuration["FileProcessing:WatchFolder"];
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        using var watcher = new FileSystemWatcher(_watchFolder, "*.csv");

        watcher.Created += async (sender, e) => await ProcessFile(e.FullPath);
        watcher.EnableRaisingEvents = true;

        while (!stoppingToken.IsCancellationRequested)
        {
            await Task.Delay(1000, stoppingToken);
        }
    }

    private async Task ProcessFile(string filePath)
    {
        try
        {
            _logger.LogInformation($"Processing file: {filePath}");

            var records = await _parser.ParseAsync<DataRecord>(filePath);

            using var scope = _serviceScopeFactory.CreateScope();
            var dataService = scope.ServiceProvider.GetRequiredService<IDataService>();

            await dataService.ProcessRecords(records);

            // Move processed file to archive
            var archivePath = Path.Combine(_watchFolder, "processed", Path.GetFileName(filePath));
            File.Move(filePath, archivePath);

            _logger.LogInformation($"Successfully processed {records.Length} records from {filePath}");
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, $"Error processing file: {filePath}");

            // Move failed file to error folder
            var errorPath = Path.Combine(_watchFolder, "errors", Path.GetFileName(filePath));
            File.Move(filePath, errorPath);
        }
    }
}

Console Application with Progress Reporting

class Program
{
    static async Task Main(string[] args)
    {
        if (args.Length != 1)
        {
            Console.WriteLine("Usage: DataProcessor <file-path>");
            return;
        }

        var filePath = args[0];
        var parser = new Parser(',');

        Console.WriteLine($"Processing file: {filePath}");
        Console.WriteLine("Please wait...");

        var stopwatch = Stopwatch.StartNew();

        try
        {
            // For large files, you might want to implement progress reporting
            var records = await ParseWithProgress<DataRecord>(parser, filePath);

            stopwatch.Stop();

            var validRecords = records.Where(r => r.Errors?.Any() != true).ToArray();
            var errorRecords = records.Where(r => r.Errors?.Any() == true).ToArray();

            Console.WriteLine();
            Console.WriteLine($"Processing completed in {stopwatch.Elapsed:mm\\:ss}");
            Console.WriteLine($"Total records: {records.Length:N0}");
            Console.WriteLine($"Valid records: {validRecords.Length:N0}");
            Console.WriteLine($"Error records: {errorRecords.Length:N0}");

            if (errorRecords.Any())
            {
                Console.WriteLine("\nErrors found:");
                foreach (var error in errorRecords.Take(10)) // Show first 10 errors
                {
                    Console.WriteLine($"  Line {error.Index}: {string.Join(", ", error.Errors)}");
                }

                if (errorRecords.Length > 10)
                {
                    Console.WriteLine($"  ... and {errorRecords.Length - 10} more errors");
                }
            }
        }
        catch (Exception ex)
        {
            Console.WriteLine($"Error: {ex.Message}");
            Environment.Exit(1);
        }
    }

    static async Task<T[]> ParseWithProgress<T>(IParser parser, string filePath) where T : IFileLine, new()
    {
        var fileInfo = new FileInfo(filePath);
        var totalBytes = fileInfo.Length;
        var processedBytes = 0L;

        using var fileStream = File.OpenRead(filePath);
        using var reader = new StreamReader(fileStream);

        var lines = new List<string>();
        string line;
        var lastProgress = 0;

        while ((line = await reader.ReadLineAsync()) != null)
        {
            lines.Add(line);
            processedBytes += Encoding.UTF8.GetByteCount(line) + Environment.NewLine.Length;

            var progress = (int)((processedBytes * 100) / totalBytes);
            if (progress > lastProgress)
            {
                Console.Write($"\rReading file: {progress}%");
                lastProgress = progress;
            }
        }

        Console.Write("\rParsing data...  ");
        return await parser.ParseAsync<T>(lines.ToArray());
    }
}

Best Practices Summary

Design Principles

Single Responsibility: Each data model should represent one type of record
Fail Fast: Use validation to catch errors early in the process
Immutable Data: Consider making parsed objects immutable after creation
Error Transparency: Always check and handle the Errors property

Performance Best Practices

Use Async Methods: Always prefer ParseAsync for I/O operations
Stream Large Files: Use Stream or byte[] overloads for large files
Batch Processing: Process large datasets in smaller chunks
Caching: Reuse parser instances when possible
Memory Management: Dispose of streams and large objects promptly

Code Organization

// Good structure
/Models/
  ├── Employee.cs           // Data model with IFileLine
  ├── EmployeeConverters.cs // Custom type converters
  └── EmployeeValidator.cs  // Business validation
/Services/
  ├── IDataService.cs       // Service interface
  └── EmployeeService.cs    // Service implementation
/Configuration/
  └── ParsingExtensions.cs  // DI configuration

Error Handling Strategy

Parse-time Errors: Use the Errors property for field-level issues
Business Validation: Implement separate validation after parsing
File-level Errors: Use try-catch for file access issues
Logging: Always log processing results and errors

Conclusion

Parsley.Net provides a powerful, flexible, and performant solution for parsing structured text files in .NET applications. Its combination of simplicity and extensibility makes it suitable for everything from small utility scripts to large-scale enterprise applications.

Key Takeaways

Simplicity: Minimal configuration required for basic scenarios
Flexibility: Extensive customization options for complex requirements
Performance: Optimized for both small files and large-scale processing
Reliability: Comprehensive error handling and validation support
Integration: Seamless integration with modern .NET patterns and practices

When to Use Parsley.Net

✅ Perfect for:

CSV/TSV file processing
Data migration and ETL operations
Configuration file parsing
Legacy system integration
Batch data processing

❌ Consider alternatives for:

JSON/XML processing (use System.Text.Json or XmlSerializer)
Binary file formats
Real-time streaming data (consider specialized streaming libraries)
Database direct access (use Entity Framework or similar)

Getting Support

Documentation: This wiki provides comprehensive guidance
GitHub Issues: Report issues or request features
Source Code: View the source on GitHub
NuGet Package: Download from NuGet

Happy parsing with Parsley.Net! 🚀

JSON Predicate

CØDE N!NJΔ — Sun, 10 Aug 2025 21:53:49 +0000

This article provides complete information on JSONPredicate libray.

Source Code

Home
Getting Started
Expression Syntax
Operators Reference
Advanced Usage
Best Practices
Performance Considerations
API Reference
Examples
Troubleshooting
Contributing
FAQ

Home

This library provides a powerful and intuitive way to evaluate string-based predicate expressions against JSON objects using JSONPath syntax in .NET applications.

What is JSONPathPredicate?

JSONPathPredicate is a lightweight .NET library that allows developers to write expressive queries against JSON objects using a simple, SQL-like syntax combined with JSONPath for property access. It's designed to be fast, type-safe, and easy to use for filtering, validation, and conditional logic operations.

Key Features

🎯 JSONPath Support: Navigate nested object properties using intuitive dot notation
🔧 Rich Operator Set: Support for equality, comparison, containment, and logical operators
🧮 Type Safety: Automatic type conversion and validation for seamless comparisons
🏗️ Complex Expressions: Parentheses grouping, operator precedence, and nested operations
⚡ High Performance: Optimized evaluation engine with minimal overhead
🪶 Lightweight: Minimal dependencies, fast startup and evaluation times
🕰️ DateTime Support: Built-in handling for various DateTime formats and comparisons
📚 Multi-Framework: Supports .NET Framework 4.6.2, .NET Standard 2.0/2.1, and .NET 9.0

Quick Example

var customer = new {
    profile = new {
        name = "John Doe",
        age = 25,
        isActive = true
    },
    orders = new[] { "premium", "urgent" },
    score = 95.5,
    lastLogin = DateTime.Parse("2024-08-01T10:30:00Z")
};

// Simple equality check
bool isJohn = JSONPredicate.Evaluate("profile.name eq `John Doe`", customer);

// Complex logical expression
bool isEligible = JSONPredicate.Evaluate(
    "profile.age gte 18 and profile.isActive eq true and score gt 90", 
    customer);

// Array containment with date comparison
bool hasRecentActivity = JSONPredicate.Evaluate(
    "orders in (`premium`, `vip`) and lastLogin gt `2024-07-01`", 
    customer);

Getting Started

Installation

Install JSONPathPredicate via NuGet Package Manager:

Package Manager Console

Install-Package JsonPredicate

.NET CLI

dotnet add package JsonPredicate

PackageReference

<PackageReference Include="JsonPredicate" Version="1.0.0" />

Framework Support

Framework	Version	Support Status
.NET Framework	4.6.2+	✅ Full Support
.NET Standard	2.0, 2.1	✅ Full Support
.NET Core	2.0+	✅ Full Support
.NET	5.0, 6.0, 7.0, 8.0, 9.0	✅ Full Support

Basic Usage

Import the namespace:

   using JSONPathPredicate;

Create your data object:

   var data = new {
       user = new {
           name = "Alice",
           age = 30,
           roles = new[] { "admin", "user" }
       },
       settings = new {
           theme = "dark",
           notifications = true
       }
   };

Evaluate expressions:

   bool result = JSONPredicate.Evaluate("user.age gte 18", data);
   // Returns: true

Your First Expression

Let's break down a simple expression:

string expression = "user.name eq `Alice`";
bool result = JSONPredicate.Evaluate(expression, data);

This expression has three parts:

JSONPath (user.name): Navigates to the nested property
Operator (eq): Specifies the comparison type
Value (Alice): The value to compare against (wrapped in backticks)

Expression Syntax

Basic Structure

Every JSONPathPredicate expression follows this pattern:

[JSONPath] [Operator] [Value]

For complex expressions:

[Expression] [Logical Operator] [Expression]

JSONPath Navigation

JSONPath allows you to navigate through nested object properties using dot notation:

Pattern	Description	Example
`property`	Root level property	`name`
`object.property`	Nested property	`user.name`
`object.nested.property`	Deeply nested property	`user.profile.address.city`

Examples:

var data = new {
    customer = new {
        profile = new {
            personal = new {
                firstName = "John",
                lastName = "Doe"
            },
            address = new {
                street = "123 Main St",
                city = "New York",
                zipCode = "10001"
            }
        },
        preferences = new {
            theme = "dark",
            language = "en-US"
        }
    }
};

// Access deeply nested properties
bool result1 = JSONPredicate.Evaluate("customer.profile.personal.firstName eq `John`", data);
bool result2 = JSONPredicate.Evaluate("customer.profile.address.city eq `New York`", data);
bool result3 = JSONPredicate.Evaluate("customer.preferences.theme eq `dark`", data);

Value Syntax

Values in expressions must be properly formatted:

String Values

Wrap string values in backticks, single quotes, or double quotes:

// All equivalent
"name eq `John`"
"name eq 'John'"
"name eq \"John\""

Numeric Values

Numbers can be written directly:

"age eq 25"
"score eq 95.5"
"count eq -10"

Boolean Values

Boolean values are case-insensitive:

"isActive eq true"
"isEnabled eq false"

DateTime Values

DateTime values should be in ISO 8601 format:

"createdAt eq `2024-08-01T10:30:00Z`"
"birthDate gt `1990-01-01`"

Parentheses for Grouping

Use parentheses to control evaluation order:

// Without parentheses - AND has higher precedence
"status eq `active` or role eq `admin` and age gte 18"
// Evaluates as: status eq 'active' or (role eq 'admin' and age gte 18)

// With parentheses - explicit grouping
"(status eq `active` or role eq `admin`) and age gte 18"
// Evaluates as: (status eq 'active' or role eq 'admin') and age gte 18

Comments and Whitespace

Expressions are whitespace-tolerant:

// All equivalent
"user.name eq `John`"
"user.name    eq    `John`"
"   user.name eq `John`   "

Operators Reference

Comparison Operators

Equality Operator (`eq`)

Tests for equality with automatic type conversion and case-insensitive string comparison.

// String comparison (case-insensitive)
"name eq `john`" // matches "John", "JOHN", "john"

// Numeric comparison
"age eq 25" // matches integer 25
"score eq 95.5" // matches double 95.5

// Boolean comparison
"isActive eq true" // matches boolean true

// DateTime comparison
"createdAt eq `2024-01-01T00:00:00Z`" // exact DateTime match

Type Conversion Examples:

var data = new { count = "25" }; // String value
bool result = JSONPredicate.Evaluate("count eq 25", data); // Returns true

Inequality Operator (`not`)

Tests for non-equality.

"status not `inactive`" // true if status is not "inactive"
"age not 0" // true if age is not 0
"isDeleted not true" // true if isDeleted is not true

Contains/In Operator (`in`)

Tests if a value is contained within a collection or if collections intersect.

Single value against collection:

var data = new { tags = new[] { "vip", "premium", "gold" } };

// Check if tags contain specific values
"tags in (`vip`, `platinum`)" // true (vip exists in tags)
"tags in (`basic`, `standard`)" // false (none exist in tags)

Single value membership:

var data = new { role = "admin" };

// Check if role is in allowed list
"role in (`admin`, `moderator`, `user`)" // true

Collection intersection:

var data = new { 
    userTags = new[] { "premium", "early-access" },
    requiredTags = new[] { "premium", "vip" }
};

// Check if collections have any common elements
"userTags in requiredTags" // true (both have "premium")

Comparison Operators (`gt`, `gte`, `lt`, `lte`)

Greater Than (gt)

"age gt 18" // age > 18
"score gt 90.5" // score > 90.5
"createdAt gt `2024-01-01`" // date after 2024-01-01

Greater Than or Equal (gte)

"age gte 18" // age >= 18
"rating gte 4.5" // rating >= 4.5

Less Than (lt)

"age lt 65" // age < 65
"price lt 100.00" // price < 100.00

Less Than or Equal (lte)

"discount lte 50" // discount <= 50
"temperature lte 32.0" // temperature <= 32.0

Numeric Type Handling:
The library automatically handles different numeric types:

var data = new { 
    intValue = 25,
    doubleValue = 25.0,
    floatValue = 25.0f,
    decimalValue = 25.0m
};

// All these return true
"intValue eq 25"
"doubleValue eq 25"
"floatValue eq 25"
"decimalValue eq 25"

Logical Operators

AND Operator (`and`)

Requires all conditions to be true. Has higher precedence than OR.

// Simple AND
"age gte 18 and isActive eq true"

// Multiple AND conditions
"status eq `active` and role eq `admin` and score gt 80"

// Mixed with comparison operators
"price gte 10 and price lte 100 and category eq `electronics`"

OR Operator (`or`)

Requires at least one condition to be true. Has lower precedence than AND.

// Simple OR
"role eq `admin` or role eq `moderator`"

// Multiple OR conditions
"status eq `premium` or status eq `vip` or status eq `gold`"

// Mixed with AND (AND has precedence)
"isActive eq true and role eq `admin` or status eq `override`"
// Evaluates as: (isActive eq true and role eq 'admin') or status eq 'override'

Operator Precedence

The evaluation order follows these precedence rules:

Parentheses (highest precedence)
Comparison operators (eq, not, in, gt, gte, lt, lte)
AND operator
OR operator (lowest precedence)

Examples:

// Without parentheses
"a eq 1 or b eq 2 and c eq 3"
// Evaluates as: a eq 1 or (b eq 2 and c eq 3)

// With parentheses for different grouping
"(a eq 1 or b eq 2) and c eq 3"
// Evaluates as: (a eq 1 or b eq 2) and c eq 3

Special Cases and Edge Conditions

Null Handling

var data = new { nullableField = (string)null };

"nullableField eq `test`" // Returns false
"nullableField not `test`" // Returns true

Empty Collections

var data = new { tags = new string[0] };

"tags in (`anything`)" // Returns false

Type Mismatches

var data = new { textNumber = "25", realNumber = 25 };

"textNumber eq 25" // Returns true (automatic conversion)
"textNumber gt 20" // Returns true (automatic conversion)

Advanced Usage

Complex Nested Expressions

JSONPathPredicate excels at handling complex logical expressions with multiple levels of nesting:

var customer = new {
    profile = new {
        name = "Alice Johnson",
        age = 28,
        membership = "premium",
        verified = true
    },
    account = new {
        balance = 1500.50,
        currency = "USD",
        status = "active",
        lastTransaction = DateTime.Parse("2024-07-15T14:30:00Z")
    },
    preferences = new {
        notifications = true,
        theme = "dark",
        language = "en-US"
    },
    tags = new[] { "vip", "early-adopter", "premium" }
};

// Complex eligibility check
bool isEligibleForOffer = JSONPredicate.Evaluate(@"
    (profile.age gte 18 and profile.age lte 65) and 
    (profile.membership in (`premium`, `vip`, `gold`)) and
    (account.balance gt 1000 and account.status eq `active`) and
    (account.lastTransaction gt `2024-06-01` and profile.verified eq true)
", customer);

// Multi-criteria filtering with fallbacks
bool hasSpecialAccess = JSONPredicate.Evaluate(@"
    (profile.membership eq `premium` and account.balance gt 5000) or
    (tags in (`vip`, `beta-tester`) and profile.verified eq true) or
    (profile.age gt 65 and account.status eq `active`)
", customer);

Working with Arrays and Collections

Array Containment Patterns

var user = new {
    roles = new[] { "user", "admin", "moderator" },
    permissions = new[] { "read", "write", "delete", "admin" },
    tags = new[] { "premium", "verified", "early-adopter" },
    favoriteCategories = new[] { "electronics", "books", "home" }
};

// Check for specific role
bool isAdmin = JSONPredicate.Evaluate("roles in (`admin`)", user);

// Check for any admin permissions
bool hasAdminPerms = JSONPredicate.Evaluate("permissions in (`admin`, `super-admin`)", user);

// Check for multiple tag requirements
bool isPremiumUser = JSONPredicate.Evaluate(
    "tags in (`premium`, `vip`) and roles in (`user`, `admin`)", user);

// Complex array operations
bool canAccessFeature = JSONPredicate.Evaluate(@"
    (roles in (`admin`, `moderator`) and permissions in (`read`, `write`)) or
    (tags in (`premium`, `vip`) and permissions in (`read`))
", user);

Array Intersection Logic

When both sides of an in operation are arrays, the operation checks for intersection:

var data = new {
    userCategories = new[] { "electronics", "books", "sports" },
    allowedCategories = new[] { "electronics", "home", "garden" },
    blockedCategories = new[] { "adult", "gambling" }
};

// Check if user has access to any allowed categories
bool hasAccess = JSONPredicate.Evaluate(
    "userCategories in allowedCategories", data); // true (electronics matches)

// Check if user is accessing blocked content
bool isBlocked = JSONPredicate.Evaluate(
    "userCategories in blockedCategories", data); // false (no intersection)

DateTime Operations and Formats

JSONPathPredicate provides robust DateTime handling with support for multiple formats:

Supported DateTime Formats

var events = new {
    createdAt = DateTime.Parse("2024-08-01T10:30:00Z"),
    updatedAt = "2024-08-02T15:45:00Z", // String will be parsed
    scheduledFor = "2024-08-10",
    deadline = "2024-12-31T23:59:59Z"
};

// ISO 8601 with timezone
"createdAt eq `2024-08-01T10:30:00Z`"

// Date only format
"scheduledFor eq `2024-08-10`"

// Different timezone representations
"createdAt eq `2024-08-01T05:30:00-05:00`" // EST timezone

// Comparison operations
"createdAt gt `2024-07-01` and deadline lt `2025-01-01`"

DateTime Range Queries

// Date range filtering
bool inDateRange = JSONPredicate.Evaluate(@"
    createdAt gte `2024-08-01` and 
    createdAt lte `2024-08-31`
", events);

// Business hours check
var transaction = new {
    timestamp = DateTime.Parse("2024-08-01T14:30:00Z"),
    amount = 250.00
};

bool duringBusinessHours = JSONPredicate.Evaluate(@"
    timestamp gte `2024-08-01T09:00:00Z` and 
    timestamp lte `2024-08-01T17:00:00Z`
", transaction);

Relative Date Comparisons

var document = new {
    createdAt = DateTime.Parse("2024-01-15T10:00:00Z"),
    modifiedAt = DateTime.Parse("2024-07-20T14:30:00Z"),
    expiresAt = DateTime.Parse("2024-12-31T23:59:59Z")
};

// Check if document was created this year
bool thisYear = JSONPredicate.Evaluate("createdAt gte `2024-01-01`", document);

// Check if recently modified (within last 30 days of a reference point)
bool recentlyModified = JSONPredicate.Evaluate(
    "modifiedAt gte `2024-07-01`", document);

Type Coercion and Conversion

JSONPathPredicate automatically handles type conversions for seamless comparisons:

Numeric Conversions

var mixedData = new {
    stringNumber = "42",
    intNumber = 42,
    floatNumber = 42.0f,
    doubleNumber = 42.0,
    decimalNumber = 42.0m
};

// All these expressions return true due to automatic conversion
bool result1 = JSONPredicate.Evaluate("stringNumber eq 42", mixedData);
bool result2 = JSONPredicate.Evaluate("intNumber eq 42.0", mixedData);
bool result3 = JSONPredicate.Evaluate("floatNumber eq 42", mixedData);
bool result4 = JSONPredicate.Evaluate("stringNumber eq floatNumber", mixedData);

String and Character Handling

var textData = new {
    singleChar = 'A',
    charString = "A",
    name = "John",
    upperName = "JOHN"
};

// Case-insensitive string comparisons
bool match1 = JSONPredicate.Evaluate("name eq `john`", textData); // true
bool match2 = JSONPredicate.Evaluate("name eq `JOHN`", textData); // true

// Character-string interoperability
bool match3 = JSONPredicate.Evaluate("singleChar eq `A`", textData); // true
bool match4 = JSONPredicate.Evaluate("charString eq singleChar", textData); // true

Boolean Conversions

var flags = new {
    isActive = true,
    isEnabled = "true",
    isValid = 1, // Truthy value
    isFlagged = false
};

// Boolean comparisons with type conversion
bool result = JSONPredicate.Evaluate("isActive eq true", flags);

Performance Optimization Patterns

Expression Caching Strategy

For applications that repeatedly evaluate the same expressions, consider implementing a caching strategy:

public class PredicateCache
{
    private readonly Dictionary<string, CompiledPredicate> _cache = new();

    public bool Evaluate(string expression, object data)
    {
        // In a real implementation, you might want to compile expressions
        // for better performance on repeated evaluations
        return JSONPredicate.Evaluate(expression, data);
    }
}

Efficient Expression Design

Structure your expressions for optimal performance:

// Good: More selective conditions first
"isActive eq true and role eq `admin` and complexCalculation gt 100"

// Better: Short-circuit on most selective condition
"role eq `admin` and isActive eq true and complexCalculation gt 100"

// Good: Use specific comparisons when possible
"status eq `active`" // Better than "status not `inactive`"

Best Practices

Expression Design Guidelines

1. Use Clear and Descriptive JSONPaths

// Good: Clear property navigation
"user.profile.contactInfo.email eq `john@example.com`"

// Avoid: Unclear abbreviated paths
"u.p.ci.e eq `john@example.com`"

2. Consistent Value Quoting

Choose a quoting style and stick with it throughout your application:

// Good: Consistent backtick usage
"name eq `John` and city eq `New York`"

// Avoid: Mixed quoting styles
"name eq `John` and city eq 'New York'"

3. Logical Expression Structure

Structure complex expressions for readability:

// Good: Logical grouping with clear precedence
"(user.isActive eq true and user.role eq `admin`) or user.permissions in (`override`)"

// Good: Multi-line for complex expressions
string complexRule = @"
    (account.balance gt 1000 and account.status eq `active`) and
    (user.verified eq true and user.age gte 18) and
    (subscription.tier in (`premium`, `enterprise`))";

4. Performance-Conscious Design

// Good: Most selective condition first
"user.role eq `admin` and user.isActive eq true and user.lastLogin gt `2024-01-01`"

// Good: Use positive conditions when possible
"status eq `active`" // Instead of "status not `inactive`"

Error Handling Patterns

1. Expression Validation

public class PredicateValidator
{
    public bool TryValidateExpression(string expression, out string error)
    {
        try
        {
            // Test with dummy object to validate syntax
            var testObj = new { test = "value" };
            JSONPredicate.Evaluate("test eq `value`", testObj);
            error = null;
            return true;
        }
        catch (ArgumentException ex)
        {
            error = ex.Message;
            return false;
        }
    }
}

2. Safe Evaluation Pattern

public static class SafeJSONPredicate
{
    public static bool TryEvaluate(string expression, object data, out bool result)
    {
        try
        {
            result = JSONPredicate.Evaluate(expression, data);
            return true;
        }
        catch (Exception)
        {
            result = false;
            return false;
        }
    }

    public static bool EvaluateWithDefault(string expression, object data, bool defaultValue = false)
    {
        try
        {
            return JSONPredicate.Evaluate(expression, data);
        }
        catch (Exception)
        {
            return defaultValue;
        }
    }
}

Security Considerations

1. Input Sanitization

When accepting user-provided expressions, implement proper validation:

public class ExpressionSanitizer
{
    private readonly HashSet<string> _allowedOperators = new HashSet<string>
    {
        "eq", "not", "in", "gt", "gte", "lt", "lte", "and", "or"
    };

    private readonly HashSet<string> _allowedPaths = new HashSet<string>
    {
        "user.name", "user.age", "user.role", "account.balance"
        // Define allowed JSONPaths
    };

    public bool IsValidExpression(string expression)
    {
        // Implement validation logic here
        // Check for allowed operators and paths only
        return true; // Simplified for example
    }
}

2. Path Restriction

Limit which properties can be accessed:

public class RestrictedJSONPredicate
{
    private readonly HashSet<string> _allowedPaths;

    public RestrictedJSONPredicate(IEnumerable<string> allowedPaths)
    {
        _allowedPaths = new HashSet<string>(allowedPaths);
    }

    public bool Evaluate(string expression, object data)
    {
        // Extract paths from expression and validate
        var paths = ExtractPaths(expression);
        if (paths.Any(p => !_allowedPaths.Contains(p)))
        {
            throw new UnauthorizedAccessException("Expression contains restricted paths");
        }

        return JSONPredicate.Evaluate(expression, data);
    }

    private IEnumerable<string> ExtractPaths(string expression)
    {
        // Implementation to extract JSONPaths from expression
        // This is a simplified example
        return new List<string>();
    }
}

Testing Strategies

1. Unit Testing Expressions

[TestFixture]
public class BusinessRuleTests
{
    private readonly object _testCustomer = new {
        profile = new {
            name = "John Doe",
            age = 35,
            membershipLevel = "gold"
        },
        account = new {
            balance = 2500.00,
            status = "active"
        },
        preferences = new {
            emailNotifications = true
        }
    };

    [Test]
    public void PremiumCustomerRule_ShouldReturnTrue_WhenCriteriaMet()
    {
        var rule = "profile.age gte 18 and account.balance gt 1000 and profile.membershipLevel in (`gold`, `platinum`)";

        var result = JSONPredicate.Evaluate(rule, _testCustomer);

        Assert.That(result, Is.True);
    }

    [TestCase("profile.age", 35, true)]
    [TestCase("profile.age", 17, false)]
    public void AgeValidation_ShouldWorkCorrectly(string path, int age, bool expected)
    {
        var customer = new { profile = new { age = age } };
        var rule = "profile.age gte 18";

        var result = JSONPredicate.Evaluate(rule, customer);

        Assert.That(result, Is.EqualTo(expected));
    }
}

2. Expression Coverage Testing

public class ExpressionCoverageHelper
{
    public static void TestAllOperators(object testData)
    {
        var operators = new[] { "eq", "not", "in", "gt", "gte", "lt", "lte" };
        var logicalOps = new[] { "and", "or" };

        foreach (var op in operators)
        {
            // Test each operator with various data types
            TestOperator(op, testData);
        }
    }

    private static void TestOperator(string op, object data)
    {
        // Implementation to systematically test operators
    }
}

Performance Considerations

Evaluation Performance

Benchmarking Results

Based on internal testing, JSONPathPredicate provides excellent performance characteristics:

Expression Type	Evaluations/sec	Memory Usage
Simple equality	~2,000,000	< 1KB
Complex logical	~500,000	< 2KB
Array operations	~300,000	< 3KB
DateTime comparisons	~400,000	< 2KB

Performance Optimization Tips

Order Conditions by Selectivity

   // Good: Most selective first
   "user.role eq `admin` and user.isActive eq true"

   // Less optimal: Less selective first
   "user.isActive eq true and user.role eq `admin`"

Use Positive Conditions

   // Preferred
   "status eq `active`"

   // Less efficient
   "status not `inactive`"

Minimize Deep Nesting

   // Consider flattening very deep paths
   "user.profile.settings.preferences.theme eq `dark`"

Memory Usage Patterns

Object Serialization Overhead

JSONPathPredicate serializes objects to JSON for evaluation. Consider these patterns:

// Good: Lightweight evaluation objects
var slimData = new {
    userId = user.Id,
    role = user.Role,
    isActive = user.IsActive
};
bool result = JSONPredicate.Evaluate("role eq `admin`", slimData);

// Less optimal: Heavy objects with unnecessary data
bool result2 = JSONPredicate.Evaluate("role eq `admin`", fullUserObjectWithManyProperties);

Caching Strategies

For repeated evaluations, consider implementing caching:

public class CachedPredicateEvaluator
{
    private readonly LRUCache<string, object> _objectCache = new(1000);

    public bool Evaluate(string expression, object data)
    {
        string dataKey = GenerateKey(data);

        if (!_objectCache.TryGetValue(dataKey, out object cachedData))
        {
            cachedData = data;
            _objectCache[dataKey] = cachedData;
        }

        return JSONPredicate.Evaluate(expression, cachedData);
    }
}

Scalability Considerations

High-Volume Scenarios

For applications processing thousands of evaluations per second:

public class HighPerformancePredicateService
{
    private readonly ConcurrentDictionary<string, CompiledExpression> _expressionCache = new();

    public async Task<bool[]> EvaluateBatchAsync(string expression, IEnumerable<object> dataItems)
    {
        var tasks = dataItems.Select(data => 
            Task.Run(() => JSONPredicate.Evaluate(expression, data))
        );

        return await Task.WhenAll(tasks);
    }

    public bool EvaluateWithMetrics(string expression, object data, out TimeSpan duration)
    {
        var stopwatch = Stopwatch.StartNew();
        var result = JSONPredicate.Evaluate(expression, data);
        duration = stopwatch.Elapsed;
        return result;
    }
}

Memory-Efficient Patterns

// Pattern: Use projection for large objects
public static class EfficientEvaluation
{
    public static bool EvaluateProjected<T>(string expression, T source, Func<T, object> projector)
    {
        var projectedData = projector(source);
        return JSONPredicate.Evaluate(expression, projectedData);
    }
}

// Usage
var result = EfficientEvaluation.EvaluateProjected(
    "name eq `John` and age gt 18",
    fullUserObject,
    user => new { name = user.FullName, age = user.Age }
);

API Reference

JSONPredicate Class

The main entry point for evaluating predicate expressions.

Methods

`Evaluate(string expression, object obj)`

Evaluates a predicate expression against an object.

Parameters:

expression (string): The predicate expression to evaluate
obj (object): The object to evaluate the expression against

Returns:

bool: True if the expression evaluates to true, false otherwise

Exceptions:

ArgumentException: Thrown when the expression format is invalid
FormatException: Thrown when DateTime parsing fails
InvalidOperationException: Thrown for unsupported operations

Example:

var data = new { name = "John", age = 25 };
bool result = JSONPredicate.Evaluate("name eq `John` and age gte 18", data);

Expression Grammar

The formal grammar for JSONPathPredicate expressions:

Expression := OrExpression

OrExpression := AndExpression ('or' AndExpression)*

AndExpression := AtomicExpression ('and' AtomicExpression)*

AtomicExpression := '(' Expression ')'
                 | JSONPath Operator Value

JSONPath := Identifier ('.' Identifier)*

Operator := 'eq' | 'not' | 'in' | 'gt' | 'gte' | 'lt' | 'lte'

Value := StringLiteral | NumberLiteral | BooleanLiteral | ArrayLiteral

StringLiteral := '`' [^`]* '`' | '\'' [^']* '\'' | '"' [^"]* '"'

NumberLiteral := '-'? [0-9]+ ('.' [0-9]+)?

BooleanLiteral := 'true' | 'false'

ArrayLiteral := '(' Value (',' Value)* ')'

Identifier := [a-zA-Z_][a-zA-Z0-9_]*

Internal Components

While these are internal implementation details, understanding them can help with troubleshooting:

DataTypes Class

Handles type comparisons and conversions.

JsonPath Class

Manages JSONPath navigation and property resolution.

Examples

Real-World Use Cases

1. User Authentication and Authorization

public class AuthService
{
    public bool IsAuthorized(User user, string resource, string action)
    {
        var context = new {
            user = new {
                id = user.Id,
                role = user.Role,
                permissions = user.Permissions.ToArray(),
                isActive = user.IsActive,
                accountExpiry = user.AccountExpiry
            },
            resource = resource,
            action = action,
            currentTime = DateTime.UtcNow
        };

        // Admin override
        if (JSONPredicate.Evaluate("user.role eq `admin` and user.isActive eq true", context))
            return true;

        // Check specific permissions
        var permissionRule = $"user.permissions in (`{resource}:{action}`, `{resource}:*`, `*:*`)";
        if (JSONPredicate.Evaluate($"{permissionRule} and user.isActive eq true", context))
            return true;

        // Account expiry check
        if (JSONPredicate.Evaluate("user.accountExpiry lt currentTime", context))
            return false;

        return false;
    }
}

2. E-commerce Product Filtering

public class ProductFilterService
{
    public IEnumerable<Product> FilterProducts(IEnumerable<Product> products, ProductFilterCriteria criteria)
    {
        var filterExpression = BuildFilterExpression(criteria);

        return products.Where(product => 
        {
            var productData = new {
                name = product.Name,
                price = product.Price,
                category = product.Category,
                brand = product.Brand,
                rating = product.AverageRating,
                inStock = product.StockQuantity > 0,
                tags = product.Tags.ToArray(),
                releaseDate = product.ReleaseDate,
                onSale = product.SalePrice.HasValue
            };

            return JSONPredicate.Evaluate(filterExpression, productData);
        });
    }

    private string BuildFilterExpression(ProductFilterCriteria criteria)
    {
        var conditions = new List<string>();

        if (criteria.MinPrice.HasValue)
            conditions.Add($"price gte {criteria.MinPrice.Value}");

        if (criteria.MaxPrice.HasValue)
            conditions.Add($"price lte {criteria.MaxPrice.Value}");

        if (!string.IsNullOrEmpty(criteria.Category))
            conditions.Add($"category eq `{criteria.Category}`");

        if (criteria.InStockOnly)
            conditions.Add("inStock eq true");

        if (criteria.MinRating.HasValue)
            conditions.Add($"rating gte {criteria.MinRating.Value}");

        if (criteria.Tags?.Any() == true)
        {
            var tagsList = string.Join("`, `", criteria.Tags);
            conditions.Add($"tags in (`{tagsList}`)");
        }

        if (criteria.OnSaleOnly)
            conditions.Add("onSale eq true");

        return conditions.Any() ? string.Join(" and ", conditions) : "inStock eq true";
    }
}

3. Business Rule Engine

public class BusinessRuleEngine
{
    private readonly Dictionary<string, string> _rules = new()
    {
        ["PREMIUM_ELIGIBILITY"] = @"
            (customer.age gte 21 and customer.creditScore gt 700) and
            (account.balance gt 10000 or account.monthlyIncome gt 5000) and
            customer.accountAge gte 365",

        ["DISCOUNT_ELIGIBILITY"] = @"
            (customer.loyaltyTier in (`gold`, `platinum`) and order.amount gt 100) or
            (customer.isFirstTime eq true and order.amount gt 50) or
            (customer.tags in (`employee`, `partner`) and order.amount gt 0)",

        ["FRAUD_DETECTION"] = @"
            (transaction.amount gt customer.averageTransactionAmount * 10) or
            (transaction.location not customer.usualLocations) or
            (transaction.time lt `06:00` or transaction.time gt `23:00`) and
            (transaction.amount gt 1000)"
    };

    public bool EvaluateRule(string ruleName, object context)
    {
        if (!_rules.TryGetValue(ruleName, out var expression))
            throw new ArgumentException($"Unknown rule: {ruleName}");

        return JSONPredicate.Evaluate(expression, context);
    }

    public Dictionary<string, bool> EvaluateAllRules(object context)
    {
        return _rules.ToDictionary(
            kvp => kvp.Key,
            kvp => JSONPredicate.Evaluate(kvp.Value, context)
        );
    }
}

4. Configuration-Driven Validation

public class ConfigurableValidator
{
    public class ValidationRule
    {
        public string Name { get; set; }
        public string Expression { get; set; }
        public string ErrorMessage { get; set; }
        public bool IsWarning { get; set; }
    }

    private readonly List<ValidationRule> _rules = new()
    {
        new ValidationRule
        {
            Name = "AGE_VALIDATION",
            Expression = "user.age gte 18 and user.age lte 120",
            ErrorMessage = "User age must be between 18 and 120",
            IsWarning = false
        },
        new ValidationRule
        {
            Name = "EMAIL_DOMAIN_CHECK",
            Expression = "user.email in (`@company.com`, `@partner.com`)",
            ErrorMessage = "Only company or partner email addresses are allowed",
            IsWarning = true
        }
    };

    public ValidationResult Validate(object data)
    {
        var result = new ValidationResult { IsValid = true };

        foreach (var rule in _rules)
        {
            try
            {
                bool isValid = JSONPredicate.Evaluate(rule.Expression, data);

                if (!isValid)
                {
                    if (rule.IsWarning)
                    {
                        result.Warnings.Add(rule.ErrorMessage);
                    }
                    else
                    {
                        result.Errors.Add(rule.ErrorMessage);
                        result.IsValid = false;
                    }
                }
            }
            catch (Exception ex)
            {
                result.Errors.Add($"Rule '{rule.Name}' evaluation failed: {ex.Message}");
                result.IsValid = false;
            }
        }

        return result;
    }
}

public class ValidationResult
{
    public bool IsValid { get; set; }
    public List<string> Errors { get; set; } = new();
    public List<string> Warnings { get; set; } = new();
}

5. Dynamic Content Personalization

public class ContentPersonalizationService
{
    private readonly Dictionary<string, ContentRule> _contentRules = new()
    {
        ["SHOW_PREMIUM_CTA"] = new ContentRule
        {
            Expression = "user.tier eq `free` and user.usageDays gt 7 and user.featuresUsed gt 3",
            Content = "Upgrade to Premium for unlimited access!"
        },
        ["SHOW_RENEWAL_NOTICE"] = new ContentRule
        {
            Expression = "user.subscriptionExpiry lt `2024-12-31` and user.tier in (`premium`, `pro`)",
            Content = "Your subscription expires soon. Renew now!"
        },
        ["SHOW_WELCOME_MESSAGE"] = new ContentRule
        {
            Expression = "user.isNewUser eq true and user.lastLogin eq null",
            Content = "Welcome! Let's get you started."
        }
    };

    public List<string> GetPersonalizedContent(User user)
    {
        var userContext = new {
            user = new {
                id = user.Id,
                tier = user.SubscriptionTier,
                isNewUser = user.CreatedAt > DateTime.UtcNow.AddDays(-7),
                lastLogin = user.LastLoginAt,
                subscriptionExpiry = user.SubscriptionExpiry,
                usageDays = (DateTime.UtcNow - user.CreatedAt).Days,
                featuresUsed = user.FeatureUsageCount
            }
        };

        var applicableContent = new List<string>();

        foreach (var rule in _contentRules)
        {
            if (JSONPredicate.Evaluate(rule.Value.Expression, userContext))
            {
                applicableContent.Add(rule.Value.Content);
            }
        }

        return applicableContent;
    }

    private class ContentRule
    {
        public string Expression { get; set; }
        public string Content { get; set; }
    }
}

Integration Examples

ASP.NET Core Integration

// Startup.cs or Program.cs
public void ConfigureServices(IServiceCollection services)
{
    services.AddScoped<IPredicateService, PredicateService>();
}

// Controller
[ApiController]
[Route("api/[controller]")]
public class UsersController : ControllerBase
{
    private readonly IPredicateService _predicateService;

    public UsersController(IPredicateService predicateService)
    {
        _predicateService = predicateService;
    }

    [HttpGet("filter")]
    public IActionResult FilterUsers([FromQuery] string expression)
    {
        if (string.IsNullOrEmpty(expression))
            return BadRequest("Expression is required");

        try
        {
            var users = GetUsers(); // Your data source
            var filteredUsers = users.Where(user => 
                _predicateService.EvaluateUserFilter(expression, user));

            return Ok(filteredUsers);
        }
        catch (ArgumentException ex)
        {
            return BadRequest($"Invalid expression: {ex.Message}");
        }
    }
}

Entity Framework Integration

public static class QueryableExtensions
{
    public static IQueryable<T> WherePredicate<T>(this IQueryable<T> source, string expression)
    {
        return source.AsEnumerable()
            .Where(item => JSONPredicate.Evaluate(expression, item))
            .AsQueryable();
    }

    // For in-memory evaluation after database query
    public static IEnumerable<T> FilterWithPredicate<T>(this IEnumerable<T> source, string expression)
    {
        return source.Where(item => JSONPredicate.Evaluate(expression, item));
    }
}

// Usage
var activeAdminUsers = dbContext.Users
    .Where(u => u.IsActive) // Database filter
    .ToList() // Execute database query
    .FilterWithPredicate("role eq `admin` and lastLogin gt `2024-01-01`"); // In-memory predicate

Background Service Integration

public class RuleEvaluationService : BackgroundService
{
    private readonly IServiceProvider _serviceProvider;
    private readonly ILogger<RuleEvaluationService> _logger;

    public RuleEvaluationService(IServiceProvider serviceProvider, ILogger<RuleEvaluationService> logger)
    {
        _serviceProvider = serviceProvider;
        _logger = logger;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            using var scope = _serviceProvider.CreateScope();
            var businessRuleEngine = scope.ServiceProvider.GetRequiredService<BusinessRuleEngine>();

            try
            {
                await ProcessPendingRules(businessRuleEngine);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error processing business rules");
            }

            await Task.Delay(TimeSpan.FromMinutes(5), stoppingToken);
        }
    }

    private async Task ProcessPendingRules(BusinessRuleEngine ruleEngine)
    {
        var pendingEvaluations = GetPendingEvaluations();

        foreach (var evaluation in pendingEvaluations)
        {
            var result = ruleEngine.EvaluateRule(evaluation.RuleName, evaluation.Context);
            await ProcessRuleResult(evaluation, result);
        }
    }
}

Troubleshooting

Common Issues and Solutions

1. Expression Format Errors

Problem: ArgumentException: Invalid expression format

Common Causes:

Missing operators
Incorrect syntax
Unmatched parentheses

Solutions:

// ❌ Incorrect
"user.name John" // Missing operator

// ✅ Correct
"user.name eq `John`"

// ❌ Incorrect
"user.age > 18" // Invalid operator

// ✅ Correct
"user.age gt 18"

// ❌ Incorrect
"(user.age gte 18 and user.isActive eq true" // Unmatched parenthesis

// ✅ Correct
"(user.age gte 18 and user.isActive eq true)"

2. Type Conversion Issues

Problem: Unexpected comparison results

Common Causes:

Type mismatches
String/numeric confusion
DateTime format issues

Solutions:

// Issue: String numbers not comparing correctly
var data = new { count = "25" };

// ❌ This might not work as expected in some cases
"count gt 20"

// ✅ Better approach - be explicit about types
// The library handles this automatically, but be aware of your data types

// Issue: DateTime format problems
// ❌ Incorrect format
"createdAt gt `2024-13-01`" // Invalid month

// ✅ Correct format
"createdAt gt `2024-01-01T00:00:00Z`"

3. JSONPath Resolution Issues

Problem: Properties not found or incorrect navigation

Common Causes:

Incorrect property names
Case sensitivity issues
Missing nested properties

Solutions:

var data = new {
    User = new {  // Note: Capital 'U'
        Name = "John"  // Note: Capital 'N'
    }
};

// ❌ Incorrect casing
"user.name eq `John`" // Returns false - property not found

// ✅ Correct casing
"User.Name eq `John`" // Returns true

// ❌ Non-existent path
"user.profile.name eq `John`" // Returns false if profile doesn't exist

// ✅ Check your object structure
Console.WriteLine(JsonSerializer.Serialize(data, new JsonSerializerOptions { WriteIndented = true }));

4. Array Operation Confusion

Problem: in operator not working as expected

Common Causes:

Misunderstanding of array intersection logic
Incorrect value syntax

Solutions:

var data = new { tags = new[] { "admin", "user" } };

// ❌ Incorrect - missing parentheses for multiple values
"tags in `admin`, `user`"

// ✅ Correct
"tags in (`admin`, `user`)"

// ❌ Incorrect - trying to check if string contains array
var singleTag = new { role = "admin" };
"role in tags" // This won't work - tags doesn't exist in singleTag

// ✅ Correct - check if single value is in array
"role in (`admin`, `user`, `moderator`)"

Debugging Techniques

1. Expression Breakdown

public static class PredicateDebugger
{
    public static void DebugExpression(string expression, object data)
    {
        Console.WriteLine($"Expression: {expression}");
        Console.WriteLine($"Data: {JsonSerializer.Serialize(data, new JsonSerializerOptions { WriteIndented = true })}");

        try
        {
            var result = JSONPredicate.Evaluate(expression, data);
            Console.WriteLine($"Result: {result}");
        }
        catch (Exception ex)
        {
            Console.WriteLine($"Error: {ex.Message}");
        }
    }

    public static void TestExpressionParts(string expression, object data)
    {
        // Break down complex expressions into parts
        var parts = SplitExpression(expression);

        foreach (var part in parts)
        {
            try
            {
                var result = JSONPredicate.Evaluate(part, data);
                Console.WriteLine($"'{part}' -> {result}");
            }
            catch (Exception ex)
            {
                Console.WriteLine($"'{part}' -> ERROR: {ex.Message}");
            }
        }
    }
}

2. Data Structure Inspection

public static class DataInspector
{
    public static void InspectObject(object obj)
    {
        var json = JsonSerializer.Serialize(obj, new JsonSerializerOptions 
        { 
            WriteIndented = true,
            PropertyNamingPolicy = JsonNamingPolicy.CamelCase
        });

        Console.WriteLine("Object structure:");
        Console.WriteLine(json);

        // Extract all possible JSONPaths
        var paths = ExtractAllPaths(obj);
        Console.WriteLine("\nAvailable JSONPaths:");
        foreach (var path in paths)
        {
            Console.WriteLine($"  {path}");
        }
    }

    private static List<string> ExtractAllPaths(object obj, string prefix = "")
    {
        var paths = new List<string>();
        var json = JsonSerializer.Serialize(obj);
        using var doc = JsonDocument.Parse(json);
        ExtractPathsRecursive(doc.RootElement, prefix, paths);
        return paths;
    }

    private static void ExtractPathsRecursive(JsonElement element, string currentPath, List<string> paths)
    {
        switch (element.ValueKind)
        {
            case JsonValueKind.Object:
                foreach (var prop in element.EnumerateObject())
                {
                    var newPath = string.IsNullOrEmpty(currentPath) ? prop.Name : $"{currentPath}.{prop.Name}";
                    paths.Add(newPath);
                    ExtractPathsRecursive(prop.Value, newPath, paths);
                }
                break;
        }
    }
}

3. Performance Profiling

public static class PerformanceProfiler
{
    public static void ProfileExpression(string expression, object data, int iterations = 1000)
    {
        // Warmup
        for (int i = 0; i < 100; i++)
        {
            JSONPredicate.Evaluate(expression, data);
        }

        var stopwatch = Stopwatch.StartNew();

        for (int i = 0; i < iterations; i++)
        {
            JSONPredicate.Evaluate(expression, data);
        }

        stopwatch.Stop();

        Console.WriteLine($"Expression: {expression}");
        Console.WriteLine($"Iterations: {iterations}");
        Console.WriteLine($"Total time: {stopwatch.ElapsedMilliseconds}ms");
        Console.WriteLine($"Average time: {(double)stopwatch.ElapsedTicks / iterations / TimeSpan.TicksPerMillisecond:F4}ms");
        Console.WriteLine($"Evaluations per second: {iterations / stopwatch.Elapsed.TotalSeconds:F0}");
    }
}

Error Message Guide

Error Message	Cause	Solution
"Invalid expression format: ..."	Syntax error in expression	Check expression syntax, ensure proper operator usage
"Unable to parse DateTime value: ..."	Invalid DateTime format	Use ISO 8601 format (YYYY-MM-DDTHH:mm:ssZ)
"Comparison operator not supported"	Unknown operator	Use only: eq, not, in, gt, gte, lt, lte
"Property not found: ..."	JSONPath doesn't exist	Verify object structure and property names

Contributing

Development Setup

Clone the repository

   git clone https://github.com/CodeShayk/JSONPathPredicate.git
   cd JSONPathPredicate

Install dependencies

   dotnet restore

Build the solution

   dotnet build

Run tests

   dotnet test

Contribution Guidelines

Code Standards

Follow C# coding conventions
Use meaningful variable and method names
Add XML documentation for public APIs
Include unit tests for new features
Maintain backward compatibility

Testing Requirements

All contributions must include comprehensive tests:

[TestFixture]
public class NewFeatureTests
{
    [Test]
    public void NewFeature_ValidInput_ShouldReturnExpectedResult()
    {
        // Arrange
        var testData = new { /* test object */ };
        var expression = "/* your expression */";

        // Act
        var result = JSONPredicate.Evaluate(expression, testData);

        // Assert
        Assert.That(result, Is.True);
    }

    [Test]
    public void NewFeature_InvalidInput_ShouldThrowException()
    {
        // Test error conditions
    }

    [TestCase(/* parameters */)]
    public void NewFeature_ParameterizedTests(/* parameters */)
    {
        // Parameterized test cases
    }
}

Pull Request Process

Create a feature branch

   git checkout -b feature/your-feature-name

Make your changes with tests
Ensure all tests pass

   dotnet test --verbosity normal

Update documentation if needed
Submit pull request with clear description

Reporting Issues

When reporting bugs, please include:

Environment details (.NET version, OS)
Minimal reproduction case
Expected vs actual behavior
Error messages or stack traces

// Example bug report template
var testData = new { /* minimal object */ };
var expression = "/* problematic expression */";

// Expected: true
// Actual: false (or exception)
var result = JSONPredicate.Evaluate(expression, testData);

FAQ

General Questions

Q: What's the performance impact of using JSONPathPredicate?
A: JSONPathPredicate is highly optimized for performance. Simple expressions evaluate at ~2M ops/second, complex expressions at ~500K ops/second. The main overhead is JSON serialization, so consider using lightweight projection objects for high-volume scenarios.

Q: Can I use JSONPathPredicate with Entity Framework?
A: JSONPathPredicate works with in-memory collections. For Entity Framework, first execute your database query, then apply JSONPathPredicate filters to the results in memory.

Q: Is JSONPathPredicate thread-safe?
A: Yes, JSONPathPredicate is stateless and thread-safe. You can safely call JSONPredicate.Evaluate() from multiple threads simultaneously.

Q: What's the maximum expression complexity supported?
A: There's no hard limit on expression complexity. However, very deep nesting or extremely long expressions may impact performance. Consider breaking complex rules into smaller, composed expressions.

Technical Questions

Q: How does type conversion work?
A: JSONPathPredicate automatically converts between compatible types:

String numbers to numeric types for comparisons
Case-insensitive string comparisons
DateTime string parsing with multiple format support
Boolean value interpretation

Q: Can I extend JSONPathPredicate with custom operators?
A: The current version doesn't support custom operators. This is a planned feature for future releases. Consider using composition of existing operators for now.

Q: How are null values handled?
A: Null values are handled gracefully:

null eq null returns true
null eq "anything" returns false
Missing properties are treated as null

Q: What JSONPath features are supported?
A: Currently supports:

Property navigation with dot notation (object.property)
Nested object access (object.nested.property)

Not yet supported:

Array indexing (array[0])
Wildcards (object.*)
Recursive descent (object..property)

Best Practices Questions

Q: How should I structure complex business rules?
A: Consider these patterns:

Use a rule engine pattern with named rules
Break complex expressions into smaller, testable parts
Use configuration files for business rules
Implement rule versioning for changes over time

Q: How do I handle user-provided expressions safely?
A: Implement validation layers:

Whitelist allowed JSONPaths
Validate expression syntax before execution
Use try-catch patterns for graceful error handling
Consider rate limiting for user-provided expressions

Q: What's the best way to debug failing expressions?
A: Use these debugging techniques:

Break complex expressions into parts
Inspect your data structure with JSON serialization
Test individual conditions separately
Use the debugging utilities provided in the troubleshooting section

FeatureOne - Feature Toggle Library.

CØDE N!NJΔ — Sun, 10 Nov 2024 20:18:51 +0000

Concept

What is a feature toggle?

Feature toggle is a mechanism that allows code to be turned “on” or “off” remotely without the need for a deploy. Feature toggles are commonly used in applications to gradually roll out new features, allowing teams to test changes on a small subset of users before releasing them to everyone.

How feature toggles work

Feature toggle is typically a logical check added to codebase to execute or ignore certain functionality in context based on evaluated status of the toggle at runitme.

In code, the functionality to be released is wrapped so that it can be controlled by the status of a feature toggle. If the status of the feature toggle is “on”, then the wrapped functionality is executed. If the status of the feature toggle is “off”, then the wrapped functionality is skipped. The statuses of each feature is provided by a store provider external to the application.

The benefits of feature toggles

The primary benefit of feature flagging is that it mitigates the risks associated with releasing changes to an application. Whether it be a new feature release or a small refactor, there is always the inherent risk of releasing new regressions. To mitigate this, changes to an application can be placed behind feature toggles, allowing them to be turned “on” or “off” in the event of an emergency.

Getting Started?

i. Installation

Install the latest nuget package as appropriate.

FeatureOne - for installing FeatureOne for custom IStorageProvider implementation.

NuGet\Install-Package FeatureOne

FeatureOne.SQL - for installing FeatureOne with SQL storage provider.

NuGet\Install-Package FeatureOne.SQL

FeatureOne.File - for installing FeatureOne with File system storage provider.

NuGet\Install-Package FeatureOne.File

ii. How to use FeatureOne

Step 1. Add Feature IsEnabled Check in Code.

In order to release a new functionality or feature - say eg. Dashboard Widget.
Add logical check in codebase to wrap the functionality under a feature toggle.

the logical check evaluates status of the toggle configured for the feature in store at runtime.

 var featureName = "dashboard_widget"; // Name of functionality or feature to toggle.
 if(Features.Current.IsEnable(featureName){ // See other IsEnable() overloads
    showDashboardWidget();
}

Step 2. Add Feature Toggle Definition to Storage

Add a toggle definition to storage ie. a store in database or file or other storage medium.
A toggle constitutes a collection of conditions that evaluate separately when the toggle is run. You can additionally specify an operator in the toggle definition to determine the overall success to include success of any constituent condition or success of all consituent conditions.

Toggles run at runtime based on consitituent conditions that evaluate separately against user claims (generally logged in user principal).

Below is a serialized JSON representation of a Feature Toggle.

{
  "feature_name":{ -- Feature name
        "toggle":{ -- Toggle definition for the feature 

            "operator":"any|all", -- Logical Operator - any (OR) & all (AND)
                                  -- ie. Evaluate overall toggle to true when `any` condition is met or
                                  --     `all` conditions are met.

            "conditions":[{ -- collection of conditions
                "type":"simple|regex" -- type of condition

                 .... other type specific properties, See below for details.                  
            }]
        }
  }
}

Condition Types

There are two types of toggle conditions that can be used out of box.

i. Simple Condition

Simple condition allows toggle with simple enable or disable of the given feature. User claims are not taken into account for this condition.

Below is the serialized representation of toggle with simple condition.

{
  "dashboard_widget":{   
      "toggle":{       
         "conditions":[{
           "type":"Simple",       -- Simple Condition.
           "isEnabled":true|false --  Enabled or disable the feature.
          }]          
      }           
  }
}

C# representation of a feature with simple toggle is

var feature = new Feature
{
  Name ="dashboard_widget",   // Feature Name
  Toggle = new Toggle         // Toggle definition
  {
    // Logical operator to be applied when evaluating consituent conditions.
    Operator = Operator.Any,  // Default is Any (Logical OR)

    Conditions = new[]
    {
        // Simple condition that can be set to true/false for feature to be enabled/disabled.
        new SimpleCondition { IsEnabled = true }
    }
  }
}

ii. Regex Condition

Regex condition allows evaluating a regex expression against specified user claim value to enable a given feature.

Below is the serialized representation of toggle with regex condition.

 {
   "dashboard_widget":{  
      "toggle":{    

          "conditions":[{
              "type":"Regex",  -- Regex Condition
              "claim":"email", -- Claim 'email' to be used for evaluation.
              "expression":"*@gbk.com" -- Regex expression to be used for evaluation.
           }]         
      }   
   }
 }

C# representation of a feature with regex toggle is


var feature = new Feature
{
  Name ="dashboard_widget",   // Feature Name
  Toggle = new Toggle         // Toggle definition
  {
    Operator = Operator.Any,  
    Conditions = new[] 
    {
        // Regex condition that evalues role of user to be administrator to enable the feature.
        new RegexCondition { Claim = "role", Expression = "administrator" }
    }
  }
}

Step 3. Implement Storage Provider.

To use FeatureOne, you need to provide implementation for Storage Provider to get all the feature toggles from storage medium of choice.
Implement IStorageProvider interface to return feature toggles from storage.
The interface has GetByName() method that returns an array of IFeature

  /// <summary>
  /// Interface to implement storage provider.
  /// </summary>
  public interface IStorageProvider
  {
        /// <summary>
        /// Implement to get storage feature toggles by a given name.
        /// </summary>
        /// <returns>Array of Features</returns>
        IFeature[] GetByName(string name);
  }

A production storage provider should be an implementation with API , SQL or File system storage backend.

An implementation option is to store features as serialized json to backend medium. Ideally, you may also want to use caching in the production implementation to optimise calls to the storage backend.

Below is an example of dummy provider implementation.

public class CustomStoreProvider : IStorageProvider
    {
        public Feature[] GetByName(string name)
        {
            return new[] {
                    new  Feature("feature-01",new Toggle(Operator.Any, new[]{ new SimpleCondition{IsEnabled=true}})),
                    new  Feature("feature-02",new Toggle(Operator.All, new SimpleCondition { IsEnabled = false }, new RegexCondition{Claim="email", Expression= "*@gbk.com" }))
                };
        }
    }

Step 4. Bootstrap Initialialization

In bootstrap code, initialize the Features class with dependencies as shown below.

i. With storage provider implementation.

   var storageProvider = new CustomStorageProviderImpl();
   Features.Initialize(() => new Features(new FeatureStore(storageProvider)));

ii. With storage provider and logger implementations.

   var logger = new CustomLoggerImpl();
   var storageProvider = new CustomStorageProviderImpl();

   Features.Initialize(() => new Features(new FeatureStore(storageProvider, logger), logger));

How to Extend FeatureOne

i. Toggle Condition

You could implement your own condition by extending the ICondition interface.
The interface provides evaluate() method that returns a boolean result of evaluating logic against list of input claims.

    /// <summary>
    /// Interface to implement toggle condition.
    /// </summary>
    public interface ICondition
    {
        /// <summary>
        /// Implement method to evaulate toggle condition.
        /// </summary>
        /// <param name="claims">List of user claims; could be empty</param>
        /// <returns></returns>
        bool Evaluate(IDictionary<string, string> claims);
    }

Example below shows sample implementation of a custom condition.

   // toggle condition to show feature after given hour during the day.
   public class TimeCondition : ICondition
   {
        public int Hour {get; set;} = 12; 

    public bool Evaluate(IDictionary<string, string> claims)
    {
        return (DateTime.Now.Hour > Hour);   
    }
   }

Example usage of above condition in toggle to allow non-admin users access to a feature only after 12 hrs.

C# representation of the feature is

var feature = new Feature
{
  Name ="feature_pen_test",   // Feature Name
  Toggle = new Toggle         // Toggle definition
  {
    Operator = Operator.Any,  // Enabled when one of below conditions are true.
    Conditions = new[]
    {
        // Custom condition - allow access after 12 o'clock
        new TimeCondition { Hour = 12 },
        // Regex condition for allowing admin users by role claim.
        new RegexCondition { Claim = "role", Expression = "^administrator$"}
    }
  }
}

JSON Serialized representation is

  {
    "feature_pen_test":{   
      "toggle":{  
        "operator":"any", -- Any below condition evaluation to true should succeed the toggle.
        "conditions":[{             
               "type":"Time", -- Time condition to allow access after 12 o'clock.
               "Hour":14
        },
        {   
               "type":"Regex", -- Regex to allow admin access
               "claim":"role",
               "expression":"^administrator$"
        }]
    }
  }

Please Note Any custom condition implementation should only include primitive type properties to work with default ICondition deserialization. When you need to implement a much complex toggle condition with non-primitive properties then you need to provide custom implementation of IConditionDeserializer to support its deserialization to toggle condition object.

ii. Logger

You could optionally provide an implementation of a logger by wrapping your favourite logging libaray under IFeatureLogger interface.
Please see the interface definition below.

This implementation is optional and when no logger is provided FeatureOne will not log any errors, warnings or information.

    /// <summary>
    /// Interface to implement custom logger.
    /// </summary>
    public interface IFeatureLogger
    {
        /// <summary>
        /// Implement the debug log method
        /// </summary>
        /// <param name="message">log message</param>
        void Debug(string message);

        /// <summary>
        /// Implement the error log method
        /// </summary>
        /// <param name="message">log message</param>
        /// <param name="message">exception</param>
        void Error(string message, Exception ex = null);

        /// <summary>
        /// Implement the info log method
        /// </summary>
        /// <param name="message">log message</param>
        void Info(string message);

        /// <summary>
        /// Implement the warn log method
        /// </summary>
        /// <param name="message">log message</param>
        void Warn(string message);
    }

FeatureOne.SQL - Feature toggles with SQL Backend.

In addition to all FeatureOne offerings, the FeatureOne.SQL package provides out of box SQL storage provider.

SQL support can easily be installed as a separate nuget package.

$ dotnet add package FeatureOne.SQL --version {latest}

Step 1 - Configure Database Provider

To register a database provider, You need to add the relevant db factory with a specific ProviderName to DbProviderFactories in the bootstrap code.
ie.
DbProviderFactories.RegisterFactory("ProviderName", ProviderFactory)

After adding the provider factory you need to pass the same provider in the connection settings of SQLConfiguration.

Below is the list of most common provider factories yu could configure.

MSSQL - DbProviderFactories.RegisterFactory("System.Data.SqlClient", SqlClientFactory.Instance);

ODBC - DbProviderFactories.RegisterFactory("System.Data.Odbc", OdbcFactory.Instance);

OleDb - DbProviderFactories.RegisterFactory("System.Data.OleDb", OleDbFactory.Instance);

SQLite - DbProviderFactories.RegisterFactory("System.Data.SQLite", SQLiteFactory.Instance);

MySQL - DbProviderFactories.RegisterFactory("MySql.Data.MySqlClient", MySqlClientFactory.Instance);

PostgreSQL - DbProviderFactories.RegisterFactory("Npgsql", NpgsqlFactory.Instance);

STEP 2 - Setup Feature Table (Database)

Requires creating a feature table with columns for feature name, toggle definition and feature archival.

SQL SCRIPT below.

CREATE TABLE TFeatures (
    Id              INT NOT NULL IDENTITY PRIMARY KEY,
    Name            VARCHAR(255) NOT NULL,
    Toggle          NVARCHAR(4000) NOT NULL,
    Archived        BIT CONSTRAINT DF_TFeatures_Archived DEFAULT (0)
);

Example Table Record

Feature toggles need to be scripted to backend database in JSON format.

Please see example entries below.

| Name |Toggle | Archived |
||||
| dashboard_widget |{ "conditions":[{ "type":"Simple", "isEnabled": true }] } | false |
|pen_test_dashboard| { "operator":"any", "conditions":[{ "type":"simple", "isEnabled":false}, { "type":"Regex", "claim":"email","expression":"^[a-zA-Z0-9_.+-]+@gbk.com" }]} | false|

STEP 3 - Bootstrap initialization

See below bootstrap initialization for FeatureOne with MS SQL backend.

SQL Configuration - Set connection string and other settings.

    var sqlConfiguration = new SQLConfiguration
    {
        // provider specific connection settings.
        ConnectionSettings = new ConnectionSettings
        {
            Providername = "System.Data.SqlClient",  -- same provider name as register with db factory.
            ConnectionString ="Data Source=Powerstation; Initial Catalog=Features; Integrated Security=SSPI;"            
        },

        // Table and column name overrides.
        FeatureTable = new FeatureTable
        {
            TableName = "[Features].[dbo].[TFeatures]",  
            NameColumn = "[Name]",
            ToggleColumn = "[Toggle]",
            ArchivedColumn = "[Archived]"
        },

        // Enable cache with absolute expiry in Minutes.
        CacheSettings = new CacheSettings 
        {
            EnableCache = true,  
            Expiry = new CacheExpiry
            {
                InMinutes = 60,
                Type = CacheExpiryType.Absolute
            }
        }
    }

i. With SQL configuration.

   -- Register db factory
   DbProviderFactories.RegisterFactory("System.Data.SqlClient", SqlClientFactory.Instance);

   var storageProvider = new SQlStorageProvider(sqlConfiguration);
   Features.Initialize(() => new Features(new FeatureStore(storageProvider)));

ii. With Custom logger implementation, default is no logger.

    var logger = new CustomLoggerImpl();
    var storageProvider = new SQlStorageProvider(sqlConfiguration, logger);

    Features.Initialize(() => new Features(new FeatureStore(storageProvider, logger), logger));

iii. With other overloads - Custom cache and Toggle Condition deserializer.

    var toggleConditionDeserializer = CustomConditionDeserializerImpl(); // Implements IConditionDeserializer 
    var featureCache = CustomFeatureCache(); // Implements ICache

    var storageProvider = new SQlStorageProvider(sqlConfiguration, featureCache, toggleConditionDeserializer);

    Features.Initialize(() => new Features(new FeatureStore(storageProvider, logger), logger));

FeatureOne.File - Feature toggles with File system Backend.

In addition to all FeatureOne offerings, the FeatureOne.File package provides out of box File storage provider.

File support can easily be installed as a separate nuget package.

$ dotnet add package FeatureOne.File --version {latest}

File Setup

Requires creating a feature file with JSON feature toggles as shown below.

File - Features.json

{
    "gbk_dashboard": {
        "toggle": {
            "operator": "any",
            "conditions": [{
                    "type": "simple",
                    "isEnabled": false
                },
                {
                    "type": "Regex",
                    "claim": "email",
                    "expression": "^[a-zA-Z0-9_.+-]+@gbk.com"
                }
            ]
        }
    },
    "dashboard_widget": {
        "toggle": {
            "conditions": [{
                "type": "simple",
                "isEnabled": true
            }]
        }
    }
}

Bootstrap initialization

See below bootstrap initialization for FeatureOne with SQL backend.

File Configuration - Set file path string and cache settings.

    var configuration = new FileConfiguration
    {
        // Absolute path to the feature file.
        FilePath ="C:\Work\Features.json",

        // Enable cache with absolute expiry in Minutes.
        CacheSettings = new CacheSettings 
        {
            EnableCache = true,  
            Expiry = new CacheExpiry
            {
                InMinutes = 60,
                Type = CacheExpiryType.Absolute
            }
        }
    }

i. With File configuration.

   var storageProvider = new FileStorageProvider(configuration);
   Features.Initialize(() => new Features(new FeatureStore(configuration)));

ii. With Custom logger implementation, default is no logger.

    var logger = new CustomLoggerImpl();
    var storageProvider = new FileStorageProvider(configuration, logger);

    Features.Initialize(() => new Features(new FeatureStore(storageProvider, logger), logger));

iii. With other overloads - Custom cache and Toggle Condition deserializer.

    var toggleConditionDeserializer = CustomConditionDeserializerImpl(); // Implements IConditionDeserializer 
    var featureCache = CustomFeatureCache(); // Implements ICache

    var storageProvider = new FileStorageProvider(configuration, featureCache, toggleConditionDeserializer);

    Features.Initialize(() => new Features(new FeatureStore(storageProvider, logger), logger));

ApiAggregator - API Integration Framework

CØDE N!NJΔ — Sun, 10 Nov 2024 19:22:15 +0000

Introduction

What is ApiAggregator?

ApiAggregator is a .net utility to help combine multiple api requests to return a single aggregated response.
The framework allows fetching the whole of aggregated response or a partial response based on the list of configured apis included in the aggregator request.

When is ApiAggregator useful?

ApiAggregator is useful in many use cases. Few to list are:

For creating Level 2 (functional or BFF) apis using Level 1 (core resource) apis.
For easily extending an api without having to break existing consumers.
For on demand retrieval of data using different subsets of configured apis to fetch varied datasets per request.
and Many more.

Getting Started?

i. Installation

Install the latest version of ApiAggregator nuget package with command below.

NuGet\Install-Package ApiAggregator

ii. Using ApiAggregator

Step 1. Create Aggregated Contract

Aggregated Contract is the resultant response from all the aggregated apis. To create aggregated contract derive the class from IContract interface.

Example.

 public class Customer : IContract
 {
        public int Id { get; set; }
        public string Code { get; set; }
        public string Name { get; set; }
        public Contacts Communication { get; set; }
        public Order[] Orders { get; set; }
}

Step 2. Create Api Aggregated

Api Aggregate is the composition of apis configured to obtain an aggregated data populated contract. To create an Api Aggregate derive from ApiAggregate<TContract> class where TContract is an implementation of IContract (ie. Agggregated Contract).

Example.

internal class CustomerAggregate : ApiAggregate<Customer>
{
        /// <summary>
        /// Constructs the api aggregate with web apis and result transformers to map data to aggregated contract.
        /// </summary>
        /// <returns>Mappings</returns>
        public override IEnumerable<Mapping<Customer, IApiResult>> Construct()
        {
            return CreateAggregate.For<Customer>()
                .Map<CustomerApi, CustomerTransform>(With.Name("customer"),
                 customer => customer.Dependents
                    .Map<CommunicationApi, CommunicationTransform>(With.Name("customer.communication"))
                    .Map<OrdersApi, OrdersTransform>(With.Name("customer.orders"),
                        customerOrders => customerOrders.Dependents
                            .Map<OrderItemsApi, OrderItemsTransform>(With.Name("customer.orders.items")))
                ).Create();
        }
}

Api Aggregate comprises of apis configured in hierarchical nested graph with each api having an associated result transformers. The result from the api is fed to the associated result transformer to map data to the aggregated contract.

2.1 Api & Transformer Pair

Every Api type in the ApiAggregate definition should have a complementing Transformer type.

You need to assign a name to the api/transformer pair. See below rules for api naming convention.

Rules:

You could nest api/transformer pairs in a parent/child hierarchy. In a given parent/child hierarchy, the output of the parent api will serve as the input to the nested api to resolve its api endpoint.
The api/transformer mappings can be nested to 5 levels of dependency.
By convention, The api name should be dot separated string with a dot for every nested level. The name should includes all parent names separated by a dot for a given hierarchy. ie. For a 3 level dependency api mapping the name should be like customer.orders.items

Example.

Below is the snippet from CustomerAggregate definition for parent/child relationship between Customer & Communication apis. The api response from CustomerApi is the input to CommunicationApi for resolving its endpoint url.

   .Map<CustomerApi, CustomerTransform>(With.Name("customer"), -- Parent mapping with name
           customer => customer.Dependents
              .Map<CommunicationApi, CommunicationTransform>(With.Name("customer.communication")) -- nested mapping with dot separated name

i. Web Api Class

The purpose of a api class is to execute the web api with api engine to fetch the response.

As mentioned previously, You can configure an api in Parent or Child (nested) mode in a hierarchical graph.

To create Web Api defined as parent or nested api, you need to implement from WebApi<TResult> class,
where TResult is IApiResult interface (or ApiResult base class) implementation and is the result that will be returned from executing the api.

Upon creating the web api class, you need to provide GetUrl() method implementation to return Uri instance.

Implement the GetUrl(IRequestContext context, IApiResult parentApiResult) method to return the constructed endpoint using given parameters of the method.
For Parent Api, only IRequestContext context parameter is passed to GetUrl() method to resolve the Url endpoint.
For Nested Api, api result parameter (ie. IApiResult parentApiResult) from the parent api is additionally passed in to GetUrl() method along with IRequestContext context parameter.
Optionally, override GetRequestHeaders() method to provide a dictionary of outgoing headers for the api request.
Optionally, override GetResponseHeaders() method to provide any list of incoming headers from the api response.
IApiResult implementation exposes Headers property for subscribed response headers received as part of the api response.

Important:

The api endpoint needs to be resolved before executing the api with ApiEngine.
IApiResult parentApiResult parameter is null for apis configured in parent mode.

Examples.

See example CustomerApi implemented to be configured and run in parent mode.

public class CustomerApi : WebApi<CustomerResult>
{
    public CustomerApi() : base(Endpoints.BaseAddress)
    {
    }

    // Override to construct the api endpoint.
    protected override Uri GetUrl(IRequestContext context, IApiResult parentApiResult)
    {
        // Executes as root or level 1 api. parentApiResult should be null.
        var customerContext = (CustomerContext)context;

        return new Uri(string.Format(Endpoints.Customer, customerContext.CustomerId));
    }

    // Override to pass custom outgoing headers with the api request.
    protected override IDictionary<string, string> GetRequestHeaders()
    {
        return new Dictionary<string, string>
        {
            { "x-meta-branch-code", "Geneva" }
        };
    }

    // Override to get custom incoming headers with the api response.
    protected override IEnumerable<string> GetResponseHeaders()
    {
        return new[] { "x-meta-branch-code" };
    }
}

See example CommunicationApi implemented to be configured and run as nested api to customer api below.

internal class CommunicationApi : WebApi<CommunicationResult>
    {
        public CommunicationApi() : base(Endpoints.BaseAddress)
        {
        }

        protected override Uri GetUrl(IRequestContext context, IApiResult parentApiResult)
        {
            var customer = (CustomerResult)parentApiResult;
            return new Uri(string.Format(Endpoints.Communication, customer.Id));
        }
    }

ii. Result Transformer Class

The purpose of the transformer class is to map the response fetched by the linked api to the aggregated contract.

To define a transformer class, you need to implement ResultTransformer<TResult, TContract> class.

where TContract is Aggregated Contract implementing IContract. eg. Customer.
where TResult is Api Result from associated Query. It is an implementation of IApiResult interface.

Example.

CustomerTransformer is implemented to map CustomerResult recevied from CustomerApi to Customer Aggregated Contract.

public class CustomerTransform : ResultTransformer<CustomerResult, Customer>
    {
        public override void Transform(CustomerResult apiResult, Customer contract)
        {
            var customer = contract ?? new Customer();
            customer.Id = apiResult.Id;
            customer.Name = apiResult.Name;
            customer.Code = apiResult.Code;
        }
    }

Step 3. ApiAggregator Setup

ApiAggregator needs to setup with required dependencies.

i. IoC Registrations

With ServiceCollection, you need to register the below dependencies.

    // Register core services
    services.AddTransient(typeof(IApiBuilder<>), typeof(ApiBuilder<>));
    services.AddTransient(typeof(IContractBuilder<>), typeof(ContractBuilder<>));
    services.AddTransient(typeof(IApiAggregator<>), typeof(ApiAggregator<>));
    services.AddTransient<IApiExecutor, ApiExecutor>();
     services.AddTransient<IApiEnginne, ApiEngine>();

    // Register instance of IApiNameMatcher.
    services.AddTransient(c => new StringColonSeparatedMatcher());

    // Enable logging
    services.AddLogging();

    // Enable HttpClient
     services.AddHttpClient();

    // Register api aggregate definitions. eg CustomerAggregate
    services.AddTransient<IApiAggregate<Customer>, CustomerAggregate>();

ii. With Fluent Registration Api

You could also acheieve the above registrations using fluent registration below.

    // Enable logging
    services.AddLogging();

    // Enable HttpClient
     services.AddHttpClient();

    // Fluent registration.
     services.UseApiAggregator()
             .AddApiAggregate<Customer>(new CustomerAggregate());

Note: Above examples to enable HttpClient is basic. However, you could additionally implement circuit breaking & retry policies.

Please see Circuit Breaker pattern for more details.

Step 4. Use IApiAggregator<`TContract`>

i. IApiAggrgator (DI)

To use Api aggregator, Inject IApiAggrgator where TContract is IContract, using constructor & property injection method or explicity Resolve using service provider

Example. IServiceProvider.GetService(typeof(IApiAggrgator<Customer>))

ii. Call Aggregator.GetData() method

You need to call the GetData() method with an instance of parameter class derived from IRequestContext interface.

The IRequestContext provides a Names property which is a list of string to include all the api names to be included for the given request to fetch aggregated response.
When no names are passed in the paramter then entire aggregated response for all configured apis is returned.
When subset of apis are included using names then the returned aggregated response only includes api responses from included apis.
When nested api with dot separated api name (eg. customer.orders.items) is included then all parent apis also get included for the dependency.

Example - Control Flow

Example execution flow for a nested api included in the GetData() parameter.

Forem: CØDE N!NJΔ

[Boost]

DataFuse.Net - Data Integration Framework.

[Boost]

ActiveForge - Active Record based ORM

[Boost]

DataFuse.Net - Data Integration Framework.

DataFuse.Net - Data Integration Framework.

DataFuse Framework

Table of Contents

Real-World Use Cases

1. E-Commerce Product Page

2. Customer 360 Dashboard

3. Multi-Service Reporting

Why DataFuse?

The Problem

How DataFuse Solves This

DataFuse vs Alternatives

When to Use DataFuse

Quick Start

1. Install Packages

2. Define Your Entity

3. Create Queries

4. Create Transformers

5. Configure the Schema

6. Register with DI

7. Use It

Core Concepts

Entities

Queries

Parent-Child Query Dependencies

Execution Order

Transformers

Schema Configuration

Packages

DataFuse.Integration

DataFuse.Adapters.Abstraction

DataFuse.Adapters.SQL

DataFuse.Adapters.EntityFramework

DataFuse.Adapters.MongoDB

DataFuse.Adapters.WebAPI

Compatibility

Query Implementation Guide

SQL Queries (Dapper)

Entity Framework Queries

MongoDB Queries

Web API Queries

Query Result Types

Transformer Guide

Basic Transformers

Collection Transformers

Context-Aware Transformers

Grandchild Transformers

Advanced Features

Caching

Selective Loading

Parallel Execution

Conditional Query Execution

Cross-Source Dependencies

Custom Query Engines

Path Matching

Architecture

Execution Flow

Component Responsibilities

Configuration Patterns

Best Practices

Query Design

Transformer Design

Schema Design

DI Registration

Migrating from Schemio (v2.x) to DataFuse (v3.0.0)

Step 1: Update NuGet Packages

Step 2: Update Namespaces

Step 3: Update DI Registration

Step 4: Verify

New in v3.0.0

Getting Support

[Boost]

SourceFlow.Net - Event Sourcing Framework. - DEV Community

ActiveForge - Active Record based ORM