Forem: Simo Roikonen

From CRUD to Real-Time Analytics: Why Business Events Matter

Simo Roikonen — Fri, 17 Jan 2025 07:16:22 +0000

Many operational systems (e.g. SaaS platforms and transactional systems) are built around CRUD operations and batch processing, but these approaches often fall short when businesses demand real-time insights.

A common intermediate step is Change Data Capture (CDC), which reflects changes at the database level in near-real time. However, CDC primarily captures raw data changes (e.g., inserts, updates, deletes) and doesn’t provide the context of what those changes mean in the real world. For example, CDC might log an update to a database row, but it doesn’t tell you that the event was actually an “Order Shipped.”

This is where business events come in. Business events are high-quality, domain-specific data — they describe meaningful occurrences like “Payment Processed” or “Customer Registered.” Unlike raw data changes, these events provide the context and structure needed to power real-time analytics and unlock the full potential of AI and machine learning. These events, often called domain events, form the backbone of event-driven systems.

Business/Domain Events in Practice

For reliably generating, publishing and consuming these domain events, patterns like the transactional outbox, subscription, and event sourcing play crucial roles:

Transactional outbox ensures that events are published consistently as part of the same database transaction that writes the underlying data, providing robustness even during failures.

Subscription pattern enables systems to listen for and react to specific events in near real-time, enabling event-driven workflows and processes.

Event sourcing goes further by making events the source of truth, storing all changes as an immutable sequence of events — empowering systems with a full audit history and the ability to reprocess or replay events at any time.

By adopting event-driven architectures and producing event streams, organizations can move beyond traditional paradigms. This approach enables:

Timely, actionable insights through real-time analytics.
A foundation of rich, contextual data for AI and ML applications.
Real-time notifications to trigger actions and inform stakeholders instantly.
Better system design with domain-driven thinking at the core.

Conclusion: Core domains in modern scalable operational systems should embrace business/domain events over Change Data Capture (CDC). Domain-driven design and event-driven architecture pave the way for creating future-proof systems.

Scalable modeling as an event-centric approach provides a strong foundation for building long-lasting operational systems that deliver low latency and enable real-time analytics at any scale.

Scalable Modeling – An Event-centric Approach

Simo Roikonen — Fri, 29 Nov 2024 10:54:18 +0000

After 10 years in software scalability and distributed systems, I’m celebrating by gathering what I’ve learned into a webpage — a modeling technique for scalable systems aptly called ScalableModeling 🤓.

Hope it helps someone out there!

Your organization is perfectly designed to produce the results it gets

Simo Roikonen — Thu, 27 Jun 2024 15:24:03 +0000

Leaders should ensure dev teams maintain high cohesion and low coupling to improve quality and increase overall productivity.

Architects should ensure components maintain high cohesion and low coupling to enhance system maintainability, scalability and reliability.

WHY should you foster high cohesion and low coupling in teams?
🎯 Better Focus: Teams with clear, unified goals (high cohesion) are more productive and effective.
💪 Robust Code: Expertise and focus lead to higher quality, more reliable software, and easier testing.
🤝 Fosters Shared Ownership: Team members are familiar with the codebase and can seamlessly continue work if someone leaves.
🌐 Adaptability: Independent, low-coupled teams can respond quickly to new opportunities and challenges.
🔄 Flexibility: Reduced interdependencies allow for easy reconfiguration to meet new priorities.
💡 Innovation: Autonomy via low coupling fosters creativity and continuous improvement.

If you think about it, your organization is perfectly designed to produce the results it gets. So, could it be that by aligning teams and components, both having high cohesion and low coupling, you can achieve higher quality, better maintainability and faster development velocity? 🤔

Leaders can't do this alone, architects can't do this alone.

Involving architects in organizational design helps to achieve this. This way, leaders can ensure that both the team structure and the software architecture align with the principles of high cohesion and low coupling, driving value creation, speed and agility. 🚀

What strategies, tools, or methods have you found effective in fostering high cohesion and low coupling in your teams?

My two cents:
1️⃣ Domain-Driven Design (DDD): Utilizing DDD principles to structure teams around business domains. This ensures that teams have a deep understanding of the domain they are working in, which enhances cohesion and allows for more effective problem-solving and decision-making.
2️⃣ Team Topologies: Applying the Team Topologies framework to define clear team boundaries and interaction patterns. This helps in designing teams that are aligned with the software architecture and business needs, promoting low coupling and high cohesion.