Forem: Noah

Code Design - Domain Driven Hexagon

Noah — Tue, 02 Mar 2021 15:37:57 +0000

This Is A Post From the Original GitHub Repo

Repo: https://github.com/Sairyss/domain-driven-hexagon

This repo is work in progress

Domain-Driven Hexagon

Main emphasis of this project is to provide recommendations on how to design software applications. In this readme are presented some of the techniques, tools, best practices, architectural patterns and guidelines gathered from different sources.

Everything below should be seen as a recommendation. Keep in mind that different projects have different requirements, so any pattern mentioned in this readme can be replaced or skipped if needed.

Code examples are written using NodeJS, TypeScript, NestJS framework and Typeorm for the database access.

Though patterns and principles presented here are framework/language agnostic, so above technologies can be easily replaced with any alternative. No matter what language or framework is used, any application can benefit from principles described below.

Note: code examples are adapted to TypeScript and mentioned above frameworks so may not fit well for other languages. Also remember that code examples presented here are just examples and must be changed according to project's needs or personal preference.

Architecture
- Diagram
- Modules
- Application Core
- Application layer
- Application Services
- Commands and Queries
- Ports
- Domain Layer
- Entities
- Aggregates
- Domain Events
- Domain Services
- Value Objects
- Enforcing invariants of Domain Objects
- Using libraries inside application's core
- Interface Adapters
- Controllers
- DTOs
- Infrastructure
- Adapters
- Repositories
- ORM Entities
- Other things that can be a part of Infrastructure layer
- Recommendations for smaller APIs
Other recommendations and best practices
- Error Handling
- Testing
- Configuration
- Logging
- Health monitoring
- Folder and File Structure
- File names
- Static Code Analysis
- Code formatting
- Documentation
- Make application easy to setup
- Seeds
- Migrations
- Rate Limiting
- Code Generation
- Custom utility types
- Pre-push/pre-commit hooks
- Prevent massive inheritance chains
- Conventional commits
Additional resources
- Articles
- Repositories
- Documentation
- Blogs
- Videos
- Books

Architecture

Mainly based on:

And many other sources (more links below in every chapter).

Before we begin, here are the PROS and CONS of using a complete architecture like this:

Pros:

Independent of external frameworks, technologies, databases, etc. Frameworks and external resources can be plugged/unplugged with much less effort.
Easily testable and scalable.
More secure. Some security principles are baked in design itself.
The solution can be worked on and maintained by different teams, without stepping on each other's toes.
Easier to add new features. As the system grows over time, the difficulty in adding new features remains constant and relatively small.
If the solution is properly broken apart along bounded context lines, it becomes easy to convert pieces of it into microservices if needed.

Cons:

This is a sophisticated architecture which requires a firm understanding of quality software principles, such as SOLID, Clean/Hexagonal Architecture, Domain-Driven Design, etc. Any team implementing such a solution will almost certainly require an expert to drive the solution and keep it from evolving the wrong way and accumulating technical debt.
Some of the practices presented here are not recommended for small-medium sized applications with not a lot of business logic. There is added up-front complexity to support all those building blocks and layers, boilerplate code, abstractions, data mapping etc. thus implementing a complete architecture like this is generally ill-suited to simple CRUD applications and could over-complicate such solutions. Some of the described below principles can be used in a smaller sized applications but must be implemented only after analyzing and understanding all pros and cons.

Diagram

^{Diagram is mostly based on this one + others found online}

In short, data flow looks like this (from left to right):

Request/CLI command/event is sent to the controller using plain DTO;
Controller parses this DTO, maps it to a Command/Query object format and passes it to a Application service;
Application service handles this Command/Query; it executes business logic using domain services and/or entities and uses the infrastructure layer through ports;
Infrastructure layer maps data to format that it needs, uses repositories to fetch/persist data and adapters to send events or do other I/O communications, maps data back to domain format and returns it back to Application service;
After application service finishes doing it's job, it returns data/confirmation back to Controllers;
Controllers return data back to the user (if application has presenters/views, those are returned instead).

Keep in mind that different projects can have more or less steps/layers/building blocks than described here. Add more if application requires it, and skip some if application is not that complex and doesn't need all that abstraction.

General recommendation for any project: analyze how big/complex the application will be, find a compromise and use as many layers/building blocks as needed for the project and skip ones that may over-complicate things.

Modules

This project's code examples use separation by modules (also called components). Each module gets its own folder with a dedicated codebase, and each use case inside that module gets it's own folder to store most of the things it needs (this is also called Vertical Slicing).

It is easier to work on things that change together if those things are gathered relatively close to each other. Try not to create dependencies between modules or use cases, move shared logic into a separate files and make both depend on that instead of depending on each other.

Try to make every module independent and keep interactions between modules minimal. Think of each module as a mini application bounded by a single context. Try to avoid direct imports between modules (like importing a service from other domain) since this creates tight coupling. Communication between modules can be done using events, public interfaces or through a port/adapter (more on that topic below).

This approach ensures loose coupling, and, if bounded contexts are defined and designed properly, each module can be easily separated into a microservice if needed without touching any domain logic.

A lof of people tend to create one module per entity, but this approach is not very good. Each module may have multiple entities. One thing to keep in mind is that putting entities in a single module requires those entities to have related business logic, don't group unrelated entities in one module.

Read more about modular programming benefits:

Modular programming: Beyond the spaghetti mess.

Each module is separated in layers described below.

Application Core

This is the core of the system which is built using DDD building blocks.

Dependencies point inwards. Outer layers can depend on inner layers, but inner layers never depend on outer layers.

Core layers shouldn't depend on frameworks or access external resources. Any external calls to out-of-process resources/retrieval of data from remote processes should be done through ports (interfaces), with class implementations created somewhere in infrastructure layer and injected into application's core (Dependency Injection and Dependency Inversion).

Application's Core consists of:

Domain layer:

Entities
Aggregates
Domain Services
Value Objects

Application layer:

Application Services
Commands and Queries
Ports

This is just a short list the main things that may reside in here. More building blocks may be added if needed.

Application layer

Application Services

Are also called "Workflow Services", "User Cases", "Interactors" etc.
These services orchestrate the steps required to fulfill the commands imposed by the client.

Typically used to orchestrate how the outside world interacts with your application and performs tasks required by the end users.
Contain no domain-specific business logic;
Operate on scalar types, transforming them into Domain types. A scalar type can be considered any type that's unknown to the Domain Model. This includes primitive types and types that don't belong to the Domain.
Application services declare dependencies on infrastructural services required to execute domain logic (by using ports).
Are used in order to fetch domain Entities (or anything else) from database/outside world through ports;
Execute other out-of-process communications through Ports (like event emits, sending emails etc);
In case of interacting with one Entity/Aggregate, executes its methods directly;
In case of working with multiple Entities/Aggregates, uses a Domain Service to orchestrate them;
Are basically a Command/Query handlers;
Should not depend on other application services since it may cause problems (like cyclic dependencies);

One service per use case is considered a good practice.

What are "Use Cases"?

wiki:

> In software and systems engineering, a use case is a list of actions or event steps typically defining the interactions between a role (known in the Unified Modeling Language as an actor) and a system to achieve a goal.

Use cases are, simply said, list of actions required from an application.

Example file: create-user.service.ts

More about services:

Notes: Interfaces for each Use Case and Local DTOs

Interfaces for each use case

Some people prefer having an interface for each use case (Driving Port), which Application Service implements and a Controller depends on. This is a viable option, but this project doesn't use interfaces for every use case for simplicity: it makes sense using interfaces when there are multiple implementations of a workflow, but use cases are too specific and should not have multiple implementations of the same workflow (one service per use case rule mentioned above). Controllers naturally depend on a concrete implementation thus making interfaces redundant. More on this topic here.

Local DTOs

Another thing that can be seen in some projects is local DTOs. Some people prefer never use domain objects (like entities) outside of core (in controllers, for example), and are using DTOs instead. This project doesn't use this technique to avoid extra interfaces and data mapping. Either to use local DTOs or not is a matter of taste.

Here are Martin Fowler's thoughts on local DTOs, in short (quote):

> Some people argue for them(DTOs) as part of a Service Layer API because they ensure that service layer clients aren't dependent upon an underlying Domain Model. While that may be handy, I don't think it's worth the cost of all of that data mapping.

Commands and Queries

This principle is called Command–Query Separation(CQS). When possible, methods should be separated into Commands (state-changing operations) and Queries (data-retrieval operations). To make a clear distinction between those two types of operations, input objects can be represented as Commands and Queries. Before DTO reaches the domain, it is converted into a Command/Query object.

Commands

Commands are used for state-changing actions, like creating new user and saving it to the database. Create, Update and Delete operations are considered as state-changing.

Data retrieval is responsibility of Queries, so Command methods should not return anything. Though, if needed, returning a bare minimum (like ID of a created item or a confirmation message) may not be a bad idea.

Note: Command has nothing to do with Command Pattern, it is just a convenient name to represent that this object invokes a CQS Command. Both Commands and Queries in this example are just simple objects with data.

Example of command object: create-user.command.ts

Queries

Query is used for retrieving data and should not make any state changes (like writes to the database, files etc).

Queries are usually just a data retrieval operation and have no business logic involved; so, if needed, application and domain layers can be bypassed completely. Though, if some additional non-state changing logic has to be applied before returning a query response (like calculating something), it should be done in a corresponding application service.

Validation also can be skipped, since no input is persisted in query operations. But, if needed, it can be validated to tell the user that query format is incorrect (when using enums for example).

Example of query bypassing application/domain layers completely: find-user-by-email.http.controller.ts

Note: Some simple cases may not need a Command/Query object, like find query or delete command may only need an ID so there is no point in creating an object for that.

Ports

Ports (for Driven Adapters) are interfaces that define contracts which must be implemented by infrastructure adapters in order to execute some action more related to technology details rather than business logic. Ports act like abstractions for technology details that business logic does not care about.

Ports are basically just interfaces that define what has to be done and don't care about how it is done.
Ports should be created to fit the Domain needs, not simply mimic the tools APIs.
Mock implementations can be passed to ports while testing. Mocking makes your tests faster and independent from the environment.

Note: since most ports implementations are injected and executed in application service, Application Layer can be a good place to keep those ports. But there are times when Domain Layer's business logic depends on executing some external resource, in that case those ports can be put in a Domain Layer.

Example file: repository.ports.ts

Domain Layer

This layer contains application's business rules.

Domain should only operate using domain objects, most important ones are described below.

Entities

Entities are the core of the domain. They encapsulate Enterprise wide business rules and attributes. An entity can be an object with properties and methods, or it can be a set of data structures and functions.

Entities represent business models and express what properties a particular model has, what it can do, when and at what conditions it can do it. An example of business model can be a User, Product, Booking, Ticket, Wallet etc.

Entities must always protect it's invariant:

Domain entities should always be valid entities. There are a certain number of invariants for an object that should always be true. For example, an order item object always has to have a quantity that must be a positive integer, plus an article name and price. Therefore, invariants enforcement is the responsibility of the domain entities (especially of the aggregate root) and an entity object should not be able to exist without being valid.

Entities:

Contain Domain business logic. Avoid having business logic in your services when possible, this leads to Anemic Domain Model (domain services are exception for business logic that can't be put in a single entity).
Have an identity that defines it and makes it distinguishable from others. It's identity is consistent during its life cycle.
Equality between two entities is determined by comparing their identificators (usually its id field).
Can contain other objects, such as other entities or value objects.
Are responsible for collecting all the understanding of state and how it changes in the same place.
Responsible for the coordination of operations on the objects it owns.
Know nothing about upper layers (services, controllers etc).
Domain entities data should be modelled to accommodate business logic, not some database schema.
Entities must protect their invariants, try to avoid public setters - update state using methods and execute invariant validation on each update if needed (this can be a simple validate() method that checks if business rules are not violated by update).
Must be consistent on creation. Validate Entities and other domain objects on creation and throw an error on first failure. Fail Fast.
Avoid no-arg (empty) constructors, accept and validate all required properties through a constructor.
For optional properties that require some complex setting up, Fluent interface and Builder Pattern can be used.
Make Entities partially immutable. Identify what properties shouldn't change after creation and make them readonly (for example id or createdAt).

Example files:

user.entity.ts

Aggregates

Aggregate is a cluster of domain objects that can be treated as a single unit. It encapsulates entities and value objects which conceptually belong together. It also contains a set of operations which those domain objects can be operated on.

Aggregates help to simplify the domain model by gathering multiple domain objects under a single abstraction.
Aggregates should not be influenced by data model. Associations between domain objects are not the same as database relationships.
Aggregate root is an entity that contains other entities/value objects and all logic to operate them.
Aggregate root has global identity. Entities inside the boundary have local identity, unique only within the Aggregate.
Aggregate root is a gateway to entire aggregate. Any references from outside the aggregate should only go to the aggregate root.
Any operations on an aggregate must be transactional operations. Either everything gets saved/updated/deleted or nothing.
Only Aggregate Roots can be obtained directly with database queries. Everything else must be done through traversal.
Similar to Entities, aggregates must protect their invariants through entire lifecycle. When a change to any object within the Aggregate boundary is committed, all invariants of the whole Aggregate must be satisfied.
Objects within the Aggregate can hold references to other Aggregate roots. Prefer references to external aggregates only by their globally unique identity, not by holding a direct object reference.
Try to avoid aggregates that are too big, this can lead to performance and maintaining problems.
Aggregates can publish Domain Events (more on that below).

All of this rules just come from the idea of creating a boundary around Aggregates. The boundary simplifies business model, as it forces us to consider each relationship very carefully, and within a well-defined set of rules.

Example files:

aggregate-root.base.ts - abstract base class.
user.entity.ts - aggregate implementations are similar to Entities, with some additional rules described above.

Understanding Aggregates in Domain-Driven Design
What Are Aggregates In Domain-Driven Design? <- this is a series of multiple articles, don't forget to click "Next article" at the end.
Effective Aggregate Design Part I: Modeling a Single Aggregate
Effective Aggregate Design Part II: Making Aggregates Work Together

Domain Events

Domain event indicates that something happened in a domain that you want other parts of the same domain (in-process) to be aware of.

For example, if a user buys something, you may want to:

Send confirmation email to that user;
Send notification to a corporate slack channel;
Notify shipping department;
Perform other side effects that are not concern of an original buy operation domain.

Typical approach that is usually used involves executing all this logic in a service that performs a buy operation. But this creates coupling between different subdomains.

A better approach would be publishing a Domain Event. Any side effect operations can be performed just by subscribing to a concrete Domain Event and creating as many event handlers as needed, without glueing any unrelated code to original domain's service that sends an event.

Domain events are just messages pushed to a domain event dispatcher in the same process. Out-of-process communications (like microservices) are called Integration Events. If sending a Domain Event to external process is needed then domain event handler should send an Integration Event.

Domain Events may be useful for creating an audit log to track all changes to important entities by saving each event to the database. Read more on why audit logs may be useful: Why soft deletes are evil and what to do instead.

There may be different ways on implementing Domain Events, for example using some kind of internal event bus/emitter, like Event Emitter, or using patterns like Mediator or slightly modified Observer.

Examples:

domain-events.ts - this class is responsible for providing publish/subscribe functionality for anyone who needs to emit or listen to events.
user-created.domain-event.ts - simple object that holds data related to published event.
user-created.event-handler.ts - this is an example of Domain Event Handler that executes actions and side-effects when a domain event is raised (in this case, user is created). Domain event handlers belong to Application layer.
typeorm.repository.base.ts - repository publishes all events for execution right before or right after persisting transaction.

Events can be published right before or right after insert/update/delete transaction, chose any option that is better for a particular project:

Before: to make side-effects part of that transaction. If any event fails all changes should be reverted.
After: to persist transaction even if some event fails. This makes side-effects independent, but in that case eventual consistency should be implemented.

Both options have pros and cons.

Note: this project uses custom implementation for Domain Events. Reason for not using Node Event Emitter is that event emitter executes events immediately when called instead of when we want it (before/after transaction), and also has no option to await for all events to finish, which might be useful when making those events a part of transaction.

To have a better understanding on domain events and implementation read this:

For integration events in distributed systems here are some useful patterns:

Domain Services

Eric Evans, Domain-Driven Design:

Domain services are used for "a significant process or transformation in the domain that is not a natural responsibility of an ENTITY or VALUE OBJECT"

Domain Service is a specific type of domain layer class that is used to execute domain logic that relies on two or more Entities.
Domain Services are used when putting the logic on a particular Entity would break encapsulation and require the Entity to know about things it really shouldn't be concerned with.
Domain services are very granular where as application services are a facade purposed with providing an API.
Domain services operate only on types belonging to the Domain. They contain meaningful concepts that can be found within the Ubiquitous Language. They hold operations that don't fit well into Value Objects or Entities.

Value objects

Some Attributes and behaviors can be moved out of the entity itself and put into Value Objects.

Value Objects:

Have no identity. Equality is determined through structural property.
Are immutable.
Can be used as an attribute of entities and other value objects.
Explicitly defines and enforces important constraints (invariants).

Value object shouldn’t be just a convenient grouping of attributes but should form a well-defined concept in the domain model. This is true even if it contains only one attribute. When modeled as a conceptual whole, it carries meaning when passed around, and it can uphold its constraints.

Imagine you have a User entity which needs to have an address of a user. Usually an address is simply a complex value that has no identity in the domain and is composed of multiple other values, like country, street, postalCode etc; so it can be modeled and treated as a Value Object with it's own business logic.

Value object isn’t just a data structure that holds values. It can also encapsulate logic associated with the concept it represents.

Example files:

address.value-object.ts

Enforcing invariants of Domain Objects

Replacing primitives with Value Objects

Most of the code bases operate on primitive types – strings, numbers etc. In the Domain Model, this level of abstraction may be too low.

Significant business concepts can be expressed using specific types and classes. Value Objects can be used instead primitives to avoid primitives obsession.
So, for example, email of type string:



email: string;

could be represented as a Value Object instead:



email: Email;

Now the only way to make an email is to create a new instance of Email class first, this ensures it will be validated on creation and a wrong value won't get into Entities.

Also an important behavior of the domain primitive is encapsulated in one place. By having the domain primitive own and control domain operations, you reduce the risk of bugs caused by lack of detailed domain knowledge of the concepts involved in the operation.

Creating an object for primitive values may be cumbersome, but it somewhat forces a developer to study domain more in details instead of just throwing a primitive type without even thinking what that value represents in domain.

Using Value Objects for primitive types is also called a domain primitive. The concept and naming are proposed in the book "Secure by Design".

Using Value Objects instead of primitives:

Makes code easier to understand by using ubiquitous language instead of just string.
Improves security by ensuring invariants of every property.
Encapsulates specific business rules associated with a value.

Also an alternative for creating an object may be a type alias just to give this primitive a semantic meaning.

Example files:

email.value-object.ts

At compile time

Types give useful semantic information to a developer. Good code should be easy to use correctly, and hard to use incorrectly. Types system can be a good help for that. It can prevent some nasty errors at a compile time, so IDE will show type errors right away.

The simplest example may be using enums instead of constants, and use those enums as input type for something. When passing anything that is not intended IDE will show a type error.

Or, for example, imagine that business logic requires to have contact info of a person by either having email, or phone, or both. Both email and phone could be represented as optional, for example:



interface ContactInfo {
  email?: Email;
  phone?: Phone;
}

But what happens if both are not provided by a programmer? Business rule violated. Illegal state allowed.

Solution: this could be presented as a union type



type ContactInfo = Email | Phone | [Email, Phone];

Now only either Email, or Phone, or both must be provided. If nothing is provided IDE will show a type error right away. This is a business rule validation used at compile time.

This approach can be used to make business logic safer and get an error as fast as possible (at compile time).

At runtime

Things that can't be validated at compile time (like user input) are validated at runtime.

Domain objects have to protect their invariants. Having some validation rules here will protect their state from corruption.

Value Object can represent a typed value in domain (a domain primitive). The goal here is to encapsulate validations and business logic related only to the represented fields and make it impossible to pass around raw values by forcing a creation of valid Value Objects first. This object only accepts values which make sense in its context.

If every argument and return value of a method is valid by definition, you’ll have input and output validation in every single method in your codebase without any extra effort. This will make application more resilient to errors and will protect it from a whole class of bugs and security vulnerabilities caused by invalid input data.

Data should not be trusted. There are a lot of cases when invalid data may end up in a domain. For example, if data comes from external API, database, or if it's just a programmer error.

Enforcing self-validation will inform immediately when data is corrupted. Not validating domain objects allows them to be in an incorrect state, this leads to problems.

Without domain primitives, the remaining code needs to take care of validation, formatting, comparing, and lots of other details. Entities represent long-lived objects with a distinguished identity, such as articles in a news feed, rooms in a hotel, and shopping carts in online sales. The functionality in a system often centers around changing the state of these objects: hotel rooms are booked, shopping cart contents are
paid for, and so on. Sooner or later the flow of control will be guided to some code representing these entities. And if all the data is transmitted as generic types such as int or String , responsibilities fall on the entity code to validate, compare, and format the data, among other tasks. The entity code will be burdened with a lot of
tasks, rather than focusing on the central business flow-of-state changes that it models. Using domain primitives can counteract the tendency for entities to grow overly complex.

Quote from: Secure by design: Chapter 5.3 Standing on the shoulders of domain primitives

Note: Though primitive obsession is a code smell, some people consider making a class/object for every primitive may be an overengineering. For less complex and smaller projects it definitely may be. For bigger projects, there are people who advocate for and against this approach. If creating a class for every primitive is not preferred, create classes just for those primitives that have specific rules or behavior, or just validate only outside of domain using some validation framework. Here are some thoughts on this topic: From Primitive Obsession to Domain Modelling - Over-engineering?.

Recommended to read:

Making illegal states unrepresentable
Domain Primitives: what they are and how you can use them to make more secure software
"Secure by Design" Chapter 5: Domain Primitives (a full chapter of the article above)

How to do simple validation?

For simple validation like checking for nulls, empty arrays, input length etc. a library of guards can be created.

Example file: guard.ts

Types of validation

There are some general recommendations for validation order. Cheap operations like checking for null/undefined and checking length of data come early in the list, and more expensive operations that require calling the database come later.

Preferably in this order:

Origin - Is the data from a legitimate sender? When possible, accept data only from authorized users / whitelisted IPs etc. depending on the situation.
Existence - are provided data not empty? Further validations make no sense if data is empty. Check for empty values: null/undefined, empty objects and arrays.
Size - Is it reasonably big? Before any further steps, check length/size of input data, no matter what type it is. This will prevent validating data that is too big which may block a thread entirely (sending data that is too big may be a DDoS attack).
Lexical content - Does it contain the right characters and encoding? For example, if we expect data that only contains digits, we scan it to see if there’s anything else. If we find anything else, we draw the conclusion that the data is either broken by mistake or has been maliciously crafted to fool our system.
Syntax - Is the format right? Check if data format is right. Sometimes checking syntax is as simple as using a regexp, or it may be more complex like parsing a XML or JSON.
Semantics - Does the data make sense? Check data in connection with the rest of the system (like database, other processes etc). For example, checking in a database if ID of item exists.

Read more about validation types described above:

"Secure by Design" Chapter 4.3: Validation.

Using libraries inside application's core

Whether or not to use libraries in application layer and especially domain layer is a subject of a lot of debates. In real world, injecting every library instead of importing it directly is not always practical, so exceptions can be made for some single responsibility libraries that help to implement domain logic (like number converting libraries etc).

Main recommendations to keep in mind is that libraries imported in application's core/domain shouldn't expose:

Functionality to access any out-of-process resources (http calls, database access etc);
Functionality not relevant to domain (frameworks, technology details like ORMs etc).
Functionality that brings randomness (generating random IDs, timestamps etc) since this makes tests unpredictable (though in TypeScript world it is not that big of a deal since this can be mocked by a test library without using DI);
If a library changes often or has a lot of dependencies of its own it most likely shouldn't be used in domain layer.

To use such libraries consider creating an anti-corruption layer by using adapter or facade patterns.

Anti-corruption Layer — An effective Shield

Be careful with general purpose libraries/frameworks that may scatter across many domain objects. It will be hard to replace those libraries if needed.

Tying only one or just few domain objects to some single-responsibility library should be fine. It is way easier to replace a specific library that is tied to one or few objects than a general purpose library that is everywhere.

Offload as much of irrelevant responsibilities as possible from the core, especially from domain layer.

Referencing external libs.

Interface Adapters

Interface adapters (also called driving/primary adapters) are user-facing interfaces that take input data from the user and repackage it in a form that is convenient for the use cases(services) and entities. Then they take the output from those use cases and entities and repackage it in a form that is convenient for displaying it back for the user. User can be either a person using an application or another server.

Contains Controllers and Request/Response DTOs (can also contain Views, like backend-generated HTML templates, if required).

Controllers

Controllers are used for parsing requests, triggering use cases and presenting the result back to the client.
One controller per use case is considered a good practice.
In NestJS world controllers may be a good place to use OpenAPI/Swagger decorators for documentation.

One controller per trigger type can be used to have a more clear separation. For example:

create-user.http.controller.ts for http requests (NestJS Controllers),
create-user.cli.controller.ts for command line interface access (NestJS Console)
create-user.event.controller.ts for external events (NetJS Microservices).
etc.

DTOs

Data Transfer Object (DTO) is an object that carries data between processes. This is a contract between your API and clients.

Request DTOs

Input data sent by a user.

Using Request DTOs gives a contract that a client of your API has to follow to make a correct request.

Examples:

Response DTOs

Output data returned to a user.

Using Response DTOs ensures clients only receive data described in DTOs contract, not everything that your model/entity owns (which may result in data leaks).
It also to some extent protects your clients from internal data structure changes that may happen in your API.

Examples:

Additional recommendations:

When returning a Response prefer whitelisting properties over blacklisting. This ensures that no sensitive data will leak in case if programmer forgets to blacklist newly added properties that shouldn't be returned to the user.
Interfaces for Request/Response objects should be kept somewhere in shared directory instead of module directory since they may be used by a different application (like front-end page, mobile app or microservice). Consider creating git submodule or a separate package for sharing interfaces.
Request/Response DTO classes may be a good place to use validation and sanitization decorators like class-validator and class-sanitizer (make sure that all validation errors are gathered first and only then return them to the user, this is called Notification pattern. Class-validator does this by default).
Request/Response DTO classes may also be a good place to use Swagger/OpenAPI library decorators that NestJS provides.
If DTO decorators for validation/documentation are not used, DTO can be just an interface instead of class + interface.
Data can be transformed to DTO format using a separate mapper or right in the constructor if DTO classes are used.

Infrastructure

The Infrastructure is responsible strictly to keep technology. You can find there the implementations of database repositories for business entities, message brokers, I/O components, dependency injection, frameworks and any other thing that represents a detail for the architecture, mostly framework dependent, external dependencies, and so on.

It's the most volatile layer. Since the things in this layer are so likely to change, they are kept as far away as possible from the more stable domain layers. Because they are kept separate, it's relatively easy make changes or swap one component for another.

Infrastructure layer can contain Adapters, database related files like Repositories, ORM entities/Schemas, framework related files etc.

Adapters

Infrastructure adapters (also called driven/secondary adapters) enable a software system to interact with external systems by receiving, storing and providing data when requested (like persistence, message brokers, sending emails or messages, requesting 3rd party APIs etc).
Adapters also can be used to interact with different domains inside single process to avoid coupling between those domains.
Adapters are essentially an implementation of ports. They are not supposed to be called directly in any point in code, only through ports(interfaces).
Adapters can be used as Anti-Corruption Layer (ACL) for legacy code.

Adapters should have:

a port somewhere in application/domain layer that it implements;
a mapper that maps data from and to domain (if it's needed);
a DTO/interface for received data;
a validator to make sure incoming data is not corrupted (validation can reside in DTO class using decorators, or it can be validated by Value Objects).

Repositories

Repositories centralize common data access functionality. They encapsulate the logic required to access that data. Entities/aggregates can be put into a repository and then retrieved at a later time without domain even knowing where data is saved, in a database, or a file, or some other source.

We use repositories to decouple the infrastructure or technology used to access databases from the domain model layer.

Martin Fowler describes a repository as follows:

A repository performs the tasks of an intermediary between the domain model layers and data mapping, acting in a similar way to a set of domain objects in memory. Client objects declaratively build queries and send them to the repositories for answers. Conceptually, a repository encapsulates a set of objects stored in the database and operations that can be performed on them, providing a way that is closer to the persistence layer. Repositories, also, support the purpose of separating, clearly and in one direction, the dependency between the work domain and the data allocation or mapping.

The data flow here looks something like this: repository receives a domain Entity from application service, maps it to database schema/ORM format, does required operations and maps it back to domain Entity and returns it back to service.

Keep in mind that application's core is not allowed to depend on repositories directly, instead it depends on abstractions (ports/interfaces). This makes data retrieval technology-agnostic.

Examples

This project contains abstract repository class that allows to make basic CRUD operations: typeorm.repository.base.ts. This base class is then extended by a specific repository, and all specific operations that an entity may need is implemented in that specific repo: user.repository.ts.

ORM Entities

Using a single entity for domain logic and database concerns leads to a database-centric architecture. In DDD world domain model and persistance model should be separated. If ORM frameworks are used, ORM Entities can be created to represent domain entities in a database.

Since domain Entities have their data modeled so that it best accommodates domain logic, it may be not in the best shape to save in database. For that purpose ORM Entities (or Schemas) are used that have shape that is better represented in a particular database that is used.

This approach can also be useful when amount of data in database grows and there is a need to improve performance, for example by doing a re-design of some tables, database normalization, or even changing the database entirely. Without an explicit separation between Entities and ORM Entities/Schemas any change to the database will lead to change in your Entities, since data can spread across multiple tables rather than being in one table. This may force a team to do a complete refactoring of a domain layer which may cause unexpected bugs and challenges.

Note: separating Entities and ORM Entities may be an overkill for smaller applications, consider all pros and cons before making this decision.

Example files:

user.orm-entity.ts
user.orm-mapper.ts <- ORM Entities should also have a corresponding mapper to map from domain to persistence and back.

Framework related files;
Application logger implementation;
Infrastructure related events (Nest-event)
Periodic cron jobs or tasks launchers (NestJS Schedule);
Other technology related files.

Recommendations for smaller APIs

Be careful when implementing any complex architecture in small-medium sized projects with not a lot of business logic. Some of the building blocks/patterns may fit well, but others may be an overengineering.

For example:

Separating code into modules/layers/use-cases, using some building blocks like controllers/services/entities, respecting boundaries and dependency injections etc. may be a good idea for any project.
But practices like creating an object for every primitive, using Value Objects to separate business logic into smaller classes, dividing Entities and ORM Entities etc. in projects that are more data-centric and have little or no business logic may only complicate such solutions and add extra boilerplate code, data mapping etc. without adding much benefit.

Some principles/patterns can be implemented in a simplified form, some can be skipped. Follow YAGNI principle and don't overengineer.

Before implementing any pattern always analyze if benefit given by using it worth extra code complexity.

Effective design argues that we need to know the price of a pattern is worth paying - that's its own skill.

However, remember:

It's easier to refactor over-design than it is to refactor no design.

Martin Fowler blog: Yagni

Other recommendations and best practices

Error Handling

Exception types

Consider extending Error object to make custom exception types for different situations. For example: DomainException etc. This is especially relevant in NodeJS world since there is no exceptions for different situations by default.

Keep in mind that application's core shouldn't throw HTTP exceptions or statuses since it shouldn't know in what context it is used, since it can be used by anything: HTTP controller, Microservice event handler, Command Line Interface etc.

When used in HTTP context, for returning proper status code back to user an instanceof check can be performed in exception interceptor or in a controller and appropriate HTTP exception can be returned depending on exception type.

Exception interceptor example: exception.interceptor.ts - notice how custom exceptions are converted to nest.js exceptions.

Adding a name string with type name for every exception is a good practice, since when that exception is transferred to another process instanceof check cannot be performed anymore so a name string is used instead. Exception name enum types can be stored in a separate file so they can be reused on a receiving side: exception.types.ts.

When using microservices, all exception types can be packed into a library and reused in each microservice for consistency.

Differentiate between programmer errors and operational errors

Application should be protected not only from operational errors (like incorrect user input), but from a programmer errors as well by throwing exceptions when something is not used as intended.

For example:

Operational errors can happen when validation error is thrown by validating user input, it means that input body is incorrect and a 400 Bad Request exception should be returned to the user with details of what fields are incorrect (notification pattern). In this case user can fix the input body and retry the request.
On the other hand, programmer error means something unexpected occurs in the program. For example, when exception happens on a new domain object creation, sometimes it can mean that a class is not used as intended and some rule is violated, for example a programmer did a mistake by assigning an incorrect value to a constructor, or value got mutated at some point and is no longer valid. In this case user cannot do anything to fix this, only a programmer can, so it may be more appropriate to throw a different type of exception that should be logged and then returned to the user as 500 Internal Server Error, in this case without adding much additional details to the response since it may cause a leak of some sensitive data.

Error Serialization

By default, in NodeJS Error objects serialize to JSON with output like this:



{
  name: 'ValidationError';
}

Consider serializing errors by creating a toJSON() method so it can be easily sent to other processes as a plain object.

Error metadata

Consider adding optional metadata object to exceptions (if language doesn't support anything similar by default) and pass some useful technical information about the error when throwing. This will make debugging easier.

Important to keep in mind: never log or add to metadata any sensitive information (like passwords, emails, phone or credit card numbers etc) since this information may leak into log files, and if log files are not protected properly this information can leak or be seen by developers who have access to log files. Aim adding only technical information to your logs.

Other recommendations

If translations of error messages to other languages is needed, consider storing those error messages in a separate object/class rather than inline string literals. This will make it easier to implement localization by adding conditional getters.

Example files:

exception.base.ts - Exception abstract base class
domain.exception.ts - Domain Exception class example
Check exceptions folder to see more examples (some of them are exceptions from other languages like C# or Java)

Alternatives to exceptions

There is an alternative approach of not throwing exceptions, but returning some kind of Result object type with a Success or a Failure (an Either monad from functional languages like Haskell). Unlike throwing exceptions, this approach allows to define types for exceptional outcomes and will force us to handle those cases explicitly instead of using try/catch. For example:



class User {
  // ...
  public createUser(): Either<User, EmailInvalidException> {
    // ...code for creating a user
    if (invalidEmail) {
      return EmailInvalidException; // <- returning instead of throwing
    }
    return User;
  }
}

This approach has its advantages and may work nicely in some languages, especially in functional languages which support Either type natively, but is not widely used in TypeScript/Javascript world.

It also has some downsides:

It goes against Fail-fast principle. Instead of terminating a program flow, this approach continues program execution and allows it to run in an incorrect state, which may lead to more unexpected errors.
It adds extra complexity. Exception cases returned somewhere deep inside application have to be handled by functions in upper layers until it reaches controllers which may add a lot of extra if statements.

For most projects this approach may be an overkill. Use it only if you really need it and know what you are doing.

"Secure by Design" Chapter 9.2: Handling failures without exceptions

Testing

Software Testing helps catching bugs early. Properly tested software product ensures reliability, security and high performance which further results in time saving, cost effectiveness and customer satisfaction.

Lets review two types of software testing:

White Box testing.
Black Box testing.

Testing module/use-case internal structures (creating a test for every file/class) is called White Box testing. White Box testing is widely used technique, but it has disadvantages. It creates coupling to implementation details, so every time you decide to refactor business logic code this may also cause a refactoring of corresponding tests.

To solve this and get the most out of your tests, prefer Black Box testing (also called Behavioral Testing). This means that tests should focus on testing user-facing behavior users care about (your code's public API, for example createUser() method in Application Service), not the implementation details of individual units it has inside. This avoids coupling, protects tests from changes that may happen while refactoring, makes tests easier to understand and maintain thus saving time.

Tests that are independent of implementation details are easier to maintain since they don't need to be changed each time you make a change to the implementation.

Try to avoid White Box testing when possible. Though, there are cases when White Box testing may be needed, for example:

High complexity in implementation details that are hard to cover using Black Box testing.
There is a need to increase code coverage.
There is a suspicion that some area of the program may be under-tested.
Sometimes it makes more sense to create a separate White Box unit test for a class with specific logic than cluttering a Black Box test file with those test cases.
Some parts of the code can't be properly tested by Black Box testing.
etc.

Use White Box testing only when it is really needed and as an addition to Black Box testing, not the other way around.

It's all about investing only in the tests that yield the biggest return on your effort.

Behavioral tests can be divided in two parts:

Fast: Use cases tests in isolation, with all I/O mocked and injected. This makes tests fast so they can be run all the time (after each change or before every commit). This will inform you when something fails as fast as possible. Finding bugs early is critical and saves a lot of time.
Slow: Full End to End (e2e) tests which test a use case from end-user standpoint. Instead of injecting I/O mocks those tests usually have all infrastructure up and running: like database, API routes etc. Those tests check how everything works together and are slower so can be run only before pushing/deploying. Though e2e tests live in the same project/repository, it is a good practice to have e2e tests independent from project's code. In bigger projects e2e tests are usually written by a separate QA team.

Note: some people try to make e2e tests faster by using in-memory or embedded databases (like sqlite3). This makes tests faster, but reduces the reliability of those tests. In real e2e testing this should be avoided. Read more: Don't use In-Memory Databases for Tests.

Load Testing

For projects with a bigger user base you might want to implement some kind of load testing to see how program behaves with a lot of concurrent users. Artillery is a nice tool for that based on NodeJS. Though, there are plenty of other tools to choose from: Top 6 Tools for API & Load Testing.

Example files: // TODO

Store all configurable variables/parameters in config files. Try to avoid using in-line literals/primitives. This will make it easier to find and maintain all configurable parameters when they are in one place.
Never store sensitive configuration variables (passwords/API keys/secret keys etc) in plain text in a configuration files or source code.
Store sensitive configuration variables, or variables that change depending on environment, as environment variables (dotenv is a nice package for that) or as a Docker/Kubernetes secrets.
Create hierarchical config files that are grouped into sections. If possible, create multiple files for different configs (like database config, API config, tasks config etc).
Application should fail and provide the immediate feedback if the required environment variables are not present at start-up.
For most projects plain object configs may be enough, but there are other options, for example: NestJS Configuration, rc, nconf or any other package.

Example files:

ormconfig.ts - this is typeorm database config file. Notice process.env - those are environmental variables.
.env.example - this is dotenv example file. This file should only store dummy example secret keys, never store actual development/production secrets in it. This file later is renamed to .env and populated with real keys for every environment (local, dev or prod). Don't forget to add .env to .gitignore file to avoid pushing it to repo and leaking all keys.

Logging

Try to log all meaningful events in a program that can be useful to anybody in your team.
Use proper log levels: log/info for events that are meaningful during production, debug for events useful while developing/debugging, and warn/error for unwanted behavior on any stage.
Write meaningful log messages and include metadata that may be useful. Try to avoid cryptic messages that only you understand.
Never log sensitive data: passwords, emails, credit card numbers etc. since this data will end up in log files. If log files are not stored securely this data can be leaked.
Avoid default logging tools (like console.log). Use mature logger libraries (for example Winston) that support features like enabling/disabling log levels, convenient log formats that are easy to parse (like JSON) etc.
Consider including user id in logs. It will facilitate investigating if user creates an incident ticket.
In distributed systems a gateway can generate an unique id for each request and pass it to every system that processes this request. Logging this id will make it easier to find related logs across different systems/files.
Use consistent structure across all logs. Each log line should represent one single event and contain things like timestamp, context, unique user/request id and/or id of entity/aggregate that is being modified, as well as additional metadata if required.
Use log managements systems. This will allow you to track and analyze logs as they happen in real-time. Here are some short list of log managers: Sentry, Loggly, Logstash, Splunk etc.
Send notifications of important events that happen in production to a corporate chat like Slack or even by SMS.
Don't write logs to a file from your program. Write all logs to stdout (to a terminal window) and let other tools handle writing logs to a file (for example docker supports writing logs to a file). Read more: Why should your Node.js application not handle log routing?
Logs can be visualized by using a tool like Kibana.

Make your app transparent using smart logs

Health monitoring

Additionally to logging tools, health monitoring tools are a good way to keep track of system performance, identify causes of crashes or downtime, monitor behavior, availability and load. Some health monitoring tools already include logging management and error tracking, as well as alerts and general performance monitoring.

Here are some basic recommendation on what can be monitored:

Connectivity – Verify if user can successfully send a request to the API endpoint and get a response with expected HTTP status code. This will confirm if the API endpoint is up and running. This can be achieved by creating some kind of 'heath check' endpoint.
Performance – Make sure the response time of the API is within acceptable limits. Long response times cause bad user experience.
Error rate – errors immediately affect your customers, you need to know when errors happen right away and fix them.
CPU and Memory usage – spikes in CPU and Memory usage can indicate that there are problems in your system, for example bad optimized code, unwanted process running, memory leaks etc. This can result in loss of money for your organization, especially when cloud providers are used.
Storage usage – servers run out of storage. Monitoring storage usage is essential to avoid data loss.

Choose health monitoring tools depending on your needs, here are some examples:

Sematext, AppSignal, Prometheus, Checkly, ClinicJS

Essential Guide to API Monitoring: Basics Metrics & Choosing the Best Tools

Folder and File Structure

So instead of using typical layered style when all application is divided into services, controllers etc, we divide everything by modules. Now, how to structure files inside those modules?

A lot of people tend to do the same thing as before: create a separate folders/files for services, controllers etc and keep all module's use-cases logic there, making those controllers and services bloated with responsibilities. This is the same approach that makes navigation harder.

Using this approach, every time something in a service changes, we might have to go to another folder to change controllers, and then go to dtos folder to change the corresponding dto etc.

It would be more logical to separate every module by components and have all the related files close together. Now if a use-case changes, those changes are usually made in a single use-case component, not everywhere across the module.

This is called The Common Closure Principle (CCP). Folder/file structure in this project uses this principle. Related files that usually change together (and are not used by anything else outside of that component) are stored close together, in a single use-case folder. Check user use-cases folder for examples.

And shared files (like domain objects, repositories etc) are stored apart since those are reused by multiple use-cases. Domain layer is isolated, and use-cases which are essentially wrappers around business logic are treated as components. This approach makes navigation and maintaining easier. Check user folder for an example.

The aim here should to be strategic and place classes that we, from experience, know often changes together into the same component.

Keep in mind that this project's folder/file structure is an example and might not work for everyone. Main recommendations here are:

Separate you application into modules;
Keep files that change together close to each other (Common Closure Principle);
Group files by their behavior that changes together, not by type of functionality that file provides;
Keep files that are reused by multiple components apart;
Respect boundaries in your code, keeping files together doesn't mean inner layers can import outer layers;
Try to avoid a lot of nested folders;
Move files around until it feels right.

There are different approaches to file/folder structuring, like explicitly separating each layer into a corresponding folder. This defines boundaries more clearly but is harder to navigate. Choose what suits better for the project/personal preference.

File names

Consider giving a descriptive type names to files after a dot ".", like *.service.ts or *.entity.ts. This makes it easier to differentiate what files does what and makes it easier to find those files using fuzzy search (CTRL+P for Windows/Linux and ⌘+P for MacOS in VSCode to try it out).

Angular Style Guides: Separate file names with dots and dashes.

Static Code Analysis

Static code analysis is a method of debugging by examining source code before a program is run.

For JavasScript and TypeScript, Eslint with typescript-eslint plugin and some rules (like airbnb) can be a great tool to enforce writing better code.

Using any type is a bad practice. Consider disallowing it (and other things that may cause problems):



// .eslintrc.js file
  rules: {
    '@typescript-eslint/no-explicit-any': 'error',
    // ...
  }

Also, enabling strict mode in tsconfig.json is recommended:



  "compilerOptions": {
    "strict": true,
    // ...
  }

Example file: .eslintrc.js

Code Spell Checker may be a good addition to eslint.

Code formatting

The way code looks adds to our understanding of it. Good style makes reading code a pleasurable and consistent experience.

Consider using code formatters like Prettier to maintain same code styles in the project.

Why Coding Style Matters

Documentation

Create documentation that may help users/other developers to use your program.

Simple readme file in a git repository that describes basic app functionality, available CLI commands, how to setup a new project etc.
OpenAPI (Swagger) or GraphQL specifications. Document in details every endpoint. Add description and examples of every request, response, properties and exceptions that endpoints may return or receive as body/parameters. This will help greatly to other developers and users of your API.

Example files:

user.response.dto.ts - notice @ApiProperty() decorators. This is NestJS Swagger module.
create-user.http.controller.ts - notice @ApiOperation() and @ApiResponse() decorators.

Make application easy to setup

There are a lot of projects out there which take effort to configure after downloading it. Everything has to be set up manually: database, all configs etc. If new developer joins the team he has to waste a lot of time just to make application work.

This is a bad practice and should be avoided. Setting up project after downloading it should be as easy as launching one or few commands in terminal. Consider adding scripts to do this automatically:

package.json scripts
docker-compose file
Makefile
Database seeding and migrations (described below)
or any other tools.

Seeds

To avoid manually creating data in the database, seeding is a great solution to populate database with data for development and testing purposes (e2e testing). Wiki description.

This project uses typeorm-seeding package.

Example file: user.seeds.ts

Migrations

Migrations are used for database table/schema changes:

Database migration refers to the management of incremental, reversible changes and version control to relational database schemas. A schema migration is performed on a database whenever it is necessary to update or revert that database's schema to some newer or older version.

Source: Wiki

Migrations should be generated every time database table schema is changed. When pushed to production it can be launched automatically.

BE CAREFUL not to drop some columns/tables that contain data by accident. Perform data migrations before table schema migrations and always backup database before doing anything.

This project uses Typeorm Migrations which automatically generates sql table schema migrations like this:

Example file: 1611765824842-CreateTables.ts

Seeds and migrations belong to Infrastructure layer.

Rate Limiting

By default there is no limit on how many request users can make to your API. This may lead to problems, like DDoS or brute force attacks, lags and performance issues etc.

To solve this, implementing Rate Limiting is essential for any API.

In NodeJS world, express-rate-limit is an option for simple APIs.
Another alternative is NGINX Rate Limiting.
Kong has rate limiting plugin.

Code Generation

Code generation can be important when using complex architectures to avoid typing boilerplate code manually.

Hygen is a great example.
This tool can generate building blocks (or entire modules) by using custom templates. Templates can be designed to follow best practices and concepts based on Clean/Hexagonal Architecture, DDD, SOLID etc.

Main advantages of automatic code generation are:

Avoid manual typing or copy-pasting of boilerplate code.
No hand-coding means less errors and faster implementations. Simple CRUD module can be generated and used right away in seconds without any manual code writing.
Using auto-generated code templates ensures that everyone in the team uses the same folder/file structures, name conventions, architectural and code styles.

Note:

To really understand and work with generated templates you need to understand what is being generated and why, so full understanding of an architecture and patterns used is required.

Custom utility types

Consider creating a bunch of shared custom utility types for different situations.

Some examples can be found in types folder.

Pre-push/pre-commit hooks

Consider launching tests/code formatting/linting every time you do git push or git commit. This prevents bad code getting in your repo. Husky is a great tool for that.

Git Hooks

Prevent massive inheritance chains

This can be achieved by making class final.

Note: in TypeScript, unlike other languages, there is no default way to make class final. But there is a way around it using a custom decorator.

Example file: final.decorator.ts

Conventional commits

Conventional commits add some useful prefixes to your commit messages, for example:

feat: added ability to delete user's profile

This creates a common language that makes easier communicating the nature of changes to teammates and also may be useful for automatic package versioning and release notes generation.

Articles

Repositories

Documentation

Blogs

Videos

Books

"Domain-Driven Design: Tackling Complexity in the Heart of Software" by Eric Evans
"Secure by Design" by Dan Bergh Johnsson, Daniel Deogun, Daniel Sawano
"Implementing Domain-Driven Design" by Vaughn Vernon
"Clean Architecture: A Craftsman's Guide to Software Structure and Design" by Robert Martin
Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems by Martin Kleppmann

Making A Deep Dream Twitter Bot

Noah — Mon, 01 Mar 2021 17:51:35 +0000

Let's get into it!

This will just be a quick walkthrough of how I made a twitter bot that can grab input from replies, generate a image from the text, and then pass that image through some Trippy deep dream filters.

This tutorial will assume that you are working inside of Google Colab

GitHub: https://github.com/OGoodness
Twitter: https://twitter.com/NoahFields_
Bot Twitter: https://twitter.com/WeDeepDream

Steps:

Getting Setup Inside of Google Collab
Creating API Keys
- Twitter
- DeepAI
  - Signup and then grab the provided API Key
Code
PROFIT!

One of my favorite outputs:

Harmful Candy
// Detect dark theme var iframe = document.getElementById('tweet-1339985791565512705-183'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1339985791565512705&theme=dark" }

Code!

First, we need to make sure Google Colab is using a compatible version of Tensorflow, you need to specify this before your imports:

%tensorflow_version 1.x

Imports

# Install Lucid and other Dependencies
!pip install --quiet lucid==0.0.5
!pip install requests
%tensorflow_version 1.x
# Import libraries
from google.colab import files
import numpy as np
import tensorflow as tf
import scipy.ndimage as nd
import random, cv2, json, os, requests, string, sched, time
from google.colab.patches import cv2_imshow
from PIL import Image

# Deep Dream Libraries
import lucid.modelzoo.vision_models as models
import lucid.optvis.objectives as objectives
import lucid.optvis.param as param
import lucid.optvis.render as render
import lucid.optvis.transform as transform
from lucid.misc.io import show, load
from lucid.misc.io.reading import read

Create Model

model = models.InceptionV1()
model.load_graphdef()

Get List of Words

Since you probably wont have a lot of user interaction at first, we are going to pull our input from a wordlist!

word_site = "https://www.mit.edu/~ecprice/wordlist.10000"

response = requests.get(word_site)
WORDS = response.content.splitlines()

Helper Functions

# Convert Images to Expected size/format
def imgToModelSize(arr):
  W = model.image_shape[0]
  w, h, _ = arr.shape
  s = float(W) / min(w,h)
  arr = nd.zoom(arr, [s, s, 1], mode="nearest")
  w, h, _ = arr.shape
  dw, dh = (w-W)//2, (h-W)//3
  return arr[dw:dw+W, dh:dh+W]

@objectives.wrap_objective
def dot_compare(layer, batch=1, cossim_pow=0):
  def inner(T):
    dot = tf.reduce_sum(T(layer)[batch] * T(layer)[0])
    mag = tf.sqrt(tf.reduce_sum(T(layer)[0]**2))
    cossim = dot/(1e-6 + mag)
    return dot * cossim ** cossim_pow
  return inner

#Get DeepAI Image from Input Text
def get_generated_image_url(input):
  headers = { "api-key": "DEEPAI API KEY"}
  url = "https://api.deepai.org/api/text2img"
  data = { "text": input.encode('utf-8') }
  r = requests.post(url, data=data, headers=headers)
  json = r.json()
  imageUrl = json["output_url"]
  return imageUrl

Twitter Interface

Functions to interact with Twitter, decently simple, just need to get the Twitter API Key

def get_tweet(url):
    tweet_id = url.split('/')[-1]
    api = get_api()
    tweet = api.get_status(tweet_id)
    return tweet

def get_api():
    # Authenticate to Twitter
    auth = tweepy.OAuthHandler("CONSUMER KEY", "CONSUMER SECRET")
    auth.set_access_token("TOKEN KEY", "TOKEN SECRET")
    # Create API object
    api = tweepy.API(auth, wait_on_rate_limit=True)
    return api

def get_replies(api, tweet):
    tweet_id = tweet.id
    user_name = tweet.user.screen_name
    max_id = None
    replies = []
    cursor = tweepy.Cursor(api.search, q='to:{}'.format(user_name),
                                since_id=tweet_id, max_id=max_id, tweet_mode='extended').items()
    iter = 0
    for reply in cursor:
      iter += 1
      if (reply.in_reply_to_status_id == tweet_id):
          replies.append(reply)
      max_id = reply.id
      if (iter > 100):
        break
    return replies

Image Processing

This is where the fancy part of the app comes in, where you actually process the image nd convert it into a weird nightmare creature!

def feature_inversion(img=None, layer=None, n_steps=512, cossim_pow=0.0):
  with tf.Graph().as_default(), tf.Session() as sess:
    img = imgToModelSize(img)

    objective = objectives.Objective.sum([
        1.0 * dot_compare(layer, cossim_pow=cossim_pow),
        objectives.blur_input_each_step(),
    ])

    t_input = tf.placeholder(tf.float32, img.shape)
    param_f = param.image(img.shape[0], decorrelate=True, fft=True, alpha=False)
    param_f = tf.stack([param_f[0], t_input])

    transforms = [
      transform.pad(8, mode='constant', constant_value=.5),
      transform.jitter(8),
      transform.random_scale([0.9, 0.95, 1.05, 1.1] + [1]*4),
      transform.random_rotate(range(-5, 5) + [0]*5),
      transform.jitter(2),
    ]

    T = render.make_vis_T(model, objective, param_f, transforms=transforms)
    loss, vis_op, t_image = T("loss"), T("vis_op"), T("input")

    tf.global_variables_initializer().run()
    for i in range(n_steps): _ = sess.run([vis_op], {t_input: img})

    result = t_image.eval(feed_dict={t_input: img})
    return Image.fromarray((result[0] * 255).astype(np.uint8))

Here we are loading the provided image and running it through a series of different algorithms.
Each one provides a slightly different output, they get trippier as you go along.
Once we have generated each image, we want to combine the images into a collage, so that it is easier to consume on Twitter. That's what the h_stack and v_stack is being used for.

def get_deep_images(imageUrl):
  sourceImage = load(imageUrl)
  collage = []
  layers = ['conv2d%d' % i for i in range(0, 3)] + [ 'mixed3a', 'mixed3b',
                                                     'mixed4a', 'mixed4b', 
                                                     'mixed4c', 'mixed4d', 
                                                     'mixed4e', 'mixed5a', 'mixed5b']
  h_stack = []
  v_stack = []
  image_iter = 0
  for layer in layers:
    image_iter += 1
    processed_image = feature_inversion(sourceImage, layer=layer)
    h_stack.append(processed_image)
    if (image_iter % 3 == 0):
      image_iter = 0
      v_stack.append(np.hstack(h_stack))
      h_stack = []
  if len(h_stack) > 0:
    v_stack.append(np.hstack(h_stack))
  collage = np.vstack(v_stack)

  v_stack = []
  for cossim in [0.0, 0.5, 1.0, 2.0]:
    processed_image = feature_inversion(sourceImage, layer='mixed4d', cossim_pow=cossim)
    v_stack.append(processed_image)
  h_stack = np.vstack(v_stack)
  collage = np.hstack([collage, h_stack])
  Image.fromarray((sourceImage * 255).astype(np.uint8)).save("source.png")
  Image.fromarray(collage).save("image.png")
  return collage

The Actual App

Now that we have the code to generate our image, process the image, and merge it into a collage. Let's put that all together and make our app work!
Currently, we are just generating the text from a random list of words, but you can use the commented section to choose a random tweet on a profile and grab replies from that.
Once we have our text, we call the above functions to generate and image, save the image, and we finally use these images in our tweet!

def run_deep_dream_comment_search(api):
  iter = 0
  #Here we are randomly generating the words
  reply_text = random.choice(WORDS) + " " + random.choice(WORDS)

  # Alternitively, take words from user reply: 
  # all_user_tweets = get_user_tweets(api, "WeDeepDream")
  # while (iter < 100 and all_replies == []):
  #   iter += 1
  #   random_tweet = random.choice(all_user_tweets)
  #   all_replies = get_replies(api, random_tweet)
  #   print "Iter: " + str(iter)
  # if (iter < 100):
  #   random_reply = random.choice(all_replies)
  #   reply_text = random_reply.full_text
  #   reply_user = random_reply.user.screen_name
  # api.update_status(status='@' + str(reply_user) + " Said: " + str(reply_text))
  # + str(reply_user) + " "

  print "Generating Images from text"
  gen_image_url = get_generated_image_url(reply_text)
  collage = get_deep_images(gen_image_url)

  # Upload images and get media_ids
  filenames = ["source.png", "image.png"]
  media_ids = []

  for filename in filenames:
      res = api.media_upload(filename)
      media_ids.append(res.media_id)

  api.update_status(status='Image generated from string: '  + str(reply_text[0:30]) + "https://github.com/OGoodness", media_ids=media_ids)
  print "Just tweeted"
  for filename in filenames:
      try:
        os.remove(filename)
      except OSError:
        print "Whoops!"
        pass

Run Repeatedly

Now that we have everything working, lets set up the collab to continuously run!
Eventually, the Collab WILL time out, but you can find ways around this, for now, a few minutes will work as a Proof Of Concept.

s = sched.scheduler(time.time, time.sleep)
api = get_api()
def do_something(sc): 
    print "Doing stuff..."
    run_deep_dream_comment_search(api)
    s.enter(600, 1, do_something, (sc,))

s.enter(10, 1, do_something, (s,))
s.run()

Example Outputs

Base Michael
// Detect dark theme var iframe = document.getElementById('tweet-1366411043614973955-968'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1366411043614973955&theme=dark" }
Finds Panel
// Detect dark theme var iframe = document.getElementById('tweet-1365382364566671361-115'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1365382364566671361&theme=dark" }
Rabbit Ellen
// Detect dark theme var iframe = document.getElementById('tweet-1365347261073428482-696'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1365347261073428482&theme=dark" }

Forem: Noah

Code Design - Domain Driven Hexagon

This Is A Post From the Original GitHub Repo

Domain-Driven Hexagon

Table of Contents

Architecture

Pros:

Cons:

Diagram

Modules

Application Core

Application's Core consists of:

Domain layer:

Application layer:

Application layer

Application Services

Interfaces for each use case

Local DTOs

Commands and Queries

Commands

Queries

Ports

Domain Layer

Entities

Aggregates

Domain Events

Domain Services

Value objects

Enforcing invariants of Domain Objects

Replacing primitives with Value Objects

At compile time

At runtime

How to do simple validation?

Types of validation

Using libraries inside application's core

Interface Adapters

Controllers

DTOs

Request DTOs

Response DTOs

Additional recommendations:

Infrastructure

Adapters

Repositories

Examples

ORM Entities

Other things that can be a part of Infrastructure layer:

Recommendations for smaller APIs

Other recommendations and best practices

Error Handling

Exception types

Differentiate between programmer errors and operational errors

Error Serialization

Error metadata

Other recommendations

Alternatives to exceptions

Testing

Load Testing

Configuration

Logging

Health monitoring

Folder and File Structure

File names

Static Code Analysis

Code formatting

Documentation

Make application easy to setup

Seeds

Migrations

Rate Limiting

Code Generation

Custom utility types

Pre-push/pre-commit hooks

Prevent massive inheritance chains

Conventional commits

Additional resources

Articles

Repositories

Documentation

Blogs