Forem: The Mighty Programmer

Finding Balance in Software Development

The Mighty Programmer — Mon, 01 Jun 2020 18:12:15 +0000

Software development is not only about code. It is also about the process and the people around it. At the core, Product (Business) Managers and Developers turn the idea of Software into reality.

In the philosophical view of Software development, there exist, two realms that affect the reality of Software: Product end and Technology end.

These are different in the perception of an application:

An Application is a set of features that support a business function. - Product View

An Application is a set of code, framework, libraries, services. - Developer View

In contrast, these are also associated in a way that a change in one realm introduces change(s) in another realm.

Tap in Product Realm

Customer Grievances, Competition Edge, UX improvements, Feedbacks and Law Changes: all these introduce changes. After a rigorous thought process (cost-benefit analysis), these changes are passed to developers to transform into code.
At developers end, they might have to write new code or adjust with the existing codebase. The complexity of adjustment depends upon changes and existing code.

complexity = f(changes, existing_code)

Patching and re-patching of code disturb the structural balance of Software. When we patch, we are so focused on the piece of code to work that we ignore the overall stability of Software.

Many times, fixing or enhancing one feature leads to disturbing other features. At one point in time, Software becomes a minefield.

Do it right from the beginning

A Software can only be well designed if we know what kind of changes to expect in future.

We are human; we can p̵r̵e̵d̵i̵c̵t̵ estimate a limited version of the future.

This option might not always be viable.

The external APIs are needed to be done right from the beginning; with time, it is harder to change interaction points because of uncontrolled dependants. It is nice to have developers with a mixed level of expertise or experience to corrected have better API design from the beginning.

Continuous Refactoring or Refactor as you go

Continuous Refactoring is a practical solution to keep the structure of Software in check. Sometimes, it may require re-structuring functionalities having no role with current business changes to keep Software structure stability or consistency.

There is risk involved with Refactoring, which may cause breaking of existing features and restless nights.

Test-Driven Development

Test-Driven Development (TDD) helps to mitigate breaking existing features risk. It helps to keep the existing-functionalities in check before and after refactoring.

You don’t need to follow the rule book of TDD. You just need to have some way of proving the existing features is not disturbed.

Tap in Technology Realm

New tools, code practices and technologies attract developers (as everyone wants to do better). When we start the development of any project, we freeze code practices: tools, frameworks, libraries and coding style. These practices stay constant for years.

More the difference of the current Software development practice to new market trends, more it is harder to keep developers motivated on the current project running on old tracks. With time, the business value of the project becomes so much that it is harder to mess up with existing practices.

At one point in the time, when developers quit; finding new developers to work on existing projects, are tougher to find.

Money becomes the only bait; after all, money solves all the problems. What for how long it would keep developers motivated?

Continuous Change

I love to borrow some concepts from Design Thinking; whether it is the design of the code, practices or process, it should be ongoing.

Change is an only forward option, and it does mean you readily accept the changes. It is all about maintaining a balance between stability and evolution.

In the software world, the definition of "right" is always temporary, contextual and opinionated. You have to (re)-discover and (re)-align your version of "right" before it is too late.

Uploading a large file from web

The Mighty Programmer — Tue, 31 Mar 2020 20:24:29 +0000

Uploading files over the web is not an easy task. It involves considerable challenges in developing a solution that works for all file sizes. Uploading files are prone to failures, users drop, and security risks. On each failure, the file needs to be re-uploaded: which adversely affect associated user experience.

Problems

Network Bandwidth & File Size

The file upload takes considerable time: more the time to upload, more the chances of failure or connection drop. With each failure, a file needs to be re-uploaded from the beginning with traditional methods.

The time to upload file can be understood by the following equation:

time_to_upload = file_size / effective_bandwidth + overhead

Note: The equation is oversimplified to explain relationships; it is not accurate; A variable is introduced for abstracting network overhead involved in the request.

It can be interpreted as:

More the size ⇒ more time.
Lesser the bandwidth ⇒ more time.

A large file on a slow network is the worst possible case.

Server Limitation

Http servers restrict the size of a file that can be uploaded. The standard limit is 2GB.
The operating system or file journalling system also imposes a limit on the file size they can handle.
Disk space availability also a factor that controls maximum file size that can be uploaded.

Security Risks

Uploading files is not free from Security Risks. The surface area of security risk depends upon the purpose of uploaded files.

Denial of Service: Server spends most of the time in serving a few requests; It expedites the hacking attempt in making Server go out of service.
Code Infusion: File upload easily enables code to be uploaded if unchecked, and it may lead to system hijacking.
File Overriding: Client provided path can trick Server into replacing any critical file.

Solution Idealization

The following idea or can be proposed to increase the user experience in uploading files:

Pre Check: Pre checking with Server before uploading whether it can handle the request or not.
Resumability: File upload should resumes if there is a failure in connecting. It can only if Server or Client stores the state of upload progress.
Verification: File should be verified and scanned to eliminate any threat.

Solution Realization

The following are iterations of implementation options (Read in order); these may be feasible or not.

Pre-checking

Pre checking with Server is an additional network request; it may not be useful for small file size, but pre-checking for large files can be helpful.

With Http Header Expect: 100-continue¹

Http header Expect: 100-continue is probing header that used to determine whether Server can receive the current request with large message body or not. If Server accepts, it sends back 100 else 417 status code. If Server accepts, a second request is trigged to upload the file.

The beauty of this mechanism is that the second request automatically trigged by Http Client. Unfortunately, It cannot be set via programming means available: fetch API or XHR (Ajax) request. It can only be set by underlying user-agent or browser. In short, A programming effort could not be made.

Also, it is not well understood by many Servers implementations even if you somehow manage to set the header.

Curl add this header on crossing 1024KB request body size³ when browsers add who knows.

It is a useful header to be practically useless. We need to pre-check through standard request.

With Two Separate Standard HTTP requests

Overall uploading process can be conceptualized as two standard HTTP requests:

A file metadata need to be sent first.
Based upon the server response file can be uploaded.

You need to develop your error and success message or codes to realise this mechanism.

Without Reserving capacity

Let’s assume a situation server has 1GB space left. Imagine, two clients asking to upload at the same time:

Both clients would get permission to upload, and after a while, both requests would be interrupted when Server gets 1 GB of combined data from both requests.

With Reserved Capacity

What if Server could reserve capacity for a file that is about to be uploaded?
It might look like a good idea, but it may not.

Server would be dealing with multiple requests at an instance, and not all of these would be successful. If unnoticed, Server may run out of storage space soon; even though Server having storage space conceptually. Also, any miscreant could learn it; place an attack on service.

You need to devise a strategy to reclaim space carefully. If you are thinking to build resumability, Server needs to wait for some time to reclaim corresponding space.

With Dynamic Capacity

We live in a cloud computing world; where you don’t need to plan capacity (only if you have unlimited money 😌 ). Most of Cloud Providers provides Object Storage.

Object Storage obscures scalability challenges associated with traditional file systems, and provide a simplified API to access entity named Objects. An object is semantically equivalent to a file.

Modern databases too include BLOB storage similar to Object Storage. Object Storages and Databases are alike in term of file system abstraction, but Databases offer their challenges of operations.

Resumability & Time

Chunking

When file size crosses a certain limit; it becomes necessary to split it and upload it in multiple requests.

A file is a sequence of bytes. We can collate some bytes into chunks. These chunks need to be individually uploaded by Client and combined by Server.

no_of_requests = file_size / chunk_size
time_to_a_request = chunk_size / bandwidth + overhead
time_to_upload  ~  time_to_a_request *  no_of_requests

It is a bit slower than traditional mechanism as multiple requests increase networking overhead (ack), but it gives ultimate control in hand:

Provide the ability to upload large files over 2GB
Resumability can be build using this idea.

Chunking is effortful; it introduces additional metadata to be exchanged to build reliable file upload. HTML 5 provides many useful utilities to realise this mechanism.

Here is a basic code snippet to illustrate the core implementation of chunking :

// file is instance of File API
const file = form.querySelector('[input=file]').files[0]
const totalSize = file.size

// 10 KB : K = 1000 : Network transimission unit
const chunkSize = 10 * 1000
const noOfChunks = Math.ceil(totalSize / chunkSize)
let offset = 0

for (let i = 0; i < noOfChunks; i++) {
  const chunk = file.slice(offset, offset + chunkSize)
  // upload  ajax request: hidden

  offset = offset + chunkSize
}

An article can be dedicated to design decisions associated with chunking; for now, you can explore Resumable.js and Tus before you mind up to build your implementation.

Compression

A file can be compressed before uploading to Server. Compression is a double edge sword as it may increase or decrease overall upload time.

total_to_upload = compression_time + upload_time

Also, Server must understand the compression algorithm in place; it is part of content-negotiation strategies.

You must have proof of concept if you introduce compression.

Security

Scanning every uploaded file is an essential task. Additionally, you can consider the following security measurements:

Integrity Check

A transferred file must be validated. Checksum Checking is a well-known practice to verify file integrity. There are many hashing algorithms to choose from MD5, SHA-1, SHA-256 or many more. Whatever algorithm is chosen for whatsoever reasons, should be supported by both Client and Server implementation.

HTTP Header: Etag is used to exchange checksum. The calculated value must be transferred over the secure channel (TLS).

Blacklisting

A file with executable permission can do more harm, especially if is application engine file like .php, .jsp, .js, .sh , and .asp.
Sandboxing or limited access is the key to protect the system.

At best, prevent users from uploading executable files.

Deep Content Disarm and Construction

Deep Content Disarm and Construction (Deep CDR) is a technique of rebuilding files from the file parts by discarding harmful components. It makes sense for files like pdf, doc or spreadsheet which allows embedded content. You can read about the technique in details here.

Closing Notes

The system attributes: kind of files, maximum allowed file size affect the implementation choices.
If you are storing files in a traditional file system, then limit the file size. It would require a considerable number of requests to place a DOS attack and hopefully detectable.
Devise a policy to define a time window to consider file upload failure and to eradicate partial-uploaded files.
With Chunking, it would seem like you are repeating implementation of TCP at higher granularity.

References

Expect Header

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Expect
When Curl sends 100 continue

https://gms.tf/when-curl-sends-100-continue.html
FileSize Limitation

https://stackoverflow.com/q/5053290/3076874

FootNotes

Bandwidth is the amount of data that can be transferred in a unit of time.
Compression is information storage optimising encoding mechanism.

Custom Event with RxJS

The Mighty Programmer — Wed, 29 Jan 2020 15:26:08 +0000

Event-Based Programming is natural to any GUI based interface. HTML DOM¹ offers inbuilt Event Notification model, but it is only useful when you are dealing with DOM Elements. There is no support for non-DOM elements.

Web applications have grown up in complexity with time; also, Javascript started participating in backend applications; which in turn made the world to see Javascript as more than HTML DOM manipulation gig.

This article showcases building up custom event publisher-subscriber model on top of Reactive Programming for non-DOM elements.

Setting up Stage

Event-Driven Programming

Event-Driven programming paradigm² introduces an intuitive flow of control based upon events.

Button is clicked => show dialog.

Vocabulary

One who emits event can be called Event Publisher or Event Emitter or Event Dispatcher.
One who listens to events is called Event Subscriber or Event Listener or Event Handler.

In general, Publisher-Subscriber or PubSub term is used collectively for event publisher and even subscriber.

Reactive Programming

Reactive Programming is a trending paradigm² these days. It is the processing of streams of data asynchronously. At core, reactive streams are written on top of the event-driven paradigm (?). You are going to build an event notification model on top of it by modelling events as a stream of data.
RxJS is a reactive programming extension for Javascript.

In Action

Problem Statement

Consider modelling Thermostat, when room temperature crosses a certain threshold (30°C), AC would turn up, and windows would close.

Design

Consider class Thermostat which encapsulates the logic of monitoring and publishes events:

above: when temperature crossed above 30°C (threshold) from lower value.
below: when temperature crossed below or equal to 30°C (threshold) from higher value.

Data Structure for holding event structure

RxJS offers Subject which can be used to emit events to multiple subscribers.

class Thermostat {
  constructor(sensor) {

    // data structure for holding subscriptions 
    this.subscriptions = {
      above: new Subject(),
      below: new Subject()
    };
  }

}

Notice, there are two Subject objects. You can also create a single Subject object named ThresholdCrossed with direction above or below. Separating the event or combining the event is Design Concern which may depend upon many factors.

Subscription API for Event

on API for registering action based upon event.

on(event, action) {
  // sanatize event left code
  this.subscriptions[event].subscribe(v => action(v));
}

Publishing Events

Consider monitor method which detects the rise and falls in temperature and publishes event accordingly.

 monitor(currentTemperature) {
  // when temperature cross above THRESHOLD
  if (
    currentTemperature > THRESHOLD 
    && this.lastRecordedTemperature <= THRESHOLD
  ) {

    // publish event
    this.subscriptions["above"].next(currentTemperature);

  } else if (
    currentTemperature <= THRESHOLD 
    && this.lastRecordedTemperature >= THRESHOLD
  ) {

    // publish event
    this.subscriptions["below"].next(currentTemperature);
  }

  this.lastRecordedTemperature = currentTemperature;
 }

Subscribing for event

When the temperature is below 30°C then ac should go off, and the window should open; the opposite should happen for above event 30°C.

thermostat.on("below", t => {
  this.ac = false;
  this.window = true;
});

thermostat.on("above", t => {
  this.ac = true;
  this.window = false;
});

Wiring all pieces together.

Fiddling

Any variation you make with the above code itself a pattern:

Keeping event subscription data in the mediator class of Event-Emitter and Event-Listener is in-memory Event Bus pattern.
If the event holds enough information that Event-Listener doesn't look back to Event-Emitter; it is Event Carried State Transfer pattern
If you store all events in datastore, it is Event Sourcing pattern (possible for backend side).
If there is a network partition between Event Publisher and Event Subscriber, then collectively, it is a Reactive System. It is build using queuing technologies (possible for backend side).

An event may contain, reference or link back to its source. In HTML DOM, each event has a field named target to point back to its source.

Footnotes

Document Object Model is the parsed tree object representation of HTML document.
A paradigm is a conceptual view of programming elements.

About Event-Driven https://www.youtube.com/watch?v=STKCRSUsyP0&vl=en

Inheritance vs Composition

The Mighty Programmer — Mon, 09 Dec 2019 05:01:09 +0000

With the introduction of OOPs, Inheritance and Composition entered our senses and still confusing us.

Choosing between Composition and Inheritance is not easy. Often, the decision depends upon the information which is not yet there.

The structure of an application plays an essential role in adding changes. Composition and Inheritance are the building blocks of an application structure. It is imperative to know when to use which one.

Definition

Inheritance

Inheritance is a way of reusing code by inheriting the structure from the referred class or Type. The referred class is called parent or base class and the referring class is called child or subclass.

Inheriting the structure suggests that subclass inherits public members (API) of the parent class. A subclass can override or add new behaviour or extend the existing behaviour of the parent class.

// Reference class 
class Animal {
   fun walk() {
     println("animal walk");
   }

   fun speak() {
      println("...")
   }
}

// Referring Class 
// Cat class "inherits the structure" from the Animal class
class Cat : Animal {

   // overriding existing behaviour
   fun speak() {
      println("meow")
   }
}


// in main
val cat = Cat();
cat.walk() // Cat can "reuse" parent class behavior 
cat.speak() // meow

Composition

Composition is a way of reusing code by interacting with the referred class without any structural imposition.

// Reference Class
class Line (val x: Int, val y:Int);
  fun draw() {
     println("line drawing")
  }
}

// Refering class 
class Shape (val sides: Array<Line>) {
  fun shapeDraw() {
     for(side in sides) {
       side.draw(); // code refering
     }
  }
}

Decision Matrix

Uniformity

I had a situation to ignore JsonNaming annotation¹ provided by Jackson library (Java ecosystem). There is no direct API provided by the library to facilitate the same.

A possible solution to have a new class dictating the desired functionality and injecting the new class object to the library. Luckily, the chosen library provides the API to inject custom class (Thanks to open-close design).

Should I compose or inherit from the existing Class? - inherit.

// class for library 
class JSONIntrospector {

    // .. rest code

    public Object findNamingStrategy(AnnotatedClass ac) {
        JsonNaming ann = _findAnnotation(ac, JsonNaming.class);
        return (ann == null) ? null : ann.value();
    }

    // .. rest code
}


// custom class
class IgnoreJsonNaming extends JSONIntrospector {

    @Override
    public Object findNamingStrategy(AnnotatedClass ac) {
        return null;
    }
}

// inject code
ObjectMapper mapper = new ObjectMapper();
mapper.setAnnotationIntrospector(new IgnoreJsonNaming());

Inheritance is more suitable in this context as it is expected to have structural similarity or uniformity.

Composition would not have worked :
If I preferred to go with Composition, I would have to implement unnecessary methods to maintain structural similarity.

Imagine, If in future, library developers add new behaviours to the referred Class, I would end up changing in the latest Class to have structural similarity.

Semantics

Inheritance and Composition not only allow to reuse the code; they also define a relationship semantic between two classes.

Inheritance : is-a
Composition : has-* (* means any quantifier)

Apple is a Fruit. (Inheritance)
Apple has seeds. (Composition)

Human is a mammal. (Inheritance)
Human has two hands. (Composition)

These keywords assist in basic analysing the requirements.

Software Model: OOA/D

Whenever we write software, we introduce so many concepts from Specification and Implementation point of view.

Through rigorous analysis, these concepts end up with numerous classes. Shaping these classes (object decomposition) and connecting these classes become our primary responsibility.

Implementation Concepts: Factory, Manager, Service, Client, Provider
e.g. RestClient, EmailService, ValidtorFactory

Sometimes, it is better to start with code at first, then evolve the design. You are free to play until the first release. Once the software is released, it is hard to reverse most of the decisions.

Multiple Concepts

Inheritance does not befit well for multi-dimensions concepts or variants or attributes. If applied, it leads to Combinatorial Explosion of classes.

Say, you are modelling Pizza (concept 1) with two allowed Toppings (concept 2) Onion and Tomato. If you choose Inheritance to connect Pizza and Topping, then you would end with four classes.

// with inheritance
class Pizza { // without topping
  fun getPrice() {
   return 100;
  }
}
class PizzaWithTomatoToppings : Pizza {
  fun getPrice() {
   return 120;
  }
}
class PizzaWithOnionTopping : Pizza {
  fun getPrice() {
   return 120;
  }
}
class PizzaWithOnionAndTomatoToppings : Pizza {
  fun getPrice() {
   return 140;
  }
}

With Composition, two classes are sufficient.

// with composition
class Pizza {
  var t1 : Topping? = null ;
  var t2 : Topping? = null ;

  fun getPrice() {
    val basePrice = 100;
    return basePrice + t1?.getPrice() ?: 0 + t2?.getPrice() ?: 0 
  }

}
class Topping(val price: Int) {
  fun getPrice() {
    return price;
  }
}

Takeaways

In Inheritance, the structure imposition introduces tight coupling between both connected classes; it restricts their future editing.
Inheritance is well suited for uniformity.
Inheritance does not great for multi-dimensions concepts or variants or attributes.
Composition does not impose any structure of any of connected Class. It is a flexible option.

References

Disable specific annotation in Jackson https://github.com/FasterXML/jackson-databind/issues/133