Forem: Dr. Gernot Starke

Speed up your video

Dr. Gernot Starke — Tue, 29 Nov 2022 13:28:01 +0000

In case you want a video to play faster, you have a few options:

some video players (like Elmedia Player (Mac only) or similar tools) allow faster or slower playback.
Use a video editing tool (like DaVinci Resolve or similar tools) to either speed up or slow down your video
Use the free ffmpeg tool on the command line to modify the duration of a video. (ffmpeg can perform a lot of other cool things with video, have a look at their website.

Let's look at an example: Below you find a 6-minute video, plus a 14 second version of the same visual content... just lot's faster.

Original (Slow) Video

High-Speed Video

`ffmpeg` Solution

ffmpeg: A complete, cross-platform solution to record, convert and stream audio and video.

Obviously, you need to install ffmpeg first. The ffmpeg website has instructions for you. (You can use a package manager like brew)
Open a terminal, head over to a directory containing your original (slow) video.

Now for the good stuff:

$ ffmpeg -i Yoda-Autoplay-compact.mp4  -vf "setpts=(PTS-STARTPTS)/5" -crf 18 yoda-autoplay-fast.mp4

The magic happens in the parameter setpts=(PTS-STARTPTS)/5, which reduces the duration to one fifth.

For the example above, I used a factor of 30 to reduce the duration from approx 6 minutes to approx 15 seconds.

Hope you find this info useful - all kudos to the ffmpeg developers.

Machine-Generated Images - with Yoda

Dr. Gernot Starke — Mon, 24 Oct 2022 09:18:41 +0000

This is part of a small series of articles showing images generated completely by AI (artificial intelligence):

The following collection contains images generated by DALL.E and Midjourney. In case I forgot some important contemporary artist, please let me know in the comment section.

The Artists and Styles

Zappa, Frank
Yokoo, Tadanori
Witt, Solomon Le-
Wiley, Kehinde
Warhol, Andy
Walker, Kara
Ting, Walasse
Takano, Aya
Street art
Stella, Frank
Sherald, Amy
Saville, Jenny
Riley, Bridget
Richter, Gerhard
Pollock, Jackson
Piccini, Patricia
Perry, Grayson
Ono, Yoko
Night, Starry
Neshat, Shirin
Mozart, Wolfgang Amadeus
Motherwell, Robert
Mendieta, Ana
Martin, Agnes
Lichtenstein, Roy
Kusame, Yayoi
Kruger, Barbara
Koons, Jeff
Karim, Anthony Abdel
Judd, Donald
Hundertwasser, Friedensreich
Hurst, Damien
Hockney, David
Grey, Alex
Graffiti
Graubner, Gotthart
Goscini (and Uderzo)
Goldsworthy, Andy
Communism
Close, Chuck
Clark, Lydia
British Upper House
Beksiński, Zdzisław
Bauer, Joseph
Basquiat, Jean-Michel
Abramovic, Marina

What is Contemporary Art?

I searched the internet for "contemporary artist" and "contemporary painter", and got literally dozens of results, containing painters, photographers, sculptors and other kind of visual artists.

The selection I present here is purely a matter of my personal taste. Chances are I forgot some important artists, as I'm software architect, no art expert.

Quoted from Wikipedia:

Contemporary art is the art of today, produced in the second half of the 20th century or in the 21st century. Contemporary artists work in a globally influenced, culturally diverse, and technologically advancing world.

And now, let's dive right in. The order is inverse alphabetic (sorted by last or family name), for a change.

Zappa, Frank

More known for his (sometimes strange) music, Frank Zappa makes for a stunning Yoda...

Yokoo, Tadanori

Witt, Solomon Le-

Wiley, Kehinde

Warhol, Andy

Walker, Kara

Ting, Walasse

Takano, Aya

Street art

Stella, Frank

v1	v2

Sherald, Amy

Saville, Jenny

Riley, Bridget

Richter, Gerhard

Pollock, Jackson

Piccini, Patricia

Perry, Grayson

Ono, Yoko

Night, Starry

(obviously, this is no artist but something like a style)

Neshat, Shirin

Mozart, Wolfgang Amadeus

(obviously, this guy is not a contemporary artist, but the image is nice anyway)

Motherwell, Robert

Mendieta, Ana

Martin, Agnes

Lichtenstein, Roy

Kusame, Yayoi

Kruger, Barbara

Koons, Jeff

Karim, Anthony Abdel

Judd, Donald

Hundertwasser, Friedensreich

Hurst, Damien

(ok, he gets three)

v1	v2	v3

Hockney, David

Grey, Alex

Graffiti

(obviously again, this is no single artist, but an art style)

v1	v2

Graubner, Gotthart

Goscini (and Uderzo)

These artists invented the Asterix comic strip.

Goldsworthy, Andy

Communism

Again, no artist, but a relict of a political movement.

Close, Chuck

Clark, Lydia

British Upper House

Ooops, another one that does not represent a single artist.

Beksiński, Zdzisław

Bauer, Joseph

Basquiat, Jean-Michel

Abramovic, Marina

Yoda, explaining art styles by example

Dr. Gernot Starke — Sun, 09 Oct 2022 08:43:19 +0000

Update Oct 9th: added Zen
Update Oct 10th: added Graffiti
Update Oct 11th: added Glitch art (thx to AntonR)
Update Oct 12th: added Mosaic
Update Oct 21st: added Cyberpunk
Update Oct 24th: added Precisionist
Update Oct 30th: added Pixel-Art

Introduction
Art Styles
- Abstract Art
- Acacemic Art
- African
- Aquarelle
- Art Deco
- Art Nouveau
- Avant-Garde
- Bauhaus
- Baroque
- Colour field
- Comic
- Contemporary Art
- Cyberpunk Art
- Dadaism
- Etching
- Expressionism
- Fauvism
- Folk art
- Futurism
- Glitch
- Graffiti
- Impressionism
- Islamic art
- Japonism
- Cubism (or Kubism)
- Modern Art
- Mosaic Art
- Naive Art
- Neoclassicism (Neo-classicism)
- Neo-Impressionism
- Neon Art
- One-Liner
- Pencil drawing
- Pixel Art
- Pointilism
- Pop Art
- Postmodern Art
- Precisionism
- Renaissance
- Rococo
- Romanticism
- Sketchnote
- Soviet Art
- Street Art
- Stuckism
- Surrealism
- Symbolism
- Ukio-e
- Vaporwave
- Zen
Appendix
- Copyrights on these images
- Acknowledgements

This article gives an overview of art styles (more precisely: painting styles), based upon computer-generated images.
It demonstrates a few capabilities of current AI-based generation systems, namely:

Introduction

For reasons only understood by some dark force, I decided to use the popular Yoda figure as the recurring subject.

In case you want to

see more Yoda images, please refer to my ART-ificial Yoda gallery, showing fictional and generated Yoda portraits by different artists, sorted by painters' age.
read more about art styles with real paintings, see
- this simplicable article or
- this artland article

And now, without further ado, let's dive into art styles. As I got completely overwhelmed by the vast number of different styles I encountered when I started researching for this post, I decided to sort them alphabetically, instead of historically.

Art Styles

Abstract Art

Abstract art does not attempt to represent an accurate depiction of a real entity, but instead uses shapes, colours and forms.

midjourney	DALL.E

Acacemic Art

Academic art is a style of painting created under the influence of European academies of art, especially the French Académie des Beaux-Arts. It combines influences of Neoclassicism and Romanticism.

African

African art describes the modern and historical paintings from native or indigenous Africans.

Craiyon	DALL.E

Aquarelle

Watercolor, also aquarelle is a painting method in which the paints are made of pigments suspended in a water-based solution, and the canvas is often wetted to make the colours slightly disperse into each other.

Midjourney	DALL.E

Art Deco

Art Deco is a style of visual arts that first appeared in France in the 1910s just before World War I.
Art Deco combined modern styles with fine craftsmanship and rich materials.

Art Nouveau

Art Nouveau is an international style of art, architecture and the decorative arts. The style is known by different names in different languages: Jugendstil in German, Stile Liberty in Italian, Modernisme in Catalan, and also known as the Modern Style in English. Art Nouveau was popular during the Belle Époque period that ended with the start of World War I in 1914

Avant-Garde

The avant-garde is a style that is experimental, radical, or unorthodox with respect to art, culture, or society.

Bauhaus

The Staatliches Bauhaus (German for 'building house'), was a German art school operational between 1919 and 1933, that combined crafts with fine arts. The school became famous for its approach to design, which attempted to unify the principles of mass production with individual artistic vision.

Baroque

The Baroque is a style of painting and other art forms that flourished in Europe from the early 17th century until the 1740s. It continued until the first decade of the 19th century. It followed Renaissance art and Mannerism and preceded the Rococo (in the past often referred to as "late Baroque") and Neoclassical styles.

Colour field

Color field painting is a style of abstract painting that emerged in New York City from the 1940s. Inspired by European modernism, related to abstract expressionism, many of its proponents were among the abstract expressionists. Color field is characterized by large fields of color spread across the canvas creating areas of unbroken surface and a flat picture plane.

Comic

Comic is an art style used to express ideas with images combined with text.

Contemporary Art

Contemporary art is the art of today, produced in the second half of the 20th century or in the 21st century. Contemporary artists work in a globally influenced, culturally diverse, and technologically advancing world.
(quoted from Wikipedia)

Cyberpunk

Cyberpunk is a subgenre of science fiction, often in dystopian futuristic setting.

Dadaism

Dadaism was an art movement of the European avant-garde in the early 20th century, starting in Zürich, Switzerland. New York Dada began 1915, and after 1920 Dada flourished in Paris. Dada artists expressed nonsense, irrationality, and anti-bourgeois protest in their works.

Etching

Etching traditionally was the process of using acid to cut into the unprotected parts of a metal surface to create a design incised in the metal.

Expressionism

Expressionism is a art movement that originated in Northern Europe around the beginning of the 20th century. It presents the world from a subjective perspective, distorting to create emotional effect.

Fauvism

Fauvism is the style of a group of early 20th-century artists whose works favoured strong color over the realistic values used by Impressionism.

Folk art

Folk art covers visual art made in the context of folk culture. Objects shown have practical utility, rather than being only decorative.

Futurism

Futurism (Italian: Futurismo) was an artistic movement from Italy in the early 20th century. It showed dynamism, speed, technology, youth, violence, and modern objects such as the car, the airplane, and other industrial products.

Glitch

Glitch art uses digital or analog errors for art purposes by either corrupting digital data or manipulating electronic devices.

Graffiti

Graffiti is art that is painted or drawn on a walls or other surfaces, often without permission and within public view. Graffiti ranges from simple written words to elaborate wall paintings, and has existed since ancient times.

Impressionism

Impressionism was a 19th-century art movement using relatively small brush strokes and emphasis on accurate depiction of light.

midjourney	DALL.E

Islamic art

Islamic art has been characterized by the absence of figures and extensive use of calligraphic, geometric and abstract floral patterns.

Japonism

Japonism, also often referred to by the French term, japonisme, refers to the incorporation of either iconography or concepts of Japanese art into European art and design.

Cubism (or Kubism)

Cubism is an early-20th-century art movement that revolutionized European painting. In Cubist artwork, objects are broken up and reassembled in an highly abstracted form.

Modern Art

Modern art includes work produced during the period from the 1860s to the 1970s, and denotes the styles and philosophies of the art created during that era.

Mosaic Art

A mosaic is an image made of small regular or irregular pieces of colored stone, glass or ceramic.

Naive Art

Naïve art is usually defined as visual art that is created by a person who lacks formal education and training.

Neoclassicism (Neo-classicism)

Neoclassicism was a cultural movement that drew inspiration from the art and culture of classical antiquity. European Neoclassicism in the visual arts began approx. 1760 in opposition to the then-dominant Rococo style.

Neo-Impressionism

Neo-Impressionism is a term coined by French art critic Félix Fénéon in 1886 to describe an art style founded by Georges Seurat. Neo-Impressionists painted modern urban scenes as well as landscapes and seashores.

Neon Art

A somewhat self-explanatory art form.

One-Liner

Maybe this is no official art style - as only very few people (I only know of La Linea) painted in it...

Pencil drawing

A pencil drawing is, surprise surprice, a drawing made with a pencil. The lines are therefore of graphite colour.

Pixel Art

Pixel art is adigital art drawn where images are created using rectangular blocks of colour (aka pixels) as the only building block.

Pointilism

Pointillism is a painting technique in which dots of color are used in patterns to form an image.

Pop Art

Pop art is an art movement that emerged in the United Kingdom and the United States during the mid- to late-1950s, challenging the traditions of fine art by including imagery from popular and mass culture, like advertising, comic books and mass-produced objects.

Postmodern Art

Postmodern art is an art movement that seeks to contradict some aspects of modernism.

Precisionism

Precisionist artworks often contain unexpected viewpoints and angles.

Renaissance

Renaissance art (1350 - 1620) is the painting, sculpture, and decorative style of the period of European history, which emerged as a distinct style in Italy.
Famous Renaissance painters were Albrecht Dürer and Jan van Eyck.

Rococo

Rococo (approx 1730) is an exceptionally ornamental style of art and decoration which combines asymmetry, scrolling curves, gilding, white and pastel colours, sculpted moulding, and frescoes to create the illusion of motion and drama.

Romanticism

Romanticism (also known Romantic era) was an artistic movement that originated in Europe towards the end of the 18th century. Romanticism idealized nature and the past, and was suspicious of science and industrialization.

Sketchnote

Sketchnoting, also referred to as visual notetaking, is the creative and highly graphic process through which someone can record thoughts by using illustrations, symbols, structures, and texts.

Soviet Art

Soviet art is a visual style that came up after the October Socialist Revolution of 1917 in Soviet Russia (1917—1922) and the Soviet Union (1922—1991).

Street Art

Street art is visual art created in public locations for open visibility.
Alternative terms are "post-graffiti", "neo-graffiti" and "guerrilla art".

Stuckism

Stuckism is an international art movement founded in 1999 by Billy Childish and Charles Thomson to promote figurative painting instead of conceptual art.

Surrealism

Surrealism is a cultural movement that developed in Europe after World War I in which artists showed illogical scenes.

Symbolism

Symbolism was a late 19th-century art movement of French and Belgian origin seeking to represent absolute truths symbolically through language and metaphorical images.

Ukio-e

Ukiyo-e is a genre of Japanese art famous from the 17th through 19th centuries. Its artists often produced of subjects like female beauties, sumo wrestlers or scenes from history ,travel, landscapes or flora and fauna.
The term ukiyo-e (浮世絵) translates as "picture of the floating world".
(Quoted from Wikipedia)

Vaporwave

Vaporwave is emerged in the early 2010s, defined partly by slowed-down, chopped and screwed samples of smooth jazz, elevator, R&B, and lounge music from the 1980s and 1990s. Vaporwave paintings incorporate early Internet imagery, late 1990s web design, glitch art, anime, 3D-rendered objects and others.

Zen

Zen art is known for elegant simplicity, often in black ink on white paper or silk.

Appendix

Copyrights on these images

The platforms used to generate these images handle copyright in slightly different manner.

Please let me make the following explicit: I did never upload, modify or otherwise use any official image of YODA, as these might or will be copyrighted and trademarked by Disney Inc. and George Lucas as the inventor and creator of the YODA character.

Concerning DALL.E copyrights

See their terms of service

You may provide input to the Services (“Input”), and receive output generated and returned by the Services based on the Input (“Output”). Input and Output are collectively “Content.” As between the parties and to the extent permitted by applicable law, you own all Input, and subject to your compliance with these Terms, OpenAI hereby assigns to you all its right, title and interest in and to Output.

Concerning Midjourney copyrights

See their terms of service:

Subject to the above license, you own all Assets you create with the Services. This does not apply if you fall under the exceptions below.

These exceptions are unpaid members and large cooperations.

Acknowledgements

Thanx to Simon for the improved version of the vaporwave Yoda.
Thanx to Anton R. for pointing me to Glitch art.

The ART-ificial Yoda Gallery

Dr. Gernot Starke — Wed, 05 Oct 2022 20:32:02 +0000

(a number of updates to this post got lost in digital nirvana - please come back in a few days for the enhanced version of this article...)

Recently, several AI (artificial intelligence) based image generation systems became popular.

This article gives a glimpse of what is currently (October 2022) possible with three major players in this league. In addition, you can learn a bit about different styles of art.

If you want to try for yourself, the first three systems require a (free) registration and will allow a few images to be generated free of charge.
Craiyon can be used without registration.

Beware: Playing around with these image generators can become an addiction - so you might want to set a timer before you start (at least I spend hours and hours ...)

Our subject

Yoda (/ˈjoʊdə/) is a fictional character in the Star Wars universe. He is a small, green humanoid alien, powerful with the Force and a leading member of the Jedi Order.
quoted from Wikipedia

Ok, you already knew this, I suppose.

All rights to the original Yoda character belong to Disney Inc. and George Lucas.

A short introduction to Yoda

DALL.E created this with the query "photo realistic image of yoda".

This image has also been generated by DALL.E, with the command "Yoda walking in desert".

"Yoda on the beach" was the phrase that made DreamStudio create the previous one.

Yoda as drawn by real artists

And now let the AI systems re-create the styles of various famous artists, starting a few hundred years back.

Yoda in style of Giovanni Bellini

Giovanni Bellini (born approx. 1430) was an Italian Renaissance painter, from the Venetian school.

Yoda in style of Hieronymus Bosch

Hieronymus Bosch (born approx 1450) was a Dutch painter, a representative of the famous Netherlandish painting school.

No, it's not guy from the current detective and crime series you saw on your favorite streaming channel...

Yoda in style of Leonardo da Vinci

Leonardo da Vinci (born 1452) was an Italian multi-talent, who was active as a painter, engineer, scientist, architect and other disciplines.

Yoda in style of Albrecht Duerer

Albrecht Dürer (born 1471) was a German painter of the Renaissance, especially famous for woodcut prints.

Yoda in style of Giuseppe Arcimboldo

Giuseppe Arcimboldo (born about 1527) was an Italian painter best known for creating portrait heads made entirely of objects.

Yoda in style of Rembrandt

Rembrandt (born 1606) was a Dutch Golden Age painter and is considered one of the greatest visual artists and the most important in Dutch art history.

Yoda in style of Francisco Goya

Francisco José de Goya y Lucientes (born 1746) was a Spanish romantic painter, considered the most important Spanish artist of the late 18th and early 19th centuries.

Yoda in style of Carl Spitzweg

Carl Spitzweg (born 1808) was a German romanticist painter of the Biedermeier era.

Yoda in style of Edgar Degas

Edgar Degas (born 1834) was a French Impressionist artist, known for his pastel and oil drawings.

Yoda in style of Thomas Moran

Thomas Moran (born 1837) was an American painter in New York whose work often showed the Rocky Mountains.

Yoda in style of Paul Cézanne

Paul Cézanne (born 1839) was a French artist and Post-Impressionist painter whose work lead the 19th-century style of painting to the radically different painting of the 20th century.

Yoda in style of van Gogh pointilism

Vincent van Gogh (born 1853) was a Dutch Post-Impressionist painter who became one of the most famous individuals in Western art.

Yoda in style of Gustav Klimt

Gustav Klimt (born 1862) was an Austrian symbolist painter.

Yoda in style of Edvard Munch

Edvard Munch (born 1863) was a Norwegian painter. His best known work, The Scream (1893), has become incredibly famous.

Yoda in style of Alfred Stieglitz

Alfred Stieglitz (born 1864) was an American photographer who made photography an accepted art form.

Yoda in style of Kandinski

Wassily Kandinsky (born 1866) was a Russian painter and art theorist. Kandinsky is generally credited as one of the pioneers of abstraction in western art.

Yoda in style of Emil Nolde

Emil Nolde (born 1867) was a German-Danish painter, one of the first Expressionists, a member of Die Brücke, known for exploring color.

Yoda in style of Paul Klee

Paul Klee (born 1879) was a Swiss-born German artist with an individual style influenced by expressionism, cubism, and surrealism.

Yoda in style of Pablo Picasso

Pablo Picasso (born 1881) was a highly influential Spanish painter, who co-founded the Cubist movement.

Yoda in style of Joan Miró

Joan Miró i Ferrà (born 1893) was a Catalan painter, known for Surrealistic paintings.

Yoda in style of Norman Rockwell

Norman Rockwell (born 1894) was an American illustrator, known for reflecting the country's culture in his images.

Yoda in style of Salvador Dali

Salvador Dali (born 1904) was a Spanish surrealist artist known for his technical skill, and bizarre images in his paintings.

Yoda in style of pointilism

Yoda in style of Roy Lichtenstein

Roy Lichtenstein (born 1923) was an American pop artist. Along with Andy Warhol and others he lead the new art movement.

Yoda in style of Robert Rauschenberg

Milton Ernest "Robert" Rauschenberg (born 1925) was an American painter whos works anticipated the Pop art movement.

Yoda in style of Bernard Buffet

Bernard Buffet (born 1928) was a French painter, known for his extensive body of primarily figurative work.

Yoda in style of Andy Warhol

Andy Warhol (born 1928) was an American visual and film artist who was a leading figure in pop art style.

Yoda in style of Walasse Ting

Walasse Ting (born 1929) was a Chinese-American visual artist, known for his colorful paintings.

Yoda in style of Solomon LeWitt

Solomon "Sol" LeWitt (born 1928) was an American known for various styles, including conceptual art and minimalism.

Yoda in style of Fernando Botero

Fernando Botero Angulo (born 1932) is a Colombian figurative artist and sculptor, who often depicts people and figures in exaggerated volume, which may represent political criticism or humor.

Yoda in style of Gerhard Richter

Gerhard Richter (born 1932) is a German visual artist, known for his abstract and photorealistic paintings. He is one of the most important contemporary German artists.

Yoda in style of Yoko Ono

Yoko Ono (born 1933) is a Japanese multimedia artist, singer, songwriter, and peace activist, formerly married to John Lennon.

Yoda in style of Keith Haring

Keith Haring (born 1958) was an American artist, whose style emerged from the New York City graffiti pop art of the 1980s.

Yoda in style of Kehinde Wiley

Kehinde Wiley (born 1977) is an African-American portrait painter based in New York City, known for naturalistic paintings.

Yoda in style of Yasutomo Oka

Yasutomo Oka (born 1983) is a Japanese painter, known for his photo-realistic paintings.

Yoda in style of Banksy

Banksy is a pseudonymous England-based street artist whose real name and identity remain unconfirmed.

Yoda in various art styles

Please see the following article for a detailed introduction to art styles in general.

Yoda in modern African style

Craiyon	DALL.E

Appendix

Copyrights on these images

The platforms used to generate these images handle copyright in slightly different manner.

Concerning DALL.E copyrights

See their terms of service

You may provide input to the Services (“Input”), and receive output generated and returned by the Services based on the Input (“Output”). Input and Output are collectively “Content.” As between the parties and to the extent permitted by applicable law, you own all Input, and subject to your compliance with these Terms, OpenAI hereby assigns to you all its right, title and interest in and to Output.

Concerning Midjourney copyrights

See their terms of service:

Subject to the above license, you own all Assets you create with the Services. This does not apply if you fall under the exceptions below.

These exceptions are unpaid members and large coorperations.

More information

Article comparing various image generation systems.
Wikipedia article on AI art
Vladimir Alexeev: (German) Article on DALL.E

Acknowledgements

Thanx to Lisa Moritz and Peter Hruschka for additional suggestions.

How to regain concentration and focus

Dr. Gernot Starke — Wed, 03 Aug 2022 12:34:00 +0000

This article was originally published at INNOQ.

No sooner had I started a task, the next thing I knew I was doing something else. I distracted myself by checking my email inbox every so often, and I was addicted to checking what was going on in the world on various news websites, constantly interrupting the original task.

You're perfectly right: This sounds completely insane. Somehow these bad habits had sneaked silently into my life.

News is to the mind what sugar is to the body.

Rolf Dobelli, Author and entrepreneur

And then two lightning bolts struck almost simultaneously: a book and a blog post.
I significantly changed my (digital) life as a result and within only two weeks I was able to get more done again, sleep better, and am significantly happier.
The short version: news diet and productive smartphone use.

Lightning strikes

How my concentration was lost…

For years, I've been producing content on a fairly regular basis in the form of talks, books, journal articles, blog posts, podcasts, and contributions to open-source projects (To quantify that: 30+ books or book editions and about a hundred articles in various magazines within the last 20 years - that's still one and a half books and five articles on average per year).

In the last few months, I ran out of steam, which made me personally very discontent. In early 2022 I published the last blog post, and in late 2021 the last articles. For the third time, I postponed the new edition of the arc42-in-Action book ...

I had plenty of ideas, but somehow I couldn't put them down on paper. Unfortunately, ideas alone are not enough: the ideas also have to be made into reality, structured, and written down in comprehensible ways.

Moreover, I was always a dedicated textbook reader, and binge-read a new book every few weeks. That, too, became more and more difficult for me - even though I was genuinely interested in the topics of the unread books on the shelf.
Now, I could chalk it all up to the pandemic, my age (don't ask...), or whatever. But finding bogus excuses is just not my thing.

"You've got to do something about that," I told myself. At first, I tried to increase my productivity and creativity by getting up earlier. No such luck. Staying up later in the evening didn't work either.

Then I tried what I've recommended (successfully!) countless times in my job as an IT consultant: A systematic analysis of the situation, a self-review.

The self-analysis

I tried to analyze my own way of working. I quickly noticed that it was becoming increasingly difficult for me to concentrate on individual tasks: No sooner had I started a task than I was already doing something else in between - or distracted myself with emails or various news websites. Likewise, I sat at preparing a lecture - and checked the INNOQ Slack channels in between. Furthermore, I continued to write on a slide - and took a brief look at my smartphone to see what's been happening on WhatsApp or Signal. And since I was holding the phone in my hand anyway, I could read a few news items on BBC-News there too.

Did you know that smartphone users unlock and look at their phones an average of 80-100 times a day (source: Cision Newswire, several other sources giving similar numbers)? I was one of them.
News websites like CNN, WSJ, and BBC had me as a permanent customer.

For me, these context switches already had addiction-like traits.
Even while I was watching a series (yes, I admit that I like to binge on one or two episodes), I would pick up the phone or tablet in between and casually check the news.
Against my better judgment, I also checked my e-mails first thing in the morning before I even drank my first espresso.

And in my inbox, there were always various news summaries, from stock-market news, and international politics up to technology updates.

Unfortunately, all this news has done me more harm than good: For my coaching and consulting engagements, I don't need to know anything about current domestic or foreign politics.
For my software architecture and engineering workshops, neither stock market trends nor the details of international terrorist acts are of any value.
My articles and books deal with software and software engineering, not with climate or politics.

A self-discovery

Constantly switching between (sophisticated) technical work, the news from around the world, and private communication on personal and professional topics, I would argue, cost me a significant amount of my ability to concentrate.

My brain (apparently) doesn't handle frequent context switches well. Moreover, my brain must have unconsciously perceived distractions as something positive - and, like Pavlov's dog, kept wanting more of them, at increasingly shorter intervals.

A (Pavlovian) dog

I noticed the lack of concentration myself. However, my own fainthearted attempts, with Pomodoro timer and calming sounds, clearly failed.
Effective remedies from the outside had to come first, two major sources helped me:

While travelling to an architecture workshop, I sat on the train, not reading the news for a change, but attempting to read a book.

For some time now, I have been using Blinkist to read short summaries of various non-fiction books, to find out for myself whether I is worth for me to read the original of these books in their entirety.

So - I'm sitting on the train and reading one of these Blinks, when I'm struck by a mental lightning bolt: I felt so caught by Rolf Dobelli's explanations, caught in my concentration trap, that I spontaneously decided to get serious. Let me quote from this book:

News is not good for us:
It clouds our mind, distorts our view of what is really important, robs us of time, makes us depressed and paralyzes our willpower.
Rolf Dobelli, Author and entrepreneur

Paralyze my willpower, rob me of time.

That was precisely my problem.

While still sitting on the train, I decided to try a strict break from any news (news, not e-mails or other messages) for a while.

However, that wasn't enough: I remembered a blog post I had read a few years ago - about the productive configuration of smartphones, which has the promising subtitle Configure Your iPhone to Work for You, Not Against You.
I therefore decided to optimize my iPhone for maximum productivity and to reduce or even eliminate distractions.

Drastic, but effective

A (currently very strict) zero-news-diet.
(Re)configure smartphone for high productivity and minimal distractions.
Reduce distractions on tablets and computers. Add to that slight changes I made to my daily habits: I've resolved (and managed for a few weeks now) to stop staring at my phone in the morning, and to stop using my iPhone at all in the evening after 8 PM at the latest (exception: setting an alarm for the next morning).

Zero News Diet

During the aforementioned train ride, I cancelled my subscription to my formerly favorite online news portal (the German "Spiegel-Online") and deleted the associated app from my iPhone. I used the initial motivation to cancel additional news subscriptions and deleted the their apps from my smartphone and iPad so that I wouldn't be tempted.

I have resolved very firmly not to read any news at all for a while. In other words, a zero news diet. That was really hard for me for two or three days. Yet, that alone was a startling realization for me: I was actually addicted to news and variety... and cold withdrawal is hard with any kind of addiction, as you know.

I purposefully invested the hours gained each day.
As a kind of early reward for my zero news diet, I treated myself to an interesting non-fiction book as a PDF (by the way, the infamous "Thinking, Fast and Slow" by the great Daniel Kahnemann).
The result: I was thrilled to have made good progress without any distractions.

To support my news diet, I instructed my browser (Firefox) to not present me with recommendations for supposedly interesting or important topics on new tabs or windows.

Firefox preferences: No more news suggestions

Reconfigure Smartphone

The above quote, "Configure Your iPhone to Work for You, Not Against You," sets the objective:
It's the subtitle of a very long blog post by Tony Stubblebine.
In addition to his many configuration suggestions, Tony also gives various advice for being more attentive and healthy, entirely worth reading in my opinion.

I summarize the things most important for me, which he illustrates in his article with many screenshots and explanations:

Turn off (almost) all notifications – so those red dots aren't constantly vying for your attention or making you feel guilty.
Hide social media apps as well as possible. Facebook, Instagram, Twitter & Co act like drugs. They deserve to be removed from the start/home screen and moved somewhere in the back of your smartphone. I deleted the Twitter app and kept Instagram only because it allows me to catch up a bit with my adult children.
Messaging apps (email, WhatsApp, Signal and co.) go into a folder, and preferably on the second screen of that folder. For me, it looks like this (K11n stands for communication):

Communication apps – unimportant ones need more “swipe” gestures
Turn on do-not-disturb-mode, the longer, the better. At least from evening until morning.
Lifting the phone should not unlock it. This setting is called "Display & Brightness/Activate on Lift" on the iPhone - and should definitely be turned off.
Turn on the screen time widget – this gives me control over all the things I spend my smartphone time on... Meanwhile, I have made a sport out of getting to the smallest value possible.
Turn on content and app restrictions. I only allow myself 15 minutes a day for email and Instagram.
Install a swipe keyboard app, so I can text faster on the phone and not have to awkwardly click each letter. I use Microsoft Swiftkey (yes correct, a Microsoft app on the iPhone) for this, others do better with Google's Gboard.
Wallpapers in muted colors. I chose a completely black home screen, and a reduced color lock screen:

iPhone backgrounds in muted colors

Reduce distractions on tablets and computers

You will find similar preference settings on your tablet and your computer, respectively. I set those to reduce notifications as much as possible:

Notifications settings, almost all turned off

In case you fall off the wagon

In case you want to be a bit stricter with yourself, there are various blockers for popular desktop platforms that can prevent access to certain websites or applications for certain times. These can improve your digital self-discipline. I tried ColdTurkey and like it a lot. However, I hope to become disciplined enough to stick to my new habits without this electronic tether… at least in the near future.

Furthermore...

Besides the aforementioned settings and the zero news diet, I like to listen to soundscapes for (supposedly) better mental focus when working intensively at my desk. I write supposedly because I like this kind of aural background, but can't prove that it helps me in any way. My family, by the way, thinks it's horrible.

If you want to try it out: My two favorites are Endel and brain.fm. Works on both smartphones and desktops (but please don’t say I didn’t warn you…).

I still haven't gotten comfortable with Pomodoro timers, they annoy me more than they help.

And finally, thanx to Łukasz, you might want to restrict your YouTube usage with the unhook extension. I tried that and immediately liked it. You have to tweak the default settings, though. No more wasted hours due to the addictive YouTube proposition algorithm...

Conclusion

Why don't you start a self-experiment, and refrain from reading sports, politics, business, and technology news as much as possible for a while? Reduce your smartphone time drastically for one or two weeks, and enjoy the time gained with a good book, a personal conversation or approach a previously abandoned activity that has been left undone so far (due to a perceived lack of time…)

Good luck – and I look forward to your feedback.

Acknowledgements

Thanks to Jochen Christ, Joachim Praetorius, Jan Seeger and Ben Wolf for reviews and constructive comments. @m and Joy Heron have drastically improved readability and wording.

This article was originally published at INNOQ, achieved a #4 rank and more than 300 comments on Hackernews

Image Sources

Dog
Lightning
Header image by Paul Skorupskas

Enable ssh access to multipass vms

Dr. Gernot Starke — Mon, 01 Aug 2022 15:03:00 +0000

The problem

You are using multipass to create lightweight virtual (Ubuntu) machines.
You want to ssh into those machines, because you cannot or don't want to use the standard shell command multipass shell <name-of-vm>.
The naive approach fails with permission denied:

$ ssh 192.168.205.7
The authenticity of host '192.168.205.7 (192.168.205.7)' can't be established.
ED25519 key fingerprint is SHA256:lWKUbxxxx.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.205.7' (ED25519) to the list of known hosts.
gernotstarke@192.168.205.7: Permission denied (publickey).

Permission denied, although there is a route to this virtual machine available...

Different approaches

Two approaches have been documented elsewhere:

Add an ssh-key to an existing virtual machine (explained by Josue Bustos here). Downside of this approach: You have to repeat it (more or less manually) for all new VMs you create with multipass launch. Therefore, let's look at a solution that will also work for new VMs...
Pass an ssh-key while creating a multipass VM (explained in detail by Ivan Krizsan here). In short: You generate a new ssh-key, and pass the public key into every new VM. I summarize the required steps below.

The solution

(I mention it again, the idea presented here have first been described by Ivan Krizsan):

Create new ssh-key for your multipass VMs
Create a reusable launch configuration for multipass VMs with cloud-init
Launch new multipass VMs with this configuration
Find out IP-address of new multipass VM
ssh into the new VM with this IP-address

Prerequisites: You have:

multipass installed. If not, it's well-documented on their website.
a terminal running a decent shell (bash, zsh).
a decent text editor (VS-Code, nano, vim or the like.)

1.: Create ssh-key

On your host machine (the one with multipass installed), change to the directory from where you will be launching multipass vms. It can be your home directory, but any other will do.

$ ssh-keygen -C vmuser -f multipass-ssh-key

vmuser can be any (dummy) username, it will later be used to log into the VM.
The parameter to -f is the filename for the generated key. You can choose a name of your liking, but there must not be an existing key with the same name in the same directory.

You will be asked to enter a passphrase, leave that empty! I know, it's not as secure as it should be, but multipass VMs are used for development and test only...

Empty passphrases are NOT suited for production environments.

2.: Create cloud-init configuration

Cloud-init is the standardized approach for cross-platform cloud and VM configuration of instance initialization.
Multipass can handle such configuration, we use it to pass the ssh key into the vm.

In the directory where you generated the ssh-key, execute the following:

$ touch cloud-init.yaml
$ code cloud-init.yaml

You can obviously use any text editor of your choice. Nowadays I use VS-Code for most of my edits.

Put the following content into this file:

users:
  - default
  - name: vmuser
    sudo: ALL=(ALL) NOPASSWD:ALL
    ssh_authorized_keys:
    - <content of YOUR public key>

The <content of YOUR public key> starts with the letters ssh-rsa and ends with the username you supplied in step 1. Both have to be included!
Remember, it's yaml: Sensitive to spaces and dashes.
In case you generated an ed25519 type of key, it starts with ssh-ed25519. If you don't know what I'm talking about, ignore this last line.

3.: Launch multipass VM with ssh configuration

You will use a slightly different launch command for your VM, by adding the cloud-init parameter:

$ multipass launch -n testvm --cloud-init cloud-init.yaml

testvm is the name of your new vm, you can choose any name or even leave it blank. In the latter case, multipass will create a random name for you.
cloud-init.yaml is the name of the configuration file we created in step 2.

4.: Find IP address of new VM

$ multipass info testvm
Name:           testvm
State:          Running
IPv4:           192.168.205.4
Release:        Ubuntu 20.04.4 LTS
Image hash:     692406940d6a (Ubuntu 20.04 LTS)
Load:           0.00 0.00 0.00
Disk usage:     1.5G out of 4.7G
Memory usage:   140.5M out of 976.9M
Mounts:         --

testvm is the name of the VM you were using in step 3 (launch VM).
The output of the info command contains the IPv4 address of this VM, you need that in the next step.

5.: ssh into new VM

$ ssh vmuser@192.168.205.4 -i multipass-ssh-key -o StrictHostKeyChecking=no

Welcome to Ubuntu 20.04.4 LTS (GNU/Linux 5.4.0-122-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

vmuser is the username for which you created the ssh key
Please replace the IP address by the one you found out in step 4.

And voilá - you made it.
The command itself should be obvious... you pass the name of the ssh-key with the -i commend (short for identity). In addition, you turn strict host checking off.

You might see additional messages, depending on your host machine ssh configuration - but you should be logged in your multipass VM by now.

May the power of the shell be with you.

Acknowledgements

Thanx to Ivan Krizsan) for providing the initial solution and his nice writeup. I took me a while to cut through numerous web-searches to find his post, therefore I decided to write it up again under a slightly different title.

All credits and kudos to him! All faults a clearly mine.

Brief introduction to arc42

Dr. Gernot Starke — Mon, 21 Feb 2022 14:17:56 +0000

Brief Introduction to arc42

arc42 ¹ is a template for architecture communication and documentation.
It is a proven, practical and highly pragmatic approach and takes the pain out of documentation.

arc42 provides everything you ever need to communicate and document your software architecture.
It's tool and technology agnostic, therefore can you use arc42 for arbitrary systems.
arc42 is open source and completely free to use.

This article provides a quick overview of arc42 and its parts.
For more information, please consult the (rather extensive) documentation [docsˆ] available on their documentation website (which, since V8/January 2022, contains several practical examples).

arc42 answers the following two questions in a pragmatic way and can be tailored to your specific needs:

What should you document/communicate about your architecture?
How should you document/communicate?

If you work under time or resource constraints, you will most likely skip certain parts of arc42. Our upcoming post on "lean documentation" will give guidance for such cases.

Before we dive in...

Everything is optional - there is no need to fill in every section of the template. Treat arc42 documentation like the compartments of a cabinet, similar to the image below: The cabinet has a value, even if certain compartments remain empty.

(image by Jan Kolar from Unsplash)

The order of these compartements is optimized for reading and understandability. You create your content in any order you like or your project requires. For example, in many development projects, the context (section 3) is defined and created first, followed by the top quality requirements (section 1.2).
arc42 is completely tool-agnostic. You might create and maintain arc42 in your toolchain of choice: A wiki, a collection of Office documents, plain Markdown or Asciidoc, one of the commercial UML tools or even LaTeX. Whatever your tool of choice is, arc42 has you covered.

The structure of arc42

arc42 proposes a dozen sections (aka compartments) for your documentation.
These are organized top-down to improve understandability.

1. Introduction and goals

Short description of the requirements and driving forces of your system:

Describe what the system delivers for its users in one or two sentences.
Describe the two or three major use cases or features of the system.
You should provide an extract or abstract of requirements.
Restrict yourself to the top three (max five) quality requirements for the architecture, which have highest priority for the major stakeholders.
Include a table of important stakeholders with their expectation regarding architecture or documentation.

2. Constraints

Anything that constrains teams in design and implementation decisions or decision about related processes. Constraints can sometimes go beyond individual systems and might valid for whole organizations and companies (e.g. company-wide technology choices or government regulations).
Time and money are common constraints in many organizations.

3. Context and scope

The context delimits your system from its (external) communication partners (neighboring systems and users).
It specifies or documents the external interfaces.
You should always document the context from a business or domain perspective.
If infrastructure or specific hardware plays an important role, you might also show a technical perspective.

4. Solution strategy

Summary of the fundamental decisions and solution strategies that shape the architecture. Can include technology, top-level decomposition, approaches to achieve top quality goals and relevant organizational decisions.

5. Building block view

Usually, boxes and arrows, showing the high-level code structure of the system. To phrase it a little more formal: The building block view explains the static structure of the system and contains, abstractions of source-code. It refines the context view, where the complete system is depicted as blackbox.

Level 1 contains the major subsystems, components or parts of the system. In some cases, this coarse overview provides enough structural overview.
Level 2 refines one or more elements from level 1: Create a separate whitebox (plus explanations) for every level-1 building block you like to explain in detail.

You can drill down to an arbitrary level of detail, but keep in mind the following:

Recommendation
Try to stick to level-1, as it often gives enough guidance and understanding for most stakeholders.

6. Runtime view

The runtime view explains the behaviour or processing of one or several building blocks.
It serves as companion to the static building block view from section 5 above.
The runtime view might explain important use cases or features, interactions at critical external interfaces, error and exception behaviour.

Recommendation
Use runtime scenarios primarily during development to validate or design building block structures. They help to identify missing or unnecessary dependencies. Keep only a small number of scenarios in your documentation

7. Deployment view

Software needs hardware to execute on, that's where the deployment view comes into play:
It shows the technical infrastructure with environments, computers, processors, networks and network-topologies.
In addition, it maps the software building blocks to those infrastructure elements.

8. Cross-cutting concepts

(aka cross-cutting concerns)

Overall, principal regulations and solution approaches relevant in multiple parts (→ cross-cutting) of the system. Concepts are often related to multiple building blocks. Include different topics like domain models, architecture patterns and -styles, rules for using specific technology and implementation rules.

9. Architecture decisions

Keep a collection of architecturally significant decisions that are important, expensive, critical, large scale or risky including rationales, that are not recorded elsewhere.

Please use your judgement to decide whether an architectural decision should be documented here or whether you better document it locally (e.g. within the building block view or any cross-cutting concept).
You may have already captured the most important decisions of your architecture in the solution strategy.

Recommendation:
Use Architecture Decision Records to capture decisions in a simple, structured yet pragmatic way.

10. Quality requirements

Quality requirements, often described as scenarios.
You may use a quality tree to provide a high-level overview.
The most important quality goals should have been already described in section 1.2. (quality).

Let's look at some examples:

Topic	Scenario
Performance	System shall performs all checks on a 100MByte file in less than 10 seconds.
Operability	The temperature range in which the correct functioning of the device is ensured should range from -25 degrees to +85 degrees
Maintainability	A new additional verification algorithm can be included in less than one day.

11. Risks and technical debt

Known technical risks or technical debt. What potential problems exist within or around the system? What does the development team feel miserable about.

In case somebody reviewed or audited your system, their findings might be included here.

12. Glossary

Important domain and technical terms that stakeholders use when discussing the system.
Please include only specific terms - and avoid explaining REST, HTTPS or other words that have common explanations.

In addition, the glossary can serve as the translation reference if you work in a multi-language (international) environment.

Like what you saw?

Give it a try! Document a few of your architecture decisions, sketch your scope and context and explain a few key technical concepts - and you have your first version of your architecture documentation.

Download the template in a format you like. Unpack and start documenting. In case you have questions or want to see examples, consult the extensive documentation site ².

In case you want to know more, we have you covered:

I wrote about the principles of technical documentation. That deep-dive article explains the fundamental requirements your technical documentation needs to fulfill - and many proven approaches how to achieve these.
Another article on lean documentation is in the making, please stay tuned. Until then, watch the video "Sparsame Dokumentation" (oops, sorry, it's in German)

Comments or questions?

Leave a comment right below - or open an issue on the arc42 site. Otherwise, thanx for reading this article.

https://arc42.org. The template has been created by Dr. Peter Hruschka and Dr. Gernot Starke. They published the first version in 2005, after having used predecessor versions for many years in different large and medium-sized projects. ↩
https://docs.arc42.org. The extensive documentation website, containing more than 140 tips and 30 examples on how to use arc42. ↩

Principles of technical documentation

Dr. Gernot Starke — Sat, 29 Jan 2022 16:12:10 +0000

This article collects fundamental requirements for technical documentation, especially software architecture documentation, together with ideas how to satisfy those.

TL;DR

The content of technical documentation needs to be correct, current, understandable and relevant. It needs to be maintainable, referenceable and use proper language. Furthermore, the whole documentation and its content need to be easy to find. To create and maintain such documentation, you should put everything under version control, prefer established tools and automate certain parts. Finally, you should document incrementally and continuously - and not defer documentation to a later stage of development.

To help you achieve these goals, the article summarizes more than 30 simple practices that will facilitate your documentation task.

0. Introduction

During several years of consulting work, I have noticed that different development teams across organizations in various domains repeated the same documentation mistakes over and over again. Therefore I decided to summarize some fundamental requirements for technical documentation

I deliberately mention modeling and diagramming only briefly as these will be subject of another post.

A counterexample

Imagine you are planning a trip to Scotland with your family. You want to visit the famous castles and legendary landscapes, and walk in the footsteps of the brave Scottish knights.

To best prepare for this wonderful trip, you visit your local bookstore and ask for a suitable guidebook. The clerk proposes the “Adventure Travel Guide to Scotland”.

Fig. 0.1: Your map of Scotland

Back home, you treat yourself to a decent cup of tea, recline in your favorite chair and start reading. After a couple of pages, you start to wonder if you got the right book (aka documentation):

To your disappointment the documentation is not useful, as the authors start in Scotland, but quickly deviate to a lengthy coverage of the Spanish island of Fuerteventura
Next thing, parts of the documentation seem quite outdated to you. The only map you can find in your new book, despite being visually appealing, was printed¹ in 1747, more than 250 years ago (see Fig 0.1). Where the heck are the airports and highways?
It contains some serious mistakes - the city of Edinburgh is misspelled Edembourg²

Hopefully, your technical documentation better fits its title and is
less than 250 years behind the current “version” of the system. But
maybe you get the analogy: Documentation should be there to help, not hinder.

Overview of documentation requirements

Let’s start with an overview of documentation requirements, so you know what to expect. The main part of this article covers both requirements and solution approaches in detail.

Fig. 0.2 Categories of documentation requirements

The most important of these categories cover the content of
documentation:

Requirement	Explanation
Req-1: Correct	Documentation needs to be accurate and free from errors. Wrong documentation is often worse than no documentation.
Req-2: Current	Documentation needs to be correct over time, reflecting changes performed upon code, infrastructure or interfaces of the system.
Req-3: Understandable	Documentation needs to be understood by the target audience.
Req-4: Relevant	With respect to structure, form and content, documentation shall be relevant for the tasks of its audience.

A few of these requirements are covered in the fundamental book book³.

Fig. 0.3: Content requirements

In addition, you should consider a few formal requirements, independent of documentation content:

Requirement	Explanation
Req-5: Referenceable	Use a consistent numbering schema for headings, diagrams and tables.
Req-6: Proper language	Use proper language, correct spelling and grammar, active voice, positive statements and short sentences.
Req-7: Maintainable	Maintainability is key to keeping documentation current.

Fig. 0.4: Formal and content requirements

Finally, a few requirements concern your documentation process and
tooling:

Requirement	Explanation
Req-8: Easy to find	Documentation itself should be easy to find whenever needed. Its content should be easily navigable and searchable.
Req-9: Version controlled	As the system evolves, so will your documentation, without losing its history
Req-10: Appropriate tools	Focus on content, reduce time needed for tool-setup.
Req-11: Continuously updated	Make it a habit to maintain and expand the documentation with every relevant change in your system.

That leaves us with the following overview of documentation requirements:

Fig. 0.5: All documentation requirements

In the upcoming sections I will consider these requirements one-by-one. You will find several tips and proposals on how to achieve them.

Req-1: Correct

Technical documentation absolutely needs to be correct, “free of
mistakes or errors and strictly adhere to fact and truth” (³, p3).

Some examples:

When a diagram shows relationships between certain elements, those relationships have to exist in reality.
On the other hand, if is no relationship between those two elements, there shouldn’t exist one in the underlying code!
If a specific version of any product or framework is mentioned, exactly this version shall be used in reality.

This requirement seems trivial, but correctness has to be achieved even in situations where development teams are constantly working to improve or enhance the system. They change code, update interfaces, frameworks or underlying technology. Important stakeholders will change requirements or constraints.

The issue with constantly changing systems leads to another requirement (“current”), which will be subject of a larger section. Before we dive into this subject, let’s look at some options you have to get your documentation correct.

How to achieve correctness

Have the right people create and maintain the documentation.
1. The exact same people who design and implement specific parts of the system also document these parts. The advantage: No information is lost at some potential interpersonal communication barrier. The disadvantage: These people might not be capable of producing high quality technical documentation, they might not have appropriate tools, motivation or time.
2. Have the architect(s) create that documentation. On one hand they will be able to extract the required knowledge from the responsible stakeholder. On the other hand, they should know the basics of quality technical documentation.
Have all parts of your documentation reviewed by the people who built, decided or implemented that exact thing - as they know the truth.
Prefer abstract information over detailed: Abstract information is still correct, it’s just leaving out some detail. Leaving out details reduces the chance for errors and the need for
constant updates. Have the courage to leave things undocumented (better no documentation than wrong documentation). The next section on current documentation will dive deeper into that subject.

Req-2: Current

Change is normal: Development teams have to implement new features or optimize/fix/change existing ones. They have to adjust the system to cope with changes in infrastructure, updates in frameworks or 3rd-party products. Teams have to update their systems to reflect changes in external infrastructure, data formats or a plethora of other aspects that might change during the lifetime of any software system.

Despite all those changes, the documentation has to fulfill the
correctness requirement. Teams must constantly adapt (maintain) their documentation, so it remains correct.

How to keep documentation current

Regularly review your documentation, especially those parts that relate to recently-changed elements of the system.
Leave out the obvious. Focus on important and specific aspects, elements or decisions in your system.
Document only those parts or aspects of your system and its architecture that are definitely required by stakeholders.
From time to time you should iterate over all parts of the documentation and:
- question whether this part is still required by somebody and
- you are still willing to maintain this part of the documentation in the future.
- If the answer to one of these questions is “no”, then remove the corresponding part of the documentation. As everything is version controlled, it would be available if needed later on.
Optimize your documentation (not only your system!) for maintainability. The section on Easy to find contains numerous approaches that help to optimize maintainability.
Automate the generation of certain parts of your documentation..

Req-3: Understandable

Although understandability seems like a trivial and basic requirement for technical documentation, it is pretty hard to achieve and really easy to get wrong. Let’s dig into some details of what makes documentation understandable:

Terms and symbols need to be clear and unambiguous.
Use abbreviations only after they have been written in full.
The overall organization (aka structure) of the documentation shall be optimized for consuming/reading
People tend to comprehend things better if they understand the reasons
Form, structure, language, tone and symbols should be adequate for the audience.

How to improve understandability

Know your target audience

Understandability depends on the target audience.

What people understand depends on their knowledge, experience, language skills, culture etc. A specific part of your documentation might be fully understandable by certain people, and completely alien to others. Different stakeholders will have different requirements concerning documentation.

The more you know about your stakeholders, the better you will be
prepared to meet their demands. This is closely related to Relevance.

Follow these steps:

Identify all stakeholders: Identify all roles or people that might have an interest in or need for your technical documentation
Identify what your target audience really needs, what are acceptable forms or formats for documentation
Create and maintain a stakeholder table containing that information.

This information about your stakeholders will also be important to
improve the relevance of your documentation.

Document unambiguously

Ensure your documentation is clear, precise and presented in a way that the target audience can understand. Most diagrams leave room for interpretation, especially if naming and/or semantics of elements isn’t exactly clear. Make sure that all symbols in diagrams and all specific term have a defined meaning. Add a legend to clarify visual elements if you are not using a standardized notation.

See also Explain your notation
See also Combine diagrams with text

Place details into context

Your audience shall be able to start with an overview and then “dive in” when they need more information. Therefore, organize your documentation top-down.

Technical documentation will contain technical details that are prerequisite to understanding some decisions or concepts. Ensure people can understand the context of such a detail. Add references to external documents for readers who might not meet your knowledge
prerequisites.

Fig. 3.1: Details with context

Explain specific terms

In case you use specific terms with a special or uncommon meaning or
semantics, you should define or explain these. Especially if the
semantics of your terms slightly deviate from common or standard
definitions.

Provide a glossary of such terms.

Explain reasons

Many decisions taken during development of a system are reflected in its source code. What’s missing though, are the explanations or reasons for these decisions.

Therefore, consider documenting the reasons and explain why the
development team took a particular path. Start with explaining the
requirements and constraints for a specific decision, then explain the reasons for the decision.

Use diagrams to explain structures

Structure: the way that something is built, arranged, or
organized

https://www.merriam-webster.com/dictionary/structure

Diagrams or graphical models are well-suited to convey structural
information, like building-blocks (aka components and their
dependencies, hardware elements and their relationships or even dynamic structures like processes.

Use them with care - as changing or updating diagrams is harder than
updating a short textual description. On the other hand, structural
information is traditionally conveyed via diagrams.

Restrict diagrams to higher abstraction levels. Prefer to omit details from your diagrams, instead of trying to explain low-level structures in diagrams. Stick to a single topic per diagram.

Explain your notation (or use a standard)

Graphical models or diagrams are an elegant way to convey structural
information, they can improve visual effectiveness.

Visual effectiveness is a measure of how the appearance of information and the use of visual elements within it affect the ease with which users can find, understand, and use the information.
³,p.277

Every symbol in a diagram should have a clear semantic or meaning. That makes the diagram unambiguous.

Either you provide a legend which explains the meaning of every symbol type, or you rely on a standard graphical notation, like UML⁴, C4⁵ or ArchiMate⁶;⁷.
If you rely on a standard, reference it in addition to the legend.

Combine diagrams with text

A diagram itself might leave room for interpretation, unless amended by textual or tabular clarification. See the following example, where component names give hints on their actual responsibility, but terms might be misleading. What is a “Reporter”? A journalist? A component creating a report, or delivering it?

Fig. 3.2: An example diagram that needs explanation

A table explaining important elements from a diagram can help to remove
such ambiguities - see for yourself:

Element	Responsibility / Meaning	Details
`Checker`	performs the actual checking, relies on the status- and error messages of the `Parser`	org.aim42.hsc.core
`Parser`	open-source library to parse HTML files	org.jsoup
`Suggester`	suggests solutions to problems found during checking	org.aim42.hsc.core
`Reporter`	creates a hicely formatted html report of all problems found during checking	org.aim42.hsc.core

Use helpful structure

The structure of technical documentation should be organized top-down, beginning with an overview, and gradually adding details.

Approaches like arc42⁸ or C4⁵ propose structures that are highly practical, established and free to use.

Req-4: Relevant (aka task oriented)

Documentation shall serve the purposes of its target audience, see Know your target audience above. They have specific goals or tasks that can be enabled or supported by technical documentation.

Relevance depends on the goals and tasks of the target audience.

Let’s look at some potential stakeholders, their goals and documentation needs:

Stakeholder	Task/Goal	Documentation
New developer	Efficient onboarding	Overview of requirements, solution approaches, building block structure, technical concepts, important decisions
Development team	Guide implementation	technical concepts, decisions, building block structure, some runtime scenarios
—“—	Take decision	e.g. quality requirements
Tester	Test parts of the system	some internal interfaces, technology overview
Neighbour system	Use external interface, e.g. API	context, technology overview, API details
Management	Budget decisions	context, technical decisions, used 3rd party frameworks
Auditor	Check compliance	often: deployment view, technical decisions, high-level building blocks
External reviewer	Due diligence	solution approaches, high-level structure, technical concepts

How to improve relevance

Identify tasks of target audience

You need to know what information the target audience needs from the
documentation, (see above). Create a
stakeholder table to contain information about each stakeholders’ tasks
and/or goals, so you can better estimate their documentation
requirements.

Show examples of existing documentation

It is highly effective to show each stakeholder examples of existing
documentation: Based upon specific examples, stakeholders can point out
what they need in addition, or what parts they are not interested in.

Beware of the “I need everything” trap: When stakeholders see detailed
and well-organized documentation, they might want “everything” - even
though they don’t need parts. Therefore, it might be more efficient if
you show only those parts which you estimate people will really need!

Req-5: Referenceable

Referenceable elements can be uniquely identified in the context of your
documentation. Instead of writing “see the diagram below” or “the table
above”, you should strive for “see diagram 5.2” or “the table 3.1”. You
can chose any numbering or referencing scheme you like, but you should
use it consistently.

How to make documentation referenceable

Number every section and heading of technical documentation, at least to heading level 2.
Number every diagram or image. Prefix diagram numbers by the number/id of the appropriate section. For example, the first diagram in section 3 shall be labelled “Fig. 3.1”.
Number every table, again prefixed by section-id/number.
Provide a table of contents (toc). Electronic or online versions of your documentation should contain hyperlinks within the toc. For paper or printed versions, use page numbers.

You should use appropriate tools to facilitate
referencing.

Req-6: Proper language

Your target audience shall have no reason to complain about spelling or grammar mistakes, but shall instead focus on content. Proper language will also improve readability and understandability.

How to improve documentation language

Language, wording and style of your documentation should be adequate for your target audience. Sentences should be grammatically correct.
Text should be formulated in a factual way and avoid emotional or offensive language.

Some specific advice:

Use positive statements, aka avoid negation. The human brain needs significantly longer to understand negation⁹.
Write concisely, aim for brevity. The average length of a sentence should be 15-20 words.
Use the active voice, as it makes it easier for our brain to comprehend and remember.
Aim for correct spelling and grammar.

¹⁰ contains several examples and explanations of “proper language”.

Req-7: Maintainable

The structure and organization of technical documentation should be
optimized for maintainability - so that keeping the documentation
current despite frequent changes becomes as easy as possible.

As already mentioned above, many aspects of systems change -
requirements, source code, configuration, deployment, operations, external interfaces, etc. These changes shall be easy to incorporate into the technical documentation, therefore maintainability is an important requirement.

Please note: We refer to maintainability of the documentation, not
of the system itself.

How to improve maintainability

Abstract: Omit (some) details

Leave out certain facts: Explicitly decide to not document irrelevant or overly detailed facts. Leaving out some facts reduces the amount of content you have to keep up-to-date in the future. Only document facts that:
- you or your team can promise to maintain in the future.
- important stakeholders need to know to do their work
Document abstractions instead of details. That is a special case of not documenting certain facts, namely the details. But what kind of details could be left out of documentation? Please note that I wrote could be left out, not shall be left out. In some cases you will want or need to document details!

instead of…	try this…
trying to name/show all elements	aggregate elements, especially when you can give explicit cohesion criteria
giving a specific version number	omit that version number or specify a range like “version >2.1”
showing all children of an element	omit children, point to parent
showing all dependencies between two elements	show just one (aggregate) dependency (see below for an example)
explaining interface details like parameters	just name the interface, omit details

You see the underlying principle: Leaving out details facilitates change
and will make your documentation more robust against (at least some)
changes in requirements, code or infrastructure.

Examples of omitting details

Fig. 7.1: Abstractions instead of details

In the first example, we condense three building blocks (A,B,C) into a
single one. At the same time, we reduce the details of explicit
dependencies between foo and A,B,C from 9 different to one. As this
single dependency carries several responsibilities (remember, it serves as an abstraction for nine detailed relationships), it’s a good practice to give such dependencies a name and explain some of the details in text or a table.

Fig. 7.2: Example: Abstraction in runtime view

The second example shows a detailed sequence diagram containing three participants (b1, b2 and b3), having some interactions among themselves and a few with the rest of the system (a and c). In the abstraction I made use of an interaction reference to hide details, see ¹¹ or ¹².

Keep the source of every diagram

Diagrams are created using appropriate tools. Often diagramming or modeling tools use proprietary binary data formats, which are converted and exported to a format like png, jpg, svg or pdf.

To be able to later modify diagrams, you need the original (source) representation.

At best, the source of diagrams is versioned and referenced from the diagram.

Regularly delete outdated parts

If something has become common knowledge, you may delete it from your written documentation.

In one of my projects, I tagged such parts as “to-be deleted”. In case nobody from the team objected within a week or so, that content would be removed from the active documentation and moved to an archive instead. Such tagging works great in some wikis.

Automation won’t improve your content

You can (and should) automate data collection, transformation, or even some visualization, which we will discuss in the section on tools later on.

But: The intellectual process of creating valuable content for technical documentation cannot currently be automated.

Documentation generated from source code will be restricted to
high-level abstractions of this source code. It cannot give reasons or explain things, therefore it will always miss critical information.

See ¹³ for a
coverage of automated (living) documentation.

Req-8: Easy to find

Finding information is essential. The organization, retrievability,
and visual effectiveness of information contribute most to whether
users can find the information that they need.
³,p.213

This requirements relates to both the content of the documentation and
to the documentation itself. In reality, documentation is often located
in access-controlled areas that are not covered by any search engine:
That means you have to know where to look for the documentation.

How to make information easier to find

Ensure the documentation itself can be easily found, for example by providing entry points at well-know locations (i.e. the start- or homepage of the product or team wiki).
Use a template, like arc42 ⁸
Organize your documentation top-down, see place details into context above
Use cross-references and hyperlinks where possible, so readers can jump within your structure
Provide a document overview that explains the structure of your documentation. For standards, like arc42, such overviews already exist.
Facilitate navigation and search, see Req-9: Easy to find

Finally, you should accept some repetition in documentation to make it
more comfortable for stakeholders (See ¹⁴, ARID).

Req-9: Version controlled

As there will be several people actively contributing to technical
documentation, you should version-control its artifacts similar to
source code:

It shall be possible to rollback certain documentation changes.
The ability to work on different versions of the documentation (e.g. in different branches of a version management system) will be useful, especially in larger or distributed development teams.
Documentation shall closely correspond to the identified versions or releases of the system. In case somebody needs to go back in time, for example to fix a bug in an older release of the system, the documentation of that older release shall be available without much effort.

How to version-control documentation

Obviously the straightforward approach is using version control systems
(VCS), like git, subversion or similar tools.

But in reality that’s not always as simple as it sounds:

Organizations might prefer proprietary systems for documentation, for example wikis (e.g. Confluence™).
Documentation authors, especially from non-technical roles, might not be able (or refuse to) use a VCS.

Therefore I propose the following strategy:

Ideally, add all documentation artifacts to a VCS, like git. Your VCS should facilitate branching and merging artifacts. Additionally, it should allow for versioning of binary artifacts.
In case your documentation tools produce binary or proprietary artifacts, which cannot easily be added to your VCS, take the following path:
- Export all release versions of your documentation (those you need to keep) to PDF. Ideally, the result is a single PDF file containing the complete documentation to a specific release or version.
- Add the resulting PDF to version control.
- Automate the preceding two steps (creating and adding the PDF). See automate certain parts.

Req-10: Use appropriate tools

You need a toolchain that satisfies a number of requirements. Let’s start with text processing:

structured text, with different heading levels
lists (numbered and/or bulleted)
cross-references
include images/diagrams created by other tools (at least png/jpg, svg/pdf recommended)
tables
split documents into several parts
support reviews by others than the author(s) with review comments and suggestions
the ability to generate numbers/ids for diagrams, tables and sections/headings

As you should create diagrams or models (see Use
diagrams), your diagramming tool shall support:

static structures, like components, building-blocks or technical infrastructure
dynamic structures, like activities or processes
any standard notation like UML ⁴, C4 ⁵ or ArchiMate ⁷
legends in diagrams, to enable Explain notation
flexible layout, so you can improve visual effectiveness of diagrams (e.g. place important elements in the center, leave appropriate whitespace around elements, use fonts or color to emphasize aspects)
links or cross-references to related diagrams

Prefer established tools

For development teams it might be fun to use the latest and greatest toolchain, with this and that plugin to spice up the otherwise boring task of documenting the architecture.

But: Any toolchain needs a certain amount of maintenance: Updates, backups, user support, and other mundane activities will require somebody’s attention. If you install and operate your own tooling, it might be you who needs to invest this time. If you rely on established tools, your company or organization will provide this support.

I’m a documentation tooling fanboy. I often install and test new tools, so I can develop my opinion and give better advice to my clients. This advice is, more often than not, directed towards the boring toolchains that already exist, where maintenance, backup and support are already sorted out.

In exceptional cases…

If (and only then!) there is no appropriate and usable toolchain available for documentation - you need to introduce one. Such a toolchain requires documentation, maintenance and support effort.

Automate certain parts

You should try to automate certain parts of maintaining your
documentation, to keep it current.

Some candidates for automation are the following:

creating platform-neutral representations (like PDF) of the complete documentation
publishing the documentation, e.g. in a browseable format
testing the sanity or consistency of your documentation, e.g. by checking hyperlinks

For the generation and publishing part, have a look at docToolchain ¹⁵, a lightweight, open-source approach to create and maintain technical documentation. It can read several data formats (AsciiDoc, PowerPoint®, Excel®, a variety of image formats etc) and generate numerous output formats, including static websites.

Req-11: Continuously updated

Deferring documentation to later is like procrastination, or saying “I’ll start healthy living next week.” Start right now, in small chunks. You decided to upgrade that persistence framework? Create an ADR for this decision. The team decided to split that module in two separate parts? Make an appropriate note in the building block view explaining the reason for the split.

How to enable or facilitate continuous documentation

Include documentation task in your Definition-of-Done (DoD)

For many agile teams that work in iterations, the Definition-of-Done “ensures everyone on the team knows exactly what is expected of everything the team delivers” (quoted from ¹⁶)

If something is not contained in the DoD, it won’t be done. Therefore you need to make sure that technical documentation is contained in your DoD.

Reserve small timeboxes for documentation

Invest a few minutes per day, or 15min every Friday to create, maintain or improve your documentation.

Document incrementally

Start with an overview, a birds’ eye perspective. Drill down into details only if somebody is willing to maintain these details.
Start small and add facts when they have gotten robust enough. If certain aspects or elements are still under discussion or investigation, don’t include those in your documentation.
If your team is experimenting with a certain technology, framework, structure or concept, defer the details to a later point in time.
Maintain a documentation backlog, where you collect open issues for your documentation.

Summary

Instead of adding even more text, I provide a graphical summary:

Fig 12.1: Pattern map for technical documentation

You can download a high-res pdf version from arc42.

Acknowledgements

Thanks and praise to my knowledgeable and articulate reviewers Ralf D. Müller, Joachim Praetorius, Marco Stroppel and Peter Hruschka.
Header image by (Retrosupply)[https://unsplash.com/photos/jLwVAUtLOAQ].

References

Map from Wikimedia Commons, provided to Wikimedia by Geographicus Rare Antique Maps, a specialist dealer in rare maps and other cartography. It was first drawn by Daniel de la Feuille in 1706. ↩
Edembourg actually was the correct spelling back in the 18th century... ↩
G. Hargis, Ed., Developing quality technical information: A handbook for writers and editors. Upper Saddle River, N.J: Prentice Hall Professional Technical Reference, 2004. ↩
"UML 2.5 Diagrams Overview." [Online]. https://www.uml-diagrams.org/uml-25-diagrams.html ↩
S. Brown, "The C4 model for visualising software architecture." https://c4model.com. ↩
"ArchiMate Quick Guide." [Online]. Available: https://archimatetool.gitbook.io/project. ↩
VAN HAREN PUBLISHING, ARCHIMATE(R) 3.1 SPECIFICATION. S.l.: VAN HAREN PUBLISHING, 2019. ↩
"Docs.Arc42.org." https://docs.arc42.org/home/. ↩
M. A. Sherman, "Bound to be easier? The negative prefix and sentence comprehension," Journal of Verbal Learning and Verbal Behavior, vol. 12, no. 1, pp. 76–84, Feb. 1973. ↩
C. Hibbard, "Six Principles of Technical Writing." ↩
"Sequence Diagram with References." https://sparxsystems.com/resources/gallery/diagrams/software/sw-sequence_diagram_with_references.html. ↩
"UML Sequence Diagrams - Graphical Notation Reference." https://www.uml-diagrams.org/sequence-diagrams-reference.html. ↩
C. Martraire, Living documentation: Continuous knowledge sharing by design, 1st edition. Boston, MA: Pearson Education, Inc, 2019. ↩
"Documentation Principles — Write the Docs.” https://www.writethedocs.org/guide/writing/docs-principles/. ↩
R. D. Müller, "docToolchain." http://doctoolchain.org/docToolchain/v2.0.x/. ↩
J. Sutherland, "Definition of Done," 07-Aug-2014. https://www.scruminc.com/definition-of done. ↩

Awesome presentations deserve beautiful code

Dr. Gernot Starke — Thu, 20 Jan 2022 11:43:30 +0000

Ever seen ugly source code on a slide?

Welcome to reality. Every now and then we all need to put code samples onto PowerPoint or Keynote presentations.

But wait - presentation programs are made for text, not for code:
They loose all syntax highlighting when you paste your valuable code into their limited textboxes...
Look at this horrible example:

Fig.1 - Slide with ugly code.

I like to introduce carbon - a free website that elegantly solves this problem. On their website the creators describe their goal short and concise:

"Carbon lets you create and share beautiful images of your source code".

You see a possible result below:

Fig.2 - Slide with beautiful code.

What a difference!

Nice'n Easy

Carbon is incredbly easy to use: Simply paste your code in the textbox or drag a source file.

Carbon has a number of interesting configuration options available: Themes, borders, (fake) window controls and more.

Predefines themes

Several themes are available, all except the Darcula-Pro themes are free:

Fig.3 - carbon theme (Monokai).

If you prefer a lighter theme, you are covered too:

Fig.4 - carbon theme (OneLight)

In case you need to, you may even configure your own custom theme. It's a lot of work, and for me the standard themes are way good enough.

Additional Layout Options

You can set padding, border color, shadow, width, (artificial) window controls and a few more options:

Fig.5 - some config options.

Export to png and svg

As expected, carbon can export the image of your source as png or svg:

Fig.6 - export options.

Alternatives

All right, you may also copy/paste source code as RTF (and hope that your presentation tool really preserves formatting).
You could either paste your code to Microsoft-Word (crossing fingers again for preserved formatting), and export it to PowerPoint from Word. Might work, but is less fun. And in contrast to carbon.sh, it does no good to our atmosphere :-).

But: Accessibility Issues

When you convert source code to images, you have to (manually) take care of accessibility.
Images like the ones generated by carbon.sh are not suitable for screen readers.
If you want to adhere to accessibility good practices, you should therefore include the source code shown in the alt-tag of the generated images.

Enjoy!

You never again need to include source code as plain and ugly text in any of your presentations!

One final remark: The name "carbon" was chosen by the authors because they want to reduce CO2 in the atmosphere, read the small print (...offsets) on their website.

Thanx

Kudos to Joachim Praetorius for improving my English and Lars Hupel for pointing out the accessibility issues!

Thanx to Miguel Henriques
for the header image.

Authoring Markdown with Zotero - My Workflow

Dr. Gernot Starke — Sun, 16 Jan 2022 06:58:27 +0000

This post describes an authoring workflow that combines the simplicity of markdown (for writing) with the power of a reference manager (for citing and generation of a bibliography).

This is my current writing workflow:

Collect references, articles, books etc. via Zotero, my citation manager. You might call this the “research phase”.
Edit article/text/book using markdown editor.
Zotero exports your references to a file (continuously in background)
Use make to compile markdown to final result:

4a. pandoc reads your markdown source file
4b. pandoc reads the bibliography file
4c. pandoc formats the bibliography entries following the desired citation style, described in a .csl file
4d. pandoc creates desired output format(s), like docx or html.

You like more details? Then follow along…

Why Markdown (sometimes) isn’t enough

I don’t need to advertise the advantages of markdown as free,
lightweight, simple yet powerful-enough approach to authoring arbitrary content like articles, blogposts (like this one!) or even books.

But: Markdown is optimized for writing, not for citing. If you refer to other sources, like websites, blogs, books or articles in your writing, you want to systematically reference these sources. An example:

See the original article about Markdown
[1] for further info.

In markdown source, this would be the following:

See the original article about Markdown ([@gruberDaringFireballMarkdown]) for further info.

For a blog post, one could simply include the hyperlink, but for any
printed publication a real reference is more appropriate, often
required by publishers. Such references in your bibliography will look similar to the following example:

[1] J. Gruber, “Daring Fireball: Markdown.” https://daringfireball.net/projects/markdown/.

In case you write just a single article, this isn't a big problem: Manually add your sources at the end of your text, and you’re done. But if you (like myself) publish regularly, you might want to have a central repository of your references including publication dates, publisher info, links etc. That’s where Zotero comes into play:

Zotero

Zotero is a free, open-source and user-friendly tool to help you collect, organize and cite references,
like articles, books or research papers (see [2])

Zotero is available for all major platforms (Win, Mac, Linux).

You can sync your reference library across multiple devices and add new references either by ISBN, DOI or similar ways. For online publications, powerful browser plugins are available to add any webpage to your library, together with a snapshot for offline viewing!

Required Zotero Plugins

You need to add BetterBibTeX (short bbt, see [3]) to your Zotero installation.
bbt can export your Zotero library (or a subset of it) to an arbitrary file system location - and keep this export updated! Wheneve you add a new reference to Zotero, this exported file gets updated too!

Adding References to a Text

I like the editing experience of VS-Code, mainly for its rich plugin universe. One of these plugins is the Zotero-Citation-Picker:
It gives you a search box and retrieves results directly from your Zotero library.

When you have found the correct reference, the plugin enters the unique citation key at the current cursor location.

Compile the Bibliography

Now you have a markdown text containing several of these “citation keys” - but still no list of references, the so called bibliography*.

That’s where the next player enters the field: pandoc (see [4] ).

Compile Output with Pandoc

“If you need to convert files from one markup format into another,
pandoc is your swiss-army knife.”

Although you could use pandoc directly from the command line, I prefer good old make, as I cannot remember parameters and options. Look at my Makefile:

OUTPUT=out
SOURCE=content.md # name of your markdown source file

CITATIONSTYLE=ieee.csl
BIBLIO=~/Dropbox/zotero-export/zorg-biblio.bib
TARGET=$(OUTPUT).docx $(OUTPUT).md

PANDOC=pandoc \
   --from=markdown \
   -s --bibliography $(BIBLIO) \
   --citeproc \
   --csl=$(CITATIONSTYLE) \
   --metadata link-citations=true \
   $(SOURCE)

# create docx
$(OUTPUT).docx: $(SOURCE) $(METADATA)
   $(PANDOC) --output=$(OUTPUT).docx


# create github flavored markdown
$(OUTPUT).md: $(SOURCE) $(METADATA)
   $(PANDOC) --to=gfm --output=$(OUTPUT).md

all: $(TARGET)

clean:
   rm -rf $(OUTPUT).docx $(OUTPUT).md

.PHONY: all clean $(TARGET)
.DEFAULT_GOAL := all

(This Makefile is available as Github-gist)

Yes, I know: One could wrap pandoc into a tiny Docker container - but pandoc is easy to install and is incredibly fast. A Docker container would remove the dependency to pandoc, but add the overhead of a running container.

When you try this on your own, please fulfill the following
prerequisites:

install pandoc
add the citation-style csl-file to your current directory (I didn’t manage to keep this file in any central location - help or suggestions would be appreciated)
add the Makefile to the current directory
update the BIBLIO variable in the makefile to point to your Zotero export location.

Image Credit

Splash image by Trent Erwin on Unsplash.

Bibliography

This bibliography has been auto-generated by the workflow shown above.

[1] J. Gruber, “Daring Fireball: Markdown.”
https://daringfireball.net/projects/markdown/.

[2] “Zotero | Your personal research assistant.”
https://www.zotero.org/.

[3] E. Heyns, “Better BibTeX for Zotero.” Dec-2021.
https://retorque.re/zotero-better-bibtex/

[4] “Pandoc - About pandoc.” https://pandoc.org/.

Merge PDF files with QPDF

Dr. Gernot Starke — Sat, 13 Nov 2021 13:31:15 +0000

The Problem

You want to merge (concatenate) several PDF files.
It is not possible to just append one PDF after the other, let's say by command-line append operations.

Other simple PDF utilities fail in preserving internal hyperlinks (cross references).

I needed a programatic solution that works completely automated (controlled via make) within a Docker container.

Nerdy Solution

QPDF is an extensively documented tool for:

"structural, content-preserving transformations on PDF files."
(from their website)

One (very basic) use case is merging two or more PDF files into one single output file.
To be more specific: I want to add a cover page to the main part of a book, as illustrated below.

I write the book in Markdown and use a TeX-based toolchain to generate PDF output (that's another story).

The book itself contains numerous cross-references (internal links), pointing to other sections of the book or to specific pages. These links must be preserved during merge - and my initial approaches lost all those links.

QPDF to the rescue:



qpdf book.pdf --pages cover.pdf 1 book.pdf 1-z -- result.pdf

It took me a while to figure out the exact syntax - and wrote this post so I (and other lost book-writing soules) can find it later.

Beware, though...

I suggest to keep the dimensions of book- and cover page identical (obviously you will care for that anyway).

Manual Solution, MacOS only

In case you are working on MacOS, you can simply use the preview tool to concatenate PDF files - in most cases hyperlinks and cross-references are preserved.
But: It is a manual solution.

Thanx

Thanx to the awesome Jay Berkenbilt for creating and maintaining QPDF since 2005 (!!).

Thanx to Wesley Tingey for the cover image, via Unsplash.

Quality-Driven Software Architecture

Dr. Gernot Starke — Tue, 27 Jul 2021 14:35:30 +0000

Getting your product quality right

Quality is the raison d'être for software architects:
Our systems should be reliable, performant, scalable and user-friendly. Systems should be build and maintained cost-effective and future-proof. Every IT professional knows that this combination of characteristics means hard work.
The article shows how you can methodically construct quality.

Tea with sugar?

How do you drink your morning tea: With sugar or natural? With a drop of milk and some honey? Do you prefer it with lemon? Black or green, hot or with ice? Or do you prefer coffee in the end? As you can see, the requirements for breakfast drinks, just like those for software, depend very much on the
viewer and the context. If we want to create a high-quality morning beverage (or a system in general), we definitely need to determine the specific (quality) requirements of the relevant stakeholders beforehand. Overly general requirements are only of limited help.

Nowadays, customers and users expect software to meet a large number of (quality) requirements, of which "correct function" is only one among many. In classical software development, however, functionality is often treated as the center of attention: The design and implementation activities are focussed on getting the functionality right - with potentially fatal consequences. Let us illustrate this with a small example.

Functionality is (usually) not enough

Since the triumph of digital cameras and smartphones, practically all of us have felt the need to somehow organize the many individual images on the computer.
Let's use this as an example of a requirements description for a software system: manage digital photos.
Imagine you are the software architect who shall develop such a system together with a small development team.
The customer tells you the following story:

"We want to organize photos in folders and tag them with keywords.
Later, we want to be able to search and display images according to various search criteria (such as: date, keyword, etc.). In our private collection, we can get by with a few thousand photos."

As an architect, you might reformulate these coarse statements into some features
(In real development, somebody like your product owner should handle that...)

add images
add image to folder
delete images
display images
search images for location, date, keywords, folder
change metadata (like location, date, keyword)
add folder
rename folder
remove folder

(yes, I know that it would be nice to have additional features like albums, but let us keep things simple for now)

Fig. 1: Sample photo with some attributes

Now you need to derive a solution approach from these (admittedly very rough) requirements.
A model helps us to clarify things. We can derive it from an example (see Fig. 1).
In our case, each photo has only a handful of attributes (date, location, file name, folder, etc.).
We map the keywords as a list so that each photo can have several.
Again, I'm well aware that this approach is overly simplistic - but give me a minute.

Fig. 2: A model for a "solution approach"

In our example, the domain model seems almost trivial (see Figure 2).
Only three small entities are sufficient to represent the desired functionality from a purely business perspective. Location, date and keywords are stored via key-value pairs in the Metadata class. It could hardly be simpler.

Based upon such a model, architects and develpers can start implementing business functionality and corresponding tests.
In our example, we would need to implement the classic CRUD tasks (create, read, update, delete) for our domain entities (photos, folders, and metadata), and additionally solve a few technical tasks to bring the photo system to life:

display jpeg files
create a graphical user interface for our features

From an architects' perspective, it is valid to ask the "make-or-buy" question: Has anybody else already solved my problem, and could we potentially buy or re-use an existing solution?
The answer is obviously "yes", as several commercial and open-source packages exist for photo management,
e.g. Shotwell, Darktable and Lightzone, see fig. 3. But that would be pretty boring for us developers, so we build the solution ourselves (despite it is usually very clever and efficient to prefer existing solutions over home-grown stuff...)

Fig. 3: Options for Photo Management

But ...

The customer is really happy that you delivered with a week and plays around with your solution. With a big smile on their faces they comment:

"It is great, but we require a small change."

Only three letters, sounding insignificant, that "but".
Our customer, excited by what they saw, adds the following requirements:

"We want to roll out our photo management to multiple users and provide 10Gbyte of free storage for each of them. Of course, people shall be able to share photos amongst each other. Otherwise, don't worry - the functionality will remain identical..."

Ok. We should have known there was something else coming - otherwise they wouldn't have asked us in the first place. So, not only local storage on users' machines any longer, as users get 10 Gig for free. Sharing photos among users definitely adds some more functional requirements - but having centrally managed storage and multiple users changes the whole scope and context of our development.

As you are an experienced architect and therefore know about "Murphy's law", you ask the customer for an expected number of users. They answer promptly: Millions, hopefully dozens of millions of users.

Millions of users

A quick calculation reveals that we now have to re-think our file-based storage approach:
let's assume we get 10 Million users, and everyone will add only half of the allowed free storage (5 GByte) of images to our system. Without considering backup and index space, that alone will require a quite impressive amount of storage:

    10 Million * 5 GByte 
  = 10.000.000 * 5 GByte 
  = 10.000     * 5 Terabyte
  = 10         * 5 Petabyte

50 Petabyte. Defintely more than we can squeeze on on standard SSD drive, and likely to become pretty expensive if we use a standard cloud storage provider.
Additionally, with a million of potentially concurrent users, we should think about concurrency, scaling, elasticity and a few other things not contained in our domain model.

Interestingly though, our domain model remains extremely simple, see the diagram below (Fig. 4)
Just a single domain class is added (User), denoting the owner of the photo, plus a relationship to share photos with other users.

Fig. 4: Enhanced Domain Model

Practically, we will now need some changes to functionality, namely all things related to user management (add/update/remove users, handle authentication, handle forgotten credentials and so on).

Just a single knob...

In contrast to the original requirements, no (!) relevant functionality was added, just a single quality attribute was adjusted: We had single-user operation before, and did not explicitly care for storage capacity. Our customer just turned a single knob ("capacity"), resulting in imporant under-the-hood changes.

That tiny change in requirements resulted in a completely different solution approach.
Compared to the simple functional requirements, even hardened architects and developers now suddenly get concerns: millions of users and petabytes of data confront a development team with completely new tasks:

Scalability to keep the system perfomant and responsive even with the planned number of users. This includes clustering and load balancing to be able to react to load peaks at short notice.
Storage of large amounts of data (new German Big Data), which does not work satisfactorily with conventional databases or file systems.
Distributed data storage, because for performance and risk reasons the entire data stock should be distributed over several locations or data centers.
High availability, because large numbers of international users do not tolerate long maintenance windows
Transmission costs, for example for replication of image data.

What's missing in Domain-Driven Design

A purely domain-driven approach to the design and development of our photo example might have ignored the essential quality requirements (number of users and data volume). In general, if there is too much focus on functionality and domain structure, there is a risk of overlooking important quality features or ignoring them for a long time.

Parallel to understanding and considering the domain you should therefore explicitly design the required or needed qualities into your system. But let's consider a relevant objection first:

In case you think "hey, we are nor adressing millions of users, neither giving away petabytes of free storage, this example is exaggerated".

Ok, let's turn another knob

If you are of the opinion that the quality requirements mentioned here are too ambitious, because you will never have millions of users yourself, let's choose a completely different requirement:

Your customer wants to store images for a few dozen employees, each with a maximum of one hundred images.

This results in a total volume of data that can easily be stored on a single hard disk. We can easily manage a few dozen user IDs without any capacity or storage problems. However:

The stored images are top secret (military grade security)

Again, we have to deal with problems beyond pure functionality: secure encryption algorithms, secure transfer of images to users, key- or credential management, highly secure authentication of users. These tasks are also difficult to retrofit to an already finished software.

What is "Quality"?

The term "quality" was subject of another short post here on dev.to.

In the past, many researchers and practitioners have discussed options how to treat "quality" of software based products. Their common approach is to treat quality as a hierarchy of smaller properties:

The well-known approach of ISO-25010 divides quality into eight categories, each of them again by additional characteristics (see Fig. 5).

Fig. 5: ISO 25010 Software Product Quality

Our photo management system from the introduction serves as an example: We now describe the (quality) requirements as a hierarchy: The required software quality is decomposed into subcharacteristics, which in turn are further decomposed as required. This so-called quality tree describes or defines the specific quality requirements for a system. In Fig. 6 below we turned the tree leftwise, and removed most of the branches from the "big" ISO-25010 model.

Fig. 6: Sample Quality Tree

These textual descriptions of specific quality attributes (e.g. "Photo search yields results in < 1sec on average") are called quality scenarios and has been described in many books and articles, see this nice post from Frank Rosner. Scenarios provide a practical and pragmatic way do describe quality requirements, they are easy to understand and create.

Achieving Quality

So how can you achieve this, often quite extensive, set of quality requirements for a system? We have already seen that concentrating on the purely functional or technical characteristics is often not enough.

The short answer: Explicitly consider quality requirements - don't expect that the system will be fast, secure, reliable or maintainable enough just by producing clean code.

Systematically Constructing Quality

As a software architect, you must therefore place great emphasis on achieving the important quality characteristics from the very beginning when designing and implementing your systems. Fortunately, this is quite simple in practice:

First of all, you should make quality a very important goal and pay attention to it accordingly. Then follow a classic plan-do-check-act cycle (analogous to the classical PDCA Deming cycle) with the following steps:

aim, getting specific and concrete quality requirements.
plan: define (together with your team) approaches or strategies to achieve the respective quality requirement. Approaches might be purely technical (e.g. use Elasticsearch for image search queries), organizational, business-related (e.g. restrict free storage to 1 GByte in first year of usage), or a mixture of all.
build: Implement some of these approaches.
check or test: Check the effectiveness of your approaches. To do this, you need to measure and test as close as possible to the running system or parts of it. If necessary, you now need to re-adjust. To do this, continue with step 1 (aim). This cyclical iterative procedure elevates the systematic review of goal achievement to a system. It ensures that high-priority quality requirements are taken into account in a timely manner in design and development. Neil Ford calls this "evolutionary architecture."

Fig. 7: Quality Driving Architectural Decisions

The build and check activities are completely normal architecture and project business - but the aim and plan activities require some explanation.

Aim: Getting Quality Requirements Right

In the ideal world, you receive concrete and specific quality requirements from requirements engineering or your product owner. However, in my long years in the IT industry, I have rarely encountered ideal conditions - I regularly had to rework the quality requirements for systems. The simple reason for this: The functional requirements and processes of a system can usually be described quite simply, but clients and other stakeholders usually have a hard time with the specific, precise description of maintainability, flexibility, robustness, ergonomics, efficiency and so on (the qualities of systems).

There has been a remedy for this problem for a long time: Methods such as quality scenarios can help to describe quality requirements and goals in a pragmatic and operationalized way. We have already roughly addressed this above, here again in more detail:

You remember the hierarchical nature of quality: quality scenarios form the leaves from the quality tree. We concretize these leaves with scenarios in order to describe more precisely what the respective stakeholders mean by this term. Several different scenarios may well refer to a single quality characteristic. Again, instead of gray theory, a few examples (in Table 1) will help you understand this approach. In general, scenarios describe the reaction of a system to events. Many quality characteristics can be well concretized with the help of scenarios, especially efficiency, performance, flexibility, extensibility and reliability.

Feature	Subfeature	Scenario	Priority
Reliability	Robustness	When uploading a corrupted jpg photo, the system gives a meaningful indication without crashing.	B
	Robustness	When uploading an unknown file type (such as: pdf, svg, tiff), the system gives an appropriate message and saves the data without crashing.	A
	Data Integrity	The system will not modify or corrupt photos uploaded by users under any circumstances.	A
Performance	Capacity: Number of Users	The system supports up to 10 million users, each of whom is allowed to use up to 5 GB of storage for photos and metadata.	B
Ease of Use	Learnability	Casual users should be able to save their first photo in less than 2 minutes.	B
	Learnability	Users should be able to create more complex searches without the help of documentation	C

Table 1: More Examples for Quality Scenarios

Please note the "Priority" column in Table 1: This puts the quality requirements in an appropriate order from a business point of view. The finished system will achieve all quality goals, the priority helps during for architecture and development in design and implementation decisions.

Now that we have clarified the quality requirements and made them operational and hopefully testable - we move on to the next step, strategy, tactics or other approaches to achieve these requirements.

Plan: Define Approaches to Achieve Quality

Once we know the specific requirements, we can design, decide and define specific approaches, measures, concepts and technology to achieve these requirements. In addition to established good practices of software engineering, we can rely on published strategies, tactics and patterns.

The well-known Software Engineering Institute has introduced the term "quality tactics" for this, and published a few for individual quality characteristics.

To date, there is no complete collection of such quality tactics, but the current literature provides you with a large pool: The patterns that have been so widespread for a few years already help for the quality characteristics flexibility, robustness, performance and security, etc. - as long as you consider them with a focus on quality-enhancing measures.

QDSA in Practice

In development practice, QDSA consists of really simple mechanisms:

Write down your stakeholders' explicit quality requirements.
Design, decide and define specific approaches, tactics or strategies to achieve these specific requirements.
Implement one or several, and finally
Check if your approach(es) worked out as planned.

A simple board or table in your project wiki suffices, no additional tools required.

QDSA for Photo Management

Let's come back to our photo management example one last time and dedicate this way to our demanding quality goal "high number of users and large amount of data". Table 2 contains some proposals how your team might achieve that goal. It's still a lot of work to do, but at least we now have collected some ideas which we can pursue during development.

Q-Requirement	Proposales / Approaches / Tactics / Strategies
Capacity (Number of Users): System supports up to 10 million users, each is allowed up to 5 GB of storage.	* NoSQL datastore with auto-replication * Use content-delivery network * Use Auth0 for user management * hire expert for large-scale load testing
Learnability: Casual users should be able to save their first photo in less than 2 minutes.	* get usability expert on team * add drop image function for upload * design-thinking workshop to optimize flows
...	...

Some of these proposals are expensive, others are difficult to implement. But if your customer has difficult requirements, sometimes you have to take big leaps to achieve these goals! Brainstorming, collecting and systematically analyzing proposals for action is the core of Quality-Driven Software Architecture.

Beware of Side Effects

In QDSA, you should be particularly careful at one point: Individual measures can easily impact other quality attributes. A positive effect on one characteristic such as performance can easily have negative effects on other characteristics, such as memory requirements, robustness, or comprehensibility. That's the classical side-effect: There is no free lunch. Many things you improve will have an impact on budget or schedule. You should therefore systematically question possible consequences before you start implementing your quality approaches.

The rest is classic development business, namely planning the measures in everyday life and then implementing them. Whether you work according to Scrum or other process models, with regard to QDSA there are no special features compared to normal procedures. From my point of view, the combination of DDD and QDSA is very promising: On the one hand, you get your domain structure under control, on the other hand, you explicitly address the important quality topics. Your customers or clients will be thrilled!

Quality Creates Value

Constructing quality systematically in this way creates value for your stakeholders: software systems that achieve specific quality goals. As mentioned above, many quality-enhancing approaches or strategies are initially expensive, costly or even risky with regard to other goals. In the long term, however, such investment in achieving quality goals virtually always pays off. This is why quality orientation is often unpopular with purely short-term management, because some investments do not pay off immediately, but only after some delay.

Quality can hardly be achieved without systematic design (aka architecture). And good things never come for free.

Acknowledgements

Way back in the year 2000, some software architects from Siemens Medical (mainly Mrs Christing Hofmeister) published a book called "Applied Software Architecture", describing a method to systematically designing systems based upon given requirements and constraints. They called their approach global analysis - which became the fundation of the QDSA approach described here. The Siemens guys did not focus on quality, and proposed a quite waterfallish top-down approach to architecture, not suited for modern development practices. Nevertheless - their idea of developing solution approaches based upon specific (quality) requirements or constraints remains usefull.

Image credits

The cheerful lady has been photographed by Photo by Sincerely Media on Unsplash.

The header image by Isaac M Smith

Forem: Dr. Gernot Starke

Speed up your video

Original (Slow) Video

High-Speed Video

ffmpeg Solution

Machine-Generated Images - with Yoda

The Artists and Styles

What is Contemporary Art?

Zappa, Frank

Yokoo, Tadanori

Witt, Solomon Le-

Wiley, Kehinde

Warhol, Andy

Walker, Kara

Ting, Walasse

Takano, Aya

Street art

Stella, Frank

Sherald, Amy

Saville, Jenny

Riley, Bridget

Richter, Gerhard

Pollock, Jackson

Piccini, Patricia

Perry, Grayson

Ono, Yoko

Night, Starry

Neshat, Shirin

Mozart, Wolfgang Amadeus

Motherwell, Robert

Mendieta, Ana

Martin, Agnes

Lichtenstein, Roy

Kusame, Yayoi

Kruger, Barbara

Koons, Jeff

Karim, Anthony Abdel

Judd, Donald

Hundertwasser, Friedensreich

Hurst, Damien

Hockney, David

Grey, Alex

Graffiti

Graubner, Gotthart

Goscini (and Uderzo)

Goldsworthy, Andy

Communism

Close, Chuck

Clark, Lydia

British Upper House

Beksiński, Zdzisław

Bauer, Joseph

Basquiat, Jean-Michel

Abramovic, Marina

Yoda, explaining art styles by example

Contents

Introduction

Art Styles

Abstract Art

Acacemic Art

African

Aquarelle

Art Deco

Art Nouveau

Avant-Garde

Bauhaus

Baroque

Colour field

Comic

Contemporary Art

Cyberpunk

Dadaism

Etching

Expressionism

Fauvism

Folk art

Futurism

Glitch

Graffiti

Impressionism

`ffmpeg` Solution