Forem: Dmytri Kleiner

Continuous Everything As Code

Dmytri Kleiner — Fri, 29 May 2020 00:00:00 +0000

Managing Databases in a CI/CD pipeline is a nightmare. Migration scripts are better than making changes on running servers, but become hard to manage in complex modern pipelines. Truly solving the problems requires revision control for databases so that you can clone, branch and merge your database.

Devs have it pretty good these days.

With Git, CI/CD, Infrastructure as Code, and the Public Cloud, you can have an idea in the shower, write some code after breakfast, commit to git and start collaborating with colleagues after lunch.

You can Docker up your app, Terraform some infra, create a CI/CD pipeline, and then test and have everything deployed to production by dinner.

By beer o’clock you’re sitting back and watching the hype buzz up the ranks on hacker news.

But then, after the glorious intro to this thus-far happy film, our protagonist wakes up screaming.

What horror haunts their dreams? What nightmare invades their peaceful bliss? What terror wrenches their face into a wretched scowl?

It’s The Data!

It’s the damn database.

Every application is perfect, beautiful, flawless, until you have real users and their stupid data. Then, the pain begins.

Then, when you want to push more amazing code to production, the database rears its ugly head, raises its iconic knife-fingered glove and sneers “Come to Freddy!”

As I was saying, managing Databases in a CI/CD pipeline is a nightmare.

Git, CI/CD and Cloud APIs have completely transformed the way software is made.The days of slow and deliberate software release with loads of manual work are gone, we now live in a world of continuous everything as code.

However, once an application is live, it has production data. Production data is important, especially to users, and needs to be preserved, you can’t just wipe it out when you release a new version of your software to production, and the new version of your software has schema changes and new data.

In the old days, when software changed such that the database schema also had to change, DBAs would immediately wait around for a few months, mostly hiding,until eventually, several entity–relationship diagrams later, they would simply take backups, modify the schema on the live server and hope for the best.

Since new software was deployed less often than Sylvester Stallone movies were released, and users were fine with a few hours, or even days, of downtime for each update, everything was fine. In those days “Agile” was a word you used to describe a Soviet gymnast, not a software development team.

But now, applications are developed continuously, different versions of the application need to run in test, dev and staging environments, each of which has different data and a different schema. This makes for a difficult set of challenges in the set up of each environment, and the reliable promotion of software from one environment to the next.

Moving data to production requires not only updating the databases schema with the latest changes, but also updating all existing data such that it is valid within the new schema.

Staging environments need copies of production data in order to test things properly. They also need all the new data and schema changes that are required to make the new version of the software work.

So now you’re moving data and schema changes back and forth, up and down the pipeline. But unlike with software source code, you do not have Git to clone,branch and merge your changes.

You need to do migrations. And migrations are flaky, slow, error prone, and frankly, terrifying. The only good thing about a migration is that you can store it in version control with your code.

Writing migrations as code become popular in the heyday of Ruby on Rails, they had “up” and “down” functions, which could either migrate the schema of the production database to the current version required by the software, or revert to the previous version.

Writing up and down migration functions for each schema and data change was a lot of work, but it seemed manageable, and everyone was happy enough so they got back to riding around on single gear bicycles and making fun of DBAs.

Later, clever devs working with enterprise teams at ThoughtWorks and other places, took a break from writing drab Java code and realized that “down” migrations never worked when you needed them, because new data could not be cleanly mapped back to an old schema without losing data. They stopped making them, creating “forward only” migrations, only making “up” migrations, and if/when they needed to revert, they would just make a new migration. Forward became the new back.

It all seemed peaceful enough. But then devs everywhere started seriously hating on monolithic software. Armed with JavaScript and enjoying their single origin coffee, they decided that one schema was not enough, because with micro services, feature flags and various roll-out strategies, not to mention multiple versions of mobile apps, you couldn’t just go forward. You needed to support multiple versions of your schema at the same time, and the expand and contract pattern was born. You “expand” outward, adding new tables and fields,but leaving old ones for older versions of your code, or platform. Then, when you’ve sunsetted the older versions or API endpoints, you “contract” and delete what you no longer need. Except you don’t - you forget, or you never sunset older versions. All of this becomes unmanageable quickly and productivity grinds to a halt.

It’s a lot of work, and as applications become more data-centric and designers create design systems with all kinds of amazing reactive components for devs to use in their applications, each of these components has fields and each of these fields have data, and this data needs to be stored in a database, and this database needs to updated every time new components are added, modified, or used in new ways.

And so here we are. Stuff is complicated. And despite all the awesomeness of devops, design systems, CI/CD, infrastructure as code, despite all this automation, everyone is still terrified of the production database.

How we do we deal with all this?

If only we could manage the revision of data the way we manage the revision of our source code. If only we could just merge our latest changes into the production database. If only we could simply clone our production database for staging, if only we could just push changes from a developer’s local machines to a test environment!

If only … we had something like Git, but for data.

Well friend, have you heard the good news? We’re building TerminusDB to provide distributed revision control for data. Databases in TerminusDB can be cloned,branched and merged with nary a migration script in sight.

With TerminusDB you can truly integrate data into your CI/CD pipelines.

You don’t need to fear your database anymore.

TerminusDB is free software, and is pretty new, and still under heavy development, but we’d love to have join us, the best place to get involved early is our Discord community: https://discord.com/invite/Gvdqw97

Reluctantly Remote, and loving it

Dmytri Kleiner — Tue, 24 Mar 2020 00:00:00 +0000

I have to admit, most of my career I’ve treated remote collaboration as anecessary evil.

I’m a big fan of collocation. I love sticky notes, magic charts and colourfultape. I love walls so much whenever I see pictures hanging on them I’m like,why are they wasting good wall space for those useless pictures? I love pairand mob programming. I love highly engaged, participant centred workshops. Ilove retrospectives, and I do elaborate ones featuring lots of pair and groupconversation, arts and crafts, and role playing.

For years I’ve lived this way, preaching customer and partner interaction,NIHITO, user observation, face to face collaboration and working togethershoulder to shoulder.

Then, the first shoe dropped, I joined an early stage startup building theknowledge collaboration database of the future, TerminusDB.

Suddenly, I had a distributed team, half of the engineers are in Utrecht, halfof the engineers, along with the business team are in Dublin, a developerrelations person in London, and me in Berlin.

So, I started making everyone fly around, we had workshops and collocated workin the Netherlands, and in Ireland, we started to develop our flow of designand delivery practises.

The idea was that we would do design and planning in person, on walls withstickies and stuff, then convert these to digital representations and dodelivery work back home, but try to divide systems such that related work couldbe done by work enters which are mostly collocated. So, core databasedevelopment in Utrecht and web platform work in Dublin.

This meant our CTO would basically commute from Dublin to Utrecht, and ourDevRel person would basically commute from London to Dublin, and I would alsobe travelling continuously. But hey, that’s been more or less what I’ve beenused to for basically forever, and this is the framework that I’ve beenrecommending to teams for a long time.

We chose Trello as our online kanban, and where already using Slack and Github,plus Google Apps and Zoom. So, we were somewhat remote already, but not really.

Realizing that it would be cool if we could find ways to do some of the designand planning work online, as there would never be enough time to do all of itin person, I started investigating online collaboration tools, and havingconversations with colleagues about best tools and practises.

Then, the other shoe dropped. A worldwide pandemic. Everyone is working fromhome. Travel is not an option, and our customers can not be visited.

Now what?

Day one we left our physical offices in Dublin and Utrecht, and moved into ournew virtual office: Discord.

Discord is a platform for gamers to play online gamestogether, it provides text chat channels, like Slack, but also voice channelswhich are like meeting rooms that people can have voice meetings in.

The ability to have different channels for teams as well as for companymeetings, and the ability to quickly see that people are in a given voicechannel, provides for a much more ad-hoc experience, you can see when and wherepeople are meeting, drop in and say hi, or notice meetings are happening whenmaybe you didn’t notice in your calendar.

This feels much more like an office experience than scheduled videoconferences, and allows for more casual interactions with less friction injumping on voice chats. It’s not quite collocation, but it’s not a badfacsimile.

Discord also allows a user to go live and share their screen, this was designedfor streaming the game you are playing so that others can watch you play. Italso works well for pair and mob programming and presentations.

If anything, moving to Discord has actually increased the time we spendparing and mobbing. Something we’re very happy about, not to mention thereduced air miles.

We are writing more code with greater quality, spending less money, emittingless carbon, and abiding by needed social distancing practises during thepandemic. Other than perhaps worsening the teams personal hygiene andfashionability, this has been a real improvement, and a practise we willcontinue even when the pandemic is behind us.

So far so good, however, pairing and mobbing and presentations are great forthe delivery portion of our workflow, but there is still the question of designand planning. While I had begun my investigations in earnest, now we’re all in.It’s go remote or go home! No wait. It’s stay home or no go! Go live or dietrying? Something like that.

The question is how do I facilitate agile design and planning practises when Ican’t see everyone, when I can’t get them to write sticky notes, and I don’thave a wall to stick them on.

So far, I’ve not really been a big fan of any of the online tools I’ve tried.They are either too fiddly, requiring the mouse ninja skills of a masterPhotoshop jockey, or too limited, implementing a very rigid version of popularpractises and not allowing you to paint outside the lines or vary the practicewhen the situation calls for it. They make it hard or impossible to combinepractises or invent new ones, as I am wont to do. Or, the system is socomplicate, that finding a setting or feature is like needing to explore 17levels of a 70s style text adventure, without being eaten by a Grue, only notnearly as fun.

Good luck facilitating an online session when you need to start with an hourlong introduction to the tools, and let’s put aside the issues of ill-fittingpricing models for now.

In the end, after trying every tool available online and putting myself at riskof repetitive strain injury just from the amount of “Hi, I’m Bill from CustomerSuccess” emails I will need to unsubscribe from , some hope is emerging.

With Discord as our virtual office, and with everybody getting used to voicechannels as opposed to video conferences, this leaves our eyes free to focus onreal time collaboration tools instead of video. While screen sharing and videoare very useful tools, for working sessions like planning and design workshopsthey are better avoided.

Instead of video or screen sharing, real time collaboration tools alloweveryone to work together at the same time within the collaboration app, sonobody is sharing their screen, everyone is using the same software at the sametime and seeing the same state.

The first tool we added was Lucid Meetings. Lucidis not a conferencing tool, we use Discord voice channels to talk duringmeetings.

Lucid is a real time meeting tool that allows you to create an agenda, and stepthrough it together with everyone at the same time. During the meeting,everyone sees the current agenda item being discussed.

This is super important if you will also be use other tools during the meeting,as you can provide links and instructions in the agenda. One of the biggestchallenges is keeping a group of remote people in sync. A real time agendatool like Lucid is really helpful for this.

Lucid also has a chat interface, allows everyone to write notes, add actionitems and other records, which are not only emailed to everyone after themeeting, but also kept in lucid and available for review at future meetings.

There is also a speaker queue, document attachments and other features requiredfor facilitating meetings and recording outcomes. Lucid is built by meetingexperts, includes templates and facilitator guides for common meeting types,and has a friendly and very helpful community.

So, now that I can plan the session and keep everyone on the same page, thenext, and trickiest bit is visual collaboration, how do I replace creatingsimple canvases with tape and sticky notes? Although honourable mention goes toMetroRetro and Miro, both verycool tools, our team chose Mural. Not sure why all thesetools start with an M. Maybe it’s like a secret coded message of some kind.

Mural is a real time visual collaboration tool that allows people to draw on acanvas at the same time, add sticky notes, shapes and pictures. What sets itapart are it’s “Facilitator Superpowers.”

Facilitators can build outlines, that define parts of a canvas or steps in aPractise, and then “summon” participants so that they are all looking at thesame part of the canvas at the same time, and hide and reveal portions of thecanvas as required to advance the practise. It also has plenty of populartemplates and frameworks for popular practises, but makes it easy to combinethem and make your own templates. It also has a timer, and a voting feature.

I’ve successfully created templates for my own practises with it and used themto run remote meetings.

So, while our journey into remote design and planning practises is just beginning, and many challenges remain, Discord, Lucid Meetings and Mural giveus a pretty solid base to build on.

But, OK, I confess “loving it” was an exaggeration, I still miss working faceto face, but we’re coping and even when the pandemic is over, I’m sure thatremote work and the practises we are developing will remain a key part of ourway of working.