Forem: Miguel Barba

Which language(s) would you recommend to Transform a large volume of data?

Miguel Barba — Fri, 15 Dec 2017 11:47:22 +0000

Hi!

Which language (or languages) would you recommend to create a custom ETL tool with the purpose of processing a (very) large volume of data? The main focus here should be for the T (Transform) component, since that's where all the major issues and complexity will surely arise.

I know there're lots of tools available out there that perform this kind of stuff but I'm currently considering all the scenarios and the complexity of the project itself may "force" us to pursue a custom solution so I'd like to ear from you guys and your expertise and past experiences!

Thanks :)

Data Quality: Technical Debt From Hell

Miguel Barba — Sun, 24 Sep 2017 11:59:04 +0000

This post was originally published here.

Technical debt can take many forms as most of you probably know it. Data quality is just one of those forms.

Data quality is a major issue in large organisations, where multiple systems, applications and services interact between themselves, exchanging and altering data. Incoherences will always occur. Either because someone made a mistake, either because there’s an unidentified bug somewhere, either because the system’s architecture isn’t as robust as it should or simply because people prefer to ignore it (this last one happens way more than it should, trust me!). This will contribute for a consistent and sometimes quiet but steady increase of your technical debt.

Don’t let your selves be fooled. It’s easy to start getting reckless regarding data quality, furthermore when you’re working on a data migration project for several months, for example, and you simply start to reach a point when, even though you don’t want to, you start getting tired and making mistakes. Hell, sometimes you’re just having a bad day… It can happen to any of us!

There are several ways to address this issue. Here are some ways:

Let’s start by stating the obvious. Make sure your services are the least error-prone possible. This is where all technical debt should start being addressed: before it even has a chance to exist. I know I’m being a dreamer, unrealistic or whatever but in a perfect world, this is how it should work.
It’s nice when you have a team dedicated to performing data analysis and comparisons between different systems and enforcing data repairs in order to correct the data; this action should be performed as often as possible on a regular basis.
Understanding what’s causing the incoherence; it may be very difficult to achieve it but when you manage to do it, the probability of eradicating it will be extremely high. This all looks great and relatively simple to achieve but then enters another common problem in any organisation: people. We are our worst enemies most of the cases when we end up going in opposite directions when trying to solve some problem we have in common. Fortunately, this doesn’t happen always and as time goes by the tendency it’s becoming quite the opposite, although there’s still a large margin for improvement (there always is!).

The idea of writing this post was a direct result of identifying a data incoherence carried throughout 4 system upgrades and migrations and that finally haunted someone (me in this particular case) more than 16 years later the flawed data was first created. Nightmarish, don’t you think? What’s yours?

Photo credit: T a k (https://www.flickr.com/photos/takashi/18862634/) via VisualHunt / CC BY-NC

Data Migration

Miguel Barba — Mon, 21 Aug 2017 13:13:41 +0000

I'm about to start a new data migration project. It's kind of my thing at the moment. I'll basically have to migrate a data model from an Oracle 11g database to a different data model in an Oracle 12c database and somewhere in between will take into account all the business rules and surrounding application ecosystem with all its constraints and challenges.

In my previous project, we were migrating from and to 11g. We used PL/SQL to extract and perform some data preparation. The core data transformation was later performed in a C++ module, from which XML files were generated and finally imported. It was far from being a great solution; instead, it was basically some code rework from a past project.

What about you guys? I'm looking forward to knowing what kind of experiences you had with this kind of project!

Photo credit: renatela (https://www.flickr.com/photos/renatela/1069651852/) via VisualHunt / CC BY-NC-ND

Rational Operationalization

Miguel Barba — Sat, 19 Aug 2017 14:38:57 +0000

This post was originally published here.

My first contact with support happened early in my career. I was part of a team that, amid other jobs, would operationalize procedures for an operations team to execute. It didn't take long to realize that the lower the automation level, the higher the probability for the operations team make a mistake. Several factors contributed to this:

The operations team worked in shifts, 24x7, and always with a high churn rate (there was always someone leaving or joining the team);
This team's elements had lower qualifications and a low knowledge of the business or its rules (they weren't supposed to);
The operations team executed procedures from several support teams (my team was just one among many), and therefore they didn't know the procedures in detail, how they were implemented or what were they for;
The procedures with the higher number of steps and actions to perform were far more error prone and to address this point, automation was essential;
The way the procedures were implemented and made available to the operations team had a correlation with the number of errors recorded, for example: to execute a script through a command line is more complex than to execute the exact same script by clicking a button in some dashboard; to send an email with a short description of the procedure isn't the same as making available an operational manual for the same procedure;

No less important here was our posture towards who made the mistake. In an initial stage, we almost only placed the blame in the operations team. Fortunately, we didn't take long to realize that wasn't the right way. Instead we made the only logical decision: to understand why the error had occurred (and hey, lots of times it was our problem!), how to correct it, to effectively correct it and, last but not least, to pass and explain the changes to the operations team so that they may execute the procedure properly.

Nowadays the errors still occur and they will keep occurring in the future but they are less frequent and we face them in a perspective of continuous improvement.

Photo credit: Armando G Alonso ✈︎ via VisualHunt / CC BY-NC-SA

Caves of Steel

Miguel Barba — Sun, 02 Jul 2017 21:49:55 +0000

This post was originally posted here.

In Isaac Asimov's "Caves of Steel", the humanity that still inhabits planet Earth, lives in giant cities, covered by gigantic domes from where they never leave. They end up becoming agoraphobic has a consequence of such seclusion. The population levels are alarming everywhere and, as a consequence, the vast majority accepts gladly to live in a context where limitations, social extracts and a very strong sense of hierarchy rule, everything because of the fear of losing everything they have (seen as a privilege) or never be able to reach whatever they might aspire to.

While reading the book I ended up establishing a kind of a parallel, on its own scale and with all the necessary differences (yes, my mind wanders a lot), about something that happens recurrently in our workplaces and that we tolerate and let go, without thinking about the adverse consequences that it might end up having. I'm referring to the fear of making a choice or deciding something, especially when it's hard, difficult and the output may have consequences of some kind. Almost for sure that you've been through that and, very lightly, you were one those persons in some occasions. In those moments, inside our heads, one can often listen to stuff like "leave it to them", "let them decide", "does it have to be me?" or "I'll end up having lots of work", just as an example. It happened to you already, hasn't it? We must be aware of his kind of posture, acknowledge that it exists and, more importantly, we must fight it; in ourselves and in others around us. Taking chances is an essential part of our growth when we want to grow and overcome ourselves on a daily basis and throughout our career. These moments may have the power to inflate us with trust or, at least, we should be able to learn from them. Any risk should be properly weight and calculate so that we can minimise it from the start.

But how can we do it? Why not share the responsibility? Exchange experiences, talk with more experienced people. Try to clearly understand what's at stake. If you can't explain the problem in a simple and clear way, then you haven't understood it clearly. But above everything else, believe in yourselves and don't turn your back to a challenge. To remain in your comfort zone may be a guarantee that you won't be taking a step back but will surely prevent you from taking a step forward. Don't stay inside your cave of steel.

When the old system prevails

Miguel Barba — Tue, 27 Jun 2017 22:27:52 +0000

This post has been originally published in a blog I tend to forget I have and I first thought of writing it after a frustrating day at work...

Imagine a system with a scalable and extensible solution, as long as it keeps following the rules according to which it was designed, what actually goes quite against what these two properties advocate. So, whenever someone decides to change the rules of the game, it's no longer able to provide an adequate response. Imagine a system developed using open source and proprietary technology as well. Imagine that the support for the proprietary SW is no longer available as it is deprecated and extremely old. Imagine as well that there isn't any stable version of the source code or executable file, for that matter; if anything serious happens with the proprietary component, the probability of not being able to restore it to a functional state is quite higher than it should be. Imagine that the servers in which the services are running are obsolete but, because of the proprietary component, they can't be upgraded. Imagine that from time to time the need for new developments arises, due to legal requirements and, every time that happens, the developments occur over the existing solution, although only regarding the non-proprietary part of it (yes there's a huge red line over the proprietary part!), even knowing there will be an impact in terms of technical debt.

Imagine that right next to you, there is a solution able to accommodate the existing service and its business logic, allowing simultaneously to scale, extend, change or adapt the rules, whenever's necessary. Imagine that the technical debt could be kept to a minimum or even to dissipate completely (one can always dream!). Imagine that it's also a proprietary solution but a way more flexible one; with support and still having new versions being made available on a regular basis and enable integration with custom developed modules, using open source, whenever necessary. Imagine that it runs on almost any server. Do you know any developer that wouldn't choose this approach if given the choice? I know I would but unfortunately that's not what happens in reality. Either because there is no budget available or because the stakeholders with real decision power, can't fully understand or we aren't able to prove that a minor short-term impact can end up becoming a quite bigger one in the medium or long term. Either because the decisors don't have the necessary technical knowledge that would allow them to make an informed decision or either due to a lack of political will, plain and simple.

How can one overcome such hurdles? Patience and quantification are two possible keywords. Show how much your system costs you currently (hours you spent fixing issues, processing, applying workarounds, etc). Prove how the long term costs of adopting a new solution compensate against opting for developing based on the existing one. Show how a new solution will allow becoming the process more agile, fast, efficient and less error-prone. Prove that using the programming language or framework A is better than B. Show that the support team interventions will decrease. Involve your system end-users if possible; they know their pains and the system's better than anyone else because they struggle with them on a daily basis. They are the living proof of the existing problems and potential bearers of the solution. They can be an invaluable ally.

Be patient. Be persistent.

Please, feel free to share your thoughts and experiences in the comments!