Forem: mpodlasin

1 Month Vim Only Challenge - Week 4 (Finale)

mpodlasin — Sun, 21 Nov 2021 15:52:29 +0000

This is the last article in my vim miniseries.

For the last month, I've been doing a "1 Month Vim Only Challenge", where I forced myself to use vim for 1 month, both in my free time as well as at work.

In the previous articles, I was telling you about the reasons why I started the challenge in the first place. I was also relating my experiences while doing the challenge.

And now the challenge comes to an end, as the fourth week of using vim ends. As I've said in the first article, this means that it is time for me to make the ultimate decision:

After using vim, and vim only, for 1 month straight, do I want to go back to using a "regular" editor/IDE like VSCode? Was using vim an unbearable experience? Do I look forward to never having to press :wq ever again?

Or did I fall in love with vim so much that I plan to use it for the rest of my life? Does using a mouse cause nausea in me now? Do I want to keep my fingers in the home row even if my room is on fire?

Well, let's not keep you on the edge of your seat anymore. Time to give you...

The Answer

I... I have no idea.

No, seriously. You would think that after 1 month I would be leaning strongly towards one side or another. That's what I thought back then as well.

The thing is that if you are trying to replicate - let's say - VSCode setup, vim will get you to like 90% or 95%. Meaning that it will mostly work nicely and flawlessly. But there will still be 5% to 10% of little annoyances, awkward UI/UX, or tiny bugs. Nothing major, nothing that would block you from doing the actual job done. But still enough for me to think with reverence about VSCode from time to time.

On the other hand, you are getting very real benefits with vim. The minimalism of the setup (my .vimrc is still tiny and I love it). How lightweight vim is. How just absurdly configurable and extendable it is.

I guess if I were to summarise the benefits of vim in one sentence, it would be this:

You can truly make vim your own.

You can have two people using vim and their setup would be completely different. You could have a minimalist, distraction-free setup with only bare necessities installed. And at the same time, you could turn vim into the craziest, most robust, multiwindow IDE packed with many, many features, including your own, custom functionalities, coded in the vim scripting language.

So if I am to make a rational decision of whether I should stick to using vim, I have to answer the question do the benefits of vim outweigh for me this 5-10% of awkwardness that it has compared to VSCode.

And at this point, I really don't feel equipped to answer this question.

It Has To Feel Natural

In the first article of the series, I wrote that the main criterion for me will be "Does coding in vim feel natural to me now?"

But this also turned out to be more difficult to answer than I initially expected.

The thing is that most of the time it does feel natural. But then you also encounter those moments when you want to do something that would be trivial in VSCode and... you realize you are using vim. You have to google for a key that does what you want. Or you have to start looking for a plugin that has the desired functionality. Or - in the most extreme cases - you realize that you have to write a little script in the vim scripting language, or at the very least copy someone's script from stack overflow.

Now, again, those moments happen relatively rarely, especially as you keep progressing in your vim journey. But they do happen nevertheless, so I cannot say that using vim is completely natural for me at this point. I still have this slight anxiety that I will start working on a task at my day job and I will have troubles, because something will turn out to be difficult, or at least awkward, in vim.

But on the other hand, most of the time I don't think about the fact that I am using vim at all. Most of the time it is natural. Vim's controls feel very straightforward at this point. This is ironic because controls are probably the thing that scares people the most from using vim. For me, it really wasn't an issue at all. Even with knowing just the most basic commands, you can get a lot done. And it only gets better as you continue to use the editor.

So, to summarise, even the criterium of "feeling natural" can't push me towards an ultimate decision. On some level, it feels incredibly natural, on some level it doesn't.

So?

Well, but I have to make a decision, don't I?

Yes. And I will.

I am sticking with vim for now.

From what I've written so far, it is not an "unquestionable win" for vim though. I still have lots of doubts, lots of issues to solve, lots of awkward spots to go through.

I am however willing to give vim a few more weeks of a "trial period" because there was indeed lots of stuff that vim impressed me with.

So I'll continue to use it, but not in a "challenge" mode anymore. If I encounter any issues that stop me from doing actual work or slow me down considerably, I will not be afraid to reinstall VSCode. It's important to note that VSCode does have an extension allowing you to use vim controls. So I might go in this direction at some point.

Turning "using vim" into a challenge was a great idea because otherwise, I would bail out much, much earlier.

But on the other hand, it stops me to evaluate things objectively, because I am literally forcing myself to use it. I plan now to stop forcing myself. I will just continue to use it if I feel like it. And after a few months, it will be clear if it actually won my heart over or not.

Final Words

So this is how I am finishing my little "1 Month Vim Only Challenge" article series.

I know that this is a slightly disappointing finale, where I failed to provide you more definite answers.

But I hope that this gives you a good idea of what to expect when getting into vim.

You will likely be pleasantly surprised, but not immediately won over at the same time. Whether you decide to stick with vim will be largely dependent on your personal preferences, style of work, what kind of tooling do you need, etc.

Me personally, even though I am still not sure if I will continue to use vim forever, I definitely don't regret participating in this little challenge. It showed me how mature vim community and vim tooling are. It showed me that you are not insane if you are using vim on a daily basis. It's much closer to VSCode experience than you might initially think.

And even if I decide to go back to VSCode at some point, I always have vim knowledge and experience under my belt. And there might be some moments when it could turn out to be helpful, if not necessary. Editing stuff over ssh for example. Or if they fire me and I will have to sell my MacBook and go back to coding on my ancient ThinkPad that can barely handle running VSCode...

Anyways. I hope you enjoyed this little series and I encourage you to try the same thing on your own. You might get surprised how much you will enjoy working in vim!

Cover Photo by Kolleen Gladden on Unsplash

1 Month Vim Only Challenge - Weeks 2 and 3

mpodlasin — Mon, 15 Nov 2021 10:38:42 +0000

This article is part of my series "1 Month Vim Only Challenge", where I force myself to use vim for 1 month and, at the same time, create articles talking about my progress, challenges, and frustrations.

You can read the two previous articles here and here.

Note that this article is not a tutorial or anything like that. It's just a collection of my purely subjective experiences while doing the challenge.

Crisis in Vim City

The two previous articles in the series were very optimistic. I was motivated to do the challenge and the first week went shockingly smooth.

However, it's probably not a surprise that in the second and third weeks everything started becoming more challenging. Energy and attitude from the first week slowly disappeared. I lost my rose-colored glasses and started noticing more and more problems with my workflow and setup.

The thing is, the two past weeks were extremely busy for me, both at work as well as in private life. At work, we are getting close to finishing an important project, so I had to stay focused on that. And on top of that, I decided to travel to Turkey for one week, in order to meet my coworkers.

Lack of free time, combined with pressures of delivering the project, made using vim very difficult for me. In fact, I was close to giving up multiple times. The only thing that stopped me was the fact that I framed this whole thing as a "challenge".

I didn't do that by accident when I was starting this series. I knew that there will be moments when using vim will become difficult, especially at work. So I decided to do it as a challenge, in order to literally force myself to go through the toughest times.

And it worked. As of now, I still haven't reinstalled VSCode and haven't used it even once. Yes, I am still writing this article in vim.:w

So what exactly were the problems? Let's mention some stuff, in no particular order.

Working On Multiple Files

For the longest time, I didn't know that vim has tabs functionality. I just used windows, which splits your view into multiple subwindows, effectively giving each file less and less space on the screen.

In the beginning, using windows was good enough for me. However, even this simple setup was awkward, because default keyboard shortcuts to navigate between windows are very, very uncomfortable. Pressing Ctrl+W while staying in the home row gives a solid stretch to your fingers... Try it!

Currently, I am trying to force myself to use tabs more. They are much better than windows - including better navigation shortcuts. And yet, for some bizarre reason, I still fall back to using windows again and again. I am not sure why.

Maybe it's the weird way directories are abbreviated in the name of the tab? Or the fact that the project tree disappears when you open a new tab because it operates on a tab per-tab basis?

Or perhaps, like most things in vim, it's really just a matter of getting used to. But as it stands, I still do feel a certain "awkwardness" when working with projects that require me to have multiple files open at the same time.

Navigating Large Projects

NERDTree, which I am using as my filesystem tree plugin, is very pleasant to work with and well documented. However, I still find it awkward to use the keyboard for navigating large directory structures. I believe I was much faster with a mouse.

To give you some context, the repositories I am currently working in can be massive. Those are often monorepos, containing multiple separate packages.

Perhaps this is just a matter of time, but for now, I feel that this way of navigating files inhibits me, especially when I have to find a file fast.

What would help me would be also some kind of global file search. Vim has some built-in stuff for that, like :grep command, but... I never really got it to work.

And even if I did, the UX of that command is frankly horrible, requiring you to type the command with a searched phrase and then type a second command, which shows you the search results. Why? Just give me a command where I can write something like :gs "some searched term" and see all the results immediately, preferably with a "jump to file" functionality.

Now, I have seen some plugins doing that, which looked extremely promising, but the installation and/or setup turned out to be a bit trickier than I expected. So this is also something I would have to devote some time to.

Working With TS Tooling

In the last article, I was very positive about the TypeScript tooling that I started using.

However, I started discovering some limitations fairly quickly. Those limitations led me to install another plugin - coc.nvim.

This extension is very interesting because it promises to "Make your Vim/Neovim as smart as VSCode."

It's a more general plugin, that - rather than supporting a single language or functionality - allows you to extend vim in a very similar way as you would extend VSCode. As far as I understand it, underneath it uses the same tooling that VSCode uses. Because of that, support for - let's say - TypeScript is virtually the same in vim and VSCode. Autocomplete works the same way, autoimports work the same way, etc.

This is an amazing idea because it utilizes the great and battle-tested work that was already done for VSCode.

As I've said, the installation itself is very simple. But - as the authors of the extension themselves recommend - you still have to do some keybindings configuration, in order to get the most out of it.

Because of that, I still don't really know how to achieve certain results using this plugin. I don't know for example how to check the inferred type of an expression. I also have trouble with identifying if my file has currently any TS errors because it is presented on a per-line basis and I can't see all the lines in the file at the same time.

Funnily enough, I still have the previous TS plugin installed. Mostly because I got used to how well some stuff worked there. On the flip side, the plugin started acting crazy in some (admittedly rare) cases. Presumably, because it didn't successfully detect which TypeScript version I am using in each package of the multirepo (?).

So even though both of those tools are very impressive and mature, even here I will have to do additional work of learning and configuring stuff, in order to make the workflow perfect.

Working With Eslint

There are still more things that I need to somehow solve. Because at work we are working on a pretty big multirepo, even the linting command runs very long. Of course most of my coworkers mitigate that by using a VSCode with an extension that lints a specific file during editing.

There exist of course some vim (or coc.nvim!) plugins that do the same thing, I just didn't put aside some time to try one out.

And my fear - based on the TypeScript plugins that I talked about - would be that even after installation, I would have to do some additional configuration work just to make everything work seamlessly.

Yes, It's All My Fault

At this point, I know what you are thinking. "Well, what's the problem? You just have to properly set up the plugins or add new ones and everything will work just as you would like it!".

And that's true. It's all my fault. But that's exactly the whole point of my article. If I had even a little bit of free time, setting up the workflow with vim would be simply fun.

But when you have to balance work and private life at the same time, it's surprisingly difficult to devote even twenty minutes to set up a particular plugin.

What keeps happening is that I start setting something up - say, I install a specific plugin. But then there is still a some configuration that I should do. And how to configure those plugins is often a bit poorly documented and/or just simply looks intimidating for a newcomer.

This makes it difficult to assess how long a certain configuration will take. It might be a breeze and take ten minutes, or it might turn out that things are more complex than expected, and you will have to spend a whole day, trying to get some plugin to work.

And yes, I know that most of the time, it will just take ten to twenty minutes. Those plugins are well maintained and have solid communities behind them. But when you have tasks at work that you need to execute ASAP, it's super difficult to make a rational decision and devote some time to configure your tooling.

On top of that, this ten to twenty minutes involves only one plugin. And, as you've seen before, I have problems involving at least few of them. So it would take me even more time to configure everything the way I'd like to.

The ultimate point here is simple - if you are going to take a deep dive into vim, make sure that you have enough free time to really learn and configure it properly. If you reserve some time, this process will be much more enjoyable. But if you are speeding through plugin documentation, because you have work that needs to get done, using vim might become challenging, and simply unpleasant.

It just isn't a "plug and play" tool. It gives you lots of freedom regarding configurability, remapping keybindings, etc. But this power comes also with a more demanding setup process.

With great power comes great responsibility, I guess.

PS

It is now the end of the weekend, as I am writing this last paragraph.

Just to be fair to the vim community, I need to mention that I have finally set aside some time to do more configuration work.

Most of the problems I've mentioned were easily solvable. It took me not more than 2 hours to solve around 80% of my issues

So the fear and anxiety were mostly unwarranted.

This however further emphasizes my ultimate point of the article - make sure to set aside some time devoted to configuring your vim setup. Don't tell yourself that you will "do it during work hours" etc. Devote solid 2, 3 hours in the week and put actual effort into reading docs and figuring out how the tooling works.

This will make using vim during actual work hours much more seamless and pleasant.

Cover Photo by Nik Shuliahin on Unsplash

1 Month Vim Only Challenge - Week 1

mpodlasin — Sun, 31 Oct 2021 07:13:58 +0000

It's currently Saturday.

Exactly one week ago I have started my #1MonthVimOnlyChallenge. I removed all other editors/IDEs from my computer and I forced myself to use vim for exactly 1 month, documenting my journey on the way. I plan to write short weekly articles and I am posting regular updates on my Twitter.

Let's answer the big questions first.

Did I give up? Nope.

Am I still writing this article in vim? Absolutely!:w

Am I enjoying myself? 100 percent!!!

In the article, I will summarize my various observations regarding using vim on a daily basis. This is not a tutorial, not a guide, just some random thoughts and feelings, in more or less random order.

The Setup

Probably the most surprising thing for me is that my setup is still fairly simple. I do plan to complicate it in the following week, to challenge myself a bit, but what I currently have already allowed me to be productive, even at my day job.

This is probably the moment where everyone expects me to show off my vim config. And I could to that, but there isn't really a point. The config is literally just a few lines. It just enables a few basic options, like syntax highlighting, showing line numbers, etc. Nothing beyond your standard "Write Your Own Vim Config" articles, that you can find hundreds of online.

Plugins are, I feel, more interesting. Without the plugins, vim is really just a basic text editor, even though it does have some built-in handling for languages like C, Java, and even JavaScript. However, my first attempts to open a TypeScript file without any plugins, literally hanged the vim window, presumably because syntax highlighting got very, very confused.

This immediately went away after I've installed typescript-vim, which provides syntax highlighting and other basic handling of TypeScript files. This immediately made vim work great with my TypeScript projects.

The installation of plugins is extremely easy. Essentially you just have to do a git clone into a proper directory and you are done! This seems to be a new-ish thing, introduced in vim 8. You can see that plugins still support various "vim plugin managers", which tells me that back in the day installing and managing those plugins was a huge pain in the ass. Now it's luckily trivial.

Finding the plugins is also easy. I just googled "vim typescript" and immediately saw links to GitHub repos.

typescript-vim plugin only gives you basic capabilities - it doesn't include autocompletion, type-checking inside the file, etc. That's where tsuquyomi comes in. It promises that it will "make your vim a TypeScript IDE". I am not sure if I would go that far, but it does give you many capabilities that you expect in a modern TS editor - autocomplete, navigation to type definitions, compilation errors that appear in the file, etc.

Functionally this stuff allows you to do the same things you would do in VSCode, but bear in mind that the UI/UX is... well... not as streamlined.

I don't know who decided that triggering autocomplete menu should be done by pressing Ctrl+X and then Ctrl+O. It's the least friendly and effective keyboard shortcut I could think of. Once you get used to it, it's not horrible, but next week I will almost for sure look into remapping that.

On the other hand, I love how "go to definition" functionality is handled, where Ctrl+] immediately takes you to the definition of an expression you are currently pointing at with your cursor. And then Ctrl+t immediately takes you back to the exact place where you pressed Ctrl+]. This makes navigating TS files fast, pleasant, and efficient.

There is also some stuff that is lacking, or at the very least I still don't know how to do it yet.

I still don't know how to see the inferred type of expression I am hovering over. I tried googling that, but I've seen some intimidating answers, mentioning stuff like coc.nvim etc. So I gave up back then but I plan to go back to it soon.

Also, I don't know how to do autoimports. Autoimport is a feature that frankly changed my life a few years ago. I could just type in the name of the value/class/interface/whatever and proper import would automagically appear at the top of the file. I want that, I need that, so this week I will attempt to research that as well.

Limitation Is A Blessing

Apart from what I've mentioned above, I have also installed NERDTree, which is a plugin for navigating directories and files.

And I also started experimenting with some Haskell/PureScript stuff, but this is at an earlier stage because I am doing that only in my free time, outside of work.

Oh, and I also have iceberg color scheme installed, because the default syntax highlighting colors are a bit too flamboyant for my taste.

But other than that - this is it! I have nothing more so far. And this simple setup - a few lines of vim config, 3 plugins (in my day job), and 1 color scheme - already get me like 80% to where I would like to be. I am kind of expecting that getting from 80% to 90% will be more difficult than getting from 0% to 80%, but we will see next week.

And quite frankly, limitations can be sometimes good. I still remember my CS professor, who used to discourage us from using syntax highlighting. He claimed that it made us dumber, and stopped us from actively thinking about the syntax. I don't know how much truth was in that in the end, but I did notice, that certain limitations of vim forced me to think more during coding. This especially relates to navigating large projects, working concurrently on multiple files, and writing imports by hand.

1 week is not a long time and I do plan to improve those aspects of my vimming in the following days. However, if I didn't, I wouldn't be surprised if, over a longer period of time, those limitations would give me a better understanding of the layout of my projects. Where certain files and modules are stored, how they relate to each other, how the dependency tree looks like etc.

It's kind of like using Grammarly for writing, which is helping me on a daily basis, but also tanking my actual writing skills, because I am using my brain less.

This Hurts!

After the first 2 days of the vim challenge, I started having pronounced wrist pains. This scared the sh** out of me.

I was having similar pains before, but using vim seemed to aggravate it. Not surprising really, when you think about it. Vim forces you to stay in the home row more and reach for the mouse less.

I am quite confused at this point because the mantra of touch-typing is "stay in the home row". But the issues I had made apparent that reaching for the mouse, or changing hands position from time to time is actually a good thing - it allows you to relax your muscles for a bit, instead of constantly staying in the same, a bit strenuous, position.

Contrargument here would be of course that the home row position shouldn't be strenuous at all. You either have to change your technique or setup in order to make sure that's the case.

Now, this of course is not an issue with the vim itself. Quite the opposite, vim helped me identify that the ergonomics of my writing are still suboptimal.

Luckily the pains didn't really appear after those first two days. If they do, however, I am planning on trying out a split keyboard. I wanted to do that for the longest time now, and perhaps vim will push me to finally do it.

And I want to see people's faces in a cafe, where I take out this bizarre keyboard and turn on a program that looks like a hacker's console. I am half expecting them to turn on VPNs at this point or something.

Vimming In Vim

Let's talk a bit about using vim itself. So far I was mostly touching on tangential topics like setup etc.

But how does it feel, to use vim, with its bizarre user interface, daily, including work?

Not bad actually. Bear in mind, I've already used vim during my university days, so the learning curve was much, much lower for me this time.

I just did some basic tutorials on the weekend, which they are plenty of out there. Many of them are interactive and/or involve some gamification aspect, which made the whole "relearning" process engaging and fun.

Vim has two basic modes - regular and insert. And I've noticed that, when using vim, my brain also works in two different modes, depending on the situation.

In a "regular" mode, I am simply working, not thinking much about the fact that I am currently using vim. Reaching this stage is actually much, much easier than you would expect. I think that people really overblow how complex basic vimming is. If you can wrap your head around using "WASD" for playing video games, you will equally easily switch to "HJKL" for navigating a text file.

The second mode is what I call a "golf" mode. Those are moments, where I actually slow down and attempt to do something in a smarter, more terse way. Those are fun little challenges throughout the day, that can break you out from the monotony of constant, mindless coding.

And the best thing about this is that slowly, over time, golf mode skills are bleeding over to a regular mode. You start using certain techniques and shortcuts more and more often until they become second nature and a part of your regular mode arsenal.

It really works just amazing and it gives you a real promise of becoming, over time, more and more efficient in coding.

I guess the only problem here for me is that - especially in my day job - I find myself not going into the golf mode nearly enough. When there is a pressure of finishing tasks, fixing bugs, and delivering sprints it can be hard to allow yourself for a relaxing golfing break. But this is, again, a me problem rather than a vim problem.

Apart from that, I believe that working in those two modes really can make you faster and more efficient over time.

Vim Anxiety

During this week I've posted a joke on Twitter. I have written:

"If you want to get a promotion, using vim is the best way. It discourages you from writing code, so you start doing more managerial and executive tasks."

And although I wrote this as a joke, it wasn't by accident. There is a (quite substantial) grain of truth in that.

I really did notice myself doing much more code reviews, research, and general "managerial" tasks. And I quickly realized that this is simply due to "vim anxiety". Even though on a logical level I know I can use vim and perform basically any task in it, I still have this thought at the back of my head, telling me that it will be unpleasant, awkward, and more difficult, compared to if I was using the good old VSCode.

I've also noticed that I am scared of doing pair programming where I show people my editor. You know this effect - when people are looking at your hands, suddenly your brain starts working at 5% of usual capacity. So imagine using vim and having people look at your monitor. This would be probably the most challenging thing imaginable.

And funnily enough, I will probably have to do it this week, because I pledged to my coworkers that I will prepare some kind of tutorial for them...

So... wish me luck! Next week I will tell you how it all went.

Cover Photo by Salmen Bejaoui on Unsplash

1 Month Vim Only Challenge - The Whats and The Whys

mpodlasin — Mon, 25 Oct 2021 13:05:29 +0000

The Whats

Two days ago, on Friday evening, I uninstalled VSCode and officially started my #1MonthVimOnlyChallenge.

In fact, I am currently writing this article in vim.:w

My plan is to use vim as my editor/IDE of choice for 1 month. No matter what happens, no matter where it leads me - for 1 month I am not allowed to use any other IDE/coding editor/text editor. Only vim is allowed, although I will probably check out neovim as well at some point.

This includes my personal stuff - coding side projects, writing articles etc. - but, perhaps more importantly, this includes my day job as well. And between my day job and personal stuff, I do a lot of writing on a daily basis, including weekends. So that means a lot of vim over the next 30 days.

This article serves as a kickoff for this project. I plan to follow it up with 3 or 4 short articles, released on a weekly basis, documenting my progress, setup and reflections on using vim as a primary editor.

In the last article of the series, I plan to make the ultimate decision.

Either I will go back to the usual way of doing things, or I will stick with vim for the foreseeable future. Nothing is predetermined here. My decision will fully depend on how the next 30 days will go.

Apart from weekly articles, I also plan to do more current updates on Twitter. There, on a daily basis, in real-time I will be sharing my thoughts and experiences of using vim.

Now, at this point, it's important to stress that I am not going into this challenge completely fresh. I've already used vim during my university years, the last time around 6 years ago. However, during that period, I never got to a point where using vim would actually bring me significant productivity benefits. It also never felt 100% natural.

So this will be the barrier I will try to cross. I don't have to become a vim expert. I don't have to be the fastest programmer out there. The main question after 30 days will be this:

Does this feel natural to me?

The Whys

There are actually quite a few reasons why I am trying to attempt such a challenge at this point in my life. So I'll try to summarize them briefly, as a list in no particular order.

1. I Want to Spice Things Up

I love my current job. But at a certain level of experience, many programming tasks become mundane and, frankly, a bit boring.

So I am trying to spice things up for myself. Make things more interesting. Make things a bit harder, more challenging.

Sometimes it's important to switch things up just for the sake of switching things up. It's like changing the wallpaper on a computer. There is no real reason to do that. But sometimes you just want to make things feel fresh again. And sometimes just changing the wallpaper can feel like literally having a brand new computer.

2. I Underuse VSCode Anyways

Recently my colleagues at work started sharing what VSCode plugins they use. Most of them had at least ten plugins installed and running. Meanwhile, I had... one.

Last week a junior dev asked me what was a keyboard shortcut to do a fairly basic task in VSCode. I didn't know. I realized I don't know almost any shortcuts.

So you can see that, even though I've used VSCode literally for years now, I significantly underuse it. I might invest some time in order to fix that. But if I have to exert effort, I might as well do it in a direction that interests me more. Which is vim.

3. I Like The Command Line

This probably relates to the fact that I am used to using the command line, rather than GUI tools. I enjoy it and, depending on the tool, sometimes I feel it gives me a better grasp of how things are working under the hood.

I use git in the command line exclusively. It's funny to me because people are scared of using git in the command line, while I am scared to use git in any of the GUI incarnations. I just have trust issues.

I also very, very, very rarely use a GUI debugger, which is one of the most common arguments people give me for using a solid IDE. I belong to the good old console.log school of debugging. No matter how much people are joking about that, and how much amateurish it seems, for me, 99% of debugging is always the question of "what value this variable has at a certain point?".

And for that, console.log was always more than enough for me. Probably because I do mostly front-end/browser stuff, where such logging is particularly easy and powerful.

4. I Want To See Where Vim Is At These Days

Now, there obviously are capabilities of VSCode which I am using extensively. Those are mostly related to the exceptional TypeScript support that VSCode has. So everything from syntax highlighting, through autoimports and autocomplete, up to hovering over stuff to see the inferred types.

And that's where frankly my curiosity gets me. I want to see where vim is at these days when it comes to integration with other tools.

Vim has a plugin system and a big, very involved community creating those plugins. Bare vim is very simple, but I want to see how far you can push vim these days. Is it really possible to transform this command line text editor into a fully-featured IDE? And if so, how difficult will that be?

I am curious about answers to those questions, so - apart from simply learning vim and writing in vim - I plan to invest significant time into researching and testing various plugins.

I am dying to see how the vim I am using now - just a few days into the challenge - will differ from the one I will use in one month.

5. I Want To See If The "Holy Grail" Is Real

Vim, as exotic and bizarre as it might seem, has a great promise.

It promises that - if you exert significant enough effort - in the end, it will make you faster and more productive than a regular text editor. It will transform you from a human being and make you one with the text you are writing and editing, to the point where you will not feel that there is a keyboard separating you.

It might take a long time, almost for sure more than 1 month. But in principle it's possible - promises vim and its followers.

So I want to test that promise. Of course, as I just mentioned, reaching this "holy grail" proficiency might take a lot longer than 1 month.

But writing daily for 30 days straight should at least give me a glimpse of such mastery. It should at least convince me that it is possible.

And that's probably what I am curious about the most. What is the answer? I'll tell you in 30 days.

Cover Photo by Igor Rodrigues on Unsplash

Haskell - The Most Gentle Introduction Ever (Part II)

mpodlasin — Sun, 24 Oct 2021 12:40:25 +0000

This is the second article in my little series explaining the basics of Haskell.

If you haven't yet, I would recommend you to read the first article, before diving into this one.

So far we've learned the basics of defining functions and datatypes, as well as using them, all based on the Haskell Bool type.

In this article, we will deepen our understanding of functions in particular, while also learning a bit about built-in Haskell types representing numbers.

Type of Number 5

Before we begin this section, I must warn you. In the previous article, I purposefully used the Bool type, due to its simplicity.

You might think that types representing numbers would be equally simple in Haskell. However, that's not really the case.

Not only does Haskell have quite a lot of types representing numbers, but there is also a significant effort in the language to make those types as interoperable with each other as possible. Because of that, there is a certain amount of complexity, which can be confusing to beginners.

To see that, type in the following in the ghci:

:t 5

You would probably expect to see something simple and concrete, like Number or Int. However, what we see is this:

5 :: Num p => p

Quite confusing, isn't it?

For a brief second try to ignore that whole Num p => part. If it wasn't there, what we would see, would be just 5 :: p So what is written here is that the value 5 has type p.

This is the first time we see the name of the type being written using a small letter. That's important. Indeed, p is not a specific type. It is a type variable. This means that p can be potentially many different, concrete types.

It wouldn't however make sense for 5 to have - for example - Bool type. That's why Num p => part is also written in the type description. It basically says that this p has to be a numeric type.

So, overall, 5 has type p, as long as p is a numeric type. For example, writing 5 :: Bool would be forbidden, thanks to that restriction.

The exact mechanism at play here will not be discussed right now. We still have to cover a few basics before we can explain it fully and in detail. But perhaps you've heard it about it already - it's called typeclasses. We will learn about typeclasses very soon. After we do, this whole type description will be absolutely clear to you.

For now, however, we don't need to go into specifics. Throughout this article, we will use concrete, specific types, so that you don't get confused. I am just warning you about the existence of this mechanism so that you don't get unpleasantly surprised and discouraged when you investigate the types of functions or values on your own.

Which I encourage you to do! Half of reading Haskell is reading the types, and you should be getting used to that.

Functions and Operators

Let's begin by doing some simple operations on numbers, familiar from other programming languages.

We can, for example, add two numbers. Writing in ghci:

5 + 7

results in a fairly reasonable answer:

But to get a deeper insight into what is happening there, let's write a "wrapper" function for adding numbers.

We will call it add and we will use it like so:

add 5 7

As a result, we should see the same answer as just a moment ago:

We would like to, of course, begin with a type signature of that function. What could it potentially be?

add :: ???

As I mentioned before, Haskell has many different types available for numerical values.

Let's say that we want to work only with integers for now. Even for integers, there are multiple types to choose from.

The two most basic ones are Integer and Int.

Int is a type that is "closer to the machine". It's a so-called fixed precision integer type. This means that - depending on the architecture of your computer - each Int will have a limited number of bits reserved for its value. Going out of those bounds can result in errors. This is a type very similar to C/C++ int type.

Integer on the other hand is an arbitrary precision integer type. This means that the values can potentially get bigger than those of Int. Haskell will just reserve more memory if that becomes necessary. So those integers are "arbitrarily" large, but, of course, only in principle - if you completely run out of computer memory, nothing can save you.

At the first glance, it would seem that Integer has some clear advantages. That being said, Int is still being widely used where memory efficiency is important or where we have high confidence that numbers will not become too large.

For now, we will use the Integer type. We can finally write the signature of our function:

add :: Integer -> Integer -> Integer

What we have written here is probably a bit confusing at the first glance.

So far, we've only written declarations of functions that accept a single value and return a single value.

But when adding numbers, we have to accept two values as parameters (two numbers to add) and then return a single result (a sum of those numbers). So in our type signature, the first two Integer types represent parameters and the last Integer represents the return value:

add :: Integer {- 1st parameter -} -> Integer {- 2nd parameter -} -> Integer {- return value -}

(By the way, you can see here what syntax we've used to add comments to our code.)

Let's now write the implementation:

add :: Integer -> Integer -> Integer
add x y = x + y

Simple, right?

Create a file called lesson_02.hs and write down that definition. Next, load the file in ghci (by running :l lesson_02.hs) and type:

add 5 7

As expected, you will see the reasonable response:

I will now show you a neat trick. If your function accepts two arguments - like it is the case with add - you can use an "infix notation" to call a function. Write:

5 `add` 7

You will again see 12 as a response.

Note that we didn't have to change the definition of add in any way to do that. We just used backticks and we were immediately able to use it in infix notation.

You can use this feature with literally any function that accepts two parameters - this is not a feature restricted only to functions operating on numbers.

At this point those two calls:

5 `add` 7

and

5 + 7

look eerily similar.

That's not an accident.

Indeed, you can do the reverse, and call the + operator as you would call a regular function - in front of the parameters. If it's an operator, you just have to wrap it in parentheses:

(+) 5 7

Try it in ghci. This works and returns 12 again!

Perhaps you already know where am I going with this.

I am trying to show you that the built-in + operator is indeed just a regular function.

There are of course syntactical differences (like using backticks or parentheses for certain calls), but conceptually it is good to think of + as being no different than add. Both are functions that work on the Integer type - accept two Integer numbers and return an Integer number as a result.

Indeed, you can even investigate the type of an operator in ghci, just as you would investigate the type of add function.

Typing:

:t (+)

results in:

(+) :: Num a => a -> a -> a

This means that + has type a -> a -> a, where a has to be a numeric type. So it's a function that accepts two parameters of numeric type a and returns the result of the same type.

I hope that at this point it makes sense why it is beneficial for the + operator to have such an abstract definition. A clear benefit + has over our custom add function is that + works on any numeric type. No matter if it's Integer, Int, or any other type that somehow represents a number - + can be used on it. Meanwhile, our add function works only on Integer types. For example, if you try to call it on - very similar - Int numbers, the call will fail.

So you can see that complexity introduced in number types doesn't come out of nowhere. It keeps the code typesafe, while still allowing huge flexibility. Types might seem complex, but this makes writing actual implementations a breeze.

Partial Application

Let's go back to the type of add function, which is probably still friendlier to read at this point.

add :: Integer -> Integer -> Integer

The way we have written the type definition here might be surprising to you. We see two -> arrows in the definition, almost suggesting that we are dealing with two functions.

And indeed we are!

To increase the readability even more, we can use the fact that -> is right-associative. This means that our type definition is equivalent to this:

add :: Integer -> (Integer -> Integer)

Let's focus on the part that is outside of the parentheses first:

add :: Integer -> (...)

This says that add is a function that accepts an Integer and returns something.

What is that something? To find out, we have to look inside the parentheses:

(Integer -> Integer)

That's again a function! This one also accepts an Integer as an argument. And as a result, it returns another Integer.

So if we look at the type of add again:

add :: Integer -> Integer -> Integer

we see that - in a certain sense - I was lying to you the whole time!

I was saying that this is how we describe a function that accepts two parameters. But that's false! There is no function that accepts two parameters here!

There is only a function that accepts a single parameter and then... returns another function!

And then that second function accepts yet another parameter and just then returns a result!

To state the same thing in terms of another language, here is how you would write a regular function that accepts two parameters in JavaScript:

function add(x, y) {
  return x + y;
}

However what we are creating in Haskell is something closer to this JavaScript code:

function add(x) {
  return function(y) {
    return x + y;
  }
}

Note how we have two functions here, each accepting only a single parameter.

In the code snippet above, function add accepts parameter x and the second, anonymous, function accepts the parameter y. There is no function here that accepts two parameters.

So, based on what we have said so far, in Haskell we should be able to call the add function with only one parameter and get a function, right?

Let's try that in ghci:

add 5

Regrettably, we get an error message:

<interactive>:108:1: error:
 • No instance for (Show (Integer -> Integer))
 arising from a use of ‘print’
 (maybe you haven't applied a function to enough arguments?)
 • In a stmt of an interactive GHCi command: print it

But that doesn't happen, because we did something wrong. The problem arises, because add 5 returns - as we stated - a function, and Haskell doesn't know how to print functions.

We can however check the type of the add 5 expression and this way convince ourselves that this indeed works:

:t add 5

As a result, we see:

add 5 :: Integer -> Integer

So it's a success! The expression add 5 has type Integer -> Integer, just as we wanted!

We can convince ourselves even more that that's the case, by calling that add 5 function on a number:

add 5 7

Oh... Wait. We just discovered something!

It was probably confusing so far why Haskell has this strange way of calling functions on values, especially on multiple values. We were just separating them by space, like so:

f x y z

Now it becomes clear, that Haskell simply deals with single-argument functions all the time, and calling a function on multiple values is simply an illusion!

Our call:

add 5 7

is equivalent to:

(add 5) 7

First, we apply add to 5 and as a result, we get a function of type Integer -> Integer, as we've just seen.

Then we apply that new function (add 5) on a value 7. As a result, we get an Integer - number 12.

Note that this means that in Haskell function call is left-associative:

(add 5) 7

Contrasting with what we found out before - that type definition of function is right-associative:

add :: Integer -> (Integer -> Integer)

It is of course done this way so that we get a sane default. Thanks to those properties, in the case of both defining and calling the add function, we can simply forget about parentheses:

add :: Integer -> Integer -> Integer

add 5 7

How else can we convince ourselves that the result of calling add 5 is an actual, working function? Well... let's give a name to that function and use it that way!

In your lesson_02.hs file add the following line:

addFive = add 5

Load the file in ghci.

First let's investigate the type one more time, just to be sure what we are dealing with:

:t addFive

This gives us:

addFive :: Integer -> Integer

Exactly what we expected after applying the add function to one argument.

We could have also written down that type definition explicitly in our file:

addFive :: Integer -> Integer
addFive = add 5

Do that to convince yourself that it all compiles when written this way.

Now you can use your, highly specific, addFive function to... add five to integers.

addFive 7

This results in the expected:

Now, I am sure this example with adding the number five seems a bit silly to you, and rightfully so.

But I hope that it shows you the power of partial application in Haskell, where you can easily use highly general functions, accepting a higher number of parameters, to create something more specific and fitting your particular needs.

For example, you can imagine a function that needs some kind of complex configuration in order to work - let's call it imaginaryFunction.

Let's assume that this configuration is some kind of data structure of type ComplexConfiguration. We can make that configuration the first argument of our function:

imaginaryFunction :: ComplexConfiguration -> OtherParameter -> Result

Why do we want to pass it as a parameter instead of just having it "hardcoded" inside the function? Who knows, perhaps different versions of our app need different configurations. Or perhaps we just need to change its value in our unit test suite.

If we do that, then, in the actual app, we can simply apply imaginaryFunction to a specific ComplexConfiguration:

configuredImaginaryFunction = imaginaryFunction config

After that, we can use the configuredImaginaryFunction directly, without the need to explicitly import config object, whenever we want to use imaginaryFunction in our code. Haskell just carries that config around for us!

Sweet!

More Numbers and Operations

This section will be far from a complete rundown, but I still want to quickly take you up to speed with basic operations on numbers in Haskell.

So far we've covered two numeric types - Integer and Int and we've shown that we can add them.

Just as in other languages, we can also subtract and multiply them in the usual way.

So:

5 - 2

results in:

while:

5 * 3

results in:

Other popular number types are Float and Double. Those are number types that can represent numbers beyond simple integers. The main difference between Float and Double is how big their storage capacity is. Most of the time you will likely use Double unless you have some specific reason to use Float.

Float and Double values can be added, subtracted, and multiplied just like Int and Integer values.

Let's see some examples, just to make ourselves comfortable with that:

1.1 - 0.1

results in:

1.0

1 + 0.1

result in:

1.1

Note that in Haskell's standard library there are much more numeric types available. Some of them are fairly common, others are fairly specific. There are also many more built-in functions.

This section was just meant to make you comfortable with making simple operations on basic number types. In the future we will come back to numbers for sure - there is a great deal of interesting type-level stuff at play here, and we will want to cover that for sure!

Identity

Let's now take a small break from numbers and go back to our beloved booleans.

Previously, we have written a not function, which was negating the booleans - converting True to False and False to True.

But what if we wanted to do... the opposite?

What if we wanted to create a function that... returns True when passed True and returns False when passed False?

This might sound a bit nonsensical. In a way, this function would do literally nothing. However, we will see that it will have a tremendous educational value for us, so let's curb our doubts and let's try to write it anyway.

First, let's start with the name and type signature as all Haskell programming should.

In mathematics, a function that takes a value and returns the exact same value is usually called an identity function. Since this one will operate on the Bool type, we will call it boolIdentity.

boolIdentity will receive a single argument of type Bool and return the same thing, so... Bool as well! Therefore its type signature is this:

boolIdentity :: Bool -> Bool

Now let's get started with actual implementation. Your first instinct might be to write something like this:

boolIdentity True = True
boolIdentity False = False

This will work perfectly fine and is a valid solution, but there is a way to write the same thing in a more terse way.

After all, we are returning the same value that we are receiving as a parameter, so we can simply write:

boolIdentity x = x

In the end, our whole definition looks like that:

boolIdentity :: Bool -> Bool
boolIdentity x = x

Write that down in your lesson_02.hs file and reload the file in ghci.

You can convince yourself that our function works, by running it:

boolIdentity True

This call results in:

True

And at the same time boolIdentity False returns False (hopefully not surprisingly).

Now let's create a similar function, but for numbers - let's say for Integer type. We want a function that will take an Integer value and return the same value. For example, if we call it with 5, we want to see 5 again.

Let's call it integerIdentity. Let's begin by writing the type signature:

integerIdentity :: Integer -> Integer

This was simple.

Now let's think about the implementation. Well... we want to take the parameter passed to the function and... just return it!

So we get:

identityInteger x = x

But... but this is exactly the same implementation as in the case of identity for Bool values!

Let's compare the two:

boolIdentity :: Bool -> Bool
boolIdentity x = x

integerIdentity :: Integer -> Integer
integerIdentity x = x

Everything looks the same. The only difference here is the types. In the first function, we operate on the Bool type. In the second we operate on the Integer type.

Now, if only there was a way to write that function only once. In the current state of things, we would have to write an identity function for each type in existence, which... sounds daunting, to say the least.

It luckily turns out that Haskell does have a mechanism to deal with that easily. Not only that - we've already encountered that mechanism!

Remember how the type of number 5 was Num p => p? The p in the type description was a type variable - basically a placeholder for actual, concrete types.

We've also seen that the + operator was quite general - it could be called on any numeric type. It also had a type variable in its type signature.

So the question is, can we use a type variable, to write the most generic version of the identity function possible? The answer is... absolutely!

Let's remove the two previous identity functions and replace them with only one:

identity :: a -> a
identity x = x

Note how we used a as a type variable here. The 3 type definitions we've seen so far, share the same "shape". You can see that the type definitions of identity for Bool type and for Integer type both "fit" this new type definition if you imagine variable a being a placeholder for other types:

boolIdentity :: Bool -> Bool

integerIdentity :: Integer -> Integer

identity :: a -> a

Let's run the code in ghci and convince ourselves that we can indeed use this new, generic identity on both Bool and Integer values:

identity True

works and results in:

True

And at the same time:

identity 5

works as well and results in:

At this point, it's important to emphasize a certain point. Given the implementation that we've used for the identity function:

identity x = x

we could not give it, for example, the following type:

identity :: Integer -> Bool

This type definition in itself is not absurd. You can easily imagine functions that accept integers and return true or false (for example based on some condition).

However, in this particular case, where we take argument x and immediately return it, without doing anything else, it's clearly impossible for x to "magically" change the type.

And this fact is reflected even in the most generic type definition of identity:

identity :: a -> a

Note that this definition states that identity accepts a value of type a and returns a value of that same type - namely a again.

On the flip side, the following definition:

identity :: a -> b

wouldn't be allowed. In fact, it won't compile, with an error, part of which says:

Couldn't match expected type ‘b’ with actual type ‘a’

So the compiler literally says that in place of b there should be the type variable a present.

That's because - given the current implementation - it's impossible for the value named x to just change the type out of nowhere.

And indeed, when Haskell infers the type of untyped code, it goes for the most general interpretation possible.

You can convince yourself of that, by removing the type definition from lesson_02.hs file, and leaving only the implementation:

identity x = x

And then compiling it in ghci and checking the type of identity by running:

:t identity

As an answer you will see:

identity :: p -> p

This is exactly the same type definition that we wrote by hand. It simply uses a different letter (p instead of a).

At the very end of this section, it would be good to mention, that you don't actually have to define the identity function by yourself. We only did it for educational purposes.

Prelude - the standard library for Haskell - has it always available for you under the shorter name id.

To convince yourself of that, write the following in ghci:

:t id

As a response you will see, more than familiar now, type:

id :: a -> a

Conclusion

In the second part of "The Most Gentle Introduction Ever" to Haskell, we used operators and functions on numbers and discovered that they are in fact almost the same. We've also seen that there are no functions of multiple variables in Haskell - they are just functions that return other functions.

And at the end, we expanded our understanding of what a type definition can be, by showing the simplest possible usage of type variables - using the identity function as an example.

All those things were meant to make you feel more comfortable with the concept of a function in Haskell. And in the future article, we will use a similar approach to enhance our understanding of algebraic data structures, especially custom ones.

So see you next time and thanks for reading!

Haskell - The Most Gentle Introduction Ever

mpodlasin — Thu, 29 Jul 2021 19:13:49 +0000

Who Is This Article For?

This article is the first in (hopefully) a series on functional programming in Haskell.

It doesn't assume any previous knowledge of Haskell or even functional programming for that matter.

It does however assume that you can already program in some programming language.

If you feel fairly comfortable in a language like JavaScript, Python, Java, C/C++, or anything similar, you are more than capable of going through this tutorial. You can rest assured that everything will be explained slowly and carefully.

The main point of this series will be to highlight the differences between Haskell and those "typical" languages that I've mentioned. So the less you know about Haskell and/or functional programming, the more illuminating and mindbending those articles will be for you.

I will also be showing you how learning Haskell can benefit you in writing better code even when using other, more mainstream languages. If you feel stuck when it comes to your programming skills and if you feel like you haven't been stretching your coding muscles lately - Haskell will be perfect for you!

Whether you are a senior coding veteran or a junior dev that barely started your career, Haskell will push you to be an all-around better programmer. If you need more convincing, in the past I've written about why it is beneficial to learn Haskell, even if you don't plan to code in it professionally.

Are you ready? Let's go then!

Installing Haskellers Toolbelt

If you are on this journey with me, we'll begin by installing some software needed to run Haskell.

But if you are still unsure/unconvinced, you don't have to do even that. I will be keeping all the examples as simple as possible, so it will be enough to just use a REPL like this one. The only thing you need to do there is to empty the file in the REPL - we will start from scratch - and run ghci command in the terminal. You are ready now, so if you don't want to install Haskell tools on your computer, you can safely omit the rest of this section.

Still here? All in on learning Haskell? Awesome!

The download section on haskell.org can be a bit confusing, so I would recommend you to go straight to ghcup page. You just have to copy the script from the website, paste it to your terminal and run it.

ghcup is a "Haskell toolchain installer". This means that it allows you to easily install and manage various tools related to Haskell.

The installation process guides you by hand - you will just have to answer a few questions.

The installer will ask you about installing secondary packages - stack and hls.

hls is "Haskell Language Server". It allows IDE plugins and extensions to work seamlessly with Haskell. So it's worth having it since the very beginning.

stack is a build tool that allows you to create isolated Haskell projects easily. This would be massive overkill for simple, one-file scripts that we will be writing at the beginning. However, once we move on to writing more complex programs, Stack will be extremely helpful. Whether you prefer to install it now or later - it's up to you.

After running the script, you can test if everything went well by running the ghci command in your terminal.

We've mentioned quite a few tools so far, but ghci is the one that we will actually use in this article. It is an interactive environment for running Haskell code. ghc (no "i" at the end!) stands for Glasgow Haskell Compiler. ghc is in fact its own command, that will allow us to compile code into binary executables.

However, if we just want to play around with code and test some stuff, an interactive ghc - called ghci - is perfect for that, because it allows us to run Haskell code without the compilation step. On top of that, it has some handy commands which we will use today.

If you type in ghci in the terminal and get an error, it likely means that your terminal doesn't know how to find the ghci binary. You might have to close and reopen the terminal window after finishing the installation.

If it's still not working, then - depending on the environment you are using - you will have to edit your .bashrc or .zshrc file.

Luckily ghcup is very tidy - it just installs everything in a single directory, in my case /Users/mateusz.podlasin/.ghcup. In that directory, there is a bin folder. You need to point the terminal you are using to that folder.

So, in my case, I had to add the following line to my .zshrc file:

export PATH="/Users/mateusz.podlasin/.ghcup/bin:$PATH"

After closing the terminal and opening it again, running ghci should now result in the following output:

GHCi, version 8.10.5: https://www.haskell.org/ghc/ :? for help
Prelude>

If you see this, you are ready to begin our Haskell adventure.

Playing With Booleans

A boolean (in Haskell named Bool) is one of the simplest and most familiar types that a programmer encounters regularly. No matter what programming language you've used previously, you likely know booleans very well.

That's why they will be perfect for learning the basics of Haskell.

In Haskell boolean values are written starting with a big letter, so we have True and False.

In ghci you can type in:

:t True

:t is a ghci command for checking the type of a value. Note that this command is not a part of Haskell itself, just a ghci functionality.

After clicking enter, as a result, you will see:

True :: Bool

You can read :: as "has type". So this line says that the value True has type Bool.

Just to make sure we understand that correctly, let's check the type of value False:

:t False

We see:

False :: Bool

As you probably anticipated, the value False has type Bool as well. So True and False are of the same type. And in Haskell - quite reasonably - True and False are the only values of the type Bool.

Contrary to :t, the <value> :: <type> syntax is a part of Haskell language. And this should already hint at something to you. If a language has a dedicated syntax to express the sentence "<value> has type <type>", it means that this language probably treats types fairly seriously.

As we will soon see, types are at the very heart of programming in Haskell. As a matter of fact, sometimes when coding in Haskell you will be thinking about types more than about actual values!

And while we are here, note that the name of the type - Bool - is also written starting with a big letter in Haskell, just like the names of values. This will be important later.

Let's now start writing some actual code. We will still use ghci to run it, but we need to write it in an actual file.

Create a file called lesson_1.hs (in REPL I linked, the file is already created for you and is called main.hs). Note the .hs suffix, which represents Haskell source code files.

You can create that file anywhere you want. You can also edit it with any text editor you desire.

I would recommend using Visual Studio Code. After installing VS Code, immediately go to the "Extensions" tab (on the left) and install the "Haskell" extension. It will give you many handy features. It will for example start displaying the types of values you hover over. If you have trouble with the extension "not detecting ghc", remember to edit the PATH in the .bashrc/.zshrc file, just like I described in the setup section. After that restart the editor.

After you've created the file, make sure to be - using the terminal - in the same directory where the file is located. To do that, you can leave ghci by running :q. Then switch directories to a proper one and run the ghci command once again.

In ghci try loading the file, by running:

:l lesson_1.hs

Even though your file doesn't have any code in it just yet, you should see a success message like this:

[1 of 1] Compiling Main ( lesson_1.hs, interpreted )
Ok, one module loaded.

If you don't see that, you've likely run ghci in a different directory than your file is located.

But if you see the message, you are ready to begin coding!

So we have types (Bool) and values (True and False). What can we do with them? The most basic thing, known from other languages, would be to assign a value to a variable.

Write the following line in lesson_1.hs:

x = True

Now load it again in ghci with the same command as before - :l lesson_1.hs (remember to save the file beforehand).

In ghci you can now type:

As a response you will see:

True

This shows that, indeed, we assigned the value True to the variable x.

Now let's do something that might surprise you and that will show you just like radically different Haskell is from "regular" languages.

In lesson_1.hs file let's write:

x = True
x = False

Load it again in ghci. You will see the following output:

[1 of 1] Compiling Main ( lesson_1.hs, interpreted )

test.hs:2:1: error:
 Multiple declarations of ‘x’
 Declared at: lesson_1.hs:1:1
 lesson_2.hs:2:1
 |
2 | x = False
 | ^

The most important part of that error message says that there are Multiple declarations of ‘x’.

It turns out that in Haskell, once you've assigned a value to a variable, you can't overwrite it or change it in any way. Ever.

If it's a Bool, you can't change it from True to False. If it's a number (which we will cover in future articles), you can't change its value. For example, you can't even increase the value of a numeric variable by one!

This sounds incredibly radical to someone used to traditional, imperative programming. Idioms like:

i++;

someBoolean = !someBoolean;

etc. are so prevalent in those languages that thought of only assigning a value to a variable once sounds... well... frankly just crazy!

Is it even possible to write actual, real-world programs with a language like this?

The answer is absolutely, and we will see that quite soon. But for now, you just have to accept that - once assigned a value - you can't ever change that variable anymore.

You've likely heard about immutability at this point. I already wrote about immutability in JavaScript for example.

That's the thing though. In those other, imperative languages immutability has to be introduced via some library or specific programming approach.

It's exactly the reverse in Haskell. Here immutability is the default and you have to use libraries or certain methods to achieve mutable variables/state.

That might sound cumbersome, but it's not an accident that the principle of immutability became so popular even in the mainstream, "mutable by default" languages. It really makes your code less buggy, safer, and more predictable.

Ok, enough of talky talk. So we know that once created, we can't alter that variable. But it is by no means useless. We can now call some functions using it.

Remove the x = False line from your file, so that loading the file in ghci works again and x has the value True.

Then run the following in ghci:

not x

As an answer you will see:

False

Now, if you've coded in Python or a language with similar syntax, you might think that not is some special, reserved keyword for negating booleans.

No. In Haskell, not is just a regular function.

In a typical language, a function call would look something like that:

not(x)

But this is not the case in Haskell. In Haskell, you write the function name, and then - instead of parenthesis - you provide arguments by separating them with a single space.

So to call a function f on a variable x you would write:

f x

If f accepted two parameters, instead of typical f(x, y) you would write:

f x y

If f accepted three parameters, you would write:

f x y z

And so on. You get the idea.

Let's go back to our not function. We called it on a variable, but nothing is preventing us from calling it on values directly.

In ghci type:

not True

and you will see:

False

Typing:

not False

results in the response:

True

Now, just as values, functions have types as well. We can investigate the type of not function in the same way we investigated the types of True and False - using the :t command in ghci.

Type:

:t not

and you will see the following answer:

not :: Bool -> Bool

We again see the :: symbol, which means "has type". We see the Bool type mentioned twice.

The only new symbol here is ->. As you probably expect, <something> -> <something else> reads as "function from <something> to <something else>".

So the output that we got from ghci can be read as "not has the type of function from Bool to Bool". Or, more naturally, "not is a function from Bool to Bool".

This shouldn't be surprising. When we call not on a Bool value, we expect to see the Bool value as a result - the "opposite" of what we've passed. If we called not True and got 15 as an answer, we would be extremely confused, wouldn't we?

Writing Functions

At this point, I would like to prove to you that there is nothing magical about not. If it's just a regular function, you should be able to write it by yourself, right?

Yup, and that's exactly what we will do right now. We will write our first Haskell function!

We will do that in our lesson_1.hs file.

Under the x definition, write the following line:

myNot x = if x then False else True

Let's break down what is happening here a little bit.

First, we have something that looks exactly like a call of a function - beginning with the name (myNot), and later the parameters of the function, separated by spaces. In this particular case, we have only one parameter, which we named x.

Next, we have an assignment (= character), after which we write the actual function - the part which we call "function body". In this case, the function body is just a simple if then else expression. Let's break it down.

Right after the if keyword we have to provide a condition. Our condition is really x == True. You probably recognize == from other languages. In Haskell it means the same thing - it's an equality operator.

But x == True is equivalent to simply writing x. After all, if x == True evaluates to True, this means that x itself has the value of True. So we can just write x as our condition, for brevity.

If the condition in if then else (our x) evaluates to True, the function will return the value after the then keyword. If it evaluates to False, the function will return the value after the else keyword. Quite simple.

Note that in Haskell if then else is an expression. This means that in the end, it evaluates to a value - one of the two provided after "then" and "else" keywords. In JavaScript for example this means that it has more common with a ternary operator, rather than regular if/else statement.

The following code in JavaScript:

const someVariable = condition ? firstValue : secondValue;

would be therefore equivalent to the following in Haskell:

someVariable = if condition then firstValue else secondValue

Let's return to the definition of our function:

myNot x = if x then False else True

I will admit that I've used x as a parameter name here just to confuse you a little bit. For a second you might think that there is a naming conflict between x that we defined earlier and the x from the function.

You can however convince yourself that that's not true, by loading the file again in ghci. It loads properly, without any errors. On top of that, our newly defined function myNot actually works.

Calling in ghci:

myNot True

returns

False

and vice versa.

We can even use the function on our x variable defined above it.

Running:

myNot x

results in:

False

That is correct because in the file we assigned x to True.

So there is no naming conflict, but it is true that we are "shadowing" the x variable. If we now wanted to use it in the function body, we couldn't, because we decided to give the same name to the function parameter.

So, just for clarity, let's change the name of the function parameter to b (as in "boolean").

The whole file looks now like this:

x = True

myNot b = if b then False else True

This will help us not get confused.

Perhaps the function definition that we came up with is not at all what you expected. We moaned for so long about the importance of types, but now we've written something that borderline looks like untyped Python. What's going on? Where are those scary types?

It turns out that Haskell's type system is so powerful, that most of the time it can infer what should be the type of a function - or a value - you've written. Save the file, load it in ghci and write:

t: myNot

You will see:

myNot :: Bool -> Bool

So indeed your custom myNot function has the same type as the original not. But how Haskell came to that conclusion?

It's quite straightforward.

Since you wrote if x then ..., using x as a condition, Haskell knew that parameter x had to be a Bool. That's because Haskell is (again!) quite strict here and only the value of type Bool can be used as a condition in the if then else construct. (Note that we've written "as a condition" here! It's completely fine to provide values of other types after then and else keywords - we will see that in future articles.)

And at the same time, you wrote then False else True. Here, in both cases ("then" case and "else" case) you are returning a Bool. Therefore the output of your function has type Bool as well.

Those two facts combined bring us to a conclusion that the type of myNot has to be Bool -> Bool.

Haskell inference is extremely good and the compiler may only have problems if you write something that is inherently vague.

And yet it is still recommended to put the type signature of the function in the code. You will see that in the vast majority of Haskell codebases the types are always written explicitly.

You can do it by writing the type of the function above its definition:

myNot :: Bool -> Bool
myNot x = if x then False else True

Writing down the type in the code has some major advantages.

First of all, it increases the readability of the code. Types are incredibly useful as documentation of what your function does. In Haskell, you will often be able to find out what a given function does simply by looking at its name and type signature. No need to read the implementation!

Function called not that has type Bool -> Bool? It surely must be a function negating the booleans!

Second of all, it's valuable to write the type of the function before writing the function definition itself. If you do that, Haskell's type system will "guide you" and help validate that your code works as expected.

After all, inferred type of a function may differ from what you have intended.

Writing the type beforehand is almost like sketching or designing a function, before actually writing it. Personally, I find that writing down the type first often gives me a better idea of how to implement the function.

This is an example of a skill that Haskell teaches you, that you can easily transfer to other languages. Even when I'm writing untyped JavaScript, I still always start by thinking about what kind of type signature my function will have. This helps me to write code faster and make fewer bugs, even though I have to be my own type-checker in that case.

So we have successfully replicated the not function - myNot has the same type and behaves in the same way. Running myNot True evaluates to False, running myNot False evaluates to True.

But let's stay on this topic a bit longer and try to write the same function in a completely different manner.

Let's write:

myNot :: Bool -> Bool
myNot True = False
myNot False = True

You can check for yourself that this loads properly in the ghci. You can also test that myNot still behaves the same as before.

What is happening here?

We used what is known as pattern matching.

Instead of declaring the parameter of the function as a variable named b, we can avoid naming it entirely and simply substitute it with a value that will be provided to the function once it's called.

When we make a call myNot True, Haskell looks for a definition that "fits" such a call. In this case, it's the first line (not counting the type signature). If we make a call myNot False, then it's the second line that matches that call. Hence the name "pattern matching".

Hopefully, it's clear that if you call myNot with a variable, not an actual value, pattern matching will still work just fine. In that case, Haskell simply evaluates the value of the variable and performs pattern matching then.

So with this new definition, calling:

myNot x

Still properly returns:

False

So we have two versions of the same function:

myNot :: Bool -> Bool
myNot b = if b then False else True

myNot :: Bool -> Bool
myNot True = False
myNot False = True

For all intents and purposes, these two functions behave in the same way (and have the same type). Which one you prefer depends entirely on you.

Do you want to see yet another way to write the same function? Here you go:

myNot :: Bool -> Bool
myNot b
 | b = False
 | otherwise = True

Arguably this way of writing the function looks the most exotic.

It uses what is known as "guards". A guard is basically a condition, which is placed between | and = characters. When the condition is satisfied, a function body that is defined after = gets executed.

In our case, the first guard is b == True. But - just as before - we can write the same condition as simply b. If b is True, this condition is satisfied and the function body that returns False will be executed.

Otherwise (so when b is not True), the function body that returns True will be executed.

Interestingly, there is nothing magical about the keyword otherwise. (Are you noticing a pattern here, where some "feature" of the language is not really a feature, but simply something coded in that language?)

You can convince yourself that this is true by writing otherwise in the ghci console. As a response you will see:

True

Yup. otherwise is nothing more than a regular variable, holding the value True! Just as our x is!

So, without the otherwise our "guarded" version of the function would look like this:

myNot :: Bool -> Bool
myNot b
 | b = False
 | True = True

Just as in if then else, a condition has to be a Bool value. If the last condition is set to True, like it's the case here, it will always hold and therefore it will act as a catchall case if the guards above it fail. Calling it otherwise is just done to make this a bit more readable.

One more thing to mention here is that we used indentation. You see that lines starting with | characters are moved a bit to the right. If we didn't do it and wrote the code like this:

myNot :: Bool -> Bool
myNot b
| b = False
| True = True

we would get an error while loading that file in ghci:

test.hs:3:1: error: parse error on input ‘|’
 |
3 | | b = False
 | ^

Indenting code makes it more readable for humans and - as you can see - also helps the compiler understand that a given line is still a part of the definition that began in the previous line.

How much you indent the code is not important to the compiler - it has to be at least one space. But adding a few more spaces is better to keep the code nicely formatted for humans.

So we have written myNot in 3 different ways. Which one you prefer depends on you. Nevertheless, it's extremely valuable to get to know all 3 constructs that we've used - if then else, pattern matching and guards - because they appear almost all the time in Haskell code.

We used them in separation here, but in the future, you will see that you can mix those constructs in various ways, especially when writing more complex code.

Creating Types and Values

So far we have written a custom not function for the Bool type. But what if we could recreate the Bool type itself, as well as its values? Is it even possible?

Indeed, it's possible and even simple. Let's create a MyBool type. It will have two values - MyTrue and MyFalse. It might seem that we are getting a bit possessive here (wink, wink), but that's only to avoid conflicts with already existing names.

At the top of our lesson_1.hs file let's write:

data MyBool = MyTrue | MyFalse

Easy right?

After you load that file in the ghci, you can run :t MyTrue to doublecheck that MyTrue has type MyBool, mirroring how True has type Bool:

MyTrue :: MyBool

The same is true for MyFalse:

MyFalse :: MyBool

Unfortunately, I lied to you a bit. If you type this in ghci:

MyTrue

You will see a mysterious message:

<interactive>:68:1: error:
 • No instance for (Show MyBool) arising from a use of ‘print’
 • In a stmt of an interactive GHCi command: print it

It happens because ghci tries to simply print the value, but... It doesn't know how!

This is part of the language that will cover in future articles. Luckily we don't have to worry about it for now. Haskell can create sane default for printing, you just have to command it to do that, by adding the following line to type definition:

data MyBool = MyTrue | MyFalse
 deriving (Show)

If you now call ghci:

MyTrue

It will simply print back:

MyTrue

As I said - a sane default. In the future, we will learn how to customize that printing capability, but for now, it's perfectly fine.

You can now write functions that operate on this brand new type, just as we wrote a function for the built-in Bool type.

So if we wanted myNot to work on that custom MyBool type, we would do it like that:

myNot :: MyBool -> MyBool
myNot MyTrue = MyFalse
myNot MyFalse = MyTrue

We simply replaced all occurrences of Bool with MyBool, all occurrences of True with MyTrue, and all occurrences of False with MyFalse.

We now have both a built-in version (Bool) and a custom version (MyBool) of a boolean type. Wouldn't it be convenient to write functions to switch between them?

Let's do it!

First let's write a boolToMyBool function. It will take a regular, Haskell Bool and transform it to our custom MyBool:

boolToMyBool :: Bool -> MyBool
boolToMyBool b = if b then MyTrue else MyFalse

And let's write a function that will do the reverse:

myBoolToBool :: MyBool -> Bool
myBoolToBool MyTrue = True
myBoolToBool MyFalse = False

We used two different ways to write a function on purpose here. Note that we wouldn't be able to use if then else with MyBool type as a conditional, because it works only on the built-in Bool type.

But on the other hand, we can use pattern matching in both, because pattern matching works with custom-defined values without any problems.

Great. We can now transform back and forth between built-in and custom types. Let's try that.

Load the file in ghci with both definitions and run:

boolToMyBool True

You will see MyTrue as a response.

Now let's try to transform the same value back. Let's write:

myBoolToBool (boolToMyBool True)

You will see True as the response.

What happened here is that we converted True to MyTrue using the boolToMyBool function (inside the parenthesis), and then we took that result and converted it back to True using the myBoolToBool function, all in a single expression.

Is it a bit nonsensical example? Perhaps, but it shows us how we can chain multiple function calls.

In fact, the brackets around the first function call are important.

Try to run the following in ghci:

myBoolToBool boolToMyBool True

The compiler will show a fairly elaborate error message, but if you read it carefully, you will find the following sentence:

The function ‘myBoolToBool’ is applied to two arguments,
 but its type ‘MyBool -> Bool’ has only one

Ha, so we found what is the problem. Indeed, our code looks as if we are trying to apply the myBoolToBool function to two parameters - boolToMyBool and True. Remember how f x y was representing calling a function f on two parameters? That's exactly what we are doing here! In that case, f is myBoolToBool, x is boolToMyBool and y is True.

We have to show the compiler that we want to apply the myBoolToBool function to a single parameter. And that parameter is a result of calling boolToMyBool on True. So we wrap that call in parenthesis to make that clearer (both to the compiler and us):

myBoolToBool (boolToMyBool True)

Can we use those conversion functions for something more practical than needlessly translating booleans back and forth? Absolutely - we can use them to rewrite our myNot implementation once again.

We can use the fact that the Bool type has already the not function defined. It works exactly how we want, it just operates on different types. So let's take a MyBool value, convert it to Bool, use not on it, and then convert it back to MyBool:

myNot :: MyBool -> MyBool
myNot mb = boolToMyBool (not (myBoolToBool mb))

Let's review step by step how that function works.

Since it's a function of type MyBool -> MyBool, its argument is a MyBool - that's why we called it mb. We pass that variable to a function - myBoolToBool mb - and as a result, we are getting something of type Bool. Then we are applying not on it, by writing not (myBoolToBool x). Note that at this step the type doesn't change - we are feeding not a Bool and getting a Bool again, since not is of type Bool -> Bool. At the end we convert back to MyBool, by writing boolToMyBool (not (myBoolToBool x)) - which is the return value of our function.

This could be again summarised as follows:

mb :: MyBool

(myBoolToBool mb) :: Bool

(not (myBoolToBool mb)) :: Bool

(boolToMyBool (not (myBoolToBool mb))) :: MyBool

This technique of tracking which value has which type is extremely valuable, especially when just starting with Haskell. Whenever you feel lost about how a certain function works, I would encourage you to analyze all the types of all the values in that function. You will discover that when you know all the types well, it's much easier to understand how the code works and what it does.

This is the second example of a skill that Haskell teaches you, which is immediately transferable to programming in other languages. Even if your day-to-day language is loosely typed and doesn't have types in its actual syntax, you can still use this way of thinking to better understand functions while reading code.

In time you will discover that this allows you to understand code much faster, especially the code that you haven't written yourself.

Conclusion

So we played with Bools in Haskell, serving as our introductory lesson in that language.

We've learned the basics of Haskell's syntax, we've seen how to create and use functions, and even how to create some simple datatypes.

In future articles, we will elaborate on those ideas quite a bit, so if you are hooked on Haskell by now (as I hope you are!), I would recommend to follow me on Twitter. I don't run any kind of newsletter email or anything like that, so that's the best way to get notified when a new article drops.

If you have any questions or comments regarding the article, you can reach me there as well. I would like to keep improving this article so that it serves others as best as possible.

Thanks for reading and see you soon!

I was gone from tech social media for almost half a year. Here is why. (Yes, it was burnout.)

mpodlasin — Fri, 25 Jun 2021 15:46:16 +0000

I've posted my last tech article on January 24th, 2021.

I've made my last tweet on January 28th.

Since then I became silent for almost half a year.

This article is very important to me because I want to achieve a few things with it:

Explain why I was gone for so long.
Use it as an opportunity to talk about an extreme burnout and how to fight it.
Test with this article if I even want to go back to writing tech-related content.

If you've ever experienced - or are currently experiencing - burnout, work-related depression, or anything similar, you might find this article relatable. You won't find easy answers in it, because I don't have them. But it may give you some ideas and suggestions to explore further on your own. At least I hope.

So let's begin!

What happened?

Instead of turning this section into an overly long, verbose description of the events, let's focus on the timeline more. I find it quite revealing how things had progressed:

January 11th, 2021 - I am making a Twitter thread about burnout.

This is fascinating to me because at that moment I am still convinced that I am completely and absolutely fine. And yet you can see that my brain is already circulating around the topic of burnout.

It's as if it is trying to warn me: "man, you have to slow down, otherwise I am going into a shutdown mode soon!".

January 14th - I am making a Twitter thread about my recent failures related to trying to get into Machine Learning.

This one is less obvious, but you can still see that my brain continued to focus on negative topics at that time. Burnout, failure, and depression were clearly on my mind.

And yet, again, at that point I sincerely believed I am "A-okay", not seeing the storm looming just above my head.

January 24th - I am publishing a tech article - the last one in a 3-part series.

I explicitly state on Twitter that this one was a slog to finish. It's the first article that felt like a pain to get done. I felt no joy in the process and was no longer interested in the topics described in the series.

Note also that at that time I started making YouTube videos associated with the articles, covering the same concepts in a video form.

The sheer amount of work - creating an article and a video almost every week - was so big, that I was basically working all the time. I was spending 8 hours coding at my day job, and then, after the day job was done, I was creating programming-related content.

So the structure of my day was:

wake up
code
have a meeting about code
code
eat crappy takeaway
write about code
record talking about code
sleep
repeat

Absolutely unsustainable. And yet even at that point - January 24th - I mostly believed that I am okay. Maybe less motivated and a bit more tired than usual, but nothing crazy.

February 2nd - my team leader writes to me: "Hey I've noticed you seem less energetic & cheerful than you usually are. Is everything going alright? I'm here if there's anything you would like to share."

I am essentially brushing off his question, saying that I indeed felt less motivated recently, but I will probably feel better after resting on the weekend.

Oh, how wrong was I...

February 8th - Monday. I wake up extremely late. Around 3 pm. And I still can't get out of bed. I am just laying there, unable to move at all. Complete shutdown.

I guess my brain incapacitated me to prevent me from continuing my insane work routine.

I sleep a bit more, wake up at 5 pm. Since it's winter, it's already getting dark. I am still unable to perform any meaningful tasks. I can't even get out of the bed and wash myself.

I don't really remember the rest of that day very well. It's likely that I just turned on YouTube or Netflix and that's how I finished the day.

I provided no info - or even sign of life for that matter - to my team or managers, even though it was a regular workday.

February 9th - Tuesday. The same situation. Waking up late. Not being able to move and do anything meaningful. No shower. No food. Just laying there.

I manage to motivate myself to contact my boss and ask him for an unpaid leave till the end of the week.

Everyone at work is super concerned with my situation - I am having conversations with a few people, some on Slack, some on Zoom. Thankfully actually getting a week off turns out to not be a problem at all.

How I got out of my burnout

So I hope that by now you realize just how serious my situation was.

Burnout is a popular word recently, used for a wide variety of conditions. But you can see that in my case it wasn't just simply feeling a bit less motivated or something similar.

I literally couldn't move or perform the most basic life functions, like cleaning myself, preparing a meal, or even going for a walk.

I was in some deep, deep s...

At that point, I was luckily very much aware of the severity of my situation (hard not to be, I guess).

During my week off I decided to deliberately stay away from anything tech or programming related. No coding, no writing about coding, no twitting about coding. No tech social media at all. Even such little thing as browsing Hacker News - absolutely forbidden.

I was hoping that this cold turkey tech detox would help. But I was also prepared to seek professional help if after a week I wouldn't feel any better.

Luckily I did.

A week later I communicated to my boss that I feel well enough to at least attempt coming back.

I still took things very, very slow, however. After I came back, I only did coding for work - no side projects or anything like that.

I was also taking only simpler tasks - the ones I knew I could implement fairly fast and with minimal effort. A complex, big new feature? - pass. A tricky bug to investigate? - pass.

I also kept staying away from tech social media - both producing content and consuming it. I stopped coding or even thinking about coding after work. Instead, I focused on my non-tech hobbies, which I previously neglected, such as photography.

And I believe this last thing the most important decision that contributed to my mental health and well-being in the long run.

Where I am now and what's next?

Avoiding tech social media and coding beyond the day job worked for me so well, that I did it for nearly half a year. I haven't coded a single side project in that time. I didn't create or read a single tech article (unless it was at work and necessary to do my job). Hell, I haven't even made a single tweet since January!

And during those last few months, I felt absolutely... amazing. I am now more energetic and effective at my work than before my burnout. I am cranking out issue after issue, solving difficult bugs, contributing to architectural decisions. If you'd see me at my day job, you would never, ever guess that I was in such a bad place just a few months ago.

Writing this article is the first time I am breaking my "no coding-related activities in my spare time" unspoken rule since my sudden burnout attack.

If I am being honest, it feels both weird and scary to write this article.

I know that I have this overly ambitious, creative bug in me. And I am terrified I am awaking it again with this article. I am seriously scared that writing and publishing this single article will again send me on a path of unsustainable workloads and being consumed by programming.

It seems that some people are more resilient than others and can handle more stress and higher workloads. I see people who work 10 hours a day, create side projects, and write articles regularly on top of that. For someone, it might be funny that I burned out so badly only because I wrote a few articles after work.

Maybe that's the ultimate lesson though. You have to know yourself and be aware of how much you can handle. And stay honest with yourself to not start biting more than you can chew.

That's why recreating the timeline of my burnout was so important to me. You can see that as early as few weeks before an actual burnout there were already warning signs present, which I completely ignored. I should have paid more attention to what my brain was telling me.

So what's next? I don't know. I can say that writing this article felt very good. I guess it was something that I just needed to get off my chest.

Will I go back to creating actual tech content, however? It's possible. Especially if I manage to find a sustainable tempo and topics which I enjoy writing about.

But it's also entirely possible that this is the last article I'll ever write.

And that's fine too.

Cover Photo by Justin Luebke on Unsplash

Generators in JavaScript, Part III - "Advanced" Concepts

mpodlasin — Sun, 24 Jan 2021 15:56:44 +0000

This is the last article in our 3 part series, where we are explaining in great detail what are generators and how they work.

This however doesn't mean that we are finishing dealing with generators just yet. In future articles, as I've been promising for a long time now, we will continue exploring their capabilities, this time in a more practical setting - namely using them with React.

But before we move on to that, we still need to explain some "advanced" concepts. But don't let the title fool you. The knowledge in this article is absolutely necessary to understand generators deeply. So let's get started!

yield expression

So far we only used the yield keyword either on its own, almost like a return, or we used it in such a construction:

const variable = yield something;

But it's important to clarify that you don't have to necessarily write it this way.

yield something is an expression, so you can put it wherever an expression would be acceptable in typical JavaScript.

For example, instead of storing the result of yield something in a variable, only to later console.log it:

const variable = yield something;

console.log(variable);

we might have as well simply written it like this:

console.log(yield something);

Basically, if there is a place where you would put a variable, you can also use the yield something expression directly.

So, for example, all of those examples are correct:

// we used let, instead of const
let x = yield something;

someFunction(yield something);

if (yield something) {
    // do stuff
}

After all - as we've seen - yield something gets "replaced" anyways with the value that you provided as an argument to the next call. So when writing code with yield you just have to imagine someone swapping in your code yield something for an actual value. Does it still look correct? If so, it is also correct with a yield something.

You have to be careful however when combining yield with operators, for example with a plus sign.

yield a + b actually gets interpreted as yield (a + b). If you wanted to yield only a here, you would have to write (yield a) + b.

There are some rules of operator precedence, but in my experience, it is best to just get a feel for it, by playing with some examples and getting a lot of practice. Simply make sure to double-check that your code actually yields the values that you are expecting.

An iterator is more than just next()...

Before we continue, I have to confess to you something... In my iterators series, I haven't told you the whole truth about iterators. And now, before we move to generators again, I need to add some things to what I explained so far in my previous articles.

At this point, you might believe that iterators only possess one method - next.

Although that's the only obligatory method that they need to have, there are also two methods, which your iterators might have, if you decide to implement them.

The first is a return method. This method is used to notify the iterator, that the consumer has decided to stop the iteration before it actually finished. It's kind of a declaration that - although the iteration process hasn't fully completed - a consumer doesn't intend to make more next calls.

This method is actually called by native JavaScript consumers - like a for ... of loop - if they stop iteration prematurely. For example when for ... of loop encounters a break statement or if an exception is thrown in the loop body.

Of course, as we said, this method is completely optional, so if a for ... of loop doesn't find a return method on its iterator, it will simply do nothing. But if the iterator has such a method, it will be called, to notify it that the iteration process ended faster than expected.

Let's take a simple infinite iterator, returning integers, starting from zero:

const counterIterator = {
    index: -1,

    next() {
        this.index++;

        return {
            value: this.index,
            done: false,
        }
    },

    [Symbol.iterator]() {
        return this;
    }
}

Let's add to it a return method. Interestingly, return has to obey the same interface as next. This means it has to return an object of the shape { value, done }.

The only reasonable value for done here is true because after return gets called, the iterator should indeed stop its iteration process. And for a value let's just stick to good old undefined. This property will be more important when we move on to generators.

const counterIterator = {
    index: -1,

    next() {
        this.index++;

        return {
            value: this.index,
            done: false,
        }
    },

    // new `return` method
    return() {
        console.log('return was called');

        return {
            value: undefined,
            done: true
        }
    },

    [Symbol.iterator]() {
        return this;
    }
}

As you can see, we've also added a log, to find out when that return method really gets called.

Let's now run a for ... of loop with a break:

for (let element of counterIterator) {
    if (element > 2) {
        break;
    }

    console.log(element);
}

In the loop, we are simply logging the elements returned by the iterator. If numbers returned from it become bigger than 2, we immediately stop the iteration.

Running this code logs:

0
1
2
return was called

So we see that indeed our return method was called when the break statement was encountered.

Let's now try throwing inside the loop:

try {
    for (let element of counterIterator) {
        if (element > 2) {
            throw 'error';
        }

        console.log(element);
    }
} catch {}

Since we are throwing, we had to wrap our loop in a try-catch block.

And no surprises here - the code logs exactly the same output:

0
1
2
return was called

So whether it's break or throw - if for ... of loop finishes prematurely, it lets the iterator know by calling its return method.

Okay, that's how return works. But... why it's here in the first place? return is very useful for doing cleanups. If there is some logic that is critical for an iterator to perform after the iteration ends, it should be probably put both into return and done. That's because successful iterations - the ones that were running till the end - don't call the return method, so you need to remember to do a cleanup in both cases.

We've mentioned that there are two optional methods that iterators can have. return is one of them, and the second is throw.

throw also has to obey a similar interface as next and return. It's meaning is supposed to be similar to return. The iterator is informed that the iteration process ends prematurely, but it is also encouraged to raise some kind of an error.

Intuitively, throw should be used when something goes really, really wrong. And yet, as we've seen, when for ...of loop encounters an exception, it calls return. It turns out that in that case throw doesn't get called. That's probably because a typical iterator doesn't really care about why the iteration process ends earlier than it should - it just does the necessary cleanup and that's it.

So most of the time, when writing custom iterators, it's perfectly fine to omit throw and only use return.

On the other hand, the behavior of generators will actually differ depending on whether we use return or throw. We will see that in the following sections.

return() with generators

Let's start with running return on generators first.

There are no big surprises here. When the generator gets informed via return call that the iteration process ended early, it just stops from ever returning further values.

Let's take an infinite "counter" iterator the same as before, but written as a generator:

function* counterGenerator() {
    let i = 0;

    while(true) {
        yield i;
        i++;
    }
}

Let's run it by hand, using next and return methods of its iterator:

const iterator = counterGenerator();

console.log(iterator.next());
console.log(iterator.next());
console.log(iterator.next());
console.log(iterator.return()); // a return method!
console.log(iterator.next());
console.log(iterator.next());

This logs:

{ value: 0, done: false }
{ value: 1, done: false }
{ value: 2, done: false }
{ value: undefined, done: true } // logged by `return` call
{ value: undefined, done: true }
{ value: undefined, done: true }

So we see, that while we were calling next methods, the iterator was behaving as usual.

We then called return, which immediately resulted in { value: undefined, done: true } object.

And since then, even though we came back to calling the next method, we couldn't receive further values anymore.

Now perhaps the iterator doesn't return anything, but the generator itself is still running underneath?

Let's check it, by adding some logs to the generator function:

function* counterGenerator() {
    let i = 0;

    while(true) {
        // we are now logging the value
        console.log(i);
        yield i;
        i++;
    }
}

Running the code now results in:

0                            // from generator
{ value: 0, done: false }
1                            // from generator
{ value: 1, done: false }
2                            // from generator
{ value: 2, done: false }
{ value: undefined, done: true }
{ value: undefined, done: true }
{ value: undefined, done: true }

So our doubts were unwarranted - the generator actually stops running completely after we call return on its iterator.

Having a return method on an iterator allowed us to perform some cleanup logic in case the iteration process ended earlier than expected.

Could we somehow replicate that with generators?

Indeed, we can use a try-finally construct for that.

Let's wrap our generator code in try-finally:

function* counterGenerator() {
    try {
        let i = 0;

        while(true) {
            yield i;
            i++;
        }
    } finally {
        console.log('finally was called!');
    }
}

Note that if this was a regular function with a while(true) loop inside, without any returns or errors, the finally block would never get executed because we would never finish running the try block. With generators that's different, because we can now stop executing the try section "from the outside".

In our finally block we have made a simple console.log. Let's again run the previous next and return sequence:

const iterator = counterGenerator();

console.log(iterator.next());
console.log(iterator.next());
console.log(iterator.next());
console.log(iterator.return()); // a return method
console.log(iterator.next());
console.log(iterator.next());

This logs:

{ value: 0, done: false }
{ value: 1, done: false }
{ value: 2, done: false }
finally was called!              <- log from finally block
{ value: undefined, done: true }
{ value: undefined, done: true }
{ value: undefined, done: true }

So indeed, the finally block ran after we've called return on this generator's iterator. So finally block is a place that you can use if you want to implement any kind of cleanup logic.

Now the mystery of why the return method has to return a { value, done } object will be finally (sic!) solved. After all, in regular functions, it's perfectly legal to make a return statement in a finally block. Let's try that here, replacing our console.log:

function* counterGenerator() {
    try {
        let i = 0;

        while(true) {
            yield i;
            i++;
        }
    } finally {
        return -123;
    }
}

Run the code again and you will see in the console:

{ value: 0, done: false }
{ value: 1, done: false }
{ value: 2, done: false }
{ value: -123, done: true } // result of `return` call
{ value: undefined, done: true }
{ value: undefined, done: true }

We see that now the result from the return method contains an actual value - in this case, a -123 number - instead of undefined.

Ha! So this way the generator can still communicate something to "the outside", even if the iteration process gets somehow interrupted!

throw() with generators

Let's now solve the mystery of the throw method.

With iterators, it was a bit unclear why actually this method is needed.

As we've said previously, it's supposed to signalize to the iterator, that the iteration failed in a very bad way and the iterator should raise some kind of error.

And that's exactly what the generator does!

Let's wrap our generator code in a try/catch now, instead of try/finally:

function* counterGenerator() {
    try {
        let i = 0;

        while(true) {
            yield i;
            i++;
        }
    // now it's a catch
    } catch(error) {
        console.log('caught error', error)
    }
}

We are prepared to log whatever error will be thrown in our code.

Let's run the next calls, but this time we will interrupt them with the throw method instead of return.

const iterator = counterGenerator();

console.log(iterator.next());
console.log(iterator.next());
console.log(iterator.next());
console.log(iterator.throw()); // now it's a throw
console.log(iterator.next());
console.log(iterator.next());

After running this code, you will see:

{ value: 0, done: false }
{ value: 1, done: false }
{ value: 2, done: false }
caught error undefined           <- log from catch block
{ value: undefined, done: true }
{ value: undefined, done: true }
{ value: undefined, done: true }

So we see that the error was indeed thrown, and that error was... undefined.

On top of that, just as was the case with the return method, after calling throw the generator stops running and it doesn't generate new values anymore.

We see that the error thrown in the generator was undefined. Could it possibly be that we can also pass an argument to throw, which will become our error? Let's try it!

const iterator = counterGenerator();

console.log(iterator.next());
console.log(iterator.next());
console.log(iterator.next());
console.log(iterator.throw("let's throw a string, why not, it's JS"));
console.log(iterator.next());
console.log(iterator.next());

We then see in the console:

{ value: 0, done: false }
{ value: 1, done: false }
{ value: 2, done: false }
caught error let's throw a string, why not, it's JS
{ value: undefined, done: true }
{ value: undefined, done: true }
{ value: undefined, done: true }

So we were right! Whatever you pass into the throw method as an argument will become the error object that actually gets thrown inside the generator.

One more thing. Similar to the return method, a value returned inside the catch block will become a value that gets returned by the throw method.

So this code:

function* counterGenerator() {
    try {
        let i = 0;

        while(true) {
            yield i;
            i++;
        }
    } catch {
        // now we return here
        return -666;
    }
}

Will result in this output:

{ value: 0, done: false }
{ value: 1, done: false }
{ value: 2, done: false }
{ value: -666, done: true }      // result of `throw` call
{ value: undefined, done: true }
{ value: undefined, done: true }

And although it's not visible in this example, I hope it's clear to you exactly in which place the error gets thrown inside our generator. It is exactly the place where the generator gets suspended while waiting for the next call.

To show that, let's take this example:

function* getNumbers() {
    yield 1;

    try {
        yield 2;
    } catch {
        console.log('We caught error!');
    }

    yield 3;
}

We can start this generator by calling next for the first time. That next call returns { value: 1, done: false } object and at this point the generator gets suspended on the yield 1; statement.

If now the second call to the iterator would be throw, then the error wouldn't get caught by try-catch. That's simply because the generator is still on the yield 1; line, which is not wrapped in a try-catch.

Indeed, running:

const iterator = getNumbers();

iterator.next();
iterator.throw('some error');

results in an uncaught string - some error - appearing in the console.

If however, you would run next as a second method, then this second call would return an object { value: 2, done: false } and the generator would be suspended on the yield 2; line.

If you called the throw method now, the error would be caught by try-catch and you would just see the log from the catch block.

So this code:

const iterator = getNumbers();

iterator.next();
iterator.next();
iterator.throw('some error');

simply prints:

We caught error!

Of course most of the time you won't rely on exactly which statements are supposed to throw. You will just use larger try/catch blocks. But it's still valuable to understand what exactly is happening here.

yield* - yield delegation

By now we got used to the yield keyword and its behavior doesn't seem strange to us anymore.

So let's get out of our comfort zone once more and learn about yield* now.

Yes, you've read that correctly. Apart from the yield keyword, you can use also yield* (`yield with a star character).

The * suggests that this construction has something to do with generators. But in fact, it's an operator that works on any iterable.

Its mechanism is called "yield delegation". yield* delegates execution to another iterable or generator.

We've started our generators adventure with this simple example:

js function* getNumbers() { yield 1; yield 2; yield 3; }

But using yield delegation we might have written it much simpler:

js function* getNumbers() { yield* [1, 2, 3]; }

Since an array is an iterable, we can call yield* on it, and at this point, the generator will start behaving as if it was a regular array iterator.

So running:

js for (let element of getNumbers()) { console.log(element) }

simply logs numbers:

js 1 2 3

If you think about it, it totally makes sense why another keyword - yield* - had to be introduced.

Note that this generator:

js function* getNumbers() { // look! no star here! yield [1, 2, 3]; }

simply emits one value - an array with 3 elements. Running the for ... of loop on this example results in the following log:

js [ 1, 2, 3 ]

Only after you use yield*, the control will be actually delegated to the array.

Of course nothing stops us from using yield* multiple times:

js function* getNumbers() { yield* [1, 2, 3]; yield* ['a', 'b', 'c']; }

which results in:

1 2 3 a b c

We can also combine yield and yield* in any way we want:

`js
function* getNumbers() {
yield* [1, 2, 3];

yield '---';

yield* ['a', 'b', 'c'];

}
`

which logs:

`
1
2

3

a
b
c
`

Since generators return iterators and since those iterators are iterables, this means we can use yield* also on results coming from generators, basically allowing us to nest generators, just like we nest functions.

Take two generators we already know:

`js
function* getNumbers() {
yield -3;
yield -2;
yield -1;
}

function* counterGenerator() {
let i = 0;

while(true) {
    yield i;
    i++;
}

}
`

We can easily run them one after another by creating another generator:

js function* getNumbersThenCount() { yield* getNumbers(); yield* counterGenerator(); }

Running:

js for (let element of getNumbersThenCount()) { if (element > 4) { break; } console.log(element); }

logs a sequence:

js -3 // <- getNumbers() -2 -1 0 // <- counterGenerator() 1 2 3 4

Of course in this example, since counterGenerator is infinite, getNumbersThenCount is infinite as well. If we wouldn't use break, it would run forever.

Generators as methods & some other syntax issues

I've left this section for the end because it's not really necessary to understand the how and why of generators.

But leaving it out completely would be dishonest and it might lead you to confusion when reading generators written by someone else.

Let's first start by noting, that you can easily turn object and class methods into generators, simply by prefixing the method name with a * symbol:

js const object = { *generatorMethod() { yield 1; } }

js class SomeClass { *generatorMethod() { yield 1; } }

It is also important to stress, that you can easily declare anonymous generators. This might be handy when you are writing inline generators as arguments to some other functions. Remember our runMaybe helper? With a little rewrite we could use it with an inline generator like this:

js runMaybe(function*() { // do something })

Going back to regular generator functions, it turns out, however, that the * character can be positioned in few different places.

Throughout this tutorial, we've written generators like this:

js function* generator() { yield 1; }

But interestingly, this works as well:

js function *generator() { yield 1; }

Note how the * character changed position.

Oh, and this works as well...

js function * generator() { yield 1; }

Uuuuuhm. And this also...

js function*generator() { yield 1; }

So this funny "looseness" of syntax means that you can see generators written in many ways. So don't get confused by it. In all of those cases, the behavior is exactly the same.

A similar thing applies to anonymous generator functions.

And in fact, yield* expressions are equally "loose".

So this works:

js function* getNumbers() { yield* [1, 2, 3]; }

But also this:

js function* getNumbers() { // * changed position here yield *[1, 2, 3]; }

And this:

js function* getNumbers() { yield * [1, 2, 3]; }

And - you guessed it! - this:

js function* getNumbers() { yield*[1, 2, 3]; }

In his phenomenal You Don't Know JS, the author Kyle Simpson recommends using the following syntax:

For declaring generators:

`js
function *someGenerator() {

}
`

For yield delegation:

js function *someGenerator() { yield *someIterable; }

However, as you've seen in these tutorials, I prefer:

`js
function* someGenerator() {

}
`

And for yield delegation:

js function* someGenerator() { yield* someIterable; }

That's because I see the function* string as a type declaration. So for me:

function = a regular function,
function* = a generator function.

Similarly, I like to think of a yield* as a single keyword (and hence written together), separate from yield. That's because it is basically a completely different mechanism, so in my mind, it makes sense to have a separate keyword for it.

But Kyle has some equally strong arguments, which you can read about here.

So ultimately just choose what you prefer and stick with it. In the end, it doesn't really matter. What's important is that you actually understand deeply the mechanisms under that syntax.

Conclusion

Uhh... That was a lot!

But I hope that at this point you feel that you understand generators very, very deeply.

And I am beyond excited, because finally in the future article we will be able to put all this knowledge into practice, by combining generators with React!

So if you don't want to miss those future articles, subscribe to me on Twitter.

Thanks for reading!

Generators in JavaScript, Part II - Simple use-case

mpodlasin — Fri, 08 Jan 2021 13:56:18 +0000

The behavior of generators that we've described in the previous article is not complicated, but it is certainly surprising and might be difficult to grasp at the very beginning.

So in this article, instead of introducing more concepts, we will pause a bit and use only what we've learned to this point while discovering a cool use-case for generators.

Let's say that we have a function like this:

function maybeAddNumbers() {
    const a = maybeGetNumberA();
    const b = maybeGetNumberB();

    return a + b;
}

Functions maybeGetNumberA and maybeGetNumberB return numbers, but sometimes they might also return null or undefined. That's what "maybe" in their names signalizes.

When that's the case, we shouldn't try to add those values (for example a number and null), but rather bail out immediately and just return, let's say, null again. After all, it's better to return null here, rather than some unpredictable value resulting from adding null/undefined with a number or with another null/undefined.

So we have to add a check that makes sure those numbers are actually defined:

function maybeAddNumbers() {
    const a = maybeGetNumberA();
    const b = maybeGetNumberB();

    if (a === null || a === undefined || b === null || b === undefined) {
        return null;
    }

    return a + b;
}

This works okay, but if a is either a null or an undefined, there is really no point in calling the maybeGetNumberB function at all. That's because we already know that we will return a null anyways.

So let's rewrite the function again:

function maybeAddNumbers() {
    const a = maybeGetNumberA();

    if (a === null || a === undefined) {
        return null;
    }

    const b = maybeGetNumberB();

    if (b === null || b === undefined) {
        return null;
    }

    return a + b;
}

Uuuh. From an easy to read 3-liner, this quickly grew to 10 lines of code (not counting the empty lines). This function is now filled with if cases, which you have to get through in order to understand what it does.

And this is just a toy example! You can imagine that in actual codebases, which contain much more complex logic, those checks would become even more complicated.

So what if we could use generators here and bring back the code to its simpler form?

Take a look at this:

function* maybeAddNumbers() {
    const a = yield maybeGetNumberA();
    const b = yield maybeGetNumberB();

    return a + b;
}

What if we could give that yield <something> expression the functionality of checking if <something> is an actual value and not null or undefined?

If it turned out that <something> is null or undefined, we would just bail early and return null, exactly like in the more verbose version of our code.

This way we could write code that looks almost as if it deals only with actual, defined values.
It's the generator itself that would check for you if that's really the case and it would act accordingly! Sounds magical, doesn't it?

And yet it's not only possible but also very easy to write!

Of course, generators themselves don't possess this functionality. They just return iterators and optionally allow you to inject some values back into the generator.

So we will have to write a wrapper - let's call it runMaybe - which will give the generator this capability.

So instead of calling the function directly:

const result = maybeAddNumbers();

We will be calling it as an argument to that wrapper:

const result = runMaybe(maybeAddNumbers());

This is a pattern that you will see incredibly often with generators.

Generators by themselves don't do much, but by writing custom wrappers like this one, you can grant generators custom behaviors! And that's precisely what we will do right now.

So runMaybe obviously is a function and it accepts one argument - an iterator produced by the generator:

function runMaybe(iterator) {

}

We will run this iterator in a while loop. In order to do that, we need to call the iterator for the first time and start checking its done property:

function runMaybe(iterator) {
    let result = iterator.next();

    while(!result.done) {

    }
}

Now inside a loop we have two options. If result.value is null or undefined we want to break the iteration process immediately and return null. Let's do that:

function runMaybe(iterator) {
    let result = iterator.next();

    while(!result.done) {
        if (result.value === null || result.value === undefined) {
            return null;
        }
    }
}

You can see that we are immediately stopping the iteration with the return and we are returning a null from our wrapper.

If however result.value is an actual, defined value, we want to "give it back" to the generator.

For example in yield maybeGetNumberA(), if it turns out that maybeGetNumberA() is actually a number, we just want to replace yield maybeGetNumberA() with the value of the number itself.

Even more specifically, if maybeGetNumberA() evaluated to, say, number 5, we would like to change const a = yield maybeGetNumberA(); into const a = 5;. As you can see, we don't want to change the yielded value in any way, but simply pass it back to the generator.

We remember that we can replace yield <something> with some value by passing that value as an argument to the iterators next method. So let's do that!

function runMaybe(iterator) {
    let result = iterator.next();

    while(!result.done) {
        if (result.value === null || result.value === undefined) {
            return null;
        }

        // we are passing result.value back
        // to the generator
        result = iterator.next(result.value)
    }
}

And as you can see, the new result gets now stored in the result variable again. We've specifically declared result with let so that it's possible.

We are almost there - if at any point our generator encounters a null/undefined when yielding a value, we just return a null from our runMaybe wrapper.

But we need to return something also if the iteration process finishes without encountering any null/undefined values. After all, if we receive two actual numbers in our generator, we want to return their sum from the wrapper!

Our maybeAddNumbers generator ends with a return statement.

We remember that return <something> in a generator causes its iterator to return an object { value: <something>, done: true } from a next call.

When this happens, while loop will stop running, because done property will be set to true. But that last returned value (in our specific case a + b value) still will be stored in the result.value property! So at the end we can simply return it:

function runMaybe(iterator) {
    let result = iterator.next();

    while(!result.done) {
        if (result.value === null || result.value === undefined) {
            return null;
        }

        result = iterator.next(result.value)
    }

    // just return the last value
    // after the iterator is done
    return result.value;
}

And... that's it!

Let's create dummy maybeGetNumberA and maybeGetNumberB functions. Let's make them return actual numbers first:

const maybeGetNumberA = () => 5;
const maybeGetNumberB = () => 10;

If we run our code now and log the results:

function* maybeAddNumbers() {
    const a = yield maybeGetNumberA();
    const b = yield maybeGetNumberB();

    return a + b;
}

const result = runMaybe(maybeAddNumbers());

console.log(result);

We will see - as expected - number 15 in the console.

Let's however change one of the added numbers to null:

const maybeGetNumberA = () => null;
const maybeGetNumberB = () => 10;

Now running the code logs null!

It was however important to us to make sure that maybeGetNumberB function doesn't called when the first function - maybeGetNumberA - returns null/undefined. So let's double-check if we really succeeded.

We can do it simply by adding a console.log to the second function:

const maybeGetNumberA = () => null;
const maybeGetNumberB = () => {
    console.log('B');
    return 10;
}

If we wrote our runMaybe helper correctly, the letter B should not appear in the console when running this example.

And indeed, if you run the code now, you will simply see null in the console, and nothing else. This means that our helper actually stops running the generator after it encounters a null/undefined value.

Our code also works as intended - by logging null - in any of those combinations:

const maybeGetNumberA = () => undefined;
const maybeGetNumberB = () => 10;

const maybeGetNumberA = () => 5;
const maybeGetNumberB = () => null;

const maybeGetNumberA = () => undefined;
const maybeGetNumberB = () => null;

etc.

The power of this example doesn't lay however in running this particular code.

It lays in the fact that we've created a general helper, which can handle any generator that potentially yields null/undefined values.

For example if we wrote a more complex function:

function* maybeAddFiveNumbers() {
    const a = yield maybeGetNumberA();
    const b = yield maybeGetNumberB();
    const c = yield maybeGetNumberC();
    const d = yield maybeGetNumberD();
    const e = yield maybeGetNumberE();

    return a + b + c + d + e;
}

We can run it in our runMaybe wrapper as well without any issue!

In fact, our wrapper doesn't even rely on the fact that in our examples those functions are returning numbers. Note that in runMaybe we don't mention the number type at all. So no matter what kind of values you are using in your generator - numbers, strings, objects, arrays, more complex data structures - it will still work with our helper!

This is exactly what developers find exciting about generators. They allow you to introduce custom functionality to the code that looks very regular (apart from those yield calls of course). You just need to create a wrapper that iterates over a generator in a particular way. This way, the wrapper basically "grants" the generator custom functionality!

And that functionality could be literally anything you want. Generators introduce potentially endless possibilities and the only limitation are our imaginations!

And in the following articles, we will keep exploring those possibilities, especially in combination with React. So if this sounds interesting to you, follow me on Twitter to not miss those future articles.

Thanks for reading!

Generators in JavaScript, Part I - Basics

mpodlasin — Mon, 21 Dec 2020 20:26:31 +0000

In this series I will teach you basically everything there is to know about generators in JavaScript - what they are, how to use them, and - as usual - all the intricacies involved. And as always, we will begin with some basics, to give you an overview of what the generators are.

This series doesn't assume any previous knowledge about generators. However, it does assume a very solid knowledge of iterables and iterators in JavaScript. If you don't know iterables/iterators, or don't really feel confident using them, make sure to check out my previous article, which covers them in-depth.

Know the prerequisites? Awesome! You are ready to dive into the world of generators. It is a strange, strange world, where many things are completely different from what you are used to in a regular JavaScript code.

But the actual mechanism is very simple, and even after reading this first article, you will feel confident in your capability to actually use generators by yourself.

So let's get started!

Motivation

"But why would I even want to learn about using generators?" - you might ask.

And that's a very fair question. Indeed, generators are still a fairly exotic feature, not used very commonly in most of the codebases.

But there are problems which can be solved with generators surprisingly elegantly. And indeed, in the next article, I will show just such an example. And after we master the generators, we will actually try to combine them with React to create code that is highly superior to "hooks-only" code. This, hopefully, will inspire you to seek your own use-cases for generators.

But don't think for a second that generators are still somehow "experimental". There are a lot of projects used in production codebases that lean on generators heavily.

I guess the most popular in the React world is redux-saga package, which is a middleware for Redux, allowing you to write side effects code that is extremely readable and extremely testable at the same time (which doesn't happen that often!).

I hope that this convinced you that it is absolutely worth learning generators. Are you now excited to study them? Let's do it then!

Introduction

If I was tasked with explaining generators in only one sentence, I would probably write - "it is a syntax sugar for producing iterators". Of course, this doesn't even come close to covering everything that generators are and can do. But it is not very far from the truth.

Let's take a basic, regular function, simply returning a number:

function getNumber() {
    return 5;
}

If we were to type it using TypeScript, we would say that it returns a number type:

function getNumber(): number {
    return 5;
}

In order to change a function into a generator function, we just need to add a * sign after the function keyword:

function* getNumber(): number {
    return 5;
}

But if you really were to do that in TypeScript, the compiler would start to complain. Indeed, a generator function doesn't simply return a value that is being returned in its body.

It instead returns an iterator!

If you would change the typings this way:

function* getNumber(): Iterator<number> {
    return 5;
}

TypeScript compiler would allow that without any issues.

But that's TypeScript. Let's test if function* really returns an iterator in pure JavaScript.

We can check it for example by trying to call the next method on the "thing" returned from the generator:

const probablyIterator = getNumber();

console.log(probablyIterator.next());

This not only works but it also logs { value: 5, done: true } to the console.

It's actually very reasonable behavior. In a sense, a function is an iterable that just returns one value and then is finished.

But would it be possible to return multiple values from a generator function?

The first thing that might've come to your mind is to use multiple returns:

function* getNumber() {
    return 1;
    return 2;
    return 3;
}

Now, this looks like blasphemy for someone used to regular functions. But I told you, we are in a completely different world now! Everything is possible.

However... this doesn't work. Let's run it:

const iterator = getNumber();

console.log(iterator.next());
console.log(iterator.next());
console.log(iterator.next());

You will see the following result in the console:

{ value: 1, done: true }
{ value: undefined, done: true }
{ value: undefined, done: true }

So we only got our first value, and after that, the iterator is stuck in it's "done" state. Interestingly the returned value is accessible only one time for us - further next calls just return undefined.

And this behavior is actually very reasonable. It obeys a basic rule true for all functions - return always stops executing the function body, even if there is some code after the return statement. This is true also for generator functions.

But there is a way to "return" multiple values from our generator. Exactly for that purpose the keyword yield was introduced. Let's try that:

function* getNumber() {
    yield 1;
    yield 2;
    yield 3;
}

Now let's run our code again:

const iterator = getNumber();

console.log(iterator.next());
console.log(iterator.next());
console.log(iterator.next());

A success! Now we get the following result:

{ value: 1, done: false }
{ value: 2, done: false }
{ value: 3, done: false }

So yielding values in a generator allows you to create an iterator that will return multiple values.

What happens if we call the next method more times after that? It behaves like any typical iterator by always returning a { value: undefined, done: true } object.

Note now that the last line in our generator is also a yield. Would it make any difference if we changed it to a return? Let's check

function* getNumber() {
    yield 1;
    yield 2;
    return 3; // note that we used a `return` here!
}

const iterator = getNumber();

console.log(iterator.next());
console.log(iterator.next());
console.log(iterator.next());

This code outputs:

{ value: 1, done: false }
{ value: 2, done: false }
{ value: 3, done: true }  // now done is true here!

Hmm. Interesting. So it does basically the same thing, but the done property gets set to true one step earlier.

You probably remember that the done property in the returned object basically decides whether the for ... of loop should continue running or not.

So let's check how our two versions of the getNumber generator behave with for ... of loops.

First let's run the version with 3 yields:

function* getNumber() {
    yield 1;
    yield 2;
    yield 3;
}

const iterator = getNumber();

for (let element of iterator) {
    console.log(element);
}

After running this code, we get:

1
2
3

No surprises really, that's how an iterator should behave.

Now let's do the same but for a generator with 2 yields and 1 return:

function* getNumber() {
    yield 1;
    yield 2;
    return 3; // only this line changed
}

const iterator = getNumber();

for (let element of iterator) {
    console.log(element);
}

What we get:

1
2

Huh. Very curious. But if you think about it, this is really just how iterators behave with the for ... of loop. The done property decides whether the next iteration step should be run or not.

Take a look at how in the iterables article we simulated the for ... of loop with a while:

let result = iterator.next();

while (!result.done) {
    const element = result.value;

    console.log(element);

    result = iterator.next();
}

In that code, if you would get a { value: 3, done: true } object from the iterator.next() call, the 3 would also never appear in the console.

That's because before console.log(element) gets called, we first have a !result.done condition. Since this condition is false for the { value: 3, done: true } object, while body would not be executed for the number 3.

And for ... of loops works in exactly the same way.

So the rule is fairly simple - do you want a value to appear in a for ... of loop? yield it!

Do you want to return it from a generator, but not include it in a for ... of iteration? return it!

Control flow in generators

At this point, we must clarify that in a generator function you can use all the typical control flow constructions.

For example, you might choose which number to yield based on an argument passed to the generator:

function* getNumber(beWeird) {
    yield 1;

    if(beWeird) {
        yield -100;
    } else {
        yield 2;
    }

    yield 3;
}

Calling getNumber(false) will create an iterator that returns numbers: 1, 2, 3.

Calling getNumber(true) will create an iterator that returns numbers: 1, -100, 3.

Not only that, you can even use loops in generators! And that's actually where their real power comes in.

In our iterables article, we've created an infinite iterator, which was generating numbers 0, 1, 2, 3, ... - up to infinity. It wasn't too difficult, but it also wasn't the most readable code ever.

Now we can do that with a generator in just few simple lines:

function* counterGenerator() {
    let index = 0;

    while(true) {
        yield index;
        index++;
    }
}

We simply start with an index set to 0. We then run an infinite while(true) loop. In that loop, we yield current index and then we simply bump that index by one. This way, in the following step, index will be yielded with a new value.

Beautifully simple, right?

This is the exact example that literally blew my mind when I was first learning generators. I hope that it blows your mind as well, at least a little bit.

Just look how far we've come - we were used to functions that can only ever return a single value. And now we are writing a function that "returns" basically... forever!

Sending values to a generator

On those first, simple examples we've seen that we can use generators to create typical iterators.

But it turns out that an iterator returned from a generator is a bit strange. It allows you to... pass some values back to the generator as well!

Let's enhance our previous generator example:

function* getNumber() {
    const first = yield 1;
    const second = yield 2;
    const third = yield 3;
}

As you can see, we are still simply yielding numbers from the generator, but we also assign to variables whatever those yield <number> expressions evaluate to.

Obviously, at the moment those variables are not used in any way. For the tutorial purposes, we will be simply logging them, but you could of course do with them whatever you want.

We will also put an additional log at the very beginning of the function.

function* getNumber() {
    console.log('start');

    const first = yield 1;
    console.log(first);

    const second = yield 2;
    console.log(second);

    const third = yield 3;
    console.log(third);
}

In the rest of this section, we will be running that exact generator multiple times. I would therefore advise you to copy this code somewhere, or just open this article again in a second browser tab.

It will be much easier for you to understand what is happening if you look at this generator as often as possible while we run the examples!

So let's run this new generator just as we did the previous one.

for (let element of getNumber()) {
    console.log(element);
}

What we get is:

start
1
undefined
2
undefined
3
undefined

I hope it's clear which logs come from the generator itself and which come from the for ... of loop. Just to make sure, here are the answers:

start          <- generator
1              <- loop
undefined      <- generator
2              <- loop
undefined      <- generator
3              <- loop
undefined      <- generator

So apparently yield <number> statements just evaluate to undefined. But we can change that!

To do that, we will have to abandon the for ... of loop and consume the iterator by hand.

Let's just call the next method of the iterator 4 times, to get our 3 numbers and the last object with done set to true. We will log every result coming from the next call.

const iterator = getNumber();

console.log(iterator.next());
console.log(iterator.next());
console.log(iterator.next());
console.log(iterator.next());

After running that (with the generator unchanged), we get:

start
{ value: 1, done: false }
undefined
{ value: 2, done: false }
undefined
{ value: 3, done: false }
undefined
{ value: undefined, done: true }

So not much changed here - undefined values are still here. We just swapped numbers from a for ... of loop to logging whole objects coming from next calls.

Generators utilize in a smart way the flexibility of an iterator interface. After all, an iterator has to have a next method, returning an object of shape { done, value }. But nobody said that this method can't accept some arguments! A next method that accepts some argument still obeys the interface, as long as it returns an object of expected shape!

So let's see what happens when we pass some strings to those next calls:

const iterator = getNumber();

console.log(iterator.next('a'));
console.log(iterator.next('b'));
console.log(iterator.next('c'));
console.log(iterator.next('d'));

After you run this, you'll finally see something else than undefined in the console:

start
{ value: 1, done: false }
b                                <- no more undefined
{ value: 2, done: false }
c                                <- no more undefined
{ value: 3, done: false }
d                                <- no more undefined
{ value: undefined, done: true }

Perhaps this result is surprising to you. After all, the first letter we've passed to the next was a. And yet we only see b, c and d here.

But it's actually fairly straightforward to see what is happening here if we do it step by step.

The rule is that a call to next causes the generator function to run until it encounters a yield <some value> call. When this call is encountered, the <some value> part gets returned from the next call (as a value in the { value, done } object). From this moment on, the generator simply waits for another next call. The value passed to that another next call will become the value to which the whole yield <something> expression gets evaluated.

Let's see it step by step on our example generator.

When you call next the first time, it simply begins the execution of the generator function. In our case, this means that console.log('start') will get executed.

Indeed, running:

const iterator = getNumber();

iterator.next('a');

results in the following:

start

In the generator function, after console.log('start'), we encounter the yield 1 expression. As we've explained, number 1 here will become the value returned from that first next call that we have just made.

Indeed, you can wrap the next call in console.log to make sure that's true:

const iterator = getNumber();

console.log(iterator.next('a'));

This now logs:

start
{ value: 1, done: false }

The 1 there is precisely what we yielded in the generator.

And this point, the generator is suspended. Even the statement where we encountered yield - const first = yield 1; - did not get executed fully. After all, the generator doesn't know yet what the value of the yield 1 part should be.

We will provide that value with our next next call:

const iterator = getNumber();

console.log(iterator.next('a'));
iterator.next('b');

This will print:

start
{ value: 1, done: false }
b

So we see that the generator resumed execution and basically replaced yield 1 with a value that we passed to the next call - b string.

To make sure you really understand what is happening, you can try to pass some other values at this point:

const iterator = getNumber();

console.log(iterator.next('a'));
iterator.next('this is some other string, which we created for tutorial purposes');

This will (hopefully obviously to you now) print:

start
{ value: 1, done: false }
this is some other string, which we created for tutorial purposes

You are the one who decides here what yield 1 will evaluate to.

So at this point we see, that our first yield expression uses the value provided in the second next call. This is crucial to understand in generators.

Basically, when encountering a yield <some value>, the generator is saying: "in current next call I will return you a <some value>, but in the next next call please provide me as an argument what should I replace yield <some value> with".

And this actually means that the argument passed to the first next call will never be used by the generator. There is simply no point to provide it, so we will just remove it from our example:

const iterator = getNumber();

console.log(iterator.next()); // no need to pass anything on the first `next` call
iterator.next('b');

After we've called next a second time, the generator continued to execute the code, until it encountered another yield statement - yield 2. Therefore number 2 gets returned from this next call as a value.

So this:

const iterator = getNumber();

console.log(iterator.next());
console.log(iterator.next('b'));

prints this:

start
{ value: 1, done: false }
b
{ value: 2, done: false }

What happens now? The generator does not know to what it should evaluate yield 2 in the const second = yield 2; statement. So it just waits there, suspended, until you pass it another value in the next call:

const iterator = getNumber();

console.log(iterator.next());
console.log(iterator.next('b'));
iterator.next('c');

This now logs:

start
{ value: 1, done: false }
b
{ value: 2, done: false }
c

So after that third next call, code in the generator starts being executed again, until we encounter yield 3. So 3 will be the value returned from that call:

const iterator = getNumber();

console.log(iterator.next());
console.log(iterator.next('b'));
console.log(iterator.next('c')); // we've added console.log here

This prints:

start
{ value: 1, done: false }
b
{ value: 2, done: false }
c
{ value: 3, done: false }

Now the generator is suspended at the const third = yield 3; statement. We know what to do to make it running again - another next call with a value!

const iterator = getNumber();

console.log(iterator.next());
console.log(iterator.next('b'));
console.log(iterator.next('c'));
iterator.next('d'); // we've added another next call

This prints:

start
{ value: 1, done: false }
b
{ value: 2, done: false }
c
{ value: 3, done: false }
d

And - because our generator doesn't how more yield statements in it - it doesn't have more values to return. It also runs till completion.

That's why the last { done, value } object from the next call, has no value in it and also notifies us that the iterator has finished.

So this code:

const iterator = getNumber();

console.log(iterator.next());
console.log(iterator.next('b'));
console.log(iterator.next('c'));
console.log(iterator.next('d')); // we've added console.log here

Prints this:

start
{ value: 1, done: false }
b
{ value: 2, done: false }
c
{ value: 3, done: false }
d
{ value: undefined, done: true }

And that's it! If this still seems confusing, you need to run this example by yourself, perhaps even a few times.

Help yourself by adding those successive next and console.log calls step by step just like I did. Try also to always control in which line of the generator you currently are. Remember! You have to look at the generator code at each step to really understand what is happening here!

Don't just read the article - run this example by yourself, as many times as necessary, to make sure that you actually understand what is happening!

Conclusion

In this article, we've learned the basics of generators. How to create them, how to use the yield keyword, and how to consume the generators.

I hope that those first exercises and examples got you excited to learn more. We still have a lot to cover with regards to generators, so make sure to follow me on Twitter to not miss those future articles.

Thanks for reading!

Iterables & Iterators - An In-Depth JavaScript Tutorial

mpodlasin — Thu, 10 Dec 2020 07:51:39 +0000

This article is an in-depth introduction to iterables and iterators in JavaScript. My main motivation for writing it was to prepare ourselves for learning generators next. In fact, later I plan to do some experimental stuff, like combining generators with React hooks.

As a matter of fact, I planned to start with the generators article, but it quickly became obvious to me, that they are tough to explain without having a solid understanding of iterables & iterators first.

That's why in this article we will focus on iterables & iterators only. We will assume no previous knowledge about them, but at the same time, we will go fairly in-depth. So if you know something about iterables & iterators, but you still don't feel fully comfortable using them, this article should fix that.

Introduction

As you've noticed, we are talking about iterables and iterators. They are related, but distinct concepts, so while reading the article make sure to keep tabs on which one we are talking about at any given moment.

Let's begin with iterables. What are they? An iterable is basically something that can be iterated over, like so:

for (let element of iterable) {
    // do something with an element
}

Note that we are talking only about for ... of loops here, which were introduced in ES6. for ... in loops are an older construct and we will not use it at all in this article.

You might now think, "okay, this iterable variable is simply an array!". And indeed, arrays are iterables. But even currently in native JavaScript, there are other data structures that we could use in a for ... of loop. In other words, there are more iterables in native JavaScript than just arrays.

For example, we can iterate over ES6 Maps:

const ourMap = new Map();

ourMap.set(1, 'a');
ourMap.set(2, 'b');
ourMap.set(3, 'c');

for (let element of ourMap) {
    console.log(element);
}

This code will print out:

[1, 'a']
[2, 'b']
[3, 'c']

So variable element in the code above stores in each iteration step an array of two elements. The first element is a key, and the second element is a value.

The fact that we could use for ... of loop to iterate over Map, proves to us that Maps are iterables. Once again - only iterables can be used in for ... of loops. So if something works with that loop - it is an iterable.

Funnily enough, Map constructor itself optionally accepts an iterable of key-value pairs. So this is an alternative way to construct the same Map as before:

const ourMap = new Map([
    [1, 'a'],
    [2, 'b'],
    [3, 'c'],
]);

And since - as we've just noted - Map itself is an iterable, we can create copies of Maps extremely easily:

const copyOfOurMap = new Map(ourMap);

We have now two distinct maps, although they are storing the same values under the same keys.

So we've seen two examples of iterables so far - an array and an ES6 Map.

But we still didn't explain how do they possess this magic power of being able to be iterated over.

The answer is simple - they have iterators associated with them. Read that carefully. Itera*tors, not iterables*.

In what way an iterator is associated with it's iterable? An iterable object simply has to have a function under its Symbol.iterator property. This function, when called, should return an iterator for that object.

For example, we can retrieve an array's iterator like so:

const ourArray = [1, 2, 3];

const iterator = ourArray[Symbol.iterator]();

console.log(iterator);

This code will print Object [Array Iterator] {} to the console.

So we know that our array has an associated iterator and that this iterator is some kind of object.

What is an iterator then?

It's fairly simple. An iterator is just an object that has a next method. This method, when called, should return:

next value in a sequence of values,
information whether the iterator has finished producing values or not.

Let's test it, by calling the next method of the iterator of our array:

const result = iterator.next();

console.log(result);

We will see an object { value: 1, done: false } printed in the console.

The first element of the array we created was 1, so it appeared as the value here. We also got information that the iterator is not done yet, meaning we still can call the next function and expect to see some values.

Let's do it! In fact, let's call next two more times:

console.log(iterator.next());
console.log(iterator.next());

Unsurprisingly, we get { value: 2, done: false } and { value: 3, done: false } printed, one after another.

But our array had only 3 elements. So what happens, if we try to call next yet again?

console.log(iterator.next());

This time we see { value: undefined, done: true } printed. This is information for us that the iterator has finished. There is no point calling next again. In fact, if we do so, we will receive the same { value: undefined, done: true } object over and over again. done: true is a sign for us to stop the iteration.

Now we can understand what for ... of loop does under the hood.

First [Symbol.iterator]() method is called to get an iterator,
next method is being called on that iterator in a loop until we get done: true,
after each call to next, value property is used in the loop's body.

Let's write all that in code:

const iterator = ourArray[Symbol.iterator]();

let result = iterator.next();

while (!result.done) {
    const element = result.value;

    // do some something with element

    result = iterator.next();
}

All this code is directly equivalent to:

for (let element of ourArray) {
    // do something with element
}

You can make sure that's the case by, for example, placing console.log(element) in place of // do something with element comment.

Creating our own iterator

So we know what the iterables and iterators are. The question then becomes - is it possible to write our own instances of them?

Absolutely!

There is nothing magical about iterators. They are just objects with a next method, which behaves in a specified way.

We've said which native JS values are iterables. We haven't mention objects there. Indeed, they are not iterables natively. Take an object like this:

const ourObject = {
    1: 'a',
    2: 'b',
    3: 'c'
};

When we try to iterate over that object with for (let element of ourObject), we will get an error, stating that object is not iterable.

So let's practice writing custom iterators by making such an object an iterable!

In order to do that, we would have to patch Object prototype with our custom [Symbol.iterator]() method. Since patching prototypes is a bad practice, let's just create a custom class, extending Object:

class IterableObject extends Object {
    constructor(object) {
        super();
        Object.assign(this, object);
    }
}

The constructor of our class simply takes a regular object and copies its properties onto an iterable one (although it's not really iterable yet!).

So we will be creating an interable object like this:

const iterableObject = new IterableObject({
    1: 'a',
    2: 'b',
    3: 'c'
})

In order to make the IterableObject class actually iterable, it needs to have a [Symbol.iterator]() method. Let's add it then.

class IterableObject extends Object {
    constructor(object) {
        super();
        Object.assign(this, object);
    }

    [Symbol.iterator]() {

    }
}

Now we can start writing an actual iterator!

We already know that it has to be an object, which has a next method on it. So let's start with that.

class IterableObject extends Object {
    // same as before

    [Symbol.iterator]() {
        return {
            next() {}
        }
    }
}

After every call to next, we have to return an object of shape { value, done }. Let's do just that, with some dummy values.

class IterableObject extends Object {
    // same as before

    [Symbol.iterator]() {
        return {
            next() {
                return {
                    value: undefined,
                    done: false
                }
            }
        }
    }
}

Given an iterable object:

const iterableObject = new IterableObject({
    1: 'a',
    2: 'b',
    3: 'c'
})

we would like to print it's key-value pairs, similarily to what iterating over ES6 Map did:

['1', 'a']
['2', 'b']
['3', 'c']

So in our custom iterator, under the value property we want to place an array [key, valueForThatKey].

Note that this - compared to the previous steps of the example - is our own design decision. If we wanted to write an iterator that returned only keys or only property values - we might do that as well, and it would be perfectly fine. We simply ourselves decided to return key-value pairs.

So we will need arrays of shape [key, valueForThatKey]. The easiest way to obtain them is simply to use the Object.entries method.

We can use it just before creating an iterator object in the [Symbol.iterator]() method:

class IterableObject extends Object {
    // same as before

    [Symbol.iterator]() {
        // we made an addition here
        const entries = Object.entries(this);

        return {
            next() {
                return {
                    value: undefined,
                    done: false
                }
            }
        }
    }
}

The iterator returned in that method will have an access to the entries variable thanks to a JavaScript closure.

But we also need some kind of state variable. It will tell us which key-value pair should be returned in a current next call. So let's add that as well.

class IterableObject extends Object {
    // same as before

    [Symbol.iterator]() {
        const entries = Object.entries(this);
        // we made an addition here
        let index = 0;

        return {
            next() {
                return {
                    value: undefined,
                    done: false
                }
            }
        }
    }
}

Note how we declared index variable with a let because we know that we plan to update its value after each next call.

We are now ready to return an actual value in the next method:

class IterableObject extends Object {
    // same as before

    [Symbol.iterator]() {
        const entries = Object.entries(this);
        let index = 0;

        return {
            next() {
                return {
                    // we made a change here
                    value: entries[index],
                    done: false
                }
            }
        }
    }
}

This was easy. We just used both entries and index variables to access a proper key-value pair from the entries array.

Now we have to deal with that done property because currently, it will be always set to false.

We could keep another variable - alongside entries and index - and update it after every next call. But there is an even easier way. We can simply check if index already went out of bounds of the entries array:

class IterableObject extends Object {
    // same as before

    [Symbol.iterator]() {
        const entries = Object.entries(this);
        let index = 0;

        return {
            next() {
                return {
                    value: entries[index],
                    // we made a change here
                    done: index >= entries.length
                }
            }
        }
    }
}

Indeed, our iterator is done when the index variable is equal to the length of entries or is bigger.

For example, if entries has length 3, it has values under indexes 0, 1, and 2. So when the index variable is 3 (equal to the length), or bigger, it means there are no more values to get. That's when we are done.

This code almost works. There is only one more thing we need to add.

The index variable starts with a value 0, but... we are never updating it!

It's actually kind of tricky because we should update it after we return { value, done }. But when we return it, the next method stops running immediately, even if there is some code after the return statement.

We can however create the { value, done } object, store it in a variable, update the index and just then return the object:

class IterableObject extends Object {
    // same as before

    [Symbol.iterator]() {
        const entries = Object.entries(this);
        let index = 0;

        return {
            next() {
                const result = {
                    value: entries[index],
                    done: index >= entries.length
                };

                index++;

                return result;
            }
        }
    }
}

After all these changes, this is how our IterableObject class looks so far:

class IterableObject extends Object {
    constructor(object) {
        super();
        Object.assign(this, object);
    }

    [Symbol.iterator]() {
        const entries = Object.entries(this);
        let index = 0;

        return {
            next() {
                const result = {
                    value: entries[index],
                    done: index >= entries.length
                };

                index++;

                return result;
            }
        }
    }
}

This code works perfectly fine, but it became a bit convoluted. There is actually a smarter (but less obvious) way to deal with having to update index after creating the result object. We can simply initialize index with -1!

Then, even though the index update happens before returning the object from next, everything will work just fine, because the first update will bump -1 to 0.

So let's do just that:

class IterableObject extends Object {
    // same as before

    [Symbol.iterator]() {
        const entries = Object.entries(this);
        let index = -1;

        return {
            next() {
                index++;

                return {
                    value: entries[index],
                    done: index >= entries.length
                }
            }
        }
    }
}

As you can see, now we don't have to juggle the order of creating the result object and updating index. That's because we are starting with -1. During the first next call, index will be updated to 0 and then we will return the result.

During the second call, index will be updated to 1 and we will return another result, etc...

So everything will work just as we wanted, and the code looks now much simpler than the previous version.

How we can test if it really works properly? We could manually run [Symbol.iterator]() method to create an iterator instance, then directly test the results of next calls, etc.

But there is a much simpler way! We've said that every iterable can be plugged into for ... of loop! So let's do just that and log the values returned by our custom iterable:

const iterableObject = new IterableObject({
    1: 'a',
    2: 'b',
    3: 'c'
});

for (let element of iterableObject) {
    console.log(element);
}

It works! You will see the following result printed in the console:

[ '1', 'a' ]
[ '2', 'b' ]
[ '3', 'c' ]

That's exactly what we wanted!

Isn't this cool? We've started with objects not being able to be used in for ... of loops because natively they don't have built-in iterators. But we created a custom IterableObject, which does have an associated iterator, which we have written by hand.

I hope that by now you can see and appreciate the power of iterables and iterators. It's a mechanism that allows your own data structures to cooperate with JS features like for ... of loops, in a way indistinguishable from the native data structures! That's very powerful and in certain situations, it can vastly simplify the code, especially if you plan to do iterations on your data structures often.

On top of that, we can customize what exactly such iteration will return. We've settled on returning key-value pairs from our iterator. But what if we cared only about the values themselves? No problem! We can just rewrite our iterator:

class IterableObject extends Object {
    // same as before

    [Symbol.iterator]() {
        // changed `entries` to `values`
        const values = Object.values(this);
        let index = -1;

        return {
            next() {
                index++;

                return {
                    // changed `entries` to `values`
                    value: values[index],
                    // changed `entries` to `values`
                    done: index >= values.length
                }
            }
        }
    }
}

And that's it!

If we run for ... of loop after this change, we will see the following output in the console:

a
b
c

So we really returned only the objects values, just as we wanted.

This proves how flexible your custom iterators can be. You can really make them return whatever you wish.

Iterators as... iterables

You will see people very often confusing iterators and iterables.

That's a mistake and I was trying to carefully differentiate between the two in this article, but I think I know one of the main reasons why people confuse them so often.

It turns out that iterators... are sometimes iterables as well!

What does it mean? We said that an iterable is an object that has an iterator associated with it.

It turns out that every native JavaScript iterator also has a [Symbol.iterator]() method, returning yet another iterator! This - according to our previous definition - makes that first iterator an iterable.

We can check that it is true, by taking an iterator returned from an array and calling [Symbol.iterator]() on it once more:

const ourArray = [1, 2, 3];

const iterator = ourArray[Symbol.iterator]();

const secondIterator = iterator[Symbol.iterator]();

console.log(secondIterator);

After running this code, you will see Object [Array Iterator] {}.

So not only our iterator has another iterator associated with it, but we also see that it is again an array iterator.

In fact, if we compare those two iterators with ===, it turns out that this is simply exactly the same iterator:

const iterator = ourArray[Symbol.iterator]();

const secondIterator = iterator[Symbol.iterator]();

// logs `true`
console.log(iterator === secondIterator);

This behavior of an iterator being its own iterator might seem strange at the beginning.

But it's actually fairly useful.

You cannot plug a bare iterator into the for ... of loop. for ... of accepts only an iterable - that is an object with a [Symbol.iterator]() method.

However, an iterator being its own iterator (and hence an iterable) mitigates that problem. Since native JavaScript iterators do have [Symbol.iterator]() methods on them, you can pass them to for ... of loops directly without thinking twice.

So because of that feature, both:

const ourArray = [1, 2, 3];

for (let element of ourArray) {
    console.log(element);
}

and:

const ourArray = [1, 2, 3];
const iterator = ourArray[Symbol.iterator]();

for (let element of iterator) {
    console.log(element);
}

work without any problems and do exactly the same thing.

But why would you even want to use an iterator directly in a for ... of loop like that? The answer is simple - it turns out that sometimes it is simply unavoidable.

First of all, you might want to create an iterator without any iterable to which it belongs. We will see such example later, and it's actually not that rare to create such "bare" iterators. Sometimes an iterable itself just isn't needed.

And it would be very akward if having a bare iterator meant you couldn't just consume it via for ... of. It's of course always possible to do it manually with a next method and, for example, a while loop, but we've seen that it requires quite a lot of typing and boilerplate.

It's simple - if you want to avoid that boilerplate and use your iterator in a for ... of loop, you have to make it an iterable as well.

On the other hand, you will also quite often receive iterators from methods other than [Symbol.iterator](). For example, ES6 Map has entries, values and keys methods. All of them return iterators.

If native JavaScript iterators weren't iterables as well, you couldn't just use those methods directly in for ... of loops like that:

for (let element of map.entries()) {
    console.log(element);
}

for (let element of map.values()) {
    console.log(element);
}

for (let element of map.keys()) {
    console.log(element);
}

The code above works, because iterators returned by the methods are also iterables.

If they weren't, we would have to, for example, awkardly wrap a result from map.entries() call in some kind of a dummy iterable. Luckily we don't have to, and we can just use those methods directly, without worrying too much about it.

For those reasons, it is a good practice to make your custom iterators iterables as well. Especially if they will be returned from some methods other than [Symbol.iterator]().

And it's actually very simple to make an iterator an iterable. Let's do that with our IterableObject iterator.

class IterableObject extends Object {
    // same as before

    [Symbol.iterator]() {
        // same as before

        return {
            next() {
                // same as before
            },

            [Symbol.iterator]() {
                return this;
            }
        }
    }
}

As you can see, we just created a [Symbol.iterator]() method under the next method.

We've made this iterator it's own iterator by simply returning this - so it just returned itself. We've seen that that's exactly how the array iterator behaved.

That's enough to make sure that our iterator works with for ... of loops, even when used in them directly.

State of an iterator

It should be fairly clear by now that each iterator has a state associated with it.

For example in our IterableObject iterator, we kept the state - an index variable - as a closure.

After each iteration step, that index was updated.

So what happens after the iteration process ends? It's simple - the iterator becomes useless and we can (and should!) discard it.

We can doublecheck that this happens even with iterators of native JavaScript objects.

We will take an iterator of an array and try to run it in a for ... of loop twice.

const ourArray = [1, 2, 3];

const iterator = ourArray[Symbol.iterator]();

for (let element of iterator) {
    console.log(element);
}

for (let element of iterator) {
    console.log(element);
}

You might expect to see numbers 1, 2, 3 appearing in the console twice. But this is not what happens. The result is still just:

1
2
3

But why?

We can discover that, by trying to call next manually, after the loop finishes:

const ourArray = [1, 2, 3];

const iterator = ourArray[Symbol.iterator]();

for (let element of iterator) {
    console.log(element);
}

console.log(iterator.next());

The last log prints { value: undefined, done: true } to the console.

Aaah. So after the loop finishes, the iterator is now in its "done" state. From now on it always return a { value: undefined, done: true } object.

Is there a way to "reset" the state of this iterator, in order to use it in a for ... of loop second time?

In some cases perhaps, but there is really no point. This is exactly why [Symbol.iterator] is a method and not just a property. We can simply call that method again to obtain another iterator:

const ourArray = [1, 2, 3];

const iterator = ourArray[Symbol.iterator]();

for (let element of iterator) {
    console.log(element);
}

const secondIterator = ourArray[Symbol.iterator]();

for (let element of secondIterator) {
    console.log(element);
}

Now it works as we would expect.

Right now you should be able to understand why looping over an array directly multiple times works:

const ourArray = [1, 2, 3];

for (let element of ourArray) {
    console.log(element);
}

for (let element of ourArray) {
    console.log(element);
}

That's because each of those for ... of loops uses a different iterator! After an iterator is done and a loop ends, that iterator is never used again.

Iterators vs arrays

Because we use iterators (although indirectly) in for ... of loops, they might look to you deceivingly similar to arrays.

But there are two important distinctions to be made between the iterators and the arrays.

Both of them have to do with the concept of eager and lazy values.

When you create an array, at any given moment it has a specific length and its values are already initialized.

I mean, sure, you can create an array wihout any values inside, but that's not what we mean here.

We mean that it is impossible to create an array that initializes its value only after you attempt to access that value by writing array[someIndex]. I mean, perhaps it is possible with some Proxy or other JS trickery, but by default JavaScript arrays don't behave in that way. You just create an array with values initialized beforehand and that's it.

And when saying that an array has a length, we in fact mean that the array is of a finite length. There are no infinite arrays in JavaScript.

Those two qualities point to the eagerness of arrays.

On the other hand, iterators are lazy.

To show that, we will create two custom iterators - the first one will be an infinite iterator, in contrast to finite arrays, and the second will initialize its values only when they are actually needed/requested by whoever is using the iterator.

Let's start with the infinite iterator. This might sound scary, but we will create something very simple - an iterator that starts at 0 and at each step returns the next integer in a sequence. Forever.


const counterIterator = {
    integer: -1,

    next() {
        this.integer++;
        return { value: this.integer, done: false };
    },

    [Symbol.iterator]() {
        return this;
    }
}

That's it! We start with the integer property equal to -1. At each next call we bump it by one and return it as a value in the result object.

Note that we used here the same trick as before - starting at -1 in order to return 0 as the first result.

Look also at the done property. It will be always false. This iterator never ends!

Third thing, which you've probably noticed yourself - we have made this iterator an iterable, by giving it a simple [Symbol.iterator]() implementation.

And one last note. This is the case that we've been mentioning earlier - we've created an iterator, but there is no iterable in sight! This is an iterator which doesn't need an iterable "parent" for anything.

We can now try out this iterator in a for ... of loop. We just need to remember to break out of the loop at some point. Otherwise, the code would run forever!

for (let element of counterIterator) {
    if (element > 5) {
        break;
    }

    console.log(element);
}

After running this code we will see the following in the console:

So we really created an infinite iterator, which can return you as many integers as you wish. And it was actually very easy do achieve!

Now, let's make an iterator, which doesn't create its values until they are requested.

Well... we already did it!

Have you noticed that at any given moment, our counterIterator stores only one number on the integer property? It stores only the last number that it has returned in a next call.

This is indeed the laziness we were talking about. This iterator can potentially return any number (non-negative integer, to be specific). But it only creates a number when it is actually needed - when someone is calling the next method.

This might not seem like a big benefit. After all, numbers are created fast and they don't occupy a lot of memory.

But if you are dealing with very big, memory-heavy objects in your code, sometimes swapping arrays for iterators can be extremely valuable, making your program faster more memory efficient.

The heavier the object (or the longer it takes to create it), the bigger the benefit.

Some other ways to consume iterables

So far we've been playing only with a for ... of loop, or we've been consuming our iterators manually, using the next method.

But those are not your only options!

We've already seen that Map constructor accepts an iterable as an argument.

You can also easily transform an iterable into an actual array by using Array.from method. Be careful though! As we've said, laziness is sometimes a big benefit of an iterator. Converting it to an array gets rid of all the laziness. All the values returned by an iterator get initialized immediately and then they get put into an array.

In particular, this means that trying to convert our infinite counterIterator into an array would result in a catastrophe. Array.from would just run forever and never return any result! So before converting an iterable/iterator to an array, make sure it's a safe operation.

Interestingly, iterables also play nicely with a spread operator (...). Just keep in mind that this works similarly to an Array.from, where all the values of an iterator get initialized at once.

For example, we can use the spread operator to create our own version of Array.from.

We just apply the operator on an iterable and then put the values into an array:


const arrayFromIterator = [...iterable];

We can also get all the values from an iterable and apply them to a function:


someFunction(...iterable);

Conclusion

I hope that at this point you understand why the title of this article was "Iterables and Iterators".

We've learned what are they, how do they differ, how to use them, and how to create your own instances of them.

This makes us more than ready to deal with generators. In fact, if you understand iterators well, then jumping into generators should be no problem at all!

If you've enjoyed this article and want more, remember to subscribe to me on Twitter.

Thanks for reading!

Iterables & Iterators in JS, Part III - "Advanced" concepts (YouTube tutorial)

mpodlasin — Wed, 09 Dec 2020 21:38:34 +0000

This is the last video of my 3 part Iterables & Iterators series!

This time we cover some "advanced" concepts, which are actually not that advanced - I just needed some name for this episode. ;)

And in the next video we will finally move on to long-awaited generators, so stay tuned!

Enjoy!