Forem: Bhaskar Karambelkar

Bash Shell Tricks

Bhaskar Karambelkar — Sun, 09 Jul 2017 00:22:53 +0000

This was originally published on my blog back in 2007, I am glad to see that this has aged well. I still find personally using many of the tricks mentioned here even today.

Quick Command editing

^orig^repl^ replace 'orig' in previous command with 'repl' and execute the new command.

File Name Generation

cp filename{,.bak} Copy 'filename' as 'filename.bak.
mkdir -p dir{1,2,3}/subdir{1,2} Create dir1/subdir1, dir1/subdir2, dir2/subdir1, dir2/subdir2, dir3/subdir1, and dir3/subdir2 directories

Completion

No I am not talking about completing command/file names. I am talking about completing arguments to various commands, completing filenames based on application. This is one of the touted features of
'zsh', but unknown to many is the fact that it is also available in bash.
All you need to do is install bash-completion.

With bash completion installed you can use the TAB key to complete arguments to common commands like rpm, apt, find, grep etc. Also bash-completion will complete host-names
for ssh, scp, by looking up hosts inside your $HOME/.ssh/authorized_keys file. For rpm based distros, bash-completion will even lookup package names already installed.

The feature I find most handy is file-name completion based on the context of the command. e.g. if you type tar -zxvf and then press the TAB key twice, you will get a list of only files ending in .tar.gz or .tgz rather than a list of all files in the directory.

Socket Communication

I bet a lot of you haven't heard of this, but bash can indeed perform basic socket communication via /dev/tcp, /dev/udp. These are pseudo devices that bash uses to communicate with network sockets. In fact if you did ls -l /dev/tcp /dev/udp you will get a 'No such file or directory' error message.

So how to use them, we below I present 2 examples.

One to quickly check headers from a HTTP Server. Here is a simple function you can put in your $HOME/.bashrc that will check for headers of HTTP server.

headers() {
  server=$1; port=${2:-80}
  exec 5<> /dev/tcp/$server/$port
  echo -e "HEAD / HTTP/1.0\nHost: ${server}\n\n" >&5;
  cat 0<&5;
  exec 5>&-
}

Simply invoke it by

headers <servername or ip> <port>

The port number defaults to 80 if not provided.

Second example is a quick way to check if a port is open or not. You can always use netcat or telnet, but I find this easier.

testPort() {
  server=$1; port=$2; proto=${3:-tcp}
  exec 5<>/dev/$proto/$server/$port
  (( $? == 0 )) && exec 5<&-
}

Again invoke it by

testPort <servername or ip> <port> <protocol>

The protocol can be either tcp or udp and defaults to tcp.

Arithmetic Evaluations

Bash can perform arithmetic evaluations. They are much easier to using expr tool. Below are some examples.Note the absense of $ prefix for variable names.

((count++)) #increment value of variable 'count' by one.
((total+=current)) # set total = total+current.
((current>max?max=current:max=max)) # ternary expression.

Data Scientists and Software Engineering

Bhaskar Karambelkar — Fri, 14 Apr 2017 02:29:01 +0000

Originally published on my personal blog

Coding != Software Engineering

As has been mentioned ad nauseam: Any one can code. What has not been said enough is that there is a lot more to coding than merely assembling a set of instructions in a programming language of your choice to make the machine do what you want. Not as catchy as 'Any one can code!', is it?

As a professional software developer turned data scientist, I feel compelled to share some software engineering wisdom with my fellow data scientist who may have followed a non coding heavy path. If you are an academic researcher turned data scientist, or perhaps a data analyst used to point-and-click GUI tools, or excel (the horror!), then learning and being able to code in a programming language can be a liberating, exhilarating, but also a very scary experience. But fear no more. This post and the follow up posts in this series are just for professionals like you. This series will introduce you and encourage you to explore software engineering in more detail, in order to become proficient in writing good code no matter the programming language.

So What is Software Engineering?

Instead of referencing a formal definition which you can easily look up using Google, let me tell you what I think the practice of software engineering aims to accomplish. Writing more or less working code is the easy part. What software engineering aims to accomplish is making the code portable, concise, relatively bug free, secure, performant within given constraints, and reusable, with limited man-power and budget. And believe me, this is not as easy as you might think it is, nor is this process a natural extension of the practice of coding. By that I mean that you can't learn software engineering by coding more and more, let alone master it. Proper software engineering is an art+science on to it self, of which coding skills are an important but nonetheless only a small piece.

The reason I chose to explain software engineering this way, is because, the term software engineering is in itself somewhat controversial and debated. So instead of drowning you in the controversy about the term, I present my understanding of the intent behind software engineering. But be forewarned that this is one person's opinion so take it for what it's worth.

So Why Must Data Scientists Care?

For many a reasons. As I mentioned previously more and more data analysis is now done in code rather than point-and-click GUI tools. This places the added burden of learning how to code on an data analyst / researcher who may not have had exposure to coding before. Even if you had taken a programming language class before, it was mostly to teach you the syntax of the programming language rather than teach you about software engineering.

The implications of the above are they you may write code that works but it may have all sorts of issues. It may not be portable on account of using non-portable APIs/Libraries. It may not be optimized/concise and as a result non performant at scale. It may have a large surface area for bugs and security vulnerabilities. It may not be maintainable in the long run, and hence prevent you from reproducing your results in the future. Given all this, I would go as far as to argue that if you care about the veracity and reproducibility of your research/analysis then you absolutely must care about software engineering.

Cliff Notes for Software Engineering

I must warn you this page is only meant to get you started on software engineering, and not tell you all that there is about it. Even then, following most of my advice below will make you a better coder (ahem software engineer).

Have a basic understanding of how computer systems work. The hardware, the software (including OS/Kernel), the network, the Internet. This will go a long way, I promise.
Try and pick up at least 2 or 3 programming languages. Broaden your repertoire.
Learn about various programming paradigms like imperative, object-oriented, functional etc.
Many a modern languages don't strictly fall under a single paradigm. Form a habit of recognizing which features adhere to which paradigms.
Embrace each programing language's idiosyncrasies rather that fight them. If in doubt always remember that people lot smarter than you put hours into developing the language/API/library you are using.
Have a good understanding of the standard library of a programming language. It will prevent you from inefficiently duplicating functionality that was at your disposal from the get go.
In addition to the standard library research a bit on some leading 3rd party libraries/APIs available. Someone always has the problem solved before you. Your skill lies in finding it rather than duplicating it.
Start picking up on how to distinguish efficient vs inefficient code. Efficiency can defined in terms of performance, conciseness, resource consumption etc.
Teach yourself the principles of application security and secure coding. Not being a full time software developer doesn't alleviate you from writing secure code.
Think beyond your immediate use case. Think of use cases in future or use cases by users other than yourself. "It suffices my needs" is a narrow mindset.
Write less code and more comments. Think of that someone who has to read your code six months or an year from now. Even if that someone is you, I can tell you from experience that reading properly commented code can do wonders to lower your stress levels.
Be critical of your coding abilities rather than being confident about them. Let that imposter syndrome be your motivation to improve.
Automate your testing. Be it unit tests or integration tests, take out the human as much as possible from the equation.
Learn about software delivery pipelines. Continuous integration (CI), automated deployments, devops are not just buzzwords. They play a critical part in your overall product development.
Familiarize yourself with distributed computing, cloud environments, virtualization and container technologies.

The last four ones are special and deserve to be separated out from the rest.
Always follow them no matter how big/small your program/script is, and how much you are pressed for time. Excuse the shouting because they are that much important.

NEVER CODE WITHOUT A VERSION CONTROL SYSTEM, PREFERABLY GIT
DON'T HARD CODE STRINGS, NUMBERS, FILE/DIRECTORY NAMES EVER!
ALWAYS METICULOUSLY DOCUMENT YOUR "CLEVER" HACKS. ALWAYS!

Finally...

DON'T BLINDLY COPY CODE FROM STACK OVERFLOW.

Anything More?

Yes! A lot more. Over the course of this series I will expand on each of my bullet points in a separate blog post that will deep dive in to the point. In the mean time feel free to look up software engineering, the controversy around it. If you have any comments to share find me on Twitter (link at the bottom).

Setup OSX for R

Bhaskar Karambelkar — Fri, 20 Jan 2017 01:44:23 +0000

Originally Published on my Personal Blog.

This is a rather details documentation of my OSX setup for R usage using homebrew. You can get away with a lot less than what I have installed below, but this is fairly comprehensive and I have also given some steps to verify installation of various packages.

Github API Token

Get a github account and setup an API token as described here. This helps you avoid hitting Github API limits when using homebrew.

echo 'export HOMEBREW_GITHUB_API_TOKEN="your_new_token"' >> $HOME/.bash_profile
. $HOME/.bash_profile

Install Homebrew and tap some taps

/usr/bin/ruby -e \
    "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
brew analytics off # full on paranoid mode
brew tap caskroom/cask # GUI apps
brew tap caskroom/fonts # fonts
brew tap homebrew/science
brew tap homebrew/completions
brew tap homebrew/services
brew tap homebrew/versions
brew tap osgeo/osgeo4mac # If you need Geospatial stuff

Bash

Install Bash and verify

brew install bash bash-completion2
/usr/local/bin/bash --version

GNU bash, version 4.3.46(1)-release (x86_64-apple-darwin15.5.0)

Setup bash along with some utility functions. Primarily we setup brewPkg bash function which will not only install homebrew packages but also log the output for debugging purposes if so desired.


sudo sh -c 'echo "/usr/local/bin/bash" >> /etc/shells'

mkdir -p $HOME/Library/Logs/Homebrew/$USER

echo '
if [ -f $(brew --prefix)/share/bash-completion/bash_completion ]; then
  . $(brew --prefix)/share/bash-completion/bash_completion
fi

brewPkg() {
  pkg=$1
  shift
  (
    brew install ${pkg} $*  2>&1 |
        tee $HOME/Library/Logs/Homebrew/$USER/${pkg}-$(date +"%F_%H%M").txt
  )
}

brewSrcPkg() {
  pkg=$1
  shift
  (
    brew install --build-from-source  ${pkg} $*  2>&1 |
        tee $HOME/Library/Logs/Homebrew/$USER/${pkg}-$(date +"%F_%H%M").txt
  )
}

brewSrcPkgWgcc() {
  pkg=$1
  shift
  (
    export CC=gcc-6
    export CXX=g++-6
    export HOMEBREW_CC=gcc-6
    export HOMEBREW_CXX=g++-6
    brew install --build-from-source  ${pkg} $*  2>&1 |
        tee $HOME/Library/Logs/Homebrew/$USER/${pkg}-$(date +"%F_%H%M").txt
  )
}

' >>  ~/.bash_profile

Load the new bash shell, so we can use all the auto-complete goodies. Or simply close and restart the Terminal App.

/usr/local/bin/bash -l

GCC Compiler and Autotools

Install gcc, without multilib so that OpenMP works.

brewPkg gcc --without-multilib
/usr/local/bin/gcc-6 --version

gcc-6 (Homebrew gcc 6.1.0 --without-multilib) 6.1.0

/usr/local/bin/gfortran --version

GNU Fortran (Homebrew gcc 6.1.0 --without-multilib) 6.1.0

Setup aliases for homebrew's gcc

cd /usr/local/bin
ln -s gcov-6 gcov
ln -s gcc-6 gcc
ln -s g++-6 g++
ln -s cpp-6 cpp
ln -s c++-6 c++
cd -

Install ccache to speed up compilation

brewPkg ccache
/usr/local/bin/ccache --version

ccache version 3.2.7

Install autotools, pkg-config, and cmake

brewPkg cmake pkg-config autoconf automake

Let's make sure OpenMP is working as expected.

cat > omp-test.c <<"END"
#include <omp.h>
#include <stdio.h>
int main() {
    #pragma omp parallel
    printf("Hello from thread %d, nthreads %d\n", omp_get_thread_num(), omp_get_num_threads());
}
END
gcc-6 -fopenmp -o omp-test omp-test.c 
./omp-test

You should see something similar but not exactly the same.

Hello from thread 1, nthreads 8
Hello from thread 6, nthreads 8
Hello from thread 4, nthreads 8
Hello from thread 2, nthreads 8
Hello from thread 5, nthreads 8
Hello from thread 0, nthreads 8
Hello from thread 3, nthreads 8
Hello from thread 7, nthreads 8

Misc libs

brewPkg freetype fontconfig  pixman gettext

SSL/SSH Libs

Setup and verify openssl and libressl.

brewPkg  openssl
/usr/local/opt/openssl/bin/openssl version

OpenSSL 1.0.2h  3 May 2016

brewPkg  libressl
/usr/local/opt/libressl/bin/openssl version

LibreSSL 2.3.6

brewPkg libssh2

wget and curl

brewPkg wget
/usr/local/bin/wget --version

GNU Wget 1.18 built on darwin15.6.0.

brewPkg curl
/usr/local/opt/curl/bin/curl-config --version

libcurl 7.50.0

GPG

brewPkg gpg2 --with-readline
/usr/local/bin/gpg2 --version

gpg (GnuPG) 2.0.30
libgcrypt 1.7.2

brewPkg gpgme
/usr/local/bin/gpgme-config --version

1.6.0

Java

brew cask install java
java -version

java version "1.8.0_102"
Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)

X-server

We need an X-Server. This takes a lot of time so be patient.

brew cask install xquartz

Python

Install python2

brewPkg python
pip install --upgrade pip setuptools
/usr/local/bin/python -V

Python 2.7.12

Install Python3

brewPkg python3
pip3 install --upgrade pip setuptools wheel
/usr/local/bin/python3 -V

Python 3.5.2

Git

brewPkg git --with-blk-sha1 --with-gettext \
    --with-pcre --with-persistent-https
/usr/local/bin/git --version

git version 2.9.2

Boost libs w/ dependencies

Install icu4c library for Unicode and globalization

brewPkg icu4c
/usr/local/opt/icu4c/bin/icu-config --version

57.1

Install libxml2, libiconv, and libxslt.

brewPkg libxml2 libiconv libxslt
brew link libxml2 --force
/usr/local/bin/xml2-config --version

2.9.4

Install boost. Ignore the warning at the end.

brewPkg boost --with-icu4c --with-mpi --without-single

Boost is a beast of a library, so we need some quick programs to test whether it has installed successfully.
Shamelessly copied/adapted from the Intertubes. Ignore the warning messages spewed by the compiler.

cat > first.cpp <<END
#include<iostream>
#include<boost/any.hpp>
int main()
{
    boost::any a(5);
    a = 1.61803;
    std::cout << boost::any_cast<double>(a) << std::endl;
}
END
clang++ -o first first.cpp
./first

1.61803

cat > second.cpp <<END
#include<iostream>
#include <boost/filesystem.hpp>
int main() 
{
    boost::filesystem::path full_path( boost::filesystem::current_path() );
    if ( boost::filesystem::exists( "second.cpp" ) )
    {
        std::cout << "Found second.cpp file in " << full_path << std::endl;
    } else {
        std::cerr << "Argh!, Something not working" << std::endl;
        return 1;
    }
}
END
clang++ -o second second.cpp \
    -lboost_filesystem-mt -lboost_system-mt
./second

Found second.cpp file in "/Users/brewmaster"

Latex Support

Take a Coffee break and then some, because this is a huge one.

brew cask install mactex

R

Install libsvg, librsvg, and cairo.

brewPkg cairo 
brewPkg libsvg
brewPkg librsvg
brewPkg pandoc

Openblas for speedier linear algebra in R.

brewPkg openblas --with-openmp

Test openblas. Shamelessly copied from Intertubes.

cat > test-openblas.c <<"END"
#include <cblas.h>
#include <stdio.h>

void main()
{
  int i=0;
  double A[6] = {1.0,2.0,1.0,-3.0,4.0,-1.0};         
  double B[6] = {1.0,2.0,1.0,-3.0,4.0,-1.0};  
  double C[9] = {.5,.5,.5,.5,.5,.5,.5,.5,.5}; 
  cblas_dgemm(CblasColMajor, CblasNoTrans, CblasTrans,
    3,3,2,1,A, 3, B, 3,2,C,3);

  for(i=0; i<9; i++)
    printf("%lf ", C[i]);
  printf("\n");
}
END

clang -L/usr/local/opt/openblas/lib \
    -I/usr/local/opt/openblas/include \
    -lopenblas -lpthread \
    -o test-openblas test-openblas.c

./test-openblas

11.000000 -9.000000 5.000000 -9.000000 21.000000 -1.000000 5.000000 -1.000000 3.000000

Eigen and Armadillo for Rcpp and v8 for R+v8.

brewPkg eigen
brewPkg armadillo --with-hdf5
brewPkg v8-315
brew link v8-315 --force

Test Armadillo

cd /usr/local/opt/armadillo/examples/
clang++   -O2   -o example1  example1.cpp  -larmadillo -framework Accelerate
./example1

You should see something like below and lot more.

Armadillo version: 7.200.2 (Plutocratic Climate Change Denialist)
A.n_rows: 2
A.n_cols: 3
...

Test v8

echo 'quit()' | v8

V8 version 3.15.11.18 [sample shell]

Finally install R itself, and also setup R to use Apple's clang compiler. We can also setup R to use gcc compiler to take advantage of openmp support, but I've noticed that not all R packages compile correctly when using GCC.

brewPkg r --with-openblas --with-pango
# for rJava to work.
R CMD javareconf \
    JAVA_CPPFLAGS=-I/System/Library/Frameworks/JavaVM.framework/Headers

# Setup $HOME/.r/Makevars file to properly link against homebrew packages.
mkdir $HOME/.r
cat > $HOME/.r/Makevars << END
CC=ccache clang
CXX=ccache clang++
SHLIB_CXXLD=ccache clang++
FC=gfortran-6
F77=gfortran-6
MAKE=make -j8
PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:/opt/X11/lib/pkgconfig:/usr/local/opt/icu4c/lib/pkgconfig
PKG_LIBS += -L/usr/local/opt/icu4c/lib -L/usr/local/lib
END


# Also setup a R_LIBS_USER directory to install R packages locally.
mkdir -p $HOME/Library/R/3.x/library
cat > $HOME/.Renviron <<END
export R_LIBS_USER=$HOME/Library/R/3.x/library
END

# add same stuff to .bash_profile
cat $HOME/.Renviron >> $HOME/.bash_profile

GIS Stuff

Mostly PostGIS + Geo libs.

brewPkg postgresql
brewPkg geos
brewPkg proj
brewPkg gdal2 \
    --with-armadillo --with-complete --with-libkml \
    --with-opencl --with-postgresql --with-unsupported
brewPkg postgis --with-gui

Other Programming Languages

Some other programming languages I use occasionally.

Node.js

brewPkg node
brewPkg phantomjs casperjs

Scala

brew install scala

golang

brew install golang
cat >> $HOME/.bash_profile <<END
export GOPATH=$HOME/golang
export GOROOT=/usr/local/opt/go/libexec
export PATH=$PATH:$GOPATH/bin
export PATH=$PATH:$GOROOT/bin
END

Other Interesting Stuff

brewPkg imagemagick --with-fontconfig --with-ghostscript \
    --with-librsvg --with-pango --with-webp
brewPkg vim --with-python3 --with-client-server \
    --with-lua --with-luajit --with-override-system-vi

brewPkg macvim --with-python3 --with-client-server \
    --with-lua --with-luajit --with-override-system-vim

brewPkg jq # json processing

brew cask install font-hack font-fira-code #extra fonts

brewPkg mlpack # Fast Machine Learning
brewPkg protobuf --devel # Google's Protocol Buffer Library
brewPkg gsl # GNU Scientific Library
brewPkg libyaml # YAML Support

GUI Apps

Apps related to Securing your Mac.

brew cask install blockblock knockknock \
    dhs taskexplorer kextviewr
brew cask install suspicious-package

Quicklook Plugins for Developers.

brew cask install qlcolorcode qlstephen qlmarkdown \
    quicklook-json qlprettypatch quicklook-csv betterzipql \
    qlimagesize webpquicklook

These are some GUI apps I use. Pick and chose as you like. They are not necessarily related to R or even development.

brew cask install google-chrome chrome-devtools firefox iterm2 seil \
 slate keepassx free-mind itsycal flux caffeine alfred beardedspice \
 macdown mysqlworkbench osxfuse smcfancontrol torbrowser vagrant\
 vagrant-manager vlc cog yed slack owncloud

GDAL 2 On Mac with Homebrew

Bhaskar Karambelkar — Thu, 20 Oct 2016 21:34:35 +0000

Originally published on my personal blog.

This is a small post about upgrading to gdal 2.x on a Mac using homebrew.

Assumptions

You have homebrew installed and setup.
You already have gdal 1.x installed via homebrew's default 'gdal' formula.

Install gdal 2.x

Unlink gdal 1.x using
brew unlink gdal
Tap into osgeo4mac
brew tap osgeo/osgeo4mac && brew tap --repair
Install gdal2.

brew install gdal2 --with-armadillo \
  --with-complete --with-libkml --with-unsupported

Link gdal2
brew link --force gdal2
Verify


$> gdal-config --version
2.1.1

$> gdal-config --libs
-L/usr/local/Cellar/gdal2/2.1.1/lib -lgdal

$> gdal-config --cflags
-I/usr/local/Cellar/gdal2/2.1.1/include

Post Install

If you are using R for geo-spatial stuff, be sure to reinstall the rgdal library. Also you can now install sf in R using devtools::install_github('edzer/sfr').

How to use Twitter’s Search REST API most effectively.

Bhaskar Karambelkar — Mon, 05 Jan 2015 12:49:00 +0000