Forem: Petar G. Petrov

Java and Native Code in OpenJDK 13

Petar G. Petrov — Tue, 11 Jun 2019 18:46:22 +0000

Intro
Hands-on
- The libCurl App
- The libSSH2 App
Final Thoughts
References

Intro

Last week I watched a pretty cool GOTO Conference talk [1] with Mikael Vidstedt, director of Software Engineering at Oracle, where he presents many of the upcoming and currently being worked on Java features. One of those, Project Panama, stood out as particularly interesting to me.

// Detect dark theme var iframe = document.getElementById('tweet-1136248991254028289-119'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1136248991254028289&theme=dark" }

Disclaimer: Project Panama is still in its early stages of development, so the contents of this article may not reflect its current state.

Project Panama, available in early-access JDK 13 builds, is meant as a bridge between Java and native code. How is this useful? There are certain use cases where a piece of functionality is available as a library written in C or C++ and the requirement is to integrate it into a Java application. The current solution is to use the Java Native Interface (JNI) [2] which could require a vast amount of work and is also quite error-prone. You need to have a good grasp of the native library's insides and then write all the required implementations to bridge the Java interface with the native code. From my experience, calling native library functions that may change at some later point in time and also managing heap memory allocations could be a real challenge with JNI. To quote Mikael as he says it on the video:

How many people here have written JNI or, you know, done JNI? We're so sorry for you!

This is where Panama comes into play. It provides new tools and API to simplify the bridging between Java and native code. It basically boils down to the following steps:

Generate Java bindings (interfaces) from existing C header file(s) using the jextract tool.
Invoke C functions via the jextracted Java interface using the java.foreign API.

This allows you to concentrate on writing the core logic of your application, rather than fiddling with glue code and integration details.

Hands-on

Project Panama's documentation pages [3] already provide a solid number of examples to start with. I'll just take a quick peek at how to bridge and run a libCurl Java app and then I'd like to present a more detailed example - a simple SSH client that I wrote based on libSSH2.

I'm running these examples on a macOS, but with a few tweaks you should also be able to run them on a Linux installation.

The libCurl App

How to download a web page using the native Curl library's implementation? Well, first we need to get and extract a Panama OpenJDK build archive.

Let's open up a shell and set the JAVA_HOME environment variable to where the OpenJDK build archive is extracted.

export JAVA_HOME=/opt/jdk-13.jdk

Now we need to generate the Java interfaces, the glue code that will bind Java code to the native library. This will produce a curl.jar file:

$JAVA_HOME/bin/jextract -t org.unix -L /usr/lib -lcurl \
  --record-library-path /usr/include/curl/curl.h \
  -o curl.jar

When we inspect the JAR file, we can see all the Curl API calls, as well as dependency bindings. The Curl API is available through the new java.foreign Java API.

Now for a quick example. Here's a Java piece of code that fetches a web page and displays its contents on screen.

import java.lang.invoke.*;
import java.foreign.*;
import java.foreign.memory.*;
import org.unix.curl.*;
import org.unix.curl_h;
import static org.unix.curl_h.*;
import static org.unix.easy_h.*;

public class Main {
   public static void main(String[] args) {
       try (Scope s = curl_h.scope().fork()) { 
           curl_global_init(CURL_GLOBAL_DEFAULT);
           Pointer<Void> curl = curl_easy_init();
           if(!curl.isNull()) {
               Pointer<Byte> url = s.allocateCString(args[0]);
               curl_easy_setopt(curl, CURLOPT_URL, url);
               int res = curl_easy_perform(curl);
               if (res != CURLE_OK) {
                 System.err.println("Error fetching from: " + args[0] + " ERR: " + Pointer.toString(curl_easy_strerror(res)));
                 curl_easy_cleanup(curl);
               }
           }
           curl_global_cleanup();
       }
    }
}

A couple of things to point out here. Notice how we cannot directly pass a Java String to the curl_easy_setopt() call. This call accepts a memory address pointer as the url parameter, so we first need to do a dynamic memory allocation operation using the Scope and pass a Pointer interface instance instead. As you may find in the Panama tech docs [5], the Pointer interface helps a lot when it comes to complex C-alike pointer operations like pointer arithmetic, casts, memory dereference, etc. The Scope manages the runtime life-cycle of dynamic allocated memory.

Alright, now armed with this knowledge, can you extend this code to write the contents of a Curl fetched web page to a file stream?

The libSSH2 App

Here's a more complete example application that utilizes libSSH2 [4] to implement a simple SSH client.

petarov / java-panama-ssh2

Java SSH client using libssh2 through project Panama

java-panama-ssh2

A simple Java SSH2 client using the native libssh2 library and JDK 13 Project Panama

NOTE: This is an experimental project. Stability is not guaranteed.

Install

The client has been tested on macOS and Linux. It should also be possible to run it on Windows, however, no work has been done in that direction. PRs are welcome!

Install libssh2.

macOS - brew install libssh2
CentOS - yum install libssh2.x86_64 libssh2-devel.x86_64

Get the latest Panama JDK build.

Open a console shell and configure the JAVA_HOME var to point to JDK 13.

Generate the required Java interfaces from the native libssh2 headers:

./gen_bindings.sh

Run

To compile and run use:

./run.sh [-p|-k] hostname port username [path to ssh keys]

-p uses a password login. You'll be prompted to enter your password.
-k uses a public key login. You'll need to specify the path to your keys, e.g., ~/.ssh.

License

…

View on GitHub

If you'd like to have a go and adjust it to run on Windows, I'll greatly appreciate a PR.

Final Thoughts

A few points from my side that I learned or have been thinking about while working with Panama.

The Scope is pretty powerful. You can allocate callback pointers, structs, arrays. The concept of layouts and layout types deserves more time for me to fully explore and grasp.
It comes as no surprise that writing Java code using a native library is more extensive and requires extra care, especially when it comes to not forgetting to invoke cleanup API calls that the library requires.
I/O in most native libraries requires a file descriptor, which isn't easy to get in Java [6]. This, however, is not directly related to the java.foreign API.
Some libraries define C++ style function prototypes without argument as opposed to C-style Void argument types. The Foreign Docs [3] have an example about this case when using the TensorFlow C API.
I haven't explored if it would be possible to use Panama with Go or Rust created native libraries. That would be pretty cool.

Thanks for reading!🍻

References

GOTO 2019 • Project Panama part
Java Native Interface
Panama Foreign Docs
libSSH2 - a client-side C library implementing the SSH2 protocol
Panama Binder Docs v3
Most efficient way to pass Java socket file descriptor to C binary file

Help Me Delete Your Tweets!

Petar G. Petrov — Sat, 16 Mar 2019 20:39:30 +0000

Ok, I do apologize for the clickbaity title and to make up for it here's what this is all about in brief.

tl;dr, I'm looking for hackers to help me improve a Python script.

A couple of years ago I wanted to have all my tweets and likes deleted, but still keep them somewhere for private purposes, mainly for history reference and personal archive. Because I'm not really that active on Twitter, deleting my account would probably have made more sense. But hey! I still wanted to put my uninformed opinion out there from time to time.

Long story short, I hacked this small Python command line tool that consumes the Twitter API. It finds all tweets and/or likes after a specified date-time point, puts them in an ePub book and then (optionally) removes them from Twitter. Because it's just a command line thing, it's also simple to have it run in a cron job as well. So far so good, it seems to work alright, however, there're some limitations. I couldn't figure out how to save tweet images with Tweepy and the ePub E-book generation could use more options like front and back cover images, table of contents, layout options, etc.

So I guess this is a call for contributions. If you're looking for some open source project to contribute to and this looks fun to you, you're more than welcome to join in. I'm actually not a Python developer, but a Python hacker, so this is more like a hack-it-for-fun-but-still-make-it-useful thing here

Here's the GitHub repo and thanks for reading!

petarov / shut-up-bird

🐦 Put your tweets/likes in an EPUB and delete them like a boss

Shut Up Bird

⚠️ NO LONGER MAINTAINED 😔

Archives your tweets in an EPUB book and then optionally deletes them.

Things you could do:

Get rid of your old tweets or likes but still have them nicely organized somewhere.
An annual archive of your twitter activty.
Setup a cron job to regularly clean up your status.
Archive someone else's tweets for your own viewing pleasure.
Read your own tweets in your favourite e-book reader app and cry.

Installation

Requires Python 2.7 or 3.x.

Packages

To install on ArchLinux from AUR run:

yaourt -S shut-up-bird

Manual

Run make or pip install -r requirements.txt.

Setup

Create a new Twitter application. The name shouldn't matter.

Open the app's Permissions page and make sure Read and Write is selected, otherwise shut-up-bird will not be able to delete anything.

Run without any parameters to initialize:

$ python shut-up-bird.py
Please provide your Twitter

…

View on GitHub

Home Storage with Raspberry Pi and Samba

Petar G. Petrov — Fri, 02 Nov 2018 20:53:25 +0000

Goal
Raspberry Pi
- Remove X11
- Storage User
Samba Server
- Installation
- Configuration
Clinets
- Linux Client
  - View Shares
  - fstab Mount
- Android Client
- iOS Client
References

Goal

The idea here is pretty simple. Setup a central storage on a Raspberry Pi [1] device, so we could exchange files at home without using the cloud or Skype, or any other messaging service. The whole thing wasn't so hard to setup and I do recommend it to everyone that can spare some time and effort into doing this.

Things I wanted to have:

Central storage to place and exchange files.
Accessible from desktop and mobile devices.
Available only on the internal network.
User/Password protection.
A reason to finally do something with that RPi I got.

Initially I wanted to install a dedicated cloud or storage solution on the RPi device, however, I found out it was much more simple and intuitive to configure a Samba server instead.

Raspberry Pi

Get a Raspbian Lite [2] installation and use dd to copy the image to your SD Card. Check the official installation guidelines [3].

On Linux the following should suffice:

dd bs=4M if=2018-10-09-raspbian-stretch.img of=/dev/sd? conv=fsync

Careful with the sd? part! Make sure that's your SD card device.

Remove X11 (Optional)

EDIT: If you download Raspbian Lite, you don't need to do anything here.

This might be a good idea, if you'd like to use Raspbian only as a server without a desktop environment. You'd also probably free up to 500MB SD card space.

sudo apt-get remove --purge x11-common
sudo apt-get autoremove

Storage User

Add a new user with the name storeuser. All shared files and folders will be stored in the home directory of this user.

sudo adduser storeuser

It might also be a good idea to disable shell login for this guy. In the end we only want to have this user as a basis for our Samba share configuration.

sudo usermod -s /usr/sbin/nologin storeuser

A new home folder /home/storeuser will get created. Next, create a share folder under which the shared files and folders will be saved.

sudo cd /home/storeuser && mkdir share

Adjust the read/write permissions to the share folder:

sudo chown -R storeuser /home/storeuser/share
sudo chgrp -R storeuser /home/storeuser/share
sudo chmod -R 1770 /home/storeuser/share

About the 1770 permissions. We set the sticky bit [4] to 1 to allow only the owner of this folder storeuser to edit or delete files.

Create a sample file, e.g., a README, just to have at least one file in the share folder for tests.

sudo touch /home/storeuser/share/README

Samba Server

There could be a number of ways to setup a Samba share. For the purpose of my needs I chose a single user password protected share mechanism, but you can extend this to use groups and more complex authentication schemes.

Installation

sudo apt-get install samba samba-common-bin

Configuration

Edit the smb.conf file:

nano /etc/samba/smb.conf

That's right! I used nano. I still can't exit vim on the other terminal. Deal with it. 😊

To be on the safe side add your network to the allowed hosts:

[global]

hosts allow = 192.168.1. 127.

I'm using a single subnet 192.168.1.0/24 network at home. Adjust this depending on your network configuration.

Add a workgroup name and enable Windows Internet Name Serving:

[global]

hosts allow = 192.168.1. 127.
workgroup = homeworld.net
wins support = yes

Now configure the share. Add the following at the end of the smb.conf file:

[mystorage]
   comment= my home storage
   path=/home/storeuser/share
   browseable=Yes
   writeable=Yes
   only guest=no
   create mask=1770
   directory mask=1770
   public=no

path specifies a directory to which the user of the service is to be given access. browseable controls whether this share is seen in the list of available shares in a net view and in the browse list.

Add a new storeuser password to the local Samba smbpasswd file. This user must already exist on the system. We'll use the storeuser that we created earlier.

smbpasswd -a storeuser

This is exactly the user and password that clients will use to access the Samba share.

Clients

Linux Client

I'm using Arch Linux quite often at home, so the description below mostly concerns that case. However, this should not differ too much on other Linux distributions.

View Shares

We could first have a look at the available shares in the network by using smbtree. Just like that, for fun and glory.

smbtree -U storeuser

When prompted, give the password of storeuser that you set earlier using smbpasswd. The result should look similar to this:

MYDOMAIN.TLD
    \\RASPBIAN              my raspbian server
        \\RASPBIAN\storeuser        Home Directories
        \\RASPBIAN\IPC$             IPC Service
        \\RASPBIAN\mystorage        my home storage

This prints a tree with all the known domains, the servers in those domains and the shares available on those servers.

fstab Mount

Create a target mount folder.

sudo mkdir /mnt/storage

Setup an auto mount by adding the share to /etc/fstab.

//RASPBIAN/mystorage /mnt/storage cifs username=storeuser,password=<password>,comment=noauto,x-systemd.automount,_netdev,uid=<your linux user>,gid=<your linux user group> 0 0

If you're not using systemd (for which I don't blame you 😉), remove the noauto,x-systemd.automount part.

Ok, what the heck is this _netdev trickery?

Basically, this means that we'd like to defer mounting the share until the network interface is up [5]. No point in trying to mount if connection's not been setup yet.

Let's mount the share explicitly by running:

sudo mount -a

You should see the sample README file we created earlier in /mnt/storage.

Android Client

There're many Android tools to choose from. Pretty much everything that supports SMB shares should work. In the past I was using ES File Explorer [6], but I found the ads too obstructive, so I switched to File Manager [7]

Here's a quick setup overview of ES File Explorer.

Select Network / LAN from the app menu and then add a new connection with New.
Type in the server name or IP address of your RPi device.
Type in the storeuser username and password.

You should now be able to access your Samba share with Read/Write permissions.

Setting up File Manager is pretty easy as well. Just use Network place from the menu to select your share and type in the credentials.

iOS Client

I'm using File Explorer Free [7] to access Samba shares from iOS devices. The free version is limited to just one share, which is good enough for me.

Configuration is quite easy. Add a new Linux/Unix share and configure the sever name or IP address of your RPi device.

Type in the storeuser username and password.

You should now be able to access your Samba share with Read/Write permissions.

Do you know any other good tools? Free or paid. Drop a comment below.

References

raspberrypi.org
raspberrypi.org/downloads
raspberrypi.org/documentation/installation/installing-images/README.md
en.wikipedia.org/wiki/Sticky_bit
codingberg.com/linux/systemd_when_to_use_netdev_mount_option
ES File Explorer for Android
File Manager for Android
File Explorer Free for iOS
Samba suite config file ref - samba.org/samba/docs/man/manpages-3/smb.conf.5.html
Arch Linux bud? Samba Wiki - wiki.archlinux.org/index.php/samba

Store Encrypted Files in Google Drive

Petar G. Petrov — Wed, 17 Oct 2018 17:25:36 +0000

Intro
Goal
Requirements
- Generate GnuPG Keys
- Manage GnuPG Keys
- Setup Sync Folder
Usage
- Linux and macOS
- Windows
- Decrypt Files
Conclusion
Annex - Metadata
References

Intro

By now I reckon you already know that Google Drive offers the astounding 15 GB of storage for free [1]. Microsoft's OneDrive offers just about 5 GB [2] for free and Dropbox, one of the pioneers in the field, about 2 GB [3]. Dropbox are kind of lagging behind and out of all three Google Drive seems the clear winner here in terms of offering a greater amount of storage for no cost at all.

(The amounts of storage described are from October 2018.)

Needless to say, free online storage is a great way to store your bulk of important data - documents, backups and maybe your large collection of pop music mp3s you downloaded back in the 90s. Don't look at me like that! When it comes to music preferences, we all have sinned. Now, if you are like me, keeping sensitive documents, data backups or SQL exports unprotected in the cloud does not really seem much comfortable. Google do offer extended security by using 2-Step verification process [4], which makes it quite hard for your account to get compromised. We must, however, add that the human factor [5] in security cannot be discarded.

Also, call me paranoid, but the following text in Google's Terms of Service [6] leaves me with a feeling of unease:

Our automated systems analyze your content (including emails) to provide you personally relevant product features, such as customized search results, tailored advertising, and spam and malware detection. This analysis occurs as the content is sent, received, and when it is stored.

See? My paranoia is a tad more justified now. So the question is can we store our data on Google Drive in an encrypted manner - and also how hard would be to automate that.

Goal

Alright, case in point - our goal here is to somehow establish a process for server or desktop located files to be encrypted and uploaded to a Google Drive folder. The synchronization will be one way only, meaning files will only be uploaded from the server or desktop machine to a designated Google Drive folder.

We needed a way to collect, encrypt and synchronize files. Collecting files may be realized via shell scripts like Bash on Un*x or Powershell on Windows.

To encrypt our precious data we'll use GnuPG [7], either version 1.4 or 2.1. It should be downloaded using your operating system's package manager, e.g., Aptitute or Brew, or alternatively from GnuPG's website.

For synchronization, there's gdrive [8], short for Google Drive CLI Client. I dig that short name. What's up G! It's written in Go and it's open source. It comes with pre-built binaries for an impressive range of platforms.

Oh, and here's a diagram of the intended workflow that I drew earlier, to make this article seem less boring. Behold!

Requirements

Generate GnuPG Keys

If you already have a GnuPG key pair that you want to use for encryption, you may skip this step completely. But you really shouldn't, because there're a bunch of lame jokes ahead.

Now, I'm going to first start with a fair warning that dealing with GnuPG could, at times, be somewhat irritating. Cryptography is generally complex and unfortunately using GPG could be a bit cumbersome, but we are going to do a minimal set of steps, so it should all be fine. Take a deep breath. Ready?

First, you need a place to store your GPG key rings. A key ring contains one or more public or private keys. By default, this would be a place in your home dir, i.e., /home/user/.gnupg. I recommend using another, separate directory for this setup. This way we can only store the synchronization keys there.

$ cd /var/local
$ mkdir mygpgkeys && chmod 700 mygpgkeys
$ export GNUPGHOME="/var/local/mygpgkeys"

Pull the curtains down and put your dark shades on, we're generating the GnuPG key pair next.

$ cd mygpgkeys
$ gpg --default-new-key-algo rsa4096 --gen-key

By default gpg generates 2048 bits keys, so we add an extra parameter to initiate a 4096 bit key generation. Fill in a name and an Email address. Although recommended, the Email address does not need to be a real one. You should then be asked about a password to secure your private key. Needless to say, choose a good, long password. Yeah, your cat's name doesn't count unless it's a concat of a bunch of Lovecraft's character names. It might take a while for your key to get generated depending on the available entropy on your system [9].

gpg (GnuPG) 2.1.21; Copyright (C) 2017 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

gpg: keybox '/var/local/mygpgkeys/pubring.kbx' created
Note: Use "gpg2 --full-generate-key" for a full featured key generation dialog.

GnuPG needs to construct a user ID to identify your key.

Real name: Max Mustermann
Email address: maxmustermann@example.org
You selected this USER-ID:              
    "Max Mustermann <maxmustermann@example.org>"

Change (N)ame, (E)mail, or (O)kay/(Q)uit? o
...
gpg: /var/local/mygpgkeys/trustdb.gpg: trustdb created
gpg: key 1A2B04F0E994DAF4 marked as ultimately trusted
gpg: directory '/var/local/mygpgkeys/openpgp-revocs.d' created
gpg: revocation certificate stored as '/var/local/mygpgkeys/openpgp-revocs.d/31208B5D45320FAE3D7E7FE21A2B04F0E994DAF4.rev'
public and secret key created and signed.

pub   rsa2048 2017-05-25 [SC] [expires: 2019-05-25]
        31208B5D45320FAE3D7E7FE21A2B04F0E994DAF4
        31208B5D45320FAE3D7E7FE21A2B04F0E994DAF4
uid                      Max Mustermann <maxmustermann@example.org>
sub   rsa2048 2017-05-25 [E] [expires: 2019-05-25]

GnuPG will generate a public and a secret key ring files named pubring.gpg and secring.gpg. (The file names may differ on macOS! Thanks Tim Cook.) You can get a glimpse on the available keys in your GPG key ring by running the following command:

$ gpg --list-keys

-------------------------------------------------------------------
pub   rsa2048 2017-05-25 [SC] [expires: 2019-05-25]
        31208B5D45320FAE3D7E7FE21A2B04F0E994DAF4
uid           [ultimate] Max Mustermann <maxmustermann@example.org>
sub   rsa2048 2017-05-25 [E] [expires: 2019-05-25]

We got our key pair, so let's board the synchronization train next. Oh, and by the way, you may now take your shades off.

Manage GnuPG Keys

I'll have to resort to bold measures for this one, so here goes - Do NOT lose your GPG key pair! I'm sure you've thought about that already, but take good care of the key you've just generated. You won't be able to decrypt your data, if you lose it!

A poor man solution could be to backup the keys on a dedicated flash drive and hide that under your pillow. If you're less of a paranoid, then you can put your keys in a certificate manager like Kleopatra [10]. Thunderbird users can just install and use the Enigmail [11] extension.

Setup Sync Folder

The first requirement is a Google Drive folder where you'll be uploading your encrypted files to. Notice that the unique id of the folder is given in the browser url.

Next, you need to initialize gdrive and configure a sync folder on the hard drive.

To create a sync folder run:

$ mkdir mysyncfolder && cd mysyncfolder

To initialize gdrive run the command below using the folder id you got from Google Drive. You'll need to manually copy and paste the authorization url in your browser and follow the steps to authorize gdrive to upload files on your behalf.

$ gdrive sync upload . 0BxNabiLkX8lpSGN3UVRmWEVQWWM -c /Users/max/.gdrive-test-sync

Authentication needed
Go to the following url in your browser:
https://accounts.google.com/o/oauth2/auth?access_type=offline&client_id=someverylongstringeg.apps.googleusercontent.com&redirect_uri=...

Enter verification code: 7/TWTDPBq32xYiRqqAlWp4tkcVBYgL3n5qsD_bfzXr8B0
Starting sync...
Collecting local and remote file information...
Found 0 local files and 0 remote files
Sync finished in 2.095311029s

Notice that the gdrive access token parameters are saved to a manually specified path, i.e., in /Users/max/.gdrive-test-sync. This should normally be a location that only your user is allowed to access. chmod 600 anyone?

Usage

Linux and macOS

Let's write a Bash script that will encrypt all files from a given folder on the hard drive and then upload them to a specified Google Drive folder. We'll call it sync.sh, because I'm out of original ideas.

$ touch sync.sh && chmod +x sync.sh

Open the file in an editor and add the following. Check the comments to adjust the source and destination paths where needed.

#!/bin/sh

GPG=$(which gpg)
GDRIVE=$(which gdrive)

# checks if the required tools are available 
if [ "XY$GPG" = "XY" ]; then
    echo "GPG not found!"
    exit 1
fi
if [ "XY$GDRIVE" = "XY" ]; then
    echo "gdrive not found!"
    exit 2
fi


####################################
## configuraitons
####################################

##### !IMPORTANT! #####
## Make sure to adjust the paths to your system
#######################

# key to encrypt with (the one we generated earlier)
GPG_KEY="maxmustermann@example.org"
export GNUPGHOME="/var/local/mygpgkeys"
# destination folder id in gdrive
GDRIVE_DEST="0BxNabiLkX8lpSGN3UVRmWEVQWWM"
# local sync folder path
GDRIVE_SYNC_DIR="/var/local/mysyncfolder"
# gdrive config
GDRIVE_CONFIG_PATH="/Users/max/.gdrive-test-sync"

####################################
## encrypt source files
####################################
echo

FOLDER="/<PATH-TO-MY-IMPORTANT-FILES-OR-BACKUPS>"

for f in $FOLDER/*; do
    echo "Encrypting $f ..."
    echo y | $GPG --recipient="$GPG_KEY" -e $f
    GPG_FILE="$f.gpg"

    BASE_NAME=$(basename $GPG_FILE)
    echo "Moving to $GDRIVE_SYNC_DIR/$BASE_NAME ..."
    mv $GPG_FILE $GDRIVE_SYNC_DIR/$BASE_NAME
done

## ...you may add more folders here

####################################
## sync everything
####################################

echo
echo "Sync files with Google Drive ..."

$GDRIVE sync upload $GDRIVE_SYNC_DIR $GDRIVE_DEST -c $GDRIVE_CONFIG_PATH

####################################
## cleanup
####################################

echo
if [ "XZ$GDRIVE_SYNC_DIR" != "XZ" ]; then
    echo "Removing encrypted files from $GDRIVE_SYNC_DIR ..."
    rm -f $GDRIVE_SYNC_DIR/*
fi

Now this is a very simple script that may be further extended. For example, traversing a folder tree or producing and encrypting a tar archive composed of many small sized files. I leave this up to you to adjust as needed.

Alright, let's run the sync.sh script.

$ ./sync.sh 

Encrypting LICENSE ...
Moving to /var/local/mysyncfolder/LICENSE.gpg ...
Encrypting Makefile ...
Moving to /var/local/mysyncfolder/Makefile.gpg ...
Encrypting MobileDevice.h ...
Moving to /var/local/mysyncfolder/MobileDevice.h.gpg ...
Encrypting README.md ...
Moving to /var/local/mysyncfolder/README.md.gpg ...

Sync files with Google Drive ...
Starting sync...
Collecting local and remote file information...
Found 4 local files and 4 remote files

4 local files has changed
[0001/0006] Updating LICENSE.gpg -> TestSync/LICENSE.gpg
[0002/0006] Updating Makefile.gpg -> TestSync/Makefile.gpg
[0003/0006] Updating MobileDevice.h.gpg -> TestSync/MobileDevice.h.gpg
[0004/0006] Updating README.md.gpg -> TestSync/README.md.gpg
Sync finished in 7.074881652s

Removing encrypted files from /var/local/mysyncfolder ...

If you now take a look in Google Drive, you'll find all the encrypted files placed in the target folder.

So what if you need this done on regular basis? Did I hear you say a cron job? Here's one that invokes the sync.sh script at 00:30 every day:

# m h  dom mon dow   command
30 0 * * * /var/local/sync.sh >> /var/log/sync.log

Windows

The sync workflow and script on Windows systems is pretty much the same. Of course, you would need the Windows version of GnuPG - Gpg4win and a Windows binary of the gdrive CLI.

@echo off
setlocal

REM ################################
REM ## configuraitons
REM ################################
SET RC=0
SET "CURRENT_DIR=%cd%"

SET "GDRIVE=C:\SYNC_TEST\gdrive-windows-x64.exe"

REM # key to encrypt with (the one we generated earlier)
SET "GPG_KEY=maxmustermann@example.org"
SET "GNUPGHOME=C:\SYNC_TEST\mygpgkeys"
REM # destination folder id in gdrive
SET "GDRIVE_DEST=0BxNabiLkX8lpSGN3UVRmWEVQWWM"
REM # local sync folder path
SET "GDRIVE_SYNC_DIR=C:\SYNC_TEST\mysyncfolder"
REM # gdrive config
SET "GDRIVE_CONFIG_PATH=C:\Users\<MY_USERNAME>\AppData\Local\gdrive-test"

REM ####################################
REM ## encrypt source files
REM ####################################

SET "FOLDER=C:\SYNC_TEST\sourcefiles"

PUSHD
CD %FOLDER%

for %%f in (*) do ( 
    echo Encrypting %%f ...
    gpg --recipient="%GPG_KEY%" -e %%f
    if errorlevel 1 (
        goto error_encrypt
    )

    echo Moving %%f.gpg to %GDRIVE_SYNC_DIR% ...
    move %%f.gpg %GDRIVE_SYNC_DIR%
)
POPD


REM ####################################
REM ## sync everything
REM ####################################

echo Sync files with Google Drive ...

PUSHD
CD %GDRIVE_SYNC_DIR%
%GDRIVE% sync upload %GDRIVE_SYNC_DIR% %GDRIVE_DEST% -c %GDRIVE_CONFIG_PATH%
POPD

REM ####################################
REM ## cleanup
REM ####################################
:cleanup

PUSHD
cd %GDRIVE_SYNC_DIR%
echo Removing encrypted files from %GDRIVE_SYNC_DIR% ...
del /F /Q *
POPD

goto end

:error_encrypt
echo Error encrypting file.
goto end

:end

Running the sync.cmd script would produce the following:

> sync.cmd
Encrypting LICENSE ...
Moving LICENSE.gpg to C:\SYNC_TEST\mysyncfolder ...
        1 file(s) moved.
Encrypting Makefile ...
Moving Makefile.gpg to C:\SYNC_TEST\mysyncfolder ...
        1 file(s) moved.
Encrypting MobileDevice.h ...
Moving MobileDevice.h.gpg to C:\SYNC_TEST\mysyncfolder ...
        1 file(s) moved.
Encrypting README.md ...
Moving README.md.gpg to C:\SYNC_TEST\mysyncfolder ...
        1 file(s) moved.
Sync files with Google Drive ...
Starting sync...
Collecting local and remote file information...
Found 4 local files and 6 remote files

4 local files has changed
[0001/0004] Updating LICENSE.gpg -> TestSync\LICENSE.gpg
[0002/0004] Updating Makefile.gpg -> TestSync\Makefile.gpg
[0003/0004] Updating MobileDevice.h.gpg -> TestSync\MobileDevice.h.gpg
[0004/0004] Updating README.md.gpg -> TestSync\README.md.gpg
Sync finished in 4.2520023s
Removing encrypted files from C:\SYNC_TEST\mysyncfolder ...

You can use the Windows Task Scheduler [12] to run the sync.cmd script at a desired time or interval.

Decrypt Files

So, the encrypt and sync workflow is ready and this is nice and all, but how does one get back their content in case they need it? Downloading the files from Google Drive is not an issue, but how about decrypting the content? Remember my all 90s mp3 file collection? Those are an important legacy I'd like my grandchildren to have.

Here is a simple bash script the decrypts all encrypted gpg files in a directory.

#!/bin/sh

if [ ! -d "$1" ]; then
    echo "No input folder specified!"
    exit 1
fi

GPG=$(which gpg)
if [ "XY$GPG" = "XY" ]; then
    echo "GPG not found!"
    exit 1
fi

FOLDER=$1

for f in $FOLDER/*.gpg; do
    echo "Decrypt $f ..."
    echo "<your-gpg-key-password>" | gpg --passphrase-fd 0 --batch --yes $f
done

Simply pass the directory path where all encrypted files reside and run the script to get all of them decrypted.

$ cd my-gdrive-downloaded-files && ./decrypt .

Decrypt ./LICENSE.gpg ...
gpg: encrypted with 2048-bit RSA key, ID E31A85E6729D5490, created 2017-05-25
    "Max Mustermann <maxmustermann@example.org>"
Decrypt ./Makefile.gpg ...
gpg: encrypted with 2048-bit RSA key, ID E31A85E6729D5490, created 2017-05-25
    "Max Mustermann <maxmustermann@example.org>"
Decrypt ./MobileDevice.h.gpg ...
gpg: encrypted with 2048-bit RSA key, ID E31A85E6729D5490, created 2017-05-25
    "Max Mustermann <maxmustermann@example.org>"
Decrypt ./README.md.gpg ...
gpg: encrypted with 2048-bit RSA key, ID E31A85E6729D5490, created 2017-05-25
    "Max Mustermann <maxmustermann@example.org>"

So, that's it. You got all your data encrypted. Sundar Pichai will never smile again.

Conclusion

In the end, what's all this good for? (I probably should have put that question at the beginning of the article.)

I personally use it to keep server backups safely stored in the cloud. Mostly binary files, PDFs, archives, etc. It probably doesn't make sense to store frequently changing documents like text or Word files using this method, although there's nothing stopping you.

Annex - Metadata

One topic that I did not consider when I was writing this article is Metadata [13]. You may have noticed that the original filenames in the scripts above are always preserved when uploaded to Google Drive. The name and size of a file may be enough information for complex algorithms to still extract quite a lot of meaningful info about what the purpose and contents of that file may be in your user context. A simple counter mechanism could be used to produce a hash value of each filename, thus at least preventing filename metadata extraction.

Here is a modification of the encryption script for Linux/macOS that hashes the filenames using arbitrary salt value and produces a CSV file index of all encrypted files.

# a registry of hashed files
INDEX="/var/local/gdrive_index.csv"
SALT="someveryveryveryveryveryveryverylongstring"

## reset index contents
echo "FILENAME;HASHED NAME" > $INDEX

####################################
## Encrypt source files
####################################
echo

FOLDER="/<PATH-TO-MY-IMPORTANT-FILES-OR-BACKUPS>"

for f in $FOLDER/*; do
    echo "Encrypting $f ..."
    echo y | $GPG --recipient="$GPG_KEY" -e $f

    BASE_NAME=$(basename $f)
    HASHED=$(echo $BASE_NAME.$SALT | openssl dgst -sha256)
    echo "$BASE_NAME;$HASHED" >> $INDEX

    GPG_FILE="$f.gpg"
    HASHED_GPG_FILE="$HASHED.gpg"

    echo "Moving to $GDRIVE_SYNC_DIR/$HASHED_GPG_FILE ..."
    mv $GPG_FILE $GDRIVE_SYNC_DIR/$HASHED_GPG_FILE
done

References

GDrive Pricing Guide - google.com/drive/pricing/
Microsoft OneDrive Plans - onedrive.live.com/about/en-us/plans
How much does Dropbox cost? - dropbox.com/help/billing/cost
Google's 2-Step Verification - google.com/landing/2step
The human factor is key to good security - computerweekly.com/opinion/The-human-factor-is-key-to-good-security
Google Terms of Service - google.com/intl/en/policies/terms
What’s GnuPG? - gnupg.org/faq/gnupg-faq.html#whats_gnupg
Google Drive CLI Client - github.com/prasmussen/gdrive
GPG does not have enough entropy - serverfault.com/q/214605
Kleopatra - openpgp.org/software/kleopatra
Enigmail - enigmail.net/index.php/en
Schedule a Task - technet.microsoft.com/en-us/library/cc748993(v=ws.11).aspx
Metadata - en.wikipedia.org/wiki/Metadata

SSH File System

Petar G. Petrov — Tue, 09 Oct 2018 19:16:24 +0000

Dig deep!

Get this article read to you.

Goal
The SSH File System
User Access Workflow
Setup
- Debian Linux
- macOS

Goal

A couple of weeks ago I found myself setting up an e-books server that was supposed to store my ever growing "Books I'd like to but I'll rarely find the time to read" collection. That imposed an interesting network problem I had to solve. I had a large external drive connected to another machine that I wanted to utilize, however, setting up a Samba server wasn't something I wanted to spend several cups of coffee on and NFS wasn't an option for me either. It occurred to me that I should be able to achieve my goal using SSH.

The SSH File System

First, I must say, I just love this idea. Using SSH as means to provide seamless file I/O operations and access control via asymmetric crypto is something I'll always consider from now on. The tool behind this is sshfs. It's being frequently updated and one may be sure to find it in their package manager.

However, there are a couple of things to consider.

It's expected that SSH transfers will be slower due to crypto (and of course network) operations, although sshfs does use caching and multi-threading to address this. In any case, this does not seem to be an issue for me in a local network over a wired ethernet connection.
sshfs leads to some very interesting user management cases, but this may quickly get too complex to handle. I'd say that using this within a complex access permissions scenario could be painful.

User Access Workflow

Here's a simplified workflow on what happens behind the scenes.

alice on Server A needs to access Server B's storage.
bob has full access to Server B's storage.
alice generates an ssh key pair and uses that to establish an SSH connection and impersonate Bob on Server B.
alice creates a mount point on Server A and can induce file read/write operations on Storage via the established SSH channel.

Setup

I have prepared two setup use cases - one based on Linux that I'm currently using and another one for macOS, which I mostly did for research purposes. It is most certainly possible to use sshfs on Windows, however, I have not done any setup research in that direction.

Debian Linux

Install sshfs on Server A via aptitude:

sudo apt-get install sshfs

Install FUSE (Filesystem in Userspace) via aptitude:

sudo apt-get install fuse

You would need to load the fuse module via:

modprobe fuse

Check if the module is loaded via:

lsmod | grep fuse

Run ssh-keygen on Server A to generate a key pair.

Copy the public key in $HOME/.ssh/id_rsa.pub from Server A and paste it to $HOME/.ssh/authorized_keys on Server B. Or just use scp to copy the file over.

Create a target mount point and mount it to Server A, e.g.,

mkdir $HOME/mountpoint
sshfs bob@server-b:/var/storage $HOME/mountpoint

You now have access to Server B's storage space. To unmount run:

fusermount -u $HOME/mountpoint

Mount via systemd.service

It'd be great to have this run as a service, so if Server A reboots, the mount point gets setup automatically. For systems using systemd we can do that the following way:

Create a mount file in your $HOME (or any other accessible) directory.

#!/bin/sh

CMD="$1"
MNT="/home/mountpoint"
TARGET="/var/storage/"

start() {
  exec sshfs bob@server-b:$TARGET $MNT
}

stop() {
  fusermount -u $MNT
}

case $1 in
  start|stop) "$1" ;;
esac

As a root user, create a bob-mount.service file in /lib/systemd/system and add the following inside:

[Unit]
Description=Server B Mount Service
After=network.target

[Service]
User=alice
Group=alice
Type=forking
ExecStart=/home/alice/mount start
ExecStop=/home/alice/mount stop

[Install]
WantedBy=multi-user.target

Start and stop the service via:

systemctl start bob-mount
systemctl stop bob-mount

Enable the service to run at boot time:

systemctl enable bob-mount

macOS

Installation on macOS is pretty similar. Use brew to install the following:

brew install Caskroom/cask/osxfuse
brew install Caskroom/cask/sshfs

You should see FUSE for macOS in your System Preferences afterwards. To mount a share run:

mkdir $HOME/mountpoint
sshfs bob@server-b:/var/storage $HOME/mountpoint

To unmount just run:

unmount $HOME/mountpoint

An easy way to mount after reboot is to create a mount script and put it in the crontab.

Forem: Petar G. Petrov

Java and Native Code in OpenJDK 13

Contents

Intro

Hands-on

The libCurl App

The libSSH2 App

petarov / java-panama-ssh2

Java SSH client using libssh2 through project Panama

java-panama-ssh2

Install

Run

License

Final Thoughts

References

Help Me Delete Your Tweets!

petarov / shut-up-bird

🐦 Put your tweets/likes in an EPUB and delete them like a boss

Shut Up Bird

Installation

Packages

Manual

Setup

Home Storage with Raspberry Pi and Samba

Contents

Goal

Raspberry Pi

Remove X11 (Optional)

Storage User

Samba Server

Installation

Configuration

Clients

Linux Client

View Shares

fstab Mount

Android Client

iOS Client

References

Store Encrypted Files in Google Drive

Contents

Intro

Goal

Requirements

Generate GnuPG Keys

Manage GnuPG Keys

Setup Sync Folder

Usage

Linux and macOS

Windows

Decrypt Files

Conclusion

Annex - Metadata

References

SSH File System

Contents

Goal

The SSH File System

User Access Workflow

Setup

Debian Linux

Mount via systemd.service

macOS