Forem: Sean Policarpio

Un-JAR-ing your Java apps for Docker

Sean Policarpio — Fri, 15 Dec 2023 09:45:00 +0000

After a small detour at work, I've found myself returning to the world of Java and its virtual machine.

Recently, I was tasked with preparing our backend application in a Docker image for deployment. Having done something similar with a Scala backend before, I was confident I could replicate a strategy I had learned, but with Java build tools instead. Succinctly, the strategy was to create an incremental and layered image by not using a Java JAR, but instead copying and directly executing compiled Java bytecode. After further expanding on the motivation to not use a JAR, I will demonstrate how I did this in Maven using standard plugins.

Why not use a JAR?

Having the ENTRYPOINT or CMD of your Dockerfile execute something like below does work:

java -jar your_app.jar

So why not stick with that approach? First off all, a JAR within a Docker image is redundant: both are basically archive formats. So quite simply, it's really unnecessary to have your code in a JAR if the unit of execution is the Docker image itself.

But there are more important benefits that become apparent when you start running your app in production.

Easier image inspection and debugging

With every code change you and your team make, those changes are likely reflected in your version control system. But those same changes are opaquely hidden if the JAR that bundles it all up is copied into a Docker image. I've found it extremely useful to be able to inspect the code within Docker images without the need to extract JARs. This is especially true when things go awry in production and you need to do this in your container environment (e.g. AWS, Google Cloud Platform). This could be anything from inspecting class code using javap or—something I've done several times in the past—viewing bundled resources like .yaml, .properties, or .hocon config files to confirm settings or flags. If a containers state is frequently changing, it's especially helpful when you don't need to re-extract the JAR archive everytime the container is restarted.

If you happen to bundle source code in your JARs, it is also useful to have them unbundled in your container when debugging. Ideally, you would have something like a git SHA for reference when comparing with version control (e.g. by tagging the Docker image with the SHA). But sometimes it's still helpful to dig into the image to sanity check what is exactly in there.

Diffs

Intuitively, it is also beneficial if everytime the image was built, you only copied the modified code and resources. This would allow you to reduce the Docker layer size between builds—likely reducing it into the magnitude of bytes or kilobytes. Unfortunately, Docker has had an age-old bug which doesn't allow you to optimize your subsequent image builds by detecting and copying only those files that have changed (e.g. like rsync). If this is ever improved, this would be another benefit to unbundling JARs: smaller Docker images, faster repository pushes/pulls, and less remote storage required.

In any case, being able to tell what's different between two builds is valuable and easier without a JAR. You can use diff to compare your application directory in two images, but you'd need to do some cp'ing to get everything localised first. Fortunately, a tool like container-diff makes this much easier. The following shows an example of performing a diff to show the only file modified and compiled in a revision to my Docker image:

% ./container-diff-darwin-amd64 diff daemon://java-backend-api:1 daemon://java-backend-api:2 --type=file     

-----File-----

These entries have been added to java-backend-api:1: None

These entries have been deleted from java-backend-api:1: None

These entries have been changed between java-backend-api:1 and java-backend-api:2:
FILE                                             SIZE1        SIZE2
/app/io/policarp/service/DemoServer.class        2K           2.2K

Additionally, unless you are copying an "uber" or "shaded" JAR, it's likely you must copy all individual dependencies into your Docker image too. The approach I'll demonstrate shows how you can also gain the same insight into third-party libraries by not having them also bundled as JARs.

Prepping your pom.xml

In my last Scala project, I used the sbt-assembly plugin to prep my code for copying and execution in Docker. Assuming you have a single project pom, the following excerpt shows the two plugins I used to repeat the same outcome in Maven.

<plugins>
    ...
    <plugin>
        <artifactId>maven-assembly-plugin</artifactId>
        <version>3.6.0</version>
        <configuration>
            <descriptorRefs>
                <descriptorRef>jar-with-dependencies</descriptorRef>
            </descriptorRefs>
        </configuration>
    </plugin>

    <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>exec-maven-plugin</artifactId>
        <version>3.1.1</version>
        <executions>
            <execution>
                <!-- Provides an execution to unzip the packaged JAR for use in a Docker image -->
                <id>unzip-assembly-for-docker</id>
                <configuration>
                    <executable>unzip</executable>
                    <arguments>
                        <argument>-quo</argument> <!-- quiet, update, overwrite -->
                        <argument>target/${project.build.finalName}-jar-with-dependencies.jar</argument>
                        <argument>-d</argument>
                        <argument>target/docker-image-target</argument>
                    </arguments>
                </configuration>
            </execution>
        </executions>
    </plugin>
    ...
</plugins>

maven-assembly-plugin is first used to create a JAR bundle of the application code and all the dependencies using the pre-configured descriptor jar-with-dependencies.
exec-maven-plugin is then used to unzip the assembly JAR for copying into Docker.

exec-maven-plugin is used just so we can keep everything inside of Maven. This is not only helpful for developer ergonomics, but also for keeping CI/CD pipelines simple. The caveat is that the environment where Maven is running depends on the command-line tool unzip, which is fortunately normally available.

With the above configured, you can simply run the following to prepare your application for Docker:

mvn package assembly:single exec:exec@unzip-assembly-for-docker

package will JAR your compiled app,
assembly:single will assemble your package JAR with the dependency JARs, and finally,
exec will call the unzip command to extract everything into the target/docker-image-target directory, ready for Docker.

As demonstrated earlier with my compiled DemoServer Java class, the maven-assembly-plugin will similarly copy all third-party dependency libraries as extracted JARs and give you the opportunity to perform diffs on library updates between Docker image builds. This can prove useful when trying to determine what exactly has changed between builds, even if you already have library updates recorded in version control. And if Docker ever optimises copying only changed files, we'd get an improvement with builds with respect to library updates.

Creating the Docker image

With all your compiled code in a target directory, your Dockerfile can be as simple as the following:

FROM eclipse-temurin:21.0.1_12-jre-alpine

WORKDIR /app

# add and run-as unprivileged user
RUN addgroup --system app-user-docker
RUN adduser --system --disabled-password --no-create-home app-user-docker app-user-docker
USER app-user-docker

# copy the extracted JAR code on to the image
COPY target/docker-image-target /app

EXPOSE 8080
ENTRYPOINT ["java", "io.policarp.service.DemoServer"]

Next time you're tasked with building a Java app in Docker, consider the approach I've demonstrated here. It's worth noting that if there is any reason you don't like it and/or find issues with unbundled JARs, going back (and forth) is not burdensome. Finally, in case you are wondering, this works just fine with Spring Boot.

HttpClient can't connect to a TLS proxy

Sean Policarpio — Mon, 13 Sep 2021 13:04:20 +0000

This is going to be a short and to the point post. Mainly because I'm pretty tired already from trying to figure this out 😅.

Problem

Java's HttpClient (java.net.http.HttpClient) allows you to specify a proxy to route all requests through like this:

final HttpClient client = 
  HttpClient.newBuilder()
    .proxy(ProxySelector.of(new InetSocketAddress(proxyHost, proxyPort)))
     // other builder config goes here
    .build();

What I discovered while trying to build some code to communicate with a proxy we are using at my company is that if proxyHost (which is just a host address, without any protocol) and proxyPort point to an endpoint that is encrypted with TLS, Java's current implementation won't work. This is because the implementation assumes the connection (via a CONNECT request) is established without TLS.

Code

My findings were discovered whilst debugging a small sample app and referring to the openjdk/jdk code base.

First off, you can refer to getConnection in HttpConnection, which is called when attempting to complete an HttpRequest.

    /**
     * Factory for retrieving HttpConnections. A connection can be retrieved
     * from the connection pool, or a new one created if none available.
     *
     * The given {@code addr} is the ultimate destination. Any proxies,
     * etc, are determined from the request. Returns a concrete instance which
     * is one of the following:
     *      {@link PlainHttpConnection}
     *      {@link PlainTunnelingConnection}
     *
     * The returned connection, if not from the connection pool, must have its,
     * connect() or connectAsync() method invoked, which ( when it completes
     * successfully ) renders the connection usable for requests.
     */
    public static HttpConnection getConnection(InetSocketAddress addr,
                                               HttpClientImpl client,
                                               HttpRequestImpl request,
                                               Version version) {
        // The default proxy selector may select a proxy whose  address is
        // unresolved. We must resolve the address before connecting to it.
        InetSocketAddress proxy = Utils.resolveAddress(request.proxy());
        HttpConnection c = null;
        boolean secure = request.secure();
        ConnectionPool pool = client.connectionPool();


        if (!secure) {
            c = pool.getConnection(false, addr, proxy);
            if (c != null && c.checkOpen() /* may have been eof/closed when in the pool */) {
                final HttpConnection conn = c;
                if (DEBUG_LOGGER.on())
                    DEBUG_LOGGER.log(conn.getConnectionFlow()
                                     + ": plain connection retrieved from HTTP/1.1 pool");
                return c;
            } else {
                return getPlainConnection(addr, proxy, request, client);
            }
        } else {  // secure
            if (version != HTTP_2) { // only HTTP/1.1 connections are in the pool
                c = pool.getConnection(true, addr, proxy);
            }
            if (c != null && c.isOpen()) {
                final HttpConnection conn = c;
                if (DEBUG_LOGGER.on())
                    DEBUG_LOGGER.log(conn.getConnectionFlow()
                                     + ": SSL connection retrieved from HTTP/1.1 pool");
                return c;
            } else {
                String[] alpn = null;
                if (version == HTTP_2 && hasRequiredHTTP2TLSVersion(client)) {
                    alpn = new String[] { "h2", "http/1.1" };
                }
                return getSSLConnection(addr, proxy, alpn, request, client);
            }
        }
    }


    private static HttpConnection getSSLConnection(InetSocketAddress addr,
                                                   InetSocketAddress proxy,
                                                   String[] alpn,
                                                   HttpRequestImpl request,
                                                   HttpClientImpl client) {
        if (proxy != null)
            return new AsyncSSLTunnelConnection(addr, client, alpn, proxy,
                                                proxyTunnelHeaders(request));
        else
            return new AsyncSSLConnection(addr, client, alpn);
    }

Skimming through it yourself, you will find regardless of the destination/target address you are trying to hit in request (http:// or https://), a connection must be made with the specified proxy, which again, is expecting to be communicated via TLS.

If your target is http://, then a call to getPlainConnection is made. This leads to a connection being established with the proxy via PlainProxyConnection, which from what I can see, will establish a socket connection, but not negotiate a TLS handshake when a CONNECT request is sent.

If your target is https://, then a call to getSSLConnection is made. This leads to a connection being established with the proxy via AsyncSSLTunnelConnection, which according to its own documentation is, "An SSL tunnel built on a Plain (CONNECT) TCP tunnel".

In either case, it's evident the issue is that although a socket channel is established, when communication is attempted with the socket, because the server is expecting encrypted communication, things just don't work.

https://github.com/square/okhttp/issues/3787 Discovered another client that doesn't support it either

tapir: Yet another way to REST

Sean Policarpio — Thu, 05 Dec 2019 21:52:41 +0000

Here at ClearScore, the backend developers utilize a number of tools and libraries to create the software that runs behind the scenes. In more cases than not, the software consists of services that communicate "RESTfully" via HTTP(S)—both with other backend services and with the frontend. Of the libraries in use at ClearScore, http4s was one I was excited and happy to learn I'd be using when I joined the company.

In the past, I've used various libraries to accomplish the development of HTTP services, including Spray, Akka HTTP, and Play in Scala; Spring Boot and Java Servlets in Java; Rocket and Actix-web in Rust; as well as simply using PHP or Ruby on Rails. But focusing on Scala—which is what the majority of ClearScore's backend is written in—http4s was attractive to me because of two main things: its simplicity and its adherence to functional programming.

For example, here's an HTTP service that returns the current time:

import java.time.LocalDateTime
import java.time.format.DateTimeFormatter

import cats.effect.{ExitCode, IO, IOApp}
import org.http4s.HttpRoutes
import org.http4s.dsl.io._
import org.http4s.implicits._
import org.http4s.server.blaze.BlazeServerBuilder

object Time extends IOApp {

  val service = HttpRoutes.of[IO] {
    case GET -> Root / "time" =>
      Ok(theTime())
  }

  override def run(args: List[String]): IO[ExitCode] = {
    BlazeServerBuilder[IO]
      .bindHttp(port = 8080, host = "0.0.0.0")
      .withHttpApp(service.orNotFound)
      .serve
      .compile
      .drain
      .map(_ => ExitCode.Success)
  }

  def theTime(): String = {
    LocalDateTime.now().format(DateTimeFormatter.ISO_TIME)
  }
}

Yeah, not very exciting, but simple is what we want (trust me 🙂). Take note, this particular implementation is using the Cats Effect library: put simply, this provides us with an effect type, IO, to run side-effecting input and output asynchronously (e.g. like interactions with our HTTP server or with external services, like the computers hardware clock). http4s is compatible with other effect types, but that's not important right now. Ignoring most of the code, the cool thing, in my opinion, is just four lines in this example:

val service = HttpRoutes.of[IO] {
    case GET -> Root / "time" =>
        Ok(theTime())
}

service is not a complex thing at all. At its core, it's a partial function that accepts as input a thing of type Request[F] and outputs as a result a thing of type Response[F], where F is IO in my example (but again, that's just a minor detail for now). The beauty, again, lies in the simplicity of the detail: service is easy to reason upon what it can and will do. This implies testing it is straightforward.

def test(): Unit = {
  val routes = service.orNotFound // i.e. anything outside our partial function will '404'

  val response: Response[IO] = routes(Request(GET, uri"/time")).unsafeRunSync()

  assert(response.status == Status.Ok)
  assert(response.bodyAsText.compile.string.unsafeRunSync().nonEmpty)

  val notfound: Response[IO] = routes(Request(GET, uri"/nonexistent")).unsafeRunSync()

  assert(notfound.status == Status.NotFound)
}

So why is this blog post about something else if http4s is so great?? Well, let's consider the following points.

First, any web service can grow in size, where by size I'm referring to the possible API endpoints a service can present to the outside world. Partial functions are good, but they can get unwieldy due to the size of your service. Fortunately, in http4s you can break your routing functions into smaller logical units and merge them together just before serving them up. But what does a function offer if the types are too vague? A Response[IO] doesn't tell you too much about what kind of response it will provide, unless you look directly at each and every individual implementation.

Another point that is particularly relevant to ClearScore—and most likely many other software companies today—is that we practice proper documentation of our REST API's, especially for internal use. This is because we have a lot of services. Because we are only human, we can't possibly know how every service works, all the time. Fortunately, we utilize quite a common practice in web-based software development: Swagger/OpenAPI-based documentation. In fact, every service in ClearScore has to have an HTML accessible Swagger UI clearly showing you what a REST API can do. As a new starter at ClearScore, this has been extremely useful for learning about the different services in our ecosystem.

Unfortunately, being human also means we still have other flaws: maintaining the Swagger documentation (JSON or yaml) can be painful or simply forgotten.

Enter tapir.

tapir

tapir (Typed API descRiptions) is a library from the awesome people at SoftwareMill. At ClearScore, we already depend on one of their other popular libraries, sttp. Recently though, through the introduction via one of my work colleagues ^1 , I also learned of tapir

tapir presents another level of abstraction to HTTP services, specifically those written in Scala. With http4s in mind ^2 , tapir provides a typed DSL that sits not entirely on top of http4s, but more so, nicely just over the routing logic it's designed around. By doing so, the interoperability between tapir and http4s is trivial.

So what does my service instead look like in tapir?

import sttp.tapir._
import sttp.tapir.server.http4s._

// assume we have the following implicity in scope
// serverOptions: Http4sServerOptions[IO], fs: Sync[IO], fcs: ContextShift[IO]

val timeEndpoint = endpoint.in("time").get.out(stringBody)
val service = timeEndpoint.toRoutes(_ => IO((theTime().asRight[Unit])))

Hmm, does that look better? Let's break it down and see. So the first thing to understand is tapir uses a completely different model for building your API. In my own words, I'd say it bears resemblance to the builder pattern that some of us are used to in Java and maybe in some Scala libraries. endpoint acts as an empty starting point to build our service routes. In my example, timeEndpoint is:

an endpoint with an input on the path "time",
that will be a GET method, and
that will output a String HTTP body response

The last line calls toRoutes to "provide the logic" for our essentially templated function. In this case, our GET path templates a function of Unit => IO[Either[Unit, String]]. tapir uses Either to represent the possibility of an error occurring with your logic. In our simple example, our side-effecting method theTime is forced into a right and delayed in an IO. Obviously, your real logic would do a better job at handling effects and error handling. Alternatively, tapir also provides toRouteRecoverErrors which will handle any unexpected errors from your effect and hide the use of Either from you.

Just a quick aside: I realize having a function of Unit => IO[Either[Unit, String]] for our GET isn't really helpful if we actually want to report your error to the API caller. Don't worry, tapir can map the left side of an Either to an HTTP error status code and a response using errorOut. For now, to keep things simple, I will ignore proper error handling in my examples.

In the end, service has the same type as before (HttpRoutes[IO]) and can be used without difference in our Time app. So at this point, maybe you're not entirely convinced about using a different method to create effectively the same thing yet? Let's look at a slightly bigger example. Imagine our service did this instead:

val service = HttpRoutes.of[IO] {
  case GET -> Root / "time" =>
    Ok(theTime())
  case request @ PUT -> Root / "time" =>
    ???     
  case GET -> Root / "time" / "zone" =>
    ???
  case request @ PUT -> Root / "time" / "zone" =>
    ???
}

In tapir, we could do the following:

val timeEndpoint = endpoint.in("time")
val timeZoneEndpoint = timeEndpoint.in("zone")

val a = timeEndpoint.get.out(stringBody).toRoutes(_ => IO((theTime().asRight[Unit])))
val b = timeEndpoint.put.in(stringBody).toRoutes(time => ???)
val c = timeZoneEndpoint.get.out(stringBody).toRoutes(_ => ???)
val d = timeZoneEndpoint.put.in(stringBody).toRoutes(zone => ???)

val service = a <+> b <+> c <+> d

Let's highlight some benefits:

Unlike the repeated paths in our http4s route, in the tapir example we define just once a path for "time". Furthermore, only once do we define a sub-path to "zone" (i.e. "/time/zone"). In terms of refactor, this is a major plus, especially when we decide to add or change logical paths.
In the http4s route, I've skipped the entity body deserialization steps that would have to occur in each PUT route. It's also not clear what kind of body content-type we are expecting. In tapir, I've clearly stated the body content we expect is a String ^3 .
Finally, as highlighted in the previous example, now that I've defined my routes once, the logic just needs to be plugged in (i.e. where my ??? placeholders are). tapir has done its type safe magic to state what kind of logic it expects based on the inputs and outputs I've defined. I think this is a major win because it means our logic can be referentially transparent. It's just a function that happens to match the types defined. With a better set up (i.e. using dependency injection of some sort), we could easily swap and/or test our HTTP routes and logic.

Take these benefits and multiple them a few more times with respect to the size of your REST API. I'm quite confident using the builder pattern might save you some time reasoning about what it is your REST API should do and how it should do it. Furthermore, the type safety enforced by the plug and play logic will make developing your API feel more like a block building experience.

One glaring downside to the tapir approach is that we need to combine the individual routes to create our service. In comparison to working with an increasingly complex partial functions, I've come to prefer tapir when working with larger REST APIs. In fact, that is usually my deciding factor: if the REST API will be considerably large, I'll defer to tapir to wrangle together all the varying server logic into orderly units. If the REST API will only serve a handful of endpoints, most of which probably belong to the same or related logic implementations, then I stick with writing smaller routes using the http4s DSL.

Documenting your API

The second thing that attracted me to tapir was an extension to the core library that allowed all those type safe routes I handcrafted to automagically get documentation. Let's rewrite the last code example to quickly demonstrate.

import sttp.tapir.docs.openapi._
import sttp.tapir.openapi.circe.yaml._

val timeEndpoint = endpoint.in("time")
val timeZoneEndpoint = timeEndpoint.in("zone")

object Endpoints {
  val timeBody = stringBody.description("ISO time")
  val zoneBody = stringBody.description("Zone ID")

  val getTime = timeEndpoint.get.out(timeBody).description("Retrieves the current time")
  val setTime = timeEndpoint.put.in(timeBody).description("Changes the current time")
  val getZone = timeZoneEndpoint.get.out(zoneBody).description("Retrieves the current time zone")
  val setZone = timeZoneEndpoint.put.in(zoneBody).description("Changes the current time zone")
}

import Endpoints._

val service = getTime.toRoutes(_ => IO((theTime().asRight[Unit]))) <+> 
              setTime.toRoutes(???) <+> 
              getZone.toRoutes(???) <+> 
              setZone.toRoutes(???)

val openAPI = Seq(getTime, setTime, getZone, setZone).toOpenAPI(title = "Time API", version = "V1")

// println(openAPI.toYaml) would print a completely auto-generated OpenAPI specification in yaml

I've slightly organized things a bit differently, mainly for readability. However, there are only really two things I've made in addition: I added human-readable description's to my endpoints and response/request bodies; and I merged them together into tapir's OpenAPI type. As commented in the code, if I wanted to view or write the OpenAPI specification, I could println it to get the following:

openapi: 3.0.1
info:
  title: Time API
  version: V1
paths:
  /time:
    get:
      description: Retrieves the current time
      operationId: getTime
      responses:
        '200':
          description: ISO time
          content:
            text/plain:
              schema:
                type: string
    put:
      description: Changes the current time
      operationId: putTime
      requestBody:
        description: ISO time
        content:
          text/plain:
            schema:
              type: string
        required: true
      responses:
        '200':
          description: ''
  /time/zone:
    get:
      description: Retrieves the current time zone
      operationId: getTimeZone
      responses:
        '200':
          description: Zone ID
          content:
            text/plain:
              schema:
                type: string
    put:
      description: Changes the current time zone
      operationId: putTimeZone
      requestBody:
        description: Zone ID
        content:
          text/plain:
            schema:
              type: string
        required: true
      responses:
        '200':
          description: ''

Imagine if I had many more API endpoints than this to maintain and/or refactor 😳. To top it all off, tapir also provides a library to route a Swagger UI instance from within your http4s application to serve your API spec.

import sttp.tapir.swagger.http4s.SwaggerHttp4s
val swaggerRoute = new SwaggerHttp4s(openAPI.toYaml).routes // and then just add this to your http4s instance

Auto generated documentation and effectively plug and play web application coding based on Scala types, hopefully that's enough reason for you to consider trying out tapir on your next http4s application.

^1 Thanks to Andrew Davidson, who introduced me to tapir.

^2 tapir supports http4s, Akka HTTP, and Finatra.

^3 tapir supports other robust content types, for example, it has libraries for Circe JSON integrations.

"Tapir" by brainstorm1984 is licensed under CC BY-ND 2.0

Creating a 'paged' Vec in Rust

Sean Policarpio — Fri, 24 May 2019 06:36:49 +0000

In this post I'll show you some code I wrote for paginating over a Vec collection in Rust. I needed this for a CLI tool I wrote which was meant to display all the vector entries retrieved from a remote server. In most cases, I expected to receive a lot of results, so to display them in a terminal efficiently, I couldn't reasonably render them all. I decided I would page the results. However, I realised I also wanted the following:

I wanted to track a users selection of an element in the source vector, and,
I wanted to display a users selection on the page for display purposes (e.g. for highlighting their cursor selection).

Intuitively, these indexes would differ. I realised I wanted a data structure to answer the following question:

"What elements are on the same page as element n and what is its index there?".

My approach was to retain the full vector, but to only display one 'page' at a time. That 'page' would be the one where the current index and the users cursor would be sitting.

PagedVec takes a Vec<A> and internally creates a Vec<Vec<&A>> which references all the A's from the source vector. With the impl method page_for, you can optionally get:

a) the 'page' for the source vectors index, and
b) the modulus translated index for that 'page'.

For example:

let some_vec = vec![1, 2, 3, 4, 5, 6, 7, 8];
let paged = PagedVec::from(some_vec, 3);

let page_1 = paged.page_for(0); // Some(0, vec![1, 2, 3])
let page_1 = paged.page_for(1); // Some(1, vec![1, 2, 3])
let page_1 = paged.page_for(2); // Some(2, vec![1, 2, 3])

let page_2 = paged.page_for(3); // Some(0, vec![4, 5, 6])

let page_3 = paged.page_for(7); // Some(1, vec![7, 8])

My original implementation code is below👇. The real magic comes from the standard libs chunks method; it basically does the page partitioning.

pub struct PagedVec<'a, A: 'a> {
    page_length: usize,
    pages: Vec<Vec<&'a A>>,
}

impl<'a, A> PagedVec<'a, A> {
    pub fn from(vec: &'a Vec<A>, page_length: usize) -> PagedVec<'a, A> {
        PagedVec {
            page_length,
            pages: vec.chunks(page_length).map(|slice| {
                slice.iter().collect::<Vec<&'a A>>()
            }).collect::<Vec<Vec<&'a A>>>(),
        }
    }

    pub fn page_for(&'a self, index: usize) -> Option<(usize, &'a Vec<&'a A>)> {
        self.pages.get((index as f32 / self.page_length as f32).floor() as usize)
        .map(|page| {
            (
                index % self.page_length,
                page
            )
        })
    }
}

My favourite part of this implementation: using Rust's referencing and borrowing capabilities, I was able to do this with little additional heap allocation. Quite simply, all the 'pages' are basically pointers to the data that exist in the original vector. If the elements of the vector were huge, the 'pages' wouldn't duplicate the volume of data resident in memory. Kind of like a view in a relational database.

Soon after using this, I realised I could reduce my memory usage even more by not retaining pages I wasn't actually displaying. The decision to use the implementation below was a design choice based on factors particular to my CLI tool.

pub struct PagedVec<'a, A: 'a> {
    page_length: usize,
    vec: &'a Vec<A>,
}

impl<'a, A> PagedVec<'a, A> {
    pub fn from(vec: &'a Vec<A>, page_length: usize) -> PagedVec<'a, A> {
        PagedVec { page_length, vec }
    }

    pub fn page(&'a self, index: usize) -> Option<(usize, Vec<&'a A>)> {
        let mut paged = self.vec.chunks(self.page_length);
        let opt_page = paged.nth((index as f32 / self.page_length as f32)
          .floor() as usize);

        opt_page.map(|page| (index % self.page_length, page.iter().collect::<Vec<&'a A>>()))
    }
}

In addition to chunks, the magic trick here is the use of nth; rather than creating all the pages from the source vector, nth—in conjunction with the lazy evaluation of Rust iterators—allows me to only create the pages up to the selected index. No more wasting time or memory building pages I don't need! 😎

There could be cases where it would be better to use the original implementation to keep all the pages in memory. Which one is better? It depends on your use case 🤷‍♀️.

Update 3/6/2019
After sharing my implementation, some of the Rust devs in the community were able to point out some improvements in my code.

Reddit user 2brainz was able to share a generic and completely allocation free implementation of PagedVec, below:

use std::{cmp::min, marker::PhantomData};

pub struct Paged<'a, T, V> {
    vec: &'a V,
    page_length: usize,
    phantom: PhantomData<&'a T>,
}

impl<'a, T, V> Paged<'a, T, V>
where
    V: AsRef<[T]>,
{
    pub fn new(vec: &'a V, page_length: usize) -> Paged<'a, T, V> {
        Paged {
            vec,
            page_length,
            phantom: PhantomData,
        }
    }

    pub fn page(&self, index: usize) -> Option<(usize, &'a [T])> {
        let slice = self.vec.as_ref();
        let len = slice.len();

        if index < len {
            let page_index = index % self.page_length;
            let start = index - page_index;
            let end = min(len, start + self.page_length);

            slice.get(start..end).map(|s| (page_index, s))
        } else {
            None
        }
    }
}

The savings here are based on simply working with the slice that backs the provided Vec.

Also, I am ashamed to say I did forget some of the fallacies of using floating types in any programming language. A number of Rust devs reminded me that a) with numeric division, flooring is normally implicit, and b) floats are never precise. Cheers for pointing that out Shadow0133
and 2brainz.

Rust's Bindgen + Fuse in 2019

Sean Policarpio — Thu, 23 May 2019 03:56:28 +0000

I will quickly show how I got bindgen (https://rust-lang.github.io/rust-bindgen) to generate the bindings to Fuse (libfuse) with the current stable¹ release of Rust. By doing so, this should demonstrate how to bootstrap writing your own Fuse file system in Rust.

I do realise that there are some crates that already exist that aid in making Fuse drivers in Rust, but this was more or less an excuse to also try out bindgen, which I don't believe those existing libraries utilise.

I will be using:

rust and cargo 1.33¹
bindgen 0.48.1
the libfuse source headers from https://github.com/libfuse/libfuse
an OSX install of the Fuse dynamic library (via https://osxfuse.github.io/), but you should be able to use the libraries from Linux too (e.g. libfuse-dev from Debian/Ubuntu).

Assuming you already have cargo, you can get bindgen with cargo install bindgen.

Steps

New project

Init a project, like cargo init rust-bindgen-fuse.
We need the libfuse header files. Rather than download them manually or refer to them via a package manager (e.g. Debian/Ubuntu's libfuse-dev), I'll take them directly from Github. Within the project, you can create a git submodule, e.g.git submodule git@github.com:libfuse/libfuse.git.

Generating bindings

To use bindgen and to generate the code we want, we create a "wrapper" C header file to encapsulate the headers we want, including all the transitive dependent headers. For example, this is what I have in src/fuse_wrapper.h:

#define FUSE_USE_VERSION 31

#include "../libfuse/include/fuse.h"

FUSE_USE_VERSION is a macro constant that another header file (I believe fuse_common.h) required to be defined for API compatibility. fuse.h contains all the high-level API's an implementor needs to work with, so I started there.

To test bindgen out, you can run the following:

bindgen --distrust-clang-mangling src/fuse_wrapper.h

You should see some rust code fly-by. Those are the bindings to the libfuse library we will be linking to later. The reason I used --distrust-clang-mangling was due to some linker names that were generated by bindgen; the linker couldn't correctly reference the generated names for some reason, but this flag fixed that.

For this project, we want the generated bindings to be dynamic though, so they are built for whatever Rust toolchain is currently in use. To do that, we use a cargo build script—i.e. a file named build.rs in the project root. First, add bindgen to your Cargo.toml file:

[build-dependencies]
bindgen = "0.48.1"

Note that it's a build dependency, not a code dependency. This allows its use from the build script. As for the script, here's what I used based on the documentation online.

extern crate bindgen;

use std::env;
use std::path::PathBuf;

fn main() {
    println!("cargo:rustc-link-search=/usr/local/lib");
    println!("cargo:rustc-link-lib=osxfuse");

    let bindings = bindgen::Builder::default()
        .header("src/fuse_wrapper.h")
        .trust_clang_mangling(false) // disabled due to linker name problem
        .generate()
        .expect("Unable to generate bindings");

    // Write the bindings to the $OUT_DIR/bindings.rs file.
    let out_path = PathBuf::from(env::var("OUT_DIR").unwrap());
    bindings
        .write_to_file(out_path.join("bindings.rs"))
        .expect("Couldn't write bindings!");
}

By default, cargo will execute build.rs. Note that for now, I've hardcoded this to my current local environment, which is an OSX machine. The println!'s are sending commands to rustc to link my local dynamic library. On Ubuntu, I tested this with the dynamic library fuse after installing libfuse-dev via apt. I imagine it would be trivial to make this script smarter based on some environment variables, etc. The rest of this code calls bindgen programmatically and outputs the Rust code to a file that can be referenced in your own code.

Putting it all together

Finally, with the generated code, I can call the Fuse bindings. The general way Fuse works is you provide an implementation of a C struct called fuse_operations. This contains reference functions that are used as callbacks from the Fuse kernel module. In other words you inject the behaviour your file system should exhibit directly into a runtime, if you're so inclined to call it that.

Normally, after providing your operations, your userspace application ends and the local installation of Fuse takes over from there. Basically, it takes your injected code, mounts a file system, and your application exits. The following code—in src/main.rs—shows my first rudimentary attempt at calling Fuse.

#![allow(non_upper_case_globals)]
#![allow(non_camel_case_types)]
#![allow(non_snake_case)]
include!(concat!(env!("OUT_DIR"), "/bindings.rs"));

use std::env::args;
use std::env::args_os;
use std::ffi::CString;
use std::mem::size_of;
use std::os::unix::ffi::OsStrExt;
use std::os::raw::c_int;

fn main() {
    let ops: fuse_operations = fuse_operations {
        getattr: None,
        readlink: None,
        mknod: None,
        mkdir: None,
        unlink: None,
        rmdir: None,
        symlink: None,
        rename: None,
        link: None,
        chmod: None,
        chown: None,
        truncate: None,
        open: None,
        read: None,
        write: None,
        statfs: None,
        flush: None,
        release: None,
        fsync: None,
        setxattr: None,
        getxattr: None,
        listxattr: None,
        removexattr: None,
        opendir: None,
        readdir: None,
        releasedir: None,
        fsyncdir: None,
        init: None,
        destroy: None,
        access: None,
        create: None,
        lock: None,
        utimens: None,
        bmap: None,
        ioctl: None,
        poll: None,
        write_buf: None,
        read_buf: None,
        flock: None,
        fallocate: None,
        copy_file_range: None,
    };

    let argc: i32 = args_os().len() as i32;
    let args: Vec<CString> = args_os()
        .into_iter()
        .map(|arg| {
            arg.to_str()
                .and_then(|s| {
                    CString::new(s)
                        .map(|c_string| {
                            dbg!(&c_string);
                            c_string
                        })
                        .ok()
                })
                .expect("Expected valid arg input")
        })
        .collect();

    let mut argv: Vec<*const ::std::os::raw::c_char> =
        args.iter().map(|arg| arg.as_ptr()).collect();

    unsafe {
        fuse_main_real(
            argc,
            argv.as_mut_ptr() as *mut *mut ::std::os::raw::c_char,
            &ops,
            size_of::<fuse_operations>(),
            std::ptr::null_mut(),
        );
    }
}

As you can see, my file system does bupkis. I pretty much do four things:

Create my fuse_operations that does nothing
Count the number of args from the command-line (argc)
Turn all the UTF-8 args into C-strings (argv)
Pass all those things into fuse_main_real, which is the entry-point that calls the Fuse library to mount a path among other things.

Note all the old-school C pointers; bindgen can translate a lot of things into nice Rust (e.g. Option's in the fuse_operations struct), but you still need to do a lot of C-like stuff, like providing pointers to pointers of strings.

At the very beginning, you can see a macro, include!; that is pulling in all the generated bindings. Unfortunately, my IDE doesn't know how to handle that, so I didn't get much help trying to look through the generated code. As a trick, I did the following temporarily:

bindgen --distrust-clang-mangling src/fuse_wrapper.h > src/delete/mod.rs

Then, I modified the main.rs like so:

// include!(concat!(env!("OUT_DIR"), "/bindings.rs"));
mod delete;
use delete::*;

That way I could use my IDE to look through the generated code and reference the structs and methods. Uncommenting the include! and commenting the delete module allowed me to verify I was calling the right stuff.

Trying it out

Running:

cargo run -- --help

should give you the usage info output by Fuse:

usage: target/debug/rust-bindgen-fuse mountpoint [options]

general options:
    -o opt,[opt...]        mount options
    -h   --help            print help
    -V   --version         print version

FUSE options:
    -d   -o debug          enable debug output (implies -f)
    -f                     foreground operation
    -s                     disable multi-threaded operation

fuse: no mount point

You should be able to do this then:

cargo run -- /tmp/mount

And if you run mount, you should see an entry like this:

rust-bindgen-fuse@osxfuse0 on /private/tmp/mount (osxfuse, nodev, nosuid, synchronous, mounted by kdrakon)

Obviously, since we didn't implement anything, you'll see this after running ls /tmp/mount:

ls: /tmp/mount: Function not implemented

So how do we know what wasn't implemented? Fortunately Fuse has a debug-mode:

cargo run -- /tmp/mount -d

This will spit out all the callbacks to your fuse_operations, specifically, the ones that failed/were missing.

Next?

I'm going to decipher more of the Fuse operations and what are expected from them. Have fun!

References

Here are some helpful links I found to get over some of the bumps in the road I encountered.

Artwork "Rust" by Roberto Bicchierai is licensed under CC BY-NC 4.0

originally posted as a gist on 13/03/2019 ↩

How to use more than one Trait implementation while avoiding conflicts

Sean Policarpio — Tue, 21 May 2019 04:42:48 +0000

Problem

Say you initially have code like this:

struct DeserializeError(String);
trait Deserialize<T> {
    fn into_type(self) -> Result<T, DeserializeError>
}

struct Foo {}
impl Deserialize<Foo> for Vec<u8> {
    fn into_type(self) -> Result<Foo, DeserializeError> {
        unimplemented!() // Vec<u8> -> Foo
    }
}

struct Bar {}
impl Deserialize<Bar> for Vec<u8> {
    fn into_type(self) -> Result<Bar, DeserializeError> {
        unimplemented!() // Vec<u8> -> Bar
    }
}

and later down the road, you need to write a generic implementation of Deserialize that requires the use of Deserialize<Foo>, Deserialize<Bar>, and whatever other implementations exist in your code base.

The problem I encountered was when I initially tried to do the following:

struct Wrapper<T> { foo: Foo, t: T }

// first 4 bytes are Foo, the rest will make up T
impl<T> Deserialize<Wrapper<T>> for Vec<u8> {
    fn into_type(self) -> Result<Wrapper<T>, DeserializeError> {
        self[0..=3].to_vec().into_type().and_then(|foo: Foo| {
            self[4..].to_vec().into_type().map(|some_t: T| {
                Wrapper { foo, t: some_t }
            })
        })
    }
}

Unfortunately, you'll get the following error:

|     self[4..].to_vec().into_type().map(|some_t: T| {
|                                    ^^^^^^^^^^^^^^^^^^ the trait `Deserialize<T>` 
|     is not implemented for `std::vec::Vec<u8>`

The compiler wonderfully suggested the following advice though:

consider adding a `where std::vec::Vec<u8>: Deserialize<T>` bound

Ok, so let's try that:

impl<T> Deserialize<Wrapper<T>> for Vec<u8>
    where Vec<u8>: Deserialize<T> {
    fn into_type(self) -> Result<Wrapper<T>, DeserializeError> {
        self[0..=3].to_vec().into_type().and_then(|foo: Foo| {
            self[4..].to_vec().into_type().map(|some_t: T| {
                Wrapper { foo, t: some_t }
            })
        })
    }
}

But now we get the following compiler error:

   = note: expected type `Foo`
              found type `T`

Why? Because the where bound has stated that Vec<u8> in this scope must implement Deserialize<T>, and based on my best guess, has explicitly enforced only that implementation into scope as well. So my problem is that although I don't have conflicting implementations of the trait for Vec<u8>, I had conflicting use of the trait.

Solution

This took me awhile to find, but the solution was in the second edition of "The Rust Programming Language" book. In the Advanced Traits section, the key was the subsection "Fully Qualified Syntax for Disambiguation: Calling Methods with the Same Name".

Although this section was regarding explicitly stating which method to use from multiple traits with the same name (conflicting method names, not trait names), it did the trick here.

impl<T> Deserialize<Response<T>> for Vec<u8>
    where Vec<u8>: Deserialize<T> {
    fn into_type(self) -> Result<Wrapper<T>, DeserializeError> {
        Deserialize::<Foo>::into_type(self[0..=3].to_vec()).and_then(|foo: Foo| {
            Deserialize::<T>::into_type(self[4..].to_vec()).map(|some_t: T| {
                Wrapper { foo, t: some_t }
            })
        })
    }
}

I still retain the where bound so that the implementation is in scope by the compiler, but by using the fully qualified syntax—including the generic type—I'm telling the compiler exactly which implementations I actually want to use. And of course, it compiles. From the perspective of a Scala developer, this is kind of like temporarily skipping implicit class conversion and manually calling the implementation you want.

originally posted as a gist.

How to sequence in Rust

Sean Policarpio — Tue, 21 May 2019 04:19:25 +0000

Well, I wouldn't say it's exactly the same sequencing in terms of FP languages like Scala et al., but it does the job for the given collection types in Rust; invert something of type F<G<A>> into G<F<A>>.

Usually, we'd expect F and G as types that are iterable and applicative. In the case of Rust, what I seem to understand from the standard library is that it is expected that there exists a trait implementation—FromIterator—that should allow G to be treated like an applicative. Since collect() is already iterating over a collection of G<A>, it's straightforward for an implementation of FromIterator for F to turn the results (i.e. a collection of them) of the first FromIterator into a G<F<A>>—which is what it does.

let opt = vec![Some(1), Some(2), Some(3)];
let sequenced = opt.into_iter().collect::<Option<Vec<i32>>>();
print!("{:?}", sequenced); // Some([1, 2, 3])

let opt = vec![Some(1), None, Some(3)];
let sequenced = opt.into_iter().collect::<Option<Vec<i32>>>();
print!("{:?}", sequenced); // None

let result = vec![Ok(1), Ok(2), Ok(3)];
let sequenced = result.into_iter().collect::<Result<Vec<i32>, &str>>();
print!("{:?}", sequenced); // Ok([1, 2, 3])

let result = vec![Ok(1), Err("oops"), Ok(3)];
let sequenced = result.into_iter().collect::<Result<Vec<i32>, &str>>();
print!("{:?}", sequenced); // Err("oops")

let result = vec![Ok(1), Ok(2), Ok(3)];
let sequenced = result.into_iter().map(|opt_i| opt_i.map(|i| i + 1)).collect::<Result<Vec<i32>, &str>>();
print!("{:?}", sequenced); // Ok([2, 3, 4])

originally posted as a gist.