Forem: Trey Botard

Fluree is moving to Discord

Trey Botard — Tue, 04 Jan 2022 01:06:44 +0000

The Fluree community has seen substantial growth over the last year.
With the velocity of growth and the big goals we have for the community,
we need a new home.

Discord has become an ever-present tool for communication on the web. The Fluree community has been on Slack for the past several years and for the most part we have been satisfied by the platform. However, over the past year, we've seen significant growth and as we've grown, we've begun to run up against some of the shortcomings of Slack as a platform for an open community. As we imagined where we want to take the Fluree community going forward, it became harder and harder to see how we could modify our vision to fit within the bounds of Slack. We needed a new platform for our community. After investigating the alternatives, we found Discord to be an extremely attractive option with all of the benefits provided by Slack and a ton of new functionality to boot. We did some preliminary internal testing, then tested the waters with a couple weeks of beta testing (shout out to our Beta testers!) and found the experience to be even better than we had hoped.

So, we made the exciting decision to migrate our community on over.

Why Migrate?

No message limits

The primary limitation in Slack is the message limits. If you weren't aware, Slack only allows up to 10,000 messages per workspace. There are over 700 members in the public Fluree Slack and the amount of chat happening meant that we were only able to see messages within about a 6-month window. As our community grows, that window will only shrink and the knowledge shared within this space will be lost at an accelerated pace. Discord has no message limit.

Voice chat and screen share

Another big point of differentiation between the two platforms is the availability of voice chat and screen sharing. Discord has a ton of features around this which we will begin to take full advantage of. We want to enable our members to self-organize and work on things together using the Fluree Discord as needed. This will enable just that. Not to mention the functionality the Fluree team can use to host events and activities from with Discord. (We have some fun stuff planned for this year!)

Open by default

Another issue with Slack for an open community is the way channels are structured. Channels are hidden by default and require a member to be invited or actively search for new channels to join. This is not the way Discord is set up. All channels are public by default, which creates an easy path to exploration and cross-pollination, with good tooling for a user to manage notifications per channel or category as desired. We think this is better for an open community and are excited for this change in structure.

Better moderation tooling

Fortunately, this hasn't been an issue thus far, but moderation is a vital part of community management. Creating a safe space for all of our community members is extremely important to the Fluree team and frankly the tooling for moderation in Slack is practically non-existent. Discord provides functionality which will enable us to create the spaces we want and ensure a welcoming, helpful, and safe environment for all who choose to join us.

Timeline

Here's the plan for the migration:

Thurs. Jan 6th: Open Discord server to the public.
Mon. Jan 10th, and Fri. Jan 14: Fluree DevRel team will be available for questions in the Discord to assist with the new tool.
Mon. Jan. 31st: Close public Slack channels to new comments. Begin migrating private channels to shared channels where needed.
- We will migrate private support channels to shared Slack channels from our internal Slack workspace.
Thurs. Feb 3rd: February Office Hours to be held in the new Discord server!!

Of course, the team will be available for any questions in both Discord and Slack for the duration of the migration and are more than happy to assist where needed.

We are extremely excited for this new chapter of the Fluree community. There will be bumps in the road as we migrate such a large community, but we are confident this is the right choice, both for the long term health of the Fluree community and for Fluree itself.
Keep an eye on our Twitter in the coming weeks for an invite link to the new Discord server. If you're in the Fluree Slack, we'll
be making an announcement there as well.

We look forward to seeing you in the new Fluree Discord!

Fluree in Docker - A Single Instance

Trey Botard — Thu, 07 Oct 2021 21:25:12 +0000

Managing dependencies is a pain in the tuchus. Whether in the cloud, in a private data center, or just on your laptop, ensuring that you have the correct version of Java or Node or whatever other runtime works properly with both the application and the OS and all of the other dependencies can quickly turn into a nightmare. Docker solves this by bundling all of the dependencies together in a single image such that as long as the Docker runtime is deployed on the target infra, your app should function as desired. No muss, no fuss. 🥳

Let's walk through the set up necessary to get your Fluree backend up and running. (Deployment to a public cloud will be out of scope for this post). This will be the first of a 3 part series on working with Fluree in Docker. The next one will be about setting up a Transactor Group!

Getting Started

To begin with, you will need to have Docker on your machine. Please visit the Docker Getting Started Docs to get it set up, if you don't have it configured already.
Once Docker is installed and running you will need to grab the Fluree ledger image from DockerHub. The Docker CLI is a great tool for managing images on your machine. We will use the docker pull command to grab the fluree/ledger image. Open a terminal and run:

docker pull fluree/ledger:latest

At the time of writing, this will pull the 1.0.0 beta 15 ledger image from Docker Hub, which is the most up-to-date version of Fluree. Once the Fluree ledger 1.0.0 is released, beta releases will go under their own tag. At any time, you can pull a specific release, if that is needed. You would simply define the image to pull using the tag like this: fluree/ledger:1.0.0-beta14.
You can see all of the release tags for Fluree in our Github Repo.

Set up and run the container

Ok, you have the image, now what?
There are a couple of parameters which Fluree needs in order to start up correctly.
These can be passed in when starting the container using the docker run command.

You can simply start the image by calling docker run fluree/ledger, but this will not expose the ports you will need to have in order to use the HTTP endpoints. Internally, Fluree listens on port 8090 for the HTTP calls, so this port will need to be mapped to an exposed, public port on the container. This can be done with:

docker run -p 8090:8090 fluree/ledger

The ports correspond to ExternalPort:InternalPort. So, if you have 2 containers running, you will need to map the first of the two to a different port and leave the second number as-is because that is the port which Fluree is listening on. So for your second container, you would pass -p 8091:8090 or some other port number of your choosing.
This spins up Fluree inside of your container, you will see the default configs print to the terminal and some messages about the consensus and state of the server. You can see the container and some metadata by running docker ps in another terminal.

Naming your Container

By default, Docker will assign your container both a CONTAINER_ID and a NAME, which you can see via the docker ps command or in the Docker Desktop UI. The name will be 2 randomly assigned words, but you can name your container when you initialize it with the docker run command. This is useful because you can then use that name to restart your container, if you stop it for whatever reason. (At this point stopping the container will lose any data, but in a bit, we will connect the container's file system to a local directoy for data persistence 🙌). You can do this by adding the --name flag to the run command, like this:

docker run -p 8090:8090 --name ledger1 fluree/ledger

Now, when you run docker ps, the NAMES field will be ledger1 instead of a randomized pair of words.

One caveat to using the --name flag is that if you stop your container or kill the terminal, that name will still be associated with that container. So you may need to run docker rm ledger1 to remove the container, if you want to docker run with the name ledger1 again. (This applies to the randomly-assigned container names as well. They will exist on the machine until they are removed.)

Interacting with your Instance

Now that your ledger is up and running on your machine, there are a few ways for you to interact with the ledger.

Admin UI

Every instance of Fluree contains a small utility for working with the ledgers, called the AdminUI. This is a browser based UI maintained by the Fluree team for working with instances of Fluree, whether as a single instance or in a transactor group. Once you have your Fluree instance ready with the HTTP port exposed, you can start the Admin UI by going to http://localhost:[EXPOSED PORT] in your browser.
If you used the command above to start up your instance, use 8090 for the [EXPOSED PORT].

We have a YouTube video walkthrough of the AdminUI coming soon.

HTTP Calls

Fluree also contains an HTTP client which exposes several HTTP endpoints for programmatically working with Fluree. Any of these endpoints can be hit by calling http://localhost:8090/fdb/[endpoint] or if the endpoint is ledger specific http://localhost:8090/fdb/[network]/[db][endpoint].

At startup, there are no dbs on your instance, so it is recommended to start by using the /new-db endpoint first.

Create a Persistent Volume

Fluree is a ledger and database. Both of these things require persisting data, which Docker does by default, but we'll want to configure it ourselves. To do that, we will need to do what is called mounting a volume to the container. This volume will live on after the container is spun down or goes offline for some reason in the place where we told it to be.

Fluree already has a mechanism for writing to a file system. In fact, there is a customizable parameter which enables you to specify the path in the file system where you would like Fluree to write its data. You can see all of the configuration options in the docs. The config we are interested in now is fdb-storage-file-directory. By default, this is set to /var/lib/fluree whether Fluree is being run in a container or in a JVM directly on a server. This means we can use some functionality in the Docker CLI to bind a directory in our local filesystem to this directory inside the container. Docker run has a flag called --volume (aliased to -v) where you can specify the two directories to be bound together. It will look like this -v "[local/path/to/dir]:[container/path/to/dir]".

For my example, I will tell my container to bind to Users/treybotard/Projects/fluree-data/docker-blog/. To put this together with the run command from above; your command should look like this:

docker run -p 8090:8090 -v "/Users/treybotard/Projects/fluree-data/docker-blog/:/var/lib/fluree" fluree/ledger

Now that I've run this command, I can open up the AdminUI, add a new ledger to my Fluree instance, and when I path to my docker-blog folder, I will see 2 folders, group/ and ledger/. These are the folders Fluree uses to maintain raft logs (in group/) and the encrypted indexes (in ledger/). I'm not going to go into too much detail here about this structure. If you are interested in reading more about how Fluree maintains the indexes or how Fluree interacts with the file system you can read more in the docs.

Private key file

When using the --volume command to persist your data outside of the container, if a private key is not provided on start-up, Fluree will create a default private key. This key (whether provided or generated at startup) is used for a few access controls in developement mode. What this means is that if you want to reuse the same identity to access the persisted data, you will need to store your key outside of the container and pass it in when spinning up a fresh container. This can be done a few different ways, but for simplicity's sake, we'll just look at passing the private key into the container as an environment variable with -e. Using an environment variable requires passing in -e FDB_GROUP_PRIVATE_KEY=[PRIVATE_KEY] when starting the container.

Now our command looks like this:

docker run -p 8090:8090 -v "/Users/treybotard/Projects/fluree-data/Docker-blog/:/var/lib/fluree" --name ledger1 -e FDB_GROUP_PRIVATE_KEY=123456789 fluree/ledger:latest

Docker start and stop

One very convenient way to maintain my development environment is to use the --name flag on a container with a volume mounted on it. What this enables is the stop and start functionality in the Docker CLI. If i have a container specifically for a client project or a demo I am working on, I give it a meaningful name "docker-blog" and mount it to a volume stored next to the source code where I am working. I can then conveniently run docker start docker-blog when I am ready to work on the container. It knows which ports to use and can spin up the container exactly the way it was when I ran docker stop docker-blog when I stopped working on it before.

Other Convenience Commands and Flags

`--rm`

If you are just testing out some some ledgers and are making some throw-away ledgers, I recommend adding the --rm flag. Going to spin up a named container which you forgot to spin down, but getting an error that the container name already exists gets old quickly. This will remove the container when you either CTRL+C or send SIGINT to the container some other way.

`-d` and `attach`

If you have been spun up a Fluree container before, you will have seen that the output is relatively chatty. One easy way to not have to see that in your terminal is to pass the -d flag when starting up the container. This will run Fluree in a detached process and not print to the console. You can run docker attach [container name] to reattach to the console and see the logs in your terminal. If you didn't give the container a name, you can look it up with docker ps -a to see all the containers, including the ones which are stopped.

`docker exec`

Sometimes, it can be very useful to have a bash terminal inside a running container while Fluree is running. You can use the docker exec command to run a command inside a running container. One useful way which I use this is to run docker exec -it [container name] /bin/bash. This opens a bash terminal in the working directory of my container where Fluree is running.

It can also be useful to access the file system in a container with Fluree inside it, even when Fluree is not running. To startup a container without initializing Fluree, you need to use docker run, but override the --entrypoint. That looks like this:

docker run -it --entrypoint=bash fluree/ledger

This will start up a container and give you a bash terminal with which to explore.

With these commands, a large percentage of the functionality needed to work with Fluree locally can be achieved. Next week, we'll take a look at how to set up a Transactor Group (several Fluree instances networked together) in Docker using similar commands and Docker compose!

Time traveling with Fluree

Trey Botard — Thu, 01 Jul 2021 18:18:35 +0000

Intro

When you get down to it, if you are building an app with Fluree as the backend, it is simplest to think of Fluree as a database. This can be a useful way to think about working with Fluree, but by doing so there is a lot being left on the table. The unique combination of technologies which make Fluree what it is enable some extremely powerful functionality and unlock ways of working with data which are either uncommon or simply not possible with other data stores. Let's talk about what some of those are, how to use them, and what this type of functionality could enable in your Fluree-backed application.

Background

In addition to a graph database for querying data, Fluree is built with an immutable ledger as the backbone which holds the dataset. This part of Fluree is what enables some really interesting and particularly unique functionality. 'Immutable ledger' is one of those terms which I had to Google in order to understand when I started at Fluree, so let's break that down a bit.

Fluree associates related data elements, called subjects. Each subject has an _id which is used to correlate the attributes (called predicates) and the values of those predicates together to form the "facts" about that subject.

Fluree is based on an extended version of the W3C standard for RDF, which is where this notion of SPO (Subject, Predicate, Object) comes from.

You can think of it like a row in a db table with _id being the unique identifier for the row, predicates are the columns, and the values are the fields. Each field makes up a fact about the instance of data stored in that particular row in the db. For example, in a Dog table with a Breed column, each row corresponds to a unique Dog who is described by the attributes and fields. The same idea holds in Fluree. An _id groups the related predicates, which point to values, in order to make up the "facts" of that subject. So, the dog/breed predicate would point at an object, "french bulldog", for example. At the point in time when that fact was written to the ledger, that specific subject's breed was french bulldog.

Each of these facts are stored in an immutable data structure. Immutable means that those data structures are not available to be modified or changed in any way. Instead of simply changing a value or updating a "row" in the data, Fluree will make a new true fact in the ledger and associate it with the appropriate subject. If this is a value which is being "modified" then Fluree will make two new facts; one where the old fact is false and the second a new, true fact, both of these facts are then associated with the subject and written to the ledger.

This is part of the "extension" to RDF. Each flake contains a boolean which indicates whether it is true or has been falsified. You can read more about this in the flakes page in the docs.

This brings us to what a ledger is. You can think of a ledger as discrete units or "blocks" which contain the history of the data as it is transacted. These blocks are made of groups of immutable facts which are sent to an instance of Fluree. Each block is linked to the one which came before it so there is a chain of blocks from when the ledger was created to the current block. In Fluree, this chain is queryable, which means once some data has been transacted to Fluree you have the history of every data element in that data set!

Time travel

So, back to those powerful pieces of functionality I mentioned at the beginning.
There are two ways of querying the data which enable what we call time travel in the Fluree world. There are block queries and history queries, both unlock elements of Fluree which are only possible because of the immutable data structures and the ledger. Block queries enable querying the data state at specific moments in time and history queries allow you to get an overview of all of the modifications to a particular subject.

Why

We'll get into how each of these types of queries work, but first, why does this even matter?
One of the primary benefits to having this type of view into your data is the ability to correlate events with the data state at the time that event happened. For example, say you are tracking prices for flights and you want to see what effect the weather had on flight prices or which day of the week prices tended to be the cheapest. The sky's the limit for these types of analytical queries.
You also may want to enable your users to see the state of some piece of data when it was updated. I saw a comment on a LinkedIn post once and was pretty sure that the commenter worked for the company who's post he was commenting on, but his current job title was recently updated so I couldn't tell where he worked when the comment was added, only where he currently worked - the current state of the data.
This type of functionality can be useful in a wide range of circumstances or situations. Having a way to view not only the current state of the data (table stakes for any database), but a way to see the state of a piece of data at a specified time OR for a specified range of time, can be extremely useful. Fluree goes a step further though. When you are querying some data a point in time, you are also seeing all of the facts which were true at that time as well. This includes all of the relationships which existed at the time. This is something which is not possible in any other database or data store that I am aware of. You are able to query not only the historical values of something in your data but also all of the context associated with that data as well. That is huge.

Now, think about how you would go about making something like this in your db of choice. Building out a historical view of a table in a traditional database, whether in a relational or NoSQL db, is a large and expensive maintenance burden, the size of your db will explode because of data duplication without significant optimization, and querying these db tables or documents can become relatively complex; specifically what happens to references? Does the reference point to the current table or is there a way to manage the reference such that it points to the correct row in the historical table? What happens when you want to do a join to with another table? There isn't an expedient or simple way to do either of those things, to my knowledge. One or two other data stores enable historical views, but are not able to pull in all of the context and maintain relationships as well.

Putting it all together

Both of these operations are exposed via an API within a Fluree db instance. Simply passing a JSON to the /block or /history endpoint is all that is needed to query this type of data. Let's get into how to use each of these queries. I will be using the Fluree Query Language (FlureeQL), which is a JSON-based way to query the backend. Fluree also supports querying via GraphQL, SPARQL, SQL or calling these endpoints directly from Clojure, but we'll use FlureeQL to illustrate this functionality. If you want to read more about our query surfaces, check out the query pages for more details.

Block Queries

There are two ways to query a block in Fluree. You can either issue a query against the /block endpoint which returns the flakes in that particular block or range of blocks, or you can add a "block" key to a basic query issued to the /query endpoint. This basic query method of querying can, and probably will, pull in facts which were transacted to the ledger before the specified block. When you issue a regular query with a block key, you are issuing a query as if the specified block were the current block.
Each of these types of query is beneficial, and can be useful depending on how you need to view your data.

Let's start with a query issued to the /block endpoint. This type of query currently supports 2 keys:

  "block": number,
  "prettyPrint": boolean

"prettyPrint" is a boolean, which if true, prints the results in a pretty printed, aka styled format, for easier reading, as well as separating the asserted and retracted flakes into their own arrays in order to make them easier to parse. The "block" key is much more interesting. It can take a number, a string in the form of an ISO-8601 formatted date-time or duration, or an array which specifies a range of block for the query.

For example, to query a specific block:

{
  "block": 3
}

You can query via a time stamp. This will return the first block which was transacted before this timestamp. In other words, it will give you the facts which were true at that time.

{
  "block": "2017-11-14T20:59:36.097Z"
}

You can also use an ISO-8601 formatted duration:

{
  "block": "PT5M"
}

This will return the state of the data as of 5 minutes ago.

If you would like to query a range of blocks, you can pass an array containing the blocks you would like to see. This range is inclusive, meaning the data returned will include both blocks you put in the array.

{
  "block": [5, 18]
}

You can also pass an array with a single block which will specify a lower, also inclusive, block and return the facts from that block up to the current block.

Using the /block endpoint will return an array of flakes, each of which is a fact stored in Fluree at that block or range of blocks. While this is useful, it is probably more realistic that you would want to see a specific set of data using a normal query, but have the results returned as if they had been issued at some point in the past. This is also enabled in Fluree by issuing a query to the /query endpoint which contains the "block" key-value pair. This key expects the value to be structured in the same way as the examples above, with the value being one of a number, a formatted string, or an array of block numbers. So the main difference is that this type of query will pull in data which is not limited to a specific block, it returns data as if the query had been issued when that block was the current block. For example, if you had a Dog collection of subjects in your ledger, you could issue this query to see all of the dogs which had been transacted and not deleted as of block 7:

{
  "select": ["*"],
  "from": "Dog",
  "block": 7
}

To read more on querying blocks, check out the docs pages for block queries and querying with the block key

History Queries

The way a /history query is structured and issued is relatively similar to /block queries, but are fairly different in what results are returned. As I mentioned above, a history query returns all of the modifications to a subject. I like to think of a block query showing the breadth of the data at a specific time and the history query as looking down the timeline of a specific piece of data.
For example, if you had a customer in your dataset who has connections to other customers, you could see the history of that customer's connections from when they first joined your application up to the current block. If you wanted to see the connections that customer had at a specific block or over a range of blocks, that is possible, as is using the ISO-8601 date-times or durations.

You can build a /history query using FlureeQL in JSON the same way you would with a /block query. For example, if you know the subject's _id you can simply hit the /history endpoint like this:

{
  "history": 351843720888320
}

This query will return an array of objects, each object containing the block and t numbers for that block and an array of flakes for that subject.

Another option is to issue a history query with a block key in order to constrain the results of the query to a specific timeframe in your data. That looks like this:

{
  "history": 369435906932737,
  "block": 4
}

This query will return the flakes for this _id up to block 4. You can also use a block range or use the ISO-8601 formatted string similar to the /block queries.

Using a flake format is another way you can issue a history query. This means that you can use pieces of data to identify the subject you want to query. This works via the subject, predicate, object structure of a flake. You pass the elements you want to use to query in an array as the value of the "history" key in the query JSON. The array needs to be passed as ["subject", "predicate", "object"], but you do not have to use all 3 elements in the array for the query to resolve.

Please note that the order of these within the array is important and either a subject or a predicate is required.

For example, if you want to query for the history of all subjects matching the predicate object pair dog/breed "french bulldog" in your collection, you could query the ledger like this:

{
  "history": [
    null, 
    "dog/breed", 
    "french bulldog"
  ]
}

Another way this could be done is using either a subject _id with a predicate, or substitute a two-tuple which uniquely identifies a subject for the _id.
That would look like this:

{
  "history": [
    351843720888320, 
    "dog/favFoods"
  ]
}

or with a two-tuple

{
  "history": [
    ["dog/name" "Jacques"], 
    "dog/favFoods"
  ]
}

Both of these queries will return the history of the predicate "dog/favFoods" for the dog specified, with either the subject _id or the unique identifier of ["dog/name" "Jacques"] used to identify the subject you want to inspect.
Similar to the "/block" queries, a "/history" query can also accept a "prettyPrint" key-value pair. When true this will return the history of the subject or predicate as indicated, but will separate out the retracted and asserted flakes per block into their own arrays. That looks like this:

{
  "history": 351843720888320,
  "prettyPrint": true
}

which will return something in this type of structure:

{
  "4": {
    "asserted": [
      {
        "_id": 351843720888320,
        "dog/breed": "french bulldog"
      }
    ],
    "retracted": []
  }
}

In the return JSON, each block containing data which matches the query is its own labeled object containing a named array for asserted and retracted.

"showAuth"

There is one other extremely powerful way to use "/history" queries to audit the history of who transacted the data. You can issue a "showAuth" boolean key-value pair or an array of _auth/id or _auth subject _id's in order to filter the history query to specific auth record's transactions. Because each transaction is signed by a private key which is associated cryptographically with the _auth/id, every flake in Fluree contains a record of who issued that transaction. This is the way to view that data. It looks like this:

{
  "history": 351843720888320,
  "showAuth": true
}

This will return an array of block objects, each of which will contain a named array of "auth" which consists of the auth's subject _id and the "_auth/id of the individual (man or machine) which signed that block. Which will look something like this:

[
  {
    "block": 2,
    "flakes": [
      [ 17592186044436, 40, "dog", -3, true, null ],
      [ 17592186044437, 40, "cat", -3, true, null ],
      [ 17592186044438, 40, "ferret", -3, true, null ]
    ],
    "t": -3,
    "auth": [
      105553116266496,
      "TexTgp1zpMkxJq1nThrgwkU5dp9wzaXA7BX"
    ]
  }
]

For more information on how Fluree stores and interacts with identity and authorization, please take a look at the identity section in the docs.

Wrap it up

So that's how you can go about time traveling in Fluree. There are powerful tools which come out-of-the box which enable you to do things like query as of a specific moment in time, see how a subject evolved over time in your dataset, or get all of the data which was transacted by a specific auth record. You can read more about it in our docs site or if you would prefer to engage with our community, come join us on Slack.

For more detail about this subject, you can watch our Time and Immutability Webinar on YouTube:
This has video has a publicly available demo which you can review here:

fluree / time-webinar

Demo app which uses the create-react-app template for Fluree to embed a webworker with the application. This demo showcases functionality around issuing block and history queries.

Time and Immutability Webinar Demo

This is the repository used for the demo in the Time and Immutability Webinar.

Set up

To begin working with this demo app, you will need to have Fluree running locally on your machine For detailed instruction on getting Fluree installed, please visit the installation page on the docs site.

You will also need to have Node.js installed on your machine.

The data folder contains the seed data for using this application as it is shown in the webinar. To get the data loaded into your Fluree instance, follow these steps:

Open the Admin UI and create a ledger called time/webinar.
Using either the Admin UI or a REST client of your choosing (Postman, Insomnia, etc.) transact the files in the data/ folder, in order, to your ledger
- This will transact the schema
- The airports and tags for the statuses
- The flight.json files…

View on GitHub