Forem: Jesse S

Determine root cause of Aerospike stop-writes in a new simple command

Jesse S — Wed, 06 Sep 2023 16:58:36 +0000

Like a well-prepared community bracing for the impact of a natural disaster, databases must be equipped to face unforeseen challenges. In the world of databases, few things are as dreaded as running out of memory, a situation referred to as OOM. Enter Aerospike's savior: the stop-writes condition, a protective measure designed to avert the nightmarish scenario of OOM. It acts as a vigilant sentinel, ensuring your database stays far from the brink of unrecoverable states. Think of it as your emergency go-bag or the life vest under your seat, offering respite when you need it most, though you hope never to use it.

But let's face it, encountering the stop-writes condition can still be quite unsettling and determining the root cause challenging. This is why we're introducing the show stop-writes command. With this powerful tool at your fingertips, you'll swiftly navigate through stop-writes scenarios and get your database back on track. Let's delve into the world of this invaluable command and discover how it ensures the safety and stability of your precious data.

Stop-write configuration

Before we delve into asadm's newest command, let's explore all the different ways you can configure and, more importantly, trigger stop-writes. The configuration you choose depends on what you want to protect.

To restrict a set from going over a certain size or number of records:

To restrict a namespace:

There is one other way the server protects itself, but this feature is not configurable:

cluster_clock_skew_stop_writes_sec

As you can see, there are several ways to configure stop-writes, and as a result many ways stop-writes can occur.

Uh Oh! I'm in stop-writes!

Picture this: you find yourself in the unfortunate stop-writes state, perhaps noticing a surge in client write errors.

Or you see an alert in Grafana.

Or you see a cryptic logline that reads something like

Jul 25 2023 23:02:02 GMT: WARNING (nsup): (nsup.c:936) {test} breached stop-writes limit (sys-memory), sys-memory pct:78, memory sz:0 (0 + 0 + 0 + 0) limit:3865470566, disk avail-pct:100 used-pct:0

The Need for a tool (asadm)

To fix the issue, use our new show stop-writes command, available in Aerospike Tools package 8.4.0 (asadm 2.15.0) and later.

When we start asadm, a helpful warning message displays (added in asadm 2.17.0) letting you know that you are in stop-writes and should run show stop-writes for further analysis.

In the table, you'll spot some key details that give us valuable insights into the stop-writes condition. First, the test namespace triggered the stop-writes, and the culprit behind it is the configured stop-writes-sys-memory-pct, set at 72%. By comparing this parameter to the system_free_mem_pct metric, which is at 26%, we can see that the stop-writes were activated when the system reached 74% memory usage.

You've got a couple of options to tackle this situation: either increase the memory size allocated for the database or free up some space by removing unnecessary records. Meanwhile, while you work on the permanent fix, you can temporarily raise the configured threshold to get your Aerospike cluster back up and running. So let's do just that using asadm's manage config command.

And voilà! You are now out of stop-writes and can start working on a more permanent solution. Just don't treat the problem like dishes in your sink and leave it for next week.

Learn more about Aerospike Observability & Management

Learn how Aerospike is adopting open standards like YAML, Open Telemetry and how to find clarity in complex systems in our upcoming Webinar - September 6th, 2023.

The REST Gateway 2.0 Gets a Re-Model (formerly, the REST Client)

Jesse S — Tue, 15 Nov 2022 20:32:05 +0000

Source: Photo by benjamin lehman on Unsplash

The REST Gateway provides you with a well known interface to your Aerospike Database, and a Swiss army knife-like solution to a variety of architectural problems where you might not want to integrate a full fledged Aerospike client into every application.

Re-Name and Refresh

The Aerospike REST Client is now the REST Gateway 2.0, including a major overhaul. We've added support for Aerospike Database server 6.0 and server 6.1 features, as well as complete retooling of the Operations API. These new features come with some deprecations and breaking changes, which this post will cover.
Here are the new features and changes:

A new Operations API backed by swagger documentation. Deprecation of /v1/operation and /v1/execute endpoints.
Query support with query by partition, response pagination, and secondary index filtering. Deprecation of /v1/scan endpoint.
Batch Write, Batch Delete, and Batch UDF support. Rewrite of old /v1/batch API. This causes a breaking change.
The ability to read and write to nested Complex Data Types (CDT). Rewrite of old CTX API. This causes a breaking change with /v1/operation and /v1/execute endpoints.
Prometheus-compatible metrics for spring-boot, tomcat, and the resilience4j circuit breaker via the /actuator/prometheus endpoint.

Lastly, we renamed the REST Client to the REST Gateway for a few reasons. Most notably, the Gateway was never really a client but a piece of useful middleware that uses a client under the hood. So keep this in mind when searching the documentation.

Operations V2

Rest Gateway 2.0 marks the release of a new API which we think you will find easier to understand and more robust. The old operations API had its benefits but could be difficult to use in practice. The schema of each model was highly restricted, causing some bloat, and while It worked well for basic ops, it had problems with list and map operations and lacked support for Bit and HLL ops.
Every available Operation model has been redesigned and is now documented in our API documentation.

Query by Partition

Aerospike server 6.0 added the ability to query by partition and paginate queries. Before the release of Gateway 2.0, we only supported scans, now called Primary Index Queries (PI query), which could only scan one partition at a time before returning the provided number of maxRecords. Throughput and potential use cases were limited. Now, with query partition, we can query partitions concurrently while still allowing for pagination via a returned token. We also support Secondary Index Queries (SI query) by providing an optional query filter.

Note: /v1/query endpoints currently only provide a way to perform foreground PI and SI queries. Use /v1/execute for background queries.

/v1/query provides a few new endpoints:

/v1/query/<namespace> query an entire namespace
/v1/query/<namespace>/<set> query a set
/v1/query/<namespace>/<begin>/<count> query <count> sequential partitions of a namespace starting at <begin>
/v1/query/<namespace>/<set>/<begin>/<count> query <count> sequential partitions of a set starting at <begin>

Pagination

To use pagination, provide the maxRecord query parameter (default 10,000) to specify the desired page size. To receive a token to be used for your next page request, provide the getToken=true query parameter (default: false). A token will then be received in the response body that can be used in the next request.

Filtering

Another major benefit of queries is providing an optional filter on a sindex. Furthermore, with Aerospike Server 6.1, sindexs can now be applied to nested CDTs. The following filters are currently supported, and the schemas can be found in our API documentation.

QueryEqualsStringFilter
QueryEqualLongFilter
QueryRangeFilter
QueryContainsStringFilter
QueryContainsLongFilter
QueryGeoWithinPolygonFilter
QueryGeoWithinRadiusFilter
QueryGeoContainsPointFilter

New Batch API

Aerospike Server 6.0 added support for BatchWrite, BatchDelete, and BatchUDF commands to complement the existing BatchRead command. BatchWrite allows you to define a list of operations to perform on a record. These operations can contain both reads and writes. BatchDelete allows you to delete a record and filter based on generationPolicy or use a filter expression. BatchUDF allows you to run server-side User Defined Functions (UDFs) on a given record. BatchRead allows you to read specific bins from a record or CDTs using read operations.

Before Gateway 2.0, you had to use the /v1/batch endpoint to read different bins from different records, as opposed to the /v2/operate/read endpoint, which allows you to read the same bins from multiple records. The old /v1/batch endpoint also did not support reading from CDTs. To add support for BatchWrites, BatchDeletes, and BatchUDFs, we redesigned the body of the /v1/batch endpoint to support all four types rather than just a BatchRead. Another added benefit of the new API is that every Batch object type can specify its own unique policy.

You can take a look at the batch API documentation and schemas in our swagger documentation.

Note: Batch operations are not the same as multi-record transactions. Their execution is not guaranteed to be in the order provided.

Try it out!

Thank you for reading! If you need help or have problems, feel free to open an issue on GitHub or ask a question on the Aerospike Community Forum.

New Features Coming to Asadm — Aerospike’s Admin Tool

Jesse S — Tue, 23 Feb 2021 01:29:42 +0000

Photo by NeONBRAND on Unsplash

Aerospike Admin (Asadm)

Asadm used to be Aerospike Database’s primary tool for monitoring performance and configuration, but it now has the ability to do common administrative tasks. Funnily enough, until now, Aerospike users would need to use another tool, aql, to do everyday administrative tasks like create a new user or update a user-defined function (UDF). With the latest Aerospike Tools 5.1.0, asadm will have the ability to:

Manage user and roles
Add and remove UDFs
Create and delete secondary indexes (sindexes)

As well as some other convenient features I will touch on later.

A Suite of New manage Commands to Administer your Aerospike Database

Admin+> manage acl create user superwoman roles sys-admin
Enter password for new user superwoman:
Successfully created user superwoman.
Admin+>
Admin+> show users
~~~~~~Users (2021-02-18 20:49:01 UTC)~~~~~~~
      User|                            Roles
admin     |read-write, sys-admin, user-admin
superwoman|                        sys-admin
Number of rows: 2

To access the new feature set, there is a new manage command. The manage command, you guessed it, helps you manage your users, roles, sindexes, and UDFs. There are quite a few manage sub-commands, which I will cover later on with some examples, but from a top-level view, there is manage sindex, manage udfs, and manage acl (access control list). The new manage commands are also accompanied by a new show commands show users, show roles, show udfs, and show sindex. Together, the show and manage commands can be used to administer your Aerospike Database with ease.

A New Privileged Mode in Asadm

Admin> asinfo -v build
ERROR: User must be in privileged mode to issue "asinfo"
       commands.
       Type "enable" to enter privileged mode.
Admin> enable
Admin+> asinfo -v build
ubuntu:3000 (192.168.173.203) returned:
5.4.0.3

Unlike existing asadm commands that monitor Aerospike’s DB state, the manage commands mutate data in the database, which adds a level of risk not seen before in asadm. There is an exception: the asinfo command, which allows some unfettered access to the Aerospike Database. A new “privileged” mode has been created, allowing users to access any risky commands like asinfo and asadm’s new suite of manage commands. The privileged mode is intended to keep users from inadvertently making changes to the Aerospike Database. The enable command also changes the prompt from Admin> to Admin+> to signify the added risk.

Admin> enable --warn
Admin+> manage sindex delete age-index ns test
The secondary index age-index has 654 keys indexed.
Confirm that you want to proceed by typing x4b862, or 
cancel by typing anything else.
x4b862
Successfully deleted sindex age-index.
Admin+>

Privileged mode has another level of protection added by the --warn flag, which, when possible, tries to inform you of the implications of the command. In the example above, you can see the manage sindex delete command informs the user about the number of keys currently stored in the index and requires a random six-digit hex code to continue.

To exit privileged mode and be kept from issuing asinfo or manage commands, use the disable command.

Now let's look at some examples.

Before we begin

Using Aerospike Enterprise Edition (EE) with security enabled requires the data-admin credential for manage udf and manage sindex commands. Similarly, to use the manage acl commands you must have the user-admin credential.

Managing User-Defined Functions (UDF)

There are two commands available for UDF management, manage udfs add and manage udfs remove. First, let's add a UDF module to the Aerospike DB.

Admin> enable
Admin+> manage udfs add foo.lua path path/to/foo.lua
Successfully added UDF foo.lua.

Let’s verify with the new show udf command.

Admin+> show udfs
~~~~~~~~~~UDF Modules (2021-02-18 01:11:39 UTC)~~~~~~~~~~~
    Filename|                                    Hash|Type
abc.123     |dceaf7f1acddf1d6e12a1752d499d80cfadfc24b|LUA 
bar.lua     |591d2536acb21a329040beabfd9bfaf110d35c18|LUA 
foo.lua     |f6eaf2b22d8b29b3597ef1ad9113d0907425ecd0|LUA 
mymodule.lua|7fae110826972135a3c3b8a812d43243e7c7a23c|LUA 
test.lua    |9f78214b275e306596d8e7120103585f3ee6adcc|LUA 
Number of rows: 5

We can then enter privileged mode with the --warn flag and update a UDF module.

Admin+> disable
Admin> enable --warn
Admin+> manage udfs add bar.lua path test.lua
You are about to write over an existing UDF module.
Confirm that you want to proceed by typing 6a9cb8, or
cancel by typing anything else.
6a9cb8
Successfully added UDF bar.lua.

As you can see, asadm first checks if the UDF module already existed and warned that it would be overwritten by running the command. Lastly, let’s delete a UDF module.

Admin+> manage udfs remove foo.lua
You are about to remove a UDF module that may be in use.
Confirm that you want to proceed by typing 1629e6, or
cancel by typing anything else.
1629e6
Successfully removed UDF foo.lua.

Again, asadm first prompts you to make sure that you understand the risks of issuing a command to remove a UDF.

Photo by Martin Adams on Unsplash

Managing Secondary Indexes

There are two new commands available for sindex management. They are manage sindex create and manage sindex delete. Let’s assume we have some data in the namespace “bar” and the set “testset” with the basic structure below.

PK   | name  | age
------------------
key1 | Jack  | 26
key2 | Jill  | 20
key3 | Sean  | 26
key4 | Sally | 24

Now let’s create a numeric sindex on bin “age”.

Admin+> manage sindex create numeric age-index ns bar set testset bin age
Adding a secondary index will cause longer restart times.
Confirm that you want to proceed by typing 52bf49, or
cancel by typing anything else.
52bf49
Successfully created sindex age-index.

Notice the warning provided by asadm. Next, we verify using the new show sindex command.

Admin+> show sindex
~~~~Secondary Indexes (2021-02-18 01:52:46 UTC)~~~~
    Index|Namespace|    Set|Bin|    Bin|Index|State
     Name|         |       |   |   Type| Type|     
age-index|bar      |testset|age|NUMERIC|NONE |RW   
Number of rows: 1

Now let’s delete the sindex since we no longer need it, and it causes longer restart times.

Admin+> manage sinde delete age-index ns bar
The secondary index age-index has 10 keys indexed.
Confirm that you want to proceed by typing 5ec3e3, or
cancel by typing anything else.
5ec3e3
Successfully deleted sindex age-index.

And there you have it. You can now manage your sindexes without breaking a sweat!

Photo by Jason Dent on Unsplash

Managing the Access Control List (ACL)

Configuring your ACL is essential to keeping unwanted users out of your Enterprise Edition of Aerospike Database. In Aerospike EE, users are assigned roles, and roles are assigned privileges. Note that privileges are predefined for you and cannot be changed. You can read more about configuring access control in the Aerospike technical documentation.

To use the new manage acl commands we must first make sure security is enabled in our aerospike.conf file. The default location of the config file is /etc/aerospike/aerospike.conf. To log in with your credential run asadm -U <username> and you will be prompted for your password. If you are just getting started with your Aerospike DB then the default username and password is admin, admin (be sure to change your password using the new set-password command below). As stated previously, manage acl commands require the user-admin credential.

Now on to an example.

First, let’s create a user named Fred Rogers.

Admin> enable
Admin+> manage acl create user Mr-Rogers
Enter password for new user Mr-Rogers:
Successfully created user Mr-Rogers.

To verify that we created the user, let’s use the new show users command.

Admin+> show users
~~~~~~Users (2021-02-18 00:23:40 UTC)~~~~~~
     User|                            Roles
Mr-Rogers|                               --
admin    |read-write, sys-admin, user-admin
Number of rows: 2

And there he is, but he has no roles. Now let's create a few roles.

Admin+> manage acl create role Bad-Neighbor priv read ns test set testset
Successfully created role Bad-Neighbor.
Admin+>
Admin+> manage acl create role Good-Neighbor priv sys-admin
Successfully created role Good-Neighbor.

The first role is “Bad-Neighbor,” which has the read privilege but only for the namespace “test” and set “testset”. The second privilege is named “Good-Neighbor” and has the privilege sys-admin, which is globally applied.

Now let’s verify with the new show roles command.

Admin+> show roles
~Roles (2021-02-18 00:35:02 UTC)~
          Role|       Privileges
Bad-Neighbor  |read.test.testset
Good-Neighbor |        sys-admin
data-admin    |       data-admin
read          |             read
read-write    |       read-write
read-write-udf|   read-write-udf
sys-admin     |        sys-admin
user-admin    |       user-admin
write         |            write
Number of rows: 9

There they are! Now let's assign those roles to Mr-Rogers.

Admin+> manage acl grant user Mr-Rogers roles Good-Neighbor Bad-Neighbor
Successfully granted roles to user Mr-Rogers.
Admin+>
Admin+> show users
~~~~~~Users (2021-02-18 00:39:00 UTC)~~~~~~
     User|                            Roles
Mr-Rogers|        Bad-Neighbor, Good-Neighbor
admin    |read-write, sys-admin, user-admin
Number of rows: 2

But that doesn’t look right?!?! After all, Mr-Rogers was a good neighbor. Let’s revoke the role “Bad-Neighbor” from Mr-Rogers.

Admin+> manage acl revoke user Mr-Rogers roles Bad-Neighbor
Successfully revoked roles from user Mr-Rogers.
Admin+>
Admin+> show users
~~~~~~Users (2021-02-18 00:43:00 UTC)~~~~~~
     User|                            Roles
Mr-Rogers|                     Good-Neigbor
admin    |read-write, sys-admin, user-admin
Number of rows: 2

Ahhh, that’s better. Although, we still have a bad neighbor lurking around. Let’s restrict Bad-Neighbor's access even more by issuing the role an allowlist. That is, addresses Bad-Neighbor can connect from.

Admin+> manage acl allowlist role Bad-Neighbor allow 192.168.2.1
Successfully updated allowlist for role Bad-Neighbor.
Admin+> show roles
~~~~~~Roles (2021-02-18 00:47:34 UTC)~~~~~~~
          Role|       Privileges|  Allowlist
Bad-Neighbor  |read.test.testset|192.168.2.1
Good-Neighbor |        sys-admin|         --
data-admin    |       data-admin|         --
read          |             read|         --
read-write    |       read-write|         --
read-write-udf|   read-write-udf|         --
sys-admin     |        sys-admin|         --
user-admin    |       user-admin|         --
write         |            write|         --
Number of rows: 9

Now we have another column of allowlist that is returned by show roles. “Bad-Neighbor” will only be able to connect from the address 192.168.2.1.

Asadm is your new one-stop-shop

As you can see asadm is coming closer to be the singular tool required to monitor, administer, and configure your Aerospike Database. Now, you no longer are required to use aql when you need to manage users and roles, add and remove UDFs, and create and delete secondary indexes. There are also plans in the works to add more features to make interacting with your Aerospike Database easier than ever before, so please stay tuned!