Forem: HS

Micronaut Pulsar

HS — Sat, 11 Dec 2021 20:32:36 +0000

A while ago I worked with Apache Pulsar and had interest in Micronaut. At the time there was no Micronaut Pulsar integration so I decided to give it a try. It was fun to go through certain Micronaut things and understand how and what to do but more than that I finally could contribute to open source project. Graeme Rocher is quite a nice guy I have to say. I expected more problems regarding coding style or something but it turns out if the tools pass the test you're quite OK with the contribution.

So to try to make it a bit more public I decided to publish a simple blog post about it. You can find the repository here.

What does it do?

It just makes integration between Micronaut framework and Apache Pulsar library. In the background it still uses Apache Pulsar client library for Java. Yes, that means it does still have some reflection usage in the background but I figured it's better to use existing piece of code than to rewrite whole protocol implementation in Java. I did test it against native image using Graal tools so I did some effort into making Pulsar client Java library work with it. It's not the best case scenario but it works so I guess it's good enough until either their library is rewritten to not use reflections or help is provided with rewriting this implementation.

What's supported?

Just the "basics" so to speak. Configuring Apache Pulsar client with TLS, OAuth2, or just setting the local URL. Consumers and producers are the main focus so one could easily annotate classes/interfaces and their methods to consume/send messages. I did replace Pulsars object mapper with Micronaut one so you can rely on the same processing when using JSON, and thanks to Johnas Conrad it's now replaced with new JsonMapper instead of ObjectMapper.

Annotations are made for consumers and producers but readers can have just annotation for injections - they will be automatically created for each injection point.

Protobuf mapping is not tested so the whole module is treated as snapshot - beta. If someone would like to give it a go feel free to fork and play with it. Other than that String, byte, JSON are well tested and working even Kotlin should be fine with suspend marked methods.

I assume anyone interested in this module understand Pulsars approach so I will not detail types of consumers or other things. I will just mention that most of the things are configurable through annotation properties like type of subscription, single topic, regex topic, acknowledgment timeout, and such.

Plans for the future

I wanted to build multi-tenant module which would allow reader, consumer, and producer injection point to detect tenant in context and instantiate or reuse existing one. This however would assume that producers and consumers that are made with annotations require a list in application properties or someplace to boot those consumers and readers during runtime. Otherwise it's not possible given that tenant headers are passed in requests. This would mean that consumers (or producers) would not be instantiated until request is made that would make that tenant known to the application. I think this is not usable so didn't implement it but let me know if you think otherwise.

Because module is in testing phase and I have no idea would be it be used enough I'd rather wait for requests and see where should effort be put it than to make code for something that is not used at all.

So, I'm hoping that this post would make it a bit more public and check interest in this module. If there's any I can get some feedback or maybe even couple of pull requests :D.

Given that I don't use Pulsar or Micronaut in my daily job anymore I don't have too much insights on what could be useful or what is bad so please do feel free to make suggestions or any kind of input.

Promotion bots on dev.to

HS — Mon, 25 Jan 2021 21:20:54 +0000

Have some "members" abused the good nature of this community for allowing "Read more"? I can constantly see generic names like jane doe, john, bill... posting parts of blog posts and putting Read More at the end of it. It's probably the same person but I just started tagging and reporting them. If anyone spots this feel free to flag to admins as it's clearly a some sort of promotion bot "phishing" users to it's own page.

Update: just for fun I followed the link. It looks like dev.to got a makeover (not necessarily in a good way). I think that it's basically a rip-off so maybe someone just took Forem, customized it and now trying to push users there. Some people just want to watch the world burn.

Apache Pulsar OAuth2 dev setup with Docker and KeyCloak

HS — Mon, 28 Dec 2020 00:35:41 +0000

So I've been eager to use something easier to set up than generating those TLS certificates and such for development environment, yet still somewhat more secure than just hardcoded JWT inside a file in Pulsar VM (container in this case). If you're not familiar with JWT setup for Pulsar, it's not necessary, but you can think of it as lesser OAuth2 since they both rely mainly on verifying JWT but process for logging in with Brokers and Clients can be different.

KeyCloak setup

KeyCloak requires only client app to be set up for this. App needs to Service Accounts Enabled to be on which is basically client_credentials grant type in OAuth2. In order to see this option Access Type has to be set to "confidential".

Next thing to set it the "specialised" claim. This claim would be easy to set as realm role or client role but Apache Pulsar has issues reading complex types in JWT by default, and so to skip writing custom authentication resolver we must enforce claim inside JWT to be simple string. Problem is that role claims by default go into array even if specified as non-multivalued. So one could get something like

"role": "[admin]" or
"role": ["admin"]

but we need

"role": "admin"

for Pulsar to play nice with defaults.
which is unexpected given that it should be only 1 string. To skip this hassle go and set-up Hardcoded claim.

Token claim name can be role and value can be for this purpose superuser. These values are important for Pulsar setup so keep track of what was put in here. Pulsar will require you to set up role name for the superuser which is dedicated role for managing all Pulsar stuff. On the other hand other roles might be used but then you need to remember to setup tenant namespaces to be accessible by that role. This means one would have to manage that pulsar instance manually through REST or CLI or something else to setup tenant namespaces manageable by user role you wish to use by certain app.

Next, get the public key of the realm. Easy you might thing? Well no, again because of Pulsar. Go to Realm Settings and into Keys tab.

There you should find RS256 under active tab if all defaults were left as-is. If you wish to use different algorithm please also mark that as important and keep track of it as well as of the key value. If not then please continue and click on Public Key of that RS256 row.

You should get some Base64 text. It will look like gibberish of alphanumeric and some special characters.

Now copy that value and somehow convert it to bytes and store as file. Pulsar lib for java at 2.7 will try to read it as x509 which will fail as it needs to be decoded into bytes prior to this. Steps to do it are:

Copy base64 value
Decode it back to bytes using either something online or simply program your own mini script
Store bytes into file A Groovy example:

byte[] key = "MACakje21/adkjwp9qmk4231/ea\d;qwdq=="..decodeBase64()
File f = new File("yourfilename")
f.bytes = key

Why Groovy? Well it was fast to write and has all those .decode and .bytes and... you can execute it as script through IntelliJ - Tools -> Groovy Console. I was mainly using JVM things with pulsar and Python but Python fails to use KeyCloak settings because of some previous bug which was fixed for Java and C++ but apparently not for Python which should rely on C++ lib.

Pulsar image

After setting up KeyCloak and storing that key to some file we can build a customised image that uses OAuth2 in standalone mode.

Now create a file that you will later copy into image. Please visit Pulsar docs if you want more info. If not then just treat this as credentials file for OAuth2 client app so client ID and secret from KeyCloak (go to App then first tab contains ID and Credentials tab has the secret). Below is simple example

{
    "type": "client_credentials",
    "client_id": "pulsar",
    "client_secret": "dadada7a7a7-a7ad77ad7da-da7a7ad7ad77",
    "issuer_url": "https://host.docker.internal/auth/realms/test"
}

I named it oauth2.json. _Download files conf/standalone.conf and conf/client.conf from Pulsar repository. These files will be changed to configure Pulsar image to use Authentication.

Configure standalone.conf. Set properties in that file to something like in below example@

authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken
# here goes the role you've picked in case you want KeyCloak client app to behave in superuser or else just leave default
superUserRoles=superuser

#brokers need to login to other things so they must also be set up
brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2
#oauth2.json path should be set to wherever Dockerfile is copying it
brokerClientAuthenticationParameters={"issuerUrl": "https://host.docker.internal/auth/realms/test","privateKey": "/pulsar/oauth2.json","audience": "pulsar"}
#bytes from base64, file path must be as defined by Dockerfile
tokenPublicKey=file:///pulsar/oauth_public.key

This will tell brokers to use JSON passed as a parameter, extract file under privateKey and post it's content as payload to isssuerUrl. Thus generating the JWT and refreshing when necessary. brokerClientAuthenticationPlugin is different from the authenticationProviders because broker generates JWT every now and then by authenticating to KeyCloak while main provider will only verify that token is valid (non-expired and signature is good). tokenPublicKey will instruct which file to use to verify JWT signature. It's the same as for simple JWT setup without the OAuth2. What's different here is that we need public key from the server while JWT can have key pair generated for Pulsar only and used by it or symmetric key for the same purpose. OAuth2 should not share it's private keys thus symmetric key is not a good option. KeyCloak by default uses asymmetric so it helps quite a bit.

Alternatively, if KeyCloak is set to never expire tokens, one can be generated in advance for Pulsar brokers and then copied into Pulsar container. This could be used instead of AuthenticationOAuth2 and parameters for broker would be file path to JWT token file. I would not advice alternative approach as it forces you to use never-expiring tokens on all of your realm not just that particular app.

Don't forget to configure your CLI tools. These are needed to run, add tenants/namespaces..., test topics, and do any kind of administration through CLI.

authPlugin=org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2

authParams={"issuerUrl": "https://host.docker.internal/auth/realms/test","privateKey": "/pulsar/oauth2.json","audience": "pulsar"}

So same values are used as for broker configuration plugin and params.

And finally Dockerfile. I made something like this:

FROM apachepulsar/pulsar-standalone:2.7.0
WORKDIR /pulsar
#using workdir /pulsar we copy public key and it's path will be /pulsar/oauth_public.key and this value must be same as in properties for standalone
COPY oauth_public.key oauth_public.key
#note oauth2.json is being copied from into workdir /pulsar which is linked in brokerClientAuthenticationParameters
COPY oauth2.json oauth2.json
#client for CLI tools
COPY client.conf conf/client.conf
#
COPY standalone.conf conf/standalone.conf

If you used same naming then you should be good to go with this image. One note is that I'm relying on -standalone instead of just pulsar or pulsar-all. If you choose something else you might want to add start-up command at the end to run pulsar when you trigger container run from docker. Standalone version runs automatically but others I used didn't so optionally use at the end

CMD [ "/pulsar/bin/pulsar", "standalone"]

if you used other image as a base.

Build image and run

This should be sufficient to run it. I made something similar and tested it with CLI, Java, and Python clients where I've noticed that Python complains about .well-known/... which was exception I got when Java library was not fixed to respect path of Issuer URL. What was wrong with Java lib is that URL was stripped of path and only root part was used so if you have http://keycloak/realms/test it would only take http://keycloak/ so keep that in mind while testing.

Hope it goes well.

Automate Micronaut 2.1 Docker build with Azure DevOps / Pipelines

HS — Mon, 26 Oct 2020 20:16:14 +0000

Recently Micronaut 2.1 was published with yet another feature that makes us "end-developers" work even less. It's called "Gradle Docker Plugin" and you can check it out here. Micronaut has adopted this feature and Graeme (creator of Micronaut) posted this YouTube video explaining some features including how to use this Gradle plugin with Micronaut.

Prerequisites

I assume that you've setup Container Registry in Azure and have all connections set up properly to communicate between Azure DevOps and registry.
Also I hope you have a project with Micronaut 2.1 or greater version because this feature is only available starting from 2.1.0

Pipeline setup

My first reaction to this was "nice" and then I thought "Well how can use this to tag images with other than latest default tag while using Azure Pipelines". Thing is that in order to make a custom tag I did something like this

dockerBuild {
    images = ["myproject:$project.version","myproject:latest"]
}

but azure-pipelines.yaml will contain by default something like this as a task for deploying

- task: Docker@2
    displayName: Build and push an image to container registry
      inputs:
        command: buildAndPush
        repository: $(imageRepository)
        dockerfile: $(dockerfilePath)
        containerRegistry: $(dockerRegistryServiceConnection)
        tags: |
          $(tag)

but that $(tag) variable is hard-coded to read from something like '$(Build.BuildId)' which is not something very useful to tag your own images with because you probably want to track your own versions. Top of your pipeline yaml should have something like below config

variables:
  # Container registry service connection established during pipeline creation
  dockerRegistryServiceConnection: 'some-uuid-some-uuid'
  imageRepository: 'yourAzureImageRepository'
  containerRegistry: 'yourAzureIMageRepoistory.azurecr.io'
  dockerfilePath: '$(Build.SourcesDirectory)'
  tag: '$(Build.BuildId)'
  # Agent VM image name
  vmImageName: 'ubuntu-latest'

You can remove that tag: ... part, as we are trying to put our own info in there, and the dockerfilePath as we will build image with gradle so we don't need the Dockerfile.

Instead let's make a script that will extract version from our build.gradle file. Add a bash task before task buildDocker task. Something like this should give you version:

#!/usr/bin/env bash
version=$(grep '^version[[:space:]]*"\(.\)\+"$' build.gradle | awk '{print $2}')
echo "##vso[task.setvariable variable=version]$version"

then your task may look like this:

- task: Bash@3
    inputs:
      targetType: 'inline'
      script: |
        #!/usr/bin/env bash
        version=$(grep '^version[[:space:]]*"\(.\)\+"$' build.gradle | awk '{print $2} | sed 's/"//g'')
        echo "##vso[task.setvariable variable=version]$version"

Windows pipelines?

I made a script which should give you same result when your running your pipelines on Windows platform

$versionVal = Get-Content .\build.gradle | Select-String -Pattern '^version\s*\"(?<version>.+)\"$'  | % { $_.Matches[0].Groups["version"].Value }
Write-Host "##vso[task.setvariable variable=version]$version"

I'm not 100% sure that it works I just tested it in PowerShell and I got the version number printed out so I guess it works :D.

Maven?

Well just check this link and adjust it accordingly. This is also a way to get some credits to the person who posted that answer as I saw this answer while looking out how to set up variable in Azure Pipeline from bash.

Tag the image

Because image was built inside Azure pipelines we need to tag it to make push command know where to push the image. Easy way to do it on Linux pipeline is using bash.

    - task: Bash@3
      inputs:
        targetType: 'inline'
        script: 'docker tag $(imageRepository):$(tag) $(containerRegistry)/$(imageRepository):$(tag)'

Sometimes you want your image to be tagged with latest because a lot people will configure (or leave by default) triggers that get activated only on this tag. So let's say you want your development to be the latest version but main or master to be specific version. Below task will activate only if branch name was 'development'.

    - task: Bash@3
      inputs:
        targetType: 'inline'
        script: 'docker tag $(imageRepository):$(tag) $(containerRegistry)/$(imageRepository):latest'
      condition: eq(variables['Build.SourceBranch'], 'development')

You can read more about it here if you need to configure it differently

Push the image

- task: Docker@2
    displayName: Build and push an image to container registry
      inputs:
        command: push
        repository: $(imageRepository)
        containerRegistry: $(dockerRegistryServiceConnection)
        buildContext: $(containerRegistry)
        tags: |
          $(tag)
          latest

If you didn't tag the image with the latest tag the whole pipeline will fail as there is no conditional statement. If you don't need it remove it completely.

Latest tag (optional)

If you need latest tag conditional this line should help.

${{ if eq(variables['Build.SourceBranchName'], 'development') }}:
  tags: |
    $(tag)
    latest
${{ if ne(variables['Build.SourceBranchName'], 'development') }}:
  tags: |
    $(tag)

So task could look like this replacing branchNameGoesHeere with whatever your branch name should be:

- task: Docker@2
    displayName: Build and push an image to container registry
      inputs:
        command: push
        repository: $(imageRepository)
        containerRegistry: $(dockerRegistryServiceConnection)
        buildContext: $(containerRegistry)
        ${{ if eq(variables['Build.SourceBranchName'], 'development') }}:
          tags: |
            $(tag)
            latest
        ${{ if ne(variables['Build.SourceBranchName'], 'development') }}:
          tags: |
            $(tag)

This should be it hope it work :D

Updates:
I've switched the logic for tagging images with latest as in most cases "latest" version is not the one to deploy nor keep on master/main branches. Obvious reason should be that deploying new code on push should only happen in development/testing environments while there must be a "manual" trigger to fetch the latest TESTED version that works properly and deploy it. Also prior example didn't actually work when having multiple tags and was missing tagging part so it's added now.

Collaborative writing?

HS — Wed, 09 Sep 2020 19:19:38 +0000

It's more of a suggestion for a new feature than discussion but:

Would it be useful to have collaboration on a post like having 2 people write same article? It's not super complicated feature just like having someone invite others to edit their post before publishing. If post is edited by 1 user other should not be able to edit until first person is done. And both authors are shown in the list. Could be more than 2 people could be expanded to blocking only parts for editing, or collaboration on series...

Do you know any platform with similar goal as Dev.to that implements this?

Pulsar docker image on Azure

HS — Thu, 03 Sep 2020 11:45:36 +0000

I wanted to quick set up Apache Pulsar on Azure. Long time ago this didn't go well but I stumbled on Kesque helm chart. Thanks to them it was quite easy. However Microsoft tends to charge quite high for VMs in Azure Kubernetes Service. Now for a startup it gets too high and we don't need that high throughput nor that kind of stability. However if you ever worked with messaging systems like Pulsar, they love to fail if there's too much traffic and VMs are not high-end or they are low in quantity. Also, using storage with AKS would be too much of a price hit so using File Share with Azure Container Instance was best overall solution for us for development environment.

Apache Pulsar Docker image customisation for JWT

This step is something I wanted but it's not required to make it work on Azure nor anywhere else.

Apache Pulsar has a nice Docker image. However it does not come with predefined way of using JWT or some other security. In order to do so I've created custom image that changes /pulsar/conf/standalone.conf to use JWT. This would make it easier for automating development to use simple JWT. First I generated public and private key, then used private key to generate JWTs for superuser, and then stored superuser JWT in container so brokers can communicate. This also included adding such lines to standalone.conf. If you forget to set these they will make your Pulsar not list clusters let alone tenants, namespaces, and such. Example of lines required to be changed

# line number about 352
# Enable authentication
authenticationEnabled=true

# line number about 358
# Autentication provider name list, which is comma separated list of class names
authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken

#line number about 370
superUserRoles=superuser
#line number about 374-375
brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationToken
brokerClientAuthenticationParameters=file:///pulsar/superuser.jwt

# line 384
tokenPublicKey=file:///pulsar/my-public.key

Notice the difference between org.apache.pulsar.broker.authentication.AuthenticationProviderToken for authenticationProviders and org.apache.pulsar.client.impl.auth.AuthenticationToken for borkerAuth. Also tokenPublicKey must be set for Pulsar to be able to verify those JWTs

First time I put the provider as the client so it didn't work and took about 20 min to figure out :D.

I also modified client.conf

#line 35
authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationToken
#line 42
authParams=file:///pulsar/superuser.jwt

which is used by tools in bin folder, and websocket.conf same settings as in standalone (maybe different lines).

Next, the docker image. I used official docker image as base and added lines to copy .jwt and .conf files.

FROM apachepulsar/pulsar:2.6.1
WORKDIR /pulsar
COPY my-private.key my-private.key
COPY my-public.key my-public.key
COPY superuser.jwt superuser.jwt
COPY client.conf conf/client.conf
COPY websocket.conf conf/websocket.conf
COPY standalone.conf conf/standalone.conf
CMD [ "/pulsar/bin/pulsar", "standalone", "-nfw", "-nss" ]

Public and private key have been generated before and copied to local machine. You can find in Pulsar official docs how to do it.
Client, websocket, and standalone conf are modified to use authorisation and need to be copied. I couldn't get something like RUN sed -i "s|xxx|xxx|g" xxx.conf" to work - it would not modify files. I guess it is because it doesn't see the files when building an image and it actually runs command at that point. So I went for fallback method which is copy existing files, modify then push back.
And finally, strip startup of parameters as much as possible so start Pulsar with the image. Parameters -nfw -nss are to avoid using functions feature.

Azure Container Instance

Assuming you already know how to do it, first create File Share. We need it so pulsar can store /pulsar/data directory in it in to be able to keep data in cases of restarts, failures, and such. I still haven't figured out how to connect Blob (if possible) so I'm mounting File Shares.

At the time of writing, Azure didn't support Container Instances to be created with the mount through Web interface so back to terminal:

az container create --resource-group RC_NAME --name CONTAINER_NAME \
--image YOUR_DOCKER_IMAGE_REPO/IMAGE_TAG \
--registry-username IMAGE_REPO_USERNAME \
--registry-password IMAGE_REPO_PASSWORD \
--dns-name-label SUBDOMAIN_PART_OF_URL_YOU_WANT \
--ports 8080 6650 \
--azure-file-volume-account-name STORAGE_ACC_NAME \
--azure-file-volume-account-key STORAGE_ACC_KEY \
--azure-file-volume-share-name pulsar \
--azure-file-volume-mount-path /pulsar/data \
--cpu 4 --memory 4

If your reading this article, your probably fine with 4 core / 4 gig ram.

Mounting /pulsar container path would lead to errors and your container wouldn't start. If you need more than 1 path like /pulsar/conf and /pulsar/data you would need to go through -templates. Reason it fails on /pulsar mount is that, if you haven't guessed it on first look, Pulsar would look into empty file share and wouldn't be able to discover /pulsar/bin/pulsar script and thus report error that file is not found for latest line in Dockerfile.

Should work

After couple of seconds / minutes, you should see your container instance up and running.

If you hit any problems call Azure support team, they "always help" XD.

MongoDB cross DB update

HS — Mon, 03 Aug 2020 21:38:48 +0000

Why?

I found myself in funny situation:

Spring Data for MongoDB adds _class field to anything
I use interface as a property in some cases
Interface relies on @JsonSubTypes from Jackson so I can easily have anything in DB as that property and let Jackson and Spring do the rest
I refactored code changing package names
Why write about it here? I need somewhere to put this piece of code so I don't have to Google if I do the same mistake again.

The problem

First of all, problem wasn't in the fact that aggregating class had different package name. No, in fact it works with main class easily as it's linked to a collection and there's only 1 specific class used for that particular collection. Issue is with this dynamic property supporting many different implementations while Spring is unable to detect what to instantiate since the _class field contains incorrect value. I got something like this:

org.springframework.data.mapping.model.MappingInstantiationException: Failed to instantiate com.your.old.package.name using constructor NO_CONSTRUCTOR with arguments

I did spend a lot of time trying to guess what was updated and what to change in code and somehow, it clicked. I forgot how I remembered it but I know that I just went straight to DB and started changing some values. It's easy for 1 collection in 1 database you put in

db.someCollection.updateMany(...)

and viola. But I have multitenant system. And yes I do use DB(s) per customer. Good thing only Mongo was affected by this as I do have more than 1 DB system for each customer.

So I wanted to update all of this by just one single command (not line necessarily)

The solution

Minimal version 4.2 for updateMany() with $set option

db.adminCommand( { listDatabases: 1 } ).databases.map(x => x.name).forEach(dbn => {
    let dbc = db.getSiblingDB(dbn)
    dbc.geoJsonFeature.updateMany(
        {"dynamicProperty._class": {$regex : /.*redp\.GeoPolygon.*/}},
        {$set: { "dynamicProperty._class": "some.package.name.ImplementingClass"}}
    )
})

Let's break it down a bit

db.adminCommand( { listDatabases: 1 } ) will return object containing list of databases so adding .databases will list databases as simple objects.
map the name only
use the for each to loop through all names
db.getSiblingDB(dbn) will switch which DB is used
updateMany with $set because replaceAll and replaceOne are from 4.4 version

It's a bit nasty "solution" but if you need to update multi tenant system where you need something trivial and all tenants have same schema for a certain collection this is quick and dirty solution for it.

You can also combine it with

db.getCollectionNames().forEach(c => {
    db.getCollection(c)...
})

to go wild :D.

That's all

SDN/RX Cypher tips

HS — Tue, 19 May 2020 13:45:57 +0000

I spent some time working on a project using Neo4j and Spring with Kotlin. I decided to go with WebFlux as it's reactive/all that non-blocking stuff. Now as SDN/RX is fairly new (out of beta since April 2020 I think) there has been some confusion, given that I used OGM for previous stuff I unintentionally tried to combine two approaches and it didn't work as expected. Trying to adjust to new framework took a bit and I realised some features were not only changed but missing completely.

Embedded objects (like in OGM) problem

Embedded objects at the time of writting are not yet supported as in OGM. This means that nesting object in another one inside of the code will produce relationship rather than the embedded object. For status update follow this GitHub issue

One to one implicit relationship

As stated in previous point, if you have complex/custom types for properties in your class, those will be resolved as Node to Node relatioship.
Having embedded objects represented as relationships can make custom queries a bit hard if your going with annotation approach. So let's say you have

data class LanguageStuff(
    @Id @GeneratedValue val id: Long,
    val en: String,
    val ba: String
)
data class Stuff(
   @Id @GeneratedValue val id: Long,
   val serial: String,
   //This one will be automatically generated as related node
   val description: LanguageStuff,
   //and this one
   val name: LanguageStuff
)

Even though there's no @Relationship attribute the description property will be generated as separate Node in Neo4j and therefor, will be fetched automatically each time we use common methods of querying data like letting spring resolve method name and try to generate query or generating Cypher with tools provided by SDN/RX as Cypher DSL which you can read more about here.

Custom queries using annotations

This all works well when you only need default behaviour. If you need annotated method with query including custom Cypher it gets a bit tricky to spot some things.

I did put name property intentionally using the same type as description. This is due to missing detailed note about caching and how objects are resolved by SDN/RX. Given repository:

interface Stuffpository : ReactiveNeo4jRepository<Stuff,Long> {
    @Query("MATCH (s:Stuff) RETURN s")
    fun customCypherStuff()
}

You might forget about the fact that Kotlin needs ? if things can be nullable. And there you go with the exception of could not be mapped. If you need those values you would want to fetch name and description represented with LanguageStuff data class. If your lazy enough like me you would try something like @Query("MATCH p=(s:Stuff)-[*]->(:LanguageStuff) RETURN p") and of course this doesn't work because it would be too much magic to resolve paths, but it's fun to break stuff and boring to read the docs.

Cypher "spread" operator and join properties

But if you think a bit harder you might end up with something like:

MATCH (s:Stuff)
MATCH (s)-[*]->(desc:LanguageStuff)
MATCH (s)-[*]->(name:LanguageStuff)
RETURN s{.*,description:desc,name:name}

inside your @Query value. If you're reading this by accident or you are new to this and don't know much about Cypher, might be a good note to point out that s{.*} is something like spread operator .* where each property/attribute from given s will be put to new object {}. As s only contains id and serial as real properties and name and description are related nodes, only first two will be spread from main node. Second two are explicitly set. Again this doesn't get the result

SDN/RX "hidden" naming

Graphs are not tables. There's no "real" one to one relationship so mapping nodes to data class fails. Then we can use arrays which solves problem up to some point

MATCH (s:Stuff)
MATCH (s)-[*]->(desc:LanguageStuff)
MATCH (s)-[*]->(name:LanguageStuff)
RETURN s{.*,Stuff_DESCRIPTION_LanguageStuff:[desc],Stuff_NAME_LanguageStuff:[name]}

Why the ugly names? SDN/RX by default maps relationships in naming format of ClassName_CAPITAL LETTERS OF PROPPERTY NAME IF RELATIONSHIP NAME IS NOT SET_PropertyClassName. So SDN/RX actually maps the relationships and generates special names for them for embedded objects (for now!). And therefor, queries need to fetch array named with relationship type where SDN/RX kicks in and resolves where to map what BUT we face yet another problem!

SDN/RX auto generated ID problem

You get things mapped but you will notice that both name and the description have the same value.
My best guess is that SDN/RX does some caching by type and both properties are the same type, but also, both properties have ID generated in default manner as a Long. This means that ID is not stored as separate/special property in the Neo4j Node but you need to actually resolve it by using id(node). This may be done in the background for automated way of generating queries but when you write a custom one you need to remember to know how does the mapper work since you haven't put in any details like relationship name or such.
I'm unaware am I right about "caching" where it could be that it's using type aware Map but it does ignore rest of the objects of the same type. The main problem is that Id is not being fetched so we end up with:

MATCH (s:Stuff)
MATCH (s)-[*]->(desc:LanguageStuff)
MATCH (s)-[*]->(name:LanguageStuff)
RETURN s{.*,Stuff_DESCRIPTION_LanguageStuff:[desc{.en,.ba,__internalNeo4jId__: id(desc)}],Stuff_NAME_LanguageStuff:[name{.en,.ba,__internalNeo4jId__:id(name)}]}

__internalNeo4jId__ is name to use for IDs generated and used by Neo4j to tell SDN/RX to actually map it. This way SDN/RX finally maps correct values to correct places.

As said, I'm unaware how does wrongful mapping of different objects of the same type happens but it does for SDN/RX 1.0.1 version when using generated IDs by default method. I guess using custom generator will prevent this as ID would be stored as separate attributed and then mapped back again automatically when doing object graph mapping.

That's it for now

Help: Security per record - is it possible to make optimal solution ?

HS — Wed, 06 May 2020 21:31:18 +0000

So I have a case where data structure looks more like a graph so lets say (object)->(another one)->(yes, antoher)->(even more). And this is stored in graph DB but there's "piled" data which is stored in different engine as pile of things referenced by some filed to vertex/node in graph one.

My problem is that I need security "user per object" where each user can have different access rights, and no role will help in this case as most of the users have different access rights even when they work at same position. So this is what I would call access grant per record or per object.

Now if someone has access to particular object he automatically has access to parent objects. But for example lets say I have (A)->(B) and (A)->(C). If I can access B I should be able to get A but not necessarily C.

I do plan to integrate something like HashiCorp Consul or if there's another similar thing which is easy to use but I don't think this can help at all. I also do have SSO but as any other it mainly relies on roles and such.

BTW, system is multi tenant so adding a new record to a specific user should be visible only for that particular client users, and Admin should decide who get's to see what.

So couple of ways to do it:

Have enormous amount of custom queries which filter out data - slow, hard to maintain and possible to leak data when new features are added or changes are introduced.
Middleware that handles access per object only - also slow due to a lot of network calls like Client -> load balancer -> API -> DB -> API -> middleware -> DB of the middleware -> middleware -> API -> load balancer -> Client. This could be used with some form of in memory cache which would speed up thing to some extent.
Have in-memory filter where each record stored and modified by admin has it's access towards each user cached - extra expensive to have huge amounts of RAM for such a cache
Have SSO deal with it through roles - now I have no idea how would I automate adding roles for each new object inserted to the system, and why would I have so much roles. Users have single access for SSO, so the actual roles would have to spill all over place as there's things like user from one tenant can be added to access some of the data of another tenant...

Anyone have ideas on how to deal with this in optimal way? I really do need help, maybe some software for this I didn't know exits or using SSO or Consul properly somehow for this?

Outsourcing state of mind in software development

HS — Wed, 15 Apr 2020 13:24:55 +0000

Last ~8 months I started working in a non software development company. What it means is that I used to work for companies which only had interest in developing software for others (outsourcing) and therefor, my job was to read tickets/issues/whatever on JIRA or such software and put some code into app/service/web by copying previous code of the "wise men before me" and changing it just enough to fit the need of the current task. Gradually you start learning patterns as a demand for job position and start "copying" stuff from your head instead from previous code writers or at that point stackoverflow. Now you might think well that's the way to do it but by changing jobs and getting more and more interaction with other people on the project (besides developers or IT guys) and in the end, with clients requesting the project, I started to get in conflict with my bosses, system architect, and such people by just asking questions like "is this task 100% correct", "should this be in the system at all", "I didn't understand client the same way you did", "why must we use X technology and do it in months when we can switch to Z tech for this part of the task and do it in 3 days while having it super easy to change later without bringing down the system".

I learned a lot about copy pasting without thinking like following patterns, DDD as explained by others, using libraries and frameworks for every piece of tiny problem, and how I'm not good enough to ever develop such a solution or my code always needs to be reviewed by boss (usually developers that founded the company in my case) or some external boss like system architect from company outsourcing job in absence of workforce on theirside.

What I didn't learn is how to use my own head and actually talk to people once I spot the problem... Thing was they didn't learn how to do projects just how to brag about being "full stack tech dev ops sec architecutre TDD DDD certified..." you get the point right? Now last switch was from software developer used to getting orders to being called backend developer while doing system architecture, cost planning, DevOps, project management for backend, partial involvement into new feature planing (yes, not only client requests features but I had to think of some too), team management and such. And WOW. What a responsibility change. I found out guys with 15 years of experience (whereas I have a bit less than 7 as time of writting) have less idea about development than I do simply because they never got out of "outsourcing state of mind". They got the "guy" working here before, he will make tasks in JIRA, tell them what to do and they will just put in some code. Boy what kind of well written useless code have I seen, from completely missing the point of task to skipping crucial steps because "I'll get back to it I didn't have info" - where later one is mindset in terms of "this is something we can consider template, we just swap later" and this one was wrong on so many levels.

I'm not encouraging devs to fight their bosses, go against their will, question everything and refuse to do obviously stupid stuff. I'm just saying thinking can lead to a great product, if your supervisors disagree, you can shut up do your work and change your employer ASAP orrrrr start up your own company. No I didn't have b** to risk my non existing finances and bankrupt my wife and me but you might and you should. Just because I chickened out many times doesn't mean you should. My wife had ideas, I wanted to start a lot of stuff but never was I willing to leave paid job and live under tent to try to make my own goals. I might try at some point but this post is not about that.

Have you ever had such opinion on industry that it's discouraging usefull product development in favour of well written software that misses the point of feature requests or maybe even well written useless software?

I might be under heavy stress now, work too much, learning 6 things at one just to make ends meet in product, my employers might have no respect towards effort required to build such a huge system, but that doesn't mean I will quit nor does it mean its worse than before. In fact, this is the first company where I can see how useful can thing I'm building be. I don't think I'm skilled enough or smart enough for this project to make it's architecture and make it work well but no one accepted the job before me. I wasn't told what I was getting into, and in the end when we tried to find skilled architect and other developers, one refused and that's about it no application no nothing. So someone has to build this, someone has to try and make it work, so since I'm here it might as well be me.

However do you think you would accept average salary for this kind of job? Some people might refuse this position under statement "not enough paid" or "because they like their job". But some of them will brag about their skill proficiency, how much books they read, and how what kind of products they read, but both you and I know that some of those would not stand a chance without being told WHAT to do but just given feature requests and left to handle the whole project. Why am I saying this negative stuff? Well it turns out that a lot of devs I had contact with are just fine with being ordered around and have no intention of ever knowing what to do. Why is that so bad? Well it's great to be that developer but someone has to be in charge of the project, someone has to make tasks, think ahead and so on. I wish I wasn't that guy because I do not think that I'am ready (skill-full) to do so but since I'm already in it I gotta do my best. Important thing for me was to inform people "hey guys this is too much for me" and they were like "yeah we know, we're gonna try to find someone but keep on the work".

Any similar experience? Any such negative opinions about certain people that you crossed career-path with?

Hope I didn't come out as someone who just nags about stuff.

A dev on UX

HS — Wed, 08 Apr 2020 18:04:47 +0000

So I've been working with IntelliJ (sadly on Windows) and I've got so angry when once more shifting focus made me mess up code and build to fail in 3 seconds window.

What happens is that I click Run for Spring Boot application while having my Terminal in Intellij opened. For some reason it takes about 3 seconds to start the process and meanwhile I'm writing some commands like "git add ." where my lovely tool switches focus to some file, so I'm typing "git ad" to console and "d ." into code and Intellij switches to tab Run/Debug then to Build and shows crash log because my file contains "d ." in middle of the code.

Now I instantly reported this as a bug on their tracker because it reeeeeeeeaaaaally pi*e* me of. And they are quite good compared to others. Imagine starting installer, putting it into background, writing code and stuff, refactoring things in code with famous Ctrl + F6 and all of a sudden instead of hitting Enter/Return on "Rename" you hit it on installer window asking you to accept all for installing McAfee, toolbar for your non existing IE, some stuff from 95, spyware for government, cookies for your grandmother, child support for other peoples kids and so on.

Really? A lot of installers wont do this but why is it even possible to make such app that steals focus? Have people not been multitasking lately with 11293810 core leaving stuff to run in the background instead of "You must wait until other installers finish!" message from half-core CPU times??

Who thought every software is so special and important that they need to blast you as a pop-up before Opera or before ad-blockers on other browsers? Why oh why would anyone do this? Blinking light in the taskbar or notification popups in specific corner are not good enough for snoflware software so OS just allows it to torture me?

Dear UI/UX people please talk with other people that will actually use software, see their experience, stop assuming your building most important thing and just make everything so important that it steals your focus.

Now a lot of things are amazing, great, and surprise me how well are they done and thought through when it comes to UX but some of the stuff make me write these kinds of posts.

Apache Pulsar on AKS quick setup for development

HS — Sat, 28 Mar 2020 22:10:27 +0000

THANKS to guys at kafkaesque-io there's https://github.com/kafkaesque-io/pulsar-helm-chart that will make it much easier to setup.

If you love Azure or Microsoft skip this part

I dislike Azure due to a lot of time wasted on trying to configure it where at the end I would eventually give up and use whatever was possible or just give up and tell my employer that this is not possible on Azure.

Some examples include:

Impossible to setup MongoDB as docker image via Container Instance which would enable easy setup for dev server. Short: it's not possible to attach Azure file share so mongo can use it. It will create a couple of folders and give up as "permission denied". However neo4j passed it.
Network. I had great pain setting up Spring Boot and Micronaut Neo4j clients just to realise that App Services cannot go over 230 seconds of idle TCP but Azure default load balancer or server or whatever under these connections will not properly close connections. Instead, it will just drop it leaving things like SDN/RX, which rely on TCP communication proper closing, think that connection is still there. Therefor, you'll see 20 sec connection pending until your app gets a hang of it realises no Neo4j connection in pool actually works then drop it and create new. Acquire time has nothing to do with this as connections will be visible to clients and only way to detect it is to have transaction timeout where client knows that if X amount of time is not enough for transaction there's a connection problem. Easy fix was to use connection lifetime on client settings where connections have to end in 230 seconds or 3.8 minutes. Maybe 228s as I set it just to be sure to break existing connection in pool before it ends and gets pending state.
Anything you touch or do with it burns. Each software will come with instructions for AWS and a lot with GCP. A LOOOOOOOOOOOOOOT come without any info on Azure or special info saying trillions of steps on how to enable it somehow and some of the features are not available.

I really did consider quitting my job and changing career just to be safe from the hell Microsfot puts his Azure clients through. And not to mention Blob Storage hell and prices.

Preps

However, job is a job, depression goes away and you still need to do stuff. Latest was the Apache Pulsar. I needed to get this up and running just to keep on working and I'm blocking the whole team.

Good thing kafkaesque.io guys put together helm chart that has Azure included.

First of all I want to mention that you need at least 6 nodes in AKS pool with machines that have 2 CPU and 4GB RAM. Because of companies subscription and previous AKS setup I had to remove AKS and add a new one with 5 nodes of 2 cpus + 4GB ram. Good enough.

Docker to speed up generating tokens prior to deploy

I actually used docker to run my own standalone pulsar to generate keys upfront. If you do it like that generate all keys on docker. You can jump in to pulsar docker instance like

docker container exec -it <<pulsar container id>> /bin/bash

Than generate stuff like explained in kafkaesque.io git repo for helm chart for pulsar under Authorisation section. Too lazy to explore well:

bin/pulsar tokens create-key-pair --output-private-key my-private.key --output-public-key my-public.key
bin/pulsar tokens create --private-key file:///pulsar/my-private.key --subject admin >> admin.jwt
bin/pulsar tokens create --private-key file:///pulsar/my-private.key --subject superuser >> superuser.jwt
bin/pulsar tokens create --private-key file:///pulsar/my-private.key --subject websocket >> websocket.jwt
bin/pulsar tokens create --private-key file:///pulsar/my-private.key --subject proxy >> proxy.jwt

By default everything is setup by those guys to work with those names. So it's really important that filename for keys are my-private.key and my-public.key. This is assuming that you want to do as less as possible with configuring anything.

Next you want to have all of those files on your PC for easier use

docker cp <<your pulsar docker id>>:/pulsar/my-private.key my-private.key
docker cp <<your pulsar docker id>>:/pulsar/my-private.key my-private.key
docker cp <<your pulsar docker id>>:/pulsar/my-public.key my-public.key
docker cp <<your pulsar docker id>>:/pulsar/admin.jwt admin.jwt
docker cp <<your pulsar docker id>>:/pulsar/proxy.jwt proxy.jwt
docker cp <<your pulsar docker id>>:/pulsar/websocket.jwt websocket.jwt
docker cp <<your pulsar docker id>>:/pulsar/superuser.jwt superuser.jwt

Reason for "one by one file" is that you see all of the files used in here. You can have some shorthand command if you like.

Also I had plugin for Kubernets in Visual Studio Code which made it easy to check Azure AKS and right click on them then and choose Merge into Kubeconfig. Using docker this was super easy and then you can switch via docker to AKS. Just to see what I'm talking about:

Why? Well I'm too lazy to configure stuff so this was easy. Also on Windows docker will have tray icon with right click option Kubernetes where you can see your AKS in list after previous step and click on it to switch it very easy.

Push secrets to AKS

If you skipped Docker part please note here that it's important to have same names for files in secrets as secret names also. This is because it's already set up to work with these names and probably for development you won't care enough to change it so:

kubectl create secret generic token-public-key --from-file=my-public.key --namespace pulsar
kubectl create secret generic token-private-key --from-file=my-private.key --namespace pulsar
kubectl create secret generic token-admin --from-file=admin.jwt --namespace pulsar
kubectl create secret generic token-superuser --from-file=superuser.jwt --namespace pulsar
kubectl create secret generic token-websocket --from-file=websocket.jwt --namespace pulsar
kubectl create secret generic token-proxy --from-file=proxy.jwt --namespace pulsar

Installation - for development

For development means not storage. Why? Well it's too expensive.

If you want you can add your Container Registry from Azure first:

helm registry login <<yourazure>>.azurecr.io --username <<listed on azure>>--password <<so is your_pass>>

Next, add helm repo:

helm repo add kafkaesque https://helm.kafkaesque.io
helm repo update

Why update? Well I have some form of OCD when it comes to updates so I like to update stuff as soon as I add new repo. Hope that's fine.

If you want to enable Authorisation you need to add extra line at the top

enableTokenAuth: yes

to example development yaml so:

I also removed pulsar functions because I don't need it*

persistence: no
enableAntiAffinity: no
#this is missing in their repo file
enableTokenAuth: yes

zookeeper:
  resources:
    requests:
      memory: 512Mi
      cpu: 0.3 
  configData:
    PULSAR_MEM: "\"-Xms512m -Xmx512m -Dcom.sun.management.jmxremote -Djute.maxbuffer=10485760 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:+DisableExplicitGC -XX:+PerfDisableSharedMem -Dzookeeper.forceSync=no\""

bookkeeper:
  replicaCount: 2
  resources:
    requests:
      memory: 512Mi
      cpu: 0.3 
  configData:
    PULSAR_MEM: "\"-Xms512m -Xmx512m -XX:MaxDirectMemorySize=512m -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+UseG1GC -XX:MaxGCPauseMillis=10 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintHeapAtGC -verbosegc -XX:G1LogLevel=finest\""

broker:
  component: broker
  replicaCount: 1
  resources:
    requests:
      memory: 512Mi
      cpu: 0.3 
  configData:
    PULSAR_MEM: "\"-Xms512m -Xmx512m -XX:MaxDirectMemorySize=512m -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem\""

autoRecovery:
  resources:
    requests:
      memory: 1Gi
      cpu: 1

proxy:
  replicaCount: 1 
  resources:
    requests:
      memory: 512Mi
      cpu: 0.3 
  wsResources:
    requests:
      memory: 512Mi
      cpu: 0.3
  configData:
    PULSAR_MEM: "\"-Xms512m -Xmx512m -XX:MaxDirectMemorySize=512m\""

Now apply this as installation. A little heads up (for users on Windows and helm 3+ I think but may be all of you): Docs on previous github repo link do not use name for installing repo so add it like this:

helm install pulsar --namespace pulsar kafkaesque/pulsar --values 'C:\Users\<<YourUsername>>\Desktop\test_helm\dev_values.yml'

This means that first pulsar in the command is actually name of the installation inside of your AKS or whichever proper word is used instead of installation. Anyways it's missing at the time of writing in the repo example.

Wait a bit and do a quick check with "kubectl get pods". If everything is initialised you can use

kubectl expose service pulsar-proxy --type=LoadBalancer --name pulsar-exposed

This will get you and External IP for using pulsar outside of AKS. Use that IP with 6650 or 8080 and admin.jwt (I think this one is for basic use).

Good luck.