Forem: Jesse Phillips

Removing Physical disk from LVM

Jesse Phillips — Fri, 15 Sep 2023 02:09:51 +0000

Since I've obtained new fast NVME drives, and I'm trying to manage my disk allocation with LVM I have a desire to migrate my currently Linux installation to the new drives.

I already have some LVM data spanning a couple of disks, one of which needs to be removed. Luckily for me the data I needed to move could fit within the freespace on my other disk. This results in touching a number of LVM commands and strategy.

First I needed to create space within Logical Volume so data in the other Logical Volume can fit on the disk. This includes a step of shrinking the filesystem.

Now lets move the data off the Physical Volume.

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/logical_volume_manager_administration/online_relocation

Now we tell LVM to move the Volume Group off the physical disk.

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/4/html/cluster_logical_volume_manager/vg_remove_pv

Working with LVM requires remembering the layers and ensuring the each layer gets the needed attention. There are some protections to help ensure you've taken desired steps to save your data.

One of the things I learned is that the LVM tools provide arguments to perform the Filesystem sizing for you. --resizefs added to the lvreduce or lvextend commands performs the desired operations if possible. This was nice to realize because these commands also provide the ability to resize relative to the existing size.

I'm a little surprised that asking to vgreduce does not handle the pvmove (moving data off the physical volume).

Logical Volume Manager (lvm)

Jesse Phillips — Thu, 14 Sep 2023 04:23:52 +0000

I'm a fiddler when it comes to computers, but I'm also behind the times so new Linux utilities like ip still trip me up. One of the areas I'm behind is disk management. I made some progress on GPT partitions and what is happening with UEFI. But LVM is currently on my new things list.

LVM is disk management above the partition table, but lower than the filesystem. One of the main benefits is that you can expand storage on a mount point through additional disks.

This capability is really nice but has implications when dealing with different storage mediums. I have ordered M.2 drivers and am thinking this will be where games and OS live. I can try to live within these 500GB but I expect that won't be enough and I have other desire for some backup space, movies/photos, etc. To get into the concern let me explain the LVM architecture more.

There are three primary concepts.

physical drive/partition
group
logical drive/partition

The Volume Groups (VG) owns a disk partition and holds logical partitions, like the Master boot Record (MBR) does for a disk. Unlike the MBR the group can manage space across physical disks. The VG can now allocate and resize Logical Volumes (LV) and the disk partition cannot be owned by multiple Volume Groups.

If I put all my storage under a single Volume Group then I can easily assign space to where it is needed and I don't need to allocate all of it upfront. Doing so means I also lose control of what data is stored on which drives (my OS could be moved to a slower disk). Thus I'm thinking through what types of control I desire for my storage management.

JSON in Different Languages

Jesse Phillips — Wed, 19 Oct 2022 17:43:31 +0000

JSON is a data format that uses Javascript notation to layout the structure of the data. But Javascript isn't the only language that can represent these structures in code.

// Javascript
{
  "firstitem" : "first data", 
  "list" : [
    "bold",
    "italics"
  ]
}

I'm utilizing the two main structures, generally referred to as 'object' {} and 'list' [] because that is how Javascript talks of them. Most languages can represent these in one or more ways. These structures don't produce JSON, but libraries (beyond this article) can make the translation.

// Lua
{
  firstitem = "first data", 
  list = {
    "bold",
    "italics"
 } 
}

Lua is almost the same with just object {} notation even for lists. The equal rather than colon is used for association. We also don't quote the field names, though Lua does allow spaces like JSON, ["first item"] = "first data".

// Python
{
  'firstitem' : 'first data', 
  'list' : [
    'bold',
    'italics'
  ]
}

Python looks really close, but string literals are single not double quote. We can also use tuples to build your list.

// Python
{
  'firstitem' : 'first data', 
  'list' : (
    'bold',
    'italics'
   )
}

This makes the list syntax different, but you'll get the needed output.

// c#
public class Data
{
  public string firstitem { get; set; }
  public List<string> list { get; set; }
}

C# is a statically typed language so we define a type that matches the structure. However C# has many different features and we can get away with much more.

// c#
new {
  firstitem = "first data", 
  list = new List<string> {
    "bold",
    "italics",
 } 
}

// c#
Dictionary<string, object> data = new {
  { "firstitem", "first data" },
  { "list", new List<string> {
    "bold", "italics",
  }},
};

This one is a cheat. The specification of object just places the data in to the libraries "general" representation, for example Newtonsoft's Json library would use a JObject, JArray for the type you could cast to.

// D
public class Data
{
  public string firstitem;
  public string[] list;
}

// D
new class {
  auto firstitem = "first data";
  auto list = [
    "bold",
    "italics"
  ];
}

// D
Variant[string] data = [
  "firstitem" : Variant("first data"),
  "list" : Variant([
    "bold",
    "italics",
 ])
];

Unlike in C# where all types have an object representation, there is not parent for all types with Auto-Boxing. D does however provide a type, Variant, which can hold other types and that needs to be explicitly created.

Working with Dolt Version Control Database

Jesse Phillips — Sun, 16 Oct 2022 19:59:49 +0000

I'm currently exploring the idea of building an application using Dolt and Dlang.

I'd been following along with the getting started docs:

And the key thing I missed was the relationship between the server and the Dolt database. Dolt provides a command-line tool where you can initialize a database in any folder, just like with git.

Utilizing the SQL interface requires that the Server be started with the database directory as the working directory. This can be as simple as:

$ mkdir newdb
$ cd newdb
$ dolt init
$ dolt sql-server

Or as complicated as

$ sudo useradd -r -m -d /var/lib/doltdb dolt
$ cd /var/lib/doltdb
$ sudo -u dolt dolt config --global --add user.email doltServer@company.com
$ sudo -u dolt dolt config --global --add user.name "Dolt Server Account"
$ cd /var/lib/doltdb
$ sudo -u dolt mkdir -p databases/my_db
$ cd databases/my_db
$ sudo -u dolt dolt init
$ sudo -u dolt /usr/local/bin/dolt sql-server -u root

I started with the second one here which is really about standing up the machine as a Database Server, The first option would have been perfect for my personal usage.

Once the server is up and running I could build a simple D application to connect:

$ dub init
$ sudo apt install default-libmysqlclient-dev
$ dub add arsd-official:mssql
$ gvim source/app.d

import arsd.mysql;

void main() {
    auto db = new MySql("127.0.0.1", "root", null, "getting_started");
}

The connection needed to use the loopback IP as it would not work specifying localhost. It will launch the server on 3306 which is default SQL, but it can be changed with -P

$ dolt sql-server -P 63444

new MySql("127.0.0.1", "root", null, "getting_started", 63444);

Enjoy.

You're not Making Test Automation Mistakes

Jesse Phillips — Wed, 14 Sep 2022 04:49:51 +0000

This topic comes up after reading

https://blog.testproject.io/2020/10/06/common-mistakes-in-test-automation/

I attempted to respond as a comment but it did not make it through review.

Namely I take issue with "Writing WebDriver API Methods in Tests/Page Objects" as being an issue.

The premise of the issue "Having WebDriver methods in tests is a really bad idea since if there are any changes in Selenium WebDrivers API or different versions of the library"

Wrapping your third party libraries can bring the expressed benefits, but I would claim that only happens when you can create an abstraction for your particular use. Let me put it another way.

The library writer is providing an API to control the inner workings. This API is going to expose all sorts of functionality. When writing a wrapping class two things are likely to happen, you expose all of the same functionality (possibly with the same function signatures) or you cut functionality and provide a simpler interface. You will not know how the public API for selenium will be broken so even your trimmed API could need to be modified to accommodate those changes.

An abstraction on the other hand relies not on the browser interaction but the application and that is what you page objects are providing.

Maintain Internal API During a Refactor

Jesse Phillips — Thu, 14 Oct 2021 03:18:46 +0000

I went down the wrong path recently. This topic relates to internal APIs and to discuss this I must ensure I define this.

When performing a refactor, the idea is that you can make any changes you want as long as the external contract remains the same. The internal components which talk to each other are all fair game when performing a refactor.

I have been in a position where these internal APIs were also like external APIs in that multiple people use these APIs to deliver their functionality, but we all own the maintenance and updates.

This meant they were treated more like external APIs. This is a pain to do and it feels like so much more work.

Well I found myself positioned were I thought the API was self contained enough I could just throw it out and reimplement. I think that is still the case but what a mistake.

See the biggest benefit to supporting the old API while doing a refactor is that your code still compiles and your tests still pass.

Dynamic languages can avoid this as code you're not running does not need to work to test the code you're working on. I could not handle needing to run all the tests to find issues the compiler tells me about right away.

When I continue to maintain the old API I could technically merge in progress into mainline. Sure that would create some confusion, but it still makes it a possibility.

Rebase and Merge Don't Mix

Jesse Phillips — Wed, 04 Aug 2021 04:20:44 +0000

When working with git it is possible to utilize both rebase and merge functionality. These two can live harmoniously within the same repository history, but things get messing if you try to mix them together.

This is a mistake made when working with remote repositories and local repositories while attempting to follow a rebase type workflow.

Remote
  M---R---Q
 /
A---B---C

Local
  M---R---Q (foo)
 /
A---B---C (origin/main)

Here I show that Local and Remote are in the same state. If you try to do any sync related to the remote and local git won't take any action, "Already up to date."

git switch foo
git rebase origin/main

Local
  M---R---Q (origin/foo)
 /        M'---R'---Q' (foo)
/        /
A---B---C (origin/main)

Now I'm going to work off the local repository since that is an accurate depiction of what git works with.

In this case I have rebased the branch foo onto the main branch, new commits are created, but the remote still has the same old commits.

git pull origin

Local
     M-----R----Q (origin/foo)
   /             \
  /               \
 /        M'-R'-Q'-Merge (foo)
/        /
A---B---C (origin/main)

Unless you have changed your configurations with

git config --global pull.rebase true

The use of pull will actually perform a merge for the remote branch foo. This is by no means desired. It results in merge conflicts and messy history with commits which duplicate each other, it is excessively bad when you've attempted additional commits on the local branch and you just want to get them out to the remote.

git push --force-with-lease origin foo

This is the most correct way to work in a rebase workflow. You own the branch and you can rewrite history and force push the changes out.

In general you should only be using rebase within a branch you own. However that does not mean that you can't collaborate within a rebase branch. Most optimally you would want to transfer ownership and only when you own the branch do you do any work on it.

When you take ownership of a branch:

git fetch origin foo
git reset --hard origin/foo

However git can handle commit divergence if the commit content has not been modified (--fixup, --squash).

Local
  M---R---Z (origin/foo)
 /        M'-R'-Q (foo)
/        /
A---B---C (origin/main)

Going back a little bit, lets say the remote had Z added and locally Q was created. You've rebased to main but don't have the Z commit. Sure we could:

git cherry-pick origin/foo

And there are probably even more fancy commands to handle multiple commits desired. But here is just one simple way to get the commits you want and still end up on main

git switch foo
git rebase origin/foo

Local
   M---R---Z (origin/foo)
  /         \
 /           Q' (foo)
/
A---B---C (origin/main)

After moving onto the remote updates, bring it back to main.

git rebase origin/main

Local
  M---R---Z (origin/foo)
 /        M'-R'-Z'-Q' (foo)
/        /
A---B---C (origin/main)

Now when the branch is force pushed, the changes from both sides still remain within the history and everything is linear.

Stop Using Short Hand Autocomplete

Jesse Phillips — Thu, 17 Jun 2021 03:33:15 +0000

This is just a simple plea to IDE builders.

Visual Studio provides quick help to generate code. For example if you inherent from an interface the IDE can generate methods which throw a not implemented exception.

This generative auto complete is there to get you started to build the implementation.

Well C# provides a short hand => operator to implement short method bodies. Great feature.

public class M : IFoo
{
    public string Bar() => throw new NotImplementedException();
}

The point of this short hand is so I don't need to write down so much for a simple return.

The point of code generating is so I don't need to write down all the expected structure. If you generate the simple thing, you aren't doing your job. I have to convert all those arrows into bodies so I can actually implement the methods rather then claim I didn't implement them.

Cheatsheet for Git Rebase

Jesse Phillips — Sat, 17 Apr 2021 04:13:59 +0000

I don't really use a bunch of cheatsheets, at least not in the conventional sense. Historically I recall having an ascii chart and dvorak keyboard. Yes there were others for school like imperial and metric conversion.

Though maybe their is something to this. Usually I will do a search for what I want to do and possibly pull up the link I visited last time. This can have the benefit that I my hit some updated information.

The world is quite different with information just one accurate search for the the thing you may not know you wanted. I also can get a little tired of searching for the same thing but forgetting which was the best answer.

This is my first creation, it isn't for me. I did this because I looked at a really nice cheatsheet for Git which had things I don't use or deliberately avoid. So I thought I would try my hand at the things I use almost daily.

https://cheatography.com/jessekphillips/cheat-sheets/git-rebase-workflow/

I really wanted stay away from creating groups which were focused on a command, I wanted to draw attention to a task. I also chose my tasks in such a manner that I could show that even though git has a reputation for inconsistent commands, today it really does provide some consistency.

So now that I created something for others maybe I can do something I'll use myself.

Also hidden within the sheet is my personal choices much has been explained here on dev.

Delete Your Master

Jesse Phillips ・ Apr 8 '19

#git

Scratch Diff Using Vim

Jesse Phillips — Sat, 10 Apr 2021 02:29:04 +0000

First as I have not seen this term utilized let me define what I mean by a "scratch diff." I have chosen this in reference to a "scratch pad." The idea is that you have some text and would like clarity in what is different between them, but you don't need to make a big deal out of the comparison.

Previously I have done this with creating two files and placing the text desired in both of them. Getting the filesystem involved at times just seems like too much for the desired diff. But an internet diff is also not acceptable because at times I need to manipulate the formatting/structure to get a decent comparison.

Well I finally took some time to learn the vim approach to making a scratch diff and isn't too bad.

After opening Vim

:vnew
:diffthis
<ctrl+w><ctrl+w>
:diffthis

Static Typing or Typescript

Jesse Phillips — Sun, 14 Feb 2021 02:19:28 +0000

This is in response to

Article No Longer Available

I have mixed feelings about Typescript. To quote Indian Jones, "you call that a type, this is a type."

If this is where people get their opinions of static typing, no wonder they think they should live without types.

On the other hand, trying to quantify root cause of an issue is not trivial. Generally people are not tracking these types of things and it is only a "sense" of the situation. I want to summarize some of my experiences with types and talk to an anecdote.

Experience

Types

Help with code reuse
Help with refactoring
Help guide programmers to correct usage

This first point is really about the next two; you do not need types to make reusable code. If the programmer misuses your library then it isn't very reusable. If you can't make refactoring changes because others might be using it, isn't very reusable.

Let's consider the simplist refactor, rename. Your IDE will very well replace all the usage in the code and you'll be confident things are still working the same. But you have a development team, they are working off branches, some work has practically died on other branches. How do we make sure that the rename propagates to the code in these branches?

With a static typed language, this is a build error. An annoying build error, but preventative nevertheless.

Build failure can occur from attempts to pass objects without the needed properties. And when using a language with compile time execution, these static guarantees can be quite helpful. (plug for D here)

Anecdote

Recently a bug was introduced into some backend code when types were introduced (yes static typed language can work with untyped data). What happened was the type defined expected data before passing the information on to the next system. The previous implementation would blindly pass the blob on.

The previous implementation worked perfectly, the introduction of types was caught by my extensive introduction tests. Why am I so happy that we introduced this bug?

The two systems had a mismatch in expectations for what was contained in the "fault tolerant" web. The system using the data had built its expectations off the existing production data. The type was built from the new data source which did not contain such fields. Our conversations lead to communicating our concerns to the upstream 3rd party.

As my tests use a mock of the 3rd party, had the type been built from our expectations, we would not have caught this issue as we would not have had influence from the real 3rd party API. That is on me, but it all panned out that a bug introduced by types identified issues upstream.

Substring/Slice of a String

Jesse Phillips — Fri, 29 Jan 2021 05:08:14 +0000

A Substring or Slice of a string, is a way to create a new string from an existing string. This technique usually is completed without allocation for the new string.

"hello world"

"o wor"

C# and Java

These languages provide a string class which are considered immutable as they don't expose methods s to modify the string.

The Substring (C#) and substring(Java) are used to create new strings from an existing one.

// C#
Console.WriteLine("hello world".Substring(4,5));

We supply the starting index of the original string, then specify the length of the new string. You can leave off the second number if you want it to go the end of the original.

// Java
System.out.println("hello world".substring(4,9));

While similar in syntax, the second argument is the index of the where the string ends in the original.

Python

# Python
print("hello world"[4:9])

The Python language includes a specific syntax, which resembles that used for indexing. Here the second number is an index specifier for where the slice ends within the original array.

Like C# the second number can be left out to include to the end of the original.

# Python
print("hello world"[4:])

Javascript

// Javascript
"Hello world!".slice(4, 9);

Javascript takes its behavior from Python, utilizing an index rather than length for its second parameter.

Unicode

Not a single one of these will protect you from splitting surrogates.

It is possible for this method to work well for your use case, even if unicode is within the string.

Even my preferred language requires consideration of the unicode details.

String Contained in a String in D

Jesse Phillips ・ Aug 25 '19

#dlang #tutorial #string