Forem: Jeff Culverhouse

Some(number) of ways to calculate a Fibonacci Number in Rust

Jeff Culverhouse — Wed, 22 Jul 2020 21:15:49 +0000

A Fibonacci number divided by the previous one approaches Phi, the Golden Ratio. That’s an Internet hole that can last for hours, if you let it! However, don’t believe everything you read about Phi.

Solving Fibonacci Numbers is the start of Chapter 3 on Dynamic Programming (more on that below) in Erickson’s Algorithm book. I had started with Peasant Multiplication, then the merge sort, and most fun – the n Queens Problem. With Fibonacci Number calculating though, there are a couple of obvious ways to code this and some presented questions for me with Rust.

I became interested in comparing a couple of these ways – both in thinking about the Rust code and looking at the timing. Then my eyes popped out when I saw the difference in the release build times!

Pseudo code, Page 97 from Algorithm by Jeff Erickson
Clean, Simple Recursion

First is probably closest to the pseudo code for the classic solution for Fibonacci. Define what 0 and 1 return and solve for the given number, recursing all the way back to 2 (which adds the fixed answers for 0 and 1):

Recursive Backtracing

fn backtrace_fib(fib_num: u128) -> u128 {
    if fib_num == 0 || fib_num == 1 {
        return fib_num;
    }
    backtrace_fib(fib_num - 1) + backtrace_fib(fib_num - 2)
}

Next, especially important if your function will be called multiple times, is to add memoization. That is, once you calculate an answer for a given input, remember than answer so you can quickly respond with it later, skipping the math.

I don’t like that, with the way I did this, I have to pass in my HashMap where I’m storing the answers. Is there any way to have some 'static living private variable that only this function would see, but would last the whole run of the program?? There are other places I could store away data, like memcache and redis.

Backtracing With Memoization

fn backtrace_memo_fib(memo: &mut HashMap<u128, u128>, fib_num: u128) -> u128 {
    match memo.get(&fib_num).map(|answer| answer.clone()) {
        Some(result) => result,
        None => {
            let result = match fib_num {
                0 | 1 => fib_num,
                n => backtrace_memo_fib(memo, n - 1) + backtrace_memo_fib(memo, n - 2),
            };
            memo.insert(fib_num, result.clone());
            result
        }
    }
}

So, Jeff Erickson, author of the Algorithms book, explains another way solve for a Fibonacci Number involves using dynamic programming. But this IS your grandfathers “programming” (not ours).

The word 'programming' does not refer to writing code, but rather to the older sense of planning or scheduling, typically by filling in a table. For example, sports programs and theater programs are schedules of important events (with ads); television programming involves filling each available time slot with a show (and ads); degree programs are schedules of classes to be taken (with ads).

Erickson, Jeff (2018, December 29). Algorithms. pp. 100. Retrieved from Algorithms: http://algorithms.wtf

For this one, I just kept the memoization inside the function. We then purposely fill the HashMap on our way up, referencing already-calculated answers, and so never descend into levels of recursion.

Pseudo code, Page 99 from Algorithm by Jeff Erickson

Dynamic Programming

fn dynamic_fib(fib_num: u128) -> u128 {
    let mut memo = HashMap::new();
    memo.insert(0, 0);
    memo.insert(1, 1);
    match fib_num {
        0 | 1 => {} // already set
        n => {
            for i in 2..=n {
                let result = *memo.get(&(i - 1)).unwrap() + *memo.get(&(i - 2)).unwrap();
                memo.insert(i, result);
            }
        }
    };
    *memo.get(&fib_num).unwrap()
}

So, as I was trying to figure out how to store some data in a function for future calls (if that is even possible), I came across the cached crate. “cached provides implementations of several caching structures as well as a handy macro for defining memoized functions.” How interesting – just what I needed and the memoization is even hidden from ME. Ok, I’ll do the simple, recursive, backtracing function but cache the results.

Cached Backtracing Function

#[cached(size = 200)]
fn cached_fib(fib_num: u128) -> u128 {
    if fib_num == 0 || fib_num == 1 {
        return fib_num;
    }
    cached_fib(fib_num - 1) + cached_fib(fib_num - 2)
}

Fibonacci Fight

So, I wrap all of those up into main() so I can start comparing them. That code is pretty boring, so check it out in the repo. Mostly I run each flavor of the algorithm, timing the result and print it all out. The first one looks like:

    let now = Instant::now();
    let _ = backtrace_fib(fib_num);
    let elapsed = now.elapsed();
    print_results(fib_num, "simple backtracing/recursion", elapsed);

I also added a test, just to be sure each is giving the correct answer.

#[test]
fn test_each_version() {
    assert_eq!(backtrace_fib(20), 6765);
    assert_eq!(backtrace_memo_fib(&mut HashMap::new(), 20), 6765);
    assert_eq!(dynamic_fib(20), 6765);
    assert_eq!(cached_fib(20), 6765);
}

Fibonacci Racing

And then some code to time each version, and call it a second and third time, so we can see the memoization at work.

Of course, your results will vary by CPU, but look at this run for Fibonacci Number 50 with the debug build (where the differences are huge). This is on an Amazon t2.medium EC2 server:

The first time solving will be the slowest

  Solving fib:50 with simple backtracing/recursion                       took    287244288946 ns
  Solving fib:50 with backtracing/recursion with memoization             took          211289 ns
  Solving fib:50 with dynamic programming with memoization               took          167291 ns
  Solving fib:50 with cached function                                    took          214175 ns

What about solving it a second or third time, anyone faster this time?

  Solving fib:50 with simple backtracing/recursion                       took    287262389809 ns
  Solving fib:50 with backtracing/recursion with memoization             took          202481 ns
  Solving fib:50 with dynamic programming with memoization               took          162527 ns
  Solving fib:50 with cached function                                    took            6285 ns

  Solving fib:50 with simple backtracing/recursion                       took    287273113221 ns
  Solving fib:50 with backtracing/recursion with memoization             took          185609 ns
  Solving fib:50 with dynamic programming with memoization               took          158492 ns
  Solving fib:50 with cached function                                    took            6496 ns

By the way, Fibonacci Number 50 is 12586269025 which (divided by Fib Num 49) approxmates phi as 1.618033988749895

But wow, check out these numbers with a release build!!

The first time solving will be the slowest

  Solving fib:50 with simple backtracing/recursion                       took             806 ns
  Solving fib:50 with backtracing/recursion with memoization             took           16128 ns
  Solving fib:50 with dynamic programming with memoization               took            8052 ns
  Solving fib:50 with cached function                                    took           19495 ns

What about solving it a second or third time, anyone faster this time?

  Solving fib:50 with simple backtracing/recursion                       took             596 ns
  Solving fib:50 with backtracing/recursion with memoization             took            8142 ns
  Solving fib:50 with dynamic programming with memoization               took            7576 ns
  Solving fib:50 with cached function                                    took             724 ns

  Solving fib:50 with simple backtracing/recursion                       took             596 ns
  Solving fib:50 with backtracing/recursion with memoization             took            7549 ns
  Solving fib:50 with dynamic programming with memoization               took            7327 ns
  Solving fib:50 with cached function                                    took             746 ns

By the way, Fibonacci Number 50 is 12586269025 which (divided by Fib Num 49) approxmates phi as 1.618033988749895

A Really Big Number

And it handles a Fibonacci Number of 186, though any higher and casting the answers as f64 s to divide and get Phi breaks. The fastest “first” calculation for Fibonacci Number 186 was simple backtracing/recursion and took only 820 ns and it’s kinda big:

By the way, Fibonacci Number 186 is 332825110087067562321196029789634457848 which (divided by Fib Num 185) approximates phi as 1.6180339887498947

The post Some(number) of ways to calculate a Fibonacci Number in Rust appeared first on Learning Rust.

(A Few) Advanced Variable Types in Rust

Jeff Culverhouse — Mon, 20 Jul 2020 22:26:28 +0000

Keep one eye on your code at all times!

“I haven’t seen Evil Dead II yet”. Much is made about this simple question in the movie adaption of [High Fidelity](https://en.wikipedia.org/wiki/High_Fidelity(film))_. Does “yet” mean the person does, indeed, intend to see the film? Jack Black’s character is having real trouble with the concept – not only does he know that the speaker, John Cusack’s character, has seen Evil Dead II, but what idiot wouldn’t see it, “because it’s a brilliant film. It’s so funny, and violent, and the soundtrack kicks so much ass.” I love this exchange, but I’m a fan of the film anyway. It is not always clear to me how to handle advanced variable types Rust, yet.

I think of these as wrappers that add abilities (and restrictions) to a variable. They give a variable super powers since the Rust compiler is so strict about what you can and can’t do with variables.

Box<T>

PROVIDES:

Smart pointer that forces your variable’s value to be stored on the heap instead of the stack. The Box<> variable itself is just a pointer so its size is obvious and can, itself, be stored on the stack.

RESTRICTIONS:

USEFUL WHEN:

If the size of an item cannot be determined at compile time it will complain if the default is to store it on the stack (where a calculable size is necessary). Using Box<> will force the storage on the heap where a fixed size is not needed. For example, a recursive data-structure, including enums, will not work on the stack because a concrete size cannot be calculated. Turning the recursive field into a Box<> means it stores a pointer which CAN be sized. The example in the docs being:

enum List\<T\> {
  Cons(T, Box\<List\<T\>\>),
  Nil,
}

Also useful if you have a very large-sized T, and want to transfer ownership of that variable without it being copied each time.

NOTABLY PROVIDES:

just see the rust-lang.org docs

EXAMPLES/DISCUSSION:

https://doc.rust-lang.org/stable/rust-by-example/std/box.html

https://www.koderhq.com/tutorial/rust/smart-pointer/

https://manishearth.github.io/blog/2017/01/10/rust-tidbits-box-is-special/

Setting the value of a simple Box<> variable is easy enough and getting the value back looks very normal:

fn main() {
  let answer = Box::new(42);
  println!("The answer is : {}", answer);
}

Cell<T>

PROVIDES:

You can have multiple, shared references to the Cell<> (and thus, access to the value inside with .get()) and yet still mutate the value inside (with .set()). This is called interior mutability because the value inside can be changed but mut on the Cell<> itself is not needed. The inner value can only be set by calling a method on the Cell<>.

RESTRICTIONS:

It is not possible to get a reference to what is inside the Cell , only a copy of the value. Also, Cell does not implement sync , so it cannot be given to a different thread, which ensures safety.

USEFUL WHEN:

Usually used for small values, such as counters or flags, where you need multiple shared references to the value AND be allowed to mutate it at the same time, in a guaranteed safe way.

NOTABLY PROVIDES:

.set() to set the value inside

.get() to get a copy of the value inside

.take() to get a copy of the value inside AND reset the value inside to default.

see the rust-lang.org docs

EXAMPLES/DISCUSSION:

https://hub.packtpub.com/shared-pointers-in-rust-challenges-solutions/

https://ricardomartins.cc/2016/06/08/interior-mutability

Setting the inner value of a Cell<> is only possible with a method call which is how it maintains safety:

use std::cell::Cell;
fn main() {
  let answer = Cell::new(0);
  answer.set(42);
  println!("The answer is : {}", answer.get());
}

RefCell<T>

PROVIDES:

RefCell<> is very similar to Cell<> except it adds borrow checking, but at run-time instead of compile time! This means, unlike Cell<> , it is possible to write RefCell<> code which will panic!(). You borrow() a ref to the inner value for read-only or borrow_mut() in order to change it.

RESTRICTIONS:

borrow() will panic if a borrow_mut() is in place, and borrow_mut() will panic if either type is in place.

USEFUL WHEN:

NOTABLY PROVIDES:

.borrow() to get a copy of the value at the ref

.borrow_mut() to set the value at the ref

.try_borrow() and .try_borrow_mut() will return a Result<> or error instead of a panic!().

see the rust-lang.org docs

EXAMPLES/DISCUSSION:

https://ricardomartins.cc/2016/06/08/interior-mutability (again)

You must successfully borrow_mut() the RefCell<> in order to set the value (by dereferencing) and then simply borrow() it to retrieve the value:

use std::cell::RefCell;
fn main() {
  let answer = RefCell::new(0);
  *answer.borrow_mut() = 42;
  println!("The answer is : {}", answer.borrow());
}

whereas, something as simple as this compiles, but panics at run-time. Imagine how much more obscure this code could be. Remember, any number of read-only references or exactly 1 read-write reference and nothing else – although for RefCell, this is enforced at run-time:

use std::cell::RefCell;
fn main() {
  let answer = RefCell::new(0);
  let break_things = answer.borrow_mut();
  println!("The initial value is : {}", *break_things); 
  *answer.borrow_mut() = 42;
  println!("The answer is : {}", answer.borrow());
}

Rc<T>

PROVIDES:

Adds the feature of run-time reference counting to your variable, but this is the simple, lower-cost version – it is not thread safe.

RESTRICTIONS:

Right from the docs “you cannot generally obtain a mutable reference to something inside an Rc. If you need mutability, put a Cell or RefCell inside the Rc“. So while there is a get_mut() method, it’s easy to just use a Cell<> inside.

USEFUL WHEN:

You need run-time reference counting of a variable so it hangs around until the last reference of it is gone.

NOTABLY PROVIDES:

.clone() – get a new copy of the pointer to the same value, upping the reference count by 1.

see the rust-lang.org docs

EXAMPLES/DISCUSSION:

https://blog.sentry.io/2018/04/05/you-cant-rust-that#refcounts-are-not-dirty

Note that in the example below, my_answer is still pointing to valid memory even when correct_answer is dropped, because the Rc<> had an internal count of “2” and drops it to “1”, leaving the storage of “42” still valid.

use std::rc::Rc;
fn main() {
  let correct_answer = Rc::new(42);
  let my_answer = Rc::clone(&correct_answer);
  println!("The correct answer is : {}", correct_answer);
  drop(correct_answer);

  println!("And you got : {}", my_answer);
}

Arc<T>

PROVIDES:

Arc<> is an atomic reference counter, very similar to Rc<> above but thread-safe.

RESTRICTIONS:

More expensive than Rc<>. Also note, the <T> you store must have the Send and Sync traits. So an Arc<RefCell<T>> will not work because RefCell<> is not Sync.

USEFUL WHEN:

Same as Rc<> , You need run-time reference counting of a variable so it hangs around until the last reference of it is gone, but safe across threads as long as the inner <T> is.

NOTABLY PROVIDES:

see the rust-lang.org docs

EXAMPLES/DISCUSSION:

https://medium.com/@DylanKerler1/how-arc-works-in-rust-b06192acd0a6

Same idea as with Rc<> , we just show it working across multiple threads (and then sleep for just 10ms to let those threads finish).

use std::sync::Arc;
use std::thread;
use std::time::Duration;
fn main() {
  let answer = Arc::new(42);
  for threadno in 0..5 {
    let answer = Arc::clone(&answer);
    thread::spawn(move || {
      println!("Thread {}, answer is: {}", threadno + 1, answer);
    });
  }
  let ten_ms = Duration::from_millis(10);
  thread::sleep(ten_ms);
}

Mutex<T>

PROVIDES:

Mutual exclusion lock protecting shared data, even across threads.

RESTRICTIONS:

Any thread which panics will “poison” the Mutex<> and make it inaccessible to all threads. The T stored must allow Send but Sync is not necessary.

USEFUL WHEN:

working on it!

NOTABLY PROVIDES:

see the rust-lang.org docs

EXAMPLES/DISCUSSION:

https://doc.rust-lang.org/book/ch16-03-shared-state.html

use std::sync::{Arc, Mutex};
use std::thread;
use std::time::Duration;
fn main() {
  let answer = Arc::new(Mutex::new(42));
  for thread_no in 0..5 {
    let changer = Arc::clone(&answer);
    thread::spawn(move || {
      let mut changer = changer.lock().unwrap();
      println!("Setting answer to thread_no: {}",
        thread_no + 1,
      );
      *changer = thread_no + 1;
    });
  }
  let ten_ms = Duration::from_millis(10);
  thread::sleep(ten_ms);
  if answer.is_poisoned() {
    println!("Mutex was poisoned :(");
  }
  else {
    println!("Mutex survived :)");
    let final_answer = answer.lock().unwrap();
    println!("Ended with answer: {}", final_answer);
  }
}

RwLock<T>

PROVIDES:

Similar to RefCell, but thread safe. borrow() is read(), borrow_mut is write(). They don’t return an option, they will block until they do get the lock.

RESTRICTIONS:

Any thread which panics while a write lock is in place will “poison” the RwLock<> and make it inaccessible to all threads. A panic! during a read lock does not poison the RwLock. The T stored must allow both Send and Sync.

USEFUL WHEN:

working on it!

NOTABLY PROVIDES:

see the rust-lang.org docs

EXAMPLES/DISCUSSION:

Slightly fancier example, that shows getting both read() and write() locks on the value. If nothing panics, we should see the answer at the end.

use std::sync::{Arc, RwLock};
use std::thread;
use std::time::Duration;
fn main() {
  let answer = Arc::new(RwLock::new(42));
  for thread_no in 0..5 {
    if thread_no % 2 == 1 {
      let changer = Arc::clone(&answer);
      thread::spawn(move || {
        let mut changer = changer.write().unwrap();
        println!("Setting answer to thread_no: {}",
          thread_no + 1,
        );
        *changer = thread_no + 1;
      });
    }
    else {
      let reader = Arc::clone(&answer);
      thread::spawn(move || {
        let reader = reader.read().unwrap();
        println!( "Checking answer in thread_no: {}, value is {}",
          thread_no + 1,
          *reader
        );
      });
    }
  }
  let ten_ms = Duration::from_millis(10);
  thread::sleep(ten_ms);
  if answer.is_poisoned() {
    println!("Mutex was poisoned :(");
  }
  else {
    println!("Mutex survived :)");
    let final_answer = answer.read().unwrap();
    println!("Ended with answer: {}", final_answer);
  }
}

Checking answer in thread_no: 1, value is 42
Checking answer in thread_no: 3, value is 42
Setting answer to thread_no: 2
Checking answer in thread_no: 5, value is 2
Setting answer to thread_no: 4
Mutex survived :)
Ended with answer: 4

Summary

There are more, plus many custom types, some I’ve even used like the crate once_cell. I started using that for the web app I was (am?) working on and wrote a little about it. Also, as you saw in the last two examples, you can combine types when you need multiple functionalities. I have included these examples in a GitHub repo, pointers.

I’ll probably hear about or (much more slowly) learn about mistakes I’ve made in wording here or come up with much better examples and excuses for using these various types, so I’ll try to update this post as I do. I see using this myself as a reference until I am really familiar with each of these types. Obviously, any mistakes here are mine alone as I learn Rust and not from any of the links or sources I listed!

Also, lots of help from 3 YouTubers I’ve been watching – the best examples can been seen as they write code and explain why they need something inside an Rc<> or in a Mutex<>. Check out their streams and watch over their shoulder as they code!!

Jon Gjengsetespecially, in this case, his video on this same topic!
David Pedersen
Ryan Levick who just had a video on once_cell!

The post (A Few) Advanced Variable Types in Rust appeared first on Learning Rust.

Next Algorithm: Backtracking into the n Queens Problem

Jeff Culverhouse — Fri, 17 Jul 2020 02:04:19 +0000

Stalking her prey…

This next algorithm was really fun and a bit more challenging. The Algorithms book I’m going through went through several more examples of breaking a problem into a smaller, simpler problem and letting recursion CPU its way to a solution, like the merge sort we just looked at. Then I moved on to chapter 2 about “backtracking” into the n Queens Problem!

How Many Queens?

According to the book, the “n Queens Problem” is a prime example of using backtracking to solve a problem. Backtracking is another way to reduce a hard problem down into smaller chunks that are more easily solvable. In this case, showing the solution as it is worked out with a recursion tree model really explains well the approach used here. Go see page numbered 71 of the PDF to check it out. After just a couple of pages, Erickson moves on to the related topic of Game Trees, but this n Queens Problem seemed really fun to me.

I first wanted to try and show the solutions as text-based chessboards (plus allow for the boring array version showing one solution per line). It took me a little while to setup the Rust crate crossterm to help me out with this. I had only recently heard of crossterm while watching a YouTube video by David Pedersen as he codes up a simple Git helper with Rust. I didn’t go crazy with color or UTF-8 like I could have, but feel free.

Let’s Rust

Then came the matter of converting the algorithm pseudo code from the PDF into Rust code. This was slightly more difficult as Rust is very careful about array bounds checking and I had to be a little safer with my IF checks than the pseudo code warned about.

Since it is Creative Commons, I can include the pseudo-code presented in Jeff Erickson’s Algorithm book – but go check it out via his site!

Figure 2.2, Page 71 from Algorithm by Jeff Erickson

When you run the queens_problem binary, you pass along an argument of how big your chessboard square is. Optionally you can include a second param (of anything, the code doesn’t really care) which causes it to output chessboards rather than the array dumps.

main.rs for queens_problem

use crossterm::{
    style::{Color, Colors, Print, ResetColor, SetColors},
    ExecutableCommand,
};
use std::env;
use std::io::stdout;

const QUEEN: &str = "<W>";
const EMPTY: &str = "   ";

/// Adapted from reading "Algorithms" by Jeff Erickson
/// freely available on http://jeffe.cs.illinois.edu/teaching/algorithms/
pub fn main() {
    let args: Vec<String> = env::args().collect();

    // nothing fancy at all, just brute-force it
    if args.len() < 2 || args.len() > 3 {
        println!("Usage: {} n [show]", args[0]);
        println!("  n = size of chessboard");
        println!("  [show] to actually see each chessboard solution");
        return;
    }

    let boardsize = args[1].parse::<usize>().unwrap();
    let chessboard = vec![-1i8; boardsize];
    let showboard = args.len() == 3; // don't even care what it is

    place_queens(chessboard.clone(), 0, showboard);
}

fn show_chessboard(board: Vec<i8>, showboard: bool) {
    let mut light: bool = true;

    if showboard {
        for pos in board.to_vec() {
            for cell in 0..board.len() {
                match (pos == cell as i8, light) {
                    (true, true) => draw_square(Colors::new(Color::Black, Color::Grey), QUEEN),
                    (true, false) => draw_square(Colors::new(Color::Grey, Color::Black), QUEEN),
                    (false, true) => draw_square(Colors::new(Color::Black, Color::Grey), EMPTY),
                    (false, false) => draw_square(Colors::new(Color::Grey, Color::Black), EMPTY),
                };
                light = !light;
            }
            println!();
            if board.len() % 2 == 0 {
                // to checkerboard even-sized boards
                light = !light;
            }
        }
        println!("\n");
    } else {
        let adjusted: Vec<i8> = board.iter().map(|&p| p + 1).collect(); // lets remove the 0-based confusion
        println!("{:?}", adjusted);
    }
}

fn draw_square(color: Colors, chesspiece: &str) {
    let mut stdout = stdout();

    stdout.execute(SetColors(color)).unwrap();
    stdout.execute(Print(chesspiece)).unwrap();
    stdout.execute(ResetColor).unwrap();
}

/// n Queens Problem
/// Section2.1, page 69-71
fn place_queens(mut chessboard: Vec<i8>, row: usize, showboard: bool) {
    if row == chessboard.len() {
        show_chessboard(chessboard.to_vec(), showboard);
        return;
    }
    for column in 0..chessboard.len() {
        let mut legal = true;
        for cell in 0..row {
            let pos = cell as usize;
            if chessboard[pos] == (column as i8)
                || (column + row >= cell && chessboard[pos] == ((column + row - cell) as i8))
                || (column + cell >= row && chessboard[pos] == ((column + cell - row) as i8))
            {
                legal = false;
            }
        }
        if legal {
            chessboard[row] = column as i8;
            place_queens(chessboard.clone(), row + 1, showboard);
        }
    }
}

Just a Hint of Verification

As nerdy as they look, I used the array dumps to validate my code against what Wiki article on the 8 Queens Puzzle has as the number of valid solutions for different sized boards. I verified a 4×4 up to 14×14 board and matched the expected counts perfectly. Go check out the README for this repo in Github.

The post Next Algorithm: Backtracking into the n Queens Problem appeared first on Learning Rust.

Merge Sort With Rust

Jeff Culverhouse — Thu, 16 Jul 2020 02:39:54 +0000

Quick, sort; no not with quicksort, with merge sort

Another algorithm to play with. Moving on in the Algorithms book I mentioned in the last post to section 1.4 we come across the merge sort. Another fun one to play with in Rust and to learn vectors just a little better. So let’s code up a quick merge sort with Rust!

Yes, I’m That Old

My AP computer class back around the mid 1980s was in Pascal. I later learned that Pascal was never meant to be used in production – it was a purely a teaching language – even to the extent it could be taught on paper and never even compiled. How terrible it would be not to see your first programs run!

Either way, we worked on terminals and Pascal worked out just fine for me. That language led me easily to C and, though I never used C professionally, that made for an easy transition to Perl (which I have) and to Rust! It turned out, the prior few years of AP tests had been bad enough across all the high schools, they decided to omit three sections of Pascal, unless you optionally wanted to learn them and take an extra AP test, which I did. These three subjects were: recursion (the algorithms both last time and this time use recursion), linked list (that broke my brain for a little while – I was glad later on that it happened in school because it made C so much easier to learn), and sorting (without any simple, built-ins).

Merging with Rust

I am sure I learned the simple merge sort back then, 35 years ago. Heck, that means it was only 40 years old when I learned it! The book only spends a couple of pages on it – it is just another example of a recursive algorithm. It was fun in Rust though, I learned some vector methods I wasn’t aware of like the various forms of split_at and concat. Again I made a couple of very simple tests just to help show it works.

So, here is the code I came up with:

main.rs for merge sort

use std::env;

#[test]
fn check_mergesort() {
    let letters = vec!["s", "a", "m", "p", "l", "e"];
    let shouldbe = vec!["a", "e", "l", "m", "p", "s"];
    let sorted = mergesort(letters);
    assert_eq!(sorted, shouldbe);

    let words = vec![
        "now", "is", "the", "time", "for", "all", "good", "men", "to", "come", "to", "the", "aid",
        "of", "their", "country",
    ];
    let shouldbe = vec![
        "aid", "all", "come", "country", "for", "good", "is", "men", "now", "of", "the", "the",
        "their", "time", "to", "to",
    ];
    let sorted = mergesort(words);
    assert_eq!(sorted, shouldbe);
}

/// Adapted from reading "Algorithms" by Jeff Erickson
/// freely available on http://jeffe.cs.illinois.edu/teaching/algorithms/
pub fn main() {
    let args: Vec<String> = env::args().collect();

    if args.len() < 2 {
        println!(
            "Usage: {} word word [word...] (two or more words to sort)",
            args[0]
        );
        return;
    }

    let words = args.iter().skip(1).map(AsRef::as_ref).collect();
    println!("Arrived: {:?}", words);

    let sorted = mergesort(words);
    println!("Sorted: {:?}", sorted);
}

/// Mergesort
/// Section 1.4, page 27
pub fn mergesort(words: Vec<&str>) -> Vec<&str> {
    if words.len() < 2 {
        return words;
    }

    let midpoint = (words.len() + 1) / 2;
    let (left, right) = words.split_at(midpoint);

    let left = mergesort(left.to_vec());
    let right = mergesort(right.to_vec());

    let halfsort = [left, right].concat();
    let sorted = merge(halfsort, midpoint);

    return sorted;
}

pub fn merge(words: Vec<&str>, midpoint: usize) -> Vec<&str> {
    let size = words.len();
    let mut left_index = 0;
    let mut right_index = midpoint;
    let mut sorted: Vec<&str> = Vec::new();

    for _ in 0..size {
        if right_index >= size {
            sorted.push(&words[left_index]);
            left_index += 1;
        } else if left_index >= midpoint {
            sorted.push(&words[right_index]);
            right_index += 1;
        } else if words[left_index] < words[right_index] {
            sorted.push(&words[left_index]);
            left_index += 1;
        } else {
            sorted.push(&words[right_index]);
            right_index += 1;
        }
    }
    return sorted;
}

There is not really much to say about it though – unless you have questions or suggestions. Check it out in the github repo.

The post Merge Sort With Rust appeared first on Learning Rust.

Algorithmic Rust

Jeff Culverhouse — Tue, 14 Jul 2020 16:37:02 +0000

Old school – how many of these are sold these days?

Well, I’m bored with my fake web app. Because I don’t know where to go next, because it doesn’t have a design or plan, because it is fake. I want to play (practice) writing Rust, but the weird “practice-this” sites always seem overly contrived: “replace all the capital letters in a given string with their ASCII-number”. Sigh. So, I’m trying something new – and probably will get over my head in no time. How about I dive into the world of algorithms: with Rust!

An 80’s Kid

I never had a formal computer science education – I grew up in the sweet spot of when nerdy kids just hacked on code and tried to make things work. I barely played computer games as a teenager, but it was very common for me to be up half the night typing in some code from a COMPUTE! magazine. Then, once it was finally in and working, adjusting the code to tweek how it ran. Oh wow – for instance, check out the code for RATS!. Anyway, the world is hidden with such great treasures, not all of them from the 80’s! Take, for example, a professor who, after teaching for a decade decides to revise and cleanup his lecture notes and, while doing so, turns them into a freely available book on algorithms. Sure, you can buy his book on Amazon for $30, but just visit Jeff Erickson’s site to get his 448-page PDF for free!! Little did I realize, until into the first chapter, this book/course is apparently NOT for undergraduates. Oh well, I’ll see how far I can get.

Come At Me

Anyway, as I’m reading through chapter 1 with the stress-free page turning of someone who won’t be actually tested on any of this. I come across a pseudo-code algorithm for something called “Peasant Multiplication”. But, I won’t murder the explanation here, just go download Erickson’s PDF. I will, however, show my interpretation of how to code that in Rust!

So here is what I came up for the algorithm itself and wrapped it so the user can try, plus there is a simple test:

main.rs

use std::env;

#[test]
fn check_peasant_mult() {
  use rand::Rng;
  let mut rng = rand::thread_rng();
  for _ in 0..10 {
    let n1 = rng.gen_range(1, 10000);
    let n2 = rng.gen_range(1, 10000);
    assert_eq!(peasant_multiplication(n1, n2), n1 * n2);
  }
}

/// Adapted from reading "Algorithms" by Jeff Erickson
/// freely available on http://jeffe.cs.illinois.edu/teaching/algorithms/
pub fn main() {
  let args: Vec<String> = env::args().collect();
  if args.len() != 2 {
    println!("Usage: {} x y (two positive integers)", args[0]);
    return;
  }
  let x = args[1].parse::<u32>().unwrap();
  let y = args[2].parse::<u32>().unwrap();
  let answer = peasant_multiplication(x, y);
  println!("{} x {} = {}", x, y, answer);
}

/// Peasant Multiplication
/// Section 1.2, page 23
pub fn peasant_multiplication(x: u32, y: u32) -> u32 {
  if x == 0 {
    return 0;
  }

  let x_prime = x / 2;
  let y_prime = y + y;
  let mut prod = peasant_multiplication(x_prime, y_prime);
  if x % 2 > 0 {
    prod += y;
  }
  return prod;
}

I thought it would be funny for someone to notice my multiplication algorithm required a random-number-generator if they looked into my Cargo.toml file, but TIL about the [dev-dependencies] section where you can list things you only need for testing! So:

Cargo.toml

[package]
name = "peasant_mult"
version = "0.1.0"
authors = ["Jeff Culverhouse <jeff@graystorm.com>"]
edition = "2018"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]

[dev-dependencies]
rand = "0.7"

There is absolutely no reason to do multiplication this way, of course. But it let me practice my Rust WHILE learning about algorithms in computer science. It doesn’t do much, but at least it feels worthwhile and not contrived. THIS is how some people (long, long ago) did multiplication – and it was fun to reproduce it. Multiplication inside your computer’s CPU doesn’t happen the way you think either, but I don’t know anything about that so go off on a Google tangent if you like!

Anyway, this one is simple and short enough I think I’ll just leave you the exercise of getting his PDF and comparing his quick mention of Peasant Multiplication and my implementation. You probably see something to improve or ways in which I am not writing idiomatic Rust (like that for loop in the test). Feel free to suggest some changes here! Also, I put this up on github, because why not? And I have some more coming.

The post Algorithmic Rust appeared first on Learning Rust.

Giving My App Secrets to the AWS SecretsManager

Jeff Culverhouse — Sat, 11 Jul 2020 19:10:09 +0000

Psst… want an API key?

I’m still working around my PinPointShooting app without actually making any app progress. You can go back to the start if you like or just jump in here. So far, I’ve had my app ids and secrets hidden inside a .env file, but rather than keep them so close to the code, lets move them into my Amazon Web Services (AWS) account. It will cost me an extra $2.40/month to store these 6 secrets (so far) at Amazon ($0.40/month/secret it claims), but it gets me more practice using the rusoto_core crate and its subsidiaries. So, here is how I went about giving all of my app secrets to the AWS SecretsManager.

Cargo.toml Changes

First, in Cargo.toml, I already have rusoto_core and rusoto_dynamodb for storing my session data. So I only need to add rusoto_secretsmanager:

[dependencies]slog = "2.5.0"
slog-bunyan = "2.1.0"
base64 = "0.10.1"
rand = "0.7.0"
rand_core = "0.5.0"
rust-crypto = "0.2.36"
config = "0.9.3"
serde_json = "1.0.40"
once_cell = "0.2.2"
dotenv = "0.14.1"
rocket-slog = "0.4.0"
sha2 = "0.8.0"
rusoto_core = "0.40.0"
rusoto_dynamodb = "0.40.0"
rusoto_secretsmanager = "0.40.0"
time = "0.1.42"
google-signin = "0.3.0"

Store Those Secrets in AWS

Of course, I had to log into AWS and create these 6 new secrets. In case I need more for other applications in the future, I prefixed each of these with “PPS_”. So, I created:

Secret name
PPS_google_api_client_id
PPS_google_api_client_secret
PPS_google_maps_api_key
PPS_facebook_app_id
PPS_facebook_app_secret
PPS_database_url

List of secrets I created in my AWS account

For each, I choose the “Other type of secrets (e.g. API key)” as the type. I stored the pair as “key” and the value I wanted to protect as a secret. This becomes the very simple struct:

#[derive(Debug, Deserialize)]
struct AWSSecret {
  key: String,
}

So, instead of pulling these into my global CONFIG from the .env file, I want to pull them from AWS instead. This requires changes to the settings.rs file. First I add the requirements at the top:

use rusoto_core::Region;
use rusoto_secretsmanager::{GetSecretValueRequest, SecretsManager, SecretsManagerClient};

plus I define the AWSSecret struct I mentioned above. Now, right after I pull in all the config settings, just before I try to turn that into a Lazy::New object to initialize the CONFIG, I need to add yet another place to pull settings from. Lets loop through each of the 6 and pull them down from AWS:

Pull Into Config

// now pull secrets from AWS
// note, AWS secrets include PPS_ prefix for this application
for secret in vec![
  "database_urls",
  "google_api_client_id",
  "google_api_client_secret",
  "google_maps_api_key",
  "facebook_app_id",
  "facebook_app_secret",
] {
  config
    .set(secret, get_secret(format!("PPS_{}", secret)).key)
    .unwrap();
}

So, for each Config setting stored on AWS, try to get that secret, prepending a “PPS_” in front of the secret name when checking AWS. Now I just write that get_secret() function:

get_secrets stored at AWS

fn get_secret(secret: String) -> AWSSecret {
  let secrets_manager = SecretsManagerClient::new(Region::UsEast1);
  match secrets_manager
    .get_secret_value(GetSecretValueRequest {
      secret_id: secret.clone(),
      ..Default::default()
    })
    .sync() {
      Ok(resp) => serde_json::from_str(&resp.secret_string.unwrap()).unwrap(),
      Err(err) => panic!("Could not retrieve secret {} from AWS: {:?}", secret, err),
    }
}

I’ll probably look back on this code in 6 months or a year and cringe, but this works anyway – and dies if it fails to pull a secret, which is what I want. I guess get_secret could return Some(AWSSecret) or None… but None means something in the app will break – I might as well just panic!

Of course, my AWS credentials are still stored in the .env file. Who protects the secret the protects the secrets!? Anyway, I did update the .env.sample file to reflect the new setup:

# configure for your user and rename to .env
ROCKET_ENV=development
AWS_ACCESS_KEY_ID=your_key_here
AWS_SECRET_ACCESS_KEY=your_key_here

# store as AWS secrets: key/value
#PPS_database_url (for example "postgres://user:password@localhost/pinpoint")
#PPS_google_api_client_id
#PPS_google_api_client_secret
#PPS_google_maps_api_key
#PPS_facebook_app_id
#PPS_facebook_app_secret

As always, see PinPointShooting.com to see just how little it does so far (if it happens to be running), or more exciting: go see the git repoitself.

The post Giving My App Secrets to the AWS SecretsManager appeared first on Learning Rust.

Scraping off the Dust: Redeploy of my Rust web app

Jeff Culverhouse — Thu, 02 Jul 2020 17:11:23 +0000

Life has been busy – no apologies or excuses, but, ya know, it’s 2020. Yet, I’m trying to slowly make my way back into playing with Rust. I decided to move my EC2 instance from AWS-Linux to the Ubuntu image; for one, I got tired of fighting with LetsEncode to get it to renew my SSL cert every 3 months. Also, I wanted to see how a redeploy of my Rust web app would go and if it still worked (why wouldn’t it?). So, lets see how tough it is to get my environment back to the same place. I took some notes (in case I needed to restart, sigh), so let’s go through it.

Go back to Part 1to see what this fake web app is about and how I got here – I need to reread it myself! So, first, this is what I ended up needing to add to what I got from the default Ubuntu image:

sudo apt install build-essential checkinstall zlib1g-dev pig-config libssl-dev libpq-dev postgresql postgresql-contrib -y

Lots of that was needed in order to get OpenSSL installed, I was following along hints here. Continuing those instructions, I did:

cd /usr/local/src/
sudo wget https://www.openssl.org/source/openssl-1.1.1g.tar.gz
sudo tar -xf openssl-1.1.1g.tar.gz
cd openssl-1.1.1g
sudo ./config --prefix=/usr/local/ssl --openssldir=/usr/local/ssl shared zlib
sudo make
sudo make test
sudo make install
sudo echo "/usr/local/ssl/lib" > /etc/ld.so.conf.d/openssl-1.1.1g.conf
sudo ldconfig -v
sudo mv /usr/bin/c_rehash /usr/bin/c_rehash.backup
sudo mv /usr/bin/openssl /usr/bin/openssl.backup
sudo nano /etc/environment # to add "/usr/local/ssl/bin" to the PATH

Next, instead of solely storing my code on a potentially tenuous EC2 server, I wanted to keep it backed up on my Google Drive (or whatever you like, this solution works with MANY network storage). I used rclone for my Raspberry Pi photo frame so I was familiar with that already.

curl https://rclone.org/install.sh | sudo bash
rclone config # to add google drive and authorize it

Ok, the most fun step!!

curl https://sh.rustup.rs -sSf | sh
cd ~/projects/rust
git clone git@github.com:jculverhouse/pinpoint_shooting.git

I need nginx for my app

sudo apt install nginx
sudo service nginx start

And now the much more reliable LetsEncrypt using Ubuntu 18.04

# follow instructions at https://certbot.eff.org/lets-encrypt
# setup root cronjob to renew once/week

For my Rocket-powered Rust app, I followed some reminders here to connect it to nginx. Simple enough, really, mostly relevant:

...
server_name pinpointshooting.com; # managed by Certbot
location {
  proxy_pass http://127.0.0.1:3000;
}
...

What? Nginx still has TLS 1 and 1.1 turned on by default? Followed this and removed those, tested the config, and restarted nginx. All of that I checked with SSLLabs via https://www.ssllabs.com/ssltest/analyze.html :

sudo nano /etc/nginx/nginx.conf # to remove TLS1 TLS1.1 from any line
sudo nano /etc/letsencrypt/options-ssl-nginx.conf # to remove TLS1 TLS1.1 from any line
sudo nginx -t
sudo service nginx reload

I’ll need Postgres for my PinpointShooting app as well, found some steps to follow here, plus I needed to setup for my own app and run the initial migrations to get it up-to-date. That involved another change so I could login with the password from a non-user-account.

cargo install diesel_cli --no-default-features --features posters
psql -d postgres -U postgres
  create user pinpoint password 'yeahrightgetyourown'; # save this in file .env in app dir
  create database pinpoint;
  grant all privileges on database pinpoint to pinpoint;
sudo nano /etc/postgresql/10/main/pg\_hba.conf # to edit "local all all" line to be md5 instead of peer
sudo service postgresql restart
psql -d postgres -U pinpoint # to test password above; just exit
cd ~/projects/rust/pinpointshooting
diesel migration run

Finally:

rustup default nightly # because Rocket, sigh...
cargo update
cargo build --release
target/release/pps &

And, we’re back online! Turns out, a redeploy of my Rust web app was about as easy as I could expect! If the app happens to be running, check it out here (though, there isn’t much to see or anything to do): pinpointshooting.com. Also, browse the repo and feel free to send me comments on how to be better about using idiomatic Rust!

The post Scraping off the Dust: Redeploy of my Rust web app appeared first on Learning Rust.

OAuth Requests, APIs, Diesel, and Sessions

Jeff Culverhouse — Fri, 09 Aug 2019 13:26:43 +0000

Things are starting to intertwine…

Some big changes in the repository just recently. I added Google Signin and Facebook Signin OAuth connections. I’m thinking I may not even configure an internal password on the site for users and instead just require one of those options. Probably I’ll add more, like 500px.com and/or Flickr, given the site’s purpose. A password field is still in my database though, so I haven’t given up the idea completely. Also, the OAuth requests create accounts using Diesel.

Users (now identified as shooters – we photographers haven’t given up that term) – are now written to the db. I really fought with one Diesel feature, so that bit is still commented out in the code. In addition, I added my first API to POST to – so another step with the Rocketcrate as well! I’d like to work my way into playing with a GraphQL endpoint so I can play with that as well!! (What’s the limit on Crate dependencies in a project anyway?!) I’m starting to think I won’t be able to tackle all of this in a single post – but let start!

OAuth vs Old and New Accounts

When a user arrives on the site, I check for a cookie with a session id (see my previous post). I decided, for now, I would use the User Agent (plus some random characters) as a fingerprint for creating the session id. So, when I am able to get a session_id from a cookie, I want to verify the User Agent is the same and that the session hasn’t expired. If the user arrives brand new, without a cookie, I immediately create an empty, no-user session for them. All of this is done, for now, right at the top of my index() route.

<src/routes.rs>

...
#[get("/")]
pub fn index(mut cookies: Cookies, nginx: Nginx) -> 
    rocket_contrib::templates::Template
{
    let session = get_or_setup_session(&mut cookies, &nginx);
...

After the index page loads, it shows the Google and Facebook Sign In buttons. Clicking one of those, they do the validation dance and get permission from the user. When that is granted, my app gets a token back which I send up to the server via a POST to /api/v1/tokensignin.

<src/api.rs>

use rocket::{http::Cookies, post, request::Form, FromForm};
use crate::oauth::*;
use crate::routes::Nginx;
use crate::session::*;
#[derive(FromForm)]
pub struct OAuthReq {
    pub g_token: Option<String>,  // google login req
    pub fb_token: Option<String>, // facebook login req`
    pub name: String,
    pub email: String,
}
#[post("/api/v1/tokensignin", data = "<oauth_req>")]
pub fn tokensignin(mut cookies: Cookies, nginx: Nginx,
        oauth_req: Form<OAuthReq>) -> String
{
    let mut session = get_or_setup_session(&mut cookies, &nginx);
    if let Some(token) = &oauth_req.g_token {
        match verify_google_oauth(&mut session, &token,
            &oauth_req.name, &oauth_req.email)
        {
            true => {
                session.google_oauth = true;
                save_session_to_ddb(&mut session);
                "success".to_string()
            }
            false => {
                session.google_oauth = false;
                save_session_to_ddb(&mut session);
                "failed".to_string()
            }
        }
    } else if let Some(token) = &oauth_req.fb_token {
        match verify_facebook_oauth(&mut session, &token,
            &oauth_req.name, &oauth_req.email)
        {
            true => {
                session.facebook_oauth = true;
                save_session_to_ddb(&mut session);
                "success".to_string()
            }
            false => {
                session.facebook_oauth = false;
                save_session_to_ddb(&mut session);
                "failed".to_string()
            }
        }
    } else {
        "no token sent".to_string()
    }
}

OAuth Requests via HTTP POSTs

This is how you allow for a POST and form data to come in – you setup a struct (OAuthReq in my example) of what you expect and bring that in as an input param. Plus, I am also bringing in any cookies that arrive with the request plus some Nginx headers so I have access to UserAgent. In the code so far, I’m either verifying a Google or Facebook token. Let’s look at the Google example (the Facebook one is nearly the same). Here are the relevant parts, but I’ll break some pieces down and go through it:

<src/oauth.rs>

...
pub fn verify_google_oauth(
    session: &mut Session,
    token: &String,
    name: &String,
    email: &String,
) -> bool {
    let mut google = google_signin::Client::new();
    google.audiences.push(CONFIG.google_api_client_id.clone());
    let id_info = google.verify(&token).expect("Expected token to be valid");
    let token = id_info.sub.clone();
    verify_token(session, "google".to_string(), &token, &name, &email)
}

Which leads right away to a big match:

fn verify_token(
    session: &mut Session,
    vendor: String,
    token: &String,
    name: &String,
    email: &String,
) -> bool {
    use crate::schema::oauth::dsl::*;
    use crate::schema::shooter::dsl::*;
    let connection = connect_pgsql();
    match oauth
        .filter(oauth_vendor.eq(&vendor))
        .filter(oauth_user.eq(&token))
        .first::<Oauth>(&connection)
    {

With the OK arm:

        // token WAS found in oauth table
        Ok(o) => {
            if let Some(id) = session.shooter_id {
                if id == o.shooter_id {
                    return true;
                } else {
                    return false;
                }
            } else {
                // log in user - what IS the problem with BelongsTo!?
                //if let Ok(s) = Shooter::belonging_to(&o)
                //    .load::<Shooter>(&connection)
                //{
                //    session.shooter_id = Some(shooter.shooter_id);
                //    session.shooter_name = Some(shooter.shooter_name);
                //    session.email_address = Some(shooter.email);
                return true;
                //} else {
                //    return false;
                //}
            }
        }

And the ERR arms:

        // token not found in oauth table
        Err(diesel::NotFound) => match session.shooter_id {
            Some(id) => {
                create_oauth(&connection, &vendor, token, id);
                true
            }
            None => match shooter
                .filter(shooter_email.eq(&email))
                .first::<Shooter>(&connection)
            {
                // email address WAS found in shooter table
                Ok(s) => {
                    create_oauth(&connection, &vendor, token, s.shooter_id);
                    true
                }
                // email address not found in shooter table
                Err(diesel::NotFound) => {
                    let this_shooter =
                        create_shooter(&connection, name, None,
                            email, &"active".to_string());
                    session.shooter_id = Some(this_shooter.shooter_id);
                    create_oauth(&connection, &vendor, token,
                        this_shooter.shooter_id);
                    true
                }
                Err(e) => {
                    panic!("Database error {}", e);
                }
            },
        },
        Err(e) => {
            panic!("Database error {}", e);
        }
    }
}

Simple Queries with Diesel

Breaking all that code down to smaller bits: first, I query the PgSQL database for the given oauth user:

match oauth
    .filter(oauth_vendor.eq(&vendor))
    .filter(oauth_user.eq(&token))
    .first::<Oauth>(&connection) {
        Ok(o) => { ... }
        Err(diesel::NotFound) => { ... }
        Err(e) => { ... }
}

Check the oauth table for records WHERE (filter) the oauth_vendor is (google or facebook) AND I’ve already stored the same validated oauth_user. I will get back either Ok(o) or Err(diesel::NotFound) … (or some worse error message), so I make a pattern with those 3 arms.

If we did get a hit from the DB, that session id is already tied to a shooter_id (user id) unless something is very wrong. So, IF we also have a shooter_id defined in our current session, I just need to verify that they match and return true or false. But, if we don’t have a shooter_id in our session, we know the oauth is tied to a shooter in the db, so this will log them in. Diesel has an easy way to get that parent record, which is what this should do:

// if let Ok(s) = Shooter::belonging\_to(&o).load::\<Shooter\>(&connection) {
...

I fought and fought to get this work, but you can see it is still commented out. From posts and chat around the Internet, I believe it can work – I think I either have a scope problem or my models aren’t setup correctly… this is how they look:

<src/model.rs>

...
#[derive(Identifiable, Queryable, Debug, PartialEq)]
#[table_name = "shooter"]
#[primary_key("shooter_id")]
pub struct Shooter {
    pub shooter_id: i32,
    pub shooter_name: String,
    pub shooter_password: String,
    pub shooter_status: String,
    pub shooter_email: String,
    pub shooter_real_name: String,
    pub shooter_create_time: chrono::NaiveDateTime,
    pub shooter_active_time: Option<chrono::NaiveDateTime>,
    pub shooter_inactive_time: Option<chrono::NaiveDateTime>,
    pub shooter_remove_time: Option<chrono::NaiveDateTime>,
    pub shooter_modify_time: chrono::NaiveDateTime,
}
...
#[derive(Identifiable, Associations, Queryable, Debug, PartialEq)]
#[belongs_to(Shooter, foreign_key = "shooter_id")]
#[table_name = "oauth"]
#[primary_key("oauth_id")]
pub struct Oauth {
    pub oauth_id: i32,
    pub oauth_vendor: String,
    pub oauth_user: String,
    pub shooter_id: i32,
    pub oauth_status: String,
    pub oauth_create_time: chrono::NaiveDateTime,
    pub oauth_last_use_time: chrono::NaiveDateTime,
    pub oauth_modify_time: chrono::NaiveDateTime,
}

I’ll get it to work eventually – I really hope it isn’t failing because I didn’t specifically name my primary fields just id like in the examples Diesel gives in their guides. It seems like naming shooter_id in table oauth to match shooter_id in the shooter table should make things obvious. Hopefully we aren’t forced to always use id as the primary field… no, that can’t be it.

Anyway, back to verifying. The other main case is that an oauth record with this token is NOT found in the table. Which means it is a new connection we haven’t seen before. If the session is already logged in, we just need to attach this oauth token to the logged in user and return true!

Some(id) => {
    create_oauth(&connection, &vendor, token, id);
    true
}

Otherwise, two choices – we will try to match on an existing shooter via the email address. If we find a match, we log them in and again attach this oauth token to their shooter record.

 None => match shooter
     .filter(shooter_email.eq(&email))
     .first::<Shooter>(&connection)
 {
     // email address WAS found in shooter table
     Ok(s) => {
         create_oauth(&connection, &vendor, token, s.shooter_id);
         true
     }

Otherwise, we don’t get a hit; that is, we haven’t seen this oauth token before AND we haven’t seen this validated email address before. We have to call that a brand new shooter account. I mentioned we create accounts from the OAuth requests using Diesel – this is where that happens. In this case, we create both the shooter record and the oauth record, linking them together.

// email address not found in shooter table
Err(diesel::NotFound) => {
    let this_shooter =
        create_shooter(&connection, name, None, email,
            &"active".to_string());
    session.shooter_id = Some(this_shooter.shooter_id);
    create_oauth(&connection, &vendor, token, this_shooter.shooter_id);
    true
}

Using Diesel to Insert Records

As we fall back out of the stack of functions we’ve called, because we return true here the session will get updated with the shooter_id – they are now logged in. Also, the shooter and oauth records are saved, so if they come back, they can just validate and be logged into their same account again. Here are the two methods that create those records:

<src/shooter.rs>

...
pub fn create_shooter<'a>(
    connection: &PgConnection,
    name: &'a String,
    password: Option<&'a String>,
    email: &'a String,
    status: &'a String,
) -> Shooter {
    use crate::schema::shooter::dsl::*;
    let new_shooter = NewShooter {
        shooter_name: name.to_string(),
        shooter_password: match password {
            Some(p) => p.to_string(),
            None => thread_rng()
                .sample_iter(&Alphanumeric)
                .take(64)
                .collect::<String>(),
        },
        shooter_status: status.to_string(),
        shooter_email: email.to_string(),
        shooter_real_name: name.to_string(),
    };
    diesel::insert_into(shooter)
        .values(&new_shooter)
        .get_result(connection)
        .expect("Error saving new Shooter")
}

<src/oauth.rs>

...
pub fn create_oauth<'a>(
    connection: &PgConnection,
    vendor: &'a String,
    user_id: &'a String,
    shooterid: i32,
) -> Oauth {
    use crate::schema::oauth::dsl::*;
    let new_oauth = NewOauth {
        oauth_vendor: vendor.to_string(),
        oauth_user: user_id.to_string(),
        shooter_id: shooterid,
    };
    diesel::insert_into(oauth)
        .values(&new_oauth)
        .get_result(connection)
        .expect("Error saving new Oauth")
}

As far as writing these new records to PgSQL – in both cases, we have NewShooter and NewOauth structs that allow us to set the bare minimum of fields without having to worry about the fields that PgSQL will default for us (like the create_date fields). We setup the appropriate struct and pass it to insert_into(). Adding .get_result() will return the newly created record to us, so we have access to the brand new shooter_id or oauth_id.

Complexity

If a user comes to the site, signs in with one OAuth (which creates their shooter record and attaches that oauth token) and then signs in with the other, this logic figures out they are validated to be the same person, so creates just a single shooter record with two oauth records, and both point to the one user. If they come back, they can authenticate via either third-party and are allowed back in.

Ok, more to come as I figure out other problems. I haven’t gone through that logic tightly enough to make sure I don’t have any holes – and it wouldn’t surprise me to find some. It doesn’t really matter – this is certainly teaching me Rust! Give it a try at PinpointShooting.com – but don’t be surprised if you shooter account gets deleted, constantly.

The post OAuth Requests, APIs, Diesel, and Sessions appeared first on Learning Rust.

Rust Web App Session Management with AWS

Jeff Culverhouse — Sun, 04 Aug 2019 17:35:17 +0000

Didn’t someone mention a cookie too!? Where’s my cookie!?

I’ve worked on some more bits with my pretend web application I introduced a few posts ago. Thinking about the technology needed to let users register and later log in, session management seems like a good first step to take. I would like to store the sessions in Amazon’s DynamoDB too, just for fun and experience. I’ve pushed several commits to the repository lately, so let me go through some of the changes enabling Rust web app session management with AWS DynamoDB.

First, I searched and read up a bit on best practices for session management. I found a Session Management Cheat Sheet from OWASP. Also, more generically, 12 Best Practices for User Account, Authorization, and Password Management on the Google Cloud Blog. I haven’t addressed many of these yet, but I’ll come back to resources like this several times in the future. I was trying to find language-independent whitepapers or guides on web app session management… and something not a decade old. If anyone has a definitive resource for this, please share!

Rusoto to the Rescue

First, let’s get our web application interfacing with AWS. Rusoto is a big set of crates for using AWS services. Rusoto Core (of course) and DynamoDB are all I need for now, but I am sure I’ll come back for more! To get set up, I created a sessions table in my AWS console and created a programmatic IAM role called pinpointshooting which has full access to the table. When creating a programmatic role, you are given credential access keys which I put into my .env file at the root of the pinpointshooting project. First, connecting to my AWS account with those access keys is a simple matter of:

DynamoDbClient::new(Region::UsEast1)

I had to think about the process I would use to store the session id in a cookie, and when/how it gets created, verified, updated, and deleted. I came up with a simple sketch to make it more clear in my mind. Note that I mention a cache both in my sketch and in some of the comments, but I haven’t done anything about that yet.

When a user arrives with no cookie – we need to create a session with some appropriate defaults, write it to the db, and send the cookie in the resulting response.
If the user arrives with a cookie – we need to pull the session from the DB (if it really exists) and do some verification and expiration checking. If everything looks good, we pull the session details into our struct. Otherwise, we delete that invalid session data and create a new session like above.
When the user selects to log out – we again check to make sure the session is valid, but then delete the session, delete the cookie, and log the user out.

Here are those steps above, as I have them in code at this point!

Step 1: Search for or Create the Session

// Check for sessid cookie and verify session or create new
// session to use - either way, return the session struct

pub fn get_or_setup_session(cookies: &mut Cookies) -> Session {
    let applogger = &LOGGING.logger;
    let dynamodb = connect_dynamodb();

    // if we can pull sessid from a cookie and validate it,
    // pull session from cache or from storage and return
    if let Some(cookie) = cookies.get_private("sessid") {
        debug!(applogger, "Cookie found, verifying"; "sessid" => cookie.value());
        // verify from dynamodb, update session with last-access if good
        if let Some(mut session) = 
        verify_session_in_ddb(&dynamodb, &cookie.value().to_string()) {
            save_session_to_ddb(&dynamodb, &mut session);
            return session;
        }
     }
     // otherwise, start a new, empty session to use for this user
    let mut hasher = Sha256::new();
    let randstr: String = thread_rng()
        .sample_iter(&Alphanumeric)
        .take(256)
        .collect();
    hasher.input(randstr);
    let sessid = format!("{:x}", hasher.result());
    cookies.add_private(Cookie::new("sessid", sessid.clone()));
    let mut session = Session {
        sessid: sessid.clone(),
        ..Default::default(),
    };
    save_session_to_ddb(&dynamodb, &mut session);
    session
}

This will check the (private) cookie sessid, if it came along with the http request, and try to fetch it from DynamoDB and then verify it (see below). If that succeeds, we update the session (with an updated last-access timestamp) and return the session struct. Otherwise, we create a new session id, with some appropriate defaults for now, by getting a hash of a random string of characters. Hrm, I should make sure I didn’t just happen to intersect with an existing sessid at this point! Anyway, we then just add (or update) the sessid cookie to be returned with the http response. Since we are using the private cookie feature of Rocket, the value is actually encrypted. The user can’t see what their real session id is or try to manufacture one. Lastly, if we just created a new session, we save it to the session table and return it.

Step 2: Verify a Session

// Search for sessid in dynamodb and verify session if found
// including to see if it has expired
fn verify_session_in_ddb(dynamodb: &DynamoDbClient, sessid: &String) -> Option<Session> {
    let applogger = &LOGGING.logger;
    let av = AttributeValue {
        s: Some(sessid.clone()),
        ..Default::default(),
    };
    let mut key = HashMap::new();
    key.insert("sessid".to_string(), av);
    let get_item_input = GetItemInput {
        table_name: "session".to_string(),
        key: key,
        ..Default::default()
    };
    match dynamodb.get_item(get_item_input).sync() {
        Ok(item_output) => match item_output.item {
            Some(item) => match item.get("session") {
                Some(session) => match &session.s {
                    Some(string) => {
                        let session: Session = serde_json::from_str(&string)
                            .unwrap();
                        match session.last_access {
                            Some(last) => {
                                if last > Utc::now() - 
                                Duration::minutes(CONFIG.sessions.expire) {
                                    Some(session)
                                } else {
                                    debug!(applogger, "Session expired";
                                        "sessid" =\> sessid); 
                                    delete_session_in_ddb(dynamodb, sessid);
                                    None
                                }
                            }
                            None => {
                                debug!(applogger, "'last_access' is blank for stored session"; "sessid" => sessid);
                                delete_session_in_ddb(dynamodb, sessid);
                                None
                            }
                        }
                    }
                    None => {
                        debug!(applogger, "'session' attribute is empty for stored session"; "sessid" => sessid);
                        delete_session_in_ddb(dynamodb, sessid);
                        None
                    }
                },
                None => {
                    debug!(applogger, "No 'session' attribute found for stored session"; "sessid" => sessid);
                    delete_session_in_ddb(dynamodb, sessid);
                    None
                }
            },
            None => {
                debug!(applogger, "Session not found in dynamodb"; "sessid" => sessid);
                None
            }
        },
        Err(e) => {
            crit!(applogger, "Error in dynamodb"; "err" => e.to_string());
            panic!("Error in dynamodb: {}", e.to\_string());
        }
    }
}

That’s some deep nesting – I’m still a newbie at Rust coding, so there is probably a better way to write this. If I didn’t care about logging the problems, I could probably shorten this by using the “?” operator at the end of each step. Maybe I’ll change that eventually, but for now I need to debugging logs to see that things are working.

This chain tries to retrieve the session attribute associated with the sessid (session id encoded in the cookie) and for now, just makes sure it isn’t too old. If that simple check is ok, we deserialize and return that Some(session struct). In all other cases, we return None and a new session will be generated. For cases where a session was present, but determined to be invalid or expired, we also delete the session from DynamoDB.

Step 3: Save (or Update) the Session

// Write current session to dynamodb, update last-access date/time too
fn save_session_to_ddb(dynamodb: &DynamoDbClient, session: &mut Session) {
    let applogger = &LOGGING.logger;
    session.last_access = Some(Utc::now());
    let sessid_av = AttributeValue {
        s: Some(session.sessid.clone()),
        ..Default::default()
    };
    let session_av = AttributeValue {
        s: Some(serde_json::to_string(&session).unwrap()),
        ..Default::default()
    };
    let mut item = HashMap::new();
    item.insert("sessid".to_string(), sessid_av);
    item.insert("session".to_string(), session_av);
    let put_item_input = PutItemInput {
        table_name: "session".to_string(),
        item: item,
        ..Default::default()
     };
     match dynamodb.put_item(put_item_input).sync() {
         Ok(_) => {}
         Err(e) => {
             crit!(applogger, "Error in dynamodb"; "err" => e.to_string());
             panic!("Error in dynamodb: {}", e.to_string());
         }
     };
}

Here, we simply take the session data we are passed, update the last_access to now() and then write the serialized session struct to DynamoDB. Also note, I’m storing the session id inside the session data itself. I’m not sure yet if this is handy duplication, a security concern, or irrelevant.

Step 4: Clean-up After Ourselves

Hey! Are you a security concern??

Just for completeness, here is the function to drop a session from DynamoDB once we have determined it is invalid (expired).

// Delete session from dynamodb
fn delete_session_in_ddb(dynamodb: &DynamoDbClient, sessid: &String) {
    let applogger = &LOGGING.logger;
    let av = AttributeValue {
        s: Some(sessid.clone()),
        ..Default::default()
    };
    let mut key = HashMap::new();
    key.insert("sessid".to_string(), av);
    let delete_item_input = DeleteItemInput {
        table_name: "session".to_string(),
        key: key,
        ..Default::default()
    };
    match dynamodb.delete_item(delete_item_input).sync() {
        Ok(_) => {
            debug!(applogger, "Deleted invalid session from ddb";
                "sessid" => sessid);
        }
        Err(e) => {
            crit!(applogger, "Error in dynamodb"; "err" => e.to_string());
            panic!("Error in dynamodb: {}", e.to_string());
        }
    };
}

Lots, lots, lots more to do – but I’m having fun with this. It might go faster if I actually had a plan for what this web app actually does. I mean, I have an idea, but I haven’t mocked up a single page so I’m going slow and just enjoying hacking out Rust code as I go! Thanks for coming along with me!

The post Rust Web App Session Management with AWS appeared first on Learning Rust.

Rust Functions, Modules, Packages, Crates, and You

Jeff Culverhouse — Tue, 23 Jul 2019 13:00:07 +0000

I know the code is in here… somewhere.

Come to find out, I’m learning Rust from old documentation. Both of the printed Rust books I have are for the pre-“2018 edition” and I think that’s contributing to some confusion I have about functions, modules, packages, and crates. A new version of the official book is coming out in the next month or so – I have a link to it through Amazon in the right sidebar. If you’ve been reading the online documentation, you’re ok – it is updated fir the “2018-edition”. I’ve looked at some of these parts of Rust before, but I recently found another new resource, the Edition Guide, which clears up some of my issues. Especially of interest here, is the section on Path Clarity which heavily influenced by RFC 2126 that improved this part of Rust.

I learned some of the history (and excitement) of RFC 2126 while listening to the Request for Explanation podcast, episode 10. Anyway, let’s go back to basics and have a look at Rust functions, modules, packages and crates as the language sits in mid-2019. I’ll present some examples from my web application we’ve been looking at. I’m going to cut out unnecessary bits to simplify things, so a “…” means there was more there in order for this to compile. You can always see whatever state it happens to be in, here.

Crates and Packages

A Rust crate (like Rocket or Diesel) is a binary or library of compiled code. A binary crate is runnable while a library crate is used for its functionality by being linked with another binary. A package (like my web app) ties together one or more crates with a single Cargo.toml file. The toml file configures the package ‘s dependencies and some minimal information about compiling the source. A binary crate will have a src/main.rs with a main() function which directs how the binary runs. A library crate will have a src/lib.rs which is the top layer of the library. This top layer directs which pieces inside are available to users of the library.

Rust Functions

Functions are easy – subroutines in your source code. A function starts with fn, possibly receives some parameters and might return a value. Also, a function may be scoped as public or kept private. The main() function inside src/main.rs is a special function that runs when the binary is called from the command line. It dictates the start of your program and you take control from there. You may create other functions, just avoid reserved words (or use the r# prefix to indicate you mean YOUR function, not the reserved word, for instance r#expect if you want to name a function “expect”). Very similar to functions, are methods and traits, which we’ve looked at before.

<src/lib.rs>

...
use diesel::prelude::*;
...
pub fn setup_db() -> PgConnection {
    PgConnection::establish(&CONFIG.database_url)
        .expect(&format!("Error connecting to db"))
}

setup_db() is a fairly simple function – it accepts no incoming parameters and returns a database connection struct called PgConnection. It has pub before fn to indicate it is a “public” function. Without that, my web application bin/src/pps.rs could not call this function – it would not be in scope. Without pub, setup_db() would only be callable from within src/lib.rs. Since I am designing my application as a library crate, I choose to put setup_db() in the main src/lib.rs file. My binary that I will use to “run” my web application is in src/bin/pps.rs and contains a main() function.

Let’s look at the return type, PgConnection. This is a struct defined by the database ORM library crate, Diesel. The only way I could write a function that returns this particular type of struct is because I have use diesel::prelude::*; at the top (and it’s in the toml file as well). The Diesel library crate provides prelude as a simple way to bring in all Diesel has to offer my package. Diesel provides the PgConnection struct as public (or what good would the crate be), so I can now use that struct in my code. This also gives me the (method or trait, how can you tell?) establish(). Just like you’d call String::new() for a new string, I’m calling PgConnection::establish() for a new database connection and then returning it (see, no trailing ; on the line).

Rust Modules

Functions (and other things) can be grouped together into a Module. For instance, setup_logging() is also in src/lib.rs. However, I could have wrapped it inside a named module, like so:

<src/lib.rs>

...
pub mod setting_up {
...
    use logging::LOGGING;
    use settings::CONFIG;
    pub fn setup_logging() {
        let applogger = &LOGGING.logger;
        let run_level = &CONFIG.server.run_level;
        warn!(applogger, "Service starting"; "run_level" => run_level);
     }
}

Now it is part of my setting_up module. Here also, the module needs to be pub so that my application can use it and the public functions inside it. Now all of the enums and structs and functions inside the module setting_up are contained together. As long as they are public, I can still get to them in my application.

Notice I use logging::LOGGING; and use settings::CONFIG; These bring in those two structs so I can use the global statics that are built when then the application starts. I included pub mod logging; and pub mod settings; at the top level, in src/lib.rs, so they are available anyplace deeper in my app. I just need to use them since I reference them in this module’s code.

Split, for Clarity

On the other hand, instead of defining a module, or multiple modules, inside a single file like above, you can use a different file to signify a module. This helps split out and separate your code, making it easier to take in a bit at a time. I did that here, with logging.rs:

<src/logging.rs>

...
use slog::{FnValue, *};

pub struct Logging {
    pub logger: slog::Logger,
}

pub static LOGGING: Lazy<Logging> = Lazy::new(|| {
    let logconfig = &CONFIG.logconfig;
    let logfile = &logconfig.applog_path;

    let file = OpenOptions::new()
        .create(true)
        .write(true)
        .truncate(true)
        .open(logfile)
        .unwrap();
    let applogger = slog::Logger::root(
        Mutex::new(slog_bunyan::default(file)).fuse(),
        o!("location" => FnValue(move |info| {
            format!("{}:{} {}", info.file(), info.line(), info.module(), )
          })
        ),
    );
    Logging { logger: applogger } });

I have a struct and a static instance of it, both of them public, defined in logging.rs. logging.rs becomes a module of my library crate when I specify it. At the top of src/lib.rs I have pub mod logging; which indicates my library crate uses that module file logging.rs and “exports” what it gets from that module as public (so my bin/src/pps.rs application can use what it provides).

In this case, you also see I use slog::{FnValue, *}}; which is like use slog::FnValue; (which I need for the FnValue struct) and use slog::*; which gives me the fuse struct and the o! macro. I was able to combine those into a single use statement to get just what I needed from that external crate.

The old books I have been referencing have you declaring the third-party crates you want to use in your application in your Cargo.toml file (which is still required), but also you’d have to bring each one in with an extern crate each_crate; at the top of main.rs or lib.rs. Thankfully, that’s no longer needed… 99% of the time. In fact, I had a long list of those myself – I am surprised cargo build didn’t warn me it was unneeded. Actually, I do have one crate I am using which still needs this “2015-edition” requirement: Diesel. Apparently, it is doing some fancy macro work and/or hasn’t been upgraded (yet?) for the “2018-edition” of Rust, so at the top of src/lib.rs, I have:

#[macro_use]
extern crate diesel;

A Few Standards and TOMLs

The Rust crate std is the standard library, and is included automatically. The primitive data types and a healthy list of macros and keywords are all included. But, if you need filesystem tools: use std::fs; and if you need a HashMap variable, you’ll need to use std::collections::HashMap; And yes, all external crates you depend on inside your source will need to be listed in Cargo.toml. This configuration helps you though – it updates crates automatically as minor versions become available, but does NOT update if a major version is released. You will need to do that manually, so you can test to see if the major release broke anything you depended on in your code. Here is a piece of my ever-growing Cargo.toml file for the web application so far:

...
[dependencies]
    slog = "2.5.0"
    slog-bunyan = "2.1.0"
    base64 = "0.10.1"
    rand = "0.7.0"
    rand_core = "0.5.0"
    rust-crypto = "0.2.36"
    config = "0.9.3"
    serde = "1.0.94"
    serde_derive = "1.0.94"
    serde_json = "1.0.40"
    once_cell = "0.2.2"
    dotenv = "0.14.1"
    chrono = "0.4.7"
    rocket = "0.4.2"
    rocket-slog = "0.4.0"

[dependencies.diesel]
    version = "1.4.2"
    features = ["postgres","chrono"]

[dependencies.rocket_contrib]
    version = "0.4.2"
    default-features = false
    features = ["serve","handlebars_templates","helmet","json"]

The post Rust Functions, Modules, Packages, Crates, and You appeared first on Learning Rust.

Better Logging for the Web Application

Jeff Culverhouse — Sat, 20 Jul 2019 14:00:14 +0000

See… structured logging…

When I last left my sample web application, I was doing simple logging to the terminal which is not useful for very long. I want structured logging and I want the app and Rocket to write to (separate) files. So, let’s switch out the crates log and simple_log for slog and sloggers to get better logging for my web application.

Application Logging

Slog‘s “ambition is to be The Logging Library for Rust” and it sure seems adaptable. There are several “helper” crates to log to the terminal or syslog, log as json, etc. However, slog is complex! Which seems to be the main reason sloggers came about. Sloggers brings you the most useful features of slog without the complex setup. It happens to include an example of configuring via an inline TOML string. I’ve already got my CONFIG global pulling from a TOML file, so I add this to conf/development.toml:

<conf/development.toml>

...
[logconfig]
type = "file"
format = "compact"
source_location = "module_and_line"
timezone = "local"
level = "debug"
path = "log/error.log"
rotate_size = 1073741824
rotate_keep = 2

[webservice]
weblog_path = "log/access.log"
...

Plus, I add logging.rs to set up the global LOGGING instance so everyone can log; just like everyone can pull from the global CONFIG instance:

<logging.rs>

use crate::settings::CONFIG;
use crate::sloggers::{Build, Config};
use once_cell::sync::Lazy;

#[derive(Debug)]
pub struct Logging {
    pub logger: slog::Logger,
}

pub static LOGGING: Lazy<Logging> = Lazy::new(|| {
    let logconfig = &CONFIG.logconfig;
    let builder = logconfig.try_to_builder().unwrap();
    let logger = builder.build().unwrap();

    Logging { logger: logger }
});

Sloggers is really easy to use to set up a slog instance. Here, I simply pull the logconfig from my global CONFIG, build the logger, and store it in my new OnceCell Lazy LOGGING global.

And then, Web Access Logging

At this point, application logging is going to my new log/error.log file, but Rocket logging is still coming to the screen. This is actually good – as I mentioned, I want to have them going to separate files anyway. So now with some slogging experience, it’s a simple matter to set up a second file, specifically for Rocket. I do need to add yet another crate, this time rocket_slog – so that Rocket internals can glom on to the slog instance that I make. Here is the new start_webservice function:

<src/lib.src>

...
pub fn start_webservice() {
    let logger = &LOGGING.logger;
    let weblog_path = &CONFIG.webservice.weblog_path;
    let bind_address = &CONFIG.webservice.bind_address;
    let bind_port = &CONFIG.webservice.bind_port;

    // start rocket webservice
    let version = include_str!("version.txt");
    let mut builder = FileLoggerBuilder::new(weblog_path);

    builder.level(Severity::Debug);

    let weblogger = builder.build().unwrap();
    let fairing = SlogFairing::new(weblogger);
    warn!(logger, "Waiting for connections...";
        "address" => bind_address,
        "port" => bind_port,
        "version" => version);

    rocket::ignite()
        .attach(fairing)
        .attach(Template::fairing())
        .attach(SpaceHelmet::default())
        .mount("/", routes![routes::index, routes::favicon])
        .mount("/img", StaticFiles::from("src/view/static/img"))
        .mount("/css", StaticFiles::from("src/view/static/css"))
        .mount("/js", StaticFiles::from("src/view/static/js"))
        .launch();
}

Now I have log/error.log for application logging (I’ll rename that later to log/app.log) and log/access.log for Rocket logging. But I still don’t have structured logs!

Finally, Structured Logs

I try several ways (with my limited Rust knowledge and experience) but it appears that with sloggers ease of use, you trade away options if you want to get fancy. At least, I couldn’t figure out how to get slog_json or something called slog_bunyan to work with sloggers. It looks like I have to deal with slog directly.

Later, on another night, I tackle these changes. I dump sloggers and, though much searching around, I end up with a merge of two examples I find (from slog and slog-bunyan). Thankfully, I am getting a little better at understanding the API docs that are posted for crates! Anyway, here is how the app logging setup looks now:

<src/logging.rs>

...
pub static LOGGING: Lazy<Logging> = Lazy::new(|| {
    let logconfig = &CONFIG.logconfig;
    let logfile = &logconfig.applog_path;
    let file = OpenOptions::new()
        .create(true)
        .write(true)
        .truncate(true)
        .open(logfile)
        .unwrap();
    let applogger = slog::Logger::root(
        Mutex::new(slog_bunyan::default(file)).fuse(),
        o!("location" => FnValue(move |info| {
            format!("{}:{} {}", info.file(), info.line(), info.module(), )
          })
        ),
    );
    Logging { logger: applogger } });

Notice I still pull the applog_path from the CONFIG and then create that filehandle. Next, a single but complicated call into slog returns a Logger instance, which I store into my global. The o! macro, provided by the slog crate, lets me add a JSON field to every log record that gets written. I copy from an example and add the file, line number, and module name where the log was generated. I’ll probably come back to this later and add more data. Also, notice I am using slog_bunyan for the JSON formatting. slog-json actually recommends using slog_bunyan for a “more complete output format.” Bunyan logging seems to originate as a node.js module. Lastly, I also change the weblogger in the same way – now both are JSON logs and still log to separate files.

This actually wasn’t as complicated as I feared. I’ve lost some of the easily configurable sloggers features like auto file rotation, but I’ll figure that out as well – it’s probably actually just a slog feature that I need to enable.

Results, JSON Style

Here is how both look now – I think next post I’ll work on a JSON log viewer so we can watch these easier, as we develop.

<log/app.log>

{"msg":"Service starting","v":0,"name":"slog-rs","level":40,"time":"2019-07-20T03:40:13.781382592+00:00","hostname":"ip-169-254-169-254","pid":6368,"location":"src/lib.rs:48 ppslib","run_level":"development"}
{"msg":"Waiting for connections...","v":0,"name":"slog-rs","level":40,"time":"2019-07-20T03:40:13.784899718+00:00","hostname":"ip-169-254-169-254","pid":6368,"location":"src/lib.rs:76 ppslib","version":"0.1.0","port":3000,"address":"0.0.0.0"}

<log/access.log>

{"msg":"listening","v":0,"name":"slog-rs","level":30,"time":"2019-07-20T05:31:21.068959173+00:00","hostname":"ip-169-254-169-254","pid":7700,"address":"http://0.0.0.0:3000"}
{"msg":"request","v":0,"name":"slog-rs","level":30,"time":"2019-07-20T05:31:22.402815352+00:00","hostname":"ip-169-254-169-254","pid":7700,"uri":"\"/\"","method":"Get"}
{"msg":"response","v":0,"name":"slog-rs","level":30,"time":"2019-07-20T05:31:22.404425952+00:00","hostname":"ip-169-254-169-254","pid":7700,"status":"200 OK","route":"\u001b[32mGET\u001b[0m \u001b[34m/\u001b[0m \u001b[36m(\u001b[0m\u001b[35mindex\u001b[0m\u001b[36m)\u001b[0m"}

Again, if all that was complicated to follow and you prefer to look at the git commits as I went through this ordeal, check out the repository.

The post Better Logging for the Web Application appeared first on Learning Rust.

Using Match – Bob Revisited

Jeff Culverhouse — Wed, 17 Jul 2019 14:00:28 +0000

Wait, “TG”? That's not a word! Challenge!

Remember Bob from a previous post – our tiniest of AI-responders? I got feedback from the Exercism tutor that I had avoided one of Rust's most versatile of operators and I should think about using match. So, not to be a jerk, but just to see if I could replace ALL of the ifs in my method, I quickly came up with this solution.

Switch to Using Match

pub fn reply(message: &str) -> &str {
    let message = message.trim();
    let question = message.ends_with('?');
    let any_alpha = message.chars().any(char::is_alphabetic);
    let is_yelling = message.to_ascii_uppercase() == message;
    let is_empty = message.len() == 0;
    match message.is_empty() {
        true => "Fine. Be that way!",
        _ => match any_alpha && is_yelling {
            true => match question {
                true => "Calm down, I know what I'm doing!",
                _ => "Whoa, chill out!",
            },
            _ => match !is_empty && question {
                true => "Sure.",
                _ => "Whatever.",
            },
        },
    }
}

Come On Bob, Get With the Rust

So, it may be my newness to Rust, but I’m not sure version 2 is “more maintainable” code than what I started with. I had thought this Exercism task was just about learning some string methods. Turns out, it is also about learning the match operator. Of course, you wouldn’t code any parser or AI this way anyway – this IS just for practice.

Another tiny thing the tutor pointed out, I did have let check_message = message.trim() in my earlier code. A more idiomatic way in Rust is to reassign it to the same variable, hence the let message = message.trim() in this version. My guess is: less memory allocation, fewer variables for you to keep track of, and you aren’t able to incorrectly use the wrong variable later in the method. Actually, that probably isn’t a Rust idea – that’s just a good programming tip.

Bob With a Trait

Patterns are, I'm told, very powerful in Rust and that's a big reason to start using match wherever you can. I’ve also learned a bit about traits and impls which I talked about in an earlier post. And that got me thinking maybe I’d work through the syntax to get it working with a trait instead. That idea led to this code.

pub fn reply(message: &str) -> &str {
    enum Quality {
        Empty,
        YellQuestion,
        YellStatement,
        AskQuestion,
        Statement, };

    trait HasQuality {
        fn has_quality(&self) -> Quality;
    }

    impl HasQuality for str {
        fn has_quality(&self) -> Quality {
            let message = self.trim();
            let question = message.ends_with('?');
            let any_alpha = message.chars().any(char::is_alphabetic);
            let is_yelling = message.to_ascii_uppercase() == message;
            let is_empty = message.len() == 0;

            match message.is_empty() {
                true => Quality::Empty,
                _ => match any_alpha && is_yelling {
                    true => match question {
                        true => Quality::YellQuestion,
                        _ => Quality::YellStatement,
                    },
                    _ => match !is_empty && question {
                        true => Quality::AskQuestion,
                        _ => Quality::Statement,
                    },
                },
            }
        }
    };
    match message.has_quality() {
        Quality::Empty => "Fine. Be that way!",
        Quality::YellQuestion => "Calm down, I know what I'm doing!",
        Quality::YellStatement => "Whoa, chill out!",
        Quality::AskQuestion => "Sure.",
        _ => "Whatever.",
    }
}

So here, we come up with the idea of a Quality trait and then we implement that trait for the built-in str primitive. We’ve expanded what you can do to a str with our own trait! Version 3 of Bob really helps reveal some of the Rust thought patterns I need to hone.

Of course, I made sure Version 3 still passes all of the Exercism tests for this task. This change was approaching my limit of Rust knowledge to get it working without help from a book – just obeying every compiler complaint. However, I cranked this out much faster than previous Rust code, so I think some of this learning (and blogging) is sinking in! I surprised myself enough, that I posted this solution to Exercism as well. I want to hear what the tutor has to say about this method (no pun intended!). Now, I just need to remember to keep using match, traits, impls, and other Rust-supplied power!

Update: 2019-07-17 13:30

Woah, I just got some great advice from the Exercism tutor! You can match on a (expression, expression, ...) so check this out!

impl HasQuality for str {
    fn has_quality(&self) -> Quality {
        let message = self.trim();
        let question = message.ends_with('?');
        let any_alpha = message.chars().any(char::is\_alphabetic);
        let is_yelling = message.to_ascii_uppercase() == message;
        let is_empty = message.len() == 0;

        match (is_empty, any_alpha && is_yelling, question) {
            (true, _, _) => Quality::Empty,
            (false, true, true) => Quality::YellQuestion,
            (false, true, false) => Quality::YellStatement,
            (false, false, true) => Quality::AskQuestion,
            _ => Quality::Statement,
        }
    }
};

The post Using Match – Bob Revisited appeared first on Learning Rust.