Ben Hyrman's Blog

A Hitchhiker's Guide to Strategic Design

2023-07-15T00:00:00Z

In this post, I'd like to talk about strategic software design and aligning architecture and delivery with business needs. I'll provide a high-level introduction to some of the strategic aspects of Domain Driven Design (DDD), Event Sourcing, and Event-Driven Architecture (EDA) as a way to achieve long-term objectives. Not the only way... but a way.

My hope is that, by the end of this article, you'll have some ideas on how to approach challenges that arise in the balance of shipping software that lasts. Each section only barely scratches the surface of the topic but it will give you a starting point to dig deeper as you map out your own domain.

Not every software product is a good fit for domain driven design. Event Sourcing doesn't fit every problem space. Event-Driven Architecture isn't a magic bullet for scale problems. And, not every business will survive long enough to have scale problems.

Once a business has validated their market fit, started to clarify their core value stream, and is grappling with cross-team coordination, then the topics below will be worth considering.

Cash Rules Everything Around Me

— Wu-Tang Clan

Our "Simple" Scenario

Let's pretend that we work at a small, luxury hotel chain. This chain specializes in destination properties. Business comes from a mix of individual bookings, group bookings for weddings, and corporate retreats. Our CEO wants to modernize the business and has asked us to build a new system to manage bookings and reservations.

And Off We Go

But, wait. A word of warning before we start.

Domain Driven Design is slower than other approaches. It is more deliberate. It involves aligning business, product, and technology. You should use it when you want to build software that lasts. You should use it to focus on your core domain; your core differentiator and value proposition. It might be the follow-on implementation step once you've drawn out Wardley Maps and know where to best spend your engineering hours.

Do not use these techniques to build an MVP to validate market fit.

With that, let's get started with the first tools in our kit for building software that lasts.

As you start to work with the business and product owners on a hotel reservation system, you'll start to form a high-level picture of various aspects of running a hotel.

We can use some techniques from Domain Driven Design to help bring order and focus to these aspects.

Bounded Contexts

One of the challenges with delivering a solution is understanding the current process. As you talk to key stakeholders, you'll discover that the front desk personnel have separate functions from the group sales team. And, likewise, the housekeeping staff have different tasks than the maintenance team. They'll all have distinct workflows and language.

Put Like With Like

— literally every organization book ever

A Bounded Context is a way to group the common things together within a team or function. It's very hard to get it right the first time and you likely will iterate in several sessions with your stakeholders before boundaries crystalize. One tip from Vaughn Vernon is that a Bounded Context is a linguistic boundary. That is, the terms and language within a Bounded Context are consistent.

For example, the front desk staff will talk about reservations, check-ins, check-outs, and folios. The housekeeping staff will talk about floor assignments, dirty rooms, and late checkouts. While the group sales team will talk about room blocks, room rates, and room types.

Drawing these bounded contexts will help you to design the components of the system. If you are delivering a modular monolith then you'll want to keep these boundaries in mind as you design the modules. If you are delivering a microservice architecture then you'll want to keep these boundaries in mind as you design the services.

One more tip... odds are very good that the bounded contexts you identify will align with the organizational structure of the business. That's ok.

Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure.

— Melvin E. Conway

Ubiquitious Language

Concurrently with discovering the bounded contexts, you'll want to start to build a shared language with the business. This is called a Ubiquitous Language. Referring back to Mr. Vernon's tip about linguistic barriers, we should ensure that the organization refers to the same thing with the same label within a given context.

Imagine working in IT at a yoga chain, for example, where one part of the business calls an instructor-led session a "class" while another team calls it an "event" and yet another team might call it an "activity". This is a recipe for confusion and frustration.

We should avoid confusion and frustration.

For all of the teams working within a bounded context, we should develop and mature a ubiquitous language. We want developers, designers, product managers, and business stakeholders to have a common language; a common way to describe the problem. We want this to permeate so deeply that the language used in meetings is the language used in support tickets is the language used in the software code.

Some language will be common across the entire organization. For example, a "guest" means the same thing to the front desk, housekeeping, and maintenance.

Some language will be specific to a bounded context. For example, a "Do Not Rent" list is specific to the front desk.

(aside, you never want to be a "guest" on a "do not rent" list in any context)

Context Maps

After we've discovered the bounded contexts and started to develop a ubiquitous language, we'll want to start to map the relationships between the bounded contexts. This is called a Context Map.

A Context Map will show the synapses firing in the organizational brain; the impulses that cause other parts of the business to light up.

Within our reservation system, a "check-in" in the front desk context will cause a "room assignment" in the housekeeping context. It might also trigger a "forecast for breakfast" in the kitchen context.

Event Storming / Event Modeling

Domain Driven Design is, in many ways, focused on the flow of information through the business and capturing how that information is acted on to make decisions. It captures process and workflow. Software becomes an outcome that supports and automates these workflows.

Discovering and validating the workflow (aka, finding the "things everybody knows" and making sure everybody actually knows them) is difficult.

One way to do this, and surface your boundaries and assumptions at the same time, is through Event Storming sessions. In these sessions, you'll take one, or several, long-lived business processes and model the events that occur end-to-end. These sessions must involve the stakeholders, product owners, and tech.

Stay high-level in these sessions. You can go deeper into Event Storming within a team or single business function later. However, your initial session is to capture the strategic, organizational-wide view. Look for the handoffs between teams to complete critical business functions in the core value stream of the business.

At the end of the sessions, you should have clarified and validated your bounded contexts and context maps. Pay attention to the language people use when describing their process inputs and outputs. If you think you have solidly defined borders for your bounded contexts but the language within one of those borders isn't consistent then you may have found a new context... or at least you've identified that you need some workshops.

Event-Driven Architecture

We've talked about events quite a bit so far. This is no coincidence. In this context, events tend to be asynchronously communicated information. In many ways, this models the natural flow of information in an organization.... unless you have a boss that hovers over your desk until you complete your task.

Let's imagine that you're going to your favorite restaurant this weekend. You called to make a reservation. That's an event. You give the waiter your order. Another event. That waiter then gives the order to the kitchen. That is, you guessed it, another event.

All along the way, there's valuable information being passed and translated for each bounded context. But, no one is blocked and waiting for a response.

Going back to our event storming and context maps, we can see that exchanging information between bounded contexts is a useful paradigm. When it comes to the actual implementation of our software, we can translate this into an Event-Driven Architecture.

An Event-Driven Architecture is valuable on its own. While Domain Driven Design makes the most sense when focusing on our core domain, an Event- Driven Architecture is useful for communication between supporting systems as well. It allows us to independently scale and decouple capabilities. And, it allows us to build future capabilities that tap into existing information flow without impact.

But, there is also a real cost to adopting this architecture. If multiple services must coordinate to fulfill a user's request then you must plan and design for scenarios when one or more of those services has a stale view of the world because they are delayed in processing events or a myriad of other communication and processing issues.

Event Sourcing

While it is more tactical than the other topics we've covered, I wanted to cover Event Sourcing as well. I've been in too many conversations where Event Sourcing and Event-Driven Architecture are conflated and used interchangeably.

Event Sourcing is a technique where we model the current state of the system by capturing every decision that led to that state. As an overused, but classic, example, the current state (balance) of your bank account is the sum of all events (deposits and withdrawals) that have happened.

Event Sourcing is very powerful but should not be used indiscriminately. In the hotel reservation system within the front desk bounded context, event sourcing would probably be very useful for the guest folio. You'll want to know every charge and every payment that has been made. However, that same front desk bounded context does not care about the history of the guest... that they moved or changed their name. However, that information is useful to the group sales bounded context and it might make sense to event source that there. Meanwhile, the housekeeping system might not benefit from event sourcing at all.

Choose carefully. Event Sourcing is a great way to capture the "why" behind state changes in a system. It is focused on all of the small changes within a system. Meanwhile, events in an Event-Driven Architecture tend to be "chunkier".

Back to our reservation system, internally we might capture when a room is reserved, when the reservation is moved to a new room because of some request, when the guest checked in, and when the guest checked out. But, it might only publish the check-in and check-out events and those events may be enriched with more information than internal events (or, conversely, less information if you're publishing skinny events).

As an aside, if you adopt Event Sourcing, then you will likely also adopt Command Query Responsibility Segregation (CQRS). You can, of course, adopt CQRS on its own as well and I happen to think it's one of the easiest ways to simplify a system's architecture. As it's orthogonal to this blog post's topic, I'll leave it to another post.

Conclusion

My goal in this blog post was to provide a very high-level introduction to strategic DDD and some of the supporting tactical architectures that often go hand-in-hand with DDD.

Delivery speed, real or perceived, is slower with DDD; at least at first. In my opinion, the real value of DDD is in aligning people and teams. This alignment is critical in the core value streams of the organization. Unless you're an HR services company, your payroll system is not a differentiator. On the other hand, if you work at a hotel chain then the reservation system may be a core competitive advantage. Spend time on your advantage.

Even if you adopt nothing else from strategic DDD, there's immense value in capturing your organization's ubiquitious language. If you're not ready for bounded contexts and event storming then at least start with this.

Adopting an Event-Driven Architecture brings its own set of trade-offs. Asynchronous communication between disparate services can be more scalable but will also bring challenges and trade-offs around data exchange, eventual consistency, and data staleness.

Event Sourcing is a powerful approach to baking the reason for each state change into your system. And, while you get an audit log "for free" out of the implementation, you now need to think through how events will change and be versioned over time. Even within a single system within a bounded context, it is likely that not everything in that system would benefit from being event sourced.

But, in spite of the challenges, strategic DDD is a very valuable tool in aligning everyone around delivering the core value of your organization. And, it will ensure that you can best focus on what matters vs what is important but not part of your core domain.

Adopting an Event-Driven Architecture can happen independent of DDD but it shines as the bridge between bounded contexts identified by DDD. It allows teams to be loosely-coupled and able to evolve their systems and architectures independently by adhering to messaging contracts for information exchange. And, defining these messaging contracts can be a concrete implementation of the ubiquitous language in the organization.

And, implementing Event Sourcing in your core applications and services will also reflect the business processes and ubiquitous language within the bounded context they support. You'll capture the intent with each change which can unlock other capabilities. And, since you will always have the events that led to the current state of the system, you aren't limited on when those capabilities can manifest.

I hope that you found this overview helpful.

Let's Make Some Art - Part 2

2021-09-25T00:00:00Z

In part 1, we had our computer draw some boxes. I want to go in a different direction here to lead up to part 3. This time I want to explore reading pixels from an existing image.

What are we making this time?

I bet you, like me, have looked at an image and wondered what it would look like if all of the lines were sorted by their RGB color values. Well, let's write some code and find out!

Let's Create!

First, we'll need a program to run our code. I'm going to create a new .NET 6.0 project and then add ImageSharp.

>dotnet new console

>dotnet add package SixLabors.ImageSharp.Drawing --version 1.0.0-beta13

Note, at the time of this post, ImageSharp.Drawing is still in beta so you'll need to explicitly add the version. Be sure to check before you start in case there's a newer version available. Technically, for this post, we're not even using anything from the Drawing package, but it has a dependency on ImageSharp and we'll be back to using the Drawing package in part 3 so... to keep the flow consistent... Onward!

My goal is to read in an existing file, sort the pixels by some arbitrary scale, and then write them out to a destination file. We'll use the source image's dimensions to create the destination image.

using SixLabors.ImageSharp;
using SixLabors.ImageSharp.Formats.Jpeg;
using SixLabors.ImageSharp.PixelFormats;

var srcFilename = args.Length > 0 ? args[0] : throw new ArgumentNullException("source", "Must supply a source");
var srcFile = new FileInfo(srcFilename);

if(!srcFile.Exists)
{
    Console.WriteLine("Bad src");
    return 1;
}

var destFilename = Path.GetFileNameWithoutExtension(srcFile.Name) + "_sorted.jpg";

using var srcImg = Image.Load<Rgb24>(srcFile.FullName);
var srcWidth = srcImg.Size().Width;
var srcHeight = srcImg.Size().Height;

using var destImg = new Image<Rgb24>(srcWidth, srcHeight);

for (var row = 0; row < srcHeight; row++)
{
    var pixels = srcImg.GetPixelRowSpan(row).ToArray();
    var orderedPixels = pixels.OrderBy(p => p.R + p.G + p.B).ToArray();

    for (var col = 0; col < orderedPixels.Length; col++)
    {
        destImg[col, row] = orderedPixels[col];
    }
}

destImg.SaveAsJpeg(destFilename, new JpegEncoder() { Quality = 95 });

Console.WriteLine("Done");

return 0;

The above code reads in the source file, creates a destination image the same size, then goes row by row down the source image. It reads the row of pixels, reorders it by the sum of their RGB values, and then writes to the destination a pixel at a time. No, it's not the most efficient... but we don't need it to be for generating art.

Given an input like this image I took of a controlled burn near our house, the above code will generate an image where each row is sorted by the sum of the colors in each pixel.

Messing around

While it's not a lot, the above work will be critical input in part 3. But, since this post doesn't have a lot of code... let's mess around and see what kind of mayhem we can create.

Of course, there are lots of other possibilities. We could figure out how to sort things by column instead of row. We could randomly sort each row. But... I noticed a neat method available on Image. TryGetSinglePixelSpan. It looks like, if possible, this method will give you back a Span over the entire image. There are some practical limits to this. But, who cares about practical. In my case, since my input image is 2400x1600 pixels, my Span will cover an array of 3,840,000 (because math) pixels... all in a 1-dimensional array.

Knowing absolutely nothing about the underlying implementation and not really bothering to read the excellent docs, let's see what happens if we do something like... oh, I don't know...

using SixLabors.ImageSharp;
using SixLabors.ImageSharp.Formats.Jpeg;
using SixLabors.ImageSharp.PixelFormats;

var srcFilename = args.Length > 0 ? args[0] : throw new ArgumentNullException("source", "Must supply a source");
var srcFile = new FileInfo(srcFilename);
var destFilename = Path.GetFileNameWithoutExtension(srcFile.Name) + "_sorted" + srcFile.Extension;

using var srcImg = Image.Load<Rgb24>(srcFile.FullName);
var srcWidth = srcImg.Size().Width;
var srcHeight = srcImg.Size().Height;

using var destImg = new Image<Rgb24>(srcWidth, srcHeight);

srcImg.TryGetSinglePixelSpan(out var pixels);

for (var row = 0; row < srcHeight; row++)
{
    for (var col = 0; col < srcWidth; col++)
    {
        destImg[col, row] = pixels[col + row];
    }
}

destImg.SaveAsJpeg(destFilename, new JpegEncoder() { Quality = 95 });

Console.WriteLine("Done");

Note that, because col keeps resetting to zero, the first row starts with pixel 0 in the array and continues to pixel 2399. The second row in the destination will start at pixel 1 and continue to pixel 2400. This leads to some very interesting streaked output as you can see in the generated image.

And, if you're like me, your first question is "hey, what happens if we change that little pixels[col + row] code to do some multiplication instead pixels[col * row]?"

The answer, interestingly, is a beautiful star field.

Wrapping up... for now

As with part 1, this is just meant to get you thinking of other things you might want to explore. The key take-away from this post that you'll need for part 3 (not yet written) is that it's possible to read pixels from a source image and use them when writing to a destination image.

Some ideas of where you could take this:

Randomize the sort order for each row
Sort by row and column
Randomize the span and then write it out to the destination image
Combine two source input images into one destination

Swapping two numbers in C#

2021-09-13T00:00:00Z

Occasionally in programming, we come across the need to swap two numbers. That is, if we have var x = 10 and var y = 20, we need to end up with x = 20 and y = 10.

Here's a possible way to do it in C#

public void SwapWithTemp(ref int x, ref int y)
{
    var tmp = x;
    x = y;
    y = tmp;
}

Back when I was first learning to program, there were a finite number of variables in the world and we all needed to call the central variable office to see if we could use one. As you can imagine, this was a very lengthy process so interviewers started asking candidates a brain teaser question. How can you swap two variables without using a third temp variable?

I'll show the code (in C#) and then discuss why it works

public void SwapWithXor(ref int x, ref int y)
{
    x = x ^ y;
    y = y ^ x;
    x = x ^ y;
}

The ^ in this context is the XOR, or Exclusive OR, operator. It's much easier to visualize how things work if we can see the numbers like a computer does. Since computers see everything in binary, we need to as well.

int x = 10;
int y = 20;

The computer sees

x = 0000 1010
y = 0001 0100

The rules for XOR are that it's true when one of the values is true and false if both values are the same.

This means that: 1 XOR 0 is 1; 1 XOR 1 is 0; and 0 XOR 0 is 0.

Let's step through the code and see what happens under the covers

int x = 10; //x = 0000 1010
int y = 20; //y = 0001 0100

//We will go through x and y and compare row to row. 
//Any time there's a 1 in one row and a 0 in another, make it a 1 in the output
x = x ^ y;
//  0000 1010
//  0001 0100
//x=0001 1110
//If we work left to right then we know at the end x will be 0001 1110 

//Now we do the same thing with y
y = y ^ x;
//Work left to right and compare the new value of x (0001 1110) to y
//so compare these two with an XOR
//  0001 1110
//  0001 0100
//y=0000 1010
//At the end of this, y is 0000 1010 (10 in decimal. Oh, nice, the original value of x)

//Now do the same thing with x again to get the original value of y
x = x ^ y;
//  0001 1110
//  0000 1010
//x=0001 0100

We saved using a temp variable, which honestly hasn't mattered since hard drives shrunk down from the size of small cars. But, is it better? We'll get into that in a minute.

I want to share one more way to swap numbers in C#. We can use a tuple.

public void SwapWithTuple(ref int x, ref int y)
{
    (x, y) = (y, x);
}

You will either see this and say that it's the most straightforward and idiomatic way to do a number swap in C# or you'll hate it and swear off this approach forever. We're telling the compiler to make a tuple of the values x and y and then assign it the tuple of values y and x. We'll go under the covers later and it will look more obvious then if it doesn't now.

So, three viable ways to swap two numbers. Which one makes the most sense? Writing software, in any language, can be a balancing act of writing code that is fast and code that is maintainable. If the XOR approach happened to be the fastest code, and the code was called often enough that optimizing a few nanoseconds was worth it, then it'd be worth using an approach that isn't as immediately obvious to everyone. (possibly including you six months later).

As we did in Measure Two Hundred Times, Tweak Twice, let's set up BenchmarkDotNet and profile some code.

Here's the code in .NET 6.0 / C# 10. I've included two other possible ways to swap numbers from the many (stupid) ways we came up with to swap numbers

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

var summary = BenchmarkRunner.Run<Swappy>();

public class Swappy
{
    [Benchmark]
    [Arguments(10, 20)]
    public void SwapWithTemp(ref int x, ref int y)
    {
        var tmp = x;
        x = y;
        y = tmp;
    }

    [Benchmark]
    [Arguments(10, 20)]
    public void SwapWithXor(ref int x, ref int y)
    {
        x = x ^ y;
        y = y ^ x;
        x = x ^ y;
    }

    [Benchmark]
    [Arguments(10, 20)]
    public void SwapWithTuple(ref int x, ref int y)
    {
        (x, y) = (y, x);
    }

    [Benchmark]
    [Arguments(10, 20)]
    public void SwapWithArray(ref int x, ref int y)
    {
        var scratch = new [] { x, y };
        x = scratch[1];
        y = scratch[0];
    }

    [Benchmark]
    [Arguments(10, 20)]
    public void SwapWithLinq(ref int x, ref int y)
    {
        var numbers = new [] { x, y };
        x = numbers.Last();
        y = numbers.First();
    }

    [Benchmark]
    [Arguments(10, 20)]
    public void SwapWithInterlock(ref int x, ref int y)
    {
        y = Interlocked.Exchange(ref x, y);
    }
}

Compile the above code in release mode and then run the exe from the command prompt and you'll get some output like the following:

|            Method |  x |  y |       Mean |     Error |    StdDev |
|------------------ |--- |--- |-----------:|----------:|----------:|
|      SwapWithTemp | 10 | 20 |  0.1575 ns | 0.0113 ns | 0.0106 ns |
|       SwapWithXor | 10 | 20 |  0.6337 ns | 0.0106 ns | 0.0094 ns |
|     SwapWithTuple | 10 | 20 |  0.1589 ns | 0.0023 ns | 0.0021 ns |
|     SwapWithArray | 10 | 20 |  3.4849 ns | 0.0427 ns | 0.0357 ns |
|      SwapWithLinq | 10 | 20 | 33.4991 ns | 0.4638 ns | 0.4339 ns |
| SwapWithInterlock | 10 | 20 |  3.0909 ns | 0.0334 ns | 0.0261 ns |

We can trust BenchmarkDotNet to give us useful measurements because it's handled all of the execution warmup, works to remove noise, and repeats runs to get useful data.

Wrapping up

So, at least in C#, the temp variable and tuple approach are the fastest. And, in my mind, the two most readable as well.

If you show up at an interview and they ask you to swap two numbers without using a third, you can push back and dazzle them with science.

If you like the tuple syntax, then you should go with that. If you prefer the temp, then that's fine too. While I'm not in love with the tuple version, I would say that will be the canonical version you'll encounter in the wild and I can live with that.

Before I go, I wanted to dive into the temp version and tuple version further. This time with a look at the generated IL. My go-to tool for this on Windows is ILSpy.

Using that to explore the compiled DLL from our test above, let's look at the generated IL for the SwapWithTemp version:

.maxstack 2
.locals init (
    [0] int32 tmp
)

// int num = x;
IL_0000: ldarg.1
IL_0001: ldind.i4
IL_0002: stloc.0
// x = y;
IL_0003: ldarg.1
IL_0004: ldarg.2
IL_0005: ldind.i4
IL_0006: stind.i4
// y = num;
IL_0007: ldarg.2
IL_0008: ldloc.0
IL_0009: stind.i4
// }
IL_000a: ret

It creates a local variable, num, and then does the assignment swapping.

Now let's look at the decompiled version of SwapWithTuple

.maxstack 2
.locals init (
    [0] int32,
    [1] int32
)

// int num = y;
IL_0000: ldarg.2
IL_0001: ldind.i4
IL_0002: stloc.0
// int num2 = x;
IL_0003: ldarg.1
IL_0004: ldind.i4
IL_0005: stloc.1
// x = num;
IL_0006: ldarg.1
IL_0007: ldloc.0
IL_0008: stind.i4
// y = num2;
IL_0009: ldarg.2
IL_000a: ldloc.1
IL_000b: stind.i4
// }
IL_000c: ret

It's incredibly similar to the version that uses a temp variable to swap. Except, in this case, it's using two temp variables for us; num and num2. That explains why they're so close for timings on our benchmark tests. And, while it actually wastes just a little tiny bit more than SwapWithTemp, we're in a new bountiful age of variables where everyone can have as many as they want without some central variable assignment authority getting upset.

Anyway, that covers more than any reasonable person ever cared to know about swapping two numbers in C#. If you can think of any other creative ways that you want to contribute, please stop by https://gist.github.com/hyrmn/387e9e8d4e2858daf5e89097396b88fb and leave a comment with some code.

Host your static 11ty site on Google Cloud

2021-03-17T00:00:00Z

I am working on a fake product for a fake business. Of course, I need a marketing site for my fake app! It's a static web site so I'm using my favorite site generator, 11ty. Of course, you could use the same approaches below with any static site generator or hand-rolled HTML. Many of the same steps apply.

My requirements were:

Static site hosted on Google Cloud
All content is served over HTTPS and if you go to the HTTP site then you get redirected
Content hosted on GitHub
Site automatically built and deployed when I push new content

Rather than walk you step-by-step through yet another tutorial, I'm going to link to the resources that helped me and then provide the stitching that I had to figure out to get a complete solution. I have a feeling that this will be simplified and turned into a product offering soon.

Step 1 - The basics of a static site

Follow this excellent Google Cloud guide on hosting a static website.

I want to call out a couple items.

First, you'll need to ensure that you've already validated your domain with Google. That involves adding a TXT record to your DNS entry with whoever your registrar is (Godaddy, NameCheap, etc). If you've already done this, or if Google is your DNS provider, then you can skip this.

Next, you can only make your storage bucket for your website files multi-region when you create it.

Also, to start, don't enable CDN caching when you create your HTTPS load balancer. Enable this later, but you will hate yourself while you're doing the initial development if you have this on.

On the subject regarding the load balancer, unless you know what you're doing and have a good reason to do it, just let Google manager your SSL cert.

Last, you're going to end up with just an HTTPS site at the end of this step. Don't worry, though, we'll take care of that next to add an HTTP to HTTPS redirect.

Step 2 - Add an HTTP to HTTPS redirect

Skip almost to the end of this Google Cloud guide on setting up an HTTP to HTTPS load balancer. This whole article provides a nice overview of the network topology and how and why. But, if you're in a hurry, just skip to the section on redirecting traffic to your HTTPS load balancer. You have an HTTPS load balancer from step 1 already, so now you need to set up one that will listen to HTTP.

At this point, you should be set.

Step 3 - Automate building and deploying your site

We're going to leverage Cloud Build for this. And, we're going to leverage the existing Cloud Builders that Google provides.

You'll need to create a cloudbuild.yaml (or json) file in your repository that we'll then point Cloud Run at. Here's my cloudbuild.yaml

steps:
 - name: 'gcr.io/cloud-builders/npm'
   args: ['ci']
 - name: 'gcr.io/cloud-builders/npm'
   args: ['run', 'build']
   env:
   - 'NODE_ENV=production'
 - name: gcr.io/cloud-builders/gsutil
   args: ['-m', 'rsync', '-r', '-c', '-d', './dist', 'gs://www.sharpstatus.com']

We'll cover how this is triggered next, but... when you push a change to GitHub, Cloud Run will clone your repository and then run the steps in the cloudbuild.yaml.

I'm leveraging the npm cloud builder because that's how I build my site. First, I run npm ci to restore my node packages. Then I run npm run build which runs the npm script labeled build in my package.json file. I'm using TailwindCSS and I have it configured to purge unused CSS. But, by default, it only does that for production builds. You signal that by setting your Node environment to production, which you can see I'm doing above for the second step.

As part of my 11ty configuration, I'm copying all generated assets to /dist. So the last step is to use the gsutil builder top copy (rsync) everything from my output to my storage bucket. If you'll recall, the guide in step 1 showed using gsutil rsync locally to copy files.

That's the basics of automating the build. The last step is to tie it all together and actually trigger it.

Again, I'm going to start by pointing you to the Google Cloud guide on triggering builds from GitHub. At a high level, you're going to connect Cloud Run to your repository. This involves granting permissions to the Google Cloud Build app. You'll also want to tell the trigger to run the cloudbuild.yaml file we just defined.

Wrapping Up

🤞🏻 That should be everything you need to deploy your static site. Once you're happy with your site, be sure to go back to your HTTPS load balancer that you created in step 1 and enable CDN.

It's a bit involved; I'm not going to lie. I feel like Netlify is significantly easier to get started with. In fact, this site that you're reading right now is hosted there. But, I wanted to host all of my infrastructure together and also experiment a little. I'd call the experiment a success.

Let's Make Some Art - Part 1

2021-03-15T00:00:00Z

I saw a tweet from @James_M_South, creator of the amazing ImageSharp library where he mentioned there wasn't a lot of public chatter about said library. Now, I know nothing about image processing or graphics. But, I've been wanting to learn more about creating Algorithmic Art, or Generative Art, for a while. And, James' tweet inspired me to dive in.

Generative Art?

The Wikipedia links above provide some really great in-depth links into generative art. For my purposes, I want to use an algorithm to get a computer to make a picture within the constraints I've provided. I also want to introduce some randomness so that no two runs will produce the same result. I'm going to do this as a bit of a series and you're going to explore with me. Each post will be self-contained to a specific idea and should offer up some space to explore your own ideas after.

Let's Create!

First, we'll need a program to run our code. And then I'm going to pull in the ImageSharp Drawing library to abstract away the drawing magic for me. I'm going to create a new project and then add ImageSharp

>dotnet new console

>dotnet add package SixLabors.ImageSharp.Drawing --version 1.0.0-beta11

Note, at the time of this post, ImageSharp.Drawing is still in beta so you'll need to explicitly add the version. Be sure to check before you start in case there's a newer version.

Now, let's get the basics in place. I'll create an image, fill it with a color, and then save it. Just to ensure it works.

using System;
using System.IO;

using SixLabors.ImageSharp;
using SixLabors.ImageSharp.Drawing;
using SixLabors.ImageSharp.Drawing.Processing;
using SixLabors.ImageSharp.PixelFormats;
using SixLabors.ImageSharp.Processing;

var destWidth = 1920;
var destHeight = 1080;

using var image = new Image<Rgba32>(destWidth, destHeight);

image.Mutate(ic =>
{
    ic.Fill(Color.Black);
});

using var outputStream = File.OpenWrite("generated.png");
image.SaveAsPng(outputStream);
Console.WriteLine("Done!");

This code creates a new 32-bit RGBA image. Drawing on the image happens in an image context, which we set-up above and then use that context to fill the image a solid black.

There you have it, our first algorithmic art. It's sized for a background right now at 1920x1080 (the resolution of my little monitor). But, you could make it huge, call it a Study in Black, and wait for the offers to roll in!

Let's Make It Art!

This is fun and all, but variety is the spice of life. Let's add some randomness and let that drive our colors.

var rand = new Random();
var r = (byte)rand.Next(0, 255);
var g = (byte)rand.Next(0, 255);
var b = (byte)rand.Next(0, 255);

var todaysLuckyColor = new Color(new Rgba32(r, g, b, 255));

image.Mutate(ic =>
{
    ic.Fill(todaysLuckyColor);
});

By default, our random number generator will be seeded with the current timestamp. We then use that to create random values for our red, green, and blue. We then fill the image with that color. I ended up with a horrible mustard yellow and then a wonderful violet the second time I ran it. Let's push this just a little bit further before we wrap up today's post.

Shapes! Boxes... oh my.

Drawing a single color image is cool and all, but what about a single tiny box instead?

Changing up the mutate function yet again, let's draw a square.

image.Mutate(ic =>
{
    ic.Fill(Color.White);

    var rotation = GeometryUtilities.DegreeToRadian(45);
    var square = new RegularPolygon(40, 40, 4, 50, rotation);
    ic.Fill(todaysLuckyColor, square);
});

ImageSharp.Drawing has some great utilities for making shapes. And, that's what we're after. Shapes. Or, specifically, squares. I'm asking ImageSharp to make me a 4-sided polygon that is 50 pixels wide with a centerpoint that is 40 pixels from the top and 40 pixels from the left (the top left corner is our reference point of 0,0). Technically... ImageSharp is giving us a polygon where each corner touches the edge of a circle that is 50 pixels around. Which brings up an interesting thing... if we draw this without rotating it, we get a diamond. I haven't looked at the internals, but I assume ImageSharp starts at 0 on a circle and then draws. So, we need to rotate our shape. ImageSharp uses radians. I'm not smart enough to understand radians but I know I need to rotate my diamond 45 degrees so I'll use their handy utility.

Run this program and you now have a pretty square on an expanse.

Let's go a little further and generate a lot of squares. A lot of multi-colored squares.

If we think about each square as going in a row and column, then we can look for a number that's easily divisible into 1920 and 1080. Completely by coincidence, if we divide by 40, we get 48 columns and 27 rows.

image.Mutate(ic =>
{
    ic.Fill(Color.White);

    var rotation = GeometryUtilities.DegreeToRadian(45);

    for(var row = 1; row < 27; row++)
    {
        for(var col = 1; col < 48; col++)
        {
            var r = (byte)rand.Next(0, 255);
            var g = (byte)rand.Next(0, 255);
            var b = (byte)rand.Next(0, 255);
            var squareColor = new Color(new Rgba32(r, g, b, 255));

            var polygon = new RegularPolygon(40 * col, 40 * row, 4, 20, rotation);
            ic.Fill(squareColor, polygon);
        }
    }
});

I'm using two for loops here. The first to walk down the imaginary rows in my image and the second to walk across the columns. I'm offsetting each square by 40 pixes and drawing a 20 pixel square in the middle. Because each square should be a random color, we need to move the randomized color creation into the loop as well. Try it out and show off your creation.

Wrapping up... for now

We scratched the surface of what we can do with Six Labors' ImageSharp library and used it with a very simple algorithm to make a unique piece of art... which is pretty damn cool.

Some ideas of where you could take this:

Randomize the rotation
Rotate and shrink the size as you get closer to the image edge
Randomize the location
Different shapes
Vary the shapes
Constrain the colors to a specific range
Make the colors grayscale
Overlap the shapes and play with the alpha blending

Multi-stage Docker builds for ASP.NET Core 5

2021-02-28T00:00:00Z

I am building an app with .NET 5 and ASP.NET Core 5. And, I want to run it on a cloud provider. Since .NET 5 isn't widely adopted yet (and probably won't be since it's not an LTS version), that means the easiest path to deployment is a Docker image.

That part isn't hard. When you create a new .NET project, you get a multi-stage Dockerfile for free (well, as long as you select that option). But, I want to use vanilla Node.js build options for processing and bundling my CSS and JS files. The common solutions I've seen for Docker + Node + .NET involve starting with the base .NET image and then using apt-get to install Node. I didn't want to go down that path if I could help it. I don't know why, but the idea of modifying the Docker image outside of just building stuff offended my aesthetic.

What I've learned so far is that you can start with a .NET base image, then layer Node on that, build your app, and then copy out the final assets.

FROM mcr.microsoft.com/dotnet/aspnet:5.0-buster-slim AS base
WORKDIR /app

FROM node:lts-buster-slim AS node_base
FROM mcr.microsoft.com/dotnet/sdk:5.0-buster-slim AS build
COPY --from=node_base . .

WORKDIR /src
COPY ["MyDotNetApp/MyDotNetApp.csproj", "MyDotNetApp/"]
RUN dotnet restore "MyDotNetApp/MyDotNetApp.csproj"
WORKDIR "/src/MyDotNetApp/"
COPY "MyDotNetApp/." .

ENV NODE_ENV=production
RUN npm ci
RUN npm run build
RUN dotnet build "MyDotNetApp.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "MyDotNetApp.csproj" -c Release -o /app/publish

FROM base AS final
EXPOSE 8080
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "MyDotNetApp.dll"]

You can see how everything fits together here where I am building SharpStatusApp.

The core bits:

FROM node:lts-buster-slim AS node_base
FROM mcr.microsoft.com/dotnet/sdk:5.0-buster-slim AS build
COPY --from=node_base . .

We're going to pull down Node's LTS image built on Buster-Slim and Microsoft's .NET 5 SDK image built on Buster-Slim. Then we'll copy the contents of the Node image into the .NET image. For me, this means I get the Node bits I need from an official image rather than using apt-get to figure out and install my own dependencies.

On to the build part:

ENV NODE_ENV=production
RUN npm ci
RUN npm run build
RUN dotnet build "MyDotNetApp.csproj" -c Release -o /app/build

PurgeCSS will automatically tree-shake the final CSS for you if it's in production mode. So, I default the NODE_ENV to production. This will also help with any other tooling I bring in later that might make different choices based on build environment.

You should use npm ci on your build server rather than npm install as ci will use your package-lock.json file to grab versions... which gives you a bit of confidence that you won't accidentally rev a version that you haven't tried locally yet.

Then it's a matter of running my npm script with npm run build and compiling my .NET app with dotnet build then we bundle things up and get a svelte(?) final image with just our assets and runtime and none of the build tools or artifacts.

Wrapping Up

Including a Node.js Docker image as a base part of your build might be old hat, but I hadn't seen any examples of it. I wanted to document what I learned in case it helped anyone else down the road. I, personally, think it's a pretty clean way to assemble an app for deployment.

If you have any questions, comments or complaints, you can always DM or skeet me @hyrmn

Living with your creations

2021-02-01T00:00:00Z

Last week, I tweeted out the three oddities of application development:

How you thought you built the system
How you built the system
How your system is used

I'm not treading any new ground here. There's been a lot of thinking by smart people on emerging properties of complex systems. But, I wanted to take a bit of time and talk about my experience and maybe share some ideas and techniques for life in a small company with a focus on a B2B SaaS product.

I've been living with, and supporting, a software product for 8 years now. So, these opinions are formed by that.

When you first set out to build a system, you'll anchor some large architectural and design decisions. You probably won't do a big design up front, but you'll make decisions and live with them. Some decisions you won't even know you've made... but not choosing is still a choice. Because communication is lossy, your mental model of the system will never match your coworkers' mental model of the system. You'll try, and you'll get pretty close but you'll all have implicit assumptions that seep in.

Over time, what you thought you built is not what you built.

You'll face market pressure; real and imagined. Enhancements and features, like water, will seek the path of least resistance in their journey to production. You'll shore up pathways and dam off bad directions, but riverbanks will still shift on you.

Heck, you will not be the same person with the same skills in two years that you are now. Your tools, libraries, frameworks and, most importantly, your thinking will change. You'll purposely take new code and new designs in a different direction. But you can't lift and shift all of the old code.

Again, over time, what you thought you built is not what you built. Multiply that by the size (and churn) of your team. Multiply it by the years your app has existed. Realize that, in some nook, some forgotten logic path, your code is lying to you. Unit tests help. Integration tests help. You still missed what happens with that I/O interaction at the edge.

How to handle code untruths

For me, this is multi-pronged.

You must revisit the concepts and models in your code and question if it still makes sense. Be zealous on this. Every design decision is an inflection point and a chance to think about past choices.

Here's an example that I'm playing with right now. We have some resources that can be deactivated. Deactivated resources shouldn't show up through any of the normal UI paths. But, you can choose to view them and reactivate them. We have other resources that can be deleted. Sometimes delete means a hard delete. Poof Gone. But, sometimes they have to be soft-deleted. A soft-deleted resource still exists but shouldn't show up through any of the normal UI paths.

That introduces cognitive load that might not be worth it. Do we really need three paths for making a resource disappear? Is the choice to live with it the easy choice or the right choice? Would addressing it help the system or just feed a need to be technically correct?

During development, leave your code better than you found it. You can't be solely focused on pulling work, completing work, pushing work and moving on. You need to revisit those old code paths. Clean behind the fridge once in a while. Can your unit tests be better? Should you add an integration test for that path you noticed but didn't touch while you were working on your feature?

In production, I've found that you absolutely will live and die by your application instrumentation and logging. Maybe you can invest in observability. Maybe you can't. But, at the very least, you need to be able to see the runtime execution path. For us, we've gotten a lot of mileage off of correlation ids from the front-end all the way back and using Seq to monitor and slice when things go wrong. Things will go wrong.

But then, the users

The thing is.. all of the above is trivial once you get to the real issue with your application. And that's that people will use it. People will use it in ways you didn't realize or understand.

We had an idea in mind that people would edit a certain document maybe a dozen times. Heck, even that seemed like a stretch. When a user saved their edits, we'd throw the current and previous state on a queue to be picked up by a backend store for reporting. Well, our mental model of our users was shattered when one user made 300 edits which caused the document to be so large that the system couldn't enqueue it.

Or, they won't use a feature you built.

There are two broad efforts you should think about with your users. First, learn how they use your software, and then either adapt to them or help them adapt. Second, write tools to analyze the data to answer meta questions.

Broadly, think about how you can test your hypotheses quickly and cheaply. Then put the tooling in place to do that.

If you're offering a web-based service, instrument the important paths through your system so that you can see how your application is being used. Talk to the people that use your system. You will have a core group of power users. You will have people that absolutely love your product and want you to succeed. Identify these people and talk to them; make their life easier.

Think about how you'll store your data and write tools to give you answers.

We store some information in a way that we can answer questions about trends about, or affecting, all of our customers. We have utilities in place to run a little bit of code across all of our tenants. Being about to go from question to answer in < 30 minutes is powerful. Why have a meeting to talk about if you should work on a feature when you can quickly show how people are using it. And who the heaviest users are.

These tools can be rough. They're for your team and you can add the polish in over time. But, you must have internal support tooling from the start.

Parting

We're a small team with a profitable product. We've out-competed much bigger companies in our space. We have great net promoter scores with our customers. Some of that is driven by the fact that we can quickly learn about, and adapt to, our customers. Accept that your system is going to evolve and be used in ways you never imagined and think about how to support your team and your customers on that journey.

How (not) to convert a string to lowercase in C#

2020-07-06T00:00:00Z

You're looking for a job, you had a good initial chat with the recruiter and the team would love to have you on-site for an interview day; starting with a technical interview. You've heard that the team you're interviewing with is a bit, well, eccentric. That's ok.

The good news, though. They encourage you to bring your own laptop configured how you'd prefer if it'd make you more comfortable. And you'll be free to use any language you want.

You show up. You think you're prepared for anything. You walk in only to be greeted by Alice and Bob. You think you're getting set up for a cryptology joke but, no, it's interview time.

Alice tells you that you've got one task. Come up with as many ways as possible, using your preferred language, to convert a string to lowercase. Given the string "This IS COOL", you should end up with "this is cool".

Luckily, since you read my blog, you are well-prepared with stupid ways to do stupid things with code.

Using ToLower()

Let's do the simplest method first, just to make sure we're jibing with Alice and Bob.

var s = "TEST STRING😁";
var lower = s.ToLower();
Debug.Assert(lower == "test string😁");

String has a .ToLower() method on it to do exactly what we need. Bob frowns slightly that you took the easy route but you assure them that you're just getting started. Meanwhile, Alice is happy that you thought to include some Unicode in your test case.

You consult MSDN and see that the String class implements an IEnumerable over a collection of Char. Not only is hello a string, it's a collection of five characters new [] {'h', 'e', 'l', 'l', 'o'};.

We can work with this.

Using ToLower() one character at a time

Let's write a for loop and lowercase a character at a time.

var s = "TEST STRING😁";
var lower = string.Empty;
for (var i = 0; i < s.Length; i++)
{
    lower += char.ToLower(s[i]);
}

Debug.Assert(lower == "test string😁");

Now we're getting somewhere (not good, but it's still somewhere). We can iterate over the string one character at a time.

Let's see what else we can do.

LINQ to the rescue

We can use LINQ. Like other languages, we have map, reduce, and filter in C# (thanks to LINQ). Unfortunately, we have different names for these methods. reduce is Aggregate in LINQ. It's common to reach for reduce when you need to do something "mathy", like maybe sum a bunch of numbers. But, C# lets us work on any collection type.

This is going to get a bit confusing, so let's start with the method signature.

TAccumulate Aggregate<TSource, TAccumulate>(this IEnumerable<TSource> source, TAccumulate seed, Func<TAccumulate, TSource, TAccumulate> func);

We are going to start with a seed in a known starting state. This will be where we'll accumulate the result of our function call as well. We're then going to walk over (enumerate) each entry in our collection (characters in our string in this case). As we do that, we will pass our current state and the entry into a Func. This can be either an inline lambda or any function that matches the signature (string, char) => string.

Confusing! Let's see it in action and hopefully understand it a bit clearer.

var s = "TEST STRING😁";
var lower = s.Aggregate(string.Empty, (current, c) => current + char.ToLower(c));

Debug.Assert(lower == "test string😁");

We're starting with an empty string as our seed (the initial value). We then start iterating through the characters in the string. On the first character, we pass in an empty string and T to our lambda function. We then concatenate the lowercased t to our empty string. We then proceed to the next character E, and pass in our current string t and the now-current character E. We do this until the end of the string and then LINQ returns our final accumulate value to us.

Bob and Alice are, frankly, very impressed with the mess you've made.

But, we're not done.

Reflecting on our progress

By now, I'm sure you've realized that there's a glaring problem with our code. That's right. It's not dynamic enough. String has two methods we can call. There's .ToLower(), which honors the rules of the current culture, and there's .ToLowerInvariant().

Let's revisit how we started our journey but, this time, we'll make it sizzle with some choose-your-own-adventure options for callers of our function.

enum WaysToLower
{
    ToLower,
    ToLowerInvariant
}

var s = "TEST STRING😁";
var lowerMethod = typeof(string).GetMethods().First(m => m.Name == WaysToLower.ToLowerInvariant.ToString());
var lower = lowerMethod.Invoke(s, null).ToString();

Debug.Assert(lower == "test string😁");

Now we can dynamically choose which function we want to invoke. We'll use reflection to find the corresponding method on String and then invoke it. The beauty of this approach is that it's slower, breaks our IDE experience, and will cause a runtime exception if we try and invoke a function that doesn't exist.

Bob and Alice are overjoyed at all of the bad code you know how to write. Because, after all, knowing how to write lots of bad code is the mark of a good programmer.

Let's get serious for a moment

Ok, all of the above code was, well, bad code. You shouldn't let it anywhere near your production systems. I wanted to highlight some ways to have fun with a trivial task in C#. But, in reality, string manipulation is not a trivial task. I'm not going to delve any more into it in this blog post, but I want to leave you with a parting example.

Strings are comprised of Char objects. But, that might not be a direct mapping with what you picture as a character. I've actually snuck this into the examples above. In linguistics, a grapheme is the smallest fundamental unit in a writing system. In English, when you learn your ABCs, you're learning your graphemes. Or, to belabor the point, A, B, and C are graphemes.

With that in mind, how many graphemes are in test string😁? How many Char objects are in test string😁? As any Millenial or Zoomer can tell you, an emoji is the smallest fundamental unit of any text message. (and, if you want even more fascinating insight into how emoji are impacting language and our ability to signal non-verbal communication in writing, check out "Because Internet" by Gretchen McCulloch)

If you showed a college student our test string, they would quickly tell you that there are twelve graphemes.

But, if you ask .NET the same question, you're going to get conflicting answers. We need two characters to represent that innocuous 😁. Calling .Length will tell us there are 13 Char objects. But, there's been a recent addition to the .NET Framework so we can be as smart as a fifth grader. If we call .EnumerateRunes(), we'll learn that there are only 12 graphemes. (both .NET and Go call them runes)

Why does this matter? In your everyday coding life, it may not. After all, the .NET runtime was smart enough to not ruin our smiley no matter what we did. You'll know when it does matter, though. And, before you get to that point, you might want to dive deeper on Character encoding in .NET.

Anyway, that's all for now.

👋🏻

Various ways to map data between classes

2020-07-01T00:00:00Z

A while ago, I wrote about using AutoMapper to copy data between objects. Today I want to revisit that and provide some alternatives for mapping. Some serious... some not so serious.

It can be useful to reshape our data by copying it to a more specific structure that's tailored to the current use case. For example, we want to return a limited amount of data in response to an API request rather than our full domain object. Or, maybe we want to let the user send us changes and we want to protect against overposting.

Let's explore some options in C# and learn something about two potentially useful C# operator type converters along the way.

To set the scene, I have a User that holds the state for some login and profile information on my to-be-announced e-commerce platform (it's a tool to find cheeses you'll love based on your favorite color... very scientific).

public class User
{
  public int Id { get; }
  public string Name { get; }
  public string EmailAddress { get; }
  public string HashedPassword { get; }
  public string FavoriteColor { get; set; }
  public DateTimeOffset AddedOn { get; }

  public User(int id, string name, string favoriteColor, DateTimeOffset addedOn)
  {
    Id = id;
    Name = name;
    FavoriteColor = favoriteColor;
    AddedOn = addedOn;
  }
}

I also have an admin screen that lists all of my users. I don't need, or want, nearly as much information for that view. So, I'm going to return a summary object. You might hear these referred to as view models. There's nothing special about them; they're data models that exist to make rendering the data via an API response or HTML page or CSV download easier. For a web app, they can look pretty darn boring.

public class UserSummary
{
  public int Id { get; set; }
  public string Name { get; set; }
}

So, how do we get the data from our domain and put it in our view model? Seems trivially easy, right? You probably came up with a few ideas already. This would be a rather dull blog post if I stopped writing here and left the rest as an exercise to the reader. Let's take a look at some options.

Inline Mapping

One option is that we can force the caller to do the mapping.

public void MapUserManually()
{
  var users = new[] 
  { 
    new User(1, "Bob Almighty", "Green", DateTimeOffset.Now), 
    new User(2, "Alice Wunder", "Orange", DateTimeOffset.Now) 
  };

  var view = users.Select(u => new UserSummary { Id = u.Id, Name = u.Name });
}

This approach can be sensible. Especially if there is only one location, or function, where the mapping occurs. However, this quickly falls down if you have, say, three controllers that rely on the same view model. If we decide that the summary must always include the email address, then we have to update every place that we map User -> UserSummary.

This is also a bit unpleasant for users of your code, either other developers on your team or you in six months, because it forces you to be intimately familiar with how to construct a valid UserSummary from a User.

There's got to be a better way

Mapping with AutoMapper

Let's start with a look at AutoMapper again.

public void MapUsingAutoMapper()
{
  var config = new MapperConfiguration(cfg => cfg.CreateMap<User, UserSummary>());
  var mapper = config.CreateMapper();
  
  var users = new[]
  {
    new User(1, "Bob Almighty", "Green", DateTimeOffset.Now),
    new User(2, "Alice Wunder", "Orange", DateTimeOffset.Now)
  };

  var view = mapper.Map<IEnumerable<UserSummary>>(users);
}

Since this is not an in-depth AutoMapper article, I've chosen to keep the configuration together with the mapping call. In a production application, you would create the mapping configuration as part of your application's startup.

With AutoMapper, cfg.CreateMap<User, UserSummary>() creates a mapping from User to UserSummary. This automatically works with scenarios like above where I am mapping a collection of Users to a collection of UserSummarys. Note, however, that the mapping is one-way only. If you want to map from UserSummary back to User then you would need to add the inverse as well by adding cfg.CreateMap<UserSummary, User>().

AutoMapper is pretty powerful and we use it extensively at my employer. However, there's no perfect solution and that includes mapping tools in .NET. If we change Name to FullName on our User domain object, but then forget to rename the same property on our model, we won't get a compilation error. Rather, AutoMapper will happily give us an object with a mapped Id and a null Name.

Note, AutoMapper has a method that you can call after you set up the configuration to ensure that all properites on the target mapping are accounted for. You can perform this check by calling config.AssertConfigurationIsValid();. But, it's still a runtime check. So, unfortunately, we lose the benefits of our compiler.

Static Map Method

This next approach is my personal favorite. There's a few reasons for that. First, it puts the responsibility for mapping in one spot. This could be a dedicated mapper object somewhere, but I like to have the mapping live right with the target. This feels natural in a "typical" namespace hierarchy as well where the view models will live closer to the end user and the domain will live further away.

Let's revisit the UserSummary and add a responsiblity for creating itself from a User

public class UserSummary
{
  public int Id { get; set; }
  public string Name { get; set; }

  public static UserSummary MapFrom(User source)
  {
    return new UserSummary
    {
      Id = source.Id,
      Name = source.Name
    };
  }
}

I've added a static method that takes in a User and gives us back a UserSummary. I like this because there is a clear home for the mapping logic; if I rename a property on my domain object then my IDE will update the mapping for me; and if I search for references on my UserSummary, I'm guided here.

In action, the call to map looks like this

public void MapUserWithStaticMapper()
{
  var users = new[]
  {
    new User(1, "Bob Almighty", "Green", DateTimeOffset.Now),
    new User(2, "Alice Wunder", "Orange", DateTimeOffset.Now)
  };

  var view = users.Select(u => UserSummary.MapFrom(u));
}

It's very similar to the first example but now the mapping has a home.

One downside of this is that now we have to write a lot of mapping code. However, there is a very nice Visual Studio add-in that will do this work for you. Check out MappingGenerator; a Visual Studio add-in by Cezary Piątek.

Depending on how frequently your mappings change and the existing structure of your app, you may find AutoMapper a better fit for you or you may prefer a static mapper.

On to misusing C# to do what we want!

I promised some "creative" alternatives for mapping as well.

Explicit type conversion

Consider the following code, var view = (UserSummary) user. We're telling C# that we want to take our User object and strong-arm it into being a UserSummary object. As our code stands above, this won't work. It won't work because, as far as C# is concerned, there's no commonality... without making User a subclass of UserSummary, C# is just going to give us a compiler error, throw it's digital hands up, and call us names.

But, this would be a pretty lackluster spot to leave this blog post if we couldn't come up with some options.

And, we do have options. A way to dazzle your friends and win that coveted trivia night contest finally.

The explicit type conversion operator.

Let's change our UserSummary class again and give C# a roadmap on how to get from A to B.

public class UserSummary
{
  public int Id { get; set; }
  public string Name { get; set; }

  public static explicit operator UserSummary(User source)
  {
    return new UserSummary { Id = source.Id, Name = source.Name };
  }
}

At the risk of repeating ourselves, let's see the explicit cast in action

public void MapUserWithExplicitCast()
{
  var users = new[]
  {
    new User(1, "Bob Almighty", "Green", DateTimeOffset.Now),
    new User(2, "Alice Wunder", "Orange", DateTimeOffset.Now)
  };

  var view = users.Select(u => (UserSummary)u);
}

If you're familiar with LINQ, you're probably wondering why you can't call users.Cast<UserSummary>(). The reason is because the explicit cast that we introduced is bound at the call sites at compile-time (that is, the code to do the conversion is written at the time you build your app) but the LINQ .Cast<>() method is a generic cast performed at runtime and, as such, misses out on that compilation switch-out.

Would I do this in production code? Not for this kind of use case... no. It forces your caller to know that they can (and should) use an explicit cast to transform the data. Compare this to the UserSummary.MapFrom() method where we get intellisense in our IDE and a clear expectation of what to pass.

That said, we've gotta have some fun in the blog. Plus, there are some valid use cases for it, such as creating a strongly typed ID value object that can pretend to be a GUID when needed. This is worth exploring if you're pursuing a Domain Driven Design.

Implicit type conversion

Let's wrap up the code examples with a look at a similar approach to explicit type conversion. In a way, it feels like the inverse. Whereas our explicit type conversion is set up with, well, an explicit cast var view = (UserSummary) user, an implicit type conversion relies on the left hand side of the assignment UserSummary view = user.

After modifying our code, we have the following

public class UserSummary
{
  public int Id { get; set; }
  public string Name { get; set; }

  public static implicit operator UserSummary(User source)
  {
    return new UserSummary { Id = source.Id, Name = source.Name };
  }
}

I should mention that you cannot have both an explicit and implicit operator that take the same source type and return the same converted type. If you try and create a class with both the explicit and implicit operators as I have them in this article, you'll get a compiler error.

Now, on to the calling code

public void MapUserWithImplicitCast()
{
  var users = new[]
  {
    new User(1, "Bob Almighty", "Green", DateTimeOffset.Now),
    new User(2, "Alice Wunder", "Orange", DateTimeOffset.Now)
  };

  var view = users.Select<User, UserSummary>(u => u);
}

We've had to change up our LINQ .Select() statement quite a bit. LINQ uses this type internally when iterating over the source collection. As it does, it will call our implicit type converter.

Again, I would not actually use this in production code although, honestly, to me the Select statement reads fairly nice.

The reason that I don't like the implicit cast here is that it hides the complexity in a way that I'd rather keep centered. And, as with our explicit cast, it forces the caller to understand too much of your internal implementation. But, it's useful to understand explicit and implicit type conversions so you can know when, and when not, to use them.

Wrapping up

We've covered a bit of ground on how to take data from over here and put it over there. I think AutoMapper or static mappers make the most sense for production applications of any notable size. I prefer static mapping methods as the maintenance can be easier and having one less dependency to manage and update might be worth the trade-off. On the other hand, AutoMapper is great when you have a consistent convention and tests in place to catch any missed mappings.

My Setup

2020-05-20T00:00:00Z

My wife and I both work remote. I mean, lots of developers do right now in the time of COVID, but we've been doing it for a while.

I bought a new computer in February 2019. My goal at the time was something portable with decent battery life. We spent about half of 2019 on the road. I haven't been ready to fully embrace that desktop life, but I did make some purchases in December. (and my wife bought me the poster)

Chair: Herman Miller Celle. I found it to be more comfortable than the Aeron and significantly cheaper. I like a metal folding chair, though, so take it with a grain of salt.

Desk: Ikea something or other. It's their manual crank adjustable desk. (I distrust technology). Even if you don't stand during the day, it's nice to be able to adjust throughout the day. I've owned and really like a Multitable in the past but this was a cheaper option.

Computer: Lenovo Thinkpad T-480s. If I were doing it over, I'd get the full T-480 with 32GB of RAM (mine has 24) and a large battery.

Monitor: LG 34" ultrawide with USB-C. The single-cable connection to the laptop is very nice. I love how it cleans the desk up. I plug my keyboard and mouse into the laptop too (and it has enough juice to power their lights).

Keyboard: Drop CTRL with Halo Clears. It's a joy to type on and the build quality is amazing. I also own a WASD Keyboard tenkeyless that I enjoy and a Keyboardio Model 01 that I keep meaning to use more and keep not using more.

Mouse: Roccat Kone XTD. It's got adjustable weighting and an optical sensor. This model is quite old and it's held up well with some tarnishing.

Headset: Some cheap-ish HyperX. They're ok but not great. I've had some AKG K240 MK II headphones in the past that I loved.

Sundry: Fuck It Ship It poster. Glorious PC Gaming Race desk pad. Walnut laptop holder. Walnut wrist wrest. Walnut headphone holder. USB-powered desk light.

My wife made the paper flowers and she's made it very clear that they're only on loan 😁

Counting lines with C#

2020-05-15T00:00:00Z

This is, I guess, part 2 in a series.

I previously wrote about writing a utility in Go to count carriage returns (or newline characters) in a file. I decided that I wanted to do the same in C# on .NET Core.

Anyway, let's take a look at some of the neat new goodies in .NET and see what a command line utility to parse a file might look like. If you want to just see the code, it's on GitHub.

Implementation Recap

I want this to function just like the Go version. One difference, though. Since I named the Go version lc, I'm going to go ahead and dub this one nlc

This means that our little application might be called like this

> nlc "path/to/your/file.txt"

or like this

> echo "Count the lines in this" | nlc

(you can substitute cat, or grep, or anything else, for echo above)

In either case, the count of carriage returns (\n) in the file will be printed out.

> nlc "path/to/your/file.txt"
109

Handling a file argument

Let's first figure out how to read a file if the user calls our application with a file path nlc c:\somedir\somefile.txt.

We create an entry point into our console application by declaring a static void Main function. This can take in a string array of command-line arguments. We don't need to support named arguments or even more than one argument, so we'll just assume that the first argument is the path to the file.

namespace nlc
{
  class Program
  {
    static void Main(string[] args)
    {
      LineCounter counter = new LineCounter();
      int count = 0;

      //...//

      try
      {
        using var file = new FileStream(args[0], 
		                                FileMode.Open,
										FileAccess.Read,
										FileShare.None, 
										bufferSize: LineCounter.BufferSize,
										FileOptions.SequentialScan);

        count = counter.CountLines(file);
      }
      catch (FileNotFoundException)
      {
        Console.WriteLine($"Could not find {args[0]}. Check the file path.");
      }
      //...//
	}
  }
}

.NET gives us different kinds of Streams for reading and writing a sequence of bytes. Or, broadly, we get different classes with a shared abstraction to do things like... read and write to files, read and write to memory, the network, etc.

So, here we're opening a FileStream in read-only mode. We're specifying a buffer size, or the number of bytes we expect to read at a time, and we're providing a hint that we'll be processing the file sequentially. This gives the runtime a chance to work with the operating system to optimize for read performance.

Handling piped input

Next, we should handle piped input. In my Go line counting article, I covered file descriptors and how stdin can be treated as just another file descriptor. Well, in .NET, the abstraction gets kicked up a notch and, you may have guessed already, we can treat it as a stream!

We can use the System.Console object to determine if the user is piping input to us on stdin and then read that as a stream if they are

if (Console.IsInputRedirected)
{
  count = counter.CountLines(Console.OpenStandardInput());
}

Count Them Lines

On to the core of the program. The LineCounter

using System;
using System.IO;

namespace nlc
{
  public class LineCounter
  {
    public const int BufferSize = 32 * 1024;
    private const byte rune = (byte)'\n';

    public int CountLines(Stream stream)
    {
      int read;
      int idxOf;
      var buffer = new Span<byte>(new byte[BufferSize]);
      int count = 0;

      while ((read = stream.Read(buffer)) > 0)
      {
        var slice = buffer.Slice(0, read);
        while ((idxOf = slice.IndexOf(rune)) > -1)
        {
          slice = slice.Slice(idxOf + 1);
          count++;
        }
      }
      return count;
    }
  }
}

As in the Go version, I'm creating a 32 kibibyte buffer. From there, we're quickly going to diverge from the Go implementation and get weird with things.

One of the shiny and amazing new offerings that came with C# 7.2 is Span<T>. Rather than loading the entire file into memory at once (which would be impossible with a large enough file!), we'll load a chunk at a time and then look for all occurances of our \n character.

Some of this looks similar to the Go version. In both, we're reading a 32kb section of file at a time. The steps we need to take in the .NET version are remarkably similar to the alternate version of the Go program.

For reference, here's that Go code again.

...
const target byte = '\n'
//...//

	var position int
	for {
		idxOf := bytes.IndexByte(buffer[position:read], target)
		if idxOf == -1 {
			break
		}

		count++
		position += idxOf + 1
	}

In both, we look at how many bytes we've read, and then we slice over that section of the file; looking for our newline character as we go. Once we find once, we advance past it in the slice and continue our search.

You can spot some philosophical differences between the .NET and Go versions. I think the one that strikes me the most is that, in .NET, I can't read past the end of the stream. The documentation has this to say

Return Value The total number of bytes read into the buffer. This can be less than the number of bytes allocated in the buffer if that many bytes are not currently available, or zero (0) if the end of the stream has been reached.

Contrast that with the Go version.

read, err = r.Read(buffer)
//...//
if err == io.EOF {
//...//
}

In Go, we expect to get back either the number of bytes read or an error. The error is our signal that the end of file has been reached.

I think this type of thinking is probably one of the harder switches for someone accustomed to writing C# or Java code when they first start with Go. In those languages, we're told that raising exceptions is expensive and should truly only be used for exceptional cases (reaching the end of a file is a normal control flow and certainly not exceptional). However, in Go, raising or returning an error is idiomatic and not costly.

Is .NET as Fast as Go?

The real reason you're reading. Maybe.

As a reminder from the last post, I'm warming up reads of a 1.6GB text file. Using our Go version lc, I get the following averages after an initial warmup call:

real    0m0.625s
user    0m0.015s
sys     0m0.015s

Using our .NET version nlc, I get the following averages after an initial warmup call:

real    0m0.619s
user    0m0.000s
sys     0m0.015s

Looks like a tie to me (and they're both faster than wc).

Of course, there are other considerations. The compiled Go version on Windows is 2.8mb. Respectable. And, easy enough to compile to multiple targets and distribute. Meanwhile, the .NET version... if you have the framework installed already... is a svelte 170kb (yes, kilobytes). You can choose to compile the .NET version standalone. I'll leave that as an exercise for the reader. (spoiler, it results in quite a bit more to bring along for a deployment).

Wrapping Up

I haven't demonstrated any tests for this program. I'll leave you to review them at your leisure.

Contrasting .NET and Go with a small utility like this has been fun and interesting. The new developments with .NET Core and C# have been exciting to watch and playing with the new Span class has been great. I would be hard-pressed to reach for .NET over Go for a utility like this though. But, I think I could easily say the inverse for other problem spaces.

Counting lines with Go

2020-05-13T00:00:00Z

I was thinking the other day about The Unix Philosophy. Broadly, you can get a lot of power from small command line utilities that do one thing well and can be chained, or composed, into more powerful use cases. For example, there's a command called wc (word count). It, well, unsurprisingly, counts words. But, you can also have it count lines if you pass in a flag wc -l.

I wanted to take the idea of "do one thing well" to the extreme. And, since I'm learning Go, I thought a line count utility would be a fitting exercise. In building this, we'll be able to learn how to read input from another program to support composability; we'll learn a bit about command line arguments; and we'll learn about reading files.

I mentioned composability a bit. Let's see what that entails. It means that this utility will output only the line count (so that it can be piped on to another command). And, it will need to accept, as input, either the output of another program (we might call this piped input). Or, it will need to be given a file location.

This means that our little application might be called like this

> lc "path/to/your/file.txt"

or like this

> echo "Count the lines in this" | lc

(you can substitute cat, or grep, or anything else, for echo above)

In either case, the count of carriage returns (\n) in the file will be printed out.

> lc "path/to/your/file.txt"
109

If you want to just see the code, it's available on GitHub.

Implementation Considerations

There are some counting assumptions that I made. I had originally chosen to have this match my editor's line count. That is, if Visual Studio Code shows x lines then my logic would also show x lines. However, I've chosen to follow the behavior of wc -l. I count carriage returns (\n). If a file does not end with a carriage return then the last line will not be counted.

While I'm not sure how I feel about this behavior, it is consistent with other tooling. A trailing carriage return is required to get an accurate count. Changing this is an exercise left to the reader.

Also, I think I might want to reuse the line-counting logic in other applications. So, I'm going to separate the command-line interface from the code that understands how to read through a stream and count returns.

Handling a file argument

Go has a nice flag library to read from the command line. We're going to abuse it a bit to read in the first argument a user passes. Remember, the first scenario is, lc "path/to/your/file.txt". In this usage, there are no named flags being passed in. I simply need arg0.

flag.Parse()
filePath := flag.Arg(0)

if filePath == "" {
	fmt.Println("Usage:\n\tlc \"path\\to\\file.txt\"")
	return
}

file, err := os.OpenFile(filePath, os.O_RDONLY, 0444)
if err != nil {
	log.Fatal(err)
}
defer file.Close()
countLines(file)

I'm opening a file for read-only access, and requiring that the file have read permissions set for the user, group, and other. We'll get a file descriptor back from the call to os.OpenFile. You can think of a file descriptor as a small reference that we'll keep around so know know where to read data from later. And, to clean up after ourselves, we'll close the file when we're done reading. We can do with with the call to defer file.Close()

Handling piped input

The other use case we have to handle is when the data is passed, or piped, directly into our utility.

> echo "Count the lines in this" | lc

This is passed in on stdin, or standard input. Unix introduced three standard streams. These are ubiquitious enough that many programming languages have some way to access them and will, as needed, abstract the implementation details for the operating system away for you. These streams are stdin, stdout (standard output), and stderr (standard error). You might read something from the user on stdin, give them some results on stdout, and log any problems to stderr. But, as you can see in my use case above, stdin might be input from anything, including another program.

And, Unix loves file descriptors. Like, it really loves them. stdin, stdout, and stderr are all file descriptors. Yep, just like the result of the os.OpenFile call earlier. This will come in handy. Trust me.

We need to probe the stdin file descriptor to see if we received any data.

stat, err := os.Stdin.Stat()
if err != nil {
	panic(err)
}

if stat.Mode() & os.ModeCharDevice == 0 {
	reader := bufio.NewReader(os.Stdin)
	countLines(reader)
}

Calling Stat() will give us information on the associated file descriptor. In this case, that's Go's reference to stdin, which Go nicely stores for us in a variable called os.Stdin.

I know stat.Mode() & os.ModeCharDevice == 0 looks a little hairy. We're asking Go for the current file mode on the file information. This is a bitmask of the current modes that are set on the file. When the 'this is character input' flag is set, then we know that stdin is open and being written to.

We could read directly from the stdin file. But, I want to buffer the input to more efficiently traverse it. Go provides a buffered I/O library for just such a use case. We give bufio.NewReader an io.Reader and get back an io.Reader. What a deal. But, it does a ton for us under the hood.

Now that we have an io.Reader, we can call into the package (that we haven't written yet), and count them carriage returns.

func countLines(r io.Reader) {
	count, err := lc.CountLines(r)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(count)
}

Count Them Lines

Long article. I know. But, we're here. All of that setup and we can deliver the actual value in 22 lines of code

func CountLines(r io.Reader) (int, error) {
	var count int
	var read int
	var err error
	var target []byte = []byte("\n")

	buffer := make([]byte, 32*1024)

	for {
		read, err = r.Read(buffer)
		if err != nil {
			break
		}

		count += bytes.Count(buffer[:read], target)
	}

	if err == io.EOF {
		return count, nil
	}

	return count, err
}

Just to recap, we want to know how many times a \n character appears in a given file. In a text file, at least, this would tell us how many lines long it is (emoji can't have carriage returns in the middle; I checked).

We don't care about any of the content. So, the question is. How can we efficiently read through the file, get what we want, and get out. I'm going to rule out using io.ReadBytes('\n') or higher-level abstractions like scanner as I want to hold as little in memory as possible. With those options, I might end up trying to read an entire file into memory before I find the first newline character.

But, we can create a byte buffer, read into that buffer, and then search just that buffer. When we're done, we'll move on to the next chunk.

So, let's create a 32 kibibyte buffer

buffer := make([]byte, 32*1024)

and then read from our file into that

read, err = r.Read(buffer)

This might give us a buffer with the following content

Hello World\nThis is a\nThree line file

The variable read will tell us how many bytes were read in. This is very important information because we're reusing our buffer and not re-initializing it between reads. Suppose we read 32kb of data on the first call to r.Read(buffer) but only 5kb of data on the second call. Our buffer will still contain 32kb of data. 5kb from the last read followed by 27kb of old data...

Next, we can use the bytes.Count method in the Go standard library to find the number of newline characters. We'll store the result in our counter variable.

count += bytes.Count(buffer[:read], target)

We will loop "forever". In reality, we will read incrementially to the end of the file. Then, we'll try and read one more time and Go will return an end of file error. We'll check to see when this is encountered and return our results then. In our use case, an end of file error is expected so... well, we shouldn't return it to the caller.

if err == io.EOF {
	return count, nil
}

return count, err

An Alternative Way to Read

Calling bytes.Count(buffer[:read], target) is a very specific choice that I can make for this application. However, it might not always work for us. Suppose we were looking for a slightly more complicated pattern. Go has a way for us to do that. bytes.IndexByte will return the index position of the first occurance of a byte in a byte array. If no occurance is found, then a -1 is returned.

So, while it's more complicated, we can look for our \n character, and then look at the next slice of the array after that character, and then look at the next slice... continuing on until we're out of things to look at. Then we'd move on to the next chunk of file.

In that implementation, we would replace count += bytes.Count(buffer[:read], target) with the following

...
const target byte = '\n'
...

	var position int
	for {
		idxOf := bytes.IndexByte(buffer[position:read], target)
		if idxOf == -1 {
			break
		}

		count++
		position += idxOf + 1
	}

Is it Fast?

I am only concerned with if this is comparatively fast when measured against wc. Getting true benchmarking numbers are outside of the scope of my efforts here. I've run both programs several times which ensures the operating system and my storage have both done any caching they plan to do.

Using time (Unix timing utility) wc on my machine (a midrange dev laptop with an NVMe SSD drive) to parse a 1.6GB text file of lorem ipsum text, I get the following averages after an initial warmup call:

real    0m0.822s
user    0m0.156s
sys     0m0.655s

Using lc to parse the same file, I get the following averages after an initial warmup call:

real    0m0.625s
user    0m0.015s
sys     0m0.015s

So, I'm happy with how this experiment went.

Wrapping Up?

I haven't demonstrated any tests for this program. I'll leave you to review them at your leisure. Or, wait for the next exciting installment.

I've previously covered how I set Go up locally and added %GOPATH%\bin to my path. So, from within the lc project directory, I can run go install and have a shiny new command line utility to use.

Honestly, thinking up bespoke little utilities has been a lot of fun. And, once you unlock the power of chaining them together, you'll think of many new use cases. Just keep the Unix philosophy in mind.

Formatting Go code with goimports

2020-01-20T00:00:00Z

I've found a couple things that help when writing Go in Visual Studio Code. If you're using VS Code then you've probably been prompted to install some Go-related extensions already. But, there are a couple of settings that you might not have enabled. These settings will make things a little more enjoyable. At least they did for me

Format on Save

Go to your VS Code settings. You can either press ctrl + , or go to File -> Preferences -> Settings. Ensure that Format on Save is enabled:

Every time you save the file in VS Code, it will run it through your format tool. Yay!

Switch from gofmt to goimports

The gofmt tool does one thing. It formats your code to ensure it follows the Go standards. Useful, but not helpful. For example, Go will throw a compilation error if your source contains an unused import. gofmt doesn't care; it's happy to just make sure things look decent.

The following won't compile

package main

import (
	"fmt"
	"log"
)

func main() {
	fmt.Println("Hello world!")
}

> go run .
.\main.go:5:2: imported and not used: "log"

There's another option. goimports. In addition to the formatting work of gofmt, goimports will remove unused imports, lay out imports so that the standard library ones are grouped first, and list everything alphabetically. It will also try and resolve and import any missing imports.

The link above has steps to get running with some editors but VS Code is missing. Fortunately, it's uncomplicated.

First, we need to install goimports. Open a command prompt and do that

> go get golang.org/x/tools/cmd/goimports

Next, go to to your VS Code settings.

Search for go: format tool and pick goimports from the list.

That's it. Now you'll get a little more help when writing code.

Next Steps

Of course, gofmt and goimports aren't the only two tools in town for code tidying. Search around, find others, and see if you find one that meshes nicely with your workflow. For example, check out goreturns.

Happy formatting.

Go 1.13 on Windows - File Structure

2020-01-15T00:00:00Z

I've been playing with Go and I thought it might be helpful to others to document a bit of what I've learned about Go on Windows. The Go Docs are very thorough but I'm putting things down so I can remember and maybe it will help someone else as well. Just keep in mind, these are the notes of a Go novice

The Go Path

When you install Go, it should add its bin directory to your path automatically. For me, everything related to the Go tooling went in C:\Go\bin

In addition to this, Go likes all of the Go code to live under one common root. It defaults to %USERPROFILE%\go. This most likely maps to c:\Users\{Your user name}. You can override this easily enough. I want everything to live under c:\code\go. To make this change, you need to set the GOPATH environment variable.

You can do this from a command prompt by running

setx GOPATH c:\code\go

Or, if you prefer, navigate to the environment variables UI and add a User Variable with the above value.

Your Go directory structure is going to closely mirror where the code comes from. If you run go get github.com/hyrmn/GoTcpEchoServer then that code will end up in c:\code\go\src\github.com\hyrmn\GoTcpEchoServer. So, while you can develop 100% locally, if you're going to eventually push the code to a remote repository then you should probably think about making the structures match now.

The full structure is probably going to look different than how you normally organize your code.

We'll also want a Go module definition file, go.mod. This file will live at the root of our repository. You can have more than one in your repository but it seems most common to only have one... The module is important as it defines the import path for your packaged code. This is another good reason to have your code structure match where it will ultimately live.

Hello File Structure

Let's make Go say "hi" to us.

On a new command prompt (so you can pick up the GOPATH settings), navigate to your new root.

Since we haven't done anything yet, it's going to be empty. However, Go will eventually put some top-level directories in here. Go expects all of the source code to live under src so make that directory now.

c:\code\go>md src

Your structure will now look like

c:\code\go\
└── src

For "simplicity's sake" (well, simplicity for future you when you're pushing code, pulling other people's code, etc), make a structure under here to map to your remote repository home. For me, that's GitHub.

c:\code\go\src\github.com\{replace with your username}\hello

So, for me, I would run

c:\code\go>md src\github.com\hyrmn\hello

This will eventually get pushed up to GitHub to a new repository under my account.

Navigate to that new directory

C:\code\go>cd src\github.com\hyrmn\hello

Hello World

At last, code time.

First, let's tell Go that we want this to be a module.

c:\code\go\src\github.com\hyrmn\hello>go mod init github.com/hyrmn/hello

Open up that go.mod file in your favorite editor.

module github.com/hyrmn/hello

go 1.13

This is useful, I promise.

Now, while you have your favorite editor open, create a new file main.go

package main

import "fmt"

func main() {
	fmt.Println("Hello, world.")
}

run it.

>go run .

Hello, world.

You could also package the code into an executable with go build. That will put an executable in the project directory named hello.exe

But, there's more we can do

Install the world

We can go install our program.

>go install

That will take a second to run. Finish with no output. And... nothing new in your current directory. What gives?

When you run that command, Go builds your code and, if it succeeds, puts the executable under %GOPATH%\bin. Or, in our case, c:\code\go\bin. Go has also taken the opportunity to create a pkg directory. You don't need anything in here. Go uses it as a module cache when you start ~~downloading the internet~~ pulling in modules from other developers.

Our full directory structure now looks like

.
├── bin
│   └── hello.exe
├── pkg
└── src
    └── github.com
        └── hyrmn
            └── hello

One Last Setting

Now that you have that fancy executable sitting in that bin directory, you should add your GOPATH bin to your path. One thing Go seems well suited for are fun little command line utilities (I mean, lots of serious stuff too, but I know where I'm starting). With them readily accessible in my bin directory, I just get them for free and at the ready the minute I go install

Wrapping Up

If you are starting out in Go, and if 1.13, or the changes it introduced, are still current when you read this, then I hope you find it helpful.

Exploring the Polly.Contrib.WaitAndRetry helpers

2019-09-18T00:00:00Z

If you're just here for the code, you can grab it in this Gist

In a previous post, I covered a little bit of what you can do with Polly to wait and retry SQL Server exceptions. Today I want talk about one of the contrib libraries and how it can help standardize some better practices into your retry policies.

In my code, I use Polly in several places to retry remote calls. Our software is deployed to a cloud provider and, while people should generally build for it anyway, hosting in a cloud environment really drives home that you need to bake resiliency in to any of your code that involves a network connection. That's not to say cloud hosting is unreliable; far from it. But, you can't control those small hiccups and blips that happen when underlying infrastructure is upgraded or rerouted.

So, I might have a Polly retry policy for handling saves to our RavenDB document database

var retryTimes = new List<TimeSpan>
{
    TimeSpan.FromMilliseconds(0),
    TimeSpan.FromMilliseconds(50),
    TimeSpan.FromMilliseconds(100),
    TimeSpan.FromMilliseconds(150)
};

var retryPolicy = Policy
    .Handle<WebException>(ex => TransientErrors.Contains(ex.Status))
    .OrInner<WebException>(ex => TransientErrors.Contains(ex.Status))
    .Or<TimeoutException>()
    .OrInner<TimeoutException>()
    .WaitAndRetry(retryTimes,
        (exception, timeSpan, retryCount, context) =>
        {
            LogTo.Warning(
                exception,
                "WARNING: Unable to save to Raven, will retry after {RetryTimeSpan}, Retry attempt {RetryCount}",
                timeSpan,
                retryCount);
        });

What I've defined in my code above is a linear back-off. If my first try fails, I try again immediately. On each subsequent failure, I'll wait an additional 50ms and try again. After the last wait and retry, Polly will kick the underlying error up the call chain.

There are a few things to think about when deciding on a retry policy.

First, how long should we wait after the initial failure? In my use case (hosting in the cloud), it makes sense to retry failures immediately. A connection might fail because traffic is being killed and routed through a different network path within the data center. In this case, trying again right away will (probably) succeed.

Next, how long should we wait between retries? That depends on a balance of understanding the resource being called and the execution path of the caller. The code I posted above is in the execution path for a user's action on our web-based application. The last thing I want to do is force my users to wait. And, I'm calling a well-provisioned multi-node database cluster so I expect it to respond quickly outside of minor traffic blips.

Choosing how long to wait in a wait-and-retry loop is a balance of resources. You don't want to overwhelm a remote resource by retrying too quickly. You don't want to wait too long between retries, though; especially when a user is on the line and waiting. (user patience is the most valuable resource of all.)

While my code works well for my use case, let's take a look at Polly.Contrib.WaitAndRetry and see if it can help with making things cleaner or at least clearer.

As the name alludes to, these are helper methods for wait-and-retry that are maintained outside of the official Polly library. There's so much you can do with Polly and they can't possibly bake everything into the core library.

So, let's redo that Raven retry policy with the contrib library.

var linearBackoff = Backoff.LinearBackoff(TimeSpan.FromMilliseconds(50), retryCount: 4, fastFirst: true);
var retryPolicy = Policy
    .Handle<WebException>(ex => TransientErrors.Contains(ex.Status))
    .OrInner<WebException>(ex => TransientErrors.Contains(ex.Status))
    .Or<TimeoutException>()
    .OrInner<TimeoutException>()
    .WaitAndRetry(linearBackoff,
        (exception, timeSpan, retryCount, context) =>
        {
            LogTo.Warning(
                exception,
                "WARNING: Unable to save to Raven, will retry after {RetryTimeSpan}, Retry attempt {RetryCount}",
                timeSpan,
                retryCount);
        });

Here, I'm using the contrib library to define a linear back-off. The execution is exactly like mine but I like that the intent is clearer. It's clear that I want to retry four times, that I want a linear growth of 50 milliseconds per retry, and that I want to retry the first time immediately (ok, granted, fastFirst might not be the clearest name).

The contrib library also provides a helper for a constant back-off. (I'm also going to switch to a shorter policy definition so this post isn't a scary length)

var constant = Backoff.ConstantBackoff(TimeSpan.FromMilliseconds(100), retryCount: 5, fastFirst: true);
var constantPolicy = Policy
    .Handle<FooException>()
    .WaitAndRetryAsync(constant);

In this case, it will retry the first one immediately and then wait 100 milliseconds between each additional retry (and, if we remove that fastFirst flag then it will wait 100 milliseconds before the first retry as well). A constant back-off is a good choice when your issue is predictable. For example, a batched upload to a rate-limited resource. Or, it can be a good choice when you're just not sure what a good back-off wait time should be.

Next, there's a helper for an exponentially increasing back-off

var exponential = Backoff.ExponentialBackoff(TimeSpan.FromMilliseconds(20), retryCount: 5, fastFirst: true);
var exponentialPolicy = Policy
    .Handle<FooException>()
    .WaitAndRetryAsync(exponential);

The default growth factor is 2.0 but you can provide your own value. With the default growth factor and a fast first retry, our delays will be 0ms, 20ms, 40ms, 80ms, 160ms. Since the growth is exponential, you should start with a low initial value; especially if your retry policy is in the path of a user request. An exponential back-off can be an excellent choice when you're trying to ensure that your requests don't all hit the remote resource at the same time every retry. If you're calling an overburdened service that's causing your code to retry, then some will retry immediately, some will be on their second retry which will take exponentially more time, etc.

This brings me to a potential problem with all of these retry strategies, and something that the Polly.Contrib.WaitAndRetry library solves brilliantly.

Suppose in my scenario, where I'm retrying saves to Raven in response to a user on my web app, Raven becomes overloaded and starts refusing connections. In this case, where I have overloaded my service with a larger number of calls, then there is a strong chance that my failures and my retries will be highly correlated. Service gets overwhelmed so hundreds of requests wait and retry. Then the service comes up, those requests slam the server and bring it down again. We need a way to decorrelate the retries. That is, we want some requests to take, say, 30ms to retry and others to take 70ms, or 49ms, etc.

What we want to do is add jitter into our retry strategy.

var jittered = Backoff.DecorrelatedJitterBackoffV2(
    medianFirstDelay: TimeSpan.FromMilliseconds(50) 
    retryCount: 5, 
    fastFirst: true);

var jitteredPolicy = Policy
    .Handle<FooException>()
    .WaitAndRetryAsync(jittered);

This will create a retry policy where the initial wait will be close to the first delay and then subsequent delays will, largely, follow an exponential back-off. The key here is to shift things just a little per request in an attempt to have lots of little waves crashing against your resource rather than big waves of retries. There's a much fuller write-up (along with science and graphs!) on the Polly.Contrib.WaitAndRetry readme and I would highly encourage heading there.

This is where the magic of the WaitAndRetry helpers really shines. While it's awesome for communicating intent with the simpler use cases like in my example of Backoff.LinearBackoff, it's downright amazing with the addition of the decorrelated jitter policy. George Polevoy, the author of the decorrelation approach, put a lot of hard thinking into this and it really shows. Frankly, I don't have the knowledge to have come up with it myself, but I do have the knowledge to easily take advantage of it thanks to the packaging provided by Dylan Reisenberger and Grant Dickinson.

Again, the full code for the above blog snippits can be found on this Gist.

SQL Server Retries with Dapper and Polly

2019-09-03T00:00:00Z

If you're just here for the code, you can grab it in this Gist

Ken Dale wrote a post on the RIM Dev blog on how to Retry Transient Failures Using SqlClient / ADO.NET With Polly. Their use case is similar to mine but different enough that I thought it'd be worth sharing our implementation.

Like RIM, our tech stack is on Azure and we heavily rely on SQL Azure. We write to a service bus topic; a service then uses those messages to update a reporting database. If the message cannot be written then it's automatically returned to the queue and retried later. After 5 retries, the service dead-letters the message.

To accomplish this, our database structure and message handling code is idempotent. Getting, or applying, the same message twice won't cause duplicated data.

Our first thought that attempting to process the message x times and then deadlettering would be enough. However, at least in the early days (6+ years ago) after launch, we found out that we were mistaken. Sometimes SQL Azure just didn't want to cooperate. it's gotten much better since.

Enter Polly. Polly allows for all sorts of amazing retry logic. The things you need to care about in any distributed environment. From basic retry logic like I'll show here to circuit breakers (great if you're calling a flaky remote service and you don't want their service degradation to bring your app down).

The original code, along with Dapper, has been in production for quite a while. I've since updated to use the async methods in Dapper and thought I should share what we use.

Following is a simplified version of the full code that's available on this Gist

At the time of this post, I'm using Polly v7.0 and Dapper v2.0

var retryPolicy = Policy
      .Handle<SqlException>(SqlServerTransientExceptionDetector.ShouldRetryOn)
      .Or<TimeoutException>()
      .OrInner<Win32Exception>(SqlServerTransientExceptionDetector.ShouldRetryOn)
      .WaitAndRetryAsync(RetryTimes);

public static class SqlServerTransientExceptionDetector
{
    public static bool ShouldRetryOn(SqlException ex) {
      // See Gist
    }

    public static bool ShouldRetryOn(Win32Exception ex) {
      // See Gist
    }
}

We're defining an AsyncRetryPolicy Polly policy. When an exception is raised in the called code, Polly will look to see if it's an exception we want handled. In this case, we're looking for SqlExceptions, Timeouts, and a wrapped Win32 exception. For SqlExceptions and Win32 exceptions, we're going to further look to see if we can retry it with a call to SqlServerTransientExceptionDetector. For timeout exceptions, we'll just retry automatically.

I've given Polly a set number of times to retry with a static back-off. If it exhausts the number of retry times then the exception will then be bubbled up to the calling code. Our service then throws the message back on the service bus to try again or deadletters the message to be handled out of band.

I've created two Dapper extension methods to wrap up calling Polly.

static async Task<int> ExecuteAsyncWithRetry(this IDbConnection cnn, 
  string sql, 
  object param = null) =>
    await RetryPolicy.ExecuteAsync(async () => await cnn.ExecuteAsync(sql, param));

static async Task<IEnumerable<T>> QueryAsyncWithRetry<T>(this IDbConnection cnn,
  string sql, 
  object param = null) =>
    await RetryPolicy.ExecuteAsync(async () => await cnn.QueryAsync<T>(sql, param));

The caller might then look like this

public async Task UpsertPerson(string firstName, string lastName)
{
    await _conn.ExecuteAsyncWithRetry(_upsertSql,
                    new
                    {
                        firstName,
                        lastName
                    });
}

Again, the full code for the above classes can be found on this Gist.

Measure Two Hundred Times, Tweak Twice

2019-07-25T00:00:00Z

My buddy Pure.Krome stopped by the JabbR nerd chat with a question.

With this code, would a string be created each time the response was generated?

httpResponse.Headers["Cache-Control"] = $"public,max-age={seconds},must-revalidate";

David Wengier helpfully provided an answer (yes), and why (a bling-string is just a string.Format() call when it's converted to IL). But, in TRUE NERD FASHION, we took things to the extreme with some helpful and not-helpful alternatives.

Mr. Wengier also has a talk on pragmatic performance that is worth the watch and inspired me to sit down and play with BenchmarkDotNet for the first time.

BenchmarkDotNet is a wonderful tool for micro-benchmarking. When you have a small bit of code and you want to know if one option is better or worse than another, you need a miro-benchmarking tool. It will warm your environment to establish a baseline. Then it will run though a bunch of iterations for you and present a cleaner picture than if you put a timer on each end yourself.

Below, I set up a .NET Core console app, create a class to hold the benchmarks I want to compare, and then tell the BenchmarkRunner to go to town. By default, the output will be sent to the console window.

[CoreJob]
public class Strings
{
    [Benchmark(Baseline = true)]
    [ArgumentsSource(nameof(PassingInts))]
    public string BlingString(int seconds) => $"public,max-age={seconds},must-revalidate";
}

public IEnumerable<int> PassingInts()
{
    yield return 60;
    yield return 60 * 60;
    yield return 60 * 60 * 24;
}

public class Program
{
    public static void Main(string[] args)
    {
        var summary = BenchmarkRunner.Run<Strings>();
    }
}

And, spoiler alert, the BlingString(seconds) method is not speedy. But, it's not horrible. I'm going to throw my pragmatic hat in the bushes and come up with some alternatives. Then I'll order a shiny, new pragmatic hat and discuss trade-offs.

First, the easy alternative. If we want execution speed, then let's just hard-code something in. Everything gets cached for an hour.

[Benchmark]
public string Literal() => "public,max-age=3600,must-revalidate";

If BlingString is our skateboard, then Literal() is the super-charged sport bike. There is no computation. BenchmarkDotNet is confused why it's here.

Strings.Literal: Core -> The method duration is indistinguishable from the empty method duration

That's interesting but... well, only until one of your teammates uses your cache header function when they want to cache a critical lookup for just a second. Then your high-frequency trading app makes decisions on stale data. The investors lose millions. And you swear you'll never use C# again.

So, now we know that literals are fast. And, we know that string.Format() (our bling string) is slow. If we could just cache strings we've returned before...

It turns out, in fact, that we can do just that.

One of the collection objects in .NET is the ConcurrentDictionary. It's thread-safe (well, enough for our usage) and, as importantly, it has a GetOrAdd method we can use

TValue GetOrAdd(TKey key, Func<TKey, TValue> valueFactory);

If a given key exists, then the value is returned.

If the key does not exist, then the Func is called to generate the value. The value is then added to the dictionary and returned. Broadly, this means we can store our cached strings and generate new ones only when needed.

private ConcurrentDictionary<int, string> LittleShopOfHorrors = new ConcurrentDictionary<int, string>();

[Benchmark]
[ArgumentsSource(nameof(PassingInts))]
public string LookupDictionary(int seconds) =>
    LittleShopOfHorrors.GetOrAdd(seconds, s => $"public,max-age={s},must-revalidate");

But, this has a glaring problem that Aaron Dandy pointed out "dictionaries fill up". That is, there's no limit to how big our dictionary can grow. Other developers, intentionally (adversarial) or not, might leave you with a 2GB dictionary full of unused values.

But, this part of the journey is to keep some brainstorming ideas going. We don't have our pragmatic hat on so we don't care about memory!

On to a couple of more elegant ideas proposed by David...

First, if we're fairly sure that we will often see many requests for the same value at once, we can implement a poor cache. Just track what we've seen last and use that if it matches.

private int _lastSeconds = 3600;
private string _lastHeader = "public,max-age=3600,must-revalidate";

[Benchmark()]
[ArgumentsSource(nameof(PassingInts))]
public string LastCache(int seconds)
{
    if (_lastSeconds != seconds)
    {
        _lastSeconds = seconds;
        _lastHeader = $"public,max-age={seconds},must-revalidate";
    }

    return _lastHeader;
}

This retains the flexibility of working for any number of seconds passed in while betting on most calls using the same value.

The next option is to just take all but a couple options off of the table. We can use an enum to control what is returned.

public enum CacheLengths
{
    Short = 60,
    Medium = 60 * 60,
    Long = 60 * 60 * 24
}

[Benchmark]
[ArgumentsSource(nameof(PassingEnums))]
public string LiteralByChoice(CacheLengths cacheLength)
{
    switch (cacheLength)
    {
        case CacheLengths.Short:
            return "public,max-age=60,must-revalidate";
        case CacheLengths.Medium:
            return "public,max-age=3600,must-revalidate";
        case CacheLengths.Long:
            return "public,max-age=86400,must-revalidate";
        default:
            return "public,max-age=0";
    }
}

Now, obviously, this is very inflexible. But, on the other hand, it allows the original author to provide a clearer indication of what the impact of a given value is.

Last, I came up with what I feel is a more esoteric option. I learned a lot about some parts of the framework that I didn't know before. I can keep this knowledge filed away in case I need it in the future. And, of course, we're not going to limit ourselves until we see what the runtime characteristics are.

private const int Entries = 20;
private const int StepSize = 60;

private static readonly int[] _acceptableSeconds =
    Enumerable.Range(1, Entries).Select(seconds => seconds * StepSize).ToArray();

private static readonly string[] _fuckYouImATrain =
    _acceptableSeconds.Select(seconds => $"public,max-age={seconds},must-revalidate").ToArray();

[Benchmark]
[ArgumentsSource(nameof(PassingInts))]
public string FindNearest(int seconds)
{
    var idx = Array.BinarySearch(_acceptableSeconds, RoundToNearest(seconds, StepSize));
    if (idx > -1)
        return _fuckYouImATrain[idx];
    return ~idx < Entries ? _fuckYouImATrain[~idx] : _fuckYouImATrain[Entries - 1];
}

We create some acceptable values for seconds. And then we create a string array of cache headers generated from those acceptable values.

When FindNearest(seconds) is called, we want to find the closest matching header. We know that we store strings generated in 60 second increments. So, if FindNearest(50), FindNearest(60), or FindNearest(59) is called, then we want to find the nearest match to 60 (in this case).

This code is, honestly, a mess from top to bottom:

It relies on keys and values being the same length and in the same order.
The returned value does not follow the principal of least surprise because there's no obviously correlation between what you pass in and what you receive back (better naming would help a little).
From a maintenance standpoint, you'll need to remember what operator ~ is each time you read the code;
you'll have to know why you're using either the index or the XORed compliment of the index
and you'll have to think about bounds checking.

But, if the speed is there then maybe... maybe... it's worth it.

Let's put it all together (gist here) and see what those numbers look like.

A look at the results will tell us that, by far, the bling string approach is the worst with a mean time of about 250 nanoseconds per call.

Our enum option is the best contender if we can live with the constraints while the the simple cache approach is perfect if we'll often use the same value in subsequent calls (it trends towards the bling string performance the less often that's true).

The lookup dictionary is next fastest while our binary search is slow enough that it hardly seems worth the trouble. This is because search time for the dictionary is an O(1) operation while the binary search is, on average, O(log n). (Thanks Wikipedia!)

So, pragmatic hat back on. What's the best option? That depends! Look at that, after reading all the way to here, I pull out a trite catchphrase. This is where we need to understand our system. We don't care about optimizing the 97% that doesn't matter; we need to apply ourselves to the 3% that does.

If this method is called on every single HTTP request, but it's an internal app that gets called 1000 a day, then who cares! Leave the bling string in place.

Or, if this is only called when an image is requested from blob storage and the cache time is an hour, then maybe just hard-code it and be done.

On the other hand, if this is part of some critical infrastructure where saving milliseconds will net you a comfortable end-of-year bonus, then it's time to dig in and make some harder choices that depend on understanding your system.

When we optimize code, we have to keep mutiple audiences in mind. There's the callers of our code. Which design choice keeps the API straightforward and unsurprising? There's future maintenance devs (including yourself after you've forgotten the reasoning behind your original implementation). Which change makes the code straightforward? What can we do to minimize breaking logic changes? And, of course, the runtime. Which change has a positive impact on execution time? Which change is least likely to have unintended runtime consequences if misused?

Lots of questions but no answers. But, with a solid handle on writing small benchmarks to evaluate options, you'll be better equiped to make choices within the scope of the applications that you write and maintain.

Simple echo server in Go

2018-02-05T00:00:00Z

When I first started learning to program, I had a blast writing little utilities and servers. I didn't realize it, but I've missed out on that feeling for too many years. I found it again. Thanks to my good friend Pieter, I've started making some progress on my backburner interest in Go.

I've written a TCP echo server. There's a not-very-prescriptive RFC for an echo protocol in RFC 0862. And, luckily for you, dear reader, the implementation is just as short.

I'll only cover the highlights of my server here. Which, given to the small program size, is pretty much all of it. But, you can get a copy of everything over on my GitHub repository.

My first go at Go

I'm still learning Go so if I fumble on any important point, please let me know! I'm also going to gloss over a few things. So if, like me, you're new to Go, just know for now that it's important there's a func main() in a package main. Take that as gospel, we'll get into the rest.

The init function will be run first, and it seems like a good place to put the configuration stuff.

func init() {
	flag.StringVar(&opt.port, "p", os.Getenv("PORT"), "Default listen port")
	flag.Parse()

	if opt.port == "" {
		opt.port = "7"
	}
}

I want to be able to configure the port I'll bind. Yes, the only solid part of the RFC says port 7, but what if I'm on some *nix distro with an echo server daemon already running.

I'm using the built-in flags library to allow the user to specify a port with -p. If they don't supply one, I'll look to see if an environment variable named PORT has been set already. I assume the program will be running in an isolated space which is why I have such a generic environment variable. Finally, if no port is specified, I default to port 7 (at long last, we'll obey the RFC).

Next, let's actually do something and set up our server to listen for a client to connect.

func main() {
	log.Printf("listening on port %v", opt.port)

	ln, err := net.Listen("tcp", ":"+opt.port)

	if err != nil {
		log.Fatalf("listen error, err=%s", err)
		return
	}
	defer ln.Close()

	for {
		conn, err := ln.Accept()
		if err != nil {
			log.Fatalf("accept error, err=%s", err)
			return
		}

		go handleConn(conn)
		log.Printf("connection accepted %v", conn.RemoteAddr())
	}
}

There's a lot going on here, but half of it is Go's approach to error handling. We'll skip that.

ln, err := net.Listen("tcp", ":"+opt.port)
...
defer ln.Close()

This binds my program to the specified TCP port and waits for a client to connect. And, I've indicated with the defer ln.close() that I want to tidy up and stop listening when my main() function exits. I don't think this is necessary in this code, but it's a good practice to always clean up any resources your application has open.

The next section is, for me, mind-blowing.

for {
    conn, err := ln.Accept()
    ...
    go handleConn(conn)
}

Usually, when you put your program in an infinite loop (in this case, a for loop with no signal on when to stop), bad things happen to your CPU. Here, we accept any new client connections from our listener and then pass them off to a goroutine. A goroutine is a lightweight thread that will run without tying up our main application. So, if 100 clients connect, my program will send off 100 independent goroutines and handle each client. This is infinitely easier than a lot of multithreaded code I've written in the past. It's not without its trade-offs, but that's immaterial for now.

Now, finally, on to the actual meat of the application. Echoing the what the client sends back to the client.

func handleConn(conn net.Conn) {
	defer conn.Close()
    _, err := io.Copy(conn, conn)
    
	if err == io.EOF {
		log.Printf("received EOF. client disconnected")
		return
	} else if err != nil {
		log.Fatalf("copy error, err=%s", err)
	}
}

Wait, that's it?

Note the use of the defer again. Here, we're saying we want to tidy up the server's connection to the client when we exit. I also want to point out the if err == io.EOF as a special case to the other error handling in the program so far. If the client disconnects, it may send an EOF (end of file) marker to the server. We don't care about those kinds of errors; it just means the client disconnected. So, we'll note it but not treat it as an error we care about.

On to actually echoing back to the client... (yes, yes, I promised that earlier)

Here's where we can get a lot of help from Go's standard libraries.

_, err := io.Copy(conn, conn)

We don't need to set up an temporary buffer to read from the client or anything. io.copy expects to be able to read bytes from something and write bytes to something else. In our case, the connection object satisfies both of these requirements. So, we read some bytes from the connection... and then write some bytes straight back to the connection.

This is really nice.

Wrapping up

Well, there it is, in a nutshell. A very basic Go TCP server. It might be useful if you wanted to, say, test connectivity between two computers. I also learned a bit about Docker and how to set up a Docker image to build the application, but I'm going to save that for a future post.

2017 Review

2018-01-28T00:00:00Z

I thought it would be useful, for myself, to reflect back on 2017. I haven't written anything since July 2016 so this will bleed into 2016 as well... or, at least the end of 2016.

2016

2016 kicked off what turned into an interesting 2017 for us. 2016 started with a lot of grief as we lost our dog, Bandit. I've written about life with an aging dog before. Shortly after that, we decided we had to get out of the house and travel for a bit (we both worked remote at the time). That led us west to Denver where we stopped in to visit coworkers and then off to see family in Arizona. We settled in Denver for a few months before finally realizing our goal of living abroad. We moved to the Netherlands in October!

If you're an American and want to experience life as an expat, I highly recommend the Netherlands. I tell people that it's expat life with training wheels. The government's process is streamlined (they have an expatcenter in The Hague so that you have a one-stop shop for all paperwork and registration as well as a helpdesk staffed by volunteer expats). Plus, the people are friendly and willing to help and, in the cities at least, everyone has a strong command of English. While you should learn Dutch and absorb Dutch culture, you won't feel like you'll drown in the meantime.

However, even with the training wheels, we found our first few months living abroad to be rough. Finding a new routine, adjusting to the lack of sunlight in the winter, and figuring out food labels are all small things but they can wear on you. We were fortunate to spend Christmas with friends in Hamburg, Germany.

2017

Life became infinitely better heading into spring of 2017. Sunshine, knowing enough to buy laundry detergent instead of accidentally buying fabric softener, and building new friendships all helped immensely. I feel like we've gained a bit of resiliency.

That helped when I unexpectedly lost my dad in September. My boss and company were nothing short of amazing in helping me deal with the tragedy. My wife and I hopped on a plane with no notice and went back to the US for two weeks to grieve and reminisce with my family. I still don't know that I've fully processed everything, so there may be some future writing...

Enough with the personal stuff...

Professionally, I've stepped into a team lead role. Moving from IC to line manager responsibility has been a growth opportunity for me. My manager has been great at mentoring and I've found myself turning to more blogs and books on management and leadership than tech lately. However, I've recently rekindled a joy for programming that, frankly, I haven't felt in a while.

After you're 20 years in the industry, even when you're learning and keeping up with your field, and even if you're playing around with other languages and platforms, things can start to feel... tired. When I first started programming, I was playing with things like C (later Java) and writing stupid little toys. A Finger client, a TCP echo server, a multi-threaded SMTP sender (which led to a fun discussion with the mainframe email admin at university when I needed help deleting the 100k email messages I had sent myself). I've found that joy again in golang. I don't know if I'll ever take it into business apps, but I do know I've been having a blast playing with it. My first project, of course, was a TCP echo server.

I've been fortunate to have a friend and coworker help me with my Go journey. I'm hoping we'll jam together on some substantial open source in the future.

One interesting side-effect, for me, of being interested in programming again is that it's gotten me interested in writing again as well. I have no Google analytics and no server logs. So, frankly, I don't know if anyone is reading. But, it's useful for me to have a space to think out loud.

2018

Dig into Golang more
Burn my fingers on Elixir
Finally, finally write one of my, frankly, brilliant ideas. Because I think it has merit, I'll probably write it with C# on .NET Core using some event sourcing (ES) and command query responsibility segregation (CQRS) because there's a lack examples in the .NET space that encompass a whole application.

Ben Hyrman's Blog

A Hitchhiker's Guide to Strategic Design

Our "Simple" Scenario #

And Off We Go #

Bounded Contexts #

Ubiquitious Language #

Context Maps #

Event Storming / Event Modeling #

Event-Driven Architecture #

Event Sourcing #

Conclusion #

Let's Make Some Art - Part 2

What are we making this time? #

Let's Create! #

Messing around #

Wrapping up... for now #

Swapping two numbers in C#

Wrapping up #

Host your static 11ty site on Google Cloud

Step 1 - The basics of a static site #

Step 2 - Add an HTTP to HTTPS redirect #

Step 3 - Automate building and deploying your site #

Wrapping Up #

Let's Make Some Art - Part 1

Generative Art? #

Let's Create! #

Let's Make It Art! #

Shapes! Boxes... oh my. #

Wrapping up... for now #

Multi-stage Docker builds for ASP.NET Core 5

Wrapping Up #

Living with your creations

How to handle code untruths #

But then, the users #

Parting #

How (not) to convert a string to lowercase in C#

Using ToLower() #

Using ToLower() one character at a time #

LINQ to the rescue #

Reflecting on our progress #

Let's get serious for a moment #

Various ways to map data between classes

Inline Mapping #

Mapping with AutoMapper #

Static Map Method #

On to misusing C# to do what we want! #

Explicit type conversion #

Implicit type conversion #

Wrapping up #

My Setup

Counting lines with C#

Implementation Recap #

Handling a file argument #

Handling piped input #

Count Them Lines #

Is .NET as Fast as Go? #

Wrapping Up #

Counting lines with Go

Implementation Considerations #

Handling a file argument #

Handling piped input #

Count Them Lines #

An Alternative Way to Read #

Is it Fast? #

Wrapping Up? #

Formatting Go code with goimports

Format on Save #

Switch from gofmt to goimports #

Next Steps #

Go 1.13 on Windows - File Structure

The Go Path #

Hello File Structure #

Hello World #

Install the world #

One Last Setting #

Wrapping Up #

Exploring the Polly.Contrib.WaitAndRetry helpers

SQL Server Retries with Dapper and Polly

Measure Two Hundred Times, Tweak Twice

Simple echo server in Go

Our "Simple" Scenario

And Off We Go

Bounded Contexts

Ubiquitious Language

Context Maps

Event Storming / Event Modeling

Event-Driven Architecture

Event Sourcing

Conclusion

What are we making this time?

Let's Create!

Messing around

Wrapping up... for now

Wrapping up

Step 1 - The basics of a static site

Step 2 - Add an HTTP to HTTPS redirect

Step 3 - Automate building and deploying your site

Wrapping Up

Generative Art?

Let's Create!

Let's Make It Art!

Shapes! Boxes... oh my.

Wrapping up... for now

Wrapping Up

How to handle code untruths

But then, the users

Parting

Using ToLower()

Using ToLower() one character at a time

LINQ to the rescue

Reflecting on our progress

Let's get serious for a moment

Inline Mapping

Mapping with AutoMapper

Static Map Method

On to misusing C# to do what we want!

Explicit type conversion

Implicit type conversion

Wrapping up

Implementation Recap

Handling a file argument

Handling piped input

Count Them Lines

Is .NET as Fast as Go?

Wrapping Up

Implementation Considerations

Handling a file argument

Handling piped input

Count Them Lines

An Alternative Way to Read

Is it Fast?

Wrapping Up?

Format on Save

Switch from gofmt to goimports

Next Steps

The Go Path

Hello File Structure

Hello World

Install the world

One Last Setting

Wrapping Up

My first go at Go

Wrapping up

2016

2017

2018