Friday, October 7, 2022

Mocking GCP APIs containing iterator structs

New company & language, who dis?

The astute amongst my readers may have noticed that I left Lightbend to go work with some friends who I'd built things with previously at another company. This new company presents some unique challenges, including another language to put under my belt. The new language is Go, also referred to as Golang, and it has some interesting claims. I'll inspect those claims here on my blog as I become comfortable, or overly uncomfortable, with the language.

One of those claims that made me uncomfortable is that the consumer should define interfaces rather than the API. I recognize the reason for this thinking, but, in my opinion, it is making up for the laziness or ignorance of API developers. Too many don't practice the SOLID principles, including coding to interfaces, not implementations, so the consumer takes up the burden of making their tests mockable. The easiest way to demonstrate the pain that causes is to show you the Go code I had to write to deal with this fact when mocking Google Cloud Platform's Compute Engine APIs containing iterators.

What's the problem?

If I'm just writing code to use Google's APIs, what's the problem? Well, the problem is that I like to have my code tested without paying Google for the use of their resources. The only way to do that is to mock/stub/fake their APIs in my code so that I fulfill their interface with my basic implementation, enough to stand in for my tests. After all, I'm not testing their APIs but my code using them.

Great, so this is a pretty common need. Typically, I use the client library for the API provided by GCP and write code that fulfills their interface. This is where Go is unique, and I differ from their philosophy. See, they feel the API client developers aren't the ones who should be on the hook for the interface. Instead, that's on the consumer to create. I disagree because it means that the API client developers have broken the Dependency Inversion Principle, making their code unmockable for their own testing. Maybe Google engineers don't need to mock their code, preferring to exercise their actual systems. However, the typical consumer isn't going to want to exercise their actual resources except in integration testing. It's laziness to pass the buck like this.

You have yet to convince me, Sir!

Let's pretend, though, that they're right. The API client authors aren't responsible for our needs. Let me show you where that still falls on its face by making its consumers' lives more complex than they should be. Google doesn't give us an interface for their API. Honestly, this isn't so uncommon, and I'm sure I have readers who are calling me out right about now. However, let me show you some code where the Google API Go client authors' disregard for the Dependency Inversion Principle bit me.

I recently had to work with GCP's Compute Engine APIs to fetch various data. I'll focus on fetching the Zones for a Region for this demo. Below is the Go code to do that.

import (


func (t *TopologyFetcher) fetchComputeZones(
	ctx context.Context,
	regionShortName string,
	zones []*computepb.Zone,
) ([]*computepb.Zone, error) {
	var (
		err  error
		zone *computepb.Zone

	client := t.zonesClientCreator(ctx)
	filter := "region eq .*" + regionShortName
	it := client.List(ctx, &computepb.ListZonesRequest{
		Project: t.projectID,
		Filter:  &filter,

	defer func(client *ZonesClientWrapper) {
		_ = client.Close()

	for zone, err = it.Next(); err != iterator.Done; zone, err = it.Next() {
		if err != nil {
			returnErr := fmt.Errorf("could not fetch GCP zones: %q", err)
			return nil, returnErr

		if !lo.Contains(zones, zone) {
			zones = append(zones, zone)

	return zones, nil

If you're new to Go having come from one of the modern OOP languages, like me, then the pointers will throw you a bit. The critical thing to note above is that the client library from GCP returns a struct by reference, not a value or an interface. What's most interesting about the code above is that I want to be able to mock the client, which returns an iterator when I request a List of zones.

Ok, that's a lie. Let me show you what the client really does:

func (c *ZonesClient) List(ctx context.Context, req *computepb.ListZonesRequest, opts ...gax.CallOption) *ZoneIterator {
	return c.internalClient.List(ctx, req, opts...)

That's the actual function of Google's compute.ZonesClient. Of note is that it returns a pointer to an iterator. That distinction makes all the difference, as I need to mock that iterator so I can return values from a slice (list or array in other languages). The problem is that they coded against a concretion, not an interface, so I now have to write my own interface to stub out my implementation.

Here's what not to do...

My first thought was that I'll just create the [missing] iterator interface. Looking at the one method I needed from it, I could see in my code above that I needed the following interface.

type ZoneIterator interface {
    Next() (*Zone, error)

Pretty simple. My next intuition was to create an interface for the client.

type ZonesClient interface {
    Close() error
    List(ctx cotext.Context, req *computepb.ListZonesRequest, opts ...gax.CallOption) ZoneIterator

This proved to be wrong, thanks to the pointer. See, computepb.List does not conform with that interface. It wants a pointer, but the client interface returns an iterator interface. Why did I try this, then? Because I'm used to other languages that decided, correctly, in my opinion, that pointers were the root of too many bugs and difficulties for developers and unnecessary. 

So, now what?

It took two pivots of my thinking to get this working. Thankfully, I wasn't going it alone. I took inspiration from this SO post and had the help of a Go pro, Jessie Hernandez, to figure it out when I got stuck. 

First, I was right to create the iterator interface, but I needed to wrap the client, so I could use it.

type ZonesClientWrapper struct {
    Closer func() error
    Lister func(ctx cotext.Context, req *computepb.ListZonesRequest, opts ...gax.CallOption) ZoneIterator

It's just a few tweaks on my interface above. First, the wrapper is a struct, not an interface. Why? Because I am not stating that computepg.ZonesClient is fulfilling an interface. Instead, its functionality will be wrapped inside the ZonesClientWrapper.

The second tweak is that Closer and Lister are pointers to functions I need to specify before use. Instead of an interface, I'll fulfill with either my stub or GCP's implementation; I will flesh out the wrapper as follows.

func (w *ZonesClientWrapper) Close() error {
    return w.Closer()

func (*ZonesClientWrapper) List(ctx context.Context, req *computepb.ListZonesRequest, opts ...gax.CallOption) ZoneIterator {
    return w.Lister(ctx, req, opts...)

Now we're ready to use this wrapper. Here's the wrapper being populated using Google's API.

client, _ := computepb.NewZonesRESTClient(ctx)
wrapper := &ZonesClientWrapper{
    Closer: client.Close,
    Lister: func(ctx context.Context, req *computepb.ListZonesRequest, opts ...gax.CallOption) ZoneIterator {
        return client.List(ctx, req, opts...)

And here it is with a test stub. First, we need to fulfill the iterator interface's promised functionality.

type ZoneIterator struct {
    iterCounter      int
    ReturnZones      []*computepb.Zone
    ReturnZonesError error

func (it *ZoneIterator) Next() (*computepb.Zone, error) {
    if it.ReturnZonesError != nil {
        return nil, it.ReturnZonesError
    if it.iterCounter == len(it.ReturnZones) {
        return nil, iterator.Done
    zone := it.ReturnZones[it.iterCounter]
    return zone, nil

Then, we create and wrap our stubbed client.

type ZonesClientStub struct {
    ZoneIterator *ZoneIterator

func (s *ZonesClientStub) List(_ context.Context, _ *computepb.ListZonesRequest, _ ...gax.CallOption) ZoneIterator {
    return s.ZoneIterator

func (s *ZonesClientStub) Close() error {
    return nil

// Wrap the stub before tests
BeforeEach(func() {
    zonesClient = &ZonesClientStub{}
    zonesClient.ItemIterator = &ZoneIterator{
        iterCounter:      0,
        ReturnZones:      []*computepb.Zone{},
        ReturnZonesError: nil
    zonesClientCreator = func(ctx context.Context) *ZonesClientWrapper {
        return &ZonesClientWrapper{
            Closer: zonesClient.Close,
            Lister: zonesClient.List,

Now I just use either client via the wrapper as shown in my first listing. The client := t.zonesClientCreator(ctx) you see there does one of the two above, depending on if the code is being executed by the test or in production.

My admittedly biased conclusion

This is a lot of work, and mind-bending around pointers, that I would not have to do in most other modern languages. Even in a traditional, garbage-collected OOP language, iterators are common enough to have their own interface. So, I'd just mock the interface and be done. In Go, I have to think about pointers and mock via a pointer function and a wrapper, adding a lot of complexity in my code just so I can test it. I'd be open to hearing a counter-argument but, to me, this is more work than necessary if only we coded consistently to interfaces and exposed those for external use.

Bonus round

Fetching Zones was not my only use of GCP's compute API. As such, I was repeating myself a lot until Jessie introduced me to generics support in Go. In doing so, I was also able to apply the pattern to GCP's billing API by breaking up its functionality as if it were two different clients. Here is a complete listing of the generic implementation.

type GenericIterator[T any] interface {
	Next() (*T, error)

type GenericClientWrapper[R any, T any] struct {
	Closer func() error
	Lister func(ctx context.Context, req *R, opts ...gax.CallOption) GenericIterator[T]

func (w *GenericClientWrapper[R, T]) Close() error {
	return w.Closer()

func (w *GenericClientWrapper[R, T]) List(
	ctx context.Context,
	req *R,
	opts ...gax.CallOption,
) GenericIterator[T] {
	return w.Lister(ctx, req, opts...)

type SkusClientCreator = func(ctx context.Context) *SkusClientWrapper
type SkusClientWrapper = GenericClientWrapper[billing.ListSkusRequest, billing.Sku]

skusClientCreator = func(ctx context.Context) *gcpservice.SkusClientWrapper {
	// real client - uncomment to test against GCP's API
	//client, _ := gcpbilling.NewCloudCatalogClient(ctx)
	//return &gcpservice.SkusClientWrapper{
	//	Closer: client.Close,
	//	Lister: func(
	//		ctx context.Context,
	//		req *billing.ListSkusRequest,
	//		opts ...gax.CallOption,
	//	) gcpservice.GenericIterator[billing.Sku] {
	//		return client.ListSkus(ctx, req, opts...)
	//	},

	return &gcpservice.SkusClientWrapper{
		Closer: skusClient.Close,
		Lister: skusClient.List,

type ZonesClientCreator = func(ctx context.Context) *ZonesClientWrapper
type ZonesClientWrapper = GenericClientWrapper[compute.ListZonesRequest, compute.Zone]

zonesClientCreator = func(ctx context.Context) *gcpservice.ZonesClientWrapper {
	// real client - uncomment to test against GCP's API
	//client, _ := gcpcompute.NewZonesRESTClient(ctx)
	//return &gcpservice.ZonesClientWrapper{
	//	Closer: client.Close,
	//	Lister: func(
	//		ctx context.Context,
	//		req *compute.ListZonesRequest,
	//		opts ...gax.CallOption,
	//	) gcpservice.GenericIterator[compute.Zone] {
	//		return client.List(ctx, req, opts...)
	//	},
    return &gcpservice.ZonesClientWrapper{
		Closer: zonesClient.Close,
		Lister: zonesClient.List,

Thursday, January 13, 2022

We're All Actors on the Stage of Life


Yes, you read that right. You, my friend, are an actor. So am I. "But!" I hear you say, "I'm not making Hollywood bucks!" I'll commiserate with you. Neither am I. And, yet, I maintain that we are all actors.

No, I'm not talking about how we fake it 'til we make it, imposter syndrome and all. I mean that we are all part of systems and act upon that system. While you can argue that some do so more effectively than others, we're all actors in a variety of systems. At home, you may be a parental actor. At work, you may be a manager or, more likely, a developer. In the world, you may be a more or less active part of a government by the people, at least in democratic nations. 

Oh, Behave!

I belabor that point to underscore that you already know how actors behave. First, importantly, they behave according to rules set out by the system in which they participate. Next, they behave consistently or they are considered suspect by the others participating in the system. In hierarchical systems, they often have a supervisor that can correct or dismiss them if they go astray.

The next thing to note about actors is that they are often busy executing upon those established behaviors. They're not usually waiting around to be told what to do. Rather, someone can ask them to do something and it gets prioritized against the other tasks they are already working. 

Now, let's dig into that. If you ask someone to do something, if it's a nearly instantaneous result then you might wait around for that result. Otherwise, you'd make the request and go on with your own activities while trusting that they'll get the result back to you, if such is needed, when it is ready. The only distinction between that and telling someone to do something is politeness. 

Revisiting the Coffee Shop

Now, let's say that you're not the only actor in the system who can handle the request. Leveraging my last example of a coffee shop, you're working the register and have 2 baristas in your coffee shop. As you take another customer's order, rather than ask one barista to make the order, being told that they're too busy, and then asking the other, you simply tell the baristas that an order is waiting to be filled. You may recall that this is called being message-driven and that you probably tell them via a ticketing system. One of them, when they're available, then takes the next order in the queue and fulfills it. 

So, that's it! You are now refreshed on actors. I say refreshed because you already knew all this but just hadn't had it contextualized. Hello, Actor! It's nice to meet a fellow Actor!

Like, wait a minute.

Oh, I see what you're saying. As a fellow software engineer, what does this have to do with us? I got my next post (which I promise will be in the next few days, not next few months).

Friday, November 12, 2021

Akka Serverless - The Future of Software, Part 3

 Ahem! Did You Fall Asleep?

Well, now that you mention it, I did about a month of sleeping since my last post. Thanks for the nudge awake. In that interregnum we at Lightbend were busy adding features to Akka Serverless to get it ready to emerge from open beta

Oh, no! I just realized that I never told you that you can skip the queue (and my blog) to get right to playing with what I think is the Next Big Thing™️.

Now Where Was I?

Ah, yes, I'd introduced you to the Reactive Principles and set the stage for talking about building message-driven software. If that's a puzzled look I'm seeing on your face, go back and see my previous post to catch up.

Let me sum it up:

  1. You want always-responsive software so that users don't wander off when they see a


  2. Always-responsive software comes in the form of being both resilient to failure and...
  3. ...elastic to meet demands, both growing to handle high load and shrinking when those demands pass so you aren't always paying for maximum potential.
  4. The means by which that value of responsiveness is enabled is by making your software message-driven.

Treat Yourself to a Mocha

Let's stroll over to the neighborhood coffee shop and chat about what being message-driven means. We deal with messages all the time. For example, when we walked into the coffee shop, the opening of the door triggered a chime to notify the staff that someone had come in (or left) the shop, saving us from having to shout out to get someone's attention from the back. Shouting or chiming, we send a message and they attend to that message or the message goes unheeded. 

In software, dropped messages happen all the time, especially in this "always" connected world, and we end up writing retry logic or setting up alerts to a potential issue for paging someone. However, we want resiliency because an unheeded customer walks back out that chiming door to find coffee elsewhere. 

So, what do we do? At minimum, we could add an Ack(nowledgement). There's a fast food joint that does that very effectively by having every employee greet new customers. For us, that'd be like every instance of our service acknowledging every request—definitely overkill.

To be Ack'd, though, we'd have to be noticed. At our corner coffee shop, while they serve up the best mochas around, they only staff up during rush periods in an effort to be elastic. Otherwise, they only have one barista handling all aspects of the store. Our barista was in the back when we entered and didn't hear the chime. We know this barista is no slacker so we waited while another customer walked through the door, queuing up behind us. This time, the barista heard the chime and came up front, greeting us both. We got our Ack and are ready to make an order!

Let's Chew This Over

One of my favorite things about the mocha here is that they throw in a biscotti. Oh, you meant what did we learn!

We just saw a few components of being message-driven. First, any time we work with another person, we do so by either asking them or telling them some kind of message, commonly referred to as a request or a command, accordingly. Next, we recognize that we can't expect that request or command to be fulfilled unless we know that the message has been received. Finally, in an ordered world we need to make sure that requests and commands, collectively referred to as messages, are handled in the order they're received. This is even true if we're able to process messages in parallel. 

For example, let's say that just as our barista was taking the other customer's order, another barista came on shift. We wouldn't expect that barista to wait to make the order of the customer behind us. Rather, they'd start making our order while the barista taking the order makes the other customer's order. Ours is a more complicated coffee order so comes out after the other customer's coffee. However, order was maintained even while the coffees were crafted in parallel.

There is one more thing to add, which you've seen before, to finish our message-driven paradigm. When you place an order, a ticket is printed up. If it's just the one barista and you're the sole customer, that ticket may hit the trash pretty quickly. However, it's likely you've seen them carry it over with them to the espresso machine to be sure they're making your order right. When the second barista enters the scene, they start pulling the tickets right off the printer, in effect passing your message on to them to fulfill. 

The Actor Formally Known As


I said Actor, not Singer, and “formally”, not “formerly”. *sigh*

For every system, such as this mocha production line, there are actors in the system, in this case the customer and the baristas. Those actors are passing messages to each other and they leverage a system to make sure coffee order tickets are processed in order and don't get dropped on the floor.

The stage is now set to finally introduce you to Akka, the underpinning technology to Akka Serverless. There are many coordinating components to the Akka Platform but I want to focus on the Actor.

In my next post. 😎

Monday, October 11, 2021

Akka Serverless - The Future of Software, Part 2

You Were Saying?

When last we met, Dear Reader, I was leading into why I think Akka Serverless is the future of software, at least any software built to run in the cloud, by giving you a little of my back story. Today I'm going to talk about the foundations upon which Akka Serverless is built. To do that, I have to present a few...


re·​ac·​tive | \ rē-ˈak-tiv  \

Definition of reactive

1 : of, relating to, or marked by reaction or reactance

2  a : readily responsive to a stimulus

    b occurring as a result of stress or emotional upset

reactive depression

manifesto noun

man·​i·​fes·​to | \ ˌma-nə-ˈfe-(ˌ)stō  \

plural manifestos or manifestoes

Essential Meaning of manifesto

a written statement that describes the policies, goals, and opinions of a person or group

The group's manifesto focused on helping the poor and stopping violence.

a political party's manifesto

Value Proposition of The Reactive Manifesto

: In today's distributed compute environment, where services rely on other services, we are building systems which rely on unreliable partners. Even in the best conditions where everything is built correctly, outages happen. Steps must be taken to deal with outages with the greatest of integrity and availability, Certain patterns have emerged that enables this in a fashion that is also easy to reason over as it models the interactions we have on a human level with each other.

Ok, Class. Your Reading Assignment Is...

I really highly recommend you scooch on over to The Reactive Manifesto, and the more thorough treatise, The Reactive Principles, and then come back here. Go on. I'll wait.

Hmm.... I'm betting you didn't go or, like me, you now have 2 more browser tabs open in your endless array. I'll see if I can do a TL;DR version here.

First off, the ideas around the Reactive Principles aren't new. The Reactive Manifesto was something I tripped over in my design pattern studies back in the 20-teens. Jonas BonérDave Farley, Roland Kuhn, and Martin Thompson had collaborated to bring together the ideas around 2014 but the concepts preceded them by decades. They just hadn't come to be something we business application developers really needed to know (despite how much I wanted my peers and managers to think it was necessary). Then came The Cloud.

Nobody understands the cloud!

Suddenly, in my circles anyway, everyone wanted to put their .NET applications up in Azure. Yet, no one was building applications to run effectively in Azure -- even inside Microsoft. I don't mean any slight to the engineers I was working with. It's simply that most folks weren't so forward looking, or were not as exposed to distributed computing, as Jonas, et al, were.


First, I have to say that I'm standing on the shoulders of giants, here. Further, if I can't describe something simply, I probably don't understand what I'm talking about. So, when I suggest that I'm going to Keep It Simple, it's so that my Silly Self can actually explain it. And, honestly, this software design pattern is extremely simple, conceptually speaking. 

In a distributed system, which aptly describes all aspects of The Cloud, things you have absolutely no control over break. However, that's no excuse to let the end-user suffer, either catastrophically or silently. With a minimum of 3 core concepts, you reach the 4th needed to be a Reactive System, which is always responsive software. 

Being elastic & resilient results in responsive software, all enabled by being message-driven

Responsiveness is simply defined as always responding in a timely manner if at all possible. As noted above, that means you need your system to be resilient to failure. 

You also need to have your system responsive under varying loads.  You could over-prepare, something many corporations did in their own data-centers regularly, by running a system, understanding what it takes to run it, combine that with trends from the past, and then building out infrastructure to match the highest load, plus a little for growth. Unfortunately, that often means inactive servers just so you don't hit a critical tipping point that crashes your application. When building for The Cloud, though, you can think more elastically, right-sizing your resources so that they're not too big, not too small, scaled up or down just in time to be just right.

Parenting as a Software Analogue

None of that is possible, though, without the fundamental underpinning of being message-driven. If you work from a task list, an email inbox, a ticketing system, etc., you, Dear Reader, are message-driven. However, most developers are used to thinking about method calls. Whether because they're writing object-oriented code making internal calls or making web API calls, these have an inherent flaw any parent with a young child knows. You have to wait and monitor for the expected result, handling exceptions as they arise. It ties up the code, like that parent, until that operation is complete before another can occur. Even the exceptionally brilliant parent, skilled at multitasking, will run out of threads if they are often checking on, responding to, or redirecting the efforts of their child. 

The answer to all this is to introduce asynchrony. That brings me back to your task list. As a kid, you may have had a chores list. That allowed you to do your chores, unsupervised, and let your parent check on the status of your efforts when they had the bandwidth. They still may have had to take corrective action but they have achieved a higher level of productivity. 

Missed It By *That* Much

With both software and kids, this takes some maturity but by the time I was maturing from a naïve coder to a software engineer, the .NET Framework had matured to introduce the Task Parallel Library giving us Async/Await in v4.5, released in 2012. While Java's Futures preceded that by quite some time, CompletableFutures in Java 8 showed up in 2014. Until then, there were a lot of shortcomings that made truly asynchronous programming difficult. There are also some distinct differences, notably around thread use, between how .NET and Java got the job done but they both have the same flaw. They both still, inevitably, await the result of the call. 

Because it is possible to have a call never return, it is possible to have an unresponsive application even when built using asynchronous patterns. Ever been that parent (or the child here) who, all else done, just can't get their kid to pick up their toys? Yeah, that. You either have some timeout and error-handling (pick up the toys yourself, perhaps) or you both pass out before the battle of wills resolves.

There is another step in maturity needed, which brings me back to being message-driven. I took a lot of time to get to that because it is a simple, but profound paradigm shift. However, it is one that should be very familiar to you.

Stay Tuned

Did I say TL;DR version? If you're new here, you've now learned what those who know me already knew. "Long" is my middle name. This has gotten long, though still a lot shorter than either of the two sites on defining Reactive systems. So, I will pick up on my next post with what being message-driven means and how an old pattern became new again.

Wednesday, October 6, 2021

Akka Serverless - The Future of Software, Part 1

Some History (Mine)

For those just joining the game, you might peruse my posts and see that I have dabbled as a software developer/engineer in multiple languages. I got my start with BASIC back when Saint Jobs was working closely with The Woz to bring us the Apple II. I moved onto Fortran, failed to understand pointers with Pascal, helped build some stuff with VisualBasic, built production code with C#, came to understand pointers and data structures with C++, learned a bit of Java, enjoyed data munging with Python, and finally matured to the Software Engineer (SWE) title building enterprise services with C#. Then, wanting to see what life as a SWE was like outside of Microsoft, I returned to Java and, well, hated it. 

The tooling I had to use, IntelliJ IDEA, wasn't as familiar, thus not as good to me, as Visual Studio. It wasn't bad, and better than Eclipse, just really unfamiliar. The ways to handle asynchronous communication in Java were much harder to reason about, for me, than with C#. There was more need with Java to concern myself about threads than there had been for C#. The libraries were all foreign to me so I felt like a burden on my team although I'd already gained the Senior title. I understood architecture and patterns but I just didn't know the tools I was given.

My first professional foray into Java had two big things going for it, though. I had a great team who never made me feel like a burden and actually showed that they valued my expertise. It was great to work with a truly inclusive team. The other thing going for it was that they'd selected a toolbox that alleviated my difficulties with reasoning about asynchrony and dealing with threads. That toolbox was from a small, open source company named Lightbend and the primary tool was Akka Platform.

Calamity Strikes

Unfortunately, roughly 3 months after joining that team, the company decided they would shutter the offices here in the PNW. It was calamitous in that I was really enjoying working with these folks, the project we were collaborating on, and I was getting the hang of the tools, language, and loving Akka.

Fortunately, I was welcomed back to Microsoft and got to work on a notable team building services underpinning all of Microsoft Store. My responsibilities were notable and the work was impactful. The team was not inclusive, though, and I really missed that. While there were some senior members of the team who were always approachable and helpful, others made a habit of being the opposite. Further, some of the younger folks, around the same age as my last teammates (I'm a young grandpa, for what it's worth), were expressly dismissive and exclusive in their interactions.

Good Things Come

Thing is, I'm really not hard to get along with and that role prior to my return to Microsoft, the one that went sideways 3 months in, resulted in some friendships, too. A few of the folks I had worked with landed roles at another company in the area and, while I was working back at Microsoft, they were building trust in their management. That trust resulted in the opening of a role for me to join them a year after I'd rejoined Microsoft. And to add to my luck, they were building services for their SAAS product using Akka!

I was building a multi-tenant, streaming service leveraging Kafka using Akka Streams Alpakka. It was comprised of 3 microservices, each able to scale independently to handle the processing load as we moved customers from the overburdened, individual software instances onto this leaner, scalable solution. The scaling and resiliency were enabled via Akka Clustering as the docker images were running in Kubernetes in AWS using Akka Discovery's K8s API. It was a great opportunity to both dive deep into multiple technologies while mentoring a team of folks, most of whom this would be their first experience with most of it.

Hello, Is It (Really) Me You're Looking For?

Have you heard this one? "Knock, knock." "Who's there?" "Lightbend." "Lightbend? No way!"

I was shocked to hear it myself. Here I'd found myself being really productive and actually enjoying Java despite my misgivings about it being less approachable than C#, all thanks to their flagship product Akka Platform. Lightbend is calling me to consider joining them as a Senior Solution Architect? It was true and that is what I did. 

As long-time readers of this blog will know, I had long ago discovered that I love learning and sharing what I learn. I had a real talent at communicating complex ideas at various levels and had started turning my journey from software development to mentoring and developer advocacy (DA), back before I got tapped to focus more on my craft and pivot to a Software Engineer role. (The differences between developer and SWE are hotly debated in the industry and a topic for a completely different post.) This blog was started to make the DA pivot only to languish as I focused on being a SWE.

However, as a Solution Architect/Technical Account Manager (my current title) at Lightbend, I get to turn my attention back to developer advocacy. Yay! As such, you should expect more of that kind of content to return (and less of the personal grousing about bad service).

Stay Tuned

With that in mind, I'm starting this new series on a service from Lightbend, currently in public beta, that I'm proud to be sharing with you. As the title of the article notes, that is Akka Serverless and I do believe it is the future of building software, at least the backends for highly connected, distributed, scalable, and resilient software. As always, I'm making this up on the fly so my plan is a little loose right now. However, for sure the next post will be a quick overview of the foundations that Akka Serverless is built upon, why they're important, and why Akka Serverless lets you stop worrying about them. You've heard of Platform as a Service, PAAS? I'll show you why I think of Akka Serverless as PAAS+, where the plus is that you don't have to worry about the complexities of actually leveraging the power of PAAS when building your truly cloud-first software, just your business domain and logic.

Oh! I almost forgot to mention that I also picked up a new language, thanks to Lightbend. I'm now building with Scala. I'll be using Scala in some of my demos as it reads almost like pseudocode. However, Akka Serverless supports a wide variety of programming languages, including full tutorials for Java and JavaScript. So, you never know what language I'll demo next.

As the heading says, stay tuned!

Monday, June 28, 2021

Windows 11 Requires TPM 2.0...Or Does It?

A Picture Speaks a Thousand Words

Look, Ma! No TPM!

Note the lack of a Security Devices node in the tree. That means I don't have no stinkin' TPM chip, 2.0 or otherwise. Yet, as a longstanding Windows Insider, since my days as a Microsoft developer, I just updated to the Windows 11 Insiders build after it warned me that I wouldn't be able to run it without the TPM chip.

MacOS is pretty!

For what it's worth, I think it's visually stunning. But, then, I've always been a fan of the MacOS.

Saturday, January 30, 2021

Open Letter to Ubisoft

The Long Overdue Update

I did nothing. Yet, suddenly, it resolved itself. My Ubisoft content spontaneously started working about a week, or less, after I posted this and emailed those listed below. I have no illusions that my actions got any attention at all as I got no reply to the email. However, something changed and it wasn't my system. I know because I was monitoring my OS, after the many reinstalls mentioned below, to see if updates came along that would impact the launching of the affected titles in Ubisoft Connect. One day, they just started working again. It took several months total to resolve the issue, which is a damned shame, and no word from Ubisoft as to why or what they did to resolve the issue. I still maintain that their support, last I interacted with them, could stand to improve. However, I am again happily playing Assassin's Creed on my PC and I think, for the time, I'll limit my purchases of future titles to console to reduce the risk of losing access to them again.

For those who care, here's the original...

The Letter

To: [email protected]; [email protected]

Subject: Support nightmare


I know this is the kind of email you'd likely ignore but I am at my wits end with your support team. I have been jostled from tech to tech since 15 November 2020 over an issue launching both Assassin's Creed Origins and Odyssey on a computer which previously ran both perfectly well, including playing Origins to story completion. All other titles I own from Ubisoft, and all titles I have access to on Xbox Game Pass, including the extremely taxing Flight Simulator, run perfectly well. Yet, your support team continue to give me a handful of troubleshooting tips directly from your support website, ignoring any prior interactions which would alert them to knowing those steps have already been tried. Recently, they took a different tack stating that there is likely something wrong with my memory and, after running the recommended Memtest86+ fo4 24+ hours and passing 10 times with 0 errors, suggested it was low quality RAM or that 16GB wasn't enough.

Is there no escalation path from them to figure out this problem? It is a problem that has echoes reaching back to 2018 on Steam with people seeing issues with Uplay/Ubisoft Connect. I am at a point of never purchasing another Ubisoft title or signing up for a subscription due to this abysmal support experience. 


Jack Pines



I have been a long time fan of Assassin's Creed having played several on console and repurchased several on PC recently. I recently played through AC:Origins and was enjoying AC:Odyssey on my PC. The astute gamer will recognize that those aren't current titles. I have a serious game backlog to work through as I typically get a long weekend here or there to play games. Then COVID-19 happened and I have more shut-in time to put towards video games. 

So, having played through AC:Origins, I turned my sights on Odyssey. I'd begun the game forever ago when Google was beta testing Stadia. It was fun, Stadia worked well on an ethernet network, and I scored a free copy of AC:Odyssey. And then I got too busy to play it. Now I had no where to go and time to spare. So, I got to playing, had finished the area which culminated in killing my father (SPOILERS!), cleared the next area, and was tooling around the seas when I got busy again. A month later, I come back to the game, Ubisoft+ had been introduced to the public, and I was excited because it seemed a good value to subscribe and play the newest titles on launch. 

However, along with the new product came a rebranding of Uplay to Ubisoft Connect and at that moment AC:Odyssey stopped launching. It would show the static title image, not even the full screen title screen, and crash. Stymied, I then tried launching other titles in my library. Turned out that AC:Origins exhibited the exact same behavior. I'd scored a great deal on the original Assassin's Creed Directors Cut Edition so I fired it up. That worked great. So did Asssassin's Creed III Remastered, which I have been pleasantly suprised about given the reviews; no eye-popping moments so far. Even Watchdogs 2 runs perfectly. I'd love to try out newer titles to see if they have issues but I'm unwilling to spend any more money to only risk failure.

The Nightmare

I immediately searched Ubisoft's support documents for hints to what could cause this. Nothing did. So, I created a support ticket, as noted above on 15 November 2020. It has been 2.5 months of mostly frustration as I was bounced around no less than 24 support techs, listed for your entertainment.

Ubi-Ice Cream Cake

My favorite fake name was Ubi-UwU. There was no continuity of support so the same logs and steps were repeated many, many, MANY times. They even told me they escalated me twice. The first time, I believed them. The second time, I didn't because the first time didn't feel like an escalation after all and because they basically ghosted me for 3 weeks and then came back with their finding that I might have faulty memory. Mind you, no other game from any other source has issues but I was game. 

I downloaded the recommended Memtest86+ and ran it for over 24 hours. It passed 10 times and was on its way to an 11th, all with no errors. I reported this back to Ubisoft support and they suggested I buy new, higher quality memory to swap out and test with. My gaming rig is, admittedly, an aging Alienware Area-51 R2 but I don't think they'd sell me inferior memory when they support overclocking out the door, something I don't bother doing since it's never seemed worth the effort when everything ran fine as is. When I said as much back, they told me that it wasn't the quality they meant to challenge; it was the amount. Since when is 16GB, with 8GB free, not sufficient for gaming -- especially games which ran fine with that much RAM previously?!


Assassin's Creed has been a lot of fun to play through the years. I will enjoy playing through Watchdogs 2. However, unless some miracle happens and either support pulls their collective heads out their arses or I actually get one of the C-level dudes addressed on the email I sent and pasted above, I believe Watchdogs 2 will be the end of the line for my time playing Ubisoft games. It's a damned shame as the subscription really looked like it was going to be worth it for me.

Mocking GCP APIs containing iterator structs

New company & language, who dis? The astute amongst my readers may have noticed that I left Lightbend to go work with some friends who ...