Golang: A Simple Distributed Application Pattern

In the previous discussion, we talked

about how a simple concurrent pattern can be used to provide services access, and also a sturdy batch process

to carry out tasks over a periodic time interval. For this post, we are going to expand upon that pattern again

by using Golang, but adding a

built-in language construct that is somewhat unique to languages that are designed for concurrency.

The Need

Last time, the program communication was all internal to the application. The user talked to the services directly and to the background process through the services. This time around, we will do the same thing, but we will also add the ability for applications talk to each other. Ordinarily, in a loosely-coupled, distributed environment, our applications would be "physically" separated in different containers or VMs, and connected by queues, streams, or some other connective "tissue." In this case, we are trying to operate within one application and simulate the movement of data between programs with what we have on hand rather than additional outside servers. Helpfully, Golang has that covered for us with the notion of channels.

Internal Queuing with Channels

Channels are a first-class citizen in the Golang specification, which yields a few advantages. For one, you can use them just about anywhere (e.g. function return values and parms). Additionally, they are very easy to use, with just a few notational syntaxes to remember. And lastly, they are very lightweight so you can use as many as you need. This last point is important - in part because channels are so easy to use, some programmers feel the need to use them anywhere. While there are standard concurrency performance/side-effect risks with a cavalier usage of Go routines, the bigger issue is actually adding program complexity where none was necessary. If two things don't need to happen at the same time, why force it?

Channels have many uses, some of which we will explore later for more relevant discussions (e.g. pub/sub). Today, we are going to focus on using a set of channels to act as queues between separate applications.

The Application(s)

Our application is actually going to be four applications. The first app is as was described in the previous blog post: a handful of services and a heartbeat. We will expand the services a bit so that we can have visibility into our new applications, but other than that, the foundation of the application will remain the same.

The new applications will be essentially single-purpose functions that process words as they enter, each one with a simple perspective on what a word should look like, then push that word out to a channel that will be picked up by the next link in the chain. If we diagram the application with more of an "information flow" feel to it, it would look something like this:

For the initial version of the application, a web service will take a word, put it on a channel bound for Function A. Function A will operate on it (i.e. make it lower case), then put it on a channel bound for Function B. This will continue until Function C completes the journey by performing its duty on whatever comes across its channel. As far as the business purpose of each of the functions:

• Function A feels all words should be lowercase;
• Function B, believes that TitleCase is the way to go;
• Whereas Function C, knows that the world would be a better place if all words were shouted in UPPERCASE.

The Go routine that manages the application's meta data (APP MGMT PROCESS) still simply manages the application's heartbeat and logging process.

Definitions

Defining channels for use in our application is pretty straight-forward. First we define what our application is going to need:

var (
 lc chan string
 uc chan string
 title chan string
)

Then we carve out some memory to hold them:

lc = make(chan string, maxMessages)
uc = make(chan string, maxMessages)
title = make(chan string, maxMessages)

Channels have types, so only the specified type of data can be passed through them. They can be simple strings like we have defined here, but they can also be complex structs as well. Since we want to use these channels as simple queues, we have also defined a length in each of the make statements of maxMessages, which is defined as 100. Length allows channels to be buffered, which will come in handy in a later discussion about controlling our distributed architectures resiliency.

Practice

When a message gets placed on a channel with a format that looks like: channelName <- data, for example lc <- c.Param("word") this puts a word passed in from the web service onto the lowercase channel - any receiver for that channel gets notified that a message is waiting.

// AddToLowerCaseQ ...
func AddToLowerCaseQ(c *gin.Context) {
 lc <- c.Param("word")
 origAddress = append(origAddress, c.Param("word"))
 content := gin.H{"payload": "Accepted: " + fmt.Sprintf("lc: %d, Tc: %d, UC: %d, #: %d", len(lc), len(title), len(uc), len(origAddress))}
 c.JSON(http.StatusOK, content)
}<a href="https://www.captechconsulting.com/blogs/golang-a-simple-concurrent-web-services-pattern" target="_blank" rel="noreferrer noopener">
</a>

This is the service we are using to capture user input. The service accepts a string and then adds it to the lc channel. When that receiver gets notified, it pulls the data from the channel and performs its operation on that data. Let's look at the Go routine receiver for making lowercase words:

// Make lowercase.
go func() {
 for {
 select {
 case item := <-lc:
 item = strings.ToLower(item)
 lcAddress = append(lcAddress, item)
 title <- item
 }
 } // for
}() // go func

The go func() {} just puts the bracketed code into the background and the for {} says run it forever. Without the for everything would fall straight through and we would miss anything coming in on the channel we are watching. The workhorse of our routine however, is the select {} block. The select is similar to a switch, except the select only looks for communication elements. And here, we are watching for anything coming across the lc channel with the case item := <-lc block. If nothing is there, we go back around and check again, and again, and again until we get data. Once that happens, we convert it to lowercase, append it to our trophy case so we can show our work to the web service we wrote to show it off GetLCAddress(), then push the updated data to the title channel where the TitleCase Go routine performs its work on it.

Now, yes, we could probably enhance the select statement to look for all of the channels in the program and it would work fine for our simple needs, but the goal here is to simulate multiple applications in a distributed environment so each Go routine deals with its own concern.

If you notice in the TitleCase and UPPERCASE Go routines, we are waiting 100 & 50 milliseconds respectively before we place check the channel channel again. There is no functional reason why we are doing this, but we want to be able to show progress in the GetTotals() and GetAllAddress() services (otherwise everything happens too quickly), but also because in our next installment, we are going to talk about circuit breakers, and delays are a convenient tool to make that pattern work.

Outputting the Proof

First, let's give the app something to do. The Gettysburg Address has 271 words that we can donate to the process so we will start be sending them in through the foreground process:

./gettysburg.sh which is a simple wrapper around a bunch of curl statements.

While this is running, we can check the progress from another terminal by running: curl -X GET http://localhost:7718/totals

This should yield something similar to this: {"payload":"lc: 109, Tc: 56, UC: 55, #: 209"}

What this tells you is that 209 words have been submitted, 109 have been converted to lowercase, 56 to TitleCase, and 55 to UPPERCASE. If you keep running this web service call, you'll see the totals change accordingly. When the script is complete, the result will look like this: {"payload":"lc: 271, Tc: 271, UC: 271, #: 271"}.

If you want to take a look at the results as they are building, the all service will show you what each of the result arrays look like side-by-side (this is a little modified for clarity).

<a href="https://www.captechconsulting.com/blogs/golang-a-simple-concurrent-web-services-pattern" target="_blank" rel="noreferrer noopener">"LC":["four","score","and","seven","years","ago","our","</a>fathers","brought","forth","on","this","continent","a","new","nation","conceived","in","liberty","and","dedicated","to","the","proposition","that","all","men","are","created","equal","now","we","are","engaged","in","a","great","civil","war","testing","whether","that","nation","or","any","nation","so","conceived","and","so","dedicated","can","long","endure","we","are","met","on","a","great","battlefield","of","that","war","we","have","come","to","dedicate","a","portion","of","that","field","as","a","final","resting","place","for","those","who","here","gave","their","lives","that","that","nation","might","live","it","is","altogether","fitting","and","proper","that","we","should","do","this","but","in","a","larger","sense","we","can"],

"Title":["Four","Score","And","Seven","Years","Ago","Our","Fathers","Brought","Forth","On","This","Continent","A","New","Nation","Conceived","In","Liberty","And","Dedicated","To","The","Proposition","That","All","Men","Are","Created","Equal","Now","We","Are","Engaged","In","A","Great","Civil","War","Testing","Whether","That","Nation","Or","Any","Nation","So","Conceived","And","So","Dedicated","Can","Long","Endure","We","Are"],

"UC":["FOUR","SCORE","AND","SEVEN","YEARS","AGO","OUR","FATHERS","BROUGHT","FORTH","ON","THIS","CONTINENT","A","NEW","NATION","CONCEIVED","IN","LIBERTY","AND","DEDICATED","TO","THE","PROPOSITION","THAT","ALL","MEN","ARE","CREATED","EQUAL","NOW","WE","ARE","ENGAGED","IN","A","GREAT","CIVIL","WAR","TESTING","WHETHER","THAT","NATION","OR","ANY","NATION","SO","CONCEIVED","AND","SO","DEDICATED","CAN","LONG","ENDURE","WE"]<a href="https://www.captechconsulting.com/blogs/golang-a-simple-concurrent-web-services-pattern" target="_blank" rel="noreferrer noopener">
</a>

As you can tell, the artificial delays we put in place show how our Go routines can handle backups in processing without any issues (to an extent). And because of the way we are siphoning data through channels, we don't have to worry about race conditions and the words coming out in different sequences.

Once everything is done, if you want to check the final tally of our three processes, it's pretty simple to do. To see the lower case results: curl -X GET http://localhost:7718/lc - which should get you this:

{"payload":"four score and seven years ago our fathers brought forth on this continent a new nation conceived in liberty and dedicated to the proposition that all men are created equal now we are engaged in a great civil war testing whether that nation or any nation so conceived and so dedicated can long endure we are met on a great battlefield of that war we have come to dedicate a portion of that field as a final resting place for those who here gave their lives that that nation might live it is altogether fitting and proper that we should do this but in a larger sense we can not dedicate we can not consecrate we can not hallow this ground the brave men living and dead who struggled here have consecrated it far above our poor power to add or detract the world will little note nor long remember what we say here but it can never forget what they did here it is for us the living rather to be dedicated here to the unfinished work which they who fought here have thus far so nobly advanced it is rather for us to be here dedicated to the great task remaining before us that from these honored dead we take increased devotion to that cause for which they gave the last full measure of devotion that we here highly resolve that these dead shall not have died in vain that this nation under god shall have a new birth of freedom and that government of the people by the people for the people shall not perish from the earth"}<a href="https://www.captechconsulting.com/blogs/golang-a-simple-concurrent-web-services-pattern" target="_blank" rel="noreferrer noopener">
</a>

To see it in its TitleCase version, call the title case service: curl -X GET http://localhost:7718/title - which should get you this:

{"payload":"Four Score And Seven Years Ago Our Fathers Brought Forth On This Continent A New Nation Conceived In Liberty And Dedicated To The Proposition That All Men Are Created Equal Now We Are Engaged In A Great Civil War Testing Whether That Nation Or Any Nation So Conceived And So Dedicated Can Long Endure We Are Met On A Great Battlefield Of That War We Have Come To Dedicate A Portion Of That Field As A Final Resting Place For Those Who Here Gave Their Lives That That Nation Might Live It Is Altogether Fitting And Proper That We Should Do This But In A Larger Sense We Can Not Dedicate We Can Not Consecrate We Can Not Hallow This Ground The Brave Men Living And Dead Who Struggled Here Have Consecrated It Far Above Our Poor Power To Add Or Detract The World Will Little Note Nor Long Remember What We Say Here But It Can Never Forget What They Did Here It Is For Us The Living Rather To Be Dedicated Here To The Unfinished Work Which They Who Fought Here Have Thus Far So Nobly Advanced It Is Rather For Us To Be Here Dedicated To The Great Task Remaining Before Us That From These Honored Dead We Take Increased Devotion To That Cause For Which They Gave The Last Full Measure Of Devotion That We Here Highly Resolve That These Dead Shall Not Have Died In Vain That This Nation Under God Shall Have A New Birth Of Freedom And That Government Of The People By The People For The People Shall Not Perish From The Earth"}<a href="https://www.captechconsulting.com/blogs/golang-a-simple-concurrent-web-services-pattern" target="_blank" rel="noreferrer noopener">
</a>

And finally, for the UPPERCASE version: curl -X GET http://localhost:7718/uc - looks like this:

{"payload":"FOUR SCORE AND SEVEN YEARS AGO OUR FATHERS BROUGHT FORTH ON THIS CONTINENT A NEW NATION CONCEIVED IN LIBERTY AND DEDICATED TO THE PROPOSITION THAT ALL MEN ARE CREATED EQUAL NOW WE ARE ENGAGED IN A GREAT CIVIL WAR TESTING WHETHER THAT NATION OR ANY NATION SO CONCEIVED AND SO DEDICATED CAN LONG ENDURE WE ARE MET ON A GREAT BATTLEFIELD OF THAT WAR WE HAVE COME TO DEDICATE A PORTION OF THAT FIELD AS A FINAL RESTING PLACE FOR THOSE WHO HERE GAVE THEIR LIVES THAT THAT NATION MIGHT LIVE IT IS ALTOGETHER FITTING AND PROPER THAT WE SHOULD DO THIS BUT IN A LARGER SENSE WE CAN NOT DEDICATE WE CAN NOT CONSECRATE WE CAN NOT HALLOW THIS GROUND THE BRAVE MEN LIVING AND DEAD WHO STRUGGLED HERE HAVE CONSECRATED IT FAR ABOVE OUR POOR POWER TO ADD OR DETRACT THE WORLD WILL LITTLE NOTE NOR LONG REMEMBER WHAT WE SAY HERE BUT IT CAN NEVER FORGET WHAT THEY DID HERE IT IS FOR US THE LIVING RATHER TO BE DEDICATED HERE TO THE UNFINISHED WORK WHICH THEY WHO FOUGHT HERE HAVE THUS FAR SO NOBLY ADVANCED IT IS RATHER FOR US TO BE HERE DEDICATED TO THE GREAT TASK REMAINING BEFORE US THAT FROM THESE HONORED DEAD WE TAKE INCREASED DEVOTION TO THAT CAUSE FOR WHICH THEY GAVE THE LAST FULL MEASURE OF DEVOTION THAT WE HERE HIGHLY RESOLVE THAT THESE DEAD SHALL NOT HAVE DIED IN VAIN THAT THIS NATION UNDER GOD SHALL HAVE A NEW BIRTH OF FREEDOM AND THAT GOVERNMENT OF THE PEOPLE BY THE PEOPLE FOR THE PEOPLE SHALL NOT PERISH FROM THE EARTH"}<a href="https://www.captechconsulting.com/blogs/golang-a-simple-concurrent-web-services-pattern" target="_blank" rel="noreferrer noopener">
</a>

Some Cautions

Keep in mind that the need for good programming practices remains in effect. Use the right tool for the right job. Concurrency requires you to think about solving your problems in a different way. Just because something can occur at the same time doesn't mean that it needs to. Even though Golang's concurrency model is very easy to understand and use, you can still fall victim to standard concurrency problems like race conditions. Using channels helps order the communication and processing order, but still, be mindful - concurrency and distributed patterns require a different way of thinking (e.g. separation of concerns, "read one-transaction, solve one-problem," simultaneous behavior, queue traceability, and dead-ends).

Summary

What we did in this article was expand our original concurrent pattern into a pattern where we could communicate between any number of background processes via Go channels. This pattern lets us model a distributed application in a completely self-contained environment. From here, we can prototype, test, or even instruct on cloud-native designs that normally would utilize queues, pub/subs, and streams - without having to stand up those assets.

Next time, we will expand upon this pattern by adding resiliency so we can use our setup to throw volume at various pieces of our application to see how it those pieces react under pressure. That will prove useful for some basic architecture validation before we expend a lot of effort actually constructing infrastructure.

Source code for this article can be found here.