This is the tenth article in the attempt to form a blogging habit

They say you cannot teach old dogs new tricks. It has been proven wrong many times, but the idea is, that once you have your own ways, it’s hard to deviate from them.

Last summer I had plenty of time for experimentation. I’ve also had an old development idea I wanted to take on the road for some time. As I was very much stuck in the world of the Java enterprise, I decided to give the “new age” methodologies/technologies a go. I’ll summarize the stages I’ve taken to learn, create and refactor the architecture to it’s current, still non-final stage.

tl;dr

Experimenting with various ideas is always worth it, even if you end up where you started with.

Quite a few changes got hyped in the past years in IT, NoSQL, Javascript, microservices, disposable containers, responsive design, to name a few. Some have broken out from behind the barriers of in-house development for huge companies, some were designed by standard boards, lead by rivaling corporations and some creeped out from the basement of some geek. All-in-all I think this has brought a new age of thinking, with changes not unlike the one witnessed when before the first Internet bubble went up.

To get my project on the road, I decided to go create a small set of loosely coupled microservices, running in separate docker  containers, using a cassandra instance as the db backend, with an AngularJS front-end. For the programming language of the services I’ve chosen to go with a multi-language approach. Some services were to be written in Groovy, some in Javascript using NodeJs, and of course the front-end in HTML5 using Bootstrap, and Javascript for the AngularJS.

To start with I had to get out and learn all those, and I would have to admit I kind-of skipped that chapter. I was pretty confident of my skills and experience to pick it up as I go. Also I didn’t want to invest too much time to learn things that I might dispose really quickly.

Initial architecture

Originally I started off with a multi-tiered design, that you can see on the image.

Only a single component is exposed to the outside world, the request broker. It’s nothing more, than a very simple HTTP proxy, that routes the service specific requests to their corresponding service providers. Any requests not targeted to a service are routed to the UI. It used a round-robin mechanism to direct requests to multiple services on the secure network after authorization, or to other broker instances on unsecured networks. The assumption was, that the last broker in the chain should perform authorization. In terms of patterns it performed server-side discovery as well as service registry function.

Each component is running in their separate docker container,  in accordance with the service per container pattern, and the connections between them are implemented using docker links. This allows for the coexistence of separate environments on the same machine, as well as substituting your service implementation during development. In the future, this allows to deploy the services to cloud providers, such as the Amazon ECS.

It looked like a nice design, but was not really to my liking. It was good for experimentation, learning the ropes with the technologies, and delivering the first implementation.

Issues with the implementation

Dealing with the docker links is not as easy as one would think. The docker links are static to the container,  if you recreate a container, you must recreate all the dependencies. As you can see from the diagram above, quite a few dependencies had to be recreated upon a single change. On my development environment, the services were running bare, without the containers, and that seemed to keep this overhead down. On my test server however I had to do some heavy Bash scripting to recreate the environment after a code change.

I’m sure there must be some easier alternative to this, but setting up an environment for instance and service management such as zookeeper or consul, would have taken even more time. Dynamic container linking is still in its proposal phase in docker, and the ambassador pattern is a pain to implement, with little actual possible gains.

To replace a service in the development test environment, with the on on my computer, I had to do lots of messing around with my firewall, to allow the exposed ports through.

As the docker images were created with the assumption of running multiple instances on the same environment, it’s local ports were not fixed upon instance creation. This way the exposed docker ports were randomized, and it also took some effort to reconfigure the local environment to match the exposed ports.

The broker also had a static configuration, which had to be messed about whenever I wanted to substitute a service from my development environment. The broker had no idea if any of it’s services were available, and had to be restarted if any service changed.

Refactoring for scale-up

The first thing to simplify the development was to get rid of the single service written in Groovy. It was quite heavy to start with, to run a REST service I had to start up an entire Jetty container and run my few lines of Groovlet. I reimplemented it as part of another NodeJs service. It reduced complexity, and overall startup time.

Second change was to remove the nodeJS server that ran the UI, and replaced it with an nginx instance. After all the client is basically no more than a number of static pages, all dynamic content is provided by the services.

After a while I realized, I was spending way too much time reconfiguring the environment instead of doing development, so I decided to switch to a dynamic service discovery pattern.
Client side service discovery is a well known pattern in SOA and microservices, each client would look up the services it would require from a central service repository. Each server in turn would register itself upon availability.

After a little research I decided to use an etcd data store as the basis of my service registry. (Another NoSQL to learn) Etcd is designed to be a service discovery database, it organizes data as a tree, and handles time-to-live for both the tree nodes and the values within.

Since all my NodeJS based nodes were used as both clients and servers, I had written a simple component to Broadcast availability. I’ve refactored my nodes to use the service registry, that was no more than a simple etcd data store.

Things started looking up, the hardlinks were removed from docker images and instances, the deployment script was simplified, dependency management was purged from the code. The broker no longer had a config file detailing all the endpoints. I no longer had to worry about lining up the ports with my development machine, and the test environment.

Issues with the implementation

I’ve created some NodeJS code, that implemented the client-side discovery, and that basically created a client-side registry copy for all the services. Soon I’ve realized how silly that was.

All registry changes were pushed to all nodes, no matter what. As the etcd client handles service re-registration the same way as creating the service, all nodes started receiving all re-registration events. What’s worse they were polling all the registry for changes upon timeout. I couldn’t figure out how to reduce this unnecessary load from inside the clients.
A better solution to solve the issue would have diverted me from working on the actual project implementation, but would have been similar to the one on this drawing. A separate service registry service should have been implemented, that would only pass service registration/keep-alive to the etcd and allow dynamic service lookups from the clients. This would have to be a singleton for the given docker environment and keep the latest relevant registry information.

Simple is beautiful

I’ve decided to go with a more conventional approach from here. I wanted some results, and a way simpler solution.

By now the application was properly divided into service layers, authentication, data access was separated from the business logic. So I collapsed it back to old-school three tiered architecture, and lived happily ever after.

To do that, the boilerplate code responsible for the REST services was removed from the modules, the entities were merged, error handling got unified.

Now it’s much easier to trace a call, I don’t have to walk through all the modules in different projects to reach to the database from the http request. I could even debug the AngularJS ui using WebStorm, which is a great help compared to the Chrome debugger, which was impossible using the broker.

Since the services are designed to be stateless to begin with, it is very easy to scale up, by adding a simple request based load balancer to target multiple copies of the service node. As soon as parts of the application grow big, they can be moved to their independent service module, keeping in mind the hiccups, I’ve described above.

Takeaways

It was really interesting to apply the ideas of microservices to a project. It was also useful to prove, that it takes a certain level of complexity to actually require breaking up the application as services.

I’ve had the chance to see the way an application can be broken up in reverse. The logical layers, that I always insisted on, while developing applications are the very points, the application can be taken apart and reassembled. It has also shown how really important it is to follow development standards ruthlessly. If I didn’t, and the organization of the modules were different from one another, it would have been difficult to merge them together. If viewed from the other side, it makes separating modules very difficult, when the boundaries are unclear.

Even though I struggled with some technologies at first, it was also useful, that I didn’t dig myself into the tutorials at first. I can’t even count the number of different frameworks, that I experimented with, the theories I embraced and discarded during the time. If I took any more time with a technology, or polished a component a bit longer, I might have not ended up here, but have a monstrous tech heavy framework, I’d have to spend mending, instead of doing any development on my actual project.

As long as it’s a one-man show with no actual release, there is really no reason to be too heavy on those.