software

Architecture of psbFinances

Vladimir psbFinances, software ,

Every time you start a new application, you ask yourself: what should I use to build it. Should you stick to what you know or use something new. I’ve made a few decisions about the architecture of psbFinances. For the last 9 years I’ve been working with node.js, React and MySql. I’ve built platforms for 6 startups with this stack. That’s what I used for psbFinances. But there was a couple of decisions I’ve made “inside” this stack.

Note that React is becoming less obvious choice for me these days, but that’s another topic.

TypeScript (TS)

I am big fan of Anders Hejlsberg. I was working with C# since he introduced its first beta. Even before that I’ve used his Borland Delhi. When he announced TypeScript in 2012 I was very interested. He gave many reasons why TypeScript was developed. Great tooling support (like IntelliSence, code completion) was one of them. Since then I was checking the growth of it and was asking myself whether I should use it for my projects. I’ve asked the same question many teams that I’ve lead: “should we go with pure JavaScript or with TypeScript”. Interesting situation. JavaScript developers wanted to have static typing. At the same time more C# developers were asking for dynamics.

9 years ago most of the JavaScript IDE didn’t have a good support for it. But the situation has changed a lot since then. I use WebStorm. It has a very good support for pure JavaScript development, especially in combination with JSDoc. Visual Studio Code has a great support for it too. For me using JSDoc makes code less verbose and cleaner compare to the type annotations in TS.

As for the static code analysis. Incorrect types add only a minor number of bugs that I discover in my and my teams code. Most of the bugs are in the logical flow. Type validation is also address as part of unit tests.

TypeScript allows to use the latest features of JavaScript. Browsers and node.js may not support these features yet. But you probably use Babel and that solves this issue.

That’s why I’ve decided against TypeScript for psbFinance. I don’t see enough benefits to use another “super” language on top of JavaScript.

Some developers say that “you can’t build large-scale application without TypeScript”. That’s not true. Large scale application can be built using any language. I’ve seen quite enough of them built with different languages. It’s not about language. It’s all about the skills and discipline of developers.

GraphQL

That was another tool I decided not to use. I use REST API. A debate of REST API vs GraphQL reminds me a lot about the discussion between backend developers and DBAs. DBA didn’t trust developers to be “gentle” with their databases. They insisted that database should be accessed via stored procedures only. That make sense. To some extend. “With great power there must also come great responsibility”. That debate was won by developers who now can write direct SQL statements or use some ORM helpers. I was never in the DBAs’ camp, even though I’ve done a lot of DBA work. For developers that power of accessing database with pure SQL is great. But some of them didn’t accept the responsibility. Many times I’ve seen lazy SQL code that degraded performance and introduced security issues.

That’s why it looks the same for me with GraphQL If you built multiple applications with REST API then you know how to address some challenges.

I am not saying that GraphQL is bad or that it should not be used. Far from that. What I am saying is that based on my experience and skills it doesn’t give me any significant benefits while introducing some complexity. It also means that the potential pool of developers who can contribute to psbFinances becomes smaller – not everybody know it.

Conclusion

TypeScript and GraphQL are great tools. They can be very useful in the right hands. But for me they don’t have enough benefits to use in this project. Check the code and you can see that it’s ok to live without them.

Giving birth to a software

Vladimir software

There are a few exciting moments for a software developer when he builds a software (and love doing it). First “clear” run, 1M customers using it without a problem, … For me the most exciting moment is the “birth”. At first there is nothing, nothing material – just some thoughts in the people’s heads. And then you create a folder, name it, add some files and run it. And it runs. It does nothing really, but it runs! There was nothing, and now there is something. This “something” is small, useless and ugly, but you see the future beauty of it. You see people who will use it, love it (or dislike). It will solve their problems and make their life easier.

But it’s still just yours at this moment. And then you commit. And boom, it’s part of the world, not just yours anymore. Even if it’s only your partner or code repository who knows about it. It’s out “there”.

And although I do it every day for some “exploration” projects, it’s different. That doesn’t bring that excitement. But starting an application that will “live” is like planting a tree or breaking ground for a new home. I may not see the tree grow or build the most of the home, but it’s all about starting it. Yes, I love to see a working product in the hands of the people. It feels good. But starting it …

I see quite often auto-generated comment in the code “Created by …“. I never like it. But when you commit the code, you have to give your name. And I know that there are a few products around that have my name at the root. And it brings a happy smile to my face 🙂.

A few day ago a new tree was planted …

Better TODO comment

Vladimir software ,
Here is a simple trick that makes TODO comments more efficient. You have seen TODO comment like this all over the code:
//TODO: convert to prepared statement.
var query = 'SELECT processedDateTime FROM ProcessedEvents WHERE hash = ?'
 
Yes, it’s good to have it. This is a way to track a technical debt. And here is how it can be much more useful:
 
//TODO: vvs p2 convert to prepared statement.
var query = 'SELECT processedDateTime FROM ProcessedEvents WHERE hash = ?'
 
Just 2 small pieces are added:
– vvs – your initials,
– p2 – priority (p1, p2, p3)
 
Now, you can search the whole code base for your own TODOs (or TODOs created by a specific person). And when you see a comment like this there is no need to go to Annotate or Blame to find out who has left it (save time).
 
When you leave a TODO comment you are in the best state to analyze the impact of the incomplete work. That’s why putting a priority at that moment is so crucial.
 
It’s a simple trick that can save you tons of time, especially when you manage a team and do daily reviews of all commits.

Easy mocking in Node.js

Vladimir software ,
I’m a big believer in unit testing. Not only because it allows me easy code refactoring and faster regression testing, but because I can deliver product much faster doing unit tests. Yes, writing more code (unit tests) shortens product delivery time. But this is another topic.
 
Efficient unit testing is not possible without mocking. In Node.js I was a happy user of rewire. I liked it. Until today. For whatever reason it stopped working for me. Either it doesn’t like me anymore or something wrong with me, but it refuses rewire modules.
 
So, I’ve spent some time looking for another mocking framework. There is a number of good ones around. But all of them didn’t feel 100% right. Especially when I compare them with framework available for C# or Java that I’m used to.
 
So, can mocking be easily and nicely done without framework? Sure, the same way I have never used any IoT/DI frameworks with C# – always used simple manual injection.
 
And it’s actually very easy with Node.js.
 
In the following code, I need to test create function. There are 2 calls that need to be mocked for unit tests: inboxDb.create and inboxProcessor.processEvent. Here is a manual mocking with  just a few lines of code – check the mock setters (t the bottom of the code.
 
'use strict';
var moduleName = 'inbox.controller'

var Promise = require('bluebird'),
appRoot = require('app-root-path')

var inboxDb = require(appRoot + '/server/components/inboxProcessor/inbox.db'),
MessageModel = require(appRoot + '/server/api/messaeg/message.model'),
inboxProcessor = require(appRoot + '/server/components/inboxProcessor/inboxProcessor')

/**
 * Processes incoming message.
 * @param req
 * @param res
 */
exports.create = function (req, res) {
 var payload = req.body.m
var ipAddress = req.connection.remoteAddress;
console.log('%s|create|payload:%s', moduleName, payload)

 var message = {}
 inboxDb.create(payload, ipAddress).
 then(function(result) {
 try {
 message = MessageModel.parsePayload(payload, ipAddress)
 return Promise.resolve(message)
 } catch(error) {
 console.log('%s|create|parsePayload|Error: ', moduleName, error)
 return res.send(204, 'poison')
 }
 }).then(function(event) {
 return inboxProcessor.processEvent(event, payload, undefined)
 }).then(function(result) {
 return res.json(201, result)
 }).
 catch(function(error) {
 console.log('%s|create|Error: ', moduleName, error)
 return res.send(500, error)
 })
}

/* Mock setters */
exports.setInboxDb = function(mock) {
 inboxDb = mock
}
exports.setInboxProcessor = function(mock) {
 inboxProcessor = mock
}
 
Now, with this setters in place, mocking is actually very easy in the unit test:
'use strict';

var Promise = require('bluebird'),
should = require('should'),
request = require('supertest'),
appRoot = require('app-root-path')

var app = require(appRoot + '/server/app'),
target = require('./inbox.controller')

describe('POST /api/inbox', function () {

 it('should process message', function (done) {
 target.setInboxDb(setDbMock('ok', false))
 target.setInboxProcessor(setInboxProcessorMock('ok', false))

 var data = { "m": "message - actual content is not important"}
 request(app)
 .post('/api/inbox')
 .send(data)
 .expect(201)
 .expect('Content-Type', /json/)
 .end(function (err, res) {
 if (err) return done(err)
 res.body.should.equal('ok')
 done()
 })
 })

})

function setDbMock(result, throwError) {
 var mock = {
 create: function(payload, ipAddress) {
 if (throwError) {
 var error = {message: 'db error'}
 return Promise.reject(error)
 }

 return Promise.resolve(result)
 }
 }
 return mock
}

function setInboxProcessorMock(result, throwError) {
 var mock = {
 processEvent: function(event, payload, next) {
 if (throwError) {
 var error = {message: 'error'}
 return Promise.reject(error)
 }

 return Promise.resolve(result)
 }
 }
 return mock}
 
By replacing actual invocation with our mocked functions, the code is easier to test and debug (in this case I’ve just created a wrapper that allows me to return some result or throw an error):
target.setInboxDb(setDbMock('ok', false))
target.setInboxProcessor(setInboxProcessorMock('ok', false))
 
Yes, it’s minimalistic and doesn’t have all the bells and whistles of other frameworks. But, it’s simple and will cover 80% of the mocking needs.
 
Update. I’ve opened another set of unit tests with rewire today and it worked flawlessly. I’m not sure why the first set didn’t work, but I’m glad that I’ve found this simple solution and will stick with it for now.
 

Data analysis with Hadoop and Hive. New tools. Old game.

Vladimir software ,
I’ve done a lot of data analysis in my life because more than half of my career I was a part of an actual business rather than ISV. I’ve worked for commodity exchange broker, wholesale and retail companies, Coca-Cola, Bayer and other businesses where I did a lot of analytical work on top of software development. I’ve done my first data analysis project in late 80s with Lotus 1-2-3 when circumstances forced me to pause software development and sell jeans (BTW, that was a great experience at the end – learn how to make a sale). And since then I’ve learned the power of databases and spreadsheets that allowed me to dig the data and help businesses grow.
 
Recently I’ve done another data analysis project with new tools – Hadoop and Hive. I’ve been working with them for the last 2+ years, but more from a development perspective. This was the first time I did actual data digging with them. And just an hour into the analysis I understood that although these are new tools, the actual process is the same. Yes, they provide some structural (sets, arrays, maps) and scalability advantages, but one needs the same data analytical state of mind to get the answers from the data.  And the fact that you have this power doesn’t mean that you have to use it right away. 
 

Getting answers from the data is an iterative process – you write query, analyst the result, optimize your queries and repeat the process. And that’s when you want to move fast – before you make The Query that will give you the ultimate answer. This fastness to find the path is especially important with large data sets and Hadoop latency – start small, move fast until you are really ready to “Release the Kraken”.

 
So, I think that Big Data analysis has not a lot to do with Hadoop & Co, well … unless you are on the operation side. It’s still more about how to dig the data and find a way to answer the question fast before forming it into a repeatable process suitable for the tools.
 
And that’s why learning how to use spreadsheets to analyze data is so important – it teaches how to find the answers. 

Software Structural Quality, or Lessons from Open Source Projects

Vladimir software
Software quality is important. Very important. Not a lot of people will argue with this. That’s why we have “tools” like QA and TDD – to ensure that we deliver a high quality software. But most of these “tools” address functional quality of the software. But what about the structural quality? Isn’t it important as well? How often will you see a bug like this: “Method xxxxx is difficult to read and it’s too complex”? Maybe structural quality is not that important and is primarily a topic of software craftsmanship?
 
“Software structural quality refers to how it meets non-functional requirements that support the delivery of the functional requirements, such as robustness or maintainability, the degree to which the software was produced correctly.” If the software has 0 bugs (what a noble goal), but is neither easily readable nor maintainable, is it good? “Hey, man, it works”. Yes, no doubt about that. How does that help when new features should be added or existing code should be changed? Unless code has a high structural quality it cannot be easily changed and it will take more and more time to deliver new features. Also, eventually it will effect functional quality, because changing something, even something very small in a very bad code can lead to an unpredictable behavior (read: bugs).
 
So, if a company would like to stay in a business and release new versions of their software, then sooner or later they should take care about “internal” quality. The sooner the better, before it’s too late. How may times have you heard something like this “I’m not touching that code – nobody knows what will happen”.
 
Then why do so many companies produce software that works, but is ugly inside? One of my ex colleagues once said: “If our customers could see our code, they would never buy our application”.
 
“Hey, it works. Don’t you get it?”. Yeah, I get it.
NewImage
I’ve been in many companies, but I have not seen high structural quality in code very often. That’s why it’s even more surprising when you look at the quality of the popular open source projects. Most of them are very good – easy to read & understand, nicely structured and covered with multiple unit tests. Shouldn’t it be an opposite – as my job I’ll write a high quality code, but I can be easy on myself with this for-fun projects? Funny, but it doesn’t work this way, because it’s impossible to add crap to an open source project without it being seen and accepted by everybody. But it’s absolutely possible at work – as long as I have 0 bugs, I can put whatever crap I want, nobody will see it anyway. “Don’t you have a code review process?”. Yeah, right – if I contribute crap, why would I reject your bad code.
 
So, what can open source projects teach us the about structural quality of software? 
 
First of all, there should be somebody who cares. Who cares about good code. Who will be a pioneer of “let’s be proud of what we write”.
 
Second, create a culture of good code. Yes, it’s possible to setup a process, but unless the team believes in good code, there is always a way to cheat the process.
 
Have you ever tried to open a PC for upgrade? Yeah. You know what I’m talking about. Have you tried open Mac? Mac Pro is beautiful inside. That was the way Steve Jobs wanted it – the product should be beautiful even inside.
 
And BTW, writing good code is much more fun. So, have fun 🙂 and be proud of what you write.
 
Update: Chris Hume volunteered to edit this post and make it better. Thanks, Chris.
 

What I like about C#

Vladimir software

Recently I’ve been asked what features I like most in C#. I’ve been working with it for almost 14 years now (since the end of 2000), and today I use it almost daily along with Java and JavaScript (in Node.js). It’s so fun to witnesses the growth of the language – from the birth to teenage age and to maturity.

 C# is a very nice language and I find the following features of it most useful:

– Generics brings the concept of type parameters. This makes code very clean especially when dealing with collections.

– Linq – added standard and easy-learning pattern for querying and updating data. It’s like SQL for C# collections. Linq allows developers to express what should be done instead of how (forget about for, while loops). I especially like to use it in the form of Lambda Expression. Although .NET framework has a functional language – F# – the ability to do functional programming in C# is great. 

– Parallel Processing – 

– Tasks. If you’ve ever done threading in C# (or any other language for that matter) the simplicity that Task provides in dealing with threading is a life saver. There is a reason why Microsoft recommends that we use Tasks instead of Thread/ThreadPool directly – not only it will work better, but code can be written much faster and cleaner.

– async / await

– dynamics. There is a reason why dynamically typed languages (Ruby, Python, JavaScript) are so popular. If you have never experience before it feels awkward at first, but then you get it and have a blast. I was working with dynamic language since early 90th, and having this functionality as part of C# makes life much easier in some cases.

With these features been around for a while it sometimes amuses me how often I still see that engineers don’t use them. Benefits

So you want to be a software architect. Part 3

Vladimir software

In the second post I’ve shared my thoughts about the most important skill that architect needs if he wants to see the materialization of the design. Today we will go over the ability to identify the risk and commitment to the priorities.

Identify risk

As you design the application or subsystem, it’s important to see what the risky parts are. Before you hand the design over to the development team, you must be sure that risk and unknowns were investigated and resolved.

Let’s take a look at an example. In order to speed up the development of the new versions of the product, you need to transition from a monolithic architecture to a Service-Oriented Architecture (SOA). You know how to split your business functionality into multiple independent services. You addressed the issue of internal routing and load balancing. But what method should be used to communicate among the services? There are many approaches. SOAP will be an obvious choice. But this will require your team to learn new skills of building SOAP-based services and operation team to deploy & maintain them. Maybe it’s not a big deal, of course. But you consider to build internal SOA using AJAX/Json services because your team is already familiar with this approach building Web application and operation knows how to deploy and operate them. But what about the performance? It can be slower than other SOAP approaches. Unless you measure and compare performance of both approaches, and than agree with the team that the performance “penalties” are acceptable because of the advantages that the team will utilize the existing skills, you will be risking the transition to the new architecture and delivering it on-time and with high quality. In this case you should measure first and build then, not build first and then test and measure:

  1. Identify the risk
  2. Investigate it
  3. Find alternatives
  4. Agree with the team about the approach
  5. Do it

Risk may present itself in many areas – team skills, complexity or stability of the 3rd party components that you plan to use, performance of the algorithms, scalability path and so on. By “realizing’ the new architecture in your head, identifying the risk as part of the design and addressing it in the earlier stages, you will be able to deliver the design that will be successfully implemented by the team. 

Commitment to priorities

Each new project or release has priorities, Usually they are:

  • time
  • quality
  • scope
  • resources

Priorities are defined by the management and driven by the business needs. And if they are not defined, then you should ask for them. Very often the resources are fixed – your team has the same number of people. Out of the other 3 you can balance only 2 and have to sacrifice one of them. For example: with limited time, it’s not possible to deliver all the features with high quality. And it’s easy to forget about the priority #1, if let’s say it’s time, by creating a perfect design or design an interesting feature (both take time). You have to deliver according to these priorities, even if you disagree with them sometimes. 

(To be continued)

So you want to be a software architect. Part 2. Important.

Vladimir software

Last time I’ve started talking about the skills that I think are required for a software architect. The plan was to go over other skills, leaving the one I consider the most important (as of today) for the last. But I think that this should go first.

Influence

Software architect must be able to influence other people, if he wants to see his architecture properly implemented in the final product. The last condition (proper realization of the architecture in the product) is important point whether architect should influence people or not. You see – if architect is only interested in just a set of UML diagrams that she passes to the development team and doesn’t care about the final product, than she should not worry about the influence. Because these diagrams are her final product. If this is how you see your role, than you can skip this post.

Why influence is important? In most companies developers and QA do not report to architect, they report to development/QA managers. And these managers do not report to architect. So, if manager decides to change the way that product should be built than it may be way different than what you have envisioned. And there is not too much that you can do about it. Because even a junior developer will do what he is told by his manager and not by you. This is why the only way for you to see your design in the final product is to “convince” a manager how it should be built. Otherwise you can be very disappointed (if you really care about the final product):

Your vision Actual product
84A613BA-AC7B-4EC0-8ACB-CC426098D71A 9F3C3243-EA51-4DA2-93D9-865CA009C13C

Thus, the most important skill for a software architect has nothing to do with technologies, but rather with their “political” ability to achieve their goal. You have to be a “politician” in order to see the final product the way you’ve envisioned it. And sadly enough it’s even more important than your technical skills to design it. Of course, it depends on your company, but keep this in mind.

(To be continued)

So you want to be a software architect

Vladimir software

There are many opinion about what software architect does, what skills are required and whether we need them or not. I’ve been in this role for a long time (20+ years), even before I’ve “acquired” the “official” title. I was lucky because a year after the graduation I’ve build the whole application – gathered requirements, architected, designed, coded, tested, operated the application as well as provided customer support. Since then most of my life I was building the complete application rather than working on its pieces here and there. And that’s how I’ve learned it at the first place.

First, let me be clear – I think that architect is required and important part of any software team or “part” of you if you are a MicroISV. And it doesn’t matter whether this person is called architect or engineer or he-knows-how-to or somehow else. There should be someone who knows how to build the whole building, not just superbly paint a single room.

So, some of my thoughts about the skills that good architect needs as well as pros and cons of this role.

Skills

Know how to do it yourself

I think that if you cannot sit down and write the whole application that you are designing yourself, there is a very slim chance that it will be designed properly, because when you design/architect an application, you actually write it in your head.

E2E knowledge

Architect has to know it end to end. From understanding how the business works to the work of customer support and everything in between (requirements, design, development, QA, operation). Without this knowledge it’s difficult to build application that will add value to its users, will be delivered on time, have a reasonable cost and will make the team happy. And this leads to the next required skill.

Side note: why did I put team happiness in this list? Long story short: in a long run happy team makes great product, unhappy team – crap.

Compromise

Meeting these requirements (value for the customers, time to market, cost and team happiness) during the whole lifecycle of the product is difficult.

  • Should the product quality be sacrificed to ship it on time? 
  • Should additional development effort be allocated to make a tool for operation team which will make operation easy? 
  • Should some components be build in house or purchased?
  • …..

Architect faces these questions every day. And the quality of the final product depends on his ability to find optimal solution – find a win-win path.

Keep it all in your head

You should be able to store the whole application in your head. You may not remember all the details, but overall layers, tiers, components, data and execution flow as well as storage design is a must. Application is a composition of multiple pieces (layers) and without seeing the “whole picture” it is difficult to build well balanced application. E.g.: design of particular database tables may be perfect from a DBA perspective, but will require a lot of unnecessary and badly performant code in data access and business layer.

Review code fast

High software quality is not only guaranties the happiness of your customers (functional quality), but also how fast you can deliver new features (structural quality). If your application has 0 bugs, but code is ugly and architecture resembles Winchester House, then there is no way you can add functionality fast. But how can you ensure high structural quality? Yes you can have all these static code analysis toolsin place and they help. Too some extend. But the only way to guaranty a high structural quality of your application is to review all the code. “I don’t have time to review all the code that is checked in?“. Yes you do. It takes 30 minutes to review regular check-ins of a 10 developers team. If you know how to do it right and if you teach your team to write good code (I’ll share my experience later). 

And don’t fool yourself – sooner or later you will review “that” code, because when it breaks one day badly or prevents your from adding a new functionality without total rework, you will have to review & clean it up. So, better do it daily when it’s easier to spot a small problem and fix it early.

BTW, if you know a better way to ensure high structural quality, please let me know.

(To be continued)