Has Agile gone Astray? Agile meets String Theory

 

Physics gone astray

  Theoretical Physics has gone astray. Ask any practicing physicist today and they will lament about the burgeoning number of contending versions of string theory, all promising grand unification.  Even as one version of the theory is being experimentally ‘validated’, another one crops up, promising to be a better, shinier version of the previous one.  Except, the Universe had other plans. It continues to expand, as was recently, experimentally proven. This expansion implied a positive cosmological constant, which was something that string theory had failed to predict. Yikes….30 plus years of grinding out ungodly math, and we are virtually back to the drawing board. 

Ironically, string theory set out to UNIFY all the fundamental forces of physics. One grand theory to try and explain everything in nature. Except, what emerged from string theory was a total of 10 ^ 500 possible theories!  Double Yikes!

What does this have to do with Agile?

Today’s Agile landscape suffers from the same problems as the String Theory landscape  – too many flavors, and not enough experimental proof that any of those flavors actually work.  With a new Agile flavor appearing every year, and promising to be the ultimate dev methodology, one is led to ask the question – Has Agile gone Astray? 

This post will dissect some of the leading Agile methodologies (notably Scrum and SAFe). The author makes the following claims (and narrates real-world, coding snafus to support these contentions):

1. The aspects of Agile that work well, have existed long before the advent of Agile XP/Scrum.  Some of these practices made it into Agile (especially XP). It is pointed out that any improvements that Agile seems to deliver, have to do with these common sense practices.
2. Certain Scrum practices, not only fail to improve the quality of release code,  they actively interfere with producing rock solid  code.
3. The hidden costs of doing Scrum, for small or large teams, are often ignored.  Scrum costs small dev teams upwards of 100k / year, without delivering any tangible benefits.
4. Anyone, and I mean anyone, can become scrum certified. Whether they have ever seen a line of code or not. The Scrum Alliance has become an organization that hands out certificates rather than ensures quality of their graduates. You can get a ZERO – that’s right – a ZERO on the certification exam – and still get your certificate. 

Common Sense Coding Practices

Constant Customer and Team Involvement, Testing your code before it gets to production (unit testing specifically) are all part of common sense coding practices that actually existed well before Agile (though Agile XP did a good job of enforcing these). 

  Code quality is not a result of any development methodology, but is more a function of common sense coding practices. These include practices such as code and design reviews (existed before Agile), unit testing (also existed prior to agile) and static code analysis.  Sprints, Daily Scrums, Coding Only to Current Requirements are all Agile tenets that actually degrade code quality. You read that right – these practices are not just unhelpful, they actually produce unstable, haphazardly released code. 

Agile, in it’s modern day incarnation (aka Scrum or SAFe), does little to ensure the success of a project. If success alludes to code quality, fewer defects, and mapping to actual customer needs, then Agile Scrum , Agile SAFe and all related embodiments are actually headed in the opposite direction. 

So where and how exactly did Agile start going astray….?

Agile Misguided Tenet # 1 –  A daily status update (scrum) is required to make the team more productive…

Does the sales team in your company stand up every morning and go around informing each other about what they will be working on today? Does the management team do something akin? How about the H.R. team? Or the graphic design or creative writing team?

The answer, of course, is a resounding NO !

Not because these teams are lazy or unmotivated, but because :

1. These teams have learnt how to communicate and get results without having to constantly advertise their day to day work schedule. 
2. Selling, graphics design, Management are all art forms. To provide a regimented approach to an art form only stifles it. 

Well, guess what?  Coding is as much an art form as any other endeavor, believe it or not.

Why do coders (or coding managers) feel that a regimented approach will produce better code? How did this ‘daily stand up’ practice start to begin with? 

  I believe that part of the motivation was the high failure rate of waterfall based I.T. projects, and the belief that somehow fixing developer practices would result in a higher success rate.  Not only did Agile NOT make projects more successful (see references at the end of this post), this article will argue that recent Agile practices, accomplishes the exact opposite. The success rate of Agile projects has not changed drastically from the waterfall success rate – but I do believe that most Agile teams today churn out unfinished, half-baked code.

Agile Misguided Tenet #2 – Code at the end of each sprint needs to be production ready

Solid code needs at least 3 iterations. For me, this means, write working code, write the test around it – check it in and go home (Iteration 1). Come back the next day (having slept on it) , rework it,  improve it’s readability somewhat  – and check it in (Iteration 2). Now, a week later, having forgotten all about this code that I wrote a week earlier, I am hit by inspiration (why did I use a switch statement when I could have used the strategy pattern instead??).  Having been struck by lightning, I race frantically in the middle of the night, log into my workstation – and completely rework the entire method that I wrote a week ago.

Of course, the original unit test breaks, and needs to be rewritten as well.  All this happens a day before the end of the sprint (which, remember, is meant to be production-ready code).

Now, I actually go to bed peacefully realizing that my code is in the final state that God intended it to be.  No more spaghetti, no more outdated, kludgy switch statements. All that remains is  an elegant OO pattern that could be easily extended to accommodate more switch cases. Both me and my code are resting peacefully.

The next morning, the scrum leader isn’t at all happy.   Why did I make these massive code changes the day before code freeze?  After all, my previous code was working just as well, as were the unit tests around it. Why did I feel the need to change everything the day before the code lock?

So, I  have to spend time convincing the Scrum Master that all code is really a living Work-in-Progress – like an unfinished symphony, searching for it’s missing notes.  Code just wants to reach completion, otherwise it feels discordant and experiences depression and internal turmoil . Needless to say, the SM isn’t buying any of this.

To get code to reach it’s final resting place, requires at least 3 iterations (ask ANYONE who writes code for a living).

And a 1-week or a 2-week sprint is usually not enough time to allow 3 iterations of the same code for the same stories.

Which is why, I claim, that most ‘Agile’ code is actually haphazardly written and half baked, in an unrealistic sprint to an imaginary finish line.

Agile Misguided Tenet #3 – Only Code to the current set of requirements (a.k.a – shortsightedness )

The case of the long running SOAP service

One fine day, I wrote a SOAP client that performed a background check on a loan applicant (against several external java web services). The external service took forever to return, at times, and so, the requirements asked that the client retry the process, in case it failed the first time.  Fair enough. In my mind, this could be accomplished easily with TWO configurable switches.

1. The first switch is obvious – how long (seconds) does the SOAP client wait before retrying?  No disagreement from anyone here.
2. The second one, that some of my team members were not fully on-board with, is the NUMBER of times you retry.  The original requirements only asked for a SINGLE retry.  And if that single retry failed, we just gave up and moved on to the next applicant’s background check.

I decided that this second number needed to be made configurable as well (with the default set to a SINGLE retry, so we were compliant with the original requirements).  And I coded that into the final version of the code.

As it transpired, a couple of days after we released our code, the business group threw a fit as to why so many credit applications were ‘failing’ and why were we not retrying more often?  Our team’s muted response that ‘we were only asked to retry once and then move on…..’ fell on deaf ears.  After all, this was seriously hurting business. And we did not have the time to go through a code fix and an entire release cycle!

Needless to say, I came out looking like a bit of a hero when I informed them that we could easily fix this, WITHOUT deploying any new code!  (since, I had made the number of retries configurable as well).  Had I stuck to the letter of the requirements, Agile would have not approved putting in that second switch.

The case of the method that needed to be overridden

  I am writing a new class with a few core methods (e.g. a Custom Authentication Class). Some of the core functionality in those methods MIGHT need to be overridden, even when there were no requirements that said so. 

So, I would mark the core method as virtual, provide a default implementation in the virtual method and let the inheritors of the class decide if they wanted to use the default functionality or write their own.

Again, Agile would not necessarily approve of my approach here – that some requirement in the future might need to do something, so let me code it in here to begin with. Once again, Agile would be shortsighted.

  I am simply illustrating that we need to let COMMON SENSE, and not Agile, dictate the depth of software that gets written.  If you are coding an important feature (story) that you think might change (based on your common sense), you need to accommodate that potential change into your code right there and then, instead of waiting till the worst case scenario happens.

Accommodate it regardless of whether Agile demands it or not, because common sense is more important than Agile. You are not necessarily violating the original requirements (as illustrated in the examples above), but you are also ‘future’ proofing your code (something that traditional Agile frowns upon).

Agile, by focusing only on the literal aspect of written requirements, misses the forest for the trees.  It asks developers to Put common sense aside and code only what is currently asked. This abandons all hope of future proofing code’ . And believe me, there continue to be drastic consequences of this short-sighted vision of Agile. 

The case of estimating a coding task and estimating OTHER people’s coding tasks

Scrum notoriously relies on estimates for tasks that developers are assigned. There is always a hidden level of complexity in any task, as any developer is aware. Sometimes, a task that looked like a 30 minute thing, becomes a 3 day grinder. And sometimes, the exact opposite happens.

Scrum forces you to come up with an estimate for something that, inherently, defies estimation. To make matters worse, Scrum expects you to anticipate how long OTHER people’s tasks are going to take! You don’t even know what you are going to wear to the party, but you are expected to pick out the right attire for the entire team!  

Agile Misguided Tenet # 4 – Scalable Agile ( SAFe ) can save your enterprise   

What happens when you take poor practices and scale them, horizontally and vertically, to encompass your enterprise?  That’s a rhetorical question…Seriously, SAFe (Scalable Agile Framework) promises to do just that. And it is (was) so addictive that I was truly mesmerized by it.

It promises the perfectly fine tuned enterprise, much like a humming Rolls Royce. Architectural Runways, Value Streams, Epics….SAFe sounded more like poetry, than software development.  The terminology used by SAFe was (is) beautiful.

And then it hit me. SAFe was still Agile! It was still the same bad candy, wrapped in newer, shinier wrapping paper.  As I delved a little deeper into each of the new SAFe constructs, I realized that they were not really new.

• SAFe Architectural Runways ? –  An architecture implementation in code that can be reused across projects. D-uh – we used to call this an application framework, and we used to write these for fun.
• SAFe Value Streams? We used to call them Product Lines (like MS Office, Exchange etc..)
• SAFe Epics ? We just called them ‘Releases’
• The list goes on….

There was nothing new here. And yet,  all the sexy talk around SAFe had really caught on.  Just like holding on to a live wire carrying current, I almost couldn’t let go.

Noteworthy Agile Tenets and Practices – Customer Involvement, Automated Unit Tests, Pair Programming

Not all that came out of the Agile revolution was terrible.  Increased customer interaction, Pair programming (an XP concept) and automated unit testing – are all powerful constructs that Agile popularized. And that is the key word here – popularized. It is not like Customer Interaction or Unit Testing were missing in the pre-Agile world.

Unit Tests, The Ultimate Code Detectives

I once had the unenviable task of merging thousands of lines of code (of a 3rd party UI library) with a newer version of the library. The problem was, that the old version (of the 3rd party library source code) had been heavily modified by our development team without any documentation. This meant, that when I merged (upgraded) to the new version of the library, I would have no clue whether the merge conflict was due to an actual feature that the upgraded library offered – or just some local change made by an in-house developer.  Trust me, it wasn’t fun.

Fortunately, I hit upon an idea that saved the day. What if I wrote unit tests around the older code base? That way, I would know what the expectation of each UI library component was expected to be, before I performed the merge. Then, after the upgrade (merge), even if I made a boo-boo, I could rerun my unit tests – and see if they still produced the same results as before. If they broke, I knew that chances were that a local developer change had occurred around that piece of code.  Right there, unit tests saved me about 3 months of tedious work.

The beauty of automated tests is that  they catch issues before they make it to production. They are like super detectives, catching the murderer even before the crime has been committed. By constantly scouring your innocuous looking code for breaking changes, they catch culprits before they make it to the release environment. 

Pair Programming, Two Pairs of Eyes are better than one

Pair programming is just another common sense concept – two pairs of eyes are better than one. As if that needs to be stated . But yes, Agile XP did promote this practice.

Customers, as part of the team

Agile did help popularize ‘regular customer interactions’ and, thus, deserves some of the credit for this common sense practice. Obviously, customer interaction was something that pre-Agile projects also tried their best to do.

However, once again, how much a customer actually interacts with a development team is never driven by Agile/Waterfall, but rather, by higher level intangibles.

These intangibles include budgetary constraints, existing customer relationship and of course, availability and level of interest of the customer herself.  Agile, by itself, will not bring your customer to the table. You have to bring the customer to the table. 

Unit Testing Existed Prior to Agile

Pair programming improves the quality of the software as does the writing of the unit tests. These practices improve the quality of the code internally simply by adopting common sense practices.  Writing code to test code has been around since the days of C programming language (those test programs were called ‘drivers’ – they ‘drove’ the test execution of the actual C program).

Still, Agile made unit testing popular and sexy, and deserves some credit.  However, as it turns out, unit testing is probably the most frequently ignored agile practice by Agile teams. Go figure…!

Agile (Scrum) Fails as Often as Non-Agile

What Agile evangelists forget is that the spirit of the (Agile) law always trumps the letter of the (Agile) law.  When we see a child stealing a loaf of bread, we do not arrest him and put him behind bars (even though the letter of the law demands that all robbers be punished).  We provide the child with some food and shelter and also, hopefully, some love so that they will not become a repeat offender.  All of civilized society is based on following the spirit, rather than the letter of the law. The letter is usually just a piece of paper; the spirit is what equates to the ‘common sense’ interpretation of the law. Common sense tells us that not all loaf stealers are to be treated the same. Common sense tells us that not all killers are equal (someone who kills in self-defense is usually considered a hero, not a murderer).

Scrum practices, by focusing on the letter of the law, tend to defy common sense on a regular basis.  Agile project take-offs, with their fancy architectural runways, crash just as often as other non-Agile projects. What gives?  

Coding is an extremely demanding and individual art form, one that rejects regimentation. Daily Scrums, 1 week sprints are all antithetical to producing good, artistic code.

Surveys that have compared waterfall projects to Agile projects have concluded that the failure rate for projects is about the same (yes, agile has a slight edge, but I believe that has to do with the few ‘common sense’ aspects discussed above).

I.T. Projects fail for much bigger reasons which include, but are not limited to, budget crunches, employee turnover, competitor  products etc. 

However, I believe Scrum goes a step further and encourages failure by trying to haphazardly sprint to an imaginary finish line, compromising on the quality of the code in the process.

The  Scrum Alliance, Even my plumber (no offense to plumbers) can get certified!

The Scrum Alliance is a non profit organization that charges $1.5k for handing you the title of CSM (Certified Scrum Master). I use the word ‘handing’, because you are given the certificate regardless of whether you pass or fail.  If you want to become a CST (Trainer), the cost if $5k.  I am not saying that the organization isn’t legit, but merely pointing out that there’s a lot of money being made by both, the alliance and the people who get certified by the alliance.

The first issue here is that  –  a Scrum Master who scored 100% on the exam is considered equivalent to someone who effectively scored ZERO. That is bothersome in itself. However, the broader issue that I see is that this certification can be obtained by someone who knows zilch about coding or software development. Heck, if my plumber decided to get certified, all he would have to do is pay $1500 and get his certificate!

This disconnect between actual coding and SDLC experience and getting Scrum certified should bother companies adopting Scrum.

The Hidden Costs of Adopting Scrum

Scrum has a lot of hidden costs for your team that are often ignored.  Let’s assume a 5-person team spends 12 hours in meetings/scrum activities (panning poker, backlog trimming…).   That’s 60 man-hours every sprint doing this stuff. If the average pay is $40 per hour (about $80,000 / year), that’s $2400 per sprint that is spent on ‘activities’.   For 50 work weeks, that is over $100 k / year!

Since Scrum gives the ‘impression’ of promoting efficiency, these hidden costs are often ignored in light of the bigger picture.

Revised Agile (Common Sense) Tenets

So, without further ado, here are my revised Agile tenets:

• Anuj.com’s  Revised Agile Tenet 1 – Scrap the Scrum, Replace it with a 1-1 developer interactions
  • Standing up daily and announcing your progress does nothing to make your code (or your productivity) any better than it was yesterday.  That daily stand up is simply the letter of the Agile law. If one truly thinks about the spirit of the Agile law, one would be writing and rewriting the same piece of code till it was ‘just right’. No need to stand up or to announce to anyone that you were on the third iteration of the same story.  If you need to understand what someone else is working on, simply go speak to him/her 1-1.
• Anuj.com’s Revised Agile Tenet 2 – Code to Common Sense, not to a piece of paper
  • One would also be accommodating future changes into their code, even if such were not mandated by the current set of requirements. Because, common sense, often demands that make your code ‘future proof’. 
• Anuj.com’s Revised Agile Tenet 3 – Replace Sprints with Slow Jogs (to get truly solid, unbreakable code)
  • One should not be dashing to a sprint finish line – just so one could claim they had ‘releasable’ software every two weeks. Instead, one needs to actually write ‘releasable’ software, and this means software that undergoes several rewrites for the same pieces of functionality (the 3 iteration rule).
• Anuj.com’s Revised Agile Tenet 4 – Encourage and incorporate common sense coding practices, whether they are part of Agile or not.
  • Encourage common sense coding practices such as running static code analysis, code complexity tools, automated unit tests, pair programming, customer interaction, 1-1 developer interactions and such.

Teams that I have worked closely with have produced killer software, without utilizing Agile in any fashion whatsoever.  Software written back in the late 90s still works flawlessly on both Unix and Windows, even though Agile was just an adjective applied to fast cats (like the cheetah), back then.

The code our team produced was rock solid; and at least part of it had to do with the fact that we were not rushing to a finish line every week.

Where can I learn more ?

If your team is just getting it’s feet wet with Agile, Anuj.com offers a simple, no-nonsense approach to building Agile Teams.

For a broader understanding of the emerging technology landscape – what works, what doesn’t, what’s hype and what’s not,  register for our popular,   The Ultimate Technology Seminar for Executives (TUTSE ™), Separating Hype from Reality. Seating is limited to 12 attendees. The half-day seminar covers trending topics including Cyber Security Risks and Mitigation Strategies, Data Analytics and BigData, Cloud Strategy for the Enterprise , Application Integration in 2017 and more…all presented through real-world success (or failure) stories.