Novator Solutions / Blog / Investing in a Test System Framework
 
 
 

 

 

Investing in a Test System Framework

 

Investing in a Test System Framework

 

By Henrik Ulfhielm 

In my 30-year career in the test system business in both product and consulting companies, it’s always struck me as strange how inefficient and lacking in standardisation many companies are with test system development and maintenance.

I have seen so many test systems within the same company that are built in different programming languages or without any consistency with each other, even though they are written in the same language and are in the same department. The reuse of code and documentation in these cases has been almost non-existent.

When asked why they don’t have a common platform or framework for testing, the answer is often along the line of “Person X prefers to write it in this programming language”, “Person Y only know this programming language”, “we don’t have the time to optimise our organisation”, or “I used this programming environment in my previous work.”

It can seem like a big investment upfront to get everyone using the same language or framework, and with the pressure of the next deadline looming, it can feel like you don’t have the time to change the whole infrastructure to get gains later. It is a “Catch 22” moment. You need to improve, but you don’t have the time.

Lately, I have seen a growing number of companies with a clear strategy for test and test systems with a consistent, standardised framework and way of working. These companies spend less time and money developing and maintaining their test systems.

Not having a test and test system development strategy is often expensive and has some risks. Here are some common scenarios that I’ve seen lead to high risks and huge costs in the long run:

 

 

    • Scenario 1 – “Lone wolf”

      One or several test systems are developed and supported by only one person in a department or company.
      I have been this person, and while it might make you feel important and gives you a lot of power (especially regarding salary negation), it is obviously unsustainable from the company’s perspective.
      If the person leaves the company, it will be very costly if the test systems need service or upgrades. It will be even more expensive if you don’t find the code or if it is not documented. If you are lucky, the lone wolf leaves your company and starts at a consulting company so you can hire them.Code quality also tends to be lower in these situations. Individual developer teams create less documentation, tend to focus less on architectural decisions for code maintainability and readability, and lack the duplicity in perspective that multi-person teams have.Possible consequences: Stop in production, high costs, and lower quality.

.

 

    • Scenario 2, “Anarchy – Use whatever programming language you want as long as you are comfortable.”

      Test systems are written in different programming languages. Each employee has knowledge of and prefers different programming languages, so, naturally, the test systems will be written in different languages. This is a little like Scenario 1, but if one person leaves, the consequences will not be as bad, but is more likely to happen. Other risks and costs are that you will get no reuse of code and documentation whatsoever between different test systems. If you combine this with no documentation and no version control, you can get an even worse nightmare regarding the service and support of the test systems.Possible consequences: Stop in production, very high costs, and lower quality.

.

  •  
    • Scenario 3 – “We use LabVIEW; other than that, we have no rules.”

      Test systems are written in the same programming languages with little or no reuse of code.
      This is very common, and I would say that this organisation is in better shape than scenario 2 because at least all the test system developers use the same programming language. The testing is still not cost-effective, though, and you have inconsistent quality control, but the effort to train and convince the organisation to standardise around a programming environment that is well-known is not that hard. To optimise costs and quality, you need to standardise on a test framework and start to reuse and version control your code.Possible consequences: High costs and bad quality.

.

         If you are a manager responsible for tests and test systems, it is your duty to make a strategy for tests with well-documented
         rules leading into a test platform or framework.

 

 

The challenge

If you are responsible for test and test systems and you find yourself in one of the scenarios above, the time to act is now.

The goal is to have cost-effective tests that help your company deliver quality products in time.

This applies to either validation tests for R&D or production tests.

The challenge is to sell the idea to your organisation – to both management and your team of engineers. The latter group might be more difficult since they,most likely, have preferences about the programming language, style, and architecture, and they could also be afraid to try and learn something new. Management will be more easily convinced using arguments like cost-effectiveness, better quality, and lower risk. For the management, prove what you say with calculations and examples from your own or other companies.

 

 

Convincing your organization

First, write a high-level requirement specification for your future tests and test system framework. This specification just needs to be detailed enough to make a plan and a presentation. When you have sold the test system framework implementation to your organisation, you can make a more detailed specification.

 

  This should include the following:

    • How tests should be done in your department or test procedures
    • A basic software and hardware architecture
    • Rules for reuse of code in the framework
    • What software and hardware should be used in the framework
    • Can any or part of previous test systems be reused?

 

Second, you need to make a plan. Make a document with your recommendation on implementing the test system framework based on your specifications.

It should at least include:

  • Time plan, for example a Gantt chart
  • Cost
  • Which resources are included in the project

 

Third, sell the plan to your organisation. Prepare a presentation that includes the following:

  • An overview of what the test platform is
  • Arguments why your organisation needs a test platform:
    • Cost-effectiveness – show calculations
    • Better quality – show why you get better quality
    • Lower risk – give examples from your current situation
  • Tell how you suggest that the test framework is implemented:
    • Your time plan
    • Cost
    • Which resources are included in the project
    • What can be reused from current test systems
  • An overview of the expected result mirroring what overview with a little bit more detail
  • Sum up with questions

Start by selling the idea to your closes manager and let them help you to sell it higher up in the organisation. In parallel, start selling the idea to your colleagues in your department and, if they are interested, involve them in the process.

 

 

Implementation

You won. What to do now? The short answer is to make a more detailed plan and follow it.
Follow our blog for a future post on how to implement your test plan.

 

Share on Social Media