How to stand out in the supercomputing crowd

If you deploy the same high-performance computing systems as everyone else, how can you possibly gain an edge over your rivals, asks supercomputing expert Andrew Jones.

For many organisations, HPC is a strategic activity. If you can optimise how HPC is used in your organisation then your computation — and thus your design, testing and logistics — will be done faster, or better, or more cost effectively. That optimisation creates a business edge over the competition.

So the value of HPC is allied to a need to differentiate, to step outside the norm enough to get your business ahead of the competition.

Contradiction
However, that fundamental aspect of HPC seems to conflict with the move towards commodity supercomputing products by users and vendors.

HPC is usually regarded as an IT activity. Of course, there are some situations where I would argue it is better to view HPC as a research facility rather than an IT service, even though it is based on computers.

Nevertheless, the IT service model is the most common way of viewing HPC, making it subject to the unwritten but widely observed rule of IT procurement: "If a dominant solution exists, then that becomes a de facto prime candidate for your procurement, unless you have specific needs that say otherwise".

Or, to put it another way, if your procurement process produces a different result from the one everyone else in your area has come up with, then at least ask: "Why?".

There is good and bad in this approach. For example, it sensibly encourages you towards the solution that is likely to be the most proven in terms of deployment, support and availability of skills. It gives you a greater chance of being compatible with collaborators, customers and suppliers. It also forces you to evaluate why you are different — what do you know or do that everyone else doesn't?

However, it can hinder innovation, placing perhaps too much emphasis on caution over the potential benefits of a novel solution. Plus it reduces diversity among customers and suppliers, which potentially reduces competition, future innovation and possibly raises prices.

Given HPC's focus on differentiating to create a competitive edge, the drive towards doing the same as everyone else appears unhelpful.

Choosing your ground
The apparent contradiction is fairly easily resolved. It comes down to choosing your ground for differentiation. The commodity solution can be employed in most places in your HPC operations, with the differentiation only being applied where it will create the most value.

The details of this differentiation will depend on the nature of your business, the role of HPC within it, and the HPC strategy you develop to maximise the benefit to the business.

For example, one of the most-discussed trends in the HPC community over recent years is the move towards commodity clusters — mass-market servers strung together with some kind of high-performance network — rather than dedicated supercomputers.

For HPC manufacturers, differentiation will depend on how they select their target market segment. They use the obvious benefits of commodity components wherever possible, but, for example, use a proprietary interconnect to deliver much tighter communications between processors for very large parallel-processing workloads; or include an optimised and pre-configured software stack; or focus on a more reliable and capable supply, install and support service.

So a commodity cluster might represent the most economical way of providing the raw compute resource with the largest set of skilled staff and support options — the same as your competitors.

You can then differentiate through the scale of your operation, by optimising other parts of the HPC service; for example, by choosing scalable and optimised applications, a deeper level of support to users, and integration with business-wide processes — or by reducing the operational cost of the datacentre through power or cooling design.

Getting it right
The essence of using HPC to gain a competitive edge lies, like any other business-critical activity, in understanding the how and why of your operations.

Where can economy be sought through commonality or commodity elements? Where is it most beneficial to invest effort and expertise? What part of the competitive edge do you make impossible by failing to invest? Getting this balance right will give you that competitive edge — in both performance and economy.

Two final notes of caution. First, HPC is one of the fastest-innovating technology areas around, so the balance needs to be reviewed regularly. Second, and most important, make sure you consider the whole of HPC — not just hardware, but the equally important people, software and interfacing business processes.

As vice president of HPC at the Numerical Algorithms Group, Andrew Jones leads the company's HPC services and consulting business, providing expertise in parallel, scalable and robust software development. Jones is well known in the supercomputing community. He is a former head of HPC at the University of Manchester and has more than 10 years' experience in HPC as an end user.