ZDNet UK / News and Analysis / Infrastructure / Servers

How direct memory access speeds Ethernet

By , ZDNet.co.uk, 1 May, 2003 16:37

Daily Newsletters

Sign up to ZDNet UK's daily newsletter.

Topics

Memory, direct, Access, rdma, Latency

ANALYSIS
As buses, memory and networks get faster, new bottlenecks are being found in the search for higher data throughput. One of the basic tenets of high-speed data transfer is 'thou shalt copy data only once', a commandment frequently and deeply broken in today's complex mesh of protocols and processor-mediated transfers. When a packet comes in from the network it is copied into a temporary memory buffer, then the main system processor breaks it down and moves different parts of the packet to different places in memory for analysis and action according to the various TCP, IP and higher level protocols. This often involves moving data between the kernel and main storage, sometimes several times. While the overheads associated with this are acceptable on single-user systems and light application loads, the amount of processor time taken up becomes significant with servers coping with multiple, data-intensive requests -- most typically in networked storage. One answer is to put as much of the protocol processing as possible into dedicated hardware on the network adaptor, and to give it the ability to read and write directly into the host computer's main memory, bypassing the kernel and CPU data reads and writes altogether. These are the basic ideas behind Remote Direct Memory Access, RDMA, a new standard recently completed by the RDMA Consortium -- the usual suspects, Cisco, IBM, Microsoft, Dell, HP and so on -- and a development of work done by the IntelliBand interconnect designers. As far as possible, RDMA lets applications on one computer read and write memory on another by issuing commands directly to the network interface, bypassing the kernel. Any movement of data or commands into or out of the kernel involves context switches and other time-consuming operations, so by keeping the data transfers and link management entirely in user memory space these overheads are avoided. The remote computer that's servicing the requests just needs to tell the network card where its buffers are -- everything else is handled by the hardware. There are security implications in all this -- by bypassing the kernel, you also bypass the normal protection mechanisms that stop applications from overwriting memory they don't control. RDMA introduces the idea of key values -- before an application can access remote memory, it must have the right key value to send along with the request. That key value is initially generated when the remote computer asks its network controller to register a memory buffer: the remote then informs the other computer about what memory is available and what the keys are before any data transfer can take place. Other potential problems include out-of-order packets. IP does not guarantee that packets arrive at the receiver in the same sequence as they left the sender, and higher level protocols are designed to spot and correct that. However, when the delayed -- or lost -- packet contains the information about where the subsequent data is to be placed, the network card doesn't know what to do with that data and has to drop it, asking for a retransmission later. This is known as the framing problem, and has serious bandwidth and latency implications. RDMA copes with this through a technique called Marker-based Protocol data unit Alignment, or MPA. Here, each segment of TCP data has the placement information immediately after the TCP header, so each chunk will arrive with its address intact. For large segments, the marker is inserted in the byte stream at every 512bytes, thus ensuring that it's present in a known place even when the TCP header isn't accessible. Additional logic detects when an intermediate device, such as a firewall or router, has reassembled and re-segmented the byte stream, which has the potential to move the marker relative to the segment start. One of the first uses of RDMA will be in TCP/IP Offload Engines, or TOEs -- chips that are designed to remove as much of the gruntwork in network communication as possible from central processors. Although some companies have already produced TOEs, the existence of a standard is expected to rapidly increase their acceptance. In particular, RDMA is agnostic about higher level protocols and the exact nature of the network fabric -- as long as it's Ethernet -- and so is expected to eventually percolate through to everywhere that high speed networking is being used. With near universal support from its consortium members -- Microsoft is expected to support RDMA in all versions of Windows -- it's only a matter of time before it becomes a fully mainstream product.
More enterprise IT news in ZDNet UK's Tech Update Channel.

For a weekly round-up of the enterprise IT news, sign up for the Tech Update newsletter. Let the editors know what you think in the Mailroom.

Related stories

SeaMicro swaps out Atoms for Xeons in microserver

Post your comment

In order to post a comment you need to be registered and logged in.

You can also log in with Facebook. Log in or create your ZDNet UK account below

  • Login

Will not be displayed with your comment

By signing up for this service, you indicate that you agree to our Terms and Conditions and have read and understood our Privacy Policy. Questions about membership? Find the answers in the Community FAQ

Get ZDNet UK's daily newsletter

Enter your email address to sign up

ZDNet UK Live

Moley

@GrueMaster. I prefer horses for courses rather than one size fits all. I, and I suspect most other computer users, do not really wish to have...

22 minutes ago by Moley on A tale of two distros: Ubuntu and Linux Mint
greycynic

The product that scares me every time I have to use it is the Office 2007 version of Excel. The first bug that I found was applying the median...

25 minutes ago by greycynic on Ten flawed products that derail productivity
GrueMaster

Nice review and very informative. One thing I'd like to add (in reply to whs001's 1st question), the main reason to have the same interface from...

2 hours ago by GrueMaster on A tale of two distros: Ubuntu and Linux Mint
Frederick Wrigley

I'be been using Mint 12 since the RC came out, and I am far more happy with the Cinnamon, the Mate, and, yes (with extensions), theGnome 3...

3 hours ago by Frederick Wrigley via Facebook on A tale of two distros: Ubuntu and Linux Mint
bdantas

Excellent article. One small correction, though--although a fresh installation of Linux Mint 12 will, indeed, provide the user with a version of...

3 hours ago by bdantas on A tale of two distros: Ubuntu and Linux Mint
Alan Ralph

In related news, the ISPs club together to get the members of the Home Affairs Select Committee (ya goofed on that part, ZDNet UK) copies of "The...

4 hours ago by Alan Ralph via Facebook on MPs urge ISPs to take down terrorist material
Alan Ralph

In related news, the ISPs club together to get the members of the Home Affairs Select Committee (ya goofed on that part, ZDNet UK) copies of "The...

4 hours ago by Alan Ralph via Facebook on MPs urge ISPs to take down terrorist material
Moley

For Gnome 2 die-hards, it is possible to add icons to the bottom panel (or top top panel, if you prefer) which provide the exact Gnome 2...

5 hours ago by Moley on A tale of two distros: Ubuntu and Linux Mint
ramwellian

Your comments would seem pretty naive and immature. Your 'solution' appears to be, "gee, let's all just give in to the hackers and give them...

5 hours ago by ramwellian on Cloud computing security: no more oxymoron?
BugStalker

"Interesting thought ... If you installed Win7 as a dual boot on a machine that previously only had Linux, and it wrecked your Linux installation,...

5 hours ago by BugStalker on Windows 7 Declares War on GRUB
whs001

This is an excellent summary of Ubuntu and Mint and the interface differences between them. Most such articles take a very partisan position for...

5 hours ago by whs001 on A tale of two distros: Ubuntu and Linux Mint
Moley

@ewallace. Not so clear. Anyone can obtain the text, for example from here http://www.ustr.gov/webfm_send/2379. I support ACTA so long as it and...

6 hours ago by Moley on ACTA: Facts, misconceptions and questions
45283

I think WinRT is fantastic. I just wish it was an option for people that didn't want to go through Microsoft's App Store with its attendant...

9 hours ago by 45283 on Why Windows 8 needs architectural hygiene for WOA
Burn-IT

Nine people? £30m? Who's back pocket is that lot going in? And IF they say it is for new buildings, what about all the ones the government has...

10 hours ago by Burn-IT on Police set to launch three £30m e-crime hubs
ewallace

Just to be clear, nobody knows what is in the text of ACTA, here is a photograph of the text of ACTA http://twitpic.com/8h9iju as submitted to the...

10 hours ago by ewallace on ACTA: Facts, misconceptions and questions
fgvrg56

Unfortunately main issue is that ASUS is refusing to accept that they make some mistake on this version of asus Transformer prime. 1 - GPS sensor...

11 hours ago by fgvrg56 on Asus Eee Pad Transformer Prime Wi-Fi & GPS problems?
Ben Woods

@Marcus A fair question. Just talked with Archos which said it was working on an announcement for next week....

12 hours ago by Ben Woods on Archos confirms G9 Ice Cream Sandwich update schedule
Marcus Karlsson

Any update on this, considering the claimed "first week of February"?

13 hours ago by Marcus Karlsson via Facebook on Archos confirms G9 Ice Cream Sandwich update schedule
apexwm

Bill Goodrich : Just as al_langevin pointed out, with Windows Server 2008 there is no Services for Macintosh anymore. It's gone, not available....

22 hours ago by apexwm on Windows Server 2008 drops the ball for Mac compatibility
txtrainguy

Replying to an old topic that I'm currently facing with my CEO (who is on a Mac). Our servers are primarily Windows Servers, office is about...

1 day ago by txtrainguy on Windows Server 2008 drops the ball for Mac compatibility

Latest in Servers

SeaMicro swaps out Atoms for Xeons in microserver

Featured white papers

How cloud adds strategy to IT outsourcing

SAVVIS

How cloud adds strategy to IT outsourcing

Thanks to cloud computing, IT outsourcing is back, but this time it can help both in cutting costs and helping the business... Read more

How enterprises can achieve cloud computing benefits

Citrix

How enterprises can achieve cloud computing benefits

Is your load balancer cloud-ready? A key to the cloud's success is having an IT infrastructure that can deliver a robust set of cloud-oriented capabilities and can support all cloud computing use... Read more

I/O holds up the traffic in virtual systems

Intel

I/O holds up the traffic in virtual systems

We can now host 512 virtual machines on a single physical server. That's a lot of virtual machines trying to squeeze... Read more

Inside ZDNet UK

A tale of two distros: Ubuntu and Linux Mint

A tale of two distros: Ubuntu and Linux Mint

2012 Predictions: VCE FastPath, VI SAN Probe & Hadoop

2012 Predictions: VCE FastPath, VI SAN Probe & Hadoop

BlackBerry Bold 9790

BlackBerry Bold 9790

Case study: How cloud enables growth

Case study: How cloud enables growth

Ten factors that make Ubuntu 11.10 a hit

Ten factors that make Ubuntu 11.10 a hit

Toshiba Portégé Z830-10P Review

Toshiba Portégé Z830-10P Review

BT's archive: Pictures from the vault

BT's archive: Pictures from the vault