You are viewing paulmck

Previous Entry | Next Entry

Parallel Programming: Announcement

SequentialCaveman
As some of you know, I have been working on a book on parallel programming. My thought had been to complete it, then announce it. But I finally realized that it really never will be complete, at least not as long as people keep coming up with new parallel-programming ideas, which I most definitely hope will continue for a very long time.

So, here it is!

Comments

(Anonymous)
Jan. 5th, 2011 10:28 pm (UTC)
Re: What, specifically, is the focus of the book?
In terms of interconnects, I'm thinking busses (in the case of multicores and SMP) and ethernet (for budget message-passing clusters, MOSIX and Kerrighed users and remote calls - including RPC, CORBA and Linux' TIPC). I wish that I could include fast serial links, but the Transputer died in the 1990s and Intel's iWarp died soon after. Those are more Visions of the Past, which I consider a shame.

In other words, stuff that makes a system almost transparently scalable. Explicit high-level stuff like MPI-2 and Bulk Synchronous Processing get ugly, there's masses of tedious detail, and frankly most of the important stuff (avoiding deadlocks, avoiding simultaneous writes, etc) is stuff you already cover. Some aspects of parallelism are universal.

In the case of ethernet, for example, there's the question of how to pretend to have shared memory. (Distributed Shared Memory schemes tend to be rare and are often badly written.) There's also the question of whether you can use process migration systems like MOSIX or Kerrighed to do MIMD-style parallel processing more effectively than you could on a multi-core or SMP computer.

Also on Ethernet, you can very easily pass around a lot of data to a lot of machines simultaneously, using Scalable Reliable Multicast. Which, incidentally, MPI does not use. MPI rotates round the list of destinations in a collective call and sends messages on a sequential basis. Implementations of SRM and NORM (NACK-Oriented Reliable Multicast) are widely available for most platforms, Windows and Linux included, so it's vendor-independent. They are not, however, significantly used in the industry, although one would think that MISD-style parallelism would find the mechanism to be extremely valuable.

For many busses, the time it takes to switch a lane from one direction or target to another is absurdly high. The cost of working at such fine detail is also absurdly high, since a lot of parallel processing languages (UPC, Erlang, etc) are too high a level to let you do much tuning. The question is how to make the best use of time to get the best use out of the system. And that's a hard problem.

"What area am I expert in that a lot of people need to know or will soon need to know?" If I knew that, I'd be rich. :) I'm expert in plenty of areas, I frequently find use for that expertise, but anticipating which areas would be useful for others has always been a tough one for me.