You are viewing paulmck

Previous Entry | Next Entry

inside
If the demand for parallel programmers has outstripped supply, then why not simply train up more parallel programmers? After all, there are a lot of parallel programs out there, and the developers who created them did not spring fully formed from the void.

Unfortunately, I cannot put myself forward as an expert in parallel programming courseware, because I did not learn parallel programming from a college class or a training course. Instead, I gained most of my parallel programming expertise directly from the hardware, with the aid of a logic analyzer or two. I would not have it any other way, as each new generation of hardware has something new to teach, and I have grown used to learning directly from it. Nevertheless, the fact that I have not learned much from parallel courseware makes it difficult for me to judge courseware quality.

On the other hand, I have had the good fortune to be a member of two communities distinguished by their success in quickly bringing new members up to speed on the art and science of writing parallel code. The first such community was Sequent's DYNIX/ptx engineering staff, and the second was (and is) the Linux kernel community. My experience with these two communities leads me to believe that the following conditions are required to quickly turn competent programmers into competent parallel programmers:

  1. Readily available parallel hardware.
  2. Large body of high-quality parallel code available for review and tinkering.
  3. Easy access to parallel-programming experts, preferably with a variety of viewpoints.
  4. Immediate “frank and open” expert feedback on new code.

Over the past 20 years, the situation for the first two items has improved beyond all recognition: cheap parallel hardware is readily available and there a wealth of open-source parallel software of all sorts is just waiting to be downloaded. The number of parallel-proficient developers has greatly increased, but there is still no shortage of projects lacking parallel expertise, and I have yet to see hordes of parallel-programming experts sitting around waiting to receive newbie parallel programs in need of review. That said, the shift from zero of four to two of four is a great improvement. Furthermore, one can argue that these first two conditions are the most important of the four, as they promote spontaneous generation of parallel programmers through self-education.

Of course, these communities both use what can be thought of as an apprenticeship approach, where new developers learn on the job. This can be quite effective, but it is not clear that it can produce the needed number of parallel developers. Perhaps formal education will start filling the gap.

In the meantime, what is the one most important parallel lesson?

All of this existing and expected improvement in parallel-programming expertise is encouraging, but what do you for a team of uniprocessor programmers who need to parallelize a large complex application?

There are now a number of parallel-programming classes and textbooks, so formal continuing education is becoming a feasible option, although it is too early to judge its effectiveness. As noted above, there are also a number of parallel open-source projects in full swing, so having some of your team members join one of these communities might provide effective education. Hiring a parallel-programming expert is also an option, assuming you pay close attention to team dynamics. Having parallel-programming expertise is not enough — successful parallelization also requires deep knowledge of the application being parallelized.

I will leave you with the following caution:

Raw intelligence is completely insufficient to guarantee success in parallel programming. In fact, raw intelligence in absence of parallel design and coding know-how is a negative. The more intelligent you are, the deeper the hole will be before you realize that you need to stop digging. It all comes back to the need for additional planning when parallel programming. The more intelligent you are, the more likely you are to delude yourself into believing that your seat-of-the-pants “planning” is actually working, and the worse trouble you, your team, and your project will get into. There are a very few exceptions to this rule, one example being Bob Beck, the guy who defined Sequent's approach to parallelism in the early 1980s. Unfortunately for all of us, the number of people with this level of talent, ability, and experience is quite small.

Comments

rasmatan
Dec. 13th, 2009 03:26 am (UTC)
My operating systems professor sincerely believes that sometime within the next ten to twenty years, it might be possible for normal people to buy computers for under five thousand dollars which can execute more than one instruction in parallel. I checked the calendar on my multicore laptop, and it said 2009, not 1985.

I don't think formal education is going to fill the gap.
paulmck
Dec. 13th, 2009 06:30 am (UTC)
If it makes you feel any better, some of my colleagues report that their 1980s operating-systems textbooks claimed that no more than two (or at the very most four) CPUs could be supported by an operating system kernel. Then they went to their computer-architecture class, which included a lab — where they ran programs on a 16-CPU Sequent machine. So professorial cluelessness is by no means a new phenomenon, nor is cluelessness in any other walk of life.

But I know many professors who are very aware of the multi-core revolution. And some of them would not be above saying something similar to what your professor said, just to get a rise out of the class. So, did any of his or her students challenge this extraordinary statement? If not, why not?
rasmatan
Dec. 14th, 2009 01:46 am (UTC)
I repeatedly got in trouble for contradicting almost everything he said. He marked me wrong on questions he marked right for others, he axed my participation grade, and chipped off points on my final project when he could find no flaws for "adding more features than were specified." Fortunately it was still mathematically impossible to give me anything but an A because brother, I like operating systems. When they saw my participation grade go down, none of the other students ever said anything.

I heard from the juniors that this semester, he told his new class not to be like "some people" who "dig out manuals and ask about things no-one's ever heard of." Our CS program is small (35 students, 4 professors) and everyone knows who that "some person" is 8)
paulmck
Dec. 14th, 2009 02:16 am (UTC)
Well, good for you! At least one set of students saw an alternative view, and doubly good for you for pulling an A despite your professor's displeasure. One can only hope that he is close to retirement age.

Some students have used other techniques, but they relied on a level of anonymity that might not prevail in a 35-student program.

And I really have met a number of operating-systems professors who are quite aware of recent events. I am sorry you got stuck with one who was not, but I am glad that you survived the experience, seemingly in quite good shape.
rasmatan
Dec. 14th, 2009 09:37 pm (UTC)
Well it's certainly inspired me to always be on top of things for the rest of my life. No upstart undergrad whippersnapper is gonna catch ME not knowing about pandimensional unitomitron Grindelwald clusters now being supported by Windows 17.5.
paulmck
Dec. 15th, 2009 01:30 am (UTC)
Very good!!! ;-) ;-) ;-)

The other thing to keep in mind is the old adage: “if you become a teacher, by your students you will be taught.” Failing to keep up is bad enough, but one of the most effective ways to continue failing to keep up is to refuse to learn from your students.