Paul E. McKenney's Journal

Transactional Memory Everywhere: Hardware Transactional Lock Elision

2012-05-19T17:41:36Z

My earlier posting on hardware transactional memory (http://paulmck.livejournal.com/31853.html) brought an unusual amount of private email, especially on my discussion of the hazards of eliding empty critical sections (http://kernel.org/pub/linux/kernel/people/paulmck/Answers/TransactionalMemoryEverywhere/WhyEmptyLock.html). Several people made the interesting claim that any correct code involving empty lock-based critical sections that relies entirely on memory references is guaranteed either to: (1) operate correctly under HTM or (2) result in an abort, causing execution to fall back to the lock-based code, again operating correctly. I invested more time than I care to admit unsuccessfully attempting to generate a counter-example, so I am becoming more inclined to believe them, despite the lack of a formal proof.

There is a straightforward rationale for their claim, namely that in strongly atomic HTM implementations, the results of a given transaction are not visible until after the transaction completes. So if you can see that the transaction started, it has already finished, at which point a no-op will successfully wait for it. So any sequence of code that tests variables to determine when a given critical section has started, and then uses an empty lock-based critical section to wait for it to complete will operate correctly even when the empty critical section becomes essentially a no-op under hardware transactional lock elision.

Of course, this would not be true of a weakly atomic HTM implementation, but as far as I know all the currently proposed production HTM implementations are strongly atomic.

But there are other ways to communicate than relying on memory references, for example, you could instead rely on the passage of time. Please note that relying on the passage of time is very difficult to get right, especially on today's super-scalar hardware that relies so heavily on speculative execution. And if your code is running on top of a hypervisor, relying on timing is even more risky. And if I catch you trying to sneak a timing-based patch into the Linux kernel, I will almost certainly NAK it. However, older computer systems were much more amenable to use of timing, so it would be no surprise if large legacy software made use of such techniques. The following is an example of what a timing-based algorithm might look like.

Each worker thread corresponds to a data feed out to some receiver. Each worker thread records the current time (e.g., from clock_gettime()) to a per-thread timestamp after executing each work unit. Because this is a per-thread timestamp, a normal write suffices. Because this is a real-time application, it is a fatal error for a given thread to fail to update its per-thread timestamp for more than (say) 100 microseconds. Results are placed into a shared-memory buffer, which is drained by a separate set of threads that transmit the results out over the network. Locks are used sparingly to access and update global state, and, again given that this is a real-time application, locks are granted in FIFO order within priority level.

Worker threads can finish, and when they do they must disentangle themselves from the rest of the application. Worker threads also indicate their status in a thread-local variable my_status that is initialized to -1. However, worker threads do not exit: They instead go to a thread pool to accommodate later processing requirements.

There is also a control thread that monitors the worker threads. In particular, it maintains a histogram of thread statuses for threads that have exited. This same thread is responsible for assigning new threads or reassigning old ones, and runs at a priority no higher than that of the worker threads.

Worker threads do something like the following:

	int my_status = -1;  /* Thread-local variable. */

	while (continue_working()) {
		enqueue_any_new_work(); /* Might include locks. */
		wp = dequeue_work();  /* Thread-local, no locking. */
		do_work(wp);  /* Might include locks. */
		my_timestamp = clock_gettime(...); /* Thread-local. */
	}

	acquire_lock(&departing_thread_lock);

	/*
	 * disentangle from application, might acquire other locks,
	 * can take much longer than MAX_LOOP_TIME, especially if
	 * many threads exit concurrently.
	 */

	my_status = get_return_status();

	release_lock(&departing_thread_lock);

	/* thread awaits repurposing. */

The control thread might do the following:

	for (;;) {
		for_each_thread(t) {
			ct = clock_gettime(...);
		        d = ct - per_thread(my_timestamp, t);
			if (d >= MAX_LOOP_TIME) {
				/* thread departing. */
				acquire_lock(&departing_thread_lock);
				release_lock(&departing_thread_lock);
				i = per_thread(my_status, t);
				status_hist[i]++; /* Bad idea if TLE! */
			}
		}
		/* Repurpose threads as needed. */
	}

Here the passage of time is used to ensure that the empty lock-based critical section happens after the departing worker thread has acquired the departing_thread_lock. If the empty lock is allowed to become a no-op, then the per_thread(my_status, t) access can abort the departing thread's transaction, so that the result is the initial value of -1. This cannot happen with locking.

Note that enclosing the access to my_status in a lock/transaction does not change things. This problem is not the extra-transactional access, but rather the structure of the application.

Again, please don't try this sort of thing on superscalar CPUs (which is almost all of them these days), and especially please don't try it in the Linux kernel. In fact, you should be very cautious about doing this sort of thing even on systems specially designed for hard real-time response. But the key point is that if you have a large application that is not well understood, you might want to be careful about applying HTM-based lock elision to it. This should not be a surprise: What you have is a real-time program, and there has not yet been much work on TM in real-time environments.

Speaking of real-time environments, how does priority boosting interact with HTM-based lock elision?

Acknoowledgments: Many thanks to Josh Triplett, Mathieu Desnoyers, Jon Walpole, Phil Howard, and Stephan Diestelhorst for many illuminating discussions. Please note that my acknowledgment of these people's contributions to my understanding of transactional lock elision should in no way be interpreted to imply their agreement with anything I say here. ;-)

2012 LPC Call For Proposals

2012-03-26T16:17:05Z

The Planning Committee for the Linux Plumbers Conference (LPC) is pleased to announce a call for refereed-track proposals for the 2012 edition of LPC, which will be held August 29-31 in San Diego, CA, USA. Refereed-track presentations are about 45 minutes in length, and should focus on a specific aspect of the "plumbing" in the Linux system. The Linux system's plumbing includes kernel subsystems, core libraries, windowing systems, management tools, media creation/playback, and so on. Proposals are due at 11:59PM Pacific time on May 1, 2012.

The best presentations are not about finished work, but rather problems, proposals, or proof-of-concept solutions that require face-to-face discussions and debate among people from different areas of the Linux plumbing. Ideally, the best presentations are also working sessions that result in patches to various portions of Linux's plumbing that make the Linux world a better place for its developers and (most important) its users.

A proposal should be short: just a couple of paragraphs describing the topic, why it is important, and what parts of the plumbing it touches.

Proposals are due at 11:59PM Pacific Time on Tuesday, May 1st, 2012. Authors will be notified by Tuesday May 15th, 2012. We look forward to seeing your proposal, and to seeing you in San Diego!

SUBMITTING A PROPOSAL

We are using the Launchpad infrastructure this year, but it is still easy to submit a proposal. Go to this URL, which will take you to an account-creation page. However, if Launchpad doesn't know you, in which case please create a new Launchpad account so that it does know you. Once you have a Launchpad account, the above URL will take you to the "Register a new blueprint" page.
Once you get to the "Register a new blueprint" page:

In the "Project" field, enter "lpc".
In the "Name" field, put a short identifier for your talk. Please do -not- put your own name, as Launchpad already knows who you are.
For refereed-track presentations:
- In the "Name" field, enter a short name for your presentation starting with "lpc2012-ref-" for refereed-track presentations (for example, "lpc2012-ref-Grand-Unified-FOSS-Project").
For microconference presentations:
- In the "Name" field, enter the code shown on that microconference's wiki. For a few examples:
  - Containers microconference: “lpc2012-cont-”
  - Real-time microconference: “lpc2012-rt-”
  - Scaling microconference: “lpc2012-scale-”
  - Virtualization microconference: “lpc2012-virt-”
In the "Title" field, enter your proposal's one-line title.
The "Specification URL" field can be left blank, but feel free to enter a URL pointing to additional information.
In the "Summary" field, enter:
1. Your abstract, which should be short (a couple of paragraphs), but should clearly describe the problem, the affected areas of the Linux plumbing, and the intended audience.
2. A one-paragraph bio, describing your experience.
The "Assignee", "Drafter", and "Approver" fields may be left blank.
Please leave the "Definition Status" in the "New" state. (Changing this field can result in your submission being lost.)

Then click the "Register Blueprint" button to submit your proposal! You will automatically be recorded as the person submitting the proposal.
If you have any difficulty submitting your abstract, please email the details to contact@linuxplumbersconf.org.

Transactional Memory Everywhere: 2012 Update for HTM

2012-03-09T23:16:42Z

In a recent post, I called out some of the past year's activity in the area of transactional memory (TM). This post takes a closer look at hardware transactional memory (HTM), with an eye to where it might fit into the parallel programmer's toolbox.

But first, what is HTM?

Current HTM implementations make use of processor caches and speculative execution to ensure that a designated group of statements (called a “transaction”) executes atomically from the viewpoint of any other transaction. The beginning and end of a transaction is marked by begin-transaction and commit-transaction machine instructions, respectively, and might also include abort-transaction instructions. The transaction is executed within the confines of its CPU, so that its work is not visible to other CPUs until the commit-transaction instruction is executed. An abort-transaction instruction squashes the speculative execution, discarding any changes that the transaction might have carried out, while also branching to a failure handler. The location of the failure handler is normally specified by the begin-transaction instruction, either as an explicit address or by means of condition codes. This failure handler might then either: (1) retry the transaction, (2) execute a fallback code sequence, for example, using locking, or (3) otherwise handle the transaction's failure. In this way, each transaction either executes atomically with respect to all other transactions, or is aborted with no changes to shared state.

So where does HTM fit into the parallel programmer's toolbox? Ideally, we would like a classification of the applicability of HTM similar to that shown for RCU in the diagram below:

On the other hand, I have been developing, using, and maintaining RCU in production for almost two decades. Because the oldest commercially available HTM implementation is still quite young by comparison, any attempt to similarly classify its use must necessarily rely on educated guesses, extrapolation, and speculation. But I am most certainly not going to let that stop me from making a first attempt! :-)

The remainder of this posting will look at a number of questions that need to be asked of current commercially available HTM implementations, but without focusing on any particular implementation. The answers will be primarily about HTM in the publicly documented here-and-now, though some additional speculation about possible future implementations will inevitably leak in. This additional speculation will draw on selected academic research projects.

Does HTM Belong In The Parallel Programmer's Toolbox?

The first question we need to ask is whether HTM has any place at all in the parallel programmer's toolbox. After all, as the new entrant, it must prove its worth compared to any number of concurrency-control mechanisms that have been in production use for decades. In addition, there has been no shortage of technologies that have roused great excitement but eventually failed of their promise, a historical tendency that is greatly illuminated by Ioannidis's Sixth Corrollary.

Ioannidis notwithstanding, it seems quite likely that the answer to this first question is in the affirmative. To quote Why The Grass May Not Be Greener On The Other Side: A Comparison of Locking vs. Transactional Memory, “An important TM near-term opportunity is thus update-heavy workloads using large non-partitionable data structures such as high-diameter unstructured graphs.” (Disclaimer: I co-authored this paper, which is a revision of the 2007 PLOS paper by the same name with revised presentation here.) Although there are other mechanisms that can be used to handle large update-heavy non-partitionable data structures, these other methods suffer from cache-locality problems. There is thus hope that HTM can provide performance advantages in this area. In addition, use of HTM for transactional lock elision promises to greatly reduce the number of cache misses associated with lock acquisition and release, which is likely to provide large performance advantages to HTM for short lock-based critical sections—at least in cases where none of the oft-written variables protected by the elided lock share a cache line with that lock.

Should HTM Be Used Universally?

Given that HTM very likely has a place in the parallel programmer's toolbox, the logical next question to ask is whether parallel programmers can simplify their lives by just using HTM for everything.

The answer to this question is an emphatic “no” for the following reasons:

HTM is unable to handle non-idempotent operations, such as unbuffered I/O (especially things like remote procedure calls) and system calls including memory mapping and exec(). This limitation appears to be inherent to HTM.
HTM limits maximum transaction size. So why do current commercially available HTM implementations limit transaction size and what can be done about it?
HTM currently lacks forward-progress guarantees. So why do current commercially available HTM implementations lack forward-progress guarantees?
Even with forward-progress guarantees, HTM is subject to aborts and rollbacks, which (aside from wasting energy) are failure paths. Failure code paths are in my experience difficult to work with. The possibility of failure is not handled particularly well by human brain cells, which are programmed for optimism. Failure code paths also pose difficulties for validation, particularly in cases where the probability of failure is low or in cases where multiple failures are required to reach a given code path.
HTM is subject to thorny validation issues, including the inability to usefully set breakpoints in or single-step through transaction. In addition, although transactional lock elision might greatly reduce the probability of deadlock, the fact that the fallback code might be executed means that fallback-code deadlock cannot be ignored. Finally, in many cases the fallback code will be invoked quite infrequently, which might allow fallback-code bugs (including performance problems involving high levels of lock contention) to escape detection during validation, thus inflicting themselves on users. Use of HTM will therefore likely require painstaking attention at validation time, and also tools to detect deadlock and exercise fallback code.
There are subtle semantic differences between locking and transactional lock elision. One example is interactions with outside accesses as shown by Blundell et al. Another example is the empty lock-based critical section, which waits for all prior critical sections for locking, but is a no-op when all concurrent lock-based critical sections have been elided in favor of HTM execution. This use of locking as a message-passing mechanism can also be used in non-empty critical sections. (More on this here.)

We can see that HTM is useful on the one hand, but has substantial limitations on the other. Therefore, the next section looks at where HTM is most likely to be helpful.

Where is HTM Most Likely to be Helpful?

HTM is likely to be at its best for large in-memory data structures that are difficult to statically partition but that are dynamically partitionable, in other words, the conflict probability is reasonably low. There must be a reasonable non-TM fallback algorithm for every transaction. The workload should ideally be update-heavy with small accesses and updates, and not subject to aggressive real-time constraints. Finally, if HTM is used for transactional lock elision, any empty critical sections must continue to use explicit locking.

Why is this situation best for HTM?

A thought experiment involving a red-black tree might be helpful.

First, consider a red-black tree that supports insertions, deletions, and lookups of single elements, but where all of these APIs return the exact number of elements in the tree. In this case, the resulting transactions will be very prone to conflicts on the variable in which the element count is maintained, resulting in poor performance and scalability, especially for update-heavy workloads.

This should not be surprising, because returning the size causes insertion and deletion to be strongly non-commutative. Maintaining and returning a consistent count is simply bad for parallelism.

Therefore, let's next consider a red-black tree with insertion, deletion, and lookup, but without the exact count of the number of elements in the tree. In this case, if the tree is large, the conflict probabilities between random insertion, deletion, and lookup operations is extremely low. In this case, HTM is likely to perform and scale quite well.

Finally, let's add an API that enumerates all the the nodes in the red-black tree. A transaction implementing this API will conflict with all concurrent insertion or deletion operations, severely limiting performance and scalability. In many cases, this situation can be alleviated by using hazard pointers or RCU to protect these large read-only accesses. Although the RCU/hazard-pointer reader can still conflict with updates, the conflict window is now limited to each single read-side load, instead of extending across the entire read-side operation, as is the case for transactions. (RCU and hazard pointers could help even more if their loads didn't generate conflicts, but instead returned the old value of any variable written by an in-flight transaction, but current HTM implementations do not appear to do this.)

This last example illustrates the importance of using combinations of synchronization mechanisms in ways that play to each mechanism's strengths.

Summary

In summary, although it appears that HTM will be able to earn a place in the parallel programmer's toolbox, it is not a magic wand with which to wish away all concurrency problems. Like every other tool in the toolbox, it will be necessary to carefully consider whether HTM is the right tool for a particular job. Furthermore, like every other tool in the toolbox, it may in some cases be necessary to tune, restructure, or even redesign your code in order to obtain the full benefits of HTM.

HTM should therefore be an interesting and exciting learning experience for all concerned. ;-)

(Thanks to a great number of people, especially Jon Walpole, Josh Triplett, and Andi Kleen for a number of illuminating discussions on TM in general and HTM in particular.)

2012 LPC Call For Microconference Topics

2012-03-01T14:39:47Z

The 2012 Linux Plumbers Conference (LPC) will be held on August 29-31 in the Sheraton San Diego, and we hope to see you there!

To that end, the LPC Planning Committee is pleased to announce a call for microconferences. These microconferences are working sessions that are roughly a half day in length, each focused on a specific aspect of the “plumbing” in the Linux system. The Linux system’s plumbing includes kernel subsystems, core libraries, windowing systems, media creation/playback, and so on. For reference, last year’s LPC had tracks on Audio, Bufferbloat and Networking, Cloud, Containers and Cgroups, Desktop, Development Tools, Early Boot and Init Systems, File and Storage Systems, Mobile, Power Management, Scaling, Tracing, Unified Memory Management, and Virtualization.

Please note that submissions to a given microconference should not normally cover finished work. The best submissions are instead problems, proposals, or proof-of-concept solutions that require face-to-face discussions and debate among people from different areas of the Linux plumbing. In other words, the best microconferences are working sessions that turn problems into patches representing solutions.

Leading an LPC microconference can be a fun, exciting, and rewarding activity, but please see here for the responsibilities of a microconference working session leader. If you have an idea for a good LPC microconference, and especially if you would like to lead up a particular microconference, please add it to the LPC wiki here.

Transactional Memory Everywhere: 2012 Update

2012-02-12T02:22:34Z

The past year has been a busy one for transactional memory (TM). IBM's Blue Gene/Q and Intel's Haswell TSX have joined Sun Rock and Azul's Vega 2 in offering hardware transactional memory (HTM). All of these implementations seem to be focusing on TM in the small, wisely in my opinion, given TM's weak support for many programming-in-the-large use cases, particularly I/O and system calls. A number of TM researchers seem to be coming around to this view, as evidenced by slide 279 of these lecture slides. All of this is to the good, especially given some of the unrealistically high expectations that some people have harbored for TM.

So what do these HTM implementations offer? Perhaps it can best be thought of as an extension of LL/SC or CAS that operate on more than one memory location, with the number being limited by hardware considerations such as cache geometry. As with both LL/SC and CAS, TM can fail, both due to conflicts between concurrently executing transactions and because of hardware events such as exceptions and cache-line conflicts. Interestingly enough, this means that in many cases, hardware transactions must have some other synchronization mechanism (such as locking!) as a fallback. Unfortunately, this means that you cannot count on HTM to simplify deadlock cycles. However, it might well increase performance for some common types of large non-statically-partitionable data structures subject to smallish updates. The canonical example of such a data structure seems to be red-black trees.

One thing that disappoints me personally about all of these mechanisms is their limited support for RCU. Don't get me wrong, you can use RCU with HTM updates. However, bare RCU readers will typically cause HTM updaters to unconditionally abort. Although RCU's purpose is in fact to favor readers, this is entirely too much of a good thing. For ideas for improved interactions between TM and RCU, please see an earlier posting. It is also unclear to what extent HTM helps in cases involving severe real-time-response constraints.

There have also been some interesting software transactional memory (STM) papers. Pankratius and Adl-Tabatabai published a productivity comparison in which the highest productivity was attained by teams that used both STM and locking. Although one can argue that inevitable transactions will subsume the uses of locking in the Pankratius paper, inevitable transactions are normally implemented as a global lock. sharply limiting I/O scalability.

Dragojevic et al. published what is often viewed as a riposte to Cascaval et al.'s Software Transactional Memory: Why Is It Only a Research Toy? in the CACM article entitled Why STM can be more than a Research Toy (see here for a related technical report). The Dragojevic article has rare graphs showing semi-reasonable scalability, with up to 12 times sequential execution rate on 16 CPUs, which is certainly not splendid, but neither is it altogether embarrassing. However, that speedup is the best results from a set of 17 benchmarks, many of which show negative scalability.

But a closer reading shows that Dragojevic's and Cascaval's ideas of exactly what constitutes “software transactional memory” are quite different. Dragojevic's results use a variant of STM that takes hints from the programmer in order to indicate which data is shared or not. This is an interesting hack, and it is hard to argue with the improvement over straight STM, but it does seem to back away from the earlier TM promises of transparent parallelism.

These hints raise some interesting possibilities. Suppose that the programmer also hinted that certain pairs of transactions could never conflict due to accessing disjoint data. One could take this a step further and use names for different disjoint subsets of the data. Such hints could greatly reduce or even remove the need for complex contention managers and associated aborts and rollbacks. But the best thing about this sort of scheme is that it has seen heavy production use with good performance and scalability extending back for decades. It is called “locking”. ;-)

What is the overhead of rcu_read_lock()?

2012-01-31T18:36:03Z

A recent post to LKML stated that the patch in question did not plan to hold any global locks, including rcu_read_lock(), presumably because of concerns about contention or overhead. This blog entry is intended to help address any lingering concerns about rcu_read_lock() contention and overhead.

To be fair, at first glance, rcu_read_lock()'s source code does look a bit scary and slow:

static inline void rcu_read_lock(void)
{       
        __rcu_read_lock();
        __acquire(RCU);
        rcu_lock_acquire(&rcu_lock_map);
}

However, a closer look reveals that both __acquire() and rcu_lock_acquire() compile to nothing in production kernels (CONFIG_PROVE_LOCKING=n). This leaves __rcu_read_lock(), which is compiled differently for different settings of CONFIG_PREEMPT.

Let's start with CONFIG_PREEMPT=n. We have:

static inline void __rcu_read_lock(void)
{       
        preempt_disable();
}

And, again for CONFIG_PREEMPT=n:

#define preempt_disable()               do { } while (0)

The overall result for rcu_read_lock() when CONFIG_PREEMPT=n is therefore as follows:

static inline void rcu_read_lock(void)
{       
}

Similar analysis of rcu_read_unlock()</code> reveals that it is also an empty static inline function for CONFIG_PREEMPT=n. It is to be hoped that this is sufficiently lightweight for most practical purposes. If you find a case where it is too heavyweight, I would be very interested in hearing about it!

That leaves CONFIG_PREEMPT=y, which actually has executable code in its definition of __rcu_read_lock() as follows:

void __rcu_read_lock(void)
{       
        current->rcu_read_lock_nesting++;
        barrier();
}

The first statement is a simple non-atomic increment of a simple int that is located in the task's own task_struct. The barrier in the second statement expands as follows:

#define barrier() __asm__ __volatile__("": : :"memory")

This is an empty asm that generates no code, but that does disable code-motion optimizations that might otherwise move memory references across the barrier() statement. So, the executable code for rcu_read_lock() for CONFIG_PREEMPT=y is as follows:

void rcu_read_lock(void)
{       
        current->rcu_read_lock_nesting++;
}

A similar analysis of rcu_read_unlock() for CONFIG_PREEMPT=y yields the following:

void __rcu_read_unlock(void)
{
        struct task_struct *t = current;

        if (t->rcu_read_lock_nesting != 1)
                --t->rcu_read_lock_nesting;
        else {
                t->rcu_read_lock_nesting = INT_MIN;
                if (unlikely(ACCESS_ONCE(t->rcu_read_unlock_special)))
                        rcu_read_unlock_special(t);
                t->rcu_read_lock_nesting = 0;
        }
}

The common case of a single level of rcu_read_lock() nesting executes the else clause of the first if statement, and only executes the then clause of the second if statement when the RCU read-side critical section was either preempted or ran for several milliseconds.

So in the common case, rcu_read_unlock() for CONFIG_PREEMPT=y executes two tests of task-local variables and two assignments to task-local variables.

This should be sufficiently lightweight for most purposes.

Of course, RCU is intended for read-mostly situations, and RCU updates can have significant overhead, incurring significant latency, CPU overhead, and/or cache misses. That said, there are some special cases where RCU updates can be extremely fast, as shown in Figures 12 and 13 of the supplementary materials to User-Level Implementations of Read-Copy Update. (No, the supplementary materials are not behind a paywall: Click on the “Supplemental Material(PDF)” link.)

User-Level Implementations of Read-Copy Update

2011-12-29T21:34:21Z

Woo-hoo!

User-Level Implementations of Read-Copy Update has appeared in the February 2012 issue of IEEE Transactions on Parallel and Distributed Systems. Many thanks to everyone involved, especially co-authors Jon Walpole, Michel R. Dagenais, Alan Stern (who did the symbolic-logic heavy lifting), and Mathieu Desnoyers (who is the lead author). Mathieu also managed to convince me to go once more into the breach, which was not an easy task given that I received my license to collect RCU rejection notices all the way back in 1995. ;-)

So it does feel very good to see this finally hit print!

Confessions of a Recovering Proprietary Programmer, Part VIII

2011-12-29T04:13:20Z

My presentation at the Real Time Linux Workshop this past October, titled “On migrate_disable() and Lantencies” (presentation), was a bit of a departure for me. This presentation used a Markov model to analyze the behavior of some recent changes to the scheduler for the 3.0 series of -rt kernels.

Although this approach produced some interesting results, one difficulty is that a number of the corresponding scheduler measurement simply do not fit the exponential distribution very well. This question of course came up during my talk, where an audience member suggested instead using the Erlang distribution. Unfortunately, my only memory of Erlang distributions was of a 1980s operations-research class, where I learned how to use an Erlang distribution, but not why anyone would want to, at least not beyond the professor's vague assurances that it is helpful when modeling telephony networks.

So I answered that I might consider an Erlang distribution, but that I figured that I could match the data quite well by using cascaded Markov-model stages to represent a single state in the higher-level model. The big advantage of this cascading approach is that the math and software remains the same: You simply map additional states into the model.

However, my work reducing RCU's need for scheduler-clock ticks took precedence, so it was only recently that I got time to work out the math for cascaded Markov-model stages. I got the results I expected, but I figured that I should also do a quick review of the Erlang distribution. So I got out my old copy of “Introduction to Operations Research” by Hillier and Lieberman. Imagine my surprise to learn that the Erlang distribution is exactly cascaded Markov-model stages!

So my response to the question during my talk was essentially: “I might try the Erlang distribution, but I am going to try deriving it from scratch first.” Oh well, I would much rather look foolish with the correct answer than to look smart with the wrong answer!

On the other hand, this approach did give me a very good understanding of not only how to use the Erlang distribution, but also how to derive it and why you might want to use it. :-)

The accretion of meaning

2011-12-26T22:22:09Z

Noam Chomsky once wrote “Colorless green ideas sleep furiously” as an example of a sentence that is grammatically correct but nonsensical. However, my daughter Sarah and I were discussing this earlier today, and each of us managed to generate meanings for this phrase.

Sarah:

Colorless: Uninspiring to the lumpen proletariat.
Green: Pertaining to improving the environment.
Sleep: Ignored by mainstream words and actions.
Furiously: Angering the idea's proponents.

In other words, uninspiring ideas for improving the environment are ignored by the mainstream, to the fury of their proponents.

Paul:

Colorless: Pertaining to transparency, as in window glass and also as in transparent transistors.
Sleep: Residing in an idle state, as in idle CPUs.
Furiously: Consuming excessive quantities of electrical power.

In other words, if you fail to specify CONFIG_NO_HZ=n when building the Linux kernel for a computer constructed of transparent transistors fabbed onto a pane of glass that controls that pane of glass, this colorless green idea will sleep furiously.

In both cases, the key element is the recently added meaning for the word “green”: As the word “green” has accreted a meaning, so has the sentence “Colorless green ideas sleep furiously.”

However, it turns out that Sarah's and my line of thought has been anticipated here, here, and here, and most probably elsewhere as well.

But that is OK. It turns out that Chomsky himself was also anticipated by Lucien Tesnière, the game cadavre exquis, and possibly also Bertrand Russell. Furthermore, a poet managed to accrete meaning onto Russell's “Quadruplicity drinks procrastination,” as can be seen here. The poet had originally intended to accrete meaning onto Chomsky's sentence, but, finding that others had beat him to it, turned to Russell's sentence instead.

The fact is that thinking up a new-to-you idea can be just as much fun as thinking up a new-to-the-human-race idea. Perhaps there really is nothing new under the sun. Even if there is something new under the sun, the human race is not the only thing under the sun! ;-)

Confessions of a Recovering Proprietary Programmer, Part VII

2011-12-15T21:54:19Z

My work on RCU does require a pioneering spirit. For example, there are no classes to guide my efforts other than those I teach, and there are no publications to draw from other than those I write. One saving grace is that I work in the Linux community, which means that people can (and often do!) contribute their thoughts and ideas, many of which are now reflected throughout the Linux kernel's RCU implementation. (Thank you all! You know who you are, and there are too many of you to name you all. If you want the list, the git log command is your friend.) Nevertheless, being too afraid to stray from the beaten path implies being too afraid to work on RCU.

But there are times when the RCU implementation needs a more sane approach. During those times, I must find some other outlet for my insanity: To do otherwise is to break RCU. Fortunately, this time around, an appropriate outlet was readily available in the guise of Ubuntu's new Unity window manager.

I decided to emulate the environment of a typical Linux hacker. This meant installing, configuring, and using Unity without the advice and counsel of my acquaintances within Ubuntu. To those acquaintances who might feel some irritation at the contents of the remainder of this blog entry, I do apologize in advance. However, experiments are invalid unless the environment is properly controlled, and part of the control for this experiment was to isolate myself from such help. I did search the web, including of course ubuntuforums.org.

So here are the issues I encountered, more or less in the order I encountered them:

Right-clicking doesn't give you a way to configure the desktop, aside from setting the background image (pure black for me!). I found the answer in ubuntuforums.org: install and run “ccsm” to make major changes desktop configuration.
I work with large numbers of xterms, so that each desktop has nine 24x80 xterms in a three-by-three pattern. (And yes, when screens were smaller, I used a four-by-four pattern.) I use a script creatively named “xterms” to create them, and I expect them to be automatically placed in a non-overlapping three-by-three pattern, which they did under Gnome. But under Unity, many of them will be placed directly on top of each other. The solution is to add a “sleep 0.250“ in my script's loop. So Unity appears to be trying to do the right thing, but is a bit slow on the uptake. I learned about this experimentally, mostly because when you query for “tiled” you get advice on how to make your windows never overlap. In contrast, I want my windows to be non-overlapping by default, but if I am (say) debugging rcu_start_gp(), I want to be able to expand the window from 24x80 vertically to use the full screen size, and in that case, I am happy with the resulting overlapping windows.
Unity uses “workspaces” rather than “desktops”. Unfortunately, there doesn't seem to be to identify the windows of a given type that have been minimized from a given workspace. So if I have nine xterms in one workspace for my RCU work and another nine in another workspace for working on a paper, clicking the xterm button gets me all 18, shuffled around so that I cannot easily tell which goes with which workspace. Perhaps shift-click should show only those xterms associated with the current workspace!
I tried to work around the above problem by creating multiple desktops via “ccsm”. However, although the additional desktops exist, Unity cannot show you any windows assigned to them. The only way to see them again is to reduce the number of desktops back to 1, which will force them back onto the sole remaining desktop. (This might actually be a useful feature, but...) Worse yet, when you have more than one desktop, you lose the ability to move a given window to some other workspace by right-clicking on it: Instead, you can only move it to other desktops. Longer term, Unity should more gracefully handle multiple desktops. Until then, “ccsm” should not offer to create more than one desktop. And until then, just say “no” to the temptation to create multiple desktops. Use workspaces instead.
My habit of clicking on icons at the lower right corner of my screen to move among desktops died hard, but the window-s hotkey does turn out to be very nice. When you hit window-s, you get a screen that shows you all your workspaces, and you can use the arrow keys to move among them. When you get to your destination workspace, just hit the enter key.
Unfortunately, focus does not always follow you from one workspace to another. A quick window-s;left-arrow;enter;"cd /foo";enter sequence is quite likely to cause the xterm on the previous workspace to change to directory /foo, which can be a bit disconcerting. This really badly needs fixing, as the mental effort to verify that focus has indeed followed me sometimes causes me to forget why I wanted to be in the new workspace in the first place. This can be frustrating. (And yes, I am not as young as I used to be. Then again, neither are you!)
In Gnome, deleting a window (for example, typing exit to a bash prompt) automatically transfers focus to some other window on that same desktop. In contrast, in Unity, deleting a window sometimes transfers focus. Always would be far preferable!
I searched for synaptic in vain, finally learning about the new Ubuntu Software Center icon, which is the shopping-bag icon on the left-hand toolbar. Ubuntu Software Center seems OK, though I am not sure whether or not I would be happy to do without apt-get on the command line. Fortunately, I have both.
The left-hand toolbar did grow on me due to the fact that it seems to track the most heavily used applications. Unfortunately, there is no way to use this toolbar to query for the existence of an application: if there is an instance, it moves to the workspace containing the instance and transfers focus to it, but if there is no instance it creates one. (If there are multiple instances, it displays them all and lets you choose — but without letting you know which instance goes with which workspace.) Again, perhaps a job for shift-click, though there needs to be a way to: (1) query the current workspace and (2) query all workspaces. And a way to see which instances are associated with which workspaces. And a way to see which instances have been minimized.
A natural reaction to this sort of behavior is to fire up “ccsm” and experiment with different settings. Bad move! Doing this has a high probability of fatally confusing Unity. “You too can power-cycle your laptop.”
Unity sometimes loses track of the keyboard. Moving back and forth among workspaces helps it find the keyboard again. Unless the screen is locked, in which case life appears to be quite hard, with power-cycling the only option I have found thus far. Fortunately, this seems to be quite rare, only two sightings in the several weeks that I have been using Unity. Oops, make that three sightings. Hmmm... Four sightings. OK, this problem appears to be triggered by switching to a workspace then immediately hitting shift-F1.
Under Gnome, resizing an xterm displays a handy popup showing how many rows and columns of text the xterm contains as it is being resized. Unity badly needs this feature.
Where did the application and system menus go??? Well, they are gone. Once I got used to it, the replacement works much better for me. Shift-F1 pops up a window that allows you to search for apps, so that shift-F1;chr;enter pops up an instance of the Chromium browser. Shift-F2 works as it does in Gnome, except that it displays the possible completions as icons. Unfortunately, in both cases, Unity doesn't always keep up with touch-typists. Sometimes it re-executes the previous command instead of the one you just typed, even though the display has already updated to show the new icon. This indicates some performance, coordination, and concurrency issues: Unity's right hand does not always know what my left hand is doing! It is therefore not obvious to me that the Unity development and test teams include any touch typists. One way or another, this sort of thing badly needs fixing.
Banshee is quite similar to Rhythmbox. One nice thing about Banshee is that it does not come with a pile of Internet radio stations preconfigured, so that you can easily find the ones you add. (My tastes in music are such that my favorites are never going to appear in any reasonable set of defaults.)
If you push a window beyond the bounds of the sides or bottom of the screen, it ends up spanning multiple workspaces, where the workspaces are connected in a toroidal fashion. This is sort of OK, except that windows that span multiple workspaces (up to four of them!) are not handled gracefully by the left-hand taskbar. Although this behavior was mildly entertaining for a while, I would prefer that the workspaces not be connected.
If you push a window up to the top of the screen, it maximizes. This is almost never what I want — the reason I pushed the window to the top of the screen was to have it appear at the top of the screen, not to have it monopolize the entire screen!!! On the rare occasions when I need to maximize a window, double-clicking the title bar works just fine, thank you!!!
Where did the per-window menus go? Well, they are mostly gone. You can get them back by right-clicking the title bar of a given menu, but I am growing to like the alternative, which is to move the mouse to the very top of the screen, causing the per-window menus to appear on the screen's upper bar. This leaves more screen real estate for the rest of the application. However, some applications keep the window-bar menus, and I have no idea why.
So I have a browser in one workspace, and I want one in another workspace. Clicking on the icon on the left-hand bar pops me back to the original workspace: Not what I wanted! Right-clicking on the icon pops up a menu that allows me to create a new browser instance in the current workspace. However, this works only for some applications.
The soffice command does not automatically background itself, in unhappy contrast to Maverick's ooffice command. OK, OK, I can easily type &, but it is still annoying.
If you start soffice from an xterm, it splats on that xterm every time you do “save as”. Yes, I know that you failed to open the file! I wouldn't have done “save as” if the file already existed!
The soffice application occasionally goes into contrarian mode when you resize it. For example, an attempted horizontal expansion of the window might instead be applied vertically. Sometimes soffice simply refuses to let you resize it, which can be especially frustrating if it has decided that it should be so small that there is no room to actually display the document in question. Repeatedly maximizing and unmaximizing the window seems to get it out of this passive-aggressive mode of operation.

So Unity does appear to have some potential, and I intend to keep using it for at least a little while longer. However, it does still need a fair amount of additional work.

And it used to be so simple...

2011-11-11T18:12:19Z

It is something I have done many times under many different operating systems. You write something into the startup scripts, take the system down, then regain control at startup. So I planned to take this approach for my KVM-based rcutorture setup: Mount the image file, deposit a test script in the startup scripts, unmount the image file, start qemu, then analyze the results (either deposited in the filesystem or dumped to the console). What could be easier?

So I manually mounted the file I use as the qemu disk image, my plan being to manually create simple tests as a first step towards automation.

Hmmm... Everything looks very strange and unfamiliar. Ah, yes, this is upstart. No problem, the Internet is but a mouse-click away, right? But wait, the documentation says that upstart and the more-familiar (to old guys like myself, anyway) sysvinit can co-exist on the same system. And what about this new systemd thing that I keep hearing about?

So my tests are going to have to inspect the filesystem and figure out which of these three things is installed. And maybe someone installed one of them, then switched to the other, leaving traces of their earlier choice lying around. How is my script supposed to figure all that out? Of course, if it was just me running the tests, I could simply make sure that a pre-selected boot-up mechanism was installed. But that is silly — anyone anywhere should be able to test RCU. So I need to be able to deal with any of the three boot-up mechanisms. But wait, it is far worse than that: What is my script supposed to do when someone else comes up with a fourth, fifth, or sixth boot-up mechanism? Ouch!!!

It is clearly time to just say “no”! After all, my main test (rcutorture) is coded in the kernel, so it is clearly going to be far easier to translate my driver scripts to Linux kernel code than to code up a script that can outwit all present and future perpetrators of bootup mechanisms. In-kernel coding will allow me to fully control the test using kernel boot parameters, and thus happily ignore which of sysvinit, upstart, systemd, or whatever is installed. Some times you just have to get a horse!

The -rcu tree is back on kernel.org

2011-11-02T17:04:31Z

Thanks to the key-signing last week, -rcu is back on kernel.org, though at a slightly different address: git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git

Parallel Programming: Back on kernel.org, Chinese translation

2011-10-29T06:01:31Z

The git archive for “Is Parallel Programming Hard, And, If So, What Can You Do About It?” is once again available from git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/perfbook.git. A big “thank you” to everyone who worked so hard to get kernel.org back!!!

In addition, Lu Yang, Xie Baoyou, and Yu Chen are working on another translation effort, with much of the book already translated to Chinese. A download is said to be available here, though my Chinese is not up to the task of figuring out exactly how to download it, even with the help of Google Translate.

I am hoping that this group and the group reported earlier will continue to make great progress on this!

Parallel Programming: Chinese Translation In Progress

2011-10-02T22:12:42Z

Cheedoong Drung has started a Chinese translation of “Is Parallel Programming Hard, And, If So, What Can You Do About It?”. He is making good progress, and would also be happy to have others join in. So if you have good Chinese and English skills (I guess I am one for two on that one), get in touch with Cheedoong. (His contact information is on his web site).

Linux Plumbers Conference 2011

2011-09-12T18:21:41Z

Now that Linux Plumbers Conference has come to a successful conclusion and I have had a weekend to catch up on sleep, I want to offer my heartfelt thanks to the members of the program committee, whose efforts were key to this conference's success, in email-address order: Allison Randal, Jon Corbet, James Bottomley, Jesse Barnes, Jes Sorenson, Jesse Barker, Joel Becker, James Morris, Julia Lawall, Kate Stewart, John Linville, Matthew Garret, Rafael Wysocki, Thomas Gleixner, and Tom Herbert. And of course to the speakers and attendees as well!

Almost all of the presentations are now available on the session pages linked from the schedule.

On the Development Tools Microconference: Julia Lawall noted that there were many different types of tools presented. As a result, she posted a survey to gather ideas for further directions on development tools: Please take a few minutes to fill out the survey.

On the Bufferbloat Microconference: Jim Gettys was happy to see that most people now agree that there is a serious buffer-management problem (as opposed to merely a specific active queue management (AQM) problem), he remains concerned that to few people understand how serious the problem really is. ;—)

On the Scaling Microconference: Good presentations by Mathieu Desnoyers on user-space scalability and by Andi Kleen on kernel scalability, but only a very few members of the audience were actively working on user-space scalability, despite a number of prominent user-space applications having single global locks that sharply limit scalability.

On the Tracing Microconference: There was some good discussion of coordinated tracing in virtualized environments between host and guest OSes, which should be helpful to the increasing numbers of people running such environments.

And much much more...

Parallel Programming: Temporary Home

2011-09-12T01:07:07Z

Due to some unscheduled downtime, perfbook's usual home is unavailable. I have therefore posted a recent PDF of perfbook here.

Parallel Programming: August 2011 Update

2011-08-29T16:40:37Z

This release of the parallel programming book features a new data-ownership chapter, the start of a new validation chapter, and lots of fixes, primarily from Elie De Brauwer.

I expect to put out the next release in December. I will continue on validation, and hopefully get a few other things accomplished.

As always, git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/perfbook.git will be updated in real time.

Stupid RCU Tricks: Bug Hunt Number 2

2011-08-15T13:25:57Z

Lai Jiangshan spotted another bug in the solution posted for bug hunt number 1. Here is the fixed (but still buggy) version:

  1 struct foo {
  2   struct list_head list;
  3   int key;
  4   int data;
  5 };
  6
  7 LIST_HEAD(mylist);
  8 DEFINE_SPINLOCK(mylock);
  9 struct foo *cache;
 10
 11 int search(int key, int *data)
 12 {
 13   struct foo *p;
 14
 15   rcu_read_lock();
 16   p = rcu_dereference(cache);
 17   if (p != NULL && p->key == key)
 18     goto found;
 19   list_for_each_entry_rcu(p, &mylist, list)
 20     if (p->key == key) {
 21       rcu_assign_pointer(cache, p);
 22       goto found;
 23     }
 24   rcu_read_unlock();
 25   return -ENOENT;
 26 found:
 27   *data = p->data;
 28   rcu_read_unlock();
 29   return 0;
 30 }
 31
 32 int insert(int key, int data)
 33 {
 34   struct foo *p = kmalloc(sizeof(*p), GFP_KERNEL);
 35
 36   if (p == NULL)
 37     return -ENOMEM;
 38   p->key = key;
 39   p->data = data;
 40   spin_lock(&mylock);
 41   list_add_rcu(&p->list, &mylist);
 42   spin_unlock(&mylock);
 43 }
 44
 45 int delete(int key)
 46 {
 47   struct foo *p;
 48
 49   spin_lock(&mylock);
 50   list_for_each_entry(p, &mylist, list)
 51     if (p->key == key) {
 52       list_del_rcu(&p->list);
 53       RCU_INIT_POINTER(cache, NULL);
 54       spin_unlock(&mylock);
 55       synchronize_rcu();
 56       free(p);
 57       return 0;
 58     }
 59   spin_unlock(&mylock);
 60   return -ENOENT;
 61 }

Can you spot the additional bug?

Stupid RCU Tricks: RCU-Protected Deletion

2011-07-26T16:45:52Z

Consider the following code fragment:

  1 rcu_read_lock();
  2 p = rcu_dereference(gp);
  3 if (must_delete(p)) {
  4   spin_lock(&my_lock);
  5   if (must_delete(p)) {
  6     kfree_rcu(p);
  7     rcu_assign_pointer(gp, q);
  8     already_deleted(p);
  9   }
 10   spin_unlock(&my_lock);
 11 } else {
 12   do_something_with(p);
 13 }
 14 rcu_read_unlock();

Note that the whole fragment is enclosed in an RCU read-side critical section. Line 2 picks up a local pointer to the global RCU-protected pointer gp, which we assume is never NULL. Line 3 checks to see if it is necessary to delete the structure referenced by gp, and, if so, lines 4-10 do the deleting. Otherwise, line 12 does something with the just-accessed structure.

If the structure must be deleted, we first acquire a spinlock on line 4, and then line 5 re-checks whether it is still necessary to delete the structure, but this time under the lock. If the structure still needs to be deleted, then line 6 frees it (but only after an RCU grace period has elapsed) and line 7 makes gp point to some replacement structure q. Clearly this replacement structure must have been allocated somehow, but the details of exactly how that happens are not important to this example. Next, line 8 marks the old structure as having already been deleted, which resolves races when multiple CPUs concurrently notice that the structure needs deleting, and is also the reason that line 5 rechecks under the lock. Finally, line 10 releases the spinlock.

This code fragment is a bit unconventional: Normally lines 6 and 7 would be interchanged. But the RCU read-side critical section will prevent the structure from being freed until after the global pointer is updated, right? Why or why not?

Puzzle: How many unlock paths?

2011-07-23T20:56:25Z

I was recently at dinner with some Linux kernel hackers who were showing off their smartphones. These phones have an interesting unlock mechanism: the phone displays an 3-by-3 array of circles, and the user traces out a path among the circles. If the path is correct, the phone unlocks.

The paths are not arbitrary. From what I could see, here are the rules governing them:

Paths are composed of a series of straight line segments.
Each line segment connects a pair of circles.
If the preceding line segment arrived at a given circle, then the next line segment must leave that same circle.
A given circle may be visited only once.
A line segment may pass over a given circle without visiting it only if that circle has already been visited. (Attempting to pass over a not-yet-visited circle will instead visit that circle.)

How many unique paths are there?

Stupid RCU Tricks: Bug Hunt Number 1

2011-06-05T23:29:18Z

The following code fragment contains a bug semi-similar to one recently found in the Linux kernel.

struct foo {
	struct list_head list;
	int key;
	int data;
};

LIST_HEAD(mylist);
DEFINE_SPINLOCK(mylock);
struct foo *cache;

int search(int key, int *data)
{
	struct foo *p;

	rcu_read_lock();
	p = rcu_dereference(cache);
	if (p != NULL && p->key == key)
		goto found;
	list_for_each_entry_rcu(p, &mylist, list)
		if (p->key == key) {
			rcu_assign_pointer(cache, p);
			goto found;
		}
	rcu_read_unlock();
	return -ENOENT;
found:
	*data = p->data;
	rcu_read_unlock();
	return 0;
}

int insert(int key, int data)
{
	struct foo *p = kmalloc(sizeof(*p), GFP_KERNEL);

	if (p == NULL)
		return -ENOMEM;
	p->key = key;
	p->data = data;
	spin_lock(&mylock);
	list_add_rcu(&p->list, &mylist);
	spin_unlock(&mylock);
}

int delete(int key)
{
	struct foo *p;

	spin_lock(&mylock);
	list_for_each_entry(p, &mylist, list)
		if (p->key == key) {
			list_del_rcu(&p->list);
			spin_unlock(&mylock);
			synchronize_rcu();
			kfree(p);
			return 0;
		}
	spin_unlock(&mylock);
	return -ENOENT;
}

Can you find it and fix it?

Parallel Programming: May 2011 Update

2011-05-31T12:55:49Z

Once again, a big thank you for everyone who contributed to the parallel-programming book. This edition includes:

Nicer fonts, courtesy of Tom Gunderson.
Front and back cover art, courtesy of Melissa Broussard.
Greatly expanded locking chapter (though this chapter never really will be complete).
An appendix covering recent history of RCU in the Linux kernel.
A section introducing RCU concepts as preparation for RCU fundamentals.
A section covering seqlock.
A conflicting-views section covering CPU evolution.

I expect to put out the next update in August. My guess is that I will focus on data ownership and perhaps a bit on data structures, but I do reserve the right to change plans at any point. ;–)

As always, git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/perfbook.git will be updated in real time.

Linaro Developers Summit: ARM Linux Consolidation Summing Up

2011-05-23T15:15:29Z

I wrote up a summary for LWN. In other news, Jon Corbet published a blog entry on this topic that is well worth reading. In addition, another LWN article gives an excellent parable illustrating the forces that lead to code duplication.

Linaro Developers Summit: Even More ARM Linux Consolidation

2011-05-13T10:19:46Z

Thursday's pair of sessions covered Device Tree and the how/who of ARM Linux consolidation. This was followed by a wrap-up session on Friday morning.

On Device Tree, Russell King acked the infrastructure patches, so we can now open the floodgates for its exploitation by ARM boards and SoCs. Woo-hoo!!! ;–)

Grant Likely expects to get these patches into 2.6.40, where they will serve the PowerPC as well as the ARM architectures. PowerPC will make fuller use of this infrastructure by moving to the new-style irq domain by 2.6.41. Note that we once again have Linux core code rather than ARM-specific code. This has a number of important benefits: (1) A larger number of developers can be focused on a common piece of code, which should increase quality and maintainability. (2) Use of Linux core code instead of ARM-specific code reduces the pressure on the ARM maintainers.

The ARM portion of the device-tree effort is not limited to Linaro members. For example, David Brown of Qualcomm will be starting on device tree for MSM and Grant Likely and Arnd Bergmann will be working with the Xilincs folks to get Xilincs support with device tree merged in 2.6.40.

Device Tree requires .dts files that describe a given system, and so much discussion centered around where these files might live. We eventually settled on placing them in the kernel source tree in the short term, but moving them to a new device-tree repository once this effort is well under way.

Finally, Device Tree is not complete. Over time, new bindings will need to be defined to cover additional hardware and IP blocks. But this will always be the case: new types of IP blocks will appear, and as they appear, device tree may need to evolve to accommodate them. I never could have predicted that accelerometers and GPS units would appear in hand-held battery powered devices, and so I freely admit that I cannot imagine what will appear over the next five or ten years. But I do know that whatever strange devices might appear, device tree must evolve to handle them.

The next Thursday session covered specifically how to handle the ARM-Linus interface going forward.

There will (of course) be a new git tree. After all, just as every computer-science problem seems to be solvable by an additional level of indirection, every maintainership problem seems to be solvable by an additional level of git tree. This new git tree will have at least one branch per participating ARM subarchitecture, and these branches will not normally be subject to rebasing. As you would expect, maintainers of participating ARM sub-architectures will send pull requests to a group of maintainers for this new git tree. A merge of all the branches will be sent to Stephen Rothwell's -next tree, but the branches will be individually pushed to Linus Torvalds.

Please note that this new maintainership regime is not limited to Linaro, but neither is it mandatory for non-Linaro ARM subarchitectures.

The membership of this overall ARM subarchitecture maintainers group will start with Arnd Bergman, Nicolas Pitre, and Marc Zyngier, with help from Thomas Gleixner. Russell King will also have write access to this tree. This effort will start with 2.6.41. The plan is to start small and evolve to handle the load.

Will this solve all ARM maintainership problems?

Of course not!

For but one example, there will still be a need for more maintainers for the various ARM subarchitectures, many of which are quite complex. However, this new effort will be very helpful, and the parallel ARM consolidation efforts should also help reduce the size of the ARM tsunami of Linux kernel code. We were all in favor this group evolving to meeting changing needs rather than attempting any sort of “intelligent design”.

Friday's session was wrap-up. The notes for all sessions are publicly available:

Wrap-up notes may be found at the end of this session, search for "summary of".

Look for additional discussion/flaming on the arm-linux-kernel and linux-kernel mailing lists!

Linaro Developers Summit: More ARM Linux Consolidation

2011-05-11T20:05:43Z

We had a principled discussion today. In particular, we arrived at several principles:

The only ARM-specific code is (a) ARM core architecture and (b) ARM SoC core architecture.
What would otherwise be ARM board-specific code should be handled by (a) device tree and (b) device drivers in /drivers/*.
Functionality should be grouped by purpose rather than by implementation, which will help bring out common patterns and opportunities for further consolidation.
The more SoCs there are, the greater the motivation and practicality of pulling out common code. The number of SoCs is expected to continue to increase over time.

The “ARM SoC core architecture” should consist primarily of plugins for common frameworks. These frameworks should preferably be Linux-kernel-wide rather than being confined to the ARM tree. It is quite possible that the ARM SoC core architecture will turn out to be nonexistent. This would be a good thing: the more we consolidate code into common frameworks, the better able we will be to productively develop and maintain Linux on ARM.

We also continued our earlier discussion on opportunities for consolidation and people who are best positioned to take on this work:

GPIO (Grant Likely, Linus Walleij).
Pinmux (Linus Walleij).
MM and IOMMU (move the drivers to common place: individual sub-arch maintainers, Joerg Roedel might need some help for embedded).
CPU freq drivers (move the drivers to common place: individual sub-arch maintainers).
CPU idle drivers (move the drivers to common place: individual sub-arch maintainers).
Demultiplexed interrupts (Thomas Gleixner).
Cascaded interrupt controllers (especially the 256-interrupt parts where you use 1 interrupt).
Convert timer init to "earlytimer" like SH (Magnus Damm).
API to use multiple HW timers (for example, OMAP has 32 in some cases). Do we gain anything by just having them in the kernel? Have yet to see a real good use for this where existing timer interfaces in kernel don't work. In some cases, the Linux kernel is reserving a HW timer to be used by a DSP OS. Need to understand the uses cases to create proper API to manage these. (Thomas Gleixner and Kevin Hilman to look at use cases and derive appropriate solution.)
PWM drivers (move the drivers to common place: individual sub-arch maintainers).
Industrial I/O drivers (now in staging, should stay out of arch/arm: Kyungmin Park & Jonathan Cameron).
drivers/mfd: (multi-function devices) getting overused—"just a bag of platform devices". This is OK for the device driver itself, but the subdrivers should go into the appropriate GPIO location. Again, align with hardware—IP block separate from how it is connected. (Samuel Ortiz is the maintainer, so we clearly need to work this out with with him).
pruss DSP: patches being proposed, Arnd has been reviewing. These patches are being generated by new-to-Linux developers.
General infrastructure to handle instantiation of new devices at runtime as well as at boot: DSPs, FPGAs (Mark Brown).
Near-Field Communication drivers: Nokia has driver upstream, discussion about long-term interface, leaning towards sockets-based interface.
Maintainer for drivers/misc and drivers/char: Arnd Bergmann. Goal: force drivers to move into appropriate place, rather than dumping random functionality into drivers/misc and drivers/char.

Of course, the discussion is the easy part. The proof of the pudding with be the patches.