Thursday, 29 August 2013

Locking tips while developing network driver

Wireless Network driver is the most important part of any Access Point/Router. Some of the famous drivers are ath9k, ath5k, madwifi etc. One of the most important thing that one should understand while developing/modifying drivers is Locking mechanism.

Locking must be used to avoid race conditions. However if not used properly, it will lead to deadlock or crashes. The type of lock that one need to use mainly depends on the the execution context which will try to acquire the lock. Linux offers several types of locks to handle different scenarios. Here in this post I will cover few of them which are useful (or I have seen) in Network driver context. The following discussion assumes SMP architecture.

Locks in Linux

Locks offered by Linux can be broadly divided into two categories. They are -
  • Spinlock
  • Semaphore
Both the locks ensure that "only one process enters the critical section at any time". However with spinlock, a process has to wait (busy wait) in a loop until the lock is available. With semaphore, the process goes into sleep until the lock is available.  With Spinklock CPU cycles will be wasted as the process is kept busy and with Semaphore there is a chance that the process may sleep if the lock is not available immediately.  Hence it is a good approach to use spinlocks where the lock holding time is small or where sleeping is not allowed and use semaphore if sleeping is not a sin.

Execution Contexts and Locks

In Linux kernel, at driver level, mainly there are three types of execution contexts.  They are -
  • Interrupt context
  • Bottom half context (SoftIRQ, Tasklet)
  • Kernel/User Thread Context (workqueue, idle, IOCTL)
Interrupt handlers are executed in Interrupt context. One basic principle is that the interrupt handler should complete the execution quickly and  operations including sleeping, interaction with user space for data transfer or invoking scheduler should be avoided. Now coming back to our main topic, what kind of lock is best suited in interrupt context? Since interrupt handlers should not sleep, we should use Spinlocks in Interrupt context. In Linux, spin_lock, spin_lock_irq and spin_lock_irq_save can be used in interrupt context. The difference between spin_lock and spink_lock_irq is that the latter disables interrupts on the local processor. And spin_lock_irq_save saves current interrupt state along with disabling interrupts.

Bottom halves are like Interrupts. Hence all the restrictions mentioned above for interrupts are applicable to bottom halves also. Hence in bottom-half context also, Spinlocks should be used. In Linux, spin_lock and spin_lock_bh  can be used in bottom-half context. The difference between spin_lock and spink_lock_bh is that the latter  disables bottom-halves.

For Kernel/User Thread context, it is upto the requirement whether to go for semaphores or spinlocks. In general, if the lock holding time is less than the context switch time overhead, go with spinlocks. Otherwise go with semaphores.


Linux also offers more sophisticated locks like read-write semaphores and read-write spinlocks. For example, rwlock_t is a spinlock, which can be acquired in read or write modes. Any number of processes holding the read lock can enter the critical section. However write access is exclusive.

 

 Mixed contexts 

 

In the above section we discussed about locking under a particular execution context. Now it is time to enumerate the locks to be used when the critical section could be entered under different paths each with same or different execution contexts. I will try put these in the following table. Please note that, at any place you can use read-write variants (example rwlock_t) of the corresponding lock. It purely depends on the underlying scenario. However the following choices shall work fine.



Context x Context Interrupt Context Bottom Half Context Thread Context
Interrupt Context
spin_lock_irq_save 
spin_lock_irq_save

In this scenario you may use spin_lock in interrupt context  code but not  in bottom-half context code (unless interrupts are disabled by some other way)
spin_lock_irq_save

In this scenario you may use spin_lock in interrupt context  code but not  in Thread context code (unless interrupts are disabled by some other way)
Bottom Half Context
spin_lock_irq_save
In this scenario you may use spin_lock in interrupt context  code but not  in bottom-half context code (unless interrupts are disabled by some other way).

spin_lock_bh

spin_lock_bh 
 In this scenario you may use spin_lock in bottom-half context  code but not  in Thread context code.
Thread Context spin_lock_irq_save

In this scenario you may use spin_lock in interrupt context  code but not  in Thread context code (unless interrupts are disabled by some other way).
spin_lock_bh 
 In this scenario you may use spin_lock in bottom-half context  code but not  in Thread context code.
spin_lock or
semaphore. Purely depends on the scenario.

Some trivia

Need to remember the following points while designing locking solution.

1. When you acquire multiple locks, always unlock them in the reverse order of locking. Otherwise it may lead to deadlock
2. If the interrupt handler is re-entrant, before doing anything else, always disable the interrupt corresponding to the interrupt handler and enable it only at the end.
3. Never try to acquire a semaphore while holding a spinlock. That will lead to deadlocks.

No comments:

Post a comment