Friday, 27 September 2013

Network driver descriptors: ath9k as reference

DMAing is one of the most important parts to understand for any network driver developer. In this post, we will try to explore few details of DMA in Tx  path. We will use ath9k as the reference driver. This should be similar for ath5k, madwifi or any other Atheros driver including Fusion and Aquila.

Before going into the details, let me explore the components involved. Mainly we have the host (on which the network driver is running) and the MAC (part of firmware in the wireless card). MAC consists of different sub components including Queue Control Unit (QCU) and DCF Control Unit (DCU). In Tx path the frame transmission begins at QCU and later passed to corresponding DCU for transmission into air. There exists one QCU for each category. Different categories include, Best Effort, Background, Video, Voice, Beacon, UAPSD and CAB.

 MAC is responsible for  transferring frames between host memory and card. All the transfers happen using a structure called descriptor. Host creates, populates and provides the descriptors to MAC for further processing by MAC. Please note that the MAC needs the physical addresses and not the virtual addresses. Hence the host need to map the virtual addresses of the descriptor before passing it to the MAC.


Descriptors are more like device specific. While passing the descriptors to the MAC, they are maintained as linked list. So  descriptor structure should contain  a pointer to the next descriptor. MAC processes the list of descriptors in the order and raises an interrupt (TXEOL) at the end of the list.

Also, the tx descriptor should contain the physical address of the actual buffer to be transmitted.

Descriptors not only useful for passing the frame to the MAC, but also for fetching the tx and rx status after the transmission and reception consecutively. However this information is fetched through a call to the HAL layer. For example, please see the definition of ath_tx_edma_tasklet in ath9k driver. This information is fetched from HAL layer using the function, ath9k_hw_txprocdesc.

As an example descriptor, please see below the descriptor used for Atheros based cards. It contains 24 32-bit words. This definition is from ath9k driver. From the definition we can see that we can specify multiple buffers in a single descriptor. However as of now, the current implementation passes only one buffer.

/* Transmit Control Descriptor */
struct ar9003_txc {
        u32 info;   /* descriptor information */
        u32 link;   /* link pointer */
        u32 data0;  /* data pointer to 1st buffer */
        u32 ctl3;   /* DMA control 3  */
        u32 data1;  /* data pointer to 2nd buffer */
        u32 ctl5;   /* DMA control 5  */
        u32 data2;  /* data pointer to 3rd buffer */
        u32 ctl7;   /* DMA control 7  */
        u32 data3;  /* data pointer to 4th buffer */
        u32 ctl9;   /* DMA control 9  */
        u32 ctl10;  /* DMA control 10 */
        u32 ctl11;  /* DMA control 11 */
        u32 ctl12;  /* DMA control 12 */
        u32 ctl13;  /* DMA control 13 */
        u32 ctl14;  /* DMA control 14 */
        u32 ctl15;  /* DMA control 15 */
        u32 ctl16;  /* DMA control 16 */
        u32 ctl17;  /* DMA control 17 */
        u32 ctl18;  /* DMA control 18 */
        u32 ctl19;  /* DMA control 19 */
        u32 ctl20;  /* DMA control 20 */
        u32 ctl21;  /* DMA control 21 */
        u32 ctl22;  /* DMA control 22 */
        u32 ctl23;  /* DMA control 23 */
        u32 pad[8]; /* pad to cache line (128 bytes/32 dwords) */
} __packed __aligned(4);

In ath9k and other Atheros based drivers , between the actual frame (skb) and descriptor, there is an abstraction called, "struct ath_buf". This is an intermediary structure holding the important data that should be accessed  before and after transmission. This structure encapsulates data like physical and virtual addresses of the descriptor, physical address of the frame (skb->data) and details like pointer to station information.

Creation of Descriptors:

Descriptors are created once and should be consistent between host and device access. Hence it is better to use consistent (coherent) DMA mapping. Corresponding calls are pci_alloc_consistent, dma_alloc_coherent and dmam_alloc_coherent.

As an example, please refer the definition of the function ath_descdma_setup in any of the Atheros based drivers. In ath9k,  memory is allocated using dmam_alloc_coherent. In some other drivers the memory is allocated using pci_alloc_consistent. Please observe that multiple number of descriptors are allocated.

Populate the descriptor:

When a frame is received by the driver, its physical address (after dma mapping) should be populated into a descriptor and the address of the descriptor should be passed to the hardware.

Please see the definition of the function ath_tx_setup_buffer in ath9k driver.  First an ath_buf is dequeued from the list of free buffers and the frame (skb) is dma mapped using the function call, dma_map_single.

Please note that we are using dma_map_single for skbs and dmam_alloc_coherent for  descriptors. dma_map_single is streaming DMA routine. Generally once the skb is mapped, host does not modify any of its contents (Except in some special cases like UAPSD). Hence streaming DMA is fine for mapping the skbs.

Please also observe that the physical address is saved in one of the fields (bf_buf_addr)  of ath_buf which will be used later.  Actual values of the descriptor are populated in HAL related functions. One such function is ar9003_set_txdesc. This function is invoked as a function pointer from ath9k_hw_set_txdesc which is invoked from ath_tx_fill_desc (in some drivers the corresponding function might be ath_hal_filltxdesc)  which in turn is invoked from ath_tx_send_normal.  You can see that the physical address of the frame (bf_buf_addr) is used here and populated into the corresponding field in the descriptor.

Pass the descriptor to the hardware:

Once the descriptor is populated, it is passed to the hardware.  In ath9k or other Atheros drivers, corresponding function is ath_tx_txqaddbuf. There are two different paths here. 

In case of enhanced DMA, the descriptor is directly given to the corresponding queue. Corresponding function call in ath9k is ath9k_hw_puttxbuf. Please observe that the queue number and physical address  of the descriptor (bf->bf_daddr) are passed to this function.

If the enhanced DMA is not supported and if the queue is empty then the descriptor is directly passed to the corresponding queue. In case the queue is not empty, the descriptor is appended to the queue. In ath9k driver, you can see the invocation of ath9k_hw_set_desc_link to append the frames to the tx queue. The frames in the queue are processed in FIFO order.

Friday, 20 September 2013

WiFi: Layer-3 fragmentation vs 802.11 Fragmentation

We all know that fragmentation of packets happens at layer-3. 802.11 standard also specifies fragmentation of frames. Generally 802.11 fragmentation happens at network driver level. However there are few other trivia  worth of discussing.

While layer-3 decides to fragment packets based on MTU (Maximum Transmission Unit), in wireless (802.11/WiFi) the decision is taken based on a constant called, "Fragmentation Threshold".  This fragmentation threshold is generally configurable.

Command to set Threshold
On Linux, for layer-3, MTU can be configured by ifconfig and 802.11 fragmentation threshold can be configured using iwpriv.

Sequence Number
Please note that we refer to the sequence number from 802.11 header in the following.

When packets are fragmented at layer-3, each fragment is given as a separate packet to the wireless network driver and they are considered as different frames. Each frame gets a different sequence number.

In the case of 802.11 fragmentation, the frames are fragmented by the network driver and each fragment gets the same sequence number with different fragment number.

Thursday, 12 September 2013

Specifying the list of users to access GUI in OpenWRT

LuCI is one of the most used GUI packages on OpenWRT. When you access the device using LuCI based packages, you will be prompted to provide the username and password. By default it allows only the user "root".If you are not comfortable in using root based login, you can specify new user by modifying the LuCI package.

For adding more users or removing the root user, you need to edit the file modules/admin-mini/luasrc/controller/mini/index.lua in the luci package. Open that file and look for "page.sysauth ="  (without quotes). By default you will see the line with root as the user ie  page.sysauth = "root". If you want specify any other username (say admin) instead of root, change this line to page.sysauth = "admin". Multiple usernames also can be specified with each username is separated by comma and the list is enclosed with { and }. As an example root and admin are specified as page.sysauth = {"root", "admin"}

You can use adduser command to add a new user.

Friday, 6 September 2013

Local source for LUCI on OpenWRT

OpenWRT is the most popular development platform for Wireless routers. The first thing that one can find for GUI based configuration is LUCI package. Integration of LUCI into your OpenWRT workspace is 3-step process.
  • Download
  • Install 
  • Build 

OpenWRT provides a convenient way to perform the above 3 steps easily.

To download and install, you need to specify the source and execute the download and install scripts. You can specify the source in feeds.conf.default. Open the file and check if there already exists a line which contains ""src-svn luci". If one exists, check whether the source repository is valid. As of this writing, valid source repository is  You can specify this repository in your file.

Now that specifying the sources is done, execute the commands, "./scripts/feeds update luci; ./scripts/feeds install -a -p luci". This will download the LUCI sources and install them into your workspace.

To integrate this installed package in your binaries, you need to build again(Issue make) .

Maintaining a local copy

Once you get the LUCI source, there are chances that you don't want to download it again and instead want to make changes to the existing repository and use it.You can use the modified repository as your IP too :)

When you download LUCI initially, it is downloaded into the directory called feeds. You can use it as a local repository. Following is one of the clean ways of doing it.

  • Create a directory for the local repository, say nearhop in your base OpenWRT directory (mkdir nearhop).
  • Copy feeds/luci into the direcotry, nearhop (cp -r feeds/luci nearhop).
  • In feeds.conf.default, change luci source to your local repository. ie comment the line containing "src-svn luci" (Just place a # at the beginning of this line.
  • In the same file, add  the line "src-cpy luci nearhop/luci" and save.

Now you can customize your GUI, by editing the files in the local repository (nearhop/luci in our case). Whenever you modify/add/delete any of the files, you need to install the modified package and build it using the commands, ./scripts/feeds update luci
./scripts/feeds install -a -p luci

Thats it. Enjoy your customized GUI.