Tag Archives: programming

Dev boards? Where we’re going we won’t need dev boards…

A complete tutorial for using an STM32 without a dev board

Introduction

STM32F103C8 Tray from eBay

About two years ago I started working with the Teensy 3.1 (which uses a Freescale Kinetis ARM-Cortex microcontroller) and I was super impressed with the ARM processor, both for its power and relative simplicity (it is not simple…its just relatively simple for the amount of power you get for the cost IMO). Almost all my projects before that point had consisted of AVRs and PICs (I’m in the AVR camp now), but now ARM-based microcontrollers had become serious contenders for something that I could go to instead. I soon began working on a small development board project also involving some Freescale Kinetis microcontrollers since those are what I have become the most familiar with. Sadly, I have had little success since I have been trying to make a programmer myself (the official one is a minimum of $200). During the course of this project I came across a LOT of STM32 stuff and it seemed that it was actually quite easy to set up. Lots of the projects used the STM32 Discovery and similar dev boards, which are a great tools and provide an easy introduction to ARM microcontrollers. However, my interest is more towards doing very bare metal development. Like soldering the chip to a board and hooking it up to a programmer. Who needs any of that dev board stuff? For some reason I just find doing embedded development without a development board absolutely fascinating. Some people might interpret doing things this way as a form of masochism. Perhaps I should start seeing a doctor…

Having seen how common the STM32 family was (both in dev boards and in commercial products) and noting that they were similarly priced to the Freescale Kinetis series, I looked in to exactly what I would need to program these, saw that the stuff was cheap, and bought it. After receiving my parts and soldering everything up, I plugged everything into my computer and had a program running on the STM32 in a matter of hours. Contrast that to a year spent trying to program a Kinetis KL26 with only partial success.

This post is a complete step-by-step tutorial on getting an STM32 microcontroller up and running without using a single dev board (breakout boards don’t count as dev boards for the purposes of this tutorial). I’m writing this because I could not find a standalone tutorial for doing this with an ARM microcontroller and I ended up having to piece it together bit by bit with a lot of googling. My objective is to walk you through the process of purchasing the appropriate parts, writing a small program, and uploading it to the microcontroller.

I make the following assumptions:

The reader is familiar with electronics and soldering.
The reader is familiar with flash-based microcontrollers in general (PIC, AVR, ARM, etc) and has programmed a few using a separate standalone programmer before.
The reader knows how to read a datasheet.
The reader knows C and is at least passingly familiar with the overall embedded build process of compilation-linking-flashing.
The reader knows about makefiles.
The reader is ridiculously excited about ARM microcontrollers and is strongly motivated to overlook any mistakes here and try this out for themselves (srsly tho…if you see a problem or have a suggestion, leave it in the comments. I really do appreciate feedback.)

All code, makefiles, and configuration stuff can be found in the project repository on github.

Project Repository: https://github.com/kcuzner/stm32f103c8-blink

Materials

You will require the following materials:

A computer running Linux. If you run Windows only, please don’t be dissuaded. I’m just lazy and don’t want to test this for Windows. It may require some finagling. Manufacturer support is actually better for Windows since they provide some interesting configuration and programming software that is Windows only…but who needs that stuff anyway?
A STLinkv2 Clone from eBay. Here’s one very similar to the one I bought. ~$3
Some STM32F103C8’s from eBay. Try going with the TQFP-48 package. Why this microcontroller? Because for some reason it is all over the place on eBay. I suspect that the lot I bought (and all of the ones on eBay) is probably not authentic from ST. I hear that Chinese STM32 clones abound nowadays. I got 10 for $12.80.
A breakout board for a TQFP-48 with 0.5mm pitch. Yes, you will need to solder surface mount. I found mine for $1. I’m sure you can find one for a similar price.
4x 0.1uF capacitors for decoupling. Mine are surface mount in the 0603 package. These will be soldered creatively to the breakout board between the power pins to provide some decoupling since we will probably have wires flying all over. I had mine lying around in a parts bin, left over from my development board project. Digikey is great for getting these, but I’m sure you could find them on eBay or Amazon as well.
Some dupont wires for connecting the programmer to the STM32. You will need at least 4. These are the ones that are sold for Arduinos. These came with my programmer, but you may have some in your parts box. They are dang cheap on Amazon.
Regular wires.
An LED and a resistor.

I was able to acquire all of these parts for less than $20. Now, I did have stuff like the capacitors, led, resistor, and wires lying around in parts boxes, but those are quite cheap anyway.

Side note: Here is an excellent video by the EE guru Dave Jones on surface mount soldering if the prospect is less than palatable to you: https://www.youtube.com/watch?v=b9FC9fAlfQE

Step 1: Download the datasheets

Above we decided to use the STM32F103C8 ARM Cortex-M3 microcontroller in a TQFP-48 package. This microcontroller has so many peripherals its no wonder its the one all over eBay. I could see this microcontroller easily satisfying the requirements for all of my projects. Among other things it has:

64K flash, 20K RAM
72MHz capability with an internal PLL
USB
CAN
I2C & SPI
Lots of timers
Lots of PWM
Lots of GPIO

All this for ~$1.20/part no less! Of course, its like $6 on digikey, but for my purposes having an eBay-sourced part is just fine.

Ok, so when messing with any microcontroller we need to look at its datasheet to know where to plug stuff in. For almost all ARM Microcontrollers there will be no less than 2 datasheet-like documents you will need: The part datasheet and the family reference manual. The datasheet contains information such as the specific pinouts and electrical characteristics and the family reference manual contains the detailed information on how the microcontroller works (core and peripherals). These are both extremely important and will be indispensable for doing anything at all with one of these microcontrollers bare metal.

Find the STM32F103C8 datasheet and family reference manual here (datasheet is at the top of the page, reference manual is at the bottom): http://www.st.com/en/microcontrollers/stm32f103c8.html. They are also found in the “ref” folder of the repository.

Step 2: Figure out where to solder and do it

STM32F103C8 Pins of interest

After getting the datasheet we need to solder the microcontroller down to the breakout board so that we can start working with it on a standard breadboard. If you prefer to go build your own PCB and all that (I usually do actually) then do that instead of this. However, you will still need to know which pins to hook up.

On the pin diagram posted here you will find the highlighted pins of interest for hooking this thing up. We need the following pins at a minimum:

Shown in Red/Blue: All power pins, VDD, VSS, AVDD, and AVSS. There are four pairs: 3 for the VDD/VSS and one AVDD/AVSS. The AVDD/AVSS pair is specifically used to power the analog/mixed signal circuitry and is separate to give us the opportunity to perform some additional filtering on those lines and remove supply noise induced by all the switching going on inside the microcontroller; an opportunity I won’t take for now.
Shown in Yellow/Green: The SWD (Serial Wire Debug) pins. These are used to connect to the STLinkV2 programmer that you purchased earlier. These can be used for so much more than just programming (debugging complete with breakpoints, for a start), but for now we will just use it to talk to the flash on the microcontroller.
Shown in Cyan: Two fun GPIOs to blink our LEDs with. I chose PB0 and PB1. You could choose others if you would like, but just make sure that they are actually GPIOs and not something unexpected.

Below you will find a picture of my breakout board. I soldered a couple extra pins since I want to experiment with USB.

STM32F103C8 Breakout

Very important: You may notice that I have some little tiny capacitors (0.1uF) soldered between the power pins (the one on the top is the most visible in the picture). You need to mount your capacitors between each pair of VDD/VSS pins (including AVDD/AVSS). How you do this is completely up to you, but it must be done and they should be rather close to the microcontroller itself. If you don’t it is entirely possible that when the microcontroller first turns on and powers up (specifically at the first falling edge of the internal clock cycle), the inductance created by the flying power wires we have will create a voltage spike that will either cause a malfunction or damage. I’ve broken microcontrollers by forgetting the decoupling caps and I’m not eager to do it again.

Step 3: Connect the breadboard and programmer

Cheap STLinkV2 Clone

Don’t do this with the programmer plugged in.

On the right you will see my STLinkV2 clone which I will use for this project. Barely visible is the pinout. We will need the following pins connected from the programmer onto our breadboard. These come off the header on the non-USB end of the programmer. Pinouts may vary. Double check your programmer!

3.3V: We will be using the programmer to actually power the microcontroller since that is the simplest option. I believe this pin is Pin 7 on my header.
GND: Obviously we need the ground. On mine this was Pin 4.
SWDIO: This is the data for the SWD bus. Mine has this at Pin 2.
SWCLK: This is the clock for the SWD bus. Mine has this at Pin 6.

You may notice in the above picture that I have an IDC cable coming off my programmer rather than the dupont wires. I borrowed the cable from my AVR USBASP programmer since it was more available at the time rather than finding the dupont cables that came with the STLinkV2.

Next, we need to connect the following pins on the breadboard:

STM32 [A]VSS pins 8, 23, 35, and 47 connected to ground.
STM32 [A]VDD pins 9, 24, 36, and 48 connected to 3.3V.
STM32 pin 34 to SWDIO.
STM32 pin 37 to SWCLK.
STM32 PB0 pin 18 to a resistor connected to the anode of an LED. The cathode of the LED goes to ground. Pin 19 (PB1) can also be connected in a similar fashion if you should so choose.

Here is my breadboard setup:

STM32F103C8 Breadboard Setup

Step 4: Download the STM32F1xx C headers

Project Repository: https://github.com/kcuzner/stm32f103c8-blink

Since we are going to write a program, we need the headers. These are part of the STM32CubeF1 library found here.

Visit the page and download the STM32CubeF1 zip file. It will ask for an email address. If you really don’t want to give them your email address, the necessary headers can be found in the project github repository.

Alternately, just clone the repository. You’ll miss all the fun of poking around the zip file, but sometimes doing less work is better.

The STM32CubeF1 zip file contains several components which are designed to help people get started quickly when programming STM32s. This is one thing that ST definitely does better than Freescale. It was so difficult to find the headers for the Kinetis microcontrollers that almost gave up at that point. Anyway, inside the zip file we are only interested in the following:

The contents of Drivers/CMSIS/Device/ST/STM32F1xx/Include. These headers contain the register definitions among other things which we will use in our program to reference the peripherals on the device.
Drivers/CMSIS/Device/ST/STM32F1xx/Source/Templates/gcc/startup_stm32f103xb.s. This contains the assembly code used to initialize the microcontroller immediately after reset. We could easily write this ourselves, but why reinvent the wheel?
Drivers/CMSIS/Device/ST/STM32F1xx/Source/Templates/system_stm32f1xx.c. This contains the common system startup routines referenced by the assembly file above.
Drivers/CMSIS/Device/ST/STM32F1xx/Source/Templates/gcc/linker/STM32F103XB_FLASH.ld. This is the linker script for the next model up of the microcontroller we have (we just have to change the “128K” to a “64K” near the beginning of the file in the MEMORY section (line 43 in my file) and we are good to go). This is used to tell the linker where to put all the parts of the program inside the microcontroller’s flash and RAM. Mine had a “0” on every blank line. If you see this in yours, delete those “0”s. They will cause errors.
The contents of Drivers/CMSIS/Include. These are the core header files for the ARM Cortex-M3 and the definitions contained therein are used in all the other header files we reference.

I copied all the files referenced above to various places in my project structure so they could be compiled into the final program. Please visit the repository for the exact locations and such. My objective with this tutorial isn’t really to talk too much about project structure, and so I think that’s best left as an exercise for the reader.

Step 5: Install the required software

We need to be able to compile the program and flash the resulting binary file to the microcontroller. In order to do this, we will require the following programs to be installed:

The arm-none-eabi toolchain. I use arch linux and had to install “arm-none-eabi-gcc”. On Ubuntu this is called “gcc-arm-none-eabi”. This is the cross-compiler for the ARM Cortex cores. The naming “none-eabi” comes from the fact that it is designed to compile for an environment where the program is the only thing running on the target processor. There is no underlying operating system talking to the application binary file (ABI = application binary interface, none-eabi = No ABI) in order to load it into memory and execute it. This means that it is ok with outputting raw binary executable programs. Contrast this with Linux which likes to use the ELF format (which is a part of an ABI specification) and the OS will interpret that file format and load the program from it.
arm-none-eabi binutils. In Arch the package is “arm-none-eabi-binutils”. In Ubuntu this is “binutils-arm-none-eabi”. This contains some utilities such as “objdump” and “objcopy” which we use to convert the output ELF format into the raw binary format we will use for flashing the microcontroller.
Make. We will be using a makefile, so obviously you will need make installed.
OpenOCD. I’m using 0.9.0, which I believe is available for both Arch and Ubuntu. This is the program that we will use to talk to the STLinkV2 which in turn talks to the microcontroller. While we are just going to use it to flash the microcontroller, it can be also used for debugging a program on the processor using gdb.

Once you have installed all of the above programs, you should be good to go for ARM development. As for an editor or IDE, I use vim. You can use whatever. It doesn’t matter really.

Step 6: Write and compile the program

Ok, so we need to write a program for this microcontroller. We are going to simply toggle on and off a GPIO pin (PB0). After reset, the processor uses the internal RC oscillator as its system clock and so it runs at a reasonable 8MHz or so I believe. There are a few steps that we need to go through in order to actually write to the GPIO, however:

Enable the clock to PORTB. Most ARM microcontrollers, the STM32 included, have a clock gating system that actually turns off the clock to pretty much all peripherals after system reset. This is a power saving measure as it allows parts of the microcontroller to remain dormant and not consume power until needed. So, we need to turn on the GPIO port before we can use it.
Set PB0 to a push-pull output. This microcontroller has many different options for the pins including analog input, an open-drain output, a push-pull output, and an alternate function (usually the output of a peripheral such as a timer PWM). We don’t want to run our LED open drain for now (though we certainly could), so we choose the push-pull output. Most microcontrollers have push-pull as the default method for driving their outputs.
Toggle the output state on. Once we get to this point, it’s success! We can control the GPIO by just flipping a bit in a register.
Toggle the output state off. Just like the previous step.

Here is my super-simple main program that does all of the above:

/**
 * STM32F103C8 Blink Demonstration
 *
 * Kevin Cuzner
 */

#include "stm32f1xx.h"

int main(void)
{
    //Step 1: Enable the clock to PORT B
    RCC->APB2ENR |= RCC_APB2ENR_IOPBEN;

    //Step 2: Change PB0's mode to 0x3 (output) and cfg to 0x0 (push-pull)
    GPIOB->CRL = GPIO_CRL_MODE0_0 | GPIO_CRL_MODE0_1;

    while (1)
    {
        //Step 3: Set PB0 high
        GPIOB->BSRR = GPIO_BSRR_BS0;
        for (uint16_t i = 0; i != 0xffff; i++) { }
        //Step 4: Reset PB0 low
        GPIOB->BSRR = GPIO_BSRR_BR0;
        for (uint16_t i = 0; i != 0xffff; i++) { }
    }

    return 0;
}

/**

* STM32F103C8 Blink Demonstration

* Kevin Cuzner

#include "stm32f1xx.h"

int main(void)

{

//Step 1: Enable the clock to PORT B

RCC->APB2ENR |= RCC_APB2ENR_IOPBEN;

//Step 2: Change PB0's mode to 0x3 (output) and cfg to 0x0 (push-pull)

GPIOB->CRL = GPIO_CRL_MODE0_0 | GPIO_CRL_MODE0_1;

while (1)

{

//Step 3: Set PB0 high

GPIOB->BSRR = GPIO_BSRR_BS0;

for (uint16_t i = 0; i != 0xffff; i++) { }

//Step 4: Reset PB0 low

GPIOB->BSRR = GPIO_BSRR_BR0;

for (uint16_t i = 0; i != 0xffff; i++) { }

}

return 0;

}

If we turn to our trusty family reference manual, we will see that the clock gating functionality is located in the Reset and Clock Control (RCC) module (section 7 of the manual). The gates to the various peripherals are sorted by the exact data bus they are connected to and have appropriately named registers. The PORTB module is located on the APB2 bus, and so we use the RCC->APB2ENR to turn on the clock for port B (section 7.3.7 of the manual).

The GPIO block is documented in section 9. We first talk to the low control register (CRL) which controls pins 0-7 of the 16-pin port. There are 4 bits per pin which describe the configuration grouped in to two 2-bit (see how many “2” sounding words I had there?) sections: The Mode and Configuration. The Mode sets the analog/input/output state and the Configuration handles the specifics of the particular mode. We have chosen output (Mode is 0b11) and the 50MHZ-capable output mode (Cfg is 0b00). I’m not fully sure what the 50MHz refers to yet, so I just kept it at 50MHz because that was the default value.

After talking to the CRL, we get to talk to the BSRR register. This register allows us to write a “1” to a bit in the register in order to either set or reset the pin’s output value. We start by writing to the BS0 bit to set PB0 high and then writing to the BR0 bit to reset PB0 low. Pretty straightfoward.

It’s not a complicated program. Half the battle is knowing where all the pieces fit. The STM32F1Cube zip file contains some examples which could prove quite revealing into the specifics on using the various peripherals on the device. In fact, it includes an entire hardware abstraction layer (HAL) which you could compile into your program if you wanted to. However, I have heard some bad things about it from a software engineering perspective (apparently it’s badly written and quite ugly). I’m sure it works, though.

So, the next step is to compile the program. See the makefile in the repository. Basically what we are going to do is first compile the main source file, the assembly file we pulled in from the STM32Cube library, and the C file we pulled in from the STM32Cube library. We will then link them using the linker script from the STM32Cube and then dump the output into a binary file.

# Makefile for the STM32F103C8 blink program
#
# Kevin Cuzner
#

PROJECT = blink

# Project Structure
SRCDIR = src
COMDIR = common
BINDIR = bin
OBJDIR = obj
INCDIR = include

# Project target
CPU = cortex-m3

# Sources
SRC = $(wildcard $(SRCDIR)/*.c) $(wildcard $(COMDIR)/*.c)
ASM = $(wildcard $(SRCDIR)/*.s) $(wildcard $(COMDIR)/*.s)

# Include directories
INCLUDE  = -I$(INCDIR) -Icmsis

# Linker 
LSCRIPT = STM32F103X8_FLASH.ld

# C Flags
GCFLAGS  = -Wall -fno-common -mthumb -mcpu=$(CPU) -DSTM32F103xB --specs=nosys.specs -g -Wa,-ahlms=$(addprefix $(OBJDIR)/,$(notdir $(<:.c=.lst)))
GCFLAGS += $(INCLUDE)
LDFLAGS += -T$(LSCRIPT) -mthumb -mcpu=$(CPU) --specs=nosys.specs 
ASFLAGS += -mcpu=$(CPU)

# Flashing
OCDFLAGS = -f /usr/share/openocd/scripts/interface/stlink-v2.cfg \
		   -f /usr/share/openocd/scripts/target/stm32f1x.cfg \
		   -f openocd.cfg

# Tools
CC = arm-none-eabi-gcc
AS = arm-none-eabi-as
AR = arm-none-eabi-ar
LD = arm-none-eabi-ld
OBJCOPY = arm-none-eabi-objcopy
SIZE = arm-none-eabi-size
OBJDUMP = arm-none-eabi-objdump
OCD = openocd

RM = rm -rf

## Build process

OBJ := $(addprefix $(OBJDIR)/,$(notdir $(SRC:.c=.o)))
OBJ += $(addprefix $(OBJDIR)/,$(notdir $(ASM:.s=.o)))


all:: $(BINDIR)/$(PROJECT).bin

Build: $(BINDIR)/$(PROJECT).bin

install: $(BINDIR)/$(PROJECT).bin
	$(OCD) $(OCDFLAGS)

$(BINDIR)/$(PROJECT).hex: $(BINDIR)/$(PROJECT).elf
	$(OBJCOPY) -R .stack -O ihex $(BINDIR)/$(PROJECT).elf $(BINDIR)/$(PROJECT).hex

$(BINDIR)/$(PROJECT).bin: $(BINDIR)/$(PROJECT).elf
	$(OBJCOPY) -R .stack -O binary $(BINDIR)/$(PROJECT).elf $(BINDIR)/$(PROJECT).bin

$(BINDIR)/$(PROJECT).elf: $(OBJ)
	@mkdir -p $(dir $@)
	$(CC) $(OBJ) $(LDFLAGS) -o $(BINDIR)/$(PROJECT).elf
	$(OBJDUMP) -D $(BINDIR)/$(PROJECT).elf > $(BINDIR)/$(PROJECT).lst
	$(SIZE) $(BINDIR)/$(PROJECT).elf

macros:
	$(CC) $(GCFLAGS) -dM -E - < /dev/null

cleanBuild: clean

clean:
	$(RM) $(BINDIR)
	$(RM) $(OBJDIR)

# Compilation
$(OBJDIR)/%.o: $(SRCDIR)/%.c
	@mkdir -p $(dir $@)
	$(CC) $(GCFLAGS) -c $< -o $@

$(OBJDIR)/%.o: $(SRCDIR)/%.s
	@mkdir -p $(dir $@)
	$(AS) $(ASFLAGS) -o $@ $<


$(OBJDIR)/%.o: $(COMDIR)/%.c
	@mkdir -p $(dir $@)
	$(CC) $(GCFLAGS) -c $< -o $@

$(OBJDIR)/%.o: $(COMDIR)/%.s
	@mkdir -p $(dir $@)
	$(AS) $(ASFLAGS) -o $@ $<

100

101

# Makefile for the STM32F103C8 blink program

# Kevin Cuzner

PROJECT = blink

# Project Structure

SRCDIR = src

COMDIR = common

BINDIR = bin

OBJDIR = obj

INCDIR = include

# Project target

CPU = cortex-m3

# Sources

SRC = $(wildcard $(SRCDIR)/*.c) $(wildcard $(COMDIR)/*.c)

ASM = $(wildcard $(SRCDIR)/*.s) $(wildcard $(COMDIR)/*.s)

# Include directories

INCLUDE = -I$(INCDIR) -Icmsis

# Linker

LSCRIPT = STM32F103X8_FLASH.ld

# C Flags

GCFLAGS = -Wall -fno-common -mthumb -mcpu=$(CPU) -DSTM32F103xB --specs=nosys.specs -g -Wa,-ahlms=$(addprefix $(OBJDIR)/,$(notdir $(<:.c=.lst)))

GCFLAGS += $(INCLUDE)

LDFLAGS += -T$(LSCRIPT) -mthumb -mcpu=$(CPU) --specs=nosys.specs

ASFLAGS += -mcpu=$(CPU)

# Flashing

OCDFLAGS = -f /usr/share/openocd/scripts/interface/stlink-v2.cfg \

-f /usr/share/openocd/scripts/target/stm32f1x.cfg \

-f openocd.cfg

# Tools

CC = arm-none-eabi-gcc

AS = arm-none-eabi-as

AR = arm-none-eabi-ar

LD = arm-none-eabi-ld

OBJCOPY = arm-none-eabi-objcopy

SIZE = arm-none-eabi-size

OBJDUMP = arm-none-eabi-objdump

OCD = openocd

RM = rm -rf

## Build process

OBJ := $(addprefix $(OBJDIR)/,$(notdir $(SRC:.c=.o)))

OBJ += $(addprefix $(OBJDIR)/,$(notdir $(ASM:.s=.o)))

all:: $(BINDIR)/$(PROJECT).bin

Build: $(BINDIR)/$(PROJECT).bin

install: $(BINDIR)/$(PROJECT).bin

$(OCD) $(OCDFLAGS)

$(BINDIR)/$(PROJECT).hex: $(BINDIR)/$(PROJECT).elf

$(OBJCOPY) -R .stack -O ihex $(BINDIR)/$(PROJECT).elf $(BINDIR)/$(PROJECT).hex

$(BINDIR)/$(PROJECT).bin: $(BINDIR)/$(PROJECT).elf

$(OBJCOPY) -R .stack -O binary $(BINDIR)/$(PROJECT).elf $(BINDIR)/$(PROJECT).bin

$(BINDIR)/$(PROJECT).elf: $(OBJ)

@mkdir -p $(dir $@)

$(CC) $(OBJ) $(LDFLAGS) -o $(BINDIR)/$(PROJECT).elf

$(OBJDUMP) -D $(BINDIR)/$(PROJECT).elf > $(BINDIR)/$(PROJECT).lst

$(SIZE) $(BINDIR)/$(PROJECT).elf

macros:

$(CC) $(GCFLAGS) -dM -E - < /dev/null

cleanBuild: clean

clean:

$(RM) $(BINDIR)

$(RM) $(OBJDIR)

# Compilation

$(OBJDIR)/%.o: $(SRCDIR)/%.c

@mkdir -p $(dir $@)

$(CC) $(GCFLAGS) -c $< -o $@

$(OBJDIR)/%.o: $(SRCDIR)/%.s

@mkdir -p $(dir $@)

$(AS) $(ASFLAGS) -o $@ $<

$(OBJDIR)/%.o: $(COMDIR)/%.c

@mkdir -p $(dir $@)

$(CC) $(GCFLAGS) -c $< -o $@

$(OBJDIR)/%.o: $(COMDIR)/%.s

@mkdir -p $(dir $@)

$(AS) $(ASFLAGS) -o $@ $<

The result of this makefile is that it will create a file called “bin/blink.bin” which contains our compiled program. We can then flash this to our microcontroller using openocd.

Step 7: Flashing the program to the microcontroller

Source for this step: https://github.com/rogerclarkmelbourne/Arduino_STM32/wiki/Programming-an-STM32F103XXX-with-a-generic-%22ST-Link-V2%22-programmer-from-Linux

This is the very last step. We get to do some openocd configuration. Firstly, we need to write a small configuration script that will tell openocd how to flash our program. Here it is:

# Configuration for flashing the blink program
init
reset halt
flash write_image erase bin/blink.bin 0x08000000
reset run
shutdown

# Configuration for flashing the blink program

init

reset halt

flash write_image erase bin/blink.bin 0x08000000

reset run

shutdown

Firstly, we init and halt the processor (reset halt). When the processor is first powered up, it is going to be running whatever program was previously flashed onto the microcontroller. We want to stop this execution before we overwrite the flash. Next we execute “flash write_image erase” which will first erase the flash memory (if needed) and then write our program to it. After writing the program, we then tell the processor to execute the program we just flashed (reset run) and we shutdown openocd.

Now, openocd requires knowledge of a few things. It first needs to know what programmer to use. Next, it needs to know what device is attached to the programmer. Both of these requirements must be satisfied before we can run our script above. We know that we have an stlinkv2 for a programmer and an stm32f1xx attached on the other end. It turns out that openocd actually comes with configuration files for these. On my installation these are located at “/usr/share/openocd/scripts/interface/stlink-v2.cfg” and “/usr/share/openocd/scripts/target/stm32f1x.cfg”, respectively. We can execute all three files (stlink, stm32f1, and our flashing routine (which I have named “openocd.cfg”)) with openocd as follows:

openocd -f /usr/share/openocd/scripts/interface/stlink-v2.cfg \
		   -f /usr/share/openocd/scripts/target/stm32f1x.cfg \
		   -f openocd.cfg

openocd -f /usr/share/openocd/scripts/interface/stlink-v2.cfg \

-f /usr/share/openocd/scripts/target/stm32f1x.cfg \

-f openocd.cfg

So, small sidenote: If we left off the “shutdown” command, openocd would actually continue running in “daemon” mode, listening for connections to it. If you wanted to use gdb to interact with the program running on the microcontroller, that is what you would use to do it. You would tell gdb that there is a “remote target” at port 3333 (or something like that). Openocd will be listening at that port and so when gdb starts talking to it and trying to issue debug commands, openocd will translate those through the STLinkV2 and send back the translated responses from the microcontroller. Isn’t that sick?

In the makefile earlier, I actually made this the “install” target, so running “sudo make install” will actually flash the microcontroller. Here is my output from that command for your reference:

kcuzner@kcuzner-laptop:~/Projects/ARM/stm32f103-blink$ sudo make install
arm-none-eabi-gcc -Wall -fno-common -mthumb -mcpu=cortex-m3 -DSTM32F103xB --specs=nosys.specs -g -Wa,-ahlms=obj/system_stm32f1xx.lst -Iinclude -Icmsis -c src/system_stm32f1xx.c -o obj/system_stm32f1xx.o
arm-none-eabi-gcc -Wall -fno-common -mthumb -mcpu=cortex-m3 -DSTM32F103xB --specs=nosys.specs -g -Wa,-ahlms=obj/main.lst -Iinclude -Icmsis -c src/main.c -o obj/main.o
arm-none-eabi-as -mcpu=cortex-m3 -o obj/startup_stm32f103x6.o src/startup_stm32f103x6.s
arm-none-eabi-gcc obj/system_stm32f1xx.o obj/main.o obj/startup_stm32f103x6.o -TSTM32F103X8_FLASH.ld -mthumb -mcpu=cortex-m3 --specs=nosys.specs  -o bin/blink.elf
arm-none-eabi-objdump -D bin/blink.elf > bin/blink.lst
arm-none-eabi-size bin/blink.elf
   text	   data	    bss	    dec	    hex	filename
   1756	   1092	   1564	   4412	   113c	bin/blink.elf
arm-none-eabi-objcopy -R .stack -O binary bin/blink.elf bin/blink.bin
openocd -f /usr/share/openocd/scripts/interface/stlink-v2.cfg -f /usr/share/openocd/scripts/target/stm32f1x.cfg -f openocd.cfg
Open On-Chip Debugger 0.9.0 (2016-04-27-23:18)
Licensed under GNU GPL v2
For bug reports, read
	http://openocd.org/doc/doxygen/bugs.html
Info : auto-selecting first available session transport "hla_swd". To override use 'transport select <transport>'.
Info : The selected transport took over low-level target control. The results might differ compared to plain JTAG/SWD
adapter speed: 1000 kHz
adapter_nsrst_delay: 100
none separate
Info : Unable to match requested speed 1000 kHz, using 950 kHz
Info : Unable to match requested speed 1000 kHz, using 950 kHz
Info : clock speed 950 kHz
Info : STLINK v2 JTAG v17 API v2 SWIM v4 VID 0x0483 PID 0x3748
Info : using stlink api v2
Info : Target voltage: 3.335870
Info : stm32f1x.cpu: hardware has 6 breakpoints, 4 watchpoints
target state: halted
target halted due to debug-request, current mode: Thread 
xPSR: 0x01000000 pc: 0x08000380 msp: 0x20004ffc
auto erase enabled
Info : device id = 0x20036410
Info : flash size = 64kbytes
target state: halted
target halted due to breakpoint, current mode: Thread 
xPSR: 0x61000000 pc: 0x2000003a msp: 0x20004ffc
wrote 3072 bytes from file bin/blink.bin in 0.249272s (12.035 KiB/s)
shutdown command invoked
kcuzner@kcuzner-laptop:~/Projects/ARM/stm32f103-blink$

kcuzner@kcuzner-laptop:~/Projects/ARM/stm32f103-blink$ sudo make install

arm-none-eabi-gcc -Wall -fno-common -mthumb -mcpu=cortex-m3 -DSTM32F103xB --specs=nosys.specs -g -Wa,-ahlms=obj/system_stm32f1xx.lst -Iinclude -Icmsis -c src/system_stm32f1xx.c -o obj/system_stm32f1xx.o

arm-none-eabi-gcc -Wall -fno-common -mthumb -mcpu=cortex-m3 -DSTM32F103xB --specs=nosys.specs -g -Wa,-ahlms=obj/main.lst -Iinclude -Icmsis -c src/main.c -o obj/main.o

arm-none-eabi-as -mcpu=cortex-m3 -o obj/startup_stm32f103x6.o src/startup_stm32f103x6.s

arm-none-eabi-gcc obj/system_stm32f1xx.o obj/main.o obj/startup_stm32f103x6.o -TSTM32F103X8_FLASH.ld -mthumb -mcpu=cortex-m3 --specs=nosys.specs -o bin/blink.elf

arm-none-eabi-objdump -D bin/blink.elf > bin/blink.lst

arm-none-eabi-size bin/blink.elf

text data bss dec hex filename

1756 1092 1564 4412 113c bin/blink.elf

arm-none-eabi-objcopy -R .stack -O binary bin/blink.elf bin/blink.bin

openocd -f /usr/share/openocd/scripts/interface/stlink-v2.cfg -f /usr/share/openocd/scripts/target/stm32f1x.cfg -f openocd.cfg

Open On-Chip Debugger 0.9.0 (2016-04-27-23:18)

Licensed under GNU GPL v2

For bug reports, read

http://openocd.org/doc/doxygen/bugs.html

Info : auto-selecting first available session transport "hla_swd". To override use 'transport select <transport>'.

Info : The selected transport took over low-level target control. The results might differ compared to plain JTAG/SWD

adapter speed: 1000 kHz

adapter_nsrst_delay: 100

none separate

Info : Unable to match requested speed 1000 kHz, using 950 kHz

Info : clock speed 950 kHz

Info : STLINK v2 JTAG v17 API v2 SWIM v4 VID 0x0483 PID 0x3748

Info : using stlink api v2

Info : Target voltage: 3.335870

Info : stm32f1x.cpu: hardware has 6 breakpoints, 4 watchpoints

target state: halted

target halted due to debug-request, current mode: Thread

xPSR: 0x01000000 pc: 0x08000380 msp: 0x20004ffc

auto erase enabled

Info : device id = 0x20036410

Info : flash size = 64kbytes

target state: halted

target halted due to breakpoint, current mode: Thread

xPSR: 0x61000000 pc: 0x2000003a msp: 0x20004ffc

wrote 3072 bytes from file bin/blink.bin in 0.249272s (12.035 KiB/s)

shutdown command invoked

kcuzner@kcuzner-laptop:~/Projects/ARM/stm32f103-blink$

After doing that I saw the following awesomeness:

STM32F103C8 with LED turned on

Wooo!!! The LED blinks! At this point, you have successfully flashed an ARM Cortex-M3 microcontroller with little more than a cheap programmer from eBay, a breakout board, and a few stray wires. Feel happy about yourself.

Conclusion

For me, this marks the end of one journey and the beginning of another. I can now feel free to experiment with ARM microcontrollers without having to worry about ruining a nice shiny development board. I can buy a obscenely powerful $1 STM32 microcontroller from eBay and put it into any project I want. If I were to try to do that with AVRs, I would be stuck with the ultra-low-end 8-pin ATTiny13A since that’s about it for ~$1 AVR eBay offerings (don’t worry…I’ve got plenty of ATMega328PB’s…though they weren’t $1). I sincerely hope that you found this tutorial useful and that it might serve as a springboard for doing your own dev board-free ARM development.

If you have any questions or comments (or want to let me know about any errors I may have made), let me know in the comments section here. I will try my best to help you out, although I can’t always find the time to address every issue.

Teensy 3.1 bare metal: Writing a USB driver

One of the things that has intrigued me for the past couple years is making embedded USB devices. It’s an industry standard bus that just about any piece of computing hardware can connect with yet is complex enough that doing it yourself is a bit of a chore.

Traditionally I have used the work of others, mainly the V-USB driver for AVR, to get my devices connected. Lately I have been messing around more with the ARM processor on a Teensy 3.1 which has an integrated USB module. The last microcontrollers I used that had these were the PIC18F4550s that I used in my dot matrix project. Even with those, I used microchip’s library and drivers.

Teensy 3.1

Over the thanksgiving break I started cobbling together some software with the intent of writing a driver for the USB module in the Teensy myself. I started originally with my bare metal stuff, but I ended up going with something closer to Karl Lunt’s solution. I configured code::blocks to use the arm-none-eabi compiler that I had installed and created a code blocks project for my code and used that to build it (with a post-compile event translating the generated elf file into a hex file).

This is a work in progress and the git repository will be updated as things progress since it’s not a dedicated demonstration of the USB driver.

The github repository here will be eventually turned in to a really really rudimentary 500-800ksps oscilloscope.

The code: https://github.com/kcuzner/teensy-oscilloscope

The code for this post was taken from the following commit:

https://github.com/kcuzner/teensy-oscilloscope/tree/9a5a4c9108717cfec0174709a72edeab93fcf2b8

At the end of this post, I will have outlined all of the pieces needed to have a simple USB device setup that responds with a descriptor on endpoint 0.

USB Basics

The Freescale K20 Family and their USB module

Part 1: The clocks

Part 2: The startup sequence

Part 3: The interrupt handler state machine

Part 4: Token processing & descriptors

Where to go from here

Conclusion

USB Basics

I will actually not be talking about these here as I am most definitely no expert. However, I will point to the page that I found most helpful when writing this:

http://www.usbmadesimple.co.uk/index.html

This site explained very clearly exactly what was going on with USB. Coupled with my previous knowledge, it was almost all I needed in terms of getting the protocol.

The Freescale K20 Family and their USB module

The one thing that I don’t like about all of these great microcontrollers that come out with USB support is that all of them have their very own special USB module which doesn’t work like anyone else. Sure, there are similarities, but there are no two exactly alike. Since I have a Teensy and the K20 family of microcontrollers seem to be relatively popular, I don’t feel bad about writing such specific software.

There are two documents I found to be essential to writing this driver:

The family manual. Getting a correct version for the MK20DX256VLH7 (the processor on the Teensy) can be a pain. PJRC comes to the rescue here: http://www.pjrc.com/teensy/K20P64M72SF1RM.pdf (note, the Teensies based on the MK20DX128VLH5 use a different manual)
The Kinetis Peripheral Module Quick Reference: http://cache.freescale.com/files/32bit/doc/quick_ref_guide/KQRUG.pdf. This specifies the initialization sequence and other things that will be needed for the module.

There are a few essential parts to understand about the USB module:

It needs a specific memory layout. Since it doesn’t have any dedicated user-accessible memory, it requires that the user specify where things should be. There are specific valid locations for its Buffer Descriptor Table (more on that later) and the endpoint buffers. The last one bit me for several days until I figured it out.
It has several different clock inputs and all of them must be enabled. Identifying the different signals is the most difficult part. After that, its not hard.
The module only handles the electrical aspect of things. It doesn’t handle sending descriptors or anything like that. The only real things it handles are the signaling levels, responding to USB packets in a valid manner, and routing data into buffers by endpoint. Other than that, its all user software.
The module can act as both a host (USB On-the-go (OTG)) and a device. We will be exclusively focusing on using it as a device here.

In writing this, I must confess that I looked quite a lot at the Teensyduino code along with the V-USB driver code (even though V-USB is for AVR and is pure software). Without these “references”, this would have been a very difficult project. Much of the structure found in the last to parts of this document reflects the Teensyduino USB driver since they did it quite efficiently and I didn’t spend a lot of time coming up with a “better” way to do it, given the scope of this project. I will likely make more changes as I customize it for my end use-case.

Part 1: The clocks

The K20 family of microcontrollers utilizes a miraculous hardware module which they call the “Multipurpose Clock Generator” (hereafter called the MCG). This is a module which basically allows the microcontroller to take any clock input between a few kilohertz and several megahertz and transform it into a higher frequency clock source that the microcontroller can actually use. This is how the Teensy can have a rated speed of 96Mhz but only use a 16Mhz crystal. The configuration that this project uses is the Phase Locked Loop (PLL) from the high speed crystal source. The exact setup of this configuration is done by the sysinit code.

The PLL operates by using a divider-multiplier setup where we give it a divisor to divide the input clock frequency by and then a multiplier to multiply that result by to give us the final clock speed. After that, it heads into the System Integration Module (SIM) which distributes the clock. Since the Teensy uses a 16Mhz crystal and we need a 96Mhz system clock (the reason will become apparent shortly), we set our divisor to 4 and our multiplier to 24 (see common.h). If the other type of Teensy 3 is being used (the one with the MK20DX128VLH5), the divisor would be 8 and the multiplier 36 to give us 72Mhz.

Every module on a K20 microcontroller has a gate on its clock. This saves power since there are many modules on the microcontroller that are not being used in any given application. Distributing the clock to each of these is expensive in terms of power and would be wasted if that module wasn’t used. The SIM handles this gating in the SIM_SCGC* registers. Before using any module, its clock gate must be enabled. If this is not done, the microcontroller will “crash” and stop executing when it tries to talk to the module registers (I think a handler for this can be specified, but I’m not sure). I had this happen once or twice while messing with this. So, the first step is to “turn on” the USB module by setting the appropriate bit in SIM_SCGC4 (per the family manual mentioned above, page 252):

SIM_SCGC4 |= SIM_SCGC4_USBOTG_MASK;

1	SIM_SCGC4 \|= SIM_SCGC4_USBOTG_MASK;

Now, the USB module is a bit different than the other modules. In addition to the module clock it needs a reference clock for USB. The USB module requires that this reference clock be at 48Mhz. There are two sources for this clock: an internal source generated by the MCG/SIM or an external source from a pin. We will use the internal source:

SIM_SOPT2 |= SIM_SOPT2_USBSRC_MASK | SIM_SOPT2_PLLFLLSEL_MASK;
SIM_CLKDIV2 = SIM_CLKDIV2_USBDIV(1);

1 2	SIM_SOPT2 \|= SIM_SOPT2_USBSRC_MASK \| SIM_SOPT2_PLLFLLSEL_MASK; SIM_CLKDIV2 = SIM_CLKDIV2_USBDIV(1);

The first line here selects that the USB reference clock will come from an internal source. It also specifies that the internal source will be using the output from the PLL in the MCG (the other option is the FLL (frequency lock loop), which we are not using). The second line sets the divider needed to give us 48Mhz from the PLL clock. Once again there are two values: The divider and the multiplier. The multiplier can only be 1 or 2 and the divider can be anywhere from 1 to 16. Since we have a 96Mhz clock, we simply divide by 2 (the value passed is a 1 since 0 = “divide by 1”, 1 = “divide by 2”, etc). If we were using the 72Mhz clock, we would first multiply by 2 before dividing by 3.

With that, the clock to the USB module has been activated and the module can now be initialized.

Part 2: The startup sequence

The Peripheral Module Quick Reference guide mentioned earlier contains a flowchart which outlines the exact sequence needed to initialize the USB module to act as a device. I don’t know if I can copy it here (yay copyright!), but it can be found on page 134, figure 15-6. There is another flowchart specifying the initialization sequence for using the module as a host.

Our startup sequence goes as follows:

//1: Select clock source
SIM_SOPT2 |= SIM_SOPT2_USBSRC_MASK | SIM_SOPT2_PLLFLLSEL_MASK; //we use MCGPLLCLK divided by USB fractional divider
SIM_CLKDIV2 = SIM_CLKDIV2_USBDIV(1); //(USBFRAC + 0)/(USBDIV + 1) = (1 + 0)/(1 + 1) = 1/2 for 96Mhz clock

//2: Gate USB clock
SIM_SCGC4 |= SIM_SCGC4_USBOTG_MASK;

//3: Software USB module reset
USB0_USBTRC0 |= USB_USBTRC0_USBRESET_MASK;
while (USB0_USBTRC0 & USB_USBTRC0_USBRESET_MASK);

//4: Set BDT base registers
USB0_BDTPAGE1 = ((uint32_t)table) >> 8;  //bits 15-9
USB0_BDTPAGE2 = ((uint32_t)table) >> 16; //bits 23-16
USB0_BDTPAGE3 = ((uint32_t)table) >> 24; //bits 31-24

//5: Clear all ISR flags and enable weak pull downs
USB0_ISTAT = 0xFF;
USB0_ERRSTAT = 0xFF;
USB0_OTGISTAT = 0xFF;
USB0_USBTRC0 |= 0x40; //a hint was given that this is an undocumented interrupt bit

//6: Enable USB reset interrupt
USB0_CTL = USB_CTL_USBENSOFEN_MASK;
USB0_USBCTRL = 0;

USB0_INTEN |= USB_INTEN_USBRSTEN_MASK;
//NVIC_SET_PRIORITY(IRQ(INT_USB0), 112);
enable_irq(IRQ(INT_USB0));

//7: Enable pull-up resistor on D+ (Full speed, 12Mbit/s)
USB0_CONTROL = USB_CONTROL_DPPULLUPNONOTG_MASK;

//1: Select clock source

SIM_SOPT2 |= SIM_SOPT2_USBSRC_MASK | SIM_SOPT2_PLLFLLSEL_MASK; //we use MCGPLLCLK divided by USB fractional divider

SIM_CLKDIV2 = SIM_CLKDIV2_USBDIV(1); //(USBFRAC + 0)/(USBDIV + 1) = (1 + 0)/(1 + 1) = 1/2 for 96Mhz clock

//2: Gate USB clock

SIM_SCGC4 |= SIM_SCGC4_USBOTG_MASK;

//3: Software USB module reset

USB0_USBTRC0 |= USB_USBTRC0_USBRESET_MASK;

while (USB0_USBTRC0 & USB_USBTRC0_USBRESET_MASK);

//4: Set BDT base registers

USB0_BDTPAGE1 = ((uint32_t)table) >> 8; //bits 15-9

USB0_BDTPAGE2 = ((uint32_t)table) >> 16; //bits 23-16

USB0_BDTPAGE3 = ((uint32_t)table) >> 24; //bits 31-24

//5: Clear all ISR flags and enable weak pull downs

USB0_ISTAT = 0xFF;

USB0_ERRSTAT = 0xFF;

USB0_OTGISTAT = 0xFF;

USB0_USBTRC0 |= 0x40; //a hint was given that this is an undocumented interrupt bit

//6: Enable USB reset interrupt

USB0_CTL = USB_CTL_USBENSOFEN_MASK;

USB0_USBCTRL = 0;

USB0_INTEN |= USB_INTEN_USBRSTEN_MASK;

//NVIC_SET_PRIORITY(IRQ(INT_USB0), 112);

enable_irq(IRQ(INT_USB0));

//7: Enable pull-up resistor on D+ (Full speed, 12Mbit/s)

USB0_CONTROL = USB_CONTROL_DPPULLUPNONOTG_MASK;

The first two steps were covered in the last section. The next one is relatively straightfoward: We ask the module to perform a “reset” on itself. This places the module to its initial state which allows us to configure it as needed. I don’t know if the while loop is necessary since the manual says that the reset bit always reads low and it only says we must “wait two USB clock cycles”. In any case, enough of a wait seems to be executed by the above code to allow it to reset properly.

The next section (4: Set BDT base registers) requires some explanation. Since the USB module doesn’t have a dedicated memory block, we have to provide it. The BDT is the “Buffer Descriptor Table” and contains 16 * 4 entries that look like so:

typedef struct {
    uint32_t desc;
    void* addr;
} bdt_t;

typedef struct {

uint32_t desc;

void* addr;

} bdt_t;

“desc” is a descriptor for the buffer and “addr” is the address of the buffer. The exact bits of the “desc” are explained in the manual (p. 971, Table 41-4), but they basically specify ownership of the buffer (user program or USB module) and the USB token that generated the data in the buffer (if applicable).

Each entry in the BDT corresponds to one of 4 buffers in one of the 16 USB endpoints: The RX even, RX odd, TX even, and TX odd. The RX and TX are pretty self explanatory…the module needs somewhere to read the data its going to send and somewhere to write the data it just received. The even and odd are a configuration that I have seen before in the PIC 18F4550 USB module: Ping-pong buffers. While one buffer is being sent/received by the module, the other can be in use by user code reading/writing (ping). When the user code is done with its buffers, it swaps buffers, giving the usb module control over the ones it was just using (pong). This allows seamless communication between the host and the device and minimizes the need for copying data between buffers. I have declared the BDT in my code as follows:

#define BDT_INDEX(endpoint, tx, odd) ((endpoint << 2) | (tx << 1) | odd)
__attribute__ ((section(".usbdescriptortable"), used))
static bdt_t table[(USB_N_ENDPOINTS + 1)*4]; //max endpoints is 15 + 1 control

#define BDT_INDEX(endpoint, tx, odd) ((endpoint << 2) | (tx << 1) | odd)

__attribute__ ((section(".usbdescriptortable"), used))

static bdt_t table[(USB_N_ENDPOINTS + 1)*4]; //max endpoints is 15 + 1 control

One caveat of the BDT is that it must be aligned with a 512-byte boundary in memory. Our code above showed that only 3 bytes of the 4 byte address of “table” are passed to the module. This is because the last byte is basically the index along the table (the specification of this is found in section 41.4.3, page 970 of the manual). The #define directly above the declaration is a helper macro for referencing entries in the table for specific endpoints (this is used later in the interrupt). Now, accomplishing this boundary alignment requires some modification of the linker script. Before this, I had never had any need to modify a linker script. We basically need to create a special area of memory (in the above, it is called “.usbdescriptortable” and the attribute declaration tells the compiler to place that variable’s reference inside of it) which is aligned to a 512-byte boundary in RAM. I declared mine like so:

.usbdescriptortable (NOLOAD) : {
	. = ALIGN(512);
	*(.usbdescriptortable*)
} > sram

.usbdescriptortable (NOLOAD) : {

. = ALIGN(512);

*(.usbdescriptortable*)

} > sram

The position of this in the file is mildly important, so looking at the full linker script would probably be good. This particular declaration I more or less lifted from the Teensyduino linker script, with some changes to make it fit into my linker script.

Steps 5-6 set up the interrupts. There is only one USB interrupt, but there are two registers of flags. We first reset all of the flags. Interestingly, to reset a flag we write back a ‘1’ to the particular flag bit. This has the effect of being able to set a flag register to itself to reset all of the flags since a flag bit is ‘1’ when it is triggered. After resetting the flags, we enable the interrupt in the NVIC (Nested Vector Interrupt Controller). I won’t discuss the NVIC much, but it is a fairly complex piece of hardware. It has support for lots and lots of interrupts (over 100) and separate priorities for each one. I don’t have reliable code for setting interrupt priorities yet, but eventually I’ll get around to messing with that. The “enable_irq()” call is a function that is provided in arm_cm4.c and all that it does is enable the interrupt specified by the passed vector number. These numbers are specified in the datasheet, but we have a #define specified in the mk20d7 header file (warning! 12000 lines ahead) which gives us the number.

The very last step in initialization is to set the internal pullup on D+. According to the USB specification, a pullup on D- specifies a low speed device (1.2Mbit/s) and a pullup on D+ specifies a full speed device (12Mbit/s). We want to use the higher speed grade. The Kinetis USB module does not support high speed (480Mbit/s) mode.

Part 3: The interrupt handler state machine

The USB protocol can be interpreted in the context of a state machine with each call to the interrupt being a “tick” in the machine. The interrupt handler must process all of the flags to determine what happened and where to go from there.

#define ENDP0_SIZE 64

/**
 * Endpoint 0 receive buffers (2x64 bytes)
 */
static uint8_t endp0_rx[2][ENDP0_SIZE];

//flags for endpoint 0 transmit buffers
static uint8_t endp0_odd, endp0_data = 0;

/**
 * Handler functions for when a token completes
 * TODO: Determine if this structure really will work for all kinds of handlers
 *
 * I hope this looks like a dynamic jump table to the compiler
 */
static void (*handlers[USB_N_ENDPOINTS + 2]) (uint8_t);

void USBOTG_IRQHandler(void)
{
    uint8_t status;
    uint8_t stat, endpoint;

    status = USB0_ISTAT;

    if (status & USB_ISTAT_USBRST_MASK)
    {
        //handle USB reset

        //initialize endpoint 0 ping-pong buffers
        USB0_CTL |= USB_CTL_ODDRST_MASK;
        endp0_odd = 0;
        table[BDT_INDEX(0, RX, EVEN)].desc = BDT_DESC(ENDP0_SIZE, 0);
        table[BDT_INDEX(0, RX, EVEN)].addr = endp0_rx[0];
        table[BDT_INDEX(0, RX, ODD)].desc = BDT_DESC(ENDP0_SIZE, 0);
        table[BDT_INDEX(0, RX, ODD)].addr = endp0_rx[1];
        table[BDT_INDEX(0, TX, EVEN)].desc = 0;
        table[BDT_INDEX(0, TX, ODD)].desc = 0;

        //initialize endpoint0 to 0x0d (41.5.23)
        //transmit, recieve, and handshake
        USB0_ENDPT0 = USB_ENDPT_EPRXEN_MASK | USB_ENDPT_EPTXEN_MASK | USB_ENDPT_EPHSHK_MASK;

        //clear all interrupts...this is a reset
        USB0_ERRSTAT = 0xff;
        USB0_ISTAT = 0xff;

        //after reset, we are address 0, per USB spec
        USB0_ADDR = 0;

        //all necessary interrupts are now active
        USB0_ERREN = 0xFF;
        USB0_INTEN = USB_INTEN_USBRSTEN_MASK | USB_INTEN_ERROREN_MASK |
            USB_INTEN_SOFTOKEN_MASK | USB_INTEN_TOKDNEEN_MASK |
            USB_INTEN_SLEEPEN_MASK | USB_INTEN_STALLEN_MASK;

        return;
    }
    if (status & USB_ISTAT_ERROR_MASK)
    {
        //handle error
        USB0_ERRSTAT = USB0_ERRSTAT;
        USB0_ISTAT = USB_ISTAT_ERROR_MASK;
    }
    if (status & USB_ISTAT_SOFTOK_MASK)
    {
        //handle start of frame token
        USB0_ISTAT = USB_ISTAT_SOFTOK_MASK;
    }
    if (status & USB_ISTAT_TOKDNE_MASK)
    {
        //handle completion of current token being processed
        stat = USB0_STAT;
        endpoint = stat >> 4;
        handlers[endpoint](stat);

        USB0_ISTAT = USB_ISTAT_TOKDNE_MASK;
    }
    if (status & USB_ISTAT_SLEEP_MASK)
    {
        //handle USB sleep
        USB0_ISTAT = USB_ISTAT_SLEEP_MASK;
    }
    if (status & USB_ISTAT_STALL_MASK)
    {
        //handle usb stall
        USB0_ISTAT = USB_ISTAT_STALL_MASK;
    }
}

#define ENDP0_SIZE 64

/**

* Endpoint 0 receive buffers (2x64 bytes)

static uint8_t endp0_rx[2][ENDP0_SIZE];

//flags for endpoint 0 transmit buffers

static uint8_t endp0_odd, endp0_data = 0;

/**

* Handler functions for when a token completes

* TODO: Determine if this structure really will work for all kinds of handlers

* I hope this looks like a dynamic jump table to the compiler

static void (*handlers[USB_N_ENDPOINTS + 2]) (uint8_t);

void USBOTG_IRQHandler(void)

{

uint8_t status;

uint8_t stat, endpoint;

status = USB0_ISTAT;

if (status & USB_ISTAT_USBRST_MASK)

{

//handle USB reset

//initialize endpoint 0 ping-pong buffers

USB0_CTL |= USB_CTL_ODDRST_MASK;

endp0_odd = 0;

table[BDT_INDEX(0, RX, EVEN)].desc = BDT_DESC(ENDP0_SIZE, 0);

table[BDT_INDEX(0, RX, EVEN)].addr = endp0_rx[0];

table[BDT_INDEX(0, RX, ODD)].desc = BDT_DESC(ENDP0_SIZE, 0);

table[BDT_INDEX(0, RX, ODD)].addr = endp0_rx[1];

table[BDT_INDEX(0, TX, EVEN)].desc = 0;

table[BDT_INDEX(0, TX, ODD)].desc = 0;

//initialize endpoint0 to 0x0d (41.5.23)

//transmit, recieve, and handshake

USB0_ENDPT0 = USB_ENDPT_EPRXEN_MASK | USB_ENDPT_EPTXEN_MASK | USB_ENDPT_EPHSHK_MASK;

//clear all interrupts...this is a reset

USB0_ERRSTAT = 0xff;

USB0_ISTAT = 0xff;

//after reset, we are address 0, per USB spec

USB0_ADDR = 0;

//all necessary interrupts are now active

USB0_ERREN = 0xFF;

USB0_INTEN = USB_INTEN_USBRSTEN_MASK | USB_INTEN_ERROREN_MASK |

USB_INTEN_SOFTOKEN_MASK | USB_INTEN_TOKDNEEN_MASK |

USB_INTEN_SLEEPEN_MASK | USB_INTEN_STALLEN_MASK;

return;

}

if (status & USB_ISTAT_ERROR_MASK)

{

//handle error

USB0_ERRSTAT = USB0_ERRSTAT;

USB0_ISTAT = USB_ISTAT_ERROR_MASK;

}

if (status & USB_ISTAT_SOFTOK_MASK)

{

//handle start of frame token

USB0_ISTAT = USB_ISTAT_SOFTOK_MASK;

}

if (status & USB_ISTAT_TOKDNE_MASK)

{

//handle completion of current token being processed

stat = USB0_STAT;

endpoint = stat >> 4;

handlers[endpoint](stat);

USB0_ISTAT = USB_ISTAT_TOKDNE_MASK;

}

if (status & USB_ISTAT_SLEEP_MASK)

{

//handle USB sleep

USB0_ISTAT = USB_ISTAT_SLEEP_MASK;

}

if (status & USB_ISTAT_STALL_MASK)

{

//handle usb stall

USB0_ISTAT = USB_ISTAT_STALL_MASK;

}

The above code will be executed whenever the IRQ for the USB module fires. This function is set up in the crt0.S file, but with a weak reference, allowing us to override it easily by simply defining a function called USBOTG_IRQHandler. We then proceed to handle all of the USB interrupt flags. If we don’t handle all of the flags, the interrupt will execute again, giving us the opportunity to fully process all of them.

Reading through the code is should be obvious that I have not done much with many of the flags, including USB sleep, errors, and stall. For the purposes of this super simple driver, we really only care about USB resets and USB token decoding.

The very first interrupt that we care about which will be called when we connect the USB device to a host is the Reset. The host performs this by bringing both data lines low for a certain period of time (read the USB basics stuff for more information). When we do this, we need to reset our USB state into its initial and ready state. We do a couple things in sequence:

Initialize the buffers for endpoint 0. We set the RX buffers to point to some static variables we have defined which are simply uint8_t arrays of length “ENDP0_SIZE”. The TX buffers are reset to null since nothing is going to be transmitted. One thing to note is that the ODDRST bit is flipped on in the USB0_CTL register. This is very important since it “syncronizes” the USB module with our code in terms of knowing whether the even or odd buffer should be used next for transmitting. When we do ODDRST, it sets the next buffer to be used to be the even buffer. We have a “user-space” flag (endp0_odd) which we reset at the same time so that we stay in sync with the buffer that the USB module is going to use.
We enable endpoint 0. Specifically, we say that it can transmit, receive, and handshake. Enabled endpoints always handshake, but endpoints can either send, receive, or both. Endpoint 0 is specified as a reading and writing endpoint in the USB specification. All of the other endpoints are device-specific.
We clear all of the interrupts. If this is a reset we obviously won’t be doing much else.
Set our USB address to 0. Each device on the USB bus gets an address between 0 and 127. Endpoint 0 is reserved for devices that haven’t been assigned an address yet (i.e. have been reset), so that becomes our address. We will receive an address later via a command sent to endpoint 0.
Activate all necessary interrupts. In the previous part where we discussed the initialization sequence we only enabled the reset interrupt. After being reset, we get to enable all of the interrupts that we will need to be able to process USB events.

After a reset the USB module will begin decoding tokens. While there are a couple different types of tokens, the USB module has a single interrupt for all of them. When a token is decoded the module gives us information about what endpoint the token was for and what BDT entry should be used. This information is contained in the USB0_STAT register.

The exact method for processing these tokens is up to the individual developer. My choice for the moment was to make a dynamic jump table of sorts which stores 16 function pointers which will be called in order to process the tokens. Initially, these pointers point to dummy functions that do nothing. The code for the endpoint 0 handler will be discussed in the next section.

Our code here uses USB0_STAT to determine which endpoint the token was decoded for, finds the appropriate function pointer, and calls it with the value of USB0_STAT.

Part 4: Token processing & descriptors

This is one part of the driver that isn’t something that must be done a certain way, but however it is done, it must accomplish the task correctly. My super-simple driver processes this in two stages: Processing the token type and processing the token itself.

As mentioned in the previous section, I had a handler for each endpoint that would be called after a token was decoded. The handler for endpoint 0 is as follows:

#define PID_OUT   0x1
#define PID_IN    0x9
#define PID_SOF   0x5
#define PID_SETUP 0xd

typedef struct {
    union {
        struct {
            uint8_t bmRequestType;
            uint8_t bRequest;
        };
        uint16_t wRequestAndType;
    };
    uint16_t wValue;
    uint16_t wIndex;
    uint16_t wLength;
} setup_t;

/**
 * Endpoint 0 handler
 */
static void usb_endp0_handler(uint8_t stat)
{
    static setup_t last_setup;

    //determine which bdt we are looking at here
    bdt_t* bdt = &table[BDT_INDEX(0, (stat & USB_STAT_TX_MASK) >> USB_STAT_TX_SHIFT, (stat & USB_STAT_ODD_MASK) >> USB_STAT_ODD_SHIFT)];

    switch (BDT_PID(bdt->desc))
    {
    case PID_SETUP:
        //extract the setup token
        last_setup = *((setup_t*)(bdt->addr));

        //we are now done with the buffer
        bdt->desc = BDT_DESC(ENDP0_SIZE, 1);

        //clear any pending IN stuff
        table[BDT_INDEX(0, TX, EVEN)].desc = 0;
        table[BDT_INDEX(0, TX, ODD)].desc = 0;
        endp0_data = 1;

        //run the setup
        usb_endp0_handle_setup(&last_setup);

        //unfreeze this endpoint
        USB0_CTL = USB_CTL_USBENSOFEN_MASK;
        break;
    case PID_IN:
        if (last_setup.wRequestAndType == 0x0500)
        {
            USB0_ADDR = last_setup.wValue;
        }
        break;
    case PID_OUT:
        //nothing to do here..just give the buffer back
        bdt->desc = BDT_DESC(ENDP0_SIZE, 1);
        break;
    case PID_SOF:
        break;
    }

    USB0_CTL = USB_CTL_USBENSOFEN_MASK;
}

#define PID_OUT 0x1

#define PID_IN 0x9

#define PID_SOF 0x5

#define PID_SETUP 0xd

typedef struct {

union {

struct {

uint8_t bmRequestType;

uint8_t bRequest;

};

uint16_t wRequestAndType;

};

uint16_t wValue;

uint16_t wIndex;

uint16_t wLength;

} setup_t;

/**

* Endpoint 0 handler

static void usb_endp0_handler(uint8_t stat)

{

static setup_t last_setup;

//determine which bdt we are looking at here

bdt_t* bdt = &table[BDT_INDEX(0, (stat & USB_STAT_TX_MASK) >> USB_STAT_TX_SHIFT, (stat & USB_STAT_ODD_MASK) >> USB_STAT_ODD_SHIFT)];

switch (BDT_PID(bdt->desc))

{

case PID_SETUP:

//extract the setup token

last_setup = *((setup_t*)(bdt->addr));

//we are now done with the buffer

bdt->desc = BDT_DESC(ENDP0_SIZE, 1);

//clear any pending IN stuff

table[BDT_INDEX(0, TX, EVEN)].desc = 0;

table[BDT_INDEX(0, TX, ODD)].desc = 0;

endp0_data = 1;

//run the setup

usb_endp0_handle_setup(&last_setup);

//unfreeze this endpoint

USB0_CTL = USB_CTL_USBENSOFEN_MASK;

break;

case PID_IN:

if (last_setup.wRequestAndType == 0x0500)

{

USB0_ADDR = last_setup.wValue;

}

break;

case PID_OUT:

//nothing to do here..just give the buffer back

bdt->desc = BDT_DESC(ENDP0_SIZE, 1);

break;

case PID_SOF:

break;

}

USB0_CTL = USB_CTL_USBENSOFEN_MASK;

}

The very first step in handling a token is determining the buffer which contains the data for the token transmitted. This is done by the first statement which finds the appropriate address for the buffer in the table using the BDT_INDEX macro which simply implements the addressing form found in Figure 41-3 in the family manual.

After determining where the data received is located, we need to determine which token exactly was decoded. We only do things with four of the tokens. Right now, if a token comes through that we don’t understand, we don’t really do anything. My thought is that I should be initiating an endpoint stall, but I haven’t seen anywhere that specifies what exactly I should do for an unrecognized token.

The main token that we care about with endpoint 0 is the SETUP token. The data attached to this token will be in the format described by setup_t, so the first step is that we dereference and cast the buffer into which the data was loaded into a setup_t. This token will be stored statically since we need to look at it again for tokens that follow, especially in the case of the IN token following the request to be assigned an address.

One part of processing a setup token that tripped me up for a while was what the next DATA state should be. The USB standard specifies that the data in a frame is either marked DATA0 or DATA1 and it alternates by frame. This information is stored in a flag that the USB module will read from the first 4 bytes of the BDT (the “desc” field). Immediately following a SETUP token, the next DATA transmitted must be a DATA1.

After this, the setup function is run (more on that next) and as a final step, the USB module is “unfrozen”. Whenever a token is being processed, the USB module “freezes” so that processing can occur. While I haven’t yet read enough documentation on the subject, it seems to me that this is to give the user program some time to actually handle a token before the USB module decodes another one. I’m not sure what happens if the user program takes to long, but I imagine some error flag will go off.

The guts of handling a SETUP request are as follows:

typedef struct {
    uint8_t bLength;
    uint8_t bDescriptorType;
    uint16_t bcdUSB;
    uint8_t bDeviceClass;
    uint8_t bDeviceSubClass;
    uint8_t bDeviceProtocol;
    uint8_t bMaxPacketSize0;
    uint16_t idVendor;
    uint16_t idProduct;
    uint16_t bcdDevice;
    uint8_t iManufacturer;
    uint8_t iProduct;
    uint8_t iSerialNumber;
    uint8_t bNumConfigurations;
} dev_descriptor_t;

typedef struct {
    uint8_t bLength;
    uint8_t bDescriptorType;
    uint8_t bInterfaceNumber;
    uint8_t bAlternateSetting;
    uint8_t bNumEndpoints;
    uint8_t bInterfaceClass;
    uint8_t bInterfaceSubClass;
    uint8_t bInterfaceProtocol;
    uint8_t iInterface;
} int_descriptor_t;

typedef struct {
    uint8_t bLength;
    uint8_t bDescriptorType;
    uint16_t wTotalLength;
    uint8_t bNumInterfaces;
    uint8_t bConfigurationValue;
    uint8_t iConfiguration;
    uint8_t bmAttributes;
    uint8_t bMaxPower;
    int_descriptor_t interfaces[];
} cfg_descriptor_t;

typedef struct {
    uint16_t wValue;
    uint16_t wIndex;
    const void* addr;
    uint8_t length;
} descriptor_entry_t;

/**
 * Device descriptor
 * NOTE: This cannot be const because without additional attributes, it will
 * not be placed in a part of memory that the usb subsystem can access. I
 * have a suspicion that this location is somewhere in flash, but not copied
 * to RAM.
 */
static dev_descriptor_t dev_descriptor = {
    .bLength = 18,
    .bDescriptorType = 1,
    .bcdUSB = 0x0200,
    .bDeviceClass = 0xff,
    .bDeviceSubClass = 0x0,
    .bDeviceProtocol = 0x0,
    .bMaxPacketSize0 = ENDP0_SIZE,
    .idVendor = 0x16c0, //VOTI VID/PID for use with libusb
    .idProduct = 0x05dc,
    .bcdDevice = 0x0001,
    .iManufacturer = 0,
    .iProduct = 0,
    .iSerialNumber = 0,
    .bNumConfigurations = 1
};

/**
 * Configuration descriptor
 * NOTE: Same thing about const applies here
 */
static cfg_descriptor_t cfg_descriptor = {
    .bLength = 9,
    .bDescriptorType = 2,
    .wTotalLength = 18,
    .bNumInterfaces = 1,
    .bConfigurationValue = 1,
    .iConfiguration = 0,
    .bmAttributes = 0x80,
    .bMaxPower = 250,
    .interfaces = {
        {
            .bLength = 9,
            .bDescriptorType = 4,
            .bInterfaceNumber = 0,
            .bAlternateSetting = 0,
            .bNumEndpoints = 0,
            .bInterfaceClass = 0xff,
            .bInterfaceSubClass = 0x0,
            .bInterfaceProtocol = 0x0,
            .iInterface = 0
        }
    }
};

static const descriptor_entry_t descriptors[] = {
    { 0x0100, 0x0000, &dev_descriptor, sizeof(dev_descriptor) },
    { 0x0200, 0x0000, &cfg_descriptor, 18 },
    { 0x0000, 0x0000, NULL, 0 }
};

static void usb_endp0_transmit(const void* data, uint8_t length)
{
    table[BDT_INDEX(0, TX, endp0_odd)].addr = (void *)data;
    table[BDT_INDEX(0, TX, endp0_odd)].desc = BDT_DESC(length, endp0_data);
    //toggle the odd and data bits
    endp0_odd ^= 1;
    endp0_data ^= 1;
}

/**
 * Endpoint 0 setup handler
 */
static void usb_endp0_handle_setup(setup_t* packet)
{
    const descriptor_entry_t* entry;
    const uint8_t* data = NULL;
    uint8_t data_length = 0;


    switch(packet->wRequestAndType)
    {
    case 0x0500: //set address (wait for IN packet)
        break;
    case 0x0900: //set configuration
        //we only have one configuration at this time
        break;
    case 0x0680: //get descriptor
    case 0x0681:
        for (entry = descriptors; 1; entry++)
        {
            if (entry->addr == NULL)
                break;

            if (packet->wValue == entry->wValue && packet->wIndex == entry->wIndex)
            {
                //this is the descriptor to send
                data = entry->addr;
                data_length = entry->length;
                goto send;
            }
        }
        goto stall;
        break;
    default:
        goto stall;
    }

    //if we are sent here, we need to send some data
    send:
        if (data_length > packet->wLength)
            data_length = packet->wLength;
        usb_endp0_transmit(data, data_length);
        return;

    //if we make it here, we are not able to send data and have stalled
    stall:
        USB0_ENDPT0 = USB_ENDPT_EPSTALL_MASK | USB_ENDPT_EPRXEN_MASK | USB_ENDPT_EPTXEN_MASK | USB_ENDPT_EPHSHK_MASK;
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

typedef struct {

uint8_t bLength;

uint8_t bDescriptorType;

uint16_t bcdUSB;

uint8_t bDeviceClass;

uint8_t bDeviceSubClass;

uint8_t bDeviceProtocol;

uint8_t bMaxPacketSize0;

uint16_t idVendor;

uint16_t idProduct;

uint16_t bcdDevice;

uint8_t iManufacturer;

uint8_t iProduct;

uint8_t iSerialNumber;

uint8_t bNumConfigurations;

} dev_descriptor_t;

typedef struct {

uint8_t bLength;

uint8_t bDescriptorType;

uint8_t bInterfaceNumber;

uint8_t bAlternateSetting;

uint8_t bNumEndpoints;

uint8_t bInterfaceClass;

uint8_t bInterfaceSubClass;

uint8_t bInterfaceProtocol;

uint8_t iInterface;

} int_descriptor_t;

typedef struct {

uint8_t bLength;

uint8_t bDescriptorType;

uint16_t wTotalLength;

uint8_t bNumInterfaces;

uint8_t bConfigurationValue;

uint8_t iConfiguration;

uint8_t bmAttributes;

uint8_t bMaxPower;

int_descriptor_t interfaces[];

} cfg_descriptor_t;

typedef struct {

uint16_t wValue;

uint16_t wIndex;

const void* addr;

uint8_t length;

} descriptor_entry_t;

/**

* Device descriptor

* NOTE: This cannot be const because without additional attributes, it will

* not be placed in a part of memory that the usb subsystem can access. I

* have a suspicion that this location is somewhere in flash, but not copied

* to RAM.

static dev_descriptor_t dev_descriptor = {

.bLength = 18,

.bDescriptorType = 1,

.bcdUSB = 0x0200,

.bDeviceClass = 0xff,

.bDeviceSubClass = 0x0,

.bDeviceProtocol = 0x0,

.bMaxPacketSize0 = ENDP0_SIZE,

.idVendor = 0x16c0, //VOTI VID/PID for use with libusb

.idProduct = 0x05dc,

.bcdDevice = 0x0001,

.iManufacturer = 0,

.iProduct = 0,

.iSerialNumber = 0,

.bNumConfigurations = 1

};

/**

* Configuration descriptor

* NOTE: Same thing about const applies here

static cfg_descriptor_t cfg_descriptor = {

.bLength = 9,

.bDescriptorType = 2,

.wTotalLength = 18,

.bNumInterfaces = 1,

.bConfigurationValue = 1,

.iConfiguration = 0,

.bmAttributes = 0x80,

.bMaxPower = 250,

.interfaces = {

{

.bLength = 9,

.bDescriptorType = 4,

.bInterfaceNumber = 0,

.bAlternateSetting = 0,

.bNumEndpoints = 0,

.bInterfaceClass = 0xff,

.bInterfaceSubClass = 0x0,

.bInterfaceProtocol = 0x0,

.iInterface = 0

}

};

static const descriptor_entry_t descriptors[] = {

{ 0x0100, 0x0000, &dev_descriptor, sizeof(dev_descriptor) },

{ 0x0200, 0x0000, &cfg_descriptor, 18 },

{ 0x0000, 0x0000, NULL, 0 }

};

static void usb_endp0_transmit(const void* data, uint8_t length)

{

table[BDT_INDEX(0, TX, endp0_odd)].addr = (void *)data;

table[BDT_INDEX(0, TX, endp0_odd)].desc = BDT_DESC(length, endp0_data);

//toggle the odd and data bits

endp0_odd ^= 1;

endp0_data ^= 1;

}

/**

* Endpoint 0 setup handler

static void usb_endp0_handle_setup(setup_t* packet)

{

const descriptor_entry_t* entry;

const uint8_t* data = NULL;

uint8_t data_length = 0;

switch(packet->wRequestAndType)

{

case 0x0500: //set address (wait for IN packet)

break;

case 0x0900: //set configuration

//we only have one configuration at this time

break;

case 0x0680: //get descriptor

case 0x0681:

for (entry = descriptors; 1; entry++)

{

if (entry->addr == NULL)

break;

if (packet->wValue == entry->wValue && packet->wIndex == entry->wIndex)

{

//this is the descriptor to send

data = entry->addr;

data_length = entry->length;

goto send;

}

goto stall;

break;

default:

goto stall;

}

//if we are sent here, we need to send some data

send:

if (data_length > packet->wLength)

data_length = packet->wLength;

usb_endp0_transmit(data, data_length);

return;

//if we make it here, we are not able to send data and have stalled

stall:

USB0_ENDPT0 = USB_ENDPT_EPSTALL_MASK | USB_ENDPT_EPRXEN_MASK | USB_ENDPT_EPTXEN_MASK | USB_ENDPT_EPHSHK_MASK;

}

This is the part that took me the longest once I managed to get the module talking. Handling of SETUP tokens on endpoint 0 must be done in a rather exact fashion and the slightest mistake gives some very cryptic errors.

This is a very very very minimalistic setup token handler and is not by any means complete. It does only what is necessary to get the computer to see the device successfully read its descriptors. There is no functionality for actually doing things with the USB device. Most of the space is devoted to actually returning the various descriptors. In this example, the descriptor is for a device with a single configuration and a single interface which uses no additional endpoints. In a real device, this would almost certainly not be the case (unless one uses V-USB…this is how V-USB sets up their device if no other endpoints are compiled in).

The SETUP packet comes with a “request” and a “type”. We process these as one word for simplicity. The above shows only the necessary commands to actually get this thing to connect to a Linux machine running the standard USB drivers that come with the kernel. I have not tested it on Windows and it may require some modification to work since it doesn’t implement all of the necessary functionality. A description of the functionality follows:

Set address (0x0500): This is a very simple command. All it does is wait for the next IN token. Upon receipt of this token, the address is considered “committed” and the USB module is told of its new address (see the endpoint 0 handler function above (not the setup handler)).
Set configuration (0x0900): This command can be complex, but I have stripped it down for the purposes of this example. Normally, during this command the USB module would be set up with all the requisite BDT entries for the endpoints described by the selected configuration. Since we only have one possible configuration and it doesn’t use any additional endpoints, we basically do nothing. Once I start added other endpoints to this, all of the setup for those endpoints will go in here. This is the equivalent of the RESET handler for non-zero endpoints in terms of the operations that occur. If the Set Interface command was implemented, it would have similar functionality. More about this command can be read in the referenced USB basics website.
Get descriptor (0x0680, 0x0681): In reality, this is two commands: Get descriptor and get interface. However, due to the structure we have chosen in storing the descriptors, these two commands can be merged. This is the most complex part of this particular driver and is influenced heavily by the way things are done with the Teensyduino driver since I thought they had a very efficient pattern. Basically, it uses the wIndex and wValue to find a pointer to some data to return, whether that be the device descriptor, the configuration descriptor, a string, or something else. In our case, we have only the device descriptor and the configuration descriptor. Adding a string would be trivial, however, and the exact wIndex and wValue combination for that is described in the USB basics. The wIndex for strings matches with any of the several i* (iManufacturer, iProduct, etc) which may be specified.
default: When an unrecognized command is received, we enter a stall. This is basically the USB way of saying “uhh…I don’t know what to do here” and requires the host to un-stall the endpoint before it can continue. From what I gather, there isn’t really much the user code has to do other than declare that a stall has occurred. The USB module seems to take care of the rest of that.

After handling a command and determining that it isn’t a stall, the transmission is set up. At the moment, I only have transmission set up for a maximum of 64 bytes. In reality, this is limited by the wLength transmitted with the setup packet (note the if statement before the call to usb_endp0_transmit), but as far as I have seen this is generally the same as the length of the endpoint (I could be very wrong here…so watch out for that one). However, it would be fairly straightfoward to allow it to transmit more bytes: Upon receipt of an IN token, just check if we have reached the end of what we are supposed to transmit. If not, point the next TX buffer to the correct starting point and subtract the endpoint size from the remaining length until we have transmitted all of the bytes. Although the endpoint size is 64 bytes, it is easy to transmit much more than that; it just takes multiple IN requests. The data length is given by the descriptors, so the host can determine when to stop sending IN requests.

During transmission, both the even and data flags are toggled. This ensures that we are always using the correct TX buffer (even/odd) and the DATA flag transmitted is valid.

The descriptors are the one part that can’t really be screwed up here. Screwing up the descriptors causes interesting errors when the host tries to communicate. I did not like how the “reference” usb drivers I looked at generally defined descriptors: They used a char array. This works very well for the case where there are a variable number of entries in the descriptor, but for my purposes I decided to use named structs so that I could match the values I had specified on my device to values I read from the host machine without resorting to counting bytes in the array. It’s simply for easier reading and doesn’t really give much more than that. It may even be more error prone because I am relying on the compiler packing the struct into memory in the correct order for transmission and in later versions I may end up using the char array method.

I won’t delve into a long and drawn out description of what the USB descriptor has in it, but I will give a few points:

In Linux, the device descriptor is requested first and then the configuration descriptor after that. They are two separate commands, hence the two separate descriptor entries in my descriptor table.
The device descriptor must NOT be “const”. For my compiler at least, this causes it to be placed into flash which, while a perfectly valid memory address that in general can be read, is inaccessible to the USB module. I spent a long time banging my head on this one saying “but it should work! why doesn’t it work???” Moral of the story: Anything that is pointed to by a BDT entry (transmit buffers, receive buffers) must be located in main RAM, not in the flash. It must not be const.
A device must have at least one configuration. Linux, at least, didn’t seem to like it very much when there were zero configurations and would put lots of errors into my log.
The configuration needs to have at least one interface. Specifying no interfaces caused the same problems as not specifying any configurations.
The configuration indices (bConfigurationValue) are 1-based and the interface indices (bInterfaceNumber) are zero based. I haven’t fooled around with these enough to test the veracity of this claim fully, but it was the only configuration that I managed to get things working in.
The length values are very important. If these are not correct, the host will have some serious troubles reading the descriptors. I spend a while troubleshooting these. The main one to make sure of is the wTotalLength value in the configuration descriptor. Most of the others are pretty much always going to be the same.

Where to go from here

The driver I have implemented leaves much to be desired. This isn’t meant to be a fully featured driver. Instead, its meant to be something of an introduction to getting the USB module to work on the bare metal without the support of some external dependency. A few things that would definitely need to be implemented are:

The full set of commands for the endpoint 0 SETUP token processing
A more expansive configuration that allows for having some bulk endpoints for sending data. The 64-byte limitation of packet size for endpoint 0 can cause some issues when attempting to actually utilize the full 12Mbit/s bandwidth. The USB protocol does actually add overhead and the less times that a token has to be invoked, the better.
Strings in the configuration. Right now, the configuration is essentially “blank” because it uses a shared VID/PID and doesn’t specify a manufacturer, product, or serial number. It would be rather hard to identify this device using libusb on a system with multiple devices using that VID/PID combination.
Real error handling. Right now, the interrupt basically ignores the errors. In a real application, these would need to be handled.
A better structure. I am not a real fan of how I have structured this, but my idea was to make it “expandable” without needing to recompile usb.c every time a change was made. It doesn’t achieve that yet, but in future iterations I hope to have a relatively portable usb driver module that I can port to other projects without modification, placing the other device-specific things into another, mimimalistic, file.

Conclusion

I can only hope that this discussion has been helpful. I spent a long time reading documentation, writing code, smashing my keyboard, and figuring things out and I would like to see that someone else could benefit from this. I hope as I learn more about using the modules on my Teensy that I will become more competent in understanding how many of the systems I rely on on a daily basis function.

The code I have included above isn’t always complete, so I would definitely recommend actually reading the code in the repository referenced at the beginning of this article.

If there are any mistakes in the above, please let me know in the comments or shoot me an email.

Database Abstraction in Python

As I was recently working on trying out the Flask web framework for Python, I ended up wanting to access my MySQL database. Recently at work I have been using entity framework and I have gotten quite used to having a good database abstraction that allows programmatic creation of SQL. While such frameworks exist in Python, I thought it would interesting to try writing one. This is one great example of getting carried away with a seemingly simple task.

I aimed for these things:

Tables should be represented as objects which each instance of the object representing a row
These objects should be able to generate their own insert, select, and update queries
Querying the database should be accomplished by logical predicates, not by strings
Update queries should be optimized to only update those fields which have changed
The database objects should have support for “immutable” fields that are generated by the database

I also wanted to be able to do relations between tables with foreign keys, but I have decided to stop for now on that. I have a structure outlined, but it isn’t necessary enough at this point since all I wanted was a database abstraction for my simple Flask project. I will probably implement it later.

This can be found as a gist here: https://gist.github.com/kcuzner/5246020

Example

Before going into the code, here is an example of what this abstraction can do as it stands. It directly uses the DbObject and DbQuery-inheriting objects which are shown further down in this post.

from db import *
import hashlib

def salt_password(user, unsalted):
    if user is None:
        return unsalted
    m = hashlib.sha512()
    m.update(user.username)
    m.update(unsalted)
    return m.hexdigest()

class User(DbObject):
    dbo_tablename = "users"
    primary_key = IntColumn("id", allow_none=True, mutable=False)
    username = StringColumn("username", "")
    password = PasswordColumn("password", salt_password, "")
    display_name = StringColumn("display_name", "")
    def __init__(self, **kwargs):
        DbObject.__init__(self, **kwargs)
    @classmethod
    def load(self, cur, username):
        selection = self.select('u')
        selection[0].where(selection[1].username == username)
        result = selection[0].execute(cur)
        if len(result) == 0:
            return None
        else:
            return result[0]
    def match_password(self, password):
        salted = salt_password(self, password)
        return salted == self.password

#assume there is a function get_db defined which returns a PEP-249
#database object
def main():
    db = get_db()
    cur = db.cursor()
    user = User.load(cur, "some username")
    user.password = "a new password!"
    user.save(cur)
    db.commit()

    new_user = User(username="someone", display_name="Their name")
    new_user.password = "A password that will be hashed"
    new_user.save(cur)
    db.commmit()

    print new_user.primary_key # this will now have a database assigned id

from db import *

import hashlib

def salt_password(user, unsalted):

if user is None:

return unsalted

m = hashlib.sha512()

m.update(user.username)

m.update(unsalted)

return m.hexdigest()

class User(DbObject):

dbo_tablename = "users"

primary_key = IntColumn("id", allow_none=True, mutable=False)

username = StringColumn("username", "")

password = PasswordColumn("password", salt_password, "")

display_name = StringColumn("display_name", "")

def __init__(self, **kwargs):

DbObject.__init__(self, **kwargs)

@classmethod

def load(self, cur, username):

selection = self.select('u')

selection[0].where(selection[1].username == username)

result = selection[0].execute(cur)

if len(result) == 0:

return None

else:

return result[0]

def match_password(self, password):

salted = salt_password(self, password)

return salted == self.password

#assume there is a function get_db defined which returns a PEP-249

#database object

def main():

db = get_db()

cur = db.cursor()

user = User.load(cur, "some username")

user.password = "a new password!"

user.save(cur)

db.commit()

new_user = User(username="someone", display_name="Their name")

new_user.password = "A password that will be hashed"

new_user.save(cur)

db.commmit()

print new_user.primary_key # this will now have a database assigned id

This example first loads a user using a DbSelectQuery. The user is then modified and the DbObject-level function save() is used to save it. Next, a new user is created and saved using the same function. After saving, the primary key will have been populated and will be printed.

Change Tracking Columns

I started out with columns. I needed columns that track changes and have a mapping to an SQL column name. I came up with the following:

class ColumnSet(object):
    """
    Object which is updated by ColumnInstances to inform changes
    """
    def __init__(self):
        self.__columns = {} # columns are sorted by name
        i_dict = type(self).__dict__
        for attr in i_dict:
            obj = i_dict[attr]
            if isinstance(obj, Column):
                # we get an instance of this column
                self.__columns[obj.name] = ColumnInstance(obj, self)

    @property
    def mutated(self):
        """
        Returns the mutated columns for this tracker.
        """
        output = []
        for name in self.__columns:
            column = self.get_column(name)
            if column.mutated:
                output.append(column)
        return output

    def get_column(self, name):
        return self.__columns[name]

class ColumnInstance(object):
    """
    Per-instance column data. This is used in ColumnSet objects to hold data
    specific to that particular instance
    """
    def __init__(self, column, owner):
        """
        column: Column object this is created for
        initial: Initial value
        """
        self.__column = column
        self.__owner = owner
        self.update(column.default)

    def update(self, value):
        """
        Updates the value for this instance, resetting the mutated flag
        """
        if value is None and not self.__column.allow_none:
            raise ValueError("'None' is invalid for column '" + 
                             self.__column.name + "'")
        if self.__column.validate(value):
            self.__value = value
            self.__origvalue = value
        else:
            raise ValueError("'" + str(value) + "' is not valid for column '" + 
                             self.__column.name + "'")

    @property
    def column(self):
        return self.__column

    @property
    def owner(self):
        return self.__owner

    @property
    def mutated(self):
        return self.__value != self.__origvalue

    @property
    def value(self):
        return self.__value

    @value.setter
    def value(self, value):
        if value is None and not self.__column.allow_none:
            raise ValueError("'None' is invalid for column '" + 
                             self.__column.name + "'")
        if not self.__column.mutable:
            raise AttributeError("Column '" + self.__column.name + "' is not" +
                                 " mutable")
        if self.__column.validate(value):
            self.__value = value
        else:
            raise ValueError("'" + value + "' is not valid for column '" + 
                             self.__column.name + "'")

class Column(object):
    """
    Column descriptor for a column
    """
    def __init__(self, name, default=None, allow_none=False, mutable=True):
        """
        Initializes a column

        name: Name of the column this maps to
        default: Default value
        allow_none: Whether none (db null) values are allowed
        mutable: Whether this can be mutated by a setter
        """
        self.__name = name
        self.__allow_none = allow_none
        self.__mutable = mutable
        self.__default = default

    def validate(self, value):
        """
        In a child class, this will validate values being set
        """
        raise NotImplementedError

    @property
    def name(self):
        return self.__name

    @property
    def allow_none(self):
        return self.__allow_none

    @property
    def mutable(self):
        return self.__mutable

    @property
    def default(self):
        return self.__default

    def __get__(self, owner, ownertype=None):
        """
        Gets the value for this column for the passed owner
        """
        if owner is None:
            return self
        if not isinstance(owner, ColumnSet):
            raise TypeError("Columns are only allowed on ColumnSets")
        return owner.get_column(self.name).value

    def __set__(self, owner, value):
        """
        Sets the value for this column for the passed owner
        """
        if not isinstance(owner, ColumnSet):
            raise TypeError("Columns are only allowed on ColumnSets")
        owner.get_column(self.name).value = value

class StringColumn(Column):
    def validate(self, value):
        if value is None and self.allow_none:
            print "nonevalue"
            return True
        if isinstance(value, basestring):
            print "isstr"
            return True
        print "not string", value, type(value)
        return False

class IntColumn(Column):
    def validate(self, value):
        if value is None and self.allow_none:
            return True
        if isinstance(value, int) or isinstance(value, long):
            return True
        return False

class PasswordColumn(Column):
    def __init__(self, name, salt_function, default=None, allow_none=False, 
                 mutable=True):
        """
        Create a new password column which uses the specified salt function

        salt_function: a function(self, value) which returns the salted string
        """
        Column.__init__(self, name, default, allow_none, mutable)
        self.__salt_function = salt_function
    def validate(self, value):
        return True
    def __set__(self, owner, value):
        salted = self.__salt_function(owner, value)
        super(PasswordColumn, self).__set__(owner, salted)

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

class ColumnSet(object):

"""

Object which is updated by ColumnInstances to inform changes

"""

def __init__(self):

self.__columns = {} # columns are sorted by name

i_dict = type(self).__dict__

for attr in i_dict:

obj = i_dict[attr]

if isinstance(obj, Column):

# we get an instance of this column

self.__columns[obj.name] = ColumnInstance(obj, self)

@property

def mutated(self):

"""

Returns the mutated columns for this tracker.

"""

output = []

for name in self.__columns:

column = self.get_column(name)

if column.mutated:

output.append(column)

return output

def get_column(self, name):

return self.__columns[name]

class ColumnInstance(object):

"""

Per-instance column data. This is used in ColumnSet objects to hold data

specific to that particular instance

"""

def __init__(self, column, owner):

"""

column: Column object this is created for

initial: Initial value

"""

self.__column = column

self.__owner = owner

self.update(column.default)

def update(self, value):

"""

Updates the value for this instance, resetting the mutated flag

"""

if value is None and not self.__column.allow_none:

raise ValueError("'None' is invalid for column '" +

self.__column.name + "'")

if self.__column.validate(value):

self.__value = value

self.__origvalue = value

else:

raise ValueError("'" + str(value) + "' is not valid for column '" +

self.__column.name + "'")

@property

def column(self):

return self.__column

@property

def owner(self):

return self.__owner

@property

def mutated(self):

return self.__value != self.__origvalue

@property

def value(self):

return self.__value

@value.setter

def value(self, value):

if value is None and not self.__column.allow_none:

raise ValueError("'None' is invalid for column '" +

self.__column.name + "'")

if not self.__column.mutable:

raise AttributeError("Column '" + self.__column.name + "' is not" +

" mutable")

if self.__column.validate(value):

self.__value = value

else:

raise ValueError("'" + value + "' is not valid for column '" +

self.__column.name + "'")

class Column(object):

"""

Column descriptor for a column

"""

def __init__(self, name, default=None, allow_none=False, mutable=True):

"""

Initializes a column

name: Name of the column this maps to

default: Default value

allow_none: Whether none (db null) values are allowed

mutable: Whether this can be mutated by a setter

"""

self.__name = name

self.__allow_none = allow_none

self.__mutable = mutable

self.__default = default

def validate(self, value):

"""

In a child class, this will validate values being set

"""

raise NotImplementedError

@property

def name(self):

return self.__name

@property

def allow_none(self):

return self.__allow_none

@property

def mutable(self):

return self.__mutable

@property

def default(self):

return self.__default

def __get__(self, owner, ownertype=None):

"""

Gets the value for this column for the passed owner

"""

if owner is None:

return self

if not isinstance(owner, ColumnSet):

raise TypeError("Columns are only allowed on ColumnSets")

return owner.get_column(self.name).value

def __set__(self, owner, value):

"""

Sets the value for this column for the passed owner

"""

if not isinstance(owner, ColumnSet):

raise TypeError("Columns are only allowed on ColumnSets")

owner.get_column(self.name).value = value

class StringColumn(Column):

def validate(self, value):

if value is None and self.allow_none:

print "nonevalue"

return True

if isinstance(value, basestring):

print "isstr"

return True

print "not string", value, type(value)

return False

class IntColumn(Column):

def validate(self, value):

if value is None and self.allow_none:

return True

if isinstance(value, int) or isinstance(value, long):

return True

return False

class PasswordColumn(Column):

def __init__(self, name, salt_function, default=None, allow_none=False,

mutable=True):

"""

Create a new password column which uses the specified salt function

salt_function: a function(self, value) which returns the salted string

"""

Column.__init__(self, name, default, allow_none, mutable)

self.__salt_function = salt_function

def validate(self, value):

return True

def __set__(self, owner, value):

salted = self.__salt_function(owner, value)

super(PasswordColumn, self).__set__(owner, salted)

The Column class describes the column and is implemented as a descriptor. Each ColumnSet instance contains multiple columns and holds ColumnInstance objects which hold the individual column per-object properties, such as the value and whether it has been mutated or not. Each column type has a validation function to help screen invalid data from the columns. When a ColumnSet is initiated, it scans itself for columns and at that moment creates its ColumnInstances.

Generation of SQL using logical predicates

The next thing I had to create was the database querying structure. I decided that rather than actually using the ColumnInstance or Column objects, I would use a go-between object that can be assigned a “prefix”. A common thing to do in SQL queries is to rename the tables in the query so that you can reference the same table multiple times or use different tables with the same column names. So, for example if I had a table called posts and I also had a table called users and they both shared a column called ‘last_update’, I could assign a prefix ‘p’ to the post columns and a prefix ‘u’ to the user columns so that the final column name would be ‘p.last_update’ and ‘u.last_update’ for posts and users respectively.

Another thing I wanted to do was avoid the usage of SQL in constructing my queries. This is similar to the way that LINQ works for C#: A predicate is specified and later translated into an SQL query or a series of operations in memory depending on what is going on. So, in Python one of my queries looks like so:

class Table(ColumnSet):
    some_column = StringColumn("column_1", "")
    another = IntColumn("column_2", 0)
a_variable = 5
columns = Table.get_columns('x') # columns with a prefix 'x'
query = DbQuery() # This base class just makes a where statement
query.where((columns.some_column == "4") &amp; (columns.another &gt; a_variable)
print query.sql

class Table(ColumnSet):

some_column = StringColumn("column_1", "")

another = IntColumn("column_2", 0)

a_variable = 5

columns = Table.get_columns('x') # columns with a prefix 'x'

query = DbQuery() # This base class just makes a where statement

query.where((columns.some_column == "4") & (columns.another > a_variable)

print query.sql

This would print out a tuple (" WHERE x.column_1 = %s AND x.column_2 > %s", ["4", 5]). So, how does this work? I used operator overloading to create DbQueryExpression objects. The code is like so:

class DbQueryExpression(object):
    """
    Query expression created from columns, literals, and operators
    """
    def __and__(self, other):
        return DbQueryConjunction(self, other)
    def __or__(self, other):
        return DbQueryDisjunction(self, other)

    def __str__(self):
        raise NotImplementedError
    @property
    def arguments(self):
        raise NotImplementedError

class DbQueryConjunction(DbQueryExpression):
    """
    Query expression joining together a left and right expression with an
    AND statement
    """
    def __init__(self, l, r):
        DbQueryExpression.__ini__(self)
        self.l = l
        self.r = r
    def __str__(self):
        return str(self.l) + " AND " + str(self.r)
    @property
    def arguments(self):
        return self.l.arguments + self.r.arguments

class DbQueryDisjunction(DbQueryExpression):
    """
    Query expression joining together a left and right expression with an
    OR statement
    """
    def __init__(self, l, r):
        DbQueryExpression.__init__(self)
        self.l = l
        self.r = r
    def __str__(self):
        return str(self.r) + " OR " + str(self.r)
    @property
    def arguments(self):
        return self.l.arguments + self.r.arguments

class DbQueryColumnComparison(DbQueryExpression):
    """
    Query expression comparing a combination of a column and/or a value
    """
    def __init__(self, l, op, r):
        DbQueryExpression.__init__(self)
        self.l = l
        self.op = op
        self.r = r
    def __str__(self):
        output = ""
        if isinstance(self.l, DbQueryColumn):
            prefix = self.l.prefix
            if prefix is not None:
                output += prefix + "."
            output += self.l.name
        elif self.l is None:
            output += "NULL"
        else:
            output += "%s"
        output += self.op
        if isinstance(self.r, DbQueryColumn):
            prefix = self.r.prefix
            if prefix is not None:
                output += prefix + "."
            output += self.r.name
        elif self.r is None:
            output += "NULL"
        else:
            output += "%s"
        return output
    @property
    def arguments(self):
        output = []
        if not isinstance(self.l, DbQueryColumn) and self.l is not None:
            output.append(self.l)
        if not isinstance(self.r, DbQueryColumn) and self.r is not None:
            output.append(self.r)
        return output

class DbQueryColumnSet(object):
    """
    Represents a set of columns attached to a specific DbOject type. This
    object dynamically builds itself based on a passed type. The columns
    attached to this set may be used in DbQueries
    """
    def __init__(self, dbo_type, prefix):
        d = dbo_type.__dict__
        self.__columns = {}
        for attr in d:
            obj = d[attr]
            if isinstance(obj, Column):
                column = DbQueryColumn(dbo_type, prefix, obj.name)
                setattr(self, attr, column)
                self.__columns[obj.name] = column
    def __len__(self):
        return len(self.__columns)
    def __getitem__(self, key):
        return self.__columns[key]
    def __iter__(self):
        return iter(self.__columns)

class DbQueryColumn(object):
    """
    Represents a Column object used in a DbQuery
    """
    def __init__(self, dbo_type, prefix, column_name):
        self.dbo_type = dbo_type
        self.name = column_name
        self.prefix = prefix

    def __lt__(self, other):
        return DbQueryColumnComparison(self, "&lt;", other)
    def __le__(self, other):
        return DbQueryColumnComparison(self, "&lt;=", other)
    def __eq__(self, other):
        op = "="
        if other is None:
           op = " IS "
       return DbQueryColumnComparison(self, op, other)
    def __ne__(self, other):
        op = "!="
        if other is None:
            op = " IS NOT "
        return DbQueryColumnComparison(self, op, other)
    def __gt__(self, other):
        return DbQueryColumnComparison(self, "&gt;", other)
    def __ge__(self, other):
        return DbQueryColumnComparison(self, "&gt;=", other)

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

class DbQueryExpression(object):

"""

Query expression created from columns, literals, and operators

"""

def __and__(self, other):

return DbQueryConjunction(self, other)

def __or__(self, other):

return DbQueryDisjunction(self, other)

def __str__(self):

raise NotImplementedError

@property

def arguments(self):

raise NotImplementedError

class DbQueryConjunction(DbQueryExpression):

"""

Query expression joining together a left and right expression with an

AND statement

"""

def __init__(self, l, r):

DbQueryExpression.__ini__(self)

self.l = l

self.r = r

def __str__(self):

return str(self.l) + " AND " + str(self.r)

@property

def arguments(self):

return self.l.arguments + self.r.arguments

class DbQueryDisjunction(DbQueryExpression):

"""

Query expression joining together a left and right expression with an

OR statement

"""

def __init__(self, l, r):

DbQueryExpression.__init__(self)

self.l = l

self.r = r

def __str__(self):

return str(self.r) + " OR " + str(self.r)

@property

def arguments(self):

return self.l.arguments + self.r.arguments

class DbQueryColumnComparison(DbQueryExpression):

"""

Query expression comparing a combination of a column and/or a value

"""

def __init__(self, l, op, r):

DbQueryExpression.__init__(self)

self.l = l

self.op = op

self.r = r

def __str__(self):

output = ""

if isinstance(self.l, DbQueryColumn):

prefix = self.l.prefix

if prefix is not None:

output += prefix + "."

output += self.l.name

elif self.l is None:

output += "NULL"

else:

output += "%s"

output += self.op

if isinstance(self.r, DbQueryColumn):

prefix = self.r.prefix

if prefix is not None:

output += prefix + "."

output += self.r.name

elif self.r is None:

output += "NULL"

else:

output += "%s"

return output

@property

def arguments(self):

output = []

if not isinstance(self.l, DbQueryColumn) and self.l is not None:

output.append(self.l)

if not isinstance(self.r, DbQueryColumn) and self.r is not None:

output.append(self.r)

return output

class DbQueryColumnSet(object):

"""

Represents a set of columns attached to a specific DbOject type. This

object dynamically builds itself based on a passed type. The columns

attached to this set may be used in DbQueries

"""

def __init__(self, dbo_type, prefix):

d = dbo_type.__dict__

self.__columns = {}

for attr in d:

obj = d[attr]

if isinstance(obj, Column):

column = DbQueryColumn(dbo_type, prefix, obj.name)

setattr(self, attr, column)

self.__columns[obj.name] = column

def __len__(self):

return len(self.__columns)

def __getitem__(self, key):

return self.__columns[key]

def __iter__(self):

return iter(self.__columns)

class DbQueryColumn(object):

"""

Represents a Column object used in a DbQuery

"""

def __init__(self, dbo_type, prefix, column_name):

self.dbo_type = dbo_type

self.name = column_name

self.prefix = prefix

def __lt__(self, other):

return DbQueryColumnComparison(self, "<", other)

def __le__(self, other):

return DbQueryColumnComparison(self, "<=", other)

def __eq__(self, other):

op = "="

if other is None:

op = " IS "

return DbQueryColumnComparison(self, op, other)

def __ne__(self, other):

op = "!="

if other is None:

op = " IS NOT "

return DbQueryColumnComparison(self, op, other)

def __gt__(self, other):

return DbQueryColumnComparison(self, ">", other)

def __ge__(self, other):

return DbQueryColumnComparison(self, ">=", other)

The __str__ function and arguments property return recursively generated expressions using the column prefixes (in the case of __str__) and the arguments (in the case of arguments). As can be seen, this supports parameterization of queries. To be honest, this part was the most fun since I was surprised it was so easy to make predicate expressions using a minimum of classes. One thing that I didn’t like, however, was the fact that the boolean and/or operators cannot be overloaded. For that reason I had to use the bitwise operators, so the expressions aren’t entirely correct when being read.

This DbQueryExpression is fed into my DbQuery object which actually does the translation to SQL. In the example above, we saw that I just passed a logical argument into my where function. This actually was a DbQueryExpression since my overloaded operators create DbQueryExpression objects when they are compared. The DbColumnSet object is an dynamically generated object containing the go-between column objects which is created from a DbObject. We will discuss the DbObject a little further down

The DbQuery objects are implemented as follows:

class DbQueryError(Exception):
    """
    Raised when there is an error constructing a query
    """
    def __init__(self, msg):
        self.message = msg
    def __str__(self):
        return self.message

class DbQuery(object):
    """
    Represents a base SQL Query to a database based upon some DbObjects

    All of the methods implemented here are valid on select, update, and
    delete statements.
    """
    def __init__(self, execute_filter=None):
        """
        callback: Function to call when the DbQuery is executed
        """
        self.__where = []
        self.__limit = None
        self.__orderby = []
        self.__execute_filter = execute_filter
    def where(self, expression):
        """Specify an expression to append to the WHERE clause"""
        self.__where.append(expression)
    def limit(self, value=None):
        """Specify the limit to the query"""
        self.__limit = value
    @property
    def sql(self):
        query = ""
        args = []
        if len(self.__where) &gt; 0:
            where = self.__where[0]
            for clause in self.__where[1:]:
                where = where &amp; clause
            args = where.arguments
            query += " WHERE " + str(where)
        if self.__limit is not None:
            query += " LIMIT " + self.__limit
        return query,args
    def execute(self, cur):
        """
        Executes this query on the passed cursor and returns either the result
        of the filter function or the cursor if there is no filter function.
        """
        query = self.sql
        cur.execute(query[0], query[1])
        if self.__execute_filter:
            return self.__execute_filter(self, cur)
        else:
            return cur

class DbSelectQuery(DbQuery):
    """
    Creates a select query to a database based upon DbObjects
    """
    def __init__(self, execute_filter=None):
        DbQuery.__init__(self, execute_filter)
        self.__select = []
        self.__froms = []
        self.__joins = []
        self.__orderby = []
    def select(self, *columns):
        """Specify one or more columns to select"""
        self.__select += columns
    def from_table(self, dbo_type, prefix):
        """Specify a table to select from"""
        self.__froms.append((dbo_type, prefix))
    def join(self, dbo_type, prefix, on):
        """Specify a table to join to"""
        self.__joins.append((dbo_type, prefix, on))
    def orderby(self, *columns):
        """Specify one or more columns to order by"""
        self.__orderby += columns
    @property
    def sql(self):
        query = "SELECT "
        args = []
        if len(self.__select) == 0:
            raise DbQueryError("No selection in DbSelectQuery")
        query += ','.join([col.prefix + "." + 
                 col.name for col in self.__select])
        if len(self.__froms) == 0:
            raise DbQueryError("No FROM clause in DbSelectQuery")
        for table in self.__froms:
            query += " FROM " + table[0].dbo_tablename + " " + table[1]
        if len(self.__joins) &gt; 0:
            for join in self.__joins:
                query += " JOIN " + join[0].dbo_tablename + " " + join[1] + 
                         " ON " + str(join[2])
        query_parent = super(DbSelectQuery, self).sql
        query += query_parent[0]
        args += query_parent[1]
        if len(self.__orderby) &gt; 0:
           query += " ORDER BY " + 
                    ','.join([col.prefix + "." + 
                    col.name for col in self.__orderby])
        return query,args

class DbInsertQuery(DbQuery):
    """
    Creates an insert query to a database based upon DbObjects. This does not
    include any where or limit expressions
    """
    def __init__(self, dbo_type, prefix, execute_filter=None):
        DbQuery.__init__(self, execute_filter)
        self.table = (dbo_type, prefix)
        self.__values = []
    def value(self, column, value):
        self.__values.append((column, value))
    @property
    def sql(self):
        if len(self.__values) == 0:
            raise DbQueryError("No values in insert")
        tablename = self.table[0].dbo_tablename
        query = "INSERT INTO {table} (".format(table=tablename)
        args = [val[1] for val in self.__values 
                if val[0].prefix == self.table[1]]
        query += ",".join([val[0].name for val in self.__values 
                          if val[0].prefix == self.table[1]])
        query += ") VALUES ("
        query += ",".join(["%s" for x in args])
        query += ")"
        return query,args

class DbUpdateQuery(DbQuery):
    """
    Creates an update query to a database based upon DbObjects
    """
    def __init__(self, dbo_type, prefix, execute_filter=None):
        """
        Initialize the update query

        dbo_type: table type to be updating
        prefix: Prefix the columns are known under
        """
        DbQuery.__init__(self, execute_filter)
        self.table = (dbo_type, prefix)
        self.__updates = []
    def update(self, left, right):
        self.__updates.append((left, right))
    @property
    def sql(self):
        if len(self.__updates) == 0:
            raise DbQueryError("No update in DbUpdateQuery")
        query = "UPDATE " + self.table[0].dbo_tablename + " " + self.table[1]
        args = []
        query += " SET "
        for update in self.__updates:
            if isinstance(update[0], DbQueryColumn):
                query += update[0].prefix + "." + update[0].name
            else:
                query += "%s"
                args.append(update[0])
            query += "="
            if isinstance(update[1], DbQueryColumn):
                query += update[1].prefix + "." + update[1].name
            else:
                query += "%s"
                args.append(update[1])
        query_parent = super(DbUpdateQuery, self).sql
        query += query_parent[0]
        args += query_parent[1]
        return query, args

class DbDeleteQuery(DbQuery):
    """
    Creates a delete query for a database based on a DbObject
    """
    def __init__(self, dbo_type, prefix, execute_filter=None):
        DbQuery.__init__(self, execute_filter)
        self.table = (dbo_type, prefix)
    @property
    def sql(self):
        query = "DELETE FROM " + self.table[0].dbo_tablename + " " + 
                self.table[1]
        args = []
        query_parent = super(DbDeleteQuery, self).sql
        query += query_parent[0]
        args += query_parent[1]
        return query, args

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

class DbQueryError(Exception):

"""

Raised when there is an error constructing a query

"""

def __init__(self, msg):

self.message = msg

def __str__(self):

return self.message

class DbQuery(object):

"""

Represents a base SQL Query to a database based upon some DbObjects

All of the methods implemented here are valid on select, update, and

delete statements.

"""

def __init__(self, execute_filter=None):

"""

callback: Function to call when the DbQuery is executed

"""

self.__where = []

self.__limit = None

self.__orderby = []

self.__execute_filter = execute_filter

def where(self, expression):

"""Specify an expression to append to the WHERE clause"""

self.__where.append(expression)

def limit(self, value=None):

"""Specify the limit to the query"""

self.__limit = value

@property

def sql(self):

query = ""

args = []

if len(self.__where) > 0:

where = self.__where[0]

for clause in self.__where[1:]:

where = where & clause

args = where.arguments

query += " WHERE " + str(where)

if self.__limit is not None:

query += " LIMIT " + self.__limit

return query,args

def execute(self, cur):

"""

Executes this query on the passed cursor and returns either the result

of the filter function or the cursor if there is no filter function.

"""

query = self.sql

cur.execute(query[0], query[1])

if self.__execute_filter:

return self.__execute_filter(self, cur)

else:

return cur

class DbSelectQuery(DbQuery):

"""

Creates a select query to a database based upon DbObjects

"""

def __init__(self, execute_filter=None):

DbQuery.__init__(self, execute_filter)

self.__select = []

self.__froms = []

self.__joins = []

self.__orderby = []

def select(self, *columns):

"""Specify one or more columns to select"""

self.__select += columns

def from_table(self, dbo_type, prefix):

"""Specify a table to select from"""

self.__froms.append((dbo_type, prefix))

def join(self, dbo_type, prefix, on):

"""Specify a table to join to"""

self.__joins.append((dbo_type, prefix, on))

def orderby(self, *columns):

"""Specify one or more columns to order by"""

self.__orderby += columns

@property

def sql(self):

query = "SELECT "

args = []

if len(self.__select) == 0:

raise DbQueryError("No selection in DbSelectQuery")

query += ','.join([col.prefix + "." +

col.name for col in self.__select])

if len(self.__froms) == 0:

raise DbQueryError("No FROM clause in DbSelectQuery")

for table in self.__froms:

query += " FROM " + table[0].dbo_tablename + " " + table[1]

if len(self.__joins) > 0:

for join in self.__joins:

query += " JOIN " + join[0].dbo_tablename + " " + join[1] +

" ON " + str(join[2])

query_parent = super(DbSelectQuery, self).sql

query += query_parent[0]

args += query_parent[1]

if len(self.__orderby) > 0:

query += " ORDER BY " +

','.join([col.prefix + "." +

col.name for col in self.__orderby])

return query,args

class DbInsertQuery(DbQuery):

"""

Creates an insert query to a database based upon DbObjects. This does not

include any where or limit expressions

"""

def __init__(self, dbo_type, prefix, execute_filter=None):

DbQuery.__init__(self, execute_filter)

self.table = (dbo_type, prefix)

self.__values = []

def value(self, column, value):

self.__values.append((column, value))

@property

def sql(self):

if len(self.__values) == 0:

raise DbQueryError("No values in insert")

tablename = self.table[0].dbo_tablename

query = "INSERT INTO {table} (".format(table=tablename)

args = [val[1] for val in self.__values

if val[0].prefix == self.table[1]]

query += ",".join([val[0].name for val in self.__values

if val[0].prefix == self.table[1]])

query += ") VALUES ("

query += ",".join(["%s" for x in args])

query += ")"

return query,args

class DbUpdateQuery(DbQuery):

"""

Creates an update query to a database based upon DbObjects

"""

def __init__(self, dbo_type, prefix, execute_filter=None):

"""

Initialize the update query

dbo_type: table type to be updating

prefix: Prefix the columns are known under

"""

DbQuery.__init__(self, execute_filter)

self.table = (dbo_type, prefix)

self.__updates = []

def update(self, left, right):

self.__updates.append((left, right))

@property

def sql(self):

if len(self.__updates) == 0:

raise DbQueryError("No update in DbUpdateQuery")

query = "UPDATE " + self.table[0].dbo_tablename + " " + self.table[1]

args = []

query += " SET "

for update in self.__updates:

if isinstance(update[0], DbQueryColumn):

query += update[0].prefix + "." + update[0].name

else:

query += "%s"

args.append(update[0])

query += "="

if isinstance(update[1], DbQueryColumn):

query += update[1].prefix + "." + update[1].name

else:

query += "%s"

args.append(update[1])

query_parent = super(DbUpdateQuery, self).sql

query += query_parent[0]

args += query_parent[1]

return query, args

class DbDeleteQuery(DbQuery):

"""

Creates a delete query for a database based on a DbObject

"""

def __init__(self, dbo_type, prefix, execute_filter=None):

DbQuery.__init__(self, execute_filter)

self.table = (dbo_type, prefix)

@property

def sql(self):

query = "DELETE FROM " + self.table[0].dbo_tablename + " " +

self.table[1]

args = []

query_parent = super(DbDeleteQuery, self).sql

query += query_parent[0]

args += query_parent[1]

return query, args

Each of the SELECT, INSERT, UPDATE, and DELETE query types inherits from a base DbQuery which does execution and such. I decided to make the DbQuery object take a PEP 249-style cursor object and execute the query itself. My hope is that this will make this a little more portable since, to my knowledge, I didn’t make the queries have any MySQL-specific constructions.

The different query types each implement a variety of statements corresponding to different parts of an SQL query: where(), limit(), orderby(), select(), from_table(), etc. These each take in either a DbQueryColumn (such as is the case with where(), orderby(), select(), etc) or a string to be appended to the query, such as is the case with limit(). I could easily have made limit take in two integers as well, but I was kind of rushing through because I wanted to see if this would even work. The query is built by creating the query object for the basic query type that is desired and then calling its member functions to add things on to the query.

Executing the queries can cause a callback “filter” function to be called which takes in the query and the cursor as arguments. I use this function to create new objects from the data or to update an object. It could probably be used for more clever things as well, but those two cases were my original intent in creating it. If no filter is specified, then the cursor is returned.

Table and row objects

At the highest level of this hierarchy is the DbObject. The DbObject definition actually represents a table in the database with a name and a single primary key column. Each instance represents a row. DbObjects also implement the methods for selecting records of their type and also updating themselves when they are changed. They inherit change tracking from the ColumnSet and use DbQueries to accomplish their querying goals. The code is as follows:

class DbObject(ColumnSet):
    """
    A DbObject is a set of columns linked to a table in the database. This is
    synonomous to a row. The following class attributes must be set:

    dbo_tablename : string table name
    primary_key : Column for the primary key
    """
    def __init__(self, **cols):
        ColumnSet.__init__(self)
        for name in cols:
            c = self.get_column(name)
            c.update(cols[name])

    @classmethod
    def get_query_columns(self, prefix):
        return DbQueryColumnSet(self, prefix)

    @classmethod
    def select(self, prefix):
        """
        Returns a DbSelectQuery set up for this DbObject
        """
        columns = self.get_query_columns(prefix)
        def execute(query, cur):
            output = []
            block = cur.fetchmany()
            while len(block) &gt; 0:
                for row in block:
                    values = {}
                    i = 0
                    for name in columns:
                        values[name] = row[i]
                        i += 1
                    output.append(self(**values))
                block = cur.fetchmany()
            return output
        query = DbSelectQuery(execute)
        query.select(*[columns[name] for name in columns])
        query.from_table(self, prefix)
        return query, columns

    def get_primary_key_name(self):
        return type(self).__dict__['primary_key'].name

    def save(self, cur):
        """
        Saves any changes to this object to the database
        """
        if self.primary_key is None:
            # we need to be saved
            columns = self.get_query_columns('x')
            def execute(query, cur):
                self.get_column(self.get_primary_key_name()
                                ).update(cur.lastrowid)
                selection = []
                for name in columns:
                    if name == self.get_primary_key_name():
                        continue #we have no need to update the primary key
                    column_instance = self.get_column(name)
                    if not column_instance.column.mutable:
                        selection.append(columns[name])
                if len(selection) != 0:
                    # we get to select to get additional computed values
                    def execute2(query, cur):
                        row = cur.fetchone()
                        index = 0
                        for s in selection:
                            self.get_column(s.name).update(row[index])
                            index += 1
                        return True
                    query = DbSelectQuery(execute2)
                    query.select(*selection)
                    query.from_table(type(self), 'x')
                    query.where(columns[self.get_primary_key_name()] == 
                                self.get_column(self.get_primary_key_name()
                                                ).value)
                    return query.execute(cur)
                return True
            query = DbInsertQuery(type(self), 'x', execute)
            for name in columns:
                column_instance = self.get_column(name)
                if not column_instance.column.mutable:
                    continue
                query.value(columns[name], column_instance.value)
            print query.sql
            return query.execute(cur)
        else:
            # we have been modified
            modified = self.mutated
            if len(modified) == 0:
                return True
            columns = self.get_query_columns('x')
            def execute(query, cur):
                for mod in modified:
                    mod.update(mod.value)
                return True
            query = DbUpdateQuery(type(self), 'x', execute)
            for mod in modified:
                query.update(columns[mod.column.name], mod.value)
            query.where(columns[self.get_primary_key_name()] == self.primary_key)
            return query.execute(cur)

100

101

102

class DbObject(ColumnSet):

"""

A DbObject is a set of columns linked to a table in the database. This is

synonomous to a row. The following class attributes must be set:

dbo_tablename : string table name

primary_key : Column for the primary key

"""

def __init__(self, **cols):

ColumnSet.__init__(self)

for name in cols:

c = self.get_column(name)

c.update(cols[name])

@classmethod

def get_query_columns(self, prefix):

return DbQueryColumnSet(self, prefix)

@classmethod

def select(self, prefix):

"""

Returns a DbSelectQuery set up for this DbObject

"""

columns = self.get_query_columns(prefix)

def execute(query, cur):

output = []

block = cur.fetchmany()

while len(block) > 0:

for row in block:

values = {}

i = 0

for name in columns:

values[name] = row[i]

i += 1

output.append(self(**values))

block = cur.fetchmany()

return output

query = DbSelectQuery(execute)

query.select(*[columns[name] for name in columns])

query.from_table(self, prefix)

return query, columns

def get_primary_key_name(self):

return type(self).__dict__['primary_key'].name

def save(self, cur):

"""

Saves any changes to this object to the database

"""

if self.primary_key is None:

# we need to be saved

columns = self.get_query_columns('x')

def execute(query, cur):

self.get_column(self.get_primary_key_name()

).update(cur.lastrowid)

selection = []

for name in columns:

if name == self.get_primary_key_name():

continue #we have no need to update the primary key

column_instance = self.get_column(name)

if not column_instance.column.mutable:

selection.append(columns[name])

if len(selection) != 0:

# we get to select to get additional computed values

def execute2(query, cur):

row = cur.fetchone()

index = 0

for s in selection:

self.get_column(s.name).update(row[index])

index += 1

return True

query = DbSelectQuery(execute2)

query.select(*selection)

query.from_table(type(self), 'x')

query.where(columns[self.get_primary_key_name()] ==

self.get_column(self.get_primary_key_name()

).value)

return query.execute(cur)

return True

query = DbInsertQuery(type(self), 'x', execute)

for name in columns:

column_instance = self.get_column(name)

if not column_instance.column.mutable:

continue

query.value(columns[name], column_instance.value)

print query.sql

return query.execute(cur)

else:

# we have been modified

modified = self.mutated

if len(modified) == 0:

return True

columns = self.get_query_columns('x')

def execute(query, cur):

for mod in modified:

mod.update(mod.value)

return True

query = DbUpdateQuery(type(self), 'x', execute)

for mod in modified:

query.update(columns[mod.column.name], mod.value)

query.where(columns[self.get_primary_key_name()] == self.primary_key)

return query.execute(cur)

DbObjects require that the inheriting classes define two properties: dbo_tablename and primary_key. dbo_tablename is just a string giving the name of the table in the database and primary_key is a Column that will be used as the primary key.

To select records from the database, the select() function can be called from the class. This sets up a DbSelectQuery which will return an array of the DbObject that it is called for when the query is executed.

One fallacy of this structure is that at the moment it assumes that the primary key won’t be None if it has been set. In other words, the way I did it right now does not allow for null primary keys. The reason it does this is because it says that if the primary key hasn’t been set, it needs to generate a DbInsertQuery for the object when save() is called instead of a DbUpdateQuery. Both insert and update queries do not include every field. Immutable fields are always excluded and then later selected or inferred from the cursor object.

Projects & Libraries

Musings and Projects from Kevin Cuzner

Tag Archives: programming

Dev boards? Where we’re going we won’t need dev boards…

A complete tutorial for using an STM32 without a dev board

Introduction

Materials

Step 1: Download the datasheets

Step 2: Figure out where to solder and do it

Step 3: Connect the breadboard and programmer

Step 4: Download the STM32F1xx C headers

Step 5: Install the required software

Step 6: Write and compile the program

Step 7: Flashing the program to the microcontroller

Conclusion

Teensy 3.1 bare metal: Writing a USB driver

Contents

USB Basics

The Freescale K20 Family and their USB module

Part 1: The clocks

Part 2: The startup sequence

Part 3: The interrupt handler state machine

Part 4: Token processing & descriptors

Where to go from here

Conclusion

Database Abstraction in Python

Example

Change Tracking Columns

Generation of SQL using logical predicates

Table and row objects