DIY Self-Driving - Build updates

Please Note:  Unlike most of my projects which are fully complete before they are published, this project is as much a build diary as anything else.  Whilst I am attempting to clearly document the processes, this is not a step-by-step guide, and later articles may well contradict earlier instructions.

If you decide you want to duplicate my build, please read through all the relevant articles before you start.

This post is an update to the build and testing article, rather than a brand new chapter.  Because I learned a whole load of things during early testing I think it's worth sharing at this point:

New Camera Mounting

I was never happy with the way that the camera was literally stuck onto the car, because it just looked, well, wrong.  In the end, I cut a slot into the bonnet (hood) just big enough to clear the camera down to the lens body.  Still not perfect because I need to mould a cover, but better than blu-tak

New camera mount and an access port

 Access port

The next thing corrects a really dumb mistake on my part.  When the covers are all back in place, how to we charge up the electronics batteries?  Fortunately the grille halves pop out and I was able to cut a 60mm hole with a hole-saw behind it.  Extending the cables from the battery gives me charger access

No more Ethernet

With a little searching, I fitted a D-Link DWA-131 USB Wireless adapter into the car.  This means I don't need an Ethernet port to talk to the Raspberry Pi anymore.  Of course if I'd used a Mk 3 board I wouldn't have this problem anyway but I didn't have one handy.

If you go down that road, you need to manually update /etc/network/interfaces so that it points to the wpa_supplicant.conf file or you won't connect at all, even though you added the network via raspi-conf.

Automated Builds

This is for the software, not the car itself.  Previously I needed to load all the development tools onto the Raspberry Pi and compile natively, which is rather a slow and some might say painful process.  Since I use Jenkins in my day-to-day work it was a matter of using another Raspberry Pi I had on hand as a build node.  When completed it pushes the libraries and executables to a JFrog Artifactory instance.  Now I just need to copy the files down to the car and it's all done.

Overkill? Yes.  But you have to agree it looks good

More Power!

The 1.3Ah main battery has been giving me grief, so I added a second one because there was plenty of space under the seat.  My tests so far show I should get over an hour of running time now.

DIY Self Driving - Part 7. Bringing it all together

Please Note:  Unlike most of my projects which are fully complete before they are published, this project is as much a build diary as anything else.  Whilst I am attempting to clearly document the processes, this is not a step-by-step guide, and later articles may well contradict earlier instructions.

If you decide you want to duplicate my build, please read through all the relevant articles before you start.

The theory is done.  How does it all come together?  If you have been reading from the beginning you will see that lots of details have changed as the project has progressed.  More often than not this was because either something didn't work out as expected or I figured a better way of doing things.

One thing which may surprise you is just how much wiring there actually is.  To make the process less frightening, I will break it down into tasks.

Power Distribution

I noted earlier that the car has two distinct battery supplies.  The "main" battery which powers the motor, steering and lighting is located under the seat.  This is a six-volt sealed lead-acid (SLA) battery.  The second is a Lithium-Ion (Li-Ion) pack in the front compartment.  It is used only for powering the electronics.

Distributing power for the motors is easy since they are conveniently located in the main cabin, and we can pick off the power from the wiring for the original direction switch.  The electronics battery however is all new.  To solve this, I created a simple distribution strip and attached it to the firewall.  Any electronics which needs a regulated 5V supply is connected to these strips.
Electronics Power Distribution
Because there are two different supplies, a common ground is required, and since the body of the car is plastic, this means running a wire between the the sets of wiring to do the job.  Use a medium gauge hook-up wire for this task.

Mounting all the electronics

Vibration will cause your problems in the long-run so secure everything properly.  By examining the photos, you can see I used M3 x 12 screws and nuts with a small spacer to secure the power regulator, distribution, Arduino and motor driver boards.  The Raspberry Pi I am using is an early model which has no mounting holes.  To solve this, I used some adhesive mounting hooks.

If you have not already set up your Raspberry Pi, this would be an ideal time to do so.

Wiring up

Electronics Power

Power for the electronics is switched using a relay powered off the main battery.  This sounds old-fashioned (it is), but it's also durable and effective.  The battery is connected to the voltage regulator through the relay.

The output side of the regulator goes to the electronics power distribution strip.  In my case negative is at the top, but it's up to you.  Check your strip for shorts before connecting the battery.

You need power for the Arduino, Raspberry Pi, ultrasonics, LCD and current sensor (the last two are optional).  The current sensor requires a cable run to the battery compartment, and I followed the existing wiring.  Make sure you secure the new cables to the existing ones to prevent damage as they do get close to wheels.


Completing the Arduino first makes good sense because we can use it to test subsystems on it's own.  For pin connections, always refer to the source code as the definitive reference.  The sets of connections are:
Power: To the distribution strip
Motor Driver:  Two digital outputs (D2 and D4), to the motor drive A and B inputs respectively.  Pull in an extra conductor while you are there so you can experiment with PWM control later.
Servo: PWM pin D9 to the servo control line.
Ultrasonics:  As well as power (to the dist. strip) connect the TRIG to D10 and ECHO to D12 on the Arduino.
Current Sensor (Optional):  Power (dist. strip) and a single analogue signal to A0.
LCD (Optional): Power (dist strip), SDA (Data) and SCL (Clock) to the LCD panel

Raspberry Pi

At this stage, do not connect the Serial lines to the Arduino.  We need to load the firmware and test it first.
Serial out (8) to Serial In on the Arduino
Serial in (10) to Serial Out on the Arduino
GPIO7 (26) to D5 on the Arduino
Power to the distribution strip

Current Sensor

The two screw terminals are used to connect the module in-line with the positive battery wiring.  The pin terminals connect to 5V, Ground and the analogue input on the Arduino.

Wiring all done.  Ethernet helps with development

One little problem I didn't think of at first was "How am I going to charge the electronics batteries with the front fitted in place?  This could also apply to connecting the Ethernet port on the Raspberry Pi as well.

The solution presented it when I discovered that the "grilles" were removable. (The simply clip into place).  Removing the left-hand grill, I used a 60mm hole-saw to create a discrete opening behind the grill.  A short Ethernet cable and extended battery charge cable are just within reach now.

Accessing the battery charge port

First smoke test

First of all, make sure the car is off the ground because if your work area is small the car can damage itself or something else if it gets away.

Disconnect the 5V wire from the Raspberry Pi to stop it from starting up.  Connect your development machine to the Arduino via USB and load up steering.ino.  Using the Serial Monitor function (set to 115200 bps), test both the steering and motor control functions.

Sending "." or ">" will steer the servo to the right, while "," or "<" turns left.
Sending "f" or "F" runs the drive motor forward, while "r" or "R" runs it in reverse.  "s" or "S" stops the motor.
Sending "p" or "P" triggers an ultrasonic ping.  Check it raises US_OUT in response to an object 14cm to the front of the sensor.

If your motor runs the wrong direction, reverse the connection to the motor controller.  While you are there, check the current monitor if fitted.  On mine, a positive value is a battery discharge and a negative is a charge.  This makes sense to me, but if you want to make it work like a traditional car ammeter, reverse the wires on the input to the current sensor.

You can see my car being tested in this short video:

Second smoke test

Keep the car off the ground and switch off the master power.  The reconnect the Raspberry Pi power, and plug in your camera and Ethernet (if used).  At present we are NOT starting the software automatically, so you will need to ssh into the board and start the control program manually.

What happens next really depends on what the camera is pointed at.  The drive wheel should start running forwards, and because it can't actually move the car, the front wheels will slowly move from centre to one side.  Don't worry, the software limits the movement so you can't damage the servo.

If you got a result similar to this, you can celebrate.  You earned it!

Blurred rear wheel means it's alive!
However, if things went wrong, the control program output will be a good place to start.  If it can't open the camera chances are it's not supported under Raspbian, so you will either need drivers or another camera.

If you get an "Error 2" opening the serial port, do an "ls /dev/tty*" and find your port, then update the code and recompile.  If you get "Error 5", your user is probably not in the dialout group or the port is already being used.

Update:  If you are using the 0.5.1 release of the software you will find the LCD is rather hard to read with some LCD panels.  A correction to this is on the "develop" branch of the code now, but it will be released as part of 0.6.x

Further Reading
Part 1 - Introduction
Part 2 - Preparing the car
Part 3 - Wiring Harnesses
Part 4 - Steering
Part 5 - Sensors
Part 6 - Software In Depth

DIY Self Driving - Part 6. Software in depth

Please Note:  Unlike most of my projects which are fully complete before they are published, this project is as much a build diary as anything else.  Whilst I am attempting to clearly document the processes, this is not a step-by-step guide, and later articles may well contradict earlier instructions.

If you decide you want to duplicate my build, please read through all the relevant articles before you start.

Until now this project has dealt with the mechanical and electrical components.  I have been putting off the software for a while, because many of the ideas are still forming in my mind.  But with everything else having reached a functional state, it is time to consider the overall software environment.

Software Goals

The overall goal is very simple.  To control the steering and acceleration of the car in such a way that it follow the road in front of it without impacting with other objects.


I think most will agree this is the most exciting part of the project.  It is also probably the most complex as well.  So I will start here, and save the boring stuff for later.

The vision component is tasked with the following:
  • Capturing an image of the "road" ahead of the car
  • Stabilise and reduce noise in the image
  • Reduce the image content to find the road edge
  • Determine the steering input required to centre the car in the image
The capture component relies on the Video For Linux (V4L) libraries to handle camera and import a video stream one frame at a time.  This is a stable and well-known interface, and little configuration is rarely required.

Since our car has no damping at all, there is the probability that images will be distorted due to vibration.  This is actually a function of camera performance as much as anything.  If this is a problem for your car, you can use the OpenCV stabilize APIs to generate a new image by comparing a set of previous frames.  The problem with this approach however is that it takes time, and when the car is on the move this can be a luxury we simply don't have.

Reducing the image takes a number of steps:
  • Conversion to greyscale, which removes complex colour information
  • Histogram Equalisation.  This increases the contrast of the image, resulting in a greater difference between dark and light regions (explanation)
  • Smoothing via Gaussian Blur.  This removes detail around the edge of objects, giving overall clearer lines to follow (explanation)
  • Apply a threshold.  Every pixel with a value greater than the threshold of 125 is assigned a value of 255.  Essentially the image is reduced to two values (explanation)
  • Erode the image to make the remaining edges thicker and easier to find (explanation)
Having converted the incoming frame to little more than a collection of edges, the software attempts to find the mid point between vertical edges, and then calculate how far the centre point of the frame is from the calculated midpoint.

The software determines based of the centre offset value whether right or left steering input is required, and the command is sent to the Arduino for execution.

It is important to remember here that the camera needs to be located at the front, along the longitudinal centre of the car.  If the camera is significantly offset it will be necessary to offset the steering value as well to compensate.

All the code is written in C++ because it is fast and the same language that OpenCV is written in to start with.  This reduces the interfaces required to get to the functions and will offer us a small improvement in performance over say Python (I like Python by the way).  Another reason for using a compiled rather than interpretive language is that a portion of the error detection (especially semantics) occurs during the compile phase so at least those errors won't get loaded into the car.

When developing the code I found it really useful to be able to see what the software was working out, so I added some code to display the final image and provide an overlay on the original.  Since there is no monitor attached to the Raspberry Pi it is not practical to run it this way on the car, but the code will compile under Linux, Windows and MacOS so you can define _GUI in the Makefile and run it on a graphics equipped machine.

This code is contained under the "videostream" directory in the source.


Initially I wanted to run all the functions from the Raspberry Pi but I discovered in early testing that there were performance issues, particularly around controlling the steering.  I could have changed the Pi for an Odroid XU4 or similar, but in the end I decided that providing a dedicated controller for motion made more sense.  I settled on an Arduino Uno, because it has sufficient I/O and well-proven libraries for the devices I needed to control.

The Arduino is responsible for the following functions:
  • Motor speed and direction of rotation
  • Steering servo control
  • Ultrasonic ranging
The motor and servo functions are very simple and came straight from other robot projects in the past.  They also have the advantage of being static from a software perspective.  That is, when the Arduino code sets an output or servo position, there is no refreshing of this required.  The attached devices will remain signalled until the software changes the I/O pins.

Ultrasonic ranging operates only when commanded and uses the NewPing library.  Eventually I intend to get the videostream logic to spot obstacles, but in the interests of getting something moving I have left this task on the Arduino for now.  The logic for rangefinding is very simple.  If an object is closer than 15cm to the sensor, the Arduino raises its US_OUT pin (refer to steering_hw.h).  This serves as a trigger to the Raspberry Pi to send a "stop" command.

The software on the Arduino is a simple interpretive loop.  Using a serial port (at 115200bps) the Raspberry Pi sends commands to the Arduino for processing.  Each command is a single character and is decoded using a simple switch construct.

Also built into the Arduino code is a "calibrate" routine for the steering.  When an input is taken low, the drive motor is disabled, and the steering is controlled from the keyboard of an attached computer using "." for right and "," for left.  Once the limits of travel have been determined, the values are stored in EEPROM on the Arduino.  This also helps us correct for any misalignment in the steering components.

You can (and I think should) test the software platform in a number of phases.  Because the physical control has been offloaded onto the Arduino it is easy enough to use the serial monitor function of the Arduino IDE to communicate.

Test each function by sending the relevant command over the serial monitor.  Remember that on a Mac or Linux machine (and probably Windows, I just haven't tried) you need to click the "Send" button after each command character.

Once you are happy that the controls work as expected, attach a USB-Serial interface to the Raspberry Pi and monitor the commands coming out of it, and make sure they correspond with your expectations.

If you are following my design there are a couple of important things to remember before you connect the Raspberry Pi to the Arduino:

  1. You must disable the serial port as a console
  2. Add your user (probably pi) to the "dialout" group
  3. Ensure the camera is connected.
  4. Restart the Raspberry Pi.
On the Arduino side:
  1. Since the UNO has only a single serial port, do not connect the serial lines from the Raspberry Pi to the UNO's serial port while the USB cable is plugged in.  I don't think it will damage anything but you can be sure nothing is going to work right.

Further Reading
Part 1 - Introduction
Part 2 - Preparing the car
Part 3 - Wiring Harnesses
Part 4 - Steering
Part 5 - Sensors
Part 7 - Final assembly and testing


DIY Self Driving - Part 5 - Sensors

Please Note:  Unlike most of my projects which are fully complete before they are published, this project is as much a build diary as anything else.  Whilst I am attempting to clearly document the processes, this is not a step-by-step guide, and later articles may well contradict earlier instructions.

If you decide you want to duplicate my build, please read through all the relevant articles before you start.

For navigation, here are two sensor types in the car at this stage, a video camera and ultrasonic.  The video camera is the heart of the control and navigation and will be covered in detail in the software article.  The ultrasonics are used to augment the data from the camera.

While the camera feeds a live stream into a Raspberry Pi which runs an OpenCV application to detect the road as well as obstacles, I have not (yet anyway) worked out how to judge distance from this information.  This is where the ultrasonics come in.  I have used these devices (HC-SR01) in previous projects and they are tough and reliable.  The purpose here is to give an indication of the distance to any objects immediately in the car's path.

At present, detecting an object with the ultrasonics will bring the car to a halt until the obstacle is cleared.  This isn't a perfect solution but it gets me started.

When choosing the location for the ultrasonics make sure it has an uninterrupted "view" ahead.  This is essential to prevent false echoes being received.

Mounting the ultrasonic module is not too difficult as there is a fair amount of room behind the front bumper.  However, in my car I needed to mount it PINS DOWN so as to clear the upper mount and the attachment for the bonnet (hood).  It was also necessary to bend the connecting pins back to a straight position so that the are at right angles to the board.  If I left them pointing to the bottom of the module there will not be enough room for cable attachment.

Take your time when mounting the ultrasonic module as it will be quite visible on the finished product, and the plastic is soft and easily over-cut.

Mounting front ultrasonics
If you need to repair this bumper, it would be a great idea to do it before fitting the ultrasonic module permanently.

The ultrasonic module is connected to the Arduino which uses the "NewPing" library to manage the ranging functions.

More challenging is mounting the camera.  I started out with a Logitech HD camera which produced very good results, but is rather large and did not lend itself to mounting on the car.  Recently I switched to a bare 4K module using a Sony image sensor.  Both cameras use a USB interface to the Raspberry Pi.  As with the ultrasonic module the camera must be obstruction free, and take into account the following:
  • It must not have any part of the vehicle in the frame as this can disturb the detection logic.
  • It must be mounted and aligned along the longitudinal centreline of the car.
  • Any reflective parts of the car should be masked to prevent cross-lighting.
  • You will need to experiment with a slight downwards angle to get the best result.
I mounted mine on the front edge of the bonnet (hood) and fed the cable back in through an existing hole where the maker's badge was located originally.  I chose to use a USB extension cable from the Raspberry Pi to the camera because it allows me to detach the bonnet without fiddling with the board.  Some self-adhesive clips on the underside of the bonnet keep the cables out of harms way.

Because it is a bare module it needs to be protected from the elements as well as from impacts  I am also trying to make it look less "stuck on", but it's still a work in progress.

Logitech camera works well, but it looks wrong because of the big bracket
Custom 4K module gives a better fit and image quality

If you look closely at the photo you will notice that the camera module is attached with "Blu-Tak".  This is purely temporary (I hope) whilst I create a better mounting arrangement.

When choosing a camera, if possible choose one with an auto-iris function.  This will help when the car is driving towards the sun.  Also it needs to be mounted securely so as to minimise vibration effects.  Better frame rates can be achieved using a CSI-connected camera such as the dedicated Raspberry Pi unit.  However its shorter built-in cable will make mounting more difficult.  If you can work out how to do it, all the better.

To get a better understanding of the performance, I also added an Allegro ACS712 current sensor.  These are available in a number of current ratings from 20 to 100A.  My unit is a 30A module.  It is very simply to use.  Take the voltage from the sensor, subtract the offset voltage (2.5V) because the sensor can read current flow in each direction and divide the mV  value by 66 to get the DC current in amps.  My motor starts up at around 2.6A and settles down to a steady current when running of around 1.5A.

Further Reading
Part 1 - Introduction
Part 2 - Preparing the car
Part 3 - Wiring Harnesses
Part 4 - Steering
Part 6 - Software
Part 7 - Final assembly and testing

DIY Self Driving - Part 4. Steering

Please Note:  Unlike most of my projects which are fully complete before they are published, this project is as much a build diary as anything else.  Whilst I am attempting to clearly document the processes, this is not a step-by-step guide, and later articles may well contradict earlier instructions.

If you decide you want to duplicate my build, please read through all the relevant articles before you start.

Here is where we get into the mechanical complexities of our project.  If you don't understand how motor car steering works, I would suggest Allan Staniforth's "Competition Car Suspension" and Tony Pashley's "How to build Motorcycle-Engined Racing Cars" but suffice to say my car has a very simplified design which approximates a steering rack in that it translates the rotary movement of the steering column into a linear movement which pushes/pulls the steered wheels about their pivot points.

Instead of a pinion gear driving the rack movement, this car uses a lever arrangement which I have called a "J-Bar" because it's shaped like the letter "J":
While this might look crude it is close to ideal for our purposes.  I am planning on using a very heavy duty servo motor to replicate the function.  I am confident that if I can mount the servo in plane with the rack then I can link it using a custom horn (those little bits which attach to the top of the servo) directly to the rack.

On the subject of servos...

My application is not that uncommon when you look at R/C cars.  However this is rather on the large size, so those little tiny units you can buy for less than $10 and hang off your Arduino are not going to be up to the task.

The "hobby" or "micro" servos typically have a torque value of 9g/cm.  In other words, at the end of a 1cm long arm (measured from the centre of the motor shaft) it could lift a 9 gram mass.

As a simple demonstration, hook one up to an Arduino and run the servo sweep example.  You can stop it in your fingers.  Or if you modify the program to set the position to 90 degrees, it is not difficult to move the horn.  The servo will correct this, but you can gauge the strength.

My car weighs about 8Kg all up.  If it were perfectly balanced this would mean 2Kg sits on each wheel.  As it turns out the weight is biased about 60% to the rear.  This means there is about 1.6Kg on each front wheel.  If my maths is even close to right I think that the 9g servo motor is going to have problems with the 3.2Kg of mass it has to move.  I expect the nylon gears would strip very soon.

Don't despair, there is a solution.  Wading through online catalogues showed there was a range of larger servos available.  I decided to go all-in and get a 20Kg/cm model.  Curiously its cost as about $45, so 4.5 times the price gets me 2000 times the torque.  It also has metal gears which should make it much more durable.
The Apple logo of my MacBook Pro gives you and idea of the size...
I ran some early tests and discovered two things:
  1. I can't move it with my fingers as I could with the 9g model
  2. It needs more current than the regulator on an Arduino can provide, so it will need to come off the main battery.


Before we can install, there is more disassembly.  Fortunately in my car this wasn't too hard.  The steering wheel is attached to the J-bar by a screw and nut behind the steering wheel itself, not in the wheel as you would find in a car.

With the wheel out, there was a bent pin on the other end of the J-bar which needed to be removed and then the whole thing slid out.

Connecting to the steering

Whatever your source of inspiration and self-confidence may be, now is a really good time to call on it, because unless you're using the same car and servo as me, you are pretty much on your own.  I am going to take you through my process as a guide to building your steering.

Position your servo

We need to fix the servo at a centre point in the car which gives full steering movement in either direction for the same servo position either side of centre.  It also needs to be far enough back from the rack part so that small changes in servo position result in small changes in steering angle.  To do this, I borrowed a very sophisticated set of tools.  A roll of string, tape, rule and pencil.
Essential for building a car:  String and 3 kinds of sticky tape
It really helps to have string which is a strong colour contrast when you're doing this.  Firstly mark your centre line and run a length of string down it.  Centre the steering and check that the hole in the bar lines up on this line.
Marking the centre
Now set the steering to full lock on one side and bring a string line from the position back to your estimated servo location.  Repeat for the other side as well.
Your servo goes here!
In reality, what we have laboured to prove over the last 15 or so minutes is that the designer got it right in the first place!  However, it's not wasted because when we mount our servo the hole is not visible.

Making a servo mount

One thing has been missing from the project so far, and that's power tools.  Now we will fix that.  My servo motor has a typical short output shaft so I need to mount it well proud of the body of the car.  Experimentally, I found that a block of D.A.R. pine I had in the garage moved it far enough forward, so I dummied up the mounting using tape to hold things together.
Trial mounting
You might also notice I had to relocate the wiring coming out of the pedal compartment to clear my mounting block.  This meant cutting down the old tie point and sanding it level with the rest of the body.

I fitted a second timber to each side of the servo motor to enclose it and prevent it from twisting under load when the car is on its wheels.

With that part done, I attached a piece of aluminium angle to the long sides of the block and screwed the servo to that using M3 screws and nuts.  Since it's going to be really hard to get to later, I added some "locktite" thread adhesive for peace of mind.
The finished mounting
The completed assembly could be painted to make it look nicer, but I didn't bother since I don't expect this car to roll over all that often.  When you're happy with everything it can be fixed into place with a series of screws through the body of the car.
Mounted hard to the body

Some might scoff at using wood for this project, but remember Morgan were still building entire chassis from wood well into the 1990's and people consider them to be highly desirable...

Link Arm

Now we need to connect the servo to the steering mechanism, and again some imagination will be required.  I deliberately servo below the steering (the J-Bar was above it) because I did not want the lowest point on my car to be a rather expensive motor.  To link the two I folded a "Z" shaped arm using a piece of "Make-A-Bracket" steel from Bunnings.  The purpose of the Z is threefold:
  1. It makes up for the height mismatch between servo and steering
  2. Compensates for any misalignment.
  3. It absorbs shocks transferred back from the wheels which might damage the servo.
Trial-fitting the steering link
Once you have determined the sizes for everything, drill out two of the holes in the servo horn to take M3 screws.  You will probably also need to drill the arm to suit.  The cable(zip) ties in the photo were simply to hold things in place while I shaped everything.  They will definitely not hold it securely enough to function. (I tried).
The finished product
I have decided to offload steering control from the Raspberry Pi onto an Arduino Uno board, partly because coding servo functions is something I have done before on that platform and because the Pi is busy enough with image processing.

To test everything, I wrote two small sketches:  "servo_test" cycles the steering from side to side, and is very useful for impressing sceptics.  "servo_calibrate" allows you to step the steering from side to side and find the limit positions.  It will then save these values to the on-board EEPROM for the main steering controls to use.

With all this done, you now really have something to show for your efforts.  You can see mine in the video here:

Finally, put it back over onto it's wheels and take more photos of your hard work!

Further Reading
Part 1 - Introduction
Part 2 - Preparing the car
Part 3 - Wiring Harnesses
Part 5 - Sensors
Part 6 - Software
Part 7 - Final assembly and testing


Wasting your and my time

I had a really interesting experience recently which I hope might enlighten others as much as it did me: I was approached (via LinkedIn) by ...