My new favourite tool

Life is much easier when you can see what's going on

Sometimes I think these machines are getting a little too autonomous for my liking...

Right now, Auric is being a downright pain the in the rear.  It just seems to be uncooperative.

It all started when I observed that the directional control was erratic.  Given that the steering process is a stream of adjustments based on image processing, it is normal that there will usually be a mismatch in the speeds of the motors on each side.  But when the camera is shown a static image of a receding path I would expect it to steer more or less straight, or to make only small adjustments.  This however, was not the case.

More digging hinted that not all of the commands sent from the main controller to the motor controller were being processed.  This boiled down to one of three possibilities:
  • The main controller was not sending the commands;
  • The motor controller was not receiving all the commands being sent; or
  • The motor controller was not acting on some of the commands being received.
I was able to eliminate the first option by connecting a serial interface to the output of the main controller and comparing what hit the wire with the logging output.  They matched, so I ruled out the controller and it's code as being the culprit.

This left the blame on the motor controller.  Unfortunately since it has only a single serial port, debugging was limited, and it does operate in what is basically a tight loop.  With no console available to me, I modified the controller code to flash the onboard LED (LED_BUILTIN) each time a message was received.  Firing up again, the LED was flickering constantly so that looked OK.

I then modified the code again to flash the LED each time it executed a command, and the result was a much slower flicker. Aha!

Or at least so I thought.  The commands follow a simple formula:


So R:512 would mean set the PWM value on the right side to 512, and the motor controller will respond with "0" for OK or "1" for an error.  If I ran minicom on the main controller, and entered the commands manually, it worked fine every time.

Then I wrote a smaller test program using libserialport to send a command to each motor.  What was curious here was that consistently only the first command was executed!  Now we are getting somewhere, but why?

I suspected that the line speed might be a problem, so I lowered the serial rate to 9600 baud, but the same symptoms was there.

Time for the big guns

A while ago I purchased a BitScope Micro through a special deal from TronixLabs.  I hadn't really put it to serious use, but if ever there was a time, this was it.  The BS05 (as it's known) has a range of software from basic oscilloscope through logic analyser and protocol analysis.  It also has tiny little probes which can get between pins.  The software also works under Windows, MacOS and Linux.  As a bonus, they are a local company.  Even better as far as I am concerned.

Images courtesy of BitScope
With everything running, I was going to use the protocol analysis features to decode the serial stream, but I didn't need to.  Modifying serialtest to loop through it's transmission, and report the number of bytes sent, I had one channel on the transmit from the main controller, and another channel on its receive.

This little device has helped protect my sanity!
I could see the blocks of data on both channels, and the size of the blocks seemed about right.  But wait!  The relative positions of the blocks was changing, sometimes there was a response in between the commands, other times responses seemed to overlap with a command.

This was the break I needed (See what I did there?)  I modified serial test yet again to report when a frame was received.  Now I could understand what was happening.  The RockChip UART has a deep transmit buffer, so when we wrote a command, the write was acknowledged instantly as having been sent and it was ready for the next command.  As a result, the commands would just keep arriving in the motor controller's RX buffers faster than it could deal with them and because there was no handshaking, some commands were simply lost.

The tiny probes make it easier to grab the right part on a PCB
I can't change the buffer behaviour, and I have no capacity to add hardware handshaking between the two controllers, so the solution is to make the main controller actively wait for the acknowledgement from the motor controller before sending the next command.  By making this change, I could ramp the data speed back up and things held together fine.

It was definitely a software problem of my own making; but the symptoms were pointing to anything but that.

The odd thing was that this same code model had held up just fine in the previous car even though my poor code was present, so what is the difference?  There's a explanations I can offer.  I am not yet sure which is the most correct:
  • Because the Raspberry Pi is slower, perhaps it simply didn't manifest;
  • There may be a difference in either the UART driver code or the UART itself between the two boards;
  • Maybe it was there, but because the of way the steering mechanism worked, it just wasn't obvious.

So what did I learn?

Sometimes a console just isn't enough, and no console is even harder
Debugging embedded systems has unique challenges because you can't always have the controller report what it's thinking.  Using a spare GPIO port to indicate state will usually cost very little in performance, but might be all you need.
Don't assume.  Check
Maybe if I had been less rushed in writing the code in the first place, I would not have spent a day trying to solve this, but I fell into the trap of "what the heck, it works!"
The right tools make all the difference
I have a couple of perfectly good CRO's (Cathode Ray Oscilloscope), but they are big and bulky and I couldn't be bothered to drag them out.  The BitScope on the other hand, is a little USB device, which makes it portable and powerful for my needs.

There is a lot of functionality I have yet to explore, but I am sure that with Auric's bad attitude I am going to get the opportunity!

Want to help?

If you would like to make a financial contribution to the project, please go to my Patreon Page at:

Please note:  I am not associated in any way with BitScope Designs, MetaChip Pty Ltd, or their resellers.  I am just a very happy customer, and sought their permission before writing.

Introducing Auric

Looks like something an evil genius should have, right?
In my last update, I mentioned that the "Auric"* chassis was being used as part of my self-driving experiment.  Now it's time to unveil it properly.

Auric is a T-900 tank chassis made by SZ-Doit.  It's not cheap, but you get a lot for your money:
  • The chassis itself, in a choice of anodising colours
  • Twenty (20) idler wheels
  • Four driven wheels
  • Plastic track
  • Bearings
  • Motors (9V with Hall-Effect sensors, or 12V without)
  • 16550 Battery Holder (no batteries)
  • Screws, Nuts. Washers
  • Allen keys
The only problem is that there are no instructions supplied.  SZ-Doit have a Wiki on GitHub, but for the most part you are referring to photographs of the finished product in order to assemble the machine.

Also available is a motor shield which is used with a NodeMCU module.  Since this was featured in the company's build I added one of those as well.

Some build notes

This thing is big!  It is close to 50cm long and 30cm wide, so you need plenty of space when working on this chassis.

It's normal to have two idler wheels left over when building the quad-track version.

The motor wires are not long enough to reach the controller.  I added a pair of terminal strips to the underside of the chassis, then connected from there to the controller.

Take your time lining up the drive wheel with the idler wheels.  If they are not lined up well, there will be a tendency to throw tracks.

There are two ridges on the inside of the tracks.  These go inside the the idler wheels.  There is also a notch closer to one edge of the track.  This faces out.  If you keep these things in mind, lining up the track on the drive and idler wheels becomes easier to understand.

With the mechanical side done, I loaded the demo software onto the NodeMCU, and I could not get it to work, so I didn't bother with the BlueTooth control either.  My GitHub repo contains a simple sketch that does work, and it would give you a starting point.

The shield uses an L293 driver which is different in operation to the Uno + L298 I usually run.  One thing I noticed was the the motor speed is much lower on the NodeMCU + L293 driver, so you do need to take this into account.

Auric chatting with my Logic Analyser

A Guided Tour

In a more-or-less central position is the new controller.  I like the Raspberry Pi, but I these projects have simply outgrown their capabilities.  After a lot of evaluation, I settled on the NanoPC-T4 from FriendlyArm.

Images courtesy of FriendlyElec/FriendlyArm

This board uses a RockChip RK3389 SoC, it's still ARM, and runs Ubuntu.  It's biggest advantages for me were the processor speed, and 4Gb of RAM.  It also supports adding an NVMe storage device, but I have not enabled that just yet.  There is also the usual WiFi (2.4/5G), BlueTooth, HDMI, etc.  As a bonus, the 40-pin I/O header is pin-compatible with the Raspberry Pi.

At the rear of the chassis is a NodeMCU attached to an SZ-Doit Motor Shield.  A simple serial connection (115200bps) connects the two devices together, and they use the same basic command/response protocol defined in my earlier self-driving experiments.

At the front are the two visually most interesting features.  A forward-facing camera.  You've seen this one before.  It's connected to the T4 and drives the image-processing functions.

Long exposure shows the LIDAR in operation

The blue cylinder is the X4 LIDAR unit from ydlidar.  It generates 720 samples per rotation, thus 2 samples / degree, which is rather high resolution for an inexpensive device ($US99)

To mount the LIDAR and forward camera, I had to make a bracket from two back-to-back aluminium angle pieces.

LIDAR is great for knowing what is around you, but it is even more useful if you know were you are, and where you are going at the time.  Two small modules make this possible.  The first is an HMC5883L I2C Compass/Magnetometer module.  This provides an heading record which is saved along with the LIDAR data.

Also included is a uBlox GPS receiver.  This device is only for logging at this point, but in the future I want to us it for navigation as well.

All this hardware needs a considerable amount of power.  The main processor needs 12V, which is a bit annoying.

Battery size and weight is always an issue with mobile devices, and Auric is no exception.  Fortunately RC cars have faced similar problems, so I am presently using a 4C LiPo battery pack.

This provides 7.4V, so for 12V I used a small switch-mode regulator to run the NanoPC-T4.  The motors are slightly under voltage, but I don't see this as a big issue for now, anyway.

Battery goes underneath
That's enough of the hardware for now.  In the next update, I will look at how the self-driving software was integrated into this platform, as well as how few changes were actually needed.

Want to help?

If you would like to make a financial contribution to the project, please go to my Patreon Page at:

*Auric gets it's name from Auric Goldfinger, a criminal genius obsessed with gold, in the James Bond novel and film "Goldfinger".  Since the chassis was a golden colour, it just made sense at the time.

Wasting your and my time

I had a really interesting experience recently which I hope might enlighten others as much as it did me: I was approached (via LinkedIn) by ...