My new favourite tool

-

Life is much easier when you can see what's going on

Sometimes I think these machines are getting a little too autonomous for my liking...

Right now, Auric is being a downright pain the in the rear.  It just seems to be uncooperative.

It all started when I observed that the directional control was erratic.  Given that the steering process is a stream of adjustments based on image processing, it is normal that there will usually be a mismatch in the speeds of the motors on each side.  But when the camera is shown a static image of a receding path I would expect it to steer more or less straight, or to make only small adjustments.  This however, was not the case.

 More digging hinted that not all of the commands sent from the main controller to the motor controller were being processed.  This boiled down to one of three possibilities:
  • The main controller was not sending the commands;
  • The motor controller was not receiving all the commands being sent; or
  • The motor controller was not acting on some of the commands being received.
I was able to eliminate the first option by connecting a serial interface to the output of the main controller and comparing what hit the wire with the logging output.  They matched, so I ruled out the controller and it's code as being the culprit.

 This left the blame on the motor controller.  Unfortunately since it has only a single serial port, debugging was limited, and it does operate in what is basically a tight loop.  With no console available to me, I modified the controller code to flash the onboard LED (LED_BUILTIN) each time a message was received.  Firing up again, the LED was flickering constantly so that looked OK.

 I then modified the code again to flash the LED each time it executed a command, and the result was a much slower flicker. Aha!

 Or at least so I thought.  The commands follow a simple formula:

direction:pwm_speed

So R:512 would mean set the PWM value on the right side to 512, and the motor controller will respond with "0" for OK or "1" for an error.  If I ran minicom on the main controller, and entered the commands manually, it worked fine every time.

 Then I wrote a smaller test program using libserialport to send a command to each motor.  What was curious here was that consistently only the first command was executed!  Now we are getting somewhere, but why?

 I suspected that the line speed might be a problem, so I lowered the serial rate to 9600 baud, but the same symptoms was there.

Time for the big guns

A while ago I purchased a BitScope Micro through a special deal from TronixLabs.  I hadn't really put it to serious use, but if ever there was a time, this was it.  The BS05 (as it's known) has a range of software from basic oscilloscope through logic analyser and protocol analysis.  It also has tiny little probes which can get between pins.  The software also works under Windows, MacOS and Linux.  As a bonus, they are a local company.  Even better as far as I am concerned.

Images courtesy of BitScope
With everything running, I was going to use the protocol analysis features to decode the serial stream, but I didn't need to.  Modifying serialtest to loop through it's transmission, and report the number of bytes sent, I had one channel on the transmit from the main controller, and another channel on its receive.

This little device has helped protect my sanity!
I could see the blocks of data on both channels, and the size of the blocks seemed about right.  But wait!  The relative positions of the blocks was changing, sometimes there was a response in between the commands, other times responses seemed to overlap with a command.

This was the break I needed (See what I did there?)  I modified serial test yet again to report when a frame was received.  Now I could understand what was happening.  The RockChip UART has a deep transmit buffer, so when we wrote a command, the write was acknowledged instantly as having been sent and it was ready for the next command.  As a result, the commands would just keep arriving in the motor controller's RX buffers faster than it could deal with them and because there was no handshaking, some commands were simply lost.

The tiny probes make it easier to grab the right part on a PCB
I can't change the buffer behaviour, and I have no capacity to add hardware handshaking between the two controllers, so the solution is to make the main controller actively wait for the acknowledgement from the motor controller before sending the next command.  By making this change, I could ramp the data speed back up and things held together fine.

It was definitely a software problem of my own making; but the symptoms were pointing to anything but that.

The odd thing was that this same code model had held up just fine in the previous car even though my poor code was present, so what is the difference?  There's a explanations I can offer.  I am not yet sure which is the most correct:
  • Because the Raspberry Pi is slower, perhaps it simply didn't manifest;
  • There may be a difference in either the UART driver code or the UART itself between the two boards;
  • Maybe it was there, but because the of way the steering mechanism worked, it just wasn't obvious.

So what did I learn?

Sometimes a console just isn't enough, and no console is even harder
Debugging embedded systems has unique challenges because you can't always have the controller report what it's thinking.  Using a spare GPIO port to indicate state will usually cost very little in performance, but might be all you need.
Don't assume.  Check
Maybe if I had been less rushed in writing the code in the first place, I would not have spent a day trying to solve this, but I fell into the trap of "what the heck, it works!"
The right tools make all the difference
I have a couple of perfectly good CRO's (Cathode Ray Oscilloscope), but they are big and bulky and I couldn't be bothered to drag them out.  The BitScope on the other hand, is a little USB device, which makes it portable and powerful for my needs.

There is a lot of functionality I have yet to explore, but I am sure that with Auric's bad attitude I am going to get the opportunity!

Want to help?

If you would like to make a financial contribution to the project, please go to my Patreon Page at:




Please note:  I am not associated in any way with BitScope Designs, MetaChip Pty Ltd, or their resellers.  I am just a very happy customer, and sought their permission before writing.










Comments

Post a Comment

Popular posts from this blog

The QYT KT-7900D

-
Image
But is it any good? As you might have noticed, I am now a freshly licensed radio amateur .  As such I do need a transceiver or two in order to get on the air.  This is the story of one of my new radios. I have a limited discretionary budget to use for my new hobby, and I have to say from the outset that my lovely wife has been very supportive.  As such, I want to try and get the best value for our money.  One evening I stumbled upon this little item on Banggood for a little over $A150 including shipping.  Covering both 2m and 70cm bands with a claimed 20W output it seemed almost too good to be true.  I bought one. About ten days later a tiny package arrived for me.  At just 98mm x 43mm x 126mm the photos don't convey quite how small it actually is.  The entire case serves as a heatsink so it is heavier than it looks, but certainly not bulky. The standard power cable comes with a cigarette lighter plug.  As I intended to run it from home usin...
Read more

Recording weather with Arduino, Elasticsearch and Kibana

-
What do half a Bunnings weather station, an Arduino, a NodeMCU (or 3), Elasticsearch, a fishtank and swimming pool have to do with each other? Nothing unless you like to monitor and measure pretty much everything around the house.  I do, and ever since I learned to fly, I have had a fascination with the weather (I am told boaties suffer from a related affliction). I could just buy a weather station, but where's the fun in that when for the same amount of money (OK, maybe a little more) I can make my own and bore people senseless when I go on and on about it? Actually, there is a practical purpose to this as well.  I wanted to teach myself about Elastic and Kibana.  Now you can wade through dry tutorials, but nothing in my opinion beats working with real live data.  Especially if the data is of interest to you. The design What do you want to measure? This is actually pretty important (not to mention obvious), but before embarking too far; take a...
Read more

Exploring Solar Power - Let there be smoke!

-
Image
Last time I introduced my new solar power project.  By the end of that post we were monitoring the power flow from the solar panel to the battery in a simple test environment.  This time we're going to dive in and start trying to control the charge power. What's MPPT? Maximum Power Point Tracking (or Transfer , depending on whom you ask) is a condition where we can transfer the highest power from the solar panel into the battery.  The key here is power, rather than voltage or current on their own.  Remember that power is the product of voltage and current, so it is possible that you will find that perhaps higher current and lower voltage (or vice versa) is the most efficient. What makes this a little problematic is that the peak power point varies constantly depending on the battery state of charge and temperature, so it is not a set and forget operation. This is why the first thing we did was to build a power monitor.  Now we can determine at any point of time...
Read more