Automated Testing on Arduino

11-9-2013

Changed the title as the “experimental part” is not applicable anymore. Built analogWrite, found a few bugs caused by the analogRead() ( so no problems in the preliminary release on GITHUB). Couple of days of testing and I expect to be able to publish Release 1.0 on GITHUB this weekend.

10-9-2013

Added analogRead() today !! Not on GITHUB yet. I suspect that analogWrite() will be added tomorrow. While playing with the Arduino I was thinking about the idea of having Excel itself send the testcases directly to the Arduino and put the results back in another sheet. This will allow huge testsets if required and checks can be programmed in the result sheet (with for example red Flagging if there is a difference). This would make it even more possible to reproduce tests without a memory limit.

It is still an idea. But knowing myself……

9-9-2013

To get some feedback I have put a preliminary version on GITHUB: https://github.com/nicoverduin/AutoTest

Please give some feedback.

Dit some more testing today. Normally I work with ECLIPSE with the Arduino plugin. The nice thing about it is that if I generate new files from the excel sheet, they are automatically updated if they were already open in ECLIPSE. The Arduino IDE does not do that. To get the lates version you will have to close the sketch and reopen it. I can’t fix that unfortunately. Just to let you know.

Also did some benchmarking on the code. An average digitalRead takes about 75 microseconds to process which is longer than a normal digitalRead. Not bad though I think. Most of the time is lost in the Serial Output. Even at 115200 baud.

And do not forget to import the AutoTest library in the Arduino IDE of Eclipse 🙂

8-9-2013

Well almost ready to go. Have it working  now. I define all the pins in excel and generate the testfiles. These files have to be in the same folder as the sketch so put the excel file there and it will put themin the right place. Then just compile and upload to the Arduino. And you are done. Just connect to the Arduino with a serial terminal and the testsets are filling the pinreads and all is output to Serial. Here is an  example of the input excel sheet, the Serial output and the output put back in the excel with some formatting. As you can see there are a lot of empty reads (polling for buttons) So I think I will add an option to not send output of a Read if the value is HIGH or LOW (selectable). There is already an option to add extra fields in the Serial output (like counters in this case) through a user function. I will explain all in the documentation when ready.

live case testset

serial output

live case testset results

29-8-2013

Made a lot of progress these last few days. Built an Excel file enabling the creation of testsets. The Excel sheet also contains a VBA macro generating a number of include files containing the data of the pins and their descriptions, the actual testsets and some size parameters. De generated include files are added to the sketch if it contains the following statement at the top: #include <AutoTest.h> and the library is included to the sketch. From then on when the program executes all testcases are run through the Arduino where digitalReads and digitalWrites are replaced with the testcases. The output is sent to the Serial(CSV format) allowing it to be fancied in Excel. The testcases and pin descriptions are stored in Flash Memory to minimize SRAM usage. There is of course a penalty due to more cpu work but RAM is limited and Flash is usually sufficient. But most important is that the requirement of just one statement (the include) will enable/disable the automatic testing and the possibility to rerun all the testcases again without extensive pushing of buttons, all kinds of additional Serial statements etc. Leave the include out and the sketch is in its original state.

The Excel sheet has the flexibility to be extended to more pins and testcases (max. 10000!! but you will probably be out of memory before that :)).

The diagram below shows the whole development / testing cycle.

Process Flow

Some sample files

Excel sample

Sample testcases Sample pinheaders

25-8-2013

When developing software for the Arduino where lots of logic sequences take place AND timing is not an issue I was disturbed that to test the Arduino a lot of buttons need to be pushed, led responses written down and verified and if the Software changes I’d have to do it all over again. Working for a project for a customer I added lots little pieces of code to that were activated at compile time when using the #define DEBUG statement. What happened was that every time a button changed or an output changed I called a routine that would output all the data to the Serial out as CSV format and copy it to Excel for neater formatting. Also in the code I added the testcases which would be executed in sequence. It works but it does not make the code look better.

debug

debug_1

Like I said it works but that is about all. I needed a way to make this easier. The idea I have is to reroute readDigital and writeDigital. By rerouting, these statements I now exactly when a digital pin read or written. However you cannot simply create new routines and do all the extra logic I would like to do as at link time, the linker will tell you that the function exists twice (once in your code and once in the original Arduino core libraries).

I discussed this issue with some of my fellow enthousiasts at www.Arduino.cc at their forum. The responses they gave were to write new read and write functions and call those. That was certainly not my idea. Actually I want to put in one (1) statement like for example “#define DEBUG” and “#ifdef DEBUG #include “DEBUG.h” #endif” and that is it. And then the compiler would let me do my own reads and writes without the program knowing the difference.

The solutions I eventually came up to was to create for myself in the DEBUG code both digitalRead and digitalWrite but make them  “inline”. The only thing these routines would do is call a member function of a Class which does the actual processing of the reads and writes.

The program would not know the difference as it still performs a digitalRead and digitalWrite and my new Debug class will handle the rest. The class will include upon compile time a whole bunch of testcases (and maybe in the future the expected results) and will during execution time execute each step one by one. And every time a pin changes (i.e. digitalRead and digitalWrite), output in CSV format will be sent to Serial allowing it to be processed further.

Then when I change the code, I can just run all the tests again with the change of one (1) statement.

I have tested this as a Proof of Concept and my theory works so time to start designing Automated Testing for the Arduino.

 

Geef een reactie