Arduino: Super Graphing Data Logger

The intensity of natural light in my basement.
Sections:

  1. Introduction
  2. The Results
  3. How to Make One For Yourself

Introduction

What is the Super Graphing Data Logger (SGDL)? It is an Arduino project that integrates data logging and the graphing of this data online using little more than an Arduino with the appropriate shields and sensors.   It differs from similar projects in that it doesn’t require a separate server or system to collect the data or to run script for the actual plot. Between the Arduino and the user’s browser, everything is taken care of.

If you just want to dive right in, the code is now posted on GitHub: https://github.com/evjrob/super-graphing-data-logger

Some time back I came across this neat javaScript based library for plotting and graphing called Highcharts JS. It didn’t take long for me to realize that charting with javaScript is very convenient for projects in which the server is limited in it’s capabilities, such as when using an Arduino with the Ethernet shield. Since the user’s browser does all the heavy lifting, the Arduino only needs to serve the files which is something it is perfectly capable of. This is especially true now that the Ethernet and SD libraries included in 1.0 support opening of multiple files simultaneously amongst other things. Thus the use of Highcharts allows us to create beautiful interactive charts based on data logged by the Arduino using nothing but the Arduino (and your browser, and a public javaScript CDN).

The Results

The best way to appreciate the final product is to actually play with it. While I’m not going to open up my home network and Arduino to the big wide internet, I have mirrored the pages and datafiles it produces on the webhost I used for my Has the World Ended Yet? project. You can find them here. These won’t be updated with new datapoints like the actual Arduino version will be, but they should at least give a fair impression of how the project looks and feels without the need to actually implement it.

For those who are unsure what they are looking at, I’ll offer a quick interpretation:

The list of data files available for graphing.

The list of data files available for graphing.

Going to the above page, we see that we are presented with a very basic list of the data files that can be selected from. Clicking any of them will cause  the graph for that datafile to be loaded (much more quickly than the Arduino can manage).

A graph for the first week of data collected.

A graph for the first week of data collected.

This chart for the 25-12-12.CSV file is already complete, and won’t have any new data added to it in the future, because the files for subsequent weeks have already been made. There is a lot to see though. The two data points that are at 1000 on the y-axis are from when I pointed a bright flashlight directly at the photo sensor. All of the data points between 300 and 400 on the y-axis are the result of the basement lights being on. The abnormally large gaps in the data are periods when the Arduino was powered off because I was still tweaking and developing it. Finally, the short humps that occur everyday are the result of natural light coming through one of the basement windows. By zooming in on one of them, we can see even more detail:

The intensity of natural light in my basement.

The intensity of natural light in my basement.

The first thing we notice is that the levels rise from zero to about 65 before falling and levelling out at close to 35 for two hours. This is followed by a another small increase before it ultimately decreases down to a value of ~10 where it levels out. That middle valley where the light levels are equal to 35 is due to the shadow cast  on the basement window by our neighbour’s house to the south of us. The levelling out of the light intensity at 10 after all the daylight has disappeared is because a light out in the hallway is usually on in the evening. It is eventually turned off for the night, causing the light levels to drop to zero where they will usually remain until the next morning. I must admit, I’m impressed that the cheap $1.00 photoresistor is capable of capturing this level of detail, and that these trends are so easily interpreted from the graphs.

How to Make One For Yourself

All of the code below is now conveniently hosted on GitHub (https://github.com/evjrob/super-graphing-data-logger) so you no longer need to copy and paste it if you don’t want to.

To replicate this project, a few things are necessary. You’ll obviously need an Arduino capable of connecting over Ethernet and storing files on an SD card. In my case, this is achieved through the use of an Uno with the Ethernet shield. Presumably an Arduino Ethernet model will also work fine, though I have not personally tested it. Other non official Ethernet shields and SD card adapters may also work if they use the same libraries, though I make no guarantees. For the more adventurous, it may be possible to adapt my code to achieve the same functionality using a Wifi shield. You will also need a data source of some sort. For my project I chose to use a very cheap photoresistor, which I rigged up on a small perf board to plug directly into the 5v, gnd, and A0 pins of my Arduino (or more precisely,  the headers on the Ethernet shield). It is set up in such a way that the minimum recordable light intensity is zero, while the maximum is 1024.

The photo sensor board fits like a charm.

The photo sensor board fits like a charm.

DSCF2931

One header is bent to reach A0.

The pins on the male headers don’t quite line up, so I intentionally used extra long ones and added a slight S-curve to the one that goes to A0. This can be seen in better detail above. For those who are interested, the circuit is very simple:

The circuit.

The circuit.

Before we get started, we need to make sure our SD card is good to go. It should be formatted as a FAT16 or FAT32 filesystem, the details of which are available on the official Arduino website. Once that is done, we need to ensure two things are present in the root directory of the card: the HC.htm file, and a data/ directory for our datafiles. The data directory is easily made with the same computer that was used to format the card provided one has an SD card reader of some sort. The HC.htm simply consists of the following code:

You will need to edit this file first to make sure it points towards the preferred  location of your highcharts.js files. You can leave this as the public CDN: http://cdnjs.cloudflare.com/ajax/libs/highcharts/2.3.5/highcharts.js, change it to point towards your own webhost, or it can even be on the Arduino’s SD card (this will be slow). It is not necessary to create a datafile before hand, the SGDL sketch will take care of that when it decides to record its first data point. Before we get that far though, it is necessary to make sure we have configured the EEPROM memory for the SGDL sketch. This is very easily accomplished using a separate sketch, which I have called EEPROM_config. This sketch (along with SGDL itself) requires an extra library called EEPROMAnything, which needs to be added to the Arduino’s libraries folder wherever one’s sketchbook folder is. While you’re at it, you should also add the Time library which we need for SGDL.

I have intentionally commented out the write line so that no one writes junk to the EEPROM by accident. While the EEPROM has a life of ~100,000 write cycles, I’d rather not waste any of them. Please review the sketch carefully and ensure you’ve adjusted it accordingly before uploading it to the Arduino. The most important thing is to ensure that your newFileTime is something sensible (in the near future most of all).

Now that that’s all taken care of, we’re ready to get SGDL all set up! The code will need a few adjustments for your own specific setup, mostly in regards to the Ethernet MAC and IP addresses. I trust that anyone making use of this code already knows how to configure their router to work with the Arduino, and that they can find the appropriate local IP address to update this sketch with. You may also wish to change the timeserver IP address to one that is geographically closer to yourself.

I currently have my code set up to make a measurement every 10 minutes, and to create a new data file every week. You are welcome to change those parameters, just be aware that the current data file management names files using a dd-mm-yy.csv date format, so the new file interval should be at least 24 hours. Another concern, is that the shorter the measurement interval and the longer the new data file interval is, the larger the files will be. Because the Arduino is not especially powerful, this will have consequences for the loading times of each chart.

Posted by Everett

75 comments

Awsome project. I will try it out.

Hi,

i’ve problem to get it work. What about “Super Graphing Data Logger – EEPROM config” ? Is this a.cpp file for the lib. Don’t know. Hope you can solve my issue

No the eeprom config is a file you need to flash to the board and run once before flashing the main file. It does the initial configuration to the eeprom for the main sketch.

Thanks for the reply. But after testing your codes, how can i “undo” the eeprom configuration? May be i have overread this part…..

Hallo. I have problem with main list of data files.. i have .csv files in /data directory, but main web page is empty. can you help me? measuring work fine

Unfortunately I’m on the other side of the world from my arduino at the moment, and will stay that way for about another 10 days. If you can wait until then, I’ll have a look at it.

Okay. Mine is up and running again, and everything seems to be going okay. I’ll try my best to replicate your problem, but I might need some more details about your exact setup. First of all, have you made sure that the file HC.htm is in the root directory of the SD card?

Hi Jozefiel, Is your problem solved?

I got the same problen.
The .csv file is made in the data directory but they don’t show up in the web page.

So I can’t sellect them to graph.

the data seems to be OK. I also put some other .txt files on the card, but he doesnot see them either.

Marcel

Sorry replied to the wrong subject below (I can’t remove it). Now on the right place.

Perhaps good, to give some extra information to help solving the problem.

changing the pad to the file by hand in the HC.htm gives a good graph.

I replaced <var dataFilePath = "/data/"+getDataFilename(query);
for ;

So the csv file is OK and the HC file itself seems to work.

The serial monitor gives the next information:
card initialized.
1384118384,1023
1384118391,1023
1384118398,1023
GET / HTTP/1.1
1384118405,1023
1384118412,1023
1384118419,1023

Perhaps this gives an idea where to search
Marcel

marcel
do you have solved this problem?

Daniel Sovino (@dsovino)

Excelent tip on PROGMEM use.

I was struggling on memory doing something similar with uno, sd, ethernet, 1 DHT11 & 1 TMP36, DS1307RTC instead 4 timing.

Without using progmem i wasnt able to debug (had to comment every single serial.print to compile).

good article, thanx!

Hi, your sketch is awesome, but I have a problem. I copied your main sketch and tryed to upload it, but I get the error that in line 266 ‘day’ was not declared. What must to be changed?

Sorry, it was my fail. I got it uploaded, but there are no csv files on the sd card and no entries in the list… There is only th HC file in the directiory, do you have a tip for me? ๐Ÿ˜‰

You might want to run the eeprom config sketch and make sure that the newFileTime parameter you choose is reasonable. It will then take about 10 minutes for the first data point to be created, along with the data file. Other things to look at are whether the arduino is properly connecting to the NTP time server, since it skipsmaking data points (and the file for the first data point) if it can’t connect. Another possibility is if there is something wrong with the file system on the SD card, and the arduino cannot write anything to it.

Enabling the serial monitor, and maybe adding your own serial debug comments will really help to pin point exactly where it’s hanging up.

I think this looks great,

I was looking for something lijke this, to use it for multiple temperature sensors for my wood stove with buffer, to measure the temp at different hights in the buffer.
I can change the code to measure my one wire temperatures I have something like that running at the moment it writes to SD, but I have to take out the the card to make graphs.

I would love to see the graphs online.

I think I can change your sketch to collect the data in the csv file, but I’m not sure how to get the HC.HTM working with more lines. Is this easy for you to awnser?

Thanks, Marcel

It shouldn’t be too hard, once you change the sketch to add more columns to the .csv file you just need to modify HC.htm to get this bit:

// Load data asynchronously using jQuery. On success, add the data
// to the options and initiate the chart.
// http://api.jquery.com/jQuery.get/
jQuery.get(dataFilePath, null, function(csv, state, xhr) {
var lines = [],
date,

// set up the two data series
lightLevels = [];

// inconsistency
if (typeof csv !== ‘string’) {
csv = xhr.responseText;
}

// split the data return into lines and parse them
csv = csv.split(/n/g);
jQuery.each(csv, function(i, line) {

// all data lines start with a double quote
line = line.split(‘,’);
date = parseInt(line[0], 10)*1000;

lightLevels.push([
date,
parseInt(line[1], 10)
]);

});

options.series[0].data = lightLevels;

chart = new Highcharts.Chart(options);

dealing with the extra columns after the line = line.split(‘,’); as a separate series presumably. So just add your additional series for each column up in the chart initialization, then just add some code to push these extra columns which will be stored in line[2], line[3], etc to those new series that you’ve made.

Okay,

Thanks for the response.
I hope I have time to try this at the weekend.
When I succeed, I will let you know.

Marcel

Perhaps good, to give some extra information to help solving the problem.

changing the pad to the file by hand in the HC.htm gives a good graph.

I replaced <var dataFilePath = "/data/"+getDataFilename(query);
for ;

So the csv file is OK and the HC file itself seems to work.

The serial monitor gives the next information:
card initialized.
1384118384,1023
1384118391,1023
1384118398,1023
GET / HTTP/1.1
1384118405,1023
1384118412,1023
1384118419,1023

Perhaps this gives an idea where to search
Marcel

What does the URL look like when you click on one of the links from the homepage with the file list?

Hi Everett,

My post abouve was on the wrong spot, it was a reaction on the problem of Jozefiel (I have the same problem).

The problem is, I don’t get a file list on the site.
changing the pad to the file by hand in the HC.htm gives a good graph.
I replaced <var dataFilePath = "/data/"+getDataFilename(query);
to <var dataFilePath = "/data/10-11-13.csv");

Then everything worked and I get the nice graph.

So the problem is why does my files not show up on the site?, I only see

Marcel

So are you going to the URL, eg http://192.168.1.100 or to one like http://192.168.1.100/data/ ?

Because if you go directly to the data directory url, noting gets loaded. The arduino isn’t functioning as a full file server, and doesn’t list directories in that way. It only loads that file list from the main page if you go to the root IP address for the arduino.

In my browser I have “http://192.168.178.34/” without the /data

In the browser I see “View data for the week of (dd-mm-yy):”
nothing else.

after refreshing the page, the serial monitor gives:
GET / HTTP/1.1

I’m trying for manny hours now. Something strange hapened this evening.

To see where the program is, I try to print lines for trouble shooting.

File workingDir = SD.open(“/data”);
client.println(“”);
while(true) {
File entry = workingDir.openNextFile();
if (! entry) {
Serial.println(“no more files”);
break;
}

I added the line ” Serial.println(“no more files”); ”

In the first run I finaly saw my files.
After removing the line it didnot work again.
When I put the line back it did not work again!!

On the serial monitor I see

“card initialized.
1384289043,1023 ”
after pressing refresh in the web page, the next message shows up:

“GET / HTTP/1.1
no more files”

I thought perhaps it is something with timing, so i tried some delays after the serial.print line.
doesn’t work…..

I also tried an other SD card, no results.

any tips?

Uh oh. Non replicable problems in software are the worst kind. Out of curiosity (and grasping at straws for a source of the problem) what version of the arduino IDE are you using? Also in the same line of thought, what make of arduino are you using. If it’s old enough it may be running into out of memory errors at that point in the code. If there’s any other interesting differences like those that you can think of, then feel free to mention them, and hopefully we can track down the source of your issues.

Hi Everett,

ypur questions made me think. First of al I have an arduino R3 with an ethernet shield.
After an upload I get a measage “Binaire sketch-grootte: 26.400 bytes (van een 32.256-byte maximum)” Then I thought about your saying about memory.
I tried to make it smaller just by removing 2 lines with an serial.print meassage. after this upload I saw it was a bit smaller “Binaire sketch-grootte: 25.928 bytes (van een 32.256-byte maximum)” And now it workes perfect!!

Thats great, but I don’t think I can use this because I have to make the programm bigger, because I have to ad the onewire measurements to the programm.

But your programm looks good, So if I can’t find some thing better, perhaps I buy an Arduino Mega.

Thanks for the help
Marcel

Yeah, unfortunately the sketch was pushing resource limits to begin with. Perhaps it would be possible to squeeze enough room for your additions by strategic removal of more debug strings, and of other strings to the progmem. Because the problem seems to be one of ram, not flash mem.

With the mega both the ram and flash are bigger I asume.
Wich type arduino do you use yourself?

I also have an real time clock I can try to use, in stead of the time server. I just got it and didn’t use it before, perhaps this will also safe data, but i’m not sure.
Because it is not my own code it is not easy to see exact wich part I can delete in that case. My wife is allready complaining I’m at the computer all the time ;-).

I will try to see if this will help, in the mean time I think I will order an arduino mega.
It will take some time, but, to be continued…..

I’m using an original UNO. The RTC may help a lot, and thus allow for data points to be taken even without an active internet connection. That means you wouldn’t need to worry about gaps in your dataset as much. I don’t know the exact mechanics of the rtc, but it should just be a matter of including the right libraries and redoing the whole getTime() function for the rtc instead of a ntp server. You could also dump all of the ntp config variables. It might be all you need to solve your issues, but would involve a non trivial amount of coding.

First of all I would like to thank for sharing such a great project!
My question is if this type of data logger can stream real time values.

Thank you in advance
Great job

Sorry, I never noticed your comment before. It is currently not set up to stream realtime values, though I’m sure it would be possible with highcharts and an arduino if you were willing to rewrite a bit of the code.

Thank you for your reply!
At the mean time I managed to use highcharts for graphing in real time.
Although I realized that I also need older data so I have to store them to csv files like you do!
So the only thing I dont really get is the use of EEPROM in your project. COuld you send me some sources for further studying how this concept works?
Thank you in advance

I want to say fantastic work, easy to implement too. I’m wanting to capture from multiple sensors and log in separate logs/graphs(temp, humidity, ph, ppm, waterlevel, etc, its for a home hydroponics farm), any thoughts on how to do this?

In theory yes, but as marcel ran into out of memory issues trying to implement multiple readings per line, you may also run into these issues. Using a Mega or other beefier arduino may solve these issues though.

After I received my mega, the memory issues were gone and I managed to do multiple readings. They are all in the CSV file so logging is no problem, and it is running stable.

I tried to make a graph with more lines, but I didn’t manage so far.

At the end I have to display about 10 sensors I’m not sure if this is too slow.
At the mean time I’m looking for an other solution I found and see wich works best for me.

Again compliments, so many people want to log there data and plot it in a graph without removing the sd card. I was looking for months and did not find a good solution. Bu now I tried yours and for sure with one sensor it works great.

Marcel

Hi
I have copied some of your code and am trying to get multiple line plot to work. So far no luck.
Using Mega with Ethernet card.
I collected the data to the sd card with my program and the csv file I am plotting looks like
this.
This what tdaytmp2.csv looks like
1389745800,32.1125,70.3625
1389747600,31.6625,72.1625
1389749400,31.55,80.0375
1389751200,31.4375,83.6375
etc.
I am using your sketch to serve the data. Modified it to remove the time stuff as I am only using it to serve the data for testing. I will copy what I need from it later when I get the plots to work.
The data is being sent to the client as I have turned on the serial print for testing and it looks good. In fact I can plot either field 1 or 2, but not both. I’m just changing lines in the hce.htm file select which data to plot and either one looks correct, but can’t seem to get both lines to plot at same time.

Another thing I noticed, but haven’t spent much time trouble shooting is if a select a file that has 2 digit precision ( like 55.75 instead of 55.6789 the plot is different in that there is no connecting line between data points.

The HCE.HTM file I am using is below. I put in some comments to show what I have tried changing while troubleshooting. Obviously I am doing something wrong.and would appreciate any help.
Thanks
Mike

Super Graphing Data Logger!

// this function gets the file names from ardunio .. I think
function getDataFilename(str){
point = str.lastIndexOf(“file=”)+4;

tempString = str.substring(point+1,str.length)
if (tempString.indexOf(“&”) == -1){
return(tempString);
}
else{
return tempString.substring(0,tempString.indexOf(“&”));
}

}

query = window.location.search;

var dataFilePath = “/data/”+getDataFilename(query);

$(function () {
var chart;
$(document).ready(function() {

// define the options
var options = {

chart: {
renderTo: ‘container’,
zoomType: ‘x’,
spacingRight: 20
},

title: {
text: ‘Solar Controller Temperatures’
},

subtitle: {
text: ‘Click and drag in the plot area to zoom in’
},

xAxis: {
type: ‘datetime’,
maxZoom: 2 * 3600000
},

yAxis: {
title: {
text: ‘Temperature in F’
},
min: 0,
startOnTick: false,
showFirstLabel: false
},

legend: {
enabled: false
},

tooltip: {
formatter: function() {
return ‘‘+ this.series.name +’‘+
Highcharts.dateFormat(‘%H:%M – %b %e, %Y’, this.x) +’: ‘+ this.y;
}
},

plotOptions: {
series: {
cursor: ‘pointer’,
lineWidth: 1.0,
point: {
events: {
click: function() {
hs.htmlExpand(null, {
pageOrigin: {
x: this.pageX,
y: this.pageY
},
headingText: this.series.name,
maincontentText: Highcharts.dateFormat(‘%H:%M – %b %e, %Y’, this.x) +’: ‘+
this.y,
width: 200
});
}
}
},
}
},

series: [{
name: ‘tsensor2’,
marker: {
radius: 2
}
}

,
{
name: ‘tsensor3’,
marker: {
radius: 3
}
}

]
};

// Load data asynchronously using jQuery. On success, add the data
// to the options and initiate the chart.
// http://api.jquery.com/jQuery.get/
jQuery.get(dataFilePath, null, function(csv, state, xhr) {
var lines = [],
date,

// set up the two data series
tsensor2 = []; // bottom
tsensor3 = []; //top
// tsensor5 = []; // Wh
// tsensor6 = []; // Mix
// tsensor7 = []; // door
// tsensor8 == [];
// inconsistency
if (typeof csv !== ‘string’) {
csv = xhr.responseText;
}

// split the data return into lines and parse them
csv = csv.split(/n/g);
jQuery.each(csv, function(i, line) {

// all data lines start with a double quote
line = line.split(‘,’);
date = parseInt(line[0], 10)*1000;

tsensor2.push([
date,
parseInt(line[1], 10)
]);
tsensor3.push ([
date,
parseInt(line[2], 10)
]);
});
// Running with only series[0] plots ok
options.series[0].data = tsensor2;
// options.series(1).data = tsensor3;
// If I enable both of the above commands I get no plot
// Testing using tsensor3 data for series [0] as below
// also works. i.e the plot is of tsensor3 data
// options.series(0).data = tsensor3;

chart = new Highcharts.Chart(options);
});
});

});

Please allow the chart to load, it may take up to 30 seconds

Okay, so I went and looked at the new Highcharts 3 ajax example and found it has changed a lot since I made this project.Luckily it looks almost exactly like what you want, and I made a couple modifications to get it working on the data set you posted above:
http://pastebin.com/UQyixApJ

The main issue right now is that the CSV file needs to have the first row be the series names for the legend. So for the data file above it would be:

time,tsensor1,tsensor2
1389745800,32.1125,70.3625
1389747600,31.6625,72.1625
1389749400,31.55,80.0375
1389751200,31.4375,83.6375

Luckily this should be an easy fix. Just have the arduino write that label line right after the new file is made at line 298 or so. The code would be something like this:

File dataFile = SD.open(config.workingFilename, FILE_WRITE);

// if the file is available, write to it:
if (dataFile) {
dataFile.println(“time,tsensor1,tsensor2”);
dataFile.close();
// print to the serial port too:
Serial.println(“time,tsensor1,tsensor2”);
}
// if the file isn’t open, pop up an error:
else {
Serial.println(“Error opening datafile for writing”);
}

Hopefully this fixes the issues you were having. Let me know if you need any clarification on anything.

If I add this to my sketch, the header line is inserted every time a value is written. Where exactly do I need to put it ? I thought if it’s inserted after the following line it should work :
EEPROM_writeAnything(0, config);

Anyone ?

Hi Bart,

My csv files don’t contain any header information, but if you wish for yours to have this you’ll need to insert it in this code block when the new file is made. This should ensure it gets added only once during the creation of the file. Note that it needs to be inside the curly brace after the EEPROM writeAnything line.

//Decide if it's time to make a new file or not. Files are broken
//up like this to keep loading times for each chart bearable.
//Lots of string stuff happens to make a new filename if necessary.
if (rawTime >= config.newFileTime){
int dayInt = day(rawTime);
int monthInt = month(rawTime);
int yearInt = year(rawTime);
char newFilename[18] = "";
char dayStr[3];
char monthStr[3];
char yearStr[5];
char subYear[3];
strcat(newFilename,"data/");
itoa(dayInt,dayStr,10);
if (dayInt < 10){ strcat(newFilename,"0"); } strcat(newFilename,dayStr); strcat(newFilename,"-"); itoa(monthInt,monthStr,10); if (monthInt < 10){ strcat(newFilename,"0"); } strcat(newFilename,monthStr); strcat(newFilename,"-"); itoa(yearInt,yearStr,10); //we only want the last two digits of the year memcpy( subYear, &yearStr[2], 3 ); strcat(newFilename,subYear); strcat(newFilename,".csv"); //make sure we update our config variables: config.newFileTime += FILE_INTERVAL; strcpy(config.workingFilename,newFilename); //Write the changes to EEPROM. Bad things may happen if power is lost midway through, //but it's a small risk we take. Manual fix with EEPROM_config sketch can correct it. EEPROM_writeAnything(0, config); }

Thanks very much for the help. It got me back on track. I edited the file with the pc and put in the header and it is plotting 6 lines fine. Also, updated my sketch to put in the header. I edited the titles etc, but will probably make some more changed. I noticed that the year in date is showing 1970 when clicking a data point. Also, the time isn’t correct but haven’t really looked at that. I’m pretty sure the unix time stamp is correct in the file. I trying to learn how to use Firebug so maybe I can do some checking later. My plan is to add some bars to the plot too, but haven’t collected that data yet. Great to be making progress though. I will try to bug you too much. I am making a sketch to gather temperatures from solar system and some home automation. I did a project using basic stamp a couple of years ago but decided to redo it using Ardunio. Staring up learning C ,HTML and JavaScript at 70 is hard for an old brain.

Thanks again

Mike

Hey.
Cool projects, just wondering how to add an temperature sensor to it.
trying to learn arduino but having some problems understanding how u add the sensor,

This depends, are you trying to add a temperature sensor in addition to the light sensor, or replacing the light sensor with it?

I’m thinking Off replacing the photoresistor with and ds18 temprature sensor and a ultrasonic range sensor to view tank level and the temprature off the water.

Okay, getting the DS18 attached should be as simple as attaching the V+ pin to 5V on the arduino, gnd to gnd, and data to A0. Getting the range sensor working is trickier. It depends on how the sensor communicates data. If it is analog, then using the tricks already discussed in the comments here should work. If it’s pulse width or serial based, then it’ll be necessary to write some additional code for your project to get it working.

[…] Arduino per Ethernet online stellen. .csv auf SD schreiben. von dygraphs dynamisch und echtzeitaktuell abrufen lassen vgl. http://everettsprojects.com/2012/12/31/arduino-super-graphing-data-logger/#gettingStarted […]

Hi,

This is an awesome project, thanks for sharing. Being a newbie at coding in general, it is teaching me a lot. Thanks again.

I am having trouble getting the graphing stuff to work for my application. I receive a CSV string over serial every X seconds. Ideally, I would like to dynamically update the chart as the data comes in and/or read from the stored CSV as well.

This is what the first and second lines of my csv file look like:

Time,Tap 1,Tap 2,Tap 3,Tap 4,Tap 5,Tap 6,Tap 7,Tap 8,Tap 9,Tap 10,Blue,Yellow,White,Green,BattV,SoC,BCMAmps,Disch mAh,Chg mAh,Max,Min,Difference
1399896413,16.24,16.24,16.23,16.29,16.24,16.24,16.20,16.28,16.23,16.26,80.25,81.74,85.15,79.33,162,18.00,0.00,0.00,0.00,16.29,16.20,0.10

But there are or will be many lines.

Right now, I can get the chart to load the legend, but no graph is displayed.

I was hoping you could point me in the right direction. While I’ve been tinkering with Arduino for a while and can generally figure things out, this javascript stuff has me fairly baffled. Let me know if you need any other information. I’ll continue staring at things in the meantime..

-Eli

With a datastring that long it’s likely that the chart is failing to load because the arduino is running out of memory; especially if you are using an uno or equivalent with only 2kb. It’s an existing problem that is already identified, so my money is on that. Otherwise it could just be that the high charts code is not configured right for all your datasets.

Interesting. I hadn’t ever considered that. I’m using a Mega though.

I think its more on the highcharts code side of things. I can get the graphs to load sometimes(?), but never my whole dataset. So maybe that is the problem.. Dynamic loading would probably be better in that regard..

Maybe this whole idea is just beyond me right now. Ideally, I would like to get to the point where each group of relevant data is sent to its own chart, ie: the voltage taps go to one chart, the temperatures go to another, etc.

I would still appreciate some input on the charts side of things. I am pretty sure I’m doing something wrong there. If you could get the single data string I posted to graph, I would be forever grateful – I think it would help me understand what is going on better. I spent all day yesterday tinkering and looking at the Highcharts examples, but I didn’t make a whole lot of progress.

just wondering how easy it would be to change the light sensor to a DHT11 or DHT22 to measure temp / humidity instead of light. sorry for the noob question but your code looks to be just what is needed to log and display almost anything.

It should be reasonably easy. From what I understand, the DHTXX sensors are not analog, and need to be wired up differently, then the data needs to be read using the functions in an appropriate library. If you wire it up according to the information here: https://learn.adafruit.com/dht/connecting-to-a-dhtxx-sensor and then replace line 302 (” int sensor = analogRead(analogPin); “) with the appropriate lines like “float t = dht.readTemperature();” and “float h = dht.readHumidity();”. You’ll also need to change line 307 from itoa(…) to ftoa(…).

Hope this helps!

Thank you a bunch for sharing this with all of us you really recognize what you are speaking about!
Bookmarked. Please also discuss with my web site =).
We may have a hyperlink change contract beween us

Part of creating a successful search engine optimization campaign for you is optimizing your website for the
correct keywords. Many companies also offer to train people
under their SEO reseller programs to help them get more companies looking for SEO assistance and earn a better commission at the same time but before you join any program make sure to join an SEO firm that is going to compensate you well for the business you will be bringing for the firm.

SEO and web developer professionals vary
on the prescribed keyword density, usually ranging from a one to three percent density rate.

[…] รคhnlichen Ansatz verfolgt everettsprojects. Er diente fรผr mich jedoch eher als proof-of-konzept denn als Anleitung, der ich folgen konnte. Zu […]

Hi, I am trying to use to or more sensors in my code, so used the code for a comment on 19th January. The datapoints ar shown in the chart correctly, but the date ist from 1970. What do I have to change in my code to get the right date. If I use your code for only one sensor, I get the correct date and time in the chart.
Thanks ๐Ÿ™‚

Do you know if the time in the CSV file is correct? You can easily check some of the values using a unix time conversion tool: http://www.epochconverter.com/

The time in the CSV file is correct. Do I have to change something in the HC.htm file? Or do I have to change the headline of my CSV file?

I think I have found the reason for this error. I have to multiply the time with the factor 1000 and i have done this manually in the CSV file. I’m not so good in programming, where and how do I have to change my programm?

If you have decided to multiply the values in the CSV file, then you can get the SGDL sketch to do this by replacing line 306 (currently “ultoa(rawTime,timeStr,10);”) with something like ultoa(rawTime*1000,timeStr,10);

The HC.HTM file is supposed to be multiplying the values by 1000 already though, and for some reason the code: “date = parseInt(line[0], 10)*1000;” that should be doing that is not.

Do you know of any reason this wouldn’t work with with a w5200 instead of a w5100, I have tried the example codes and get dhcp addresses but your code does not respond to pings

The only reasons I can think of are that either the W5200 currently lacks support in the default Ethernet Library, or the Ethernet Library does things a bit differently now and has broken some of the code in my sketch.

If you are using this http://www.seeedstudio.com/wiki/Ethernet_Shield_V2.0 shield, then it appears to have a separate version 2 library that is different from the one included with the Arduino by default. Maybe manually installing the V2 library will solve these problems?

[…] was for some kind soul to put Highcharts together with an Arduino โ€“ and SD card.. and VOILA.. http://everettsprojects.com/2012/12/31/arduino-super-graphing-data-logger/&nbsp; I MUST STRESS I have not tested this (give it time) โ€“ I have the sketch in front of me โ€“ […]

have any solved the problem with file list???

Hi Everett
Back in January I contacted you and you got me back on track with my project. .. A SolarController and a bunch of other stuff.
It displays data about my solar system in addition to running the system. I have several charts that I generate using highcharts.
My problem is even though they work fine with Firefox and Chrome I can’t seem to figure out why they don’t work with IE or on android tablet which I suppose is some variant of IE.
If you want to look at it these links (http://pumps-solardata.pw/hilow , http://pumps-solardata.pw/chart, http://pumps-solardata.pw/mth_yr ) show the problem in that no data gets plotted when using IE, but work fine with Firefox or Chrome

There are others, but this should give you and idea of what it does. By using Firebug on firefox I can see the get request going to the Ardunio.
GET http://pumps-solardata/pw/data/30dhilow.csv
That never happens when using IE so I pretty sure there is something in the htm file that IE doesn’t like but since I havn’t figured how to get firefox lite or IE’s debugger to work in IE I can’t figure out what is wrong.

Any ideas?
Also, note that not all the sensors readings are valid as they are on my bench instead of on the solar system. Outside temps are pretty close, but sensors are too close to the house to bereally accurate.
I didn’t include the htm files as you can see them by inspecting element or with firebug.
Thanks
Mike

Hi, Everret.
I’m an engineer in WIZnet as supplier of W5100 for Arduino.
Your project is the excellent reference to arduino ethernet shield, SD card, EEPROM interface.
So, I hope that I will introduce your project into my blog & WIZnet museum(http://wiznetmuseum.com/).
If you’re ok, shall I do?
Thank you in advance.

Hi, sorry for the very late response. I haven’t had a lot of free time to tend to this blog recently. But yes, you have my permission to include this project on the wiznet museum blog.

[…] ๋ฐค์†Œ๋‹˜์˜ ๋ธ”๋กœ๊ทธ์— ์ •๋ฆฌ๋˜์–ด ์†Œ๊ฐœ๋˜๊ณ  ์žˆ์œผ๋ฉฐ, ์›๋ณธ ๊ธ€๊ณผ ์†Œ์Šค์ฝ”๋“œ ๋“ฑ์€ Everett๋‹˜์˜ ๋ธ”๋กœ๊ทธ์—์„œ ํ™•์ธ […]

Gerrit Jan Ritman

Hello Everett,
I’m building a weather dependent furnace controller. The prototype works fine, it even stores data every 10 seconds on the SD card. To access the data I have to take the SD card out. ๐Ÿ™ I would like to make the data visible like in your SGDL, but I get the impression that string.h is missing. Other errors I get is: error: ‘prog_char’ does not name a type
and
variable ‘HeaderOK_table’ must be const in order to be put into read-only section by means of ‘__attribute__((progmem))’
Any suggestions?
Regards, Gerrit Jan

Hi Gerrit,

Sorry for the late response, but I suppose late is better than never.
It would appear to me that a lot of changes have been made to various arduino libraries since I created this project: https://www.arduino.cc/en/Reference/PROGMEM

So it would appear that the lines using progmem like this:
prog_char HeaderOK_0[] PROGMEM = “HTTP/1.1 200 OK”;

Should be changed to this syntax:
const char HeaderOK_0[] PROGMEM = “HTTP/1.1 200 OK”;

Hopefully this resolves the issues you were/are having, but if it doesn’t and you’re still interested then feel free to ask for more help. Hopefully I’ll be a lot quicker in getting a response back to you next time.

Had the same issue with Gerrit Jan using IDE 1.6.5 compiling errors , so i went to the link everett gave and made modifications as below and it worked.

Hope this helps!

* Do the same changes for header404 parts.
Cheers

—————

// Strings stored in flash mem for the Html Header (saves ram)
// const dataType variableName[] PROGMEM = {};
const char HeaderOK_0[] PROGMEM = {“HTTP/1.1 200 OK”}; //
const char HeaderOK_1[] PROGMEM = {“Content-Type: text/html”}; //
const char HeaderOK_2[] PROGMEM = {“”}; //

// A table of pointers to the flash memory strings for the header
// const char* const string_table[] PROGMEM = {string_0, string_1, string_2, string_3, string_4, string_5};
const char* const HeaderOK_table[] PROGMEM = {
HeaderOK_0,
HeaderOK_1,
HeaderOK_2
};

Thx Everett for such a great project for learning graphing!

I finally made it work after many hrs of attempting to resolve a “Error opening datafile for writing” errors.

Finally found that i did not uncomment the EEPPROM line in the initial sketch.

Cheers..hope others wont stumble on this.

Going to follow on with doing DHT11 for temperature and humid.

Cheers

Hi All,

Anyone who has recent code from the above super project?
I got the DHT22 implemented and it compiles and runs.
So far I can see only temperature. The other (humidity) graph seems a bridge too far yet. It was already a challenge getting all code adjusted to the latest version

Reading/writing one character at a time isn’t optimized in the Arduino libraries. I changed the file transfer code as follows:

while(file.available())
{
int num_bytes_read;
char byte_buffer[64];

num_bytes_read=file.read(byte_buffer,64);
client.write(byte_buffer,num_bytes_read);
}

That loads the data for a plot about 10 times faster than the original code.

Hi all,

Can you upload the latest project that works, with all files, Libraries, HC.htm included?

thank you!

Hi, I have updated the source code files and uploaded them to GitHub. These changes should make it all compile and run using the newer versions of the Arduino IDE, but let me now if you still have issues.

Leave a Reply