web

My Life As A Quant: A Neopets Stock Market Bot.

My Life As A Quant: A Neopets Stock Market Bot.

This program definitely violates the Neopets Terms of Use. Use at your own risk.

Sometime in the year 2000 or so I was really into the website Neopets. I was about 10, and everyone my age was into Neopets. I don’t remember many details, but I do remember trying to get rich using the Neopets stock market. It didn’t really work because I didn’t have the patience to properly execute the tried and true techniques every one else had already figured out. I knew this, and the brief thought of making a robot to do it for me crossed my mind. The ten year old me had no clue how to go about creating such a robot though, and I’m pretty sure that I had it in my head to build a literal robot that would use the keyboard and mouse. Maybe out of lego or something. I’m not sure.

Some 12 years later (in 2012), something jogged my memory and drew my attention to this unrealized dream of long ago. More importantly, I realized I could make it work now. Sure Neopets isn’t much of a thing anymore, but that didn’t matter to me. I had to make this happen; it was a loose end! In pursuit of this goal I turned to Python as my language of choice, which conveniently has a library called Mechanize (named after the original Perl Library). Mechanize can programatically interact with webpages as though it is a user. It’s a great module that allowed me to finish the project in the span of a couple days between two busy weeks at university. Even today, about two years later, it’s still running on an almost daily basis (sometimes computer downtime prevents it from running), and has taken the 1.5 million neopoints of seed capital and turned it into more than 20 million neopoints.

Lets start with the code:

#
# stockbot.py - A webcrawling bot that can automatically play the Neopets stock market game.
#

import mechanize
from lxml import etree
import random
import datetime
import time
import math
import copy
import sys

# Ermagerd, global variables! Bad practice! Bad practice!
logFile = open('<PATH TO FILE>/log.txt', 'a')
errorHTMLdump = open('<PATH TO FILE>/errorHTML.txt', 'a')


##
# Main function controls browser session and logging into Neopets
##  
def main():
    
    #user credentials
    userName = 
    passWord = 

    br = mechanize.Browser(factory=mechanize.RobustFactory())
    
    # User-Agent
    br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:17.0) Gecko/17.0 Firefox/17.0')]
    
    # Only necessary if you are using cron instead of anacron, and wish to hide your robotic behaviour.
    #humanizingDelay(300)
    
    # For potential issues connecting, and a URLError is raised. This sleeps for 30 seconds
    # then retries the connection up to 10 times before giving up and documenting the error.
    for attempt in range(10):
        try:
            br.open("http://www.neopets.com/login/index.phtml")
        except mechanize.URLError:
            time.sleep(30)
        else:
            break
    else:
        logFile.write(datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")+" - An error occured while trying to connect to the webpage.\n")
        sys.exit()

    br.select_form(nr=0)
    br.form['username'] = userName
    br.form['password'] = passWord

    # Login
    br.submit()
    
    bankWithdrawal(br)
    stockManager(br)
    bankDeposit(br)
    
    br.open("http://www.neopets.com/logout.phtml")

##
# Causes the script to pause for a random time to make it appear more 
# human by not always executing at the exact same time of day. maxLength
# is the maximum duration of the pause in seconds.
# minLength is optional and defaults to 0.
##
def humanizingDelay(maxLength, minLength=0):
    pauseDuration = random.uniform(minLength,maxLength)
    time.sleep(pauseDuration)

##
# Collects the daily bank interest and decides if it is necessary to withdraw
# neopoints for the day's stock purchase.
##  
def bankWithdrawal(browser):
    
    bankPage = "http://www.neopets.com/bank.phtml"
    bankHTML = browser.open(bankPage)
    
    browser.select_form(nr=3)
    browser.submit()
    
    humanizingDelay(5, minLength=2)
    
    bankHTMLString = bankHTML.read()
    if getNeopoints(bankHTMLString) < 17000:
        browser.select_form(nr=2)
        browser.form['amount'] = "17000"
        browser.submit()
        logFile.write(datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")+" - 17000 NP withdrawn from the bank.\n")


##
# Looks at the current neopoint value and deposits any excess
##  
def bankDeposit(browser):
    
    bankPage = "http://www.neopets.com/bank.phtml"
    bankHTML = browser.open(bankPage)
    
    humanizingDelay(3, minLength=1)
    
    bankHTMLString = bankHTML.read()
    if getNeopoints(bankHTMLString) > 32500:
        depositValue = getNeopoints(bankHTMLString) - 32500
        browser.select_form(nr=1)
        browser.form['amount'] = str(depositValue)
        browser.submit()
        logFile.write(datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")+" - "+str(depositValue)+" NP deposited to the bank.\n")
    
##
# Takes the current page's HTML and returns the neopoint value.
##     
def getNeopoints(pageHTML):
    startToken = "<a id='npanchor' href=\"/inventory.phtml\">"
    endToken = "</a>"
    startIndex = pageHTML.find(">",pageHTML.find(startToken))+1
    endIndex = pageHTML.find(endToken,startIndex)
    npString = pageHTML[startIndex:endIndex]
    npValue = int(npString.replace(",",""))
    
    return npValue


##
# A function performing the upper level stock market stuff.
## 
def stockManager(br):
    stockListHTML = br.open("http://www.neopets.com/stockmarket.phtml?type=list&full=true")
    portfolioHTML = br.open("http://www.neopets.com/stockmarket.phtml?type=portfolio")
   
    stockPrices = extractStockPrices(stockListHTML)
    stockHoldings = extractStockHoldings(portfolioHTML)
    
    todaysBuy = pickStockPurchase(stockPrices, stockHoldings)
    if todaysBuy != "No Stocks at 15-17NP":
        buyResult = buyStocks(todaysBuy, br)
        logFile.write(datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")+" - "+buyResult+"\n")
    else:
        logFile.write(datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")+" - No stocks purchased, none available at 15-17NP/share.\n")
        
    todaysSales = pickStockSales(stockPrices, stockHoldings)
    
    if len(todaysSales) >= 1:
        successfulSale = sellStocks(todaysSales, br)
        if successfulSale:
            for stock in todaysSales.keys():
                logFile.write(datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")+" - "+str(int(round(todaysSales.get(stock))))+" shares of ["+stock+"] sold at "+str(stockPrices.get(stock))+" NP each.\n")
        else:
            logFile.write(datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")+" - Error: Something went wrong when trying to sell stocks.\n")


##
# Extracts the Stock tickers and values into a dictionary
##  
def extractStockPrices(stockListHTML):
    
    #Put the HTML source into a string
    stockListHTMLString = stockListHTML.read()
    
    #Trim the source down to just the table we care about
    startToken = "<table cellpadding=3 cellspacing=0 border=1 align=center><tr><td align=center bgcolor='#8888ff'><font color=white><b>Logo"
    endToken = "<br clear="
    startIndex = stockListHTMLString.index(startToken)
    endIndex = stockListHTMLString.index(endToken)    
    stockTableHtml = stockListHTMLString[startIndex:endIndex]
    
    #Pick out all of the HTML stuff our xml parser will choke on
    removeSubStrings = [["<b>",""],
                        ["</b>",""],
                        ["<table cellpadding=3 cellspacing=0 border=1 align=center>","<table>"],
                        [" bgcolor='#eeeeff'",""],
                        [" bgcolor='#8888ff'",""],
                        [" align=center",""],
                        [" width=60",""],
                        [" height=60",""],
                        [" border=0",""],
                        ["<font color=",""],
                        ["'green'>",""],
                        ["'red'>",""],
                        ["'black'>",""],
                        ["white>",""],
                        ["</font>",""],
                        ["</a>",""]]
    

    for i in removeSubStrings:
        stockTableHtml = stockTableHtml.replace(i[0], i[1])

    while stockTableHtml.find("<img src=") != -1:
        tempStartIndex = stockTableHtml.find("<img src=")
        tempEndIndex = stockTableHtml.find("gif'>") + 5
        tempSubString = stockTableHtml[tempStartIndex:tempEndIndex]
        stockTableHtml = stockTableHtml.replace(tempSubString, "")
        
    while stockTableHtml.find("<a href=") != -1:
        tempStartIndex = stockTableHtml.find("<a href=")
        tempEndIndex = stockTableHtml.find("'>") + 2
        tempSubString = stockTableHtml[tempStartIndex:tempEndIndex]
        stockTableHtml = stockTableHtml.replace(tempSubString, "")       
        
    stockTableHtml = stockTableHtml.replace("<td>Logo</td>", "")
    stockTableHtml = stockTableHtml.replace("<td></td>", "")
           
    ##
    # Aggregate into dictionary
    ##  
    
    
    stockValues = {}

    table = etree.XML(stockTableHtml)
    rows = iter(table)
    next(rows)
    for row in rows:
        stockValues[row[0].text] = int(row[4].text)
        
    return stockValues
    
##
# Extracts the Stock tickers and current holdings into a dictionary
#       Only adds stocks which are present in portfolio
## 
def extractStockHoldings(portfolioHTML):
    
    #Put the HTML source into a string
    portfolioHTMLString = portfolioHTML.read()
    
    #Trim the source down to just the table we care about
    startToken = "<table align=center cellpadding=3 cellspacing=0 border=1>"
    endToken = "<span id='show_sell' style='display:none'><center><input type="
    startIndex = portfolioHTMLString.index(startToken)
    endIndex = portfolioHTMLString.index(endToken)   
    portfolioTableHtml = portfolioHTMLString[startIndex:endIndex]
    
    #Pick out all of the HTML stuff our xml parser will choke on
    while portfolioTableHtml.find("<tr id=") != -1:
        tempStartIndex = portfolioTableHtml.find("<tr id=")
        tempEndIndex = portfolioTableHtml.find("</table>\n</td>\n<tr>") + 21
        tempSubString = portfolioTableHtml[tempStartIndex:tempEndIndex]
        portfolioTableHtml = portfolioTableHtml.replace(tempSubString, "")    
        
    while portfolioTableHtml.find("<td align=\"center\" ><img") != -1:
        tempStartIndex = portfolioTableHtml.find("<td align=\"center\" ><img")
        tempEndIndex = portfolioTableHtml.find("\"></td>") + 7
        tempSubString = portfolioTableHtml[tempStartIndex:tempEndIndex]
        portfolioTableHtml = portfolioTableHtml.replace(tempSubString, "")    
        
    while portfolioTableHtml.find("<a href=") != -1:
        tempStartIndex = portfolioTableHtml.find("<a href=")
        tempEndIndex = portfolioTableHtml.find(">", tempStartIndex) + 1
        tempSubString = portfolioTableHtml[tempStartIndex:tempEndIndex]
        portfolioTableHtml = portfolioTableHtml.replace(tempSubString, "")
        
    tempStartIndex = portfolioTableHtml.find("<tr bgcolor=\"#BBBBBB\">")
    tempEndIndex = portfolioTableHtml.find("</tr>", tempStartIndex) + 5
    tempSubString = portfolioTableHtml[tempStartIndex:tempEndIndex]
    portfolioTableHtml = portfolioTableHtml.replace(tempSubString, "")  

    
    removeSubStrings2 = [["<td bgcolor='#ccccff' colspan=2>&nbsp;</td>",""],
                        ["<td bgcolor='#ccccff' align=center colspan=3><b>Today</b></td>",""],
                        ["<td bgcolor='#ccccff' align=center colspan=2><b>Holdings</b></td>",""],
                        ["<td bgcolor='#ccccff' align=center colspan=2><b>Overall</b></td>",""],
                        ["<td align=center bgcolor='#ccccff'><b>Icon</b></a></td>",""],
                        ["<tr>\n\n\n\n\n</tr>",""],
                        ["<tr><td align=\"right\" colspan=\"5\">Totals:</td><td\">4,000</td>",""],
                        ["<tr><td align=\"right\" colspan=\"5\">Totals:</td><td\">4,000</td>",""],
                        ["<tr><td align=\"right\" colspan=\"5\">Totals:</td><td\">4,000</td>",""],
                        ["<b>",""],
                        ["</b>",""],
                        ["<table align=center cellpadding=3 cellspacing=0 border=1>","<table>"],
                        [" bgcolor=\"#EEEEFF\"",""],
                        [" bgcolor=\"#FFFFFF\"",""],
                        [" bgcolor=\"#BBBBBB\"",""],
                        [" bgcolor='#ccccff'",""],
                        [" align=\"center\"",""],
                        [" align=center",""],
                        ["<font color=\"",""],
                        ["<font size=1>(profile)",""],
                        ["green\">",""],
                        ["red\">",""],
                        ["black\">",""],
                        ["</font>",""],
                        ["</a>",""],
                        ["<nobr>",""],
                        ["</nobr>",""],
                        ["<br>",""]]
    
    for i in removeSubStrings2:
        portfolioTableHtml = portfolioTableHtml.replace(i[0], i[1])
    
    
    ##
    # Aggregate into dictionary
    ##  
    
    stockHoldings = {}
    
    table = etree.XML(portfolioTableHtml)
    rows = iter(table)
    next(rows)
    for row in rows:
        stockHoldings[(row[0].text).strip()] = int(row[4].text.strip().replace(",",""))

    return stockHoldings
    
##
# Picks the stock to purchase for the day by checking which ones are at 15 NP
# If multiple stocks are at 15 NP, it buys the one we currently own the least
# amount of. If there is a tie, it decided between them randomly.
##     
def pickStockPurchase(prices, holdings): 
    
    #Make a list of the stocks at 15NP and a list of raw holdings for easy use of min later
    # If no stocks at 15NP / share then look for 16 and even 17 if necessary
    potentialPurchases = []
    holdingsValues = []
    stocksAtPrice = False
    targetPrice = 15
    while not(stocksAtPrice) and targetPrice <= 17:
        for ticker,price in  prices.iteritems():
            if price == targetPrice:
                stocksAtPrice = True
                if ticker in holdings:
                    potentialPurchases.append([ticker, holdings.get(ticker)])
                    holdingsValues.append(holdings.get(ticker))
                else:
                    potentialPurchases.append([ticker, 0])
                    holdingsValues.append(0)
        targetPrice = targetPrice + 1
                    
    
    #Go through the possible cases for potentialPurchases, act accordingly         
    if len(potentialPurchases) == 0:
        return "No Stocks at 15-17NP"
    elif len(potentialPurchases) == 1:
        return potentialPurchases[0][0]
    else:      #Most complicated possibility, pick one we own the least shares of,
                #randomly selected in event of tie.
        minHoldings = min(holdingsValues)
        shortList = []
        for stock in potentialPurchases:
            if stock[1] == minHoldings:
                shortList.append(stock[0])
        if len(shortList) == 1:
            return shortList[0]
        else:
            randomPick = random.choice(shortList)
            return randomPick
            

##
# Actually buys 1000 shares of the stock picked buy pickStockPurchase()
# Relies on the index of the form and inputs, hopefully this doesn't change.
##        
def buyStocks(ticker, browser):
    
    #The stock buying page URL
    buyPage = "http://www.neopets.com/stockmarket.phtml?type=buy"

    browser.open(buyPage)
    #The form we want isn't named, but it's the second one on the page
    browser.select_form(nr=1)
    #selecting the controls by name doesn't work, so we get them by index
    controls = browser.form.controls
    controls[2]._value = ticker # the ticker
    controls[3]._value = "1000"   # the number of shares
    
    humanizingDelay(5,minLength=1)
    response = browser.submit()
    
    #check that everything worked, return a string with the result
    if response.geturl() == "http://www.neopets.com/stockmarket.phtml?type=portfolio":
        return "Success: 1000 shares of ["+ticker+"] have been purchased."
    elif response.geturl() == "http://www.neopets.com/process_stockmarket.phtml":
        responseHTML = response.read()
        startToken = "<b>Error:"
        endToken = "</div>"
        startIndex = responseHTML.index(startToken)
        endIndex = responseHTML.index(endToken,startIndex)   
        errorString = responseHTML[startIndex:endIndex].replace("</b>","").replace("<b>","")
        return errorString
    else:
        return "Error: Unknown problem occured while buying stocks."

    
    


##
# Decides if any of the stocks in our portfolio are beyond the sale threshold.
# All of the current holdings are sold if they are beyond. Returns a Dictionary 
# of Ticker and number of stocks to sell.
##         
def pickStockSales(prices, holdings):

    sellThreshold = 50
    stocksToSell = {}
    
    # Go through the stocks we own, check if we should sell, then if so 
    # add to list at half the currently owned shares (rounded up)
    for ticker in holdings.keys():
        if prices.get(ticker) >= sellThreshold:
            stocksToSell[ticker] = holdings.get(ticker)
    return stocksToSell
      


##
# Implements the actual selling of the stocks from pickStockSales()
# Returns a boolean indicating success (for log file purposes)
##         
def sellStocks(salesList, browser):
    
    salesPage = "http://www.neopets.com/stockmarket.phtml?type=portfolio"
    browser.open(salesPage)
    #The form we want isn't named, but it's the second one on the page
    browser.select_form(nr=1)
    #We don't know the exact names of the inputs, they have a ref number, time to search:
    controls = browser.form.controls
    for control in controls:
        try:
            if len(control.name) > 10:
                if control.name[5:control.name.find("]")] in salesList:
                    control._value = "1000"
        except TypeError:
            pass
    
    humanizingDelay(5,minLength=1)
    response = browser.submit()
    
    #check that everything worked, return a string with the result
    if response.read().find("There were no successful transactions") == -1:
        return True
    else:
        errorHTMLdump.write("/n"+datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")+"/n"+response.read())
        return False
    
    
main()

The script is broken down into a pretty logical sequence as it’s very procedural. It starts with the main function which deals with logging in, collecting bank interest and withdrawing neopoints for the stock market purchase if necessary, and then finally playing the stock market. Finally, it goes to deposit the neopoints gained from the stock market if any sales were made, then logs out. Pretty simple on the face of it, but the real interesting parts are the actual interactions with the webpages. In order for our bot to be able to do anything meaningful we need to first extract the information we care about from the page, then have the script make the right decisions using that information, and finally put the calculated input values into the right fields and submit the data. For the login, and bank portions of this bot, that’s fairly simple; it’s just a matter of going to the correct url selecting the right forms, and submitting a static or otherwise easy to calculate value that is pulled out of the HTML by finding some substrings that encapsulate the data we’re after.

Screenshot from 2014-09-27 14:46:58

The list of Neodaq stocks and their prices.

The stockManager() function is a little less straight forward. The Neopets stock market game is a simple html form based game, which makes most of this possible (building a bot to interact with flash content? No thank you!). This means that we use the same techniques as we did for the login and bank pages, we’re just after information that is a presented in a slightly more complex way as an HTML table. We’re interested in the structure of the table, and we need to maintain that, just without all of the HTML bits that don’t mean anything to us. This is done through the extractStockPrices() and extractStockHoldings() functions. They both work in pretty much the same way, by taking the entire HTML page as a string, isolating the table of interest, stripping out some of the problematic HTML bits, then putting it all into a dictionary that maintains the data structure using XML.

Screenshot from 2014-09-27 14:47:28

My portfolio of Neodaq stocks.

Finally, with the data isolated from the formatting, we can figure out what it is we actually want to do. This is accompolished using the pickStockPurchase(), buyStocks(), pickStockSales(), and sellStocks() functions. The pickStockPurchases() function goes through the dictionary of stocks with their prices we just made and looks for stocks in the 15-17 NP per share range. It then selects the lowest priced one that we currently own the least of. The buyStocks() function then simply purchases 1000 shares of this stock (which is the maximum number of shares any one user can purchase per day). The pickStockSales() function then looks at our portfolio of stocks and finds any that surpass the user defined sale threshold. It puts all stocks that need to be sold into a list and then the sellStocks() function takes care of actually selling them. The sellStocks() function sells off 100% of the holdings of any stock that meets this criteria which keeps things simpler, as the sales page is built in such a way that a single stock is broken down into each of the individual purchases that went into it rather than the total holdings. We originally purchased the stock in 1000 share units, which means the bot can simple put 1000 into all of the input fields that belong to that ticker to completely divest that stock.

Now you may have noticed that there is no means of ensuring that your bank account contains enough neopoints to facilitate the 17,000 NP withdrawal at the start. This means that it is necessary to ensure your seed capital is sufficient that you won’t burn through it before you start seeing returns. How long it takes before you start seeing returns is a matter of what your sales threshold is set at. For the most part, stocks on the Neopets stock market follow a pretty predictable pattern of swinging back and forth between about 6 and 60 NP per share. It’s less common, but stocks do periodically move to above this 60 NP range before they begin falling back to the 6 NP area. What this means for you is that the higher you set your sales threshold, the longer it will be before you see returns (although the returns will be larger). So if you are starting with a smaller pool of seed capital, it will make sense to use a lower threshold so that you won’t run into an empty bank account. Personally I started with the 1.5 million NP that were in my account from long ago, and used a sales threshold of 45 NP per share to start. If I recall correctly, the balance in my bank account never dropped below about 750,000 NP before I started earning it back. Since there is a strong element of randomness in all of this it is a very good idea to have more seed capital than you think you’ll need at your chosen sale threshold. A lot of people like to pick a return of 1,000,000 NP per month as their goal. To achieve this you need to earn a daily return of almost 33,000 NP per day (12 million divided by 365 days). You’re buying 1000 shares at up to 17,000 NP each day, so this means you should sell these shares at 33 NP per share above what you paid. This works out to a nice round number of 50 NP per share. Nice. This isn’t really a bad sales threshold to start at if you have the bank balance to support it.

For those lucky enough to have a very large sum of neopoints in their bank account, the burn rate will not even be an issue. This is because the bot collects bank interest. If your bank interest is in excess of 17,000 NP per day (the maximum this bot will ever spend on stocks) then you will never see a net decrease in your account balance. You’d need to have a bank balance of 49,640,000 NP before this would be the case though, so you’re probably not going to be in this privileged position to begin with. But that’s enough theory on burn rates and returns. If you want to read and learn more about this stuff then there are plenty of places to start.

Throughout this whole process the script also keeps detailed logs of what it is doing. This allows you to quickly check up on the progress of the bot without logging in to Neopets and then trying to figure out what has happened since you last looked.

Screenshot from 2014-09-27 15:06:59

The log file.

So now that we have the script, it’d be nice to have a way to make it run automatically each day too, right? If you’re a linux or mac user, then there’re two wonderful tools called cron and anacron that can accomplish this. I prefer anacron since it will work on systems that are not guaranteed to be on when the task is is scheduled; it’ll simply ensure the task is run once per day if the computer is turned on at all during that day. Cron also works well on server like systems with near 100% uptime. Windows users will have to use the windows task scheduler which has a fairly similar mechanic to cron and anacron.

But lets just go ahead and look at the process for setting up anacron:

To schedule a task with anacron, we use anacrontab:

sudo nano /etc/anacrontab

There will be a few lines in there already. At the bottom you simply want to add the following one:

1 6 cron.daily nice python /<PATH TO FILE>/stockbot.py >/dev/null

What this does is tells the system to run the task daily, with a delay of 6 minutes after start up. It assigns the task to cron.daily label (you can change this), and then executes the command “python /<directories>/stockbot.py” and sends any CLI output to null (basically computer oblivion).

You can learn more about cron and anacron here.

And that’s that! You now have a Neopets stock market bot running daily and earning you free neopoints. Congrats. It only took about 10 years.

Posted by Everett in Programming, 1 comment
An Interactive Route Map for my Travel Blog

An Interactive Route Map for my Travel Blog

travelMap

See the finished project at: http://everett.x10.mx/maps/

For the last three months I have been on a backpacking trip, which is part of the reason why this blog has been so neglected for the past few months. A travel blog hosted at meandmypack.wordpress.com had taken precedence, and I habitually kept that one updated throughout my trip. With all of that over though, I’ve had to find things to do in my time to keep my self from becoming bored and lethargic with life back home in Canada. One such activity is the tying up of loose ends as far as documentation of my trip is concerned, and from early on I had it in my mind to make a nice map of all the places I went. Over time this idea evolved into a whole project in it’s own right, using Google maps and becoming more interactive and feature rich every time my mind drifted back to the idea of it. I couldn’t really spare the time to design it while in Europe, and that probably would have been a waste of the limited time I had there any way. So I stored the idea away and made a promise to my self to figure it out back home. Now some two weeks later, I’ve pulled it off.

To start, I began with the Google Maps API v3 Simple Polylines example code and then added in this code for adding in markers to the map. The poly line consists of a large number of latitude and longitude coordinates that I fetched from Google maps using the LatLng Marker plugin available through Google Maps Labs. With a stubbornness that could be mistaken for OCD, I made sure the PolyLine at least vaguely resembled my true route between major destinations by routing them through all of the intermediate stations that the train called at along the way. This was accomplished with the travel report from my Eurail Pass ( I knew I diligently filled it out for a reason), and the Eurail timetables. With these two tools, I could easily go back and find the true route of most of the train travel I did during my trip. Elsewhere when I didn’t travel by train I figured the route out through some combination of memory and Google. The markers for all of the main cities and attractions that I visited were simply made by selectively harvesting those coordinates from the Polyline list and then adding them to their own modified list with extra fields for the associated tag on my blog, and the blurb for the popup info box. I modified the marker code to put a link to the associated content on meandmypack.wordpress.com inside that popup box.  Finally I felt that it would be nice to calculate the distance travelled from the Polyline, which I did with the help of this code.

With all of the main features of the map coded, and a few hours spent finding the geospatial coordinates of my route, I had a decent looking finished product. I spent just a little more time on the layout and design of the page so that some information about the map was presented in a permanent box in the top left hand corner. I’ve also decided to post the final draft of the code below for easy viewing by all interested parties:

index.html


<!DOCTYPE html>
<html>
  <head>
    <meta name="viewport" content="initial-scale=1.0, user-scalable=no">
    <meta charset="utf-8">
    <title>Me and My Pack Interactive Route Map</title>
    <link href="/maps/default.css" rel="stylesheet">
    <script type="text/javascript"
      src="https://maps.googleapis.com/maps/api/js?key=AIzaSyCIxpXOSPJWNG7TnhMYq-Q2hPcM7zEQs8g&sensor=false">
    </script>
    <script>
	//Standard Google Maps API code with project specific values
	function initialize() {
	  var middleEarth = new google.maps.LatLng(52.01254, 8.2133);
	  var mapOptions = {
	    zoom: 5,
	    center: middleEarth,
	    mapTypeId: google.maps.MapTypeId.ROADMAP
	  };
	
	  var map = new google.maps.Map(document.getElementById('map-canvas'), mapOptions);
	
	  //The Polyline coordinates. Lots and Lots of them.
	  var routeCoordinates = [
	      new google.maps.LatLng(51.51120, -0.11978), 	//London
	      new google.maps.LatLng(53.95795, -1.0934),
	      new google.maps.LatLng(54.9681, -1.6173),
	      new google.maps.LatLng(55.7743, -2.0110),
	      new google.maps.LatLng(55.95324, -3.18827), 	//Edinburgh
	      new google.maps.LatLng(55.982, -3.616),
	      new google.maps.LatLng(56.077, -3.923),
	      new google.maps.LatLng(56.1387, -3.9179), 	//Wallace monument
	      new google.maps.LatLng(56.17843, -4.3821), 	//Aberfoyle
	      new google.maps.LatLng(56.23381, -4.4290),	//loch katrine
	      new google.maps.LatLng(56.2440, -4.2158),
	      new google.maps.LatLng(56.18932, -4.0510),	//doune
	      new google.maps.LatLng(56.077, -3.923),
	      new google.maps.LatLng(55.982, -3.616),
	      new google.maps.LatLng(55.95324, -3.18827), 	//Edinburgh
	      new google.maps.LatLng(55.85931, -4.25836),
	      new google.maps.LatLng(54.9617, -5.0142),
	      new google.maps.LatLng(55.0317, -5.1047),
	      new google.maps.LatLng(55.0271, -5.3356),
	      new google.maps.LatLng(54.7595, -5.6473),
	      new google.maps.LatLng(54.5971, -5.930),		//Belfast
	      new google.maps.LatLng(54.852, -5.811),
	      new google.maps.LatLng(54.982, -5.996),
	      new google.maps.LatLng(55.058, -6.062),
	      new google.maps.LatLng(55.200, -6.239),
	      new google.maps.LatLng(55.24881, -6.48898),	//Giants causeway
	      new google.maps.LatLng(54.745, -6.23),
	      new google.maps.LatLng(54.5971, -5.930),		//Belfast
	      new google.maps.LatLng(54.0011, -6.4129),
	      new google.maps.LatLng(53.34980, -6.26028),	//Dublin
	      new google.maps.LatLng(53.435, -7.941),
	      new google.maps.LatLng(53.27055, -9.0566),	//Galway
	      new google.maps.LatLng(53.271, -8.918),
	      new google.maps.LatLng(53.207, -8.868),
	      new google.maps.LatLng(53.139, -8.931),
	      new google.maps.LatLng(53.114, -9.148),
	      new google.maps.LatLng(53.016, -9.375),
	      new google.maps.LatLng(52.97184, -9.42649),	//Cliffs of Moher
	      new google.maps.LatLng(53.016, -9.375),
	      new google.maps.LatLng(53.114, -9.148),
	      new google.maps.LatLng(53.139, -8.931),
	      new google.maps.LatLng(53.207, -8.868),
	      new google.maps.LatLng(53.271, -8.918),
	      new google.maps.LatLng(53.27055, -9.0566),	//Galway
	      new google.maps.LatLng(53.435, -7.941),
	      new google.maps.LatLng(53.34980, -6.26028),	//Dublin
	      new google.maps.LatLng(53.3076, -4.6310),
	      new google.maps.LatLng(53.204, -4.141),
	      new google.maps.LatLng(53.287, -3.716),	
	      new google.maps.LatLng(53.1968, -2.8798),
	      new google.maps.LatLng(51.5901, -2.9984),
	      new google.maps.LatLng(51.572, -2.649),
	      new google.maps.LatLng(51.44877, -2.5800),
	      new google.maps.LatLng(51.37737, -2.35709),    	//Bath
	      new google.maps.LatLng(51.0705, -1.8066),     	//Salisbury
	      new google.maps.LatLng(51.17885, -1.82618),    	//Stonehenge
	      new google.maps.LatLng(51.0705, -1.8066),     	//Salisbury
	      new google.maps.LatLng(51.53216, -0.12680),  	//London
	      new google.maps.LatLng(51.1086, 1.2870),
	      new google.maps.LatLng(50.9143, 1.805),
	      new google.maps.LatLng(50.62706, 3.0853),
	      new google.maps.LatLng(50.8354, 4.3355),       	//Brussels
	      new google.maps.LatLng(51.2094, 3.2246),	    	//Bruges
	      new google.maps.LatLng(50.8453, 4.3567),       	//Brussels
	      new google.maps.LatLng(51.2191, 4.421),		//Antwerp
	      new google.maps.LatLng(51.809, 4.658),
	      new google.maps.LatLng(52.0598, 4.3099),		//Den Haag
	      new google.maps.LatLng(52.3879, 4.6386),		//Haarlem
	      new google.maps.LatLng(52.3786, 4.9004),		//Amsterdam
	      new google.maps.LatLng(52.3144, 5.113),
	      new google.maps.LatLng(52.549, 5.639),
	      new google.maps.LatLng(52.514, 6.079),		//Zwolle
	      new google.maps.LatLng(53.2173, 6.564),		//Groningen
	      new google.maps.LatLng(53.2316, 7.4657),
	      new google.maps.LatLng(53.0827, 8.815),
	      new google.maps.LatLng(53.5544, 10.005),		//Hamburg
	      new google.maps.LatLng(53.8679, 10.6700),
	      new google.maps.LatLng(54.502, 11.228),
	      new google.maps.LatLng(54.652, 11.36),
	      new google.maps.LatLng(54.7671, 11.8772),
	      new google.maps.LatLng(55.6388, 12.0887),
	      new google.maps.LatLng(55.6730, 12.564),		//Copenhagen
	      new google.maps.LatLng(55.9155, 12.5007),
	      new google.maps.LatLng(55.9641, 12.5333),		//Humlebaek
	      new google.maps.LatLng(55.9155, 12.5007),
	      new google.maps.LatLng(55.6730, 12.564),		//Copenhagen
	      new google.maps.LatLng(55.6314, 12.6768),
	      new google.maps.LatLng(55.5655, 12.8917),
	      new google.maps.LatLng(55.7048, 13.1871),
	      new google.maps.LatLng(56.0443, 12.6954),
	      new google.maps.LatLng(56.5018, 12.9995),
	      new google.maps.LatLng(56.6692, 12.8658),
	      new google.maps.LatLng(57.7104, 11.9819),		//Gothenburg
	      new google.maps.LatLng(58.2876, 12.2990),
	      new google.maps.LatLng(58.9134, 11.9315),
	      new google.maps.LatLng(58.9659, 11.552),
	      new google.maps.LatLng(59.1206, 11.3859),
	      new google.maps.LatLng(59.2857, 11.1183),
	      new google.maps.LatLng(59.4319, 10.6565),
	      new google.maps.LatLng(59.7195, 10.8347),
	      new google.maps.LatLng(59.9095, 10.7598),		//Oslo
	      new google.maps.LatLng(59.913, 10.626),
	      new google.maps.LatLng(59.7407, 10.2042),
	      new google.maps.LatLng(59.7616, 9.919),
	      new google.maps.LatLng(60.052, 10.050),
	      new google.maps.LatLng(60.1688, 10.2490),
	      new google.maps.LatLng(60.4321, 9.4734),
	      new google.maps.LatLng(60.6991, 8.9698),
	      new google.maps.LatLng(60.6261, 8.5623),
	      new google.maps.LatLng(60.5356, 8.2068),
	      new google.maps.LatLng(60.4989, 8.0399),
	      new google.maps.LatLng(60.5607, 7.5869),
	      new google.maps.LatLng(60.6019, 7.5042),
	      new google.maps.LatLng(60.7352, 7.1229),
	      new google.maps.LatLng(60.6293, 6.4098),
	      new google.maps.LatLng(60.5869, 5.8148),
	      new google.maps.LatLng(60.455, 5.736),
	      new google.maps.LatLng(60.3894, 5.3354),		//Bergen
	      new google.maps.LatLng(60.455, 5.736),
	      new google.maps.LatLng(60.5869, 5.8148),
	      new google.maps.LatLng(60.6293, 6.4098),
	      new google.maps.LatLng(60.7352, 7.1229),
	      new google.maps.LatLng(60.6019, 7.5042),
	      new google.maps.LatLng(60.6019, 7.5042),
	      new google.maps.LatLng(60.5607, 7.5869),
	      new google.maps.LatLng(60.4989, 8.0399),
	      new google.maps.LatLng(60.5356, 8.2068),
	      new google.maps.LatLng(60.6261, 8.5623),
	      new google.maps.LatLng(60.6991, 8.9698),
	      new google.maps.LatLng(60.4321, 9.4734),
	      new google.maps.LatLng(60.1688, 10.2490),
	      new google.maps.LatLng(60.052, 10.050),
	      new google.maps.LatLng(59.7616, 9.919),
	      new google.maps.LatLng(59.7407, 10.2042),
 	      new google.maps.LatLng(59.913, 10.626),
	      new google.maps.LatLng(59.9095, 10.7598),		//Oslo
	      new google.maps.LatLng(60.189, 12.005),
	      new google.maps.LatLng(59.6533, 12.5912),
	      new google.maps.LatLng(59.3776, 13.4994),
	      new google.maps.LatLng(59.4182, 13.6920),
	      new google.maps.LatLng(59.2292, 14.4394),
	      new google.maps.LatLng(59.0668, 15.1098),
	      new google.maps.LatLng(58.9964, 16.2101),
	      new google.maps.LatLng(59.1790, 17.6459),
	      new google.maps.LatLng(59.3311, 18.0551),		//Stockholm
	      new google.maps.LatLng(59.3363, 18.2067),
	      new google.maps.LatLng(59.3794, 18.2948),
	      new google.maps.LatLng(59.3594, 18.4460),
	      new google.maps.LatLng(59.3970, 18.4426),
	      new google.maps.LatLng(59.4377, 18.3880),
	      new google.maps.LatLng(59.4482, 18.4287),
	      new google.maps.LatLng(59.4769, 18.4407),
	      new google.maps.LatLng(59.5045, 18.479),
	      new google.maps.LatLng(59.5757, 18.680),
	      new google.maps.LatLng(59.7195, 19.115),
	      new google.maps.LatLng(59.759, 19.319),
	      new google.maps.LatLng(60.068, 19.925),
	      new google.maps.LatLng(60.09231, 19.9279),
	      new google.maps.LatLng(60.068, 19.925),
	      new google.maps.LatLng(60.0130, 19.8542),
	      new google.maps.LatLng(59.807, 19.878),
	      new google.maps.LatLng(59.353, 22.72),
	      new google.maps.LatLng(60.146, 25.001),
	      new google.maps.LatLng(60.16780, 24.9528),	//Helsinki
	      new google.maps.LatLng(52.51630, 13.37769),	//Berlin
	      new google.maps.LatLng(51.0398, 13.7324),
	      new google.maps.LatLng(50.901, 14.221),
	      new google.maps.LatLng(50.7726, 14.2008),
	      new google.maps.LatLng(50.6595, 14.0448),
	      new google.maps.LatLng(50.5093, 14.0601),
	      new google.maps.LatLng(50.0826, 14.4353),		//Prague
	      new google.maps.LatLng(50.0309, 15.7563),
	      new google.maps.LatLng(49.8967, 16.4462),
	      new google.maps.LatLng(49.1898, 16.6130),
	      new google.maps.LatLng(48.7545, 16.8954),
	      new google.maps.LatLng(48.17483, 16.33662),	//Vienna
	      new google.maps.LatLng(48.2082, 15.6257),
	      new google.maps.LatLng(48.2896, 14.2928),
	      new google.maps.LatLng(47.8129, 13.0470),
	      new google.maps.LatLng(48.1405, 11.5569),		//Munich
	      new google.maps.LatLng(47.9854, 10.1867),
	      new google.maps.LatLng(47.54470, 9.6803),
	      new google.maps.LatLng(47.5509, 9.7194),
	      new google.maps.LatLng(47.5155, 9.7557),
	      new google.maps.LatLng(47.5035, 9.7419),
	      new google.maps.LatLng(47.4234, 9.3690),
	      new google.maps.LatLng(47.5002, 8.7228),
	      new google.maps.LatLng(47.3784, 8.5382),		//Zurich
	      new google.maps.LatLng(47.2958, 8.5636),
	      new google.maps.LatLng(47.1736, 8.5156),
	      new google.maps.LatLng(47.1801, 8.4634),
	      new google.maps.LatLng(47.0503, 8.3093),
	      new google.maps.LatLng(46.762, 8.139),
	      new google.maps.LatLng(46.7264, 8.1843),
	      new google.maps.LatLng(46.7548, 8.0368),
	      new google.maps.LatLng(46.6913, 7.8701),		//Interlaken
	      new google.maps.LatLng(46.5989, 7.9081),
	      new google.maps.LatLng(46.5753, 7.9390),
	      new google.maps.LatLng(46.5844, 7.9601),
	      new google.maps.LatLng(46.5745, 7.9742),		//Eiger trail
	      new google.maps.LatLng(46.62418, 8.0337),
	      new google.maps.LatLng(46.6328, 7.9009),
	      new google.maps.LatLng(46.6913, 7.8701),		//Interlaken
	      new google.maps.LatLng(46.7547, 7.6290),
	      new google.maps.LatLng(46.9496, 7.4396),
	      new google.maps.LatLng(46.8028, 7.1511),
	      new google.maps.LatLng(46.5161, 6.6290),
	      new google.maps.LatLng(46.5178, 6.5081),
	      new google.maps.LatLng(46.3851, 6.2366),
	      new google.maps.LatLng(46.21013, 6.1422),		//Geneva
	      new google.maps.LatLng(45.9021, 6.1204),		//Annecy
	      new google.maps.LatLng(45.6878, 5.9084),
	      new google.maps.LatLng(45.802, 5.853),
	      new google.maps.LatLng(45.95342, 5.3423),
	      new google.maps.LatLng(45.7605, 4.8613),
	      new google.maps.LatLng(43.9412, 4.8049),
	      new google.maps.LatLng(43.6849, 4.6327),
	      new google.maps.LatLng(43.5801, 4.9996),
	      new google.maps.LatLng(43.4879, 5.2307),
	      new google.maps.LatLng(43.3042, 5.3838),		//Marseille
	      new google.maps.LatLng(43.4879, 5.2307),
	      new google.maps.LatLng(43.5801, 4.9996),
	      new google.maps.LatLng(43.6849, 4.6327),
	      new google.maps.LatLng(43.8329, 4.3658),
	      new google.maps.LatLng(43.6050, 3.8816),
	      new google.maps.LatLng(43.3370, 3.2190),
	      new google.maps.LatLng(43.1899, 3.0065),
	      new google.maps.LatLng(42.544, 2.848),
	      new google.maps.LatLng(42.2649, 2.9683),
	      new google.maps.LatLng(41.9784, 2.8171),
	      new google.maps.LatLng(41.7753, 2.7407),
	      new google.maps.LatLng(41.548, 2.227),
	      new google.maps.LatLng(41.3795, 2.1418),		//Barcelona
	      new google.maps.LatLng(41.548, 2.227),
	      new google.maps.LatLng(41.7753, 2.7407),
	      new google.maps.LatLng(41.9784, 2.8171),
	      new google.maps.LatLng(42.2649, 2.9683),
	      new google.maps.LatLng(42.544, 2.848),
	      new google.maps.LatLng(43.1899, 3.0065),
	      new google.maps.LatLng(43.2172, 2.3502),
	      new google.maps.LatLng(43.61116, 1.45425),
	      new google.maps.LatLng(43.7035, 1.8137),
	      new google.maps.LatLng(43.5995, 2.2302),		//Castres
	      new google.maps.LatLng(43.7035, 1.8137),
	      new google.maps.LatLng(43.61116, 1.45425),
	      new google.maps.LatLng(44.0139, 1.3405),
	      new google.maps.LatLng(44.2079, 0.6214),
	      new google.maps.LatLng(44.8258, -0.5553),		//Bordeaux
	      new google.maps.LatLng(44.6222, -1.002),
	      new google.maps.LatLng(44.6585, -1.1653),
	      new google.maps.LatLng(44.65592, -1.25991),	//Cap ferret
	      new google.maps.LatLng(44.6585, -1.1653),
	      new google.maps.LatLng(44.6222, -1.002),
	      new google.maps.LatLng(44.8258, -0.5553),		//Bordeaux
	      new google.maps.LatLng(44.9918, -0.440),
	      new google.maps.LatLng(45.7482, -0.6182),
	      new google.maps.LatLng(46.1528, -1.1431),
	      new google.maps.LatLng(46.409, -0.892),
	      new google.maps.LatLng(47.2182, -1.5363),
	      new google.maps.LatLng(48.1027, -1.6725),		//Rennes
	      new google.maps.LatLng(48.6357, -1.5112),		//Mont Saint Michel
	      new google.maps.LatLng(48.1027, -1.6725),
	      new google.maps.LatLng(47.99541, 0.1911),
	      new google.maps.LatLng(48.8778, 2.3605),		//Paris gare de lest
	      new google.maps.LatLng(49.2588, 4.0241),
	      new google.maps.LatLng(49.1096, 6.1771),
	      new google.maps.LatLng(49.5994, 6.1355),		//Luxembourg
	      new google.maps.LatLng(49.1096, 6.1771),
	      new google.maps.LatLng(48.5851, 7.7336),		//Strasbourg	
	      new google.maps.LatLng(48.47824, 7.9475),
	      new google.maps.LatLng(48.9936, 8.4013),
	      new google.maps.LatLng(48.7848, 9.1827),		//Stuttgart
	      new google.maps.LatLng(48.9936, 8.4013),
	      new google.maps.LatLng(50.0507, 8.5709),
	      new google.maps.LatLng(50.9433, 6.9587),		//Cologne
	      new google.maps.LatLng(51.2196, 6.7936),
	      new google.maps.LatLng(51.4291, 6.7765),
	      new google.maps.LatLng(51.53123, 7.1659),
	      new google.maps.LatLng(51.9564, 7.6352),
	      new google.maps.LatLng(52.2759, 7.4342),
	      new google.maps.LatLng(52.2092, 5.9692),
	      new google.maps.LatLng(52.1541, 5.3728),
	      new google.maps.LatLng(52.3786, 4.9004),		//Amsterdam
	      new google.maps.LatLng(52.3879, 4.6386),		//Haarlem
	      new google.maps.LatLng(52.0598, 4.3099),		//Den Haag
	      new google.maps.LatLng(51.809, 4.658),
	      new google.maps.LatLng(51.2191, 4.421),		//Antwerp
	      new google.maps.LatLng(50.8453, 4.3567),       	//Brussels
	      new google.maps.LatLng(50.8354, 4.3355),       	//Brussels
	      new google.maps.LatLng(50.62706, 3.0853),
	      new google.maps.LatLng(48.8822, 2.3563)		//Paris gare du nord 
	  ];
	  
	  var routePath = new google.maps.Polyline({
	    path: routeCoordinates,
	    strokeColor: '#FF0000',
	    strokeOpacity: 1.0,
	    strokeWeight: 2
	  });
	  
	  //Use the Polyline to calculate the distance travelled for later
	  document.getElementById("distanceTravelled").innerHTML = Math.round(routePath.inKm())+' km';
	  
	  //Add the Polyline to the map canvas
	  routePath.setMap(map);
	  
	  //variables and list for the marker's to link back to the travel blog
	  var tagURL = 'http://meandmypack.wordpress.com/tag/';
	  
	  var mainCities = [
	      [51.51120, -0.11978, 'london', 'London'],
	      [55.95324, -3.18827, 'edinburgh', 'Edinburgh'],
	      [56.23381, -4.4290, 'highlands', 'Scottish Highlands'],
	      [54.5971, -5.930, 'belfast', 'Belfast'],
	      [55.24881, -6.48898, 'giants-causeway', 'Giant\'s Causeway'],
	      [53.27055, -9.0566, 'galway', 'Galway'],
	      [52.97184, -9.42649, 'cliffs-of-moher', 'Cliffs of Moher'],
	      [53.34980, -6.26028, 'dublin', 'Dublin'],
	      [51.37737, -2.35709, 'bath', 'Bath'],
	      [51.0705, -1.8066, 'salisbury', 'Salisbury'],
	      [51.17885, -1.82618, 'stonehenge', 'Stonehenge'],
	      [51.2094, 3.2246, 'bruges', 'Bruges'],
	      [50.8354, 4.3355, 'brussels', 'Brussels'],
	      [52.3786, 4.9004, 'amsterdam', 'Amsterdam'],
	      [53.2173, 6.564, 'groningen', 'Groningen'],
	      [53.5544, 10.005, 'hamburg', 'Hamburg'],
	      [55.6730, 12.564, 'copenhagen', 'Copenhagen'],
	      [57.7104, 11.9819, 'gothenburg', 'Gothenburg'],
	      [59.9095, 10.7598, 'oslo', 'Oslo'],
	      [60.3894, 5.3354, 'bergen', 'Bergen'],
	      [59.3311, 18.0551, 'stockholm', 'Stockholm'],
	      [60.16780, 24.9528, 'helsinki', 'Helsinki'],
	      [52.51630, 13.37769, 'berlin', 'Berlin'],
	      [50.0826, 14.4353, 'prague', 'Prague'],
	      [48.17483, 16.33662, 'vienna', 'Vienna'],
	      [48.1405, 11.5569, 'munich', 'Munich'],
	      [47.3784, 8.5382, 'zurich', 'Zurich'],
	      [46.6913, 7.8701, 'interlaken', 'Interlaken'],
	      [46.5745, 7.9742, 'eiger-trail', 'The Eiger Trail'],
	      [45.9021, 6.1204, 'annecy', 'Annecy'],
	      [43.3042, 5.3838, 'marseille', 'Marseille'],
	      [41.3795, 2.1418, 'barcelona', 'Barcelona'],
	      [43.5995, 2.2302, 'castres', 'Castres'],
	      [44.8258, -0.5553, 'bordeaux', 'Bordeaux'],
	      [44.65592, -1.25991, 'cap-ferret', 'Arcachon and Cap Ferret'],
	      [48.1027, -1.6725, 'rennes', 'Rennes'],
	      [48.6357, -1.5112, 'mont-saint-michel', 'Mont Saint Michel'],
	      [49.5994, 6.1355, 'luxembourg', 'Luxembourg'],
	      [48.5851, 7.7336, 'strasbourg', 'Strasbourg'],
	      [48.7848, 9.1827, 'stuttgart', 'Stuttgart'],
	      [50.9433, 6.9587, 'cologne', 'Cologne'],
	      [52.3879, 4.6386, 'haarlem', 'Haarlem'],
	      [48.8822, 2.3563, 'paris', 'Paris']	      
	  ];
	  
	  var markers = [];
	  
	  //Stick those markers into the map canvas
	  for (var i = 0; i < mainCities.length; i++) {
	    var marker = new google.maps.Marker({
	      position: new google.maps.LatLng(mainCities[i][0], mainCities[i][1]),
	      map: map
	    });
	    var infowindow = new google.maps.InfoWindow({
	      content: '<a href="'+tagURL+mainCities[i][2]+'/" target="blank">'+mainCities[i][3]+'</a>'
	    });
	
	    makeInfoWindowEvent(map, infowindow, marker);
	    
	    markers.push(marker);
	  }
	}
	
	//The info window function from http://jsfiddle.net/yV6xv/161/
	function makeInfoWindowEvent(map, infowindow, marker) {
	  google.maps.event.addListener(marker, 'click', function() {
	    infowindow.open(map, marker);
	  });
	}
	
	//The polyline distance code from https://groups.google.com/forum/#!topic/google-maps-js-api-v3/Op87g7lBotc
	google.maps.LatLng.prototype.kmTo = function(a){ 
    	  var e = Math, ra = e.PI/180; 
    	  var b = this.lat() * ra, c = a.lat() * ra, d = b - c; 
    	  var g = this.lng() * ra - a.lng() * ra; 
    	  var f = 2 * e.asin(e.sqrt(e.pow(e.sin(d/2), 2) + e.cos(b) * e.cos(c) * e.pow(e.sin(g/2), 2))); 
    	  return f * 6378.137; 
  	}
  	
  	google.maps.Polyline.prototype.inKm = function(n){ 
    	  var a = this.getPath(n), len = a.getLength(), dist = 0; 
    	    for(var i=0; i<len-1; i++){ 
      	    dist += a.getAt(i).kmTo(a.getAt(i+1)); 
    	  } 
    	  return dist; 
  	}
  	
	
	google.maps.event.addDomListener(window, 'load', initialize);
	
    </script>
  </head>
  <body>
        <div id="map-canvas" style="float:left;width:100%;height:100%;"></div>
        <div id="info-panel" style="float:right;text-align:left;">
        <div style="margin:10px;border-width:2px;float:center;text-align:center;">
          <h3>Me and My Pack Interactive Route Map</h3>
          <b>Distance Travelled: </b>
          <div id="distanceTravelled"></div><br>
          <a href="http://meandmypack.wordpress.com" target="blank">meandmypack.wordpress.com</a><br>
          <a href="http://everettsprojects.com" target="blank">everettsprojects.com</a>
    	</div>
  </body>
</html>

default.css

html, body {
  background-color:#b0c4de;
  height: 100%;
  margin: 0;
  padding: 0;
}

#map-canvas, #map_canvas {
  height: 100%;
}

@media print {
  html, body {
    height: auto;
  }

  #map-canvas, #map_canvas {
    height: 650px;
  }
}

#info-panel {
  width: 25%;
  font-size: 12px;
  position: absolute;
  top: 10px;
  left: 90px;
  background-color: #fff;
  padding: 2px;
  border: 1px solid #999;
  background: rgba(255, 255, 255, 1);
  -webkit-border-radius: 5px;
  -moz-border-radius: 5px;
  -ms-border-radius: 5px;
  -o-border-radius: 5px;
  border-radius: 5px;
  border: outset 1px #a1b5cf;
}
Posted by Everett in Programming, Web Applications, 0 comments
Arduino: Super Graphing Data Logger

Arduino: Super Graphing Data Logger

The intensity of natural light in my basement.
Sections:

  1. Introduction
  2. The Results
  3. How to Make One For Yourself

Introduction

What is the Super Graphing Data Logger (SGDL)? It is an Arduino project that integrates data logging and the graphing of this data online using little more than an Arduino with the appropriate shields and sensors.   It differs from similar projects in that it doesn’t require a separate server or system to collect the data or to run script for the actual plot. Between the Arduino and the user’s browser, everything is taken care of.

If you just want to dive right in, the code is now posted on GitHub: https://github.com/evjrob/super-graphing-data-logger

Some time back I came across this neat javaScript based library for plotting and graphing called Highcharts JS. It didn’t take long for me to realize that charting with javaScript is very convenient for projects in which the server is limited in it’s capabilities, such as when using an Arduino with the Ethernet shield. Since the user’s browser does all the heavy lifting, the Arduino only needs to serve the files which is something it is perfectly capable of. This is especially true now that the Ethernet and SD libraries included in 1.0 support opening of multiple files simultaneously amongst other things. Thus the use of Highcharts allows us to create beautiful interactive charts based on data logged by the Arduino using nothing but the Arduino (and your browser, and a public javaScript CDN).

The Results

The best way to appreciate the final product is to actually play with it. While I’m not going to open up my home network and Arduino to the big wide internet, I have mirrored the pages and datafiles it produces on the webhost I used for my Has the World Ended Yet? project. You can find them here. These won’t be updated with new datapoints like the actual Arduino version will be, but they should at least give a fair impression of how the project looks and feels without the need to actually implement it.

For those who are unsure what they are looking at, I’ll offer a quick interpretation:

The list of data files available for graphing.

The list of data files available for graphing.

Going to the above page, we see that we are presented with a very basic list of the data files that can be selected from. Clicking any of them will cause  the graph for that datafile to be loaded (much more quickly than the Arduino can manage).

A graph for the first week of data collected.

A graph for the first week of data collected.

This chart for the 25-12-12.CSV file is already complete, and won’t have any new data added to it in the future, because the files for subsequent weeks have already been made. There is a lot to see though. The two data points that are at 1000 on the y-axis are from when I pointed a bright flashlight directly at the photo sensor. All of the data points between 300 and 400 on the y-axis are the result of the basement lights being on. The abnormally large gaps in the data are periods when the Arduino was powered off because I was still tweaking and developing it. Finally, the short humps that occur everyday are the result of natural light coming through one of the basement windows. By zooming in on one of them, we can see even more detail:

The intensity of natural light in my basement.

The intensity of natural light in my basement.

The first thing we notice is that the levels rise from zero to about 65 before falling and levelling out at close to 35 for two hours. This is followed by a another small increase before it ultimately decreases down to a value of ~10 where it levels out. That middle valley where the light levels are equal to 35 is due to the shadow cast  on the basement window by our neighbour’s house to the south of us. The levelling out of the light intensity at 10 after all the daylight has disappeared is because a light out in the hallway is usually on in the evening. It is eventually turned off for the night, causing the light levels to drop to zero where they will usually remain until the next morning. I must admit, I’m impressed that the cheap $1.00 photoresistor is capable of capturing this level of detail, and that these trends are so easily interpreted from the graphs.

How to Make One For Yourself

All of the code below is now conveniently hosted on GitHub (https://github.com/evjrob/super-graphing-data-logger) so you no longer need to copy and paste it if you don’t want to.

To replicate this project, a few things are necessary. You’ll obviously need an Arduino capable of connecting over Ethernet and storing files on an SD card. In my case, this is achieved through the use of an Uno with the Ethernet shield. Presumably an Arduino Ethernet model will also work fine, though I have not personally tested it. Other non official Ethernet shields and SD card adapters may also work if they use the same libraries, though I make no guarantees. For the more adventurous, it may be possible to adapt my code to achieve the same functionality using a Wifi shield. You will also need a data source of some sort. For my project I chose to use a very cheap photoresistor, which I rigged up on a small perf board to plug directly into the 5v, gnd, and A0 pins of my Arduino (or more precisely,  the headers on the Ethernet shield). It is set up in such a way that the minimum recordable light intensity is zero, while the maximum is 1024.

The photo sensor board fits like a charm.

The photo sensor board fits like a charm.

DSCF2931

One header is bent to reach A0.

The pins on the male headers don’t quite line up, so I intentionally used extra long ones and added a slight S-curve to the one that goes to A0. This can be seen in better detail above. For those who are interested, the circuit is very simple:

The circuit.

The circuit.

Before we get started, we need to make sure our SD card is good to go. It should be formatted as a FAT16 or FAT32 filesystem, the details of which are available on the official Arduino website. Once that is done, we need to ensure two things are present in the root directory of the card: the HC.htm file, and a data/ directory for our datafiles. The data directory is easily made with the same computer that was used to format the card provided one has an SD card reader of some sort. The HC.htm simply consists of the following code:

You will need to edit this file first to make sure it points towards the preferred  location of your highcharts.js files. You can leave this as the public CDN: http://cdnjs.cloudflare.com/ajax/libs/highcharts/2.3.5/highcharts.js, change it to point towards your own webhost, or it can even be on the Arduino’s SD card (this will be slow). It is not necessary to create a datafile before hand, the SGDL sketch will take care of that when it decides to record its first data point. Before we get that far though, it is necessary to make sure we have configured the EEPROM memory for the SGDL sketch. This is very easily accomplished using a separate sketch, which I have called EEPROM_config. This sketch (along with SGDL itself) requires an extra library called EEPROMAnything, which needs to be added to the Arduino’s libraries folder wherever one’s sketchbook folder is. While you’re at it, you should also add the Time library which we need for SGDL.

I have intentionally commented out the write line so that no one writes junk to the EEPROM by accident. While the EEPROM has a life of ~100,000 write cycles, I’d rather not waste any of them. Please review the sketch carefully and ensure you’ve adjusted it accordingly before uploading it to the Arduino. The most important thing is to ensure that your newFileTime is something sensible (in the near future most of all).

Now that that’s all taken care of, we’re ready to get SGDL all set up! The code will need a few adjustments for your own specific setup, mostly in regards to the Ethernet MAC and IP addresses. I trust that anyone making use of this code already knows how to configure their router to work with the Arduino, and that they can find the appropriate local IP address to update this sketch with. You may also wish to change the timeserver IP address to one that is geographically closer to yourself.

I currently have my code set up to make a measurement every 10 minutes, and to create a new data file every week. You are welcome to change those parameters, just be aware that the current data file management names files using a dd-mm-yy.csv date format, so the new file interval should be at least 24 hours. Another concern, is that the shorter the measurement interval and the longer the new data file interval is, the larger the files will be. Because the Arduino is not especially powerful, this will have consequences for the loading times of each chart.

Posted by Everett in Arduino, Electronics, Programming, Sensors and Data Logging, Web Applications, 75 comments
Has the world ended yet? A first attempt at web development

Has the world ended yet? A first attempt at web development

Despite the sheer nuttiness of it, everyone keeps going on about the end of the world as “predicted” by the Mayan calendar. National Geographic even had and entire day devoted to it. Building on that theme, I decided to make a very convenient (and pretty much useless) webpage that helps you figure out if the world has in fact ended: http://everett.x10.mx/end-of-the-world.php. This project was really simple, didn’t involve a lot of code or design, and was basically thrown together over the course of an hour and a half. It turns out PHP is extremely easy if your host is already configured for it, and I’m looking forward to doing some more web development related stuff both with PHP and other languages or tools. The HTML side of the page was also relatively straightforward. I’m impressed by what’s possible design wise using modern HTML and CSS3. My inspiration on that front was this amazing site: http://www.tubalr.com/

screenshot

It works as the page implies, by polling google.com for a response. If google is down, then it is assumed the world has ended, and the result is Yes.in big red letters. The PHP that does the trick is a slightly modified version of what’s posted at the following site: http://css-tricks.com/snippets/php/check-if-website-is-available/. The background is not mine, but I’ve left attribution on the image, and you can find the originals here: http://m3-f.deviantart.com/gallery/?offset=24#/d3b4qgn.

And because I see no reason not to release it, here is the entire source code for the page:

 <?php
   function Visit($url){
     $agent = "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)";$ch=curl_init();
     curl_setopt ($ch, CURLOPT_URL,$url );
     curl_setopt($ch, CURLOPT_USERAGENT, $agent);
     curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
     curl_setopt ($ch,CURLOPT_VERBOSE,false);
     curl_setopt($ch, CURLOPT_TIMEOUT, 5);
     curl_setopt($ch,CURLOPT_SSL_VERIFYPEER, FALSE);
     curl_setopt($ch,CURLOPT_SSLVERSION,3);
     curl_setopt($ch,CURLOPT_SSL_VERIFYHOST, FALSE);
     $page=curl_exec($ch);
     //echo curl_error($ch);
     $httpcode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
     curl_close($ch);
     if($httpcode>=200 && $httpcode<300) return true;
     else return false;
   }
   if (Visit("http://www.google.com")){
     $answer = "No.";
     $colour = "green";
   }
   else{
     $answer = "Yes.";
     $colour = "red";
   }
?>

<!DOCTYPE html>
<html>
  <head>
    <title>Has the World Ended Yet?</title>
<style>
  a:link {color:#FFFFFF;}
  a:visited {color:#FFFFFF;}

html {
  overflow-y: scroll;
  background: url(/backgrounds/eow.jpg) no-repeat center center fixed;
  -webkit-background-size: cover;
  -moz-background-size: cover;
  -o-background-size: cover;
  background-size: cover;

}

body {
  font-family: 'Open Sans', sans-serif;
  font-size: 24px;
  color: #fff;
  padding-bottom: 20px;
}

#main
{
  text-align: center;
  margin-top: 50px;
  margin-bottom: 20px;
  background: #000;
  background: rgba(0, 0, 0, 0.85);
  -webkit-border-radius: 5px;
  -moz-border-radius: 5px;
  -ms-border-radius: 5px;
  -o-border-radius: 5px;
  border-radius: 5px;
  -webkit-box-shadow: 0 0 5px rgba(0, 0, 0, 0.5);
  -moz-box-shadow: 0 0 5px rgba(0, 0, 0, 0.5);
  box-shadow: 0 0 5px rgba(0, 0, 0, 0.5);
  border: solid 1px #000;
  width:800px;
  margin-left:auto;
  margin-right:auto;
}
#result
{
  font-family: 'Open Sans', sans-serif;
  font-size: 112px;
  color: <?=$colour?>;
}

#disclaimer
{
  font-family: 'Open Sans', sans-serif;
  font-size: 12px;
  color: #fff;
  margin-top: 80px;
  margin-left: 100px;
  margin-right: 100px;
  margin-bottom: 50px;
}
</style>

  </head>
  <body>
    <div id="main">
        <H1>Has the world ended yet? <sup>*</sup></H1>
        <br>
        <div id="result">
            <b> <?=$answer?></b>
        </div>
        <div id="disclaimer">
            <sup>*</sup> Does not actually check if the world has ended. Result is based on the assumption that if Google.com is not responding, the world has probably ended. <br><br> <a href="http://everettsprojects.com">http://everettsprojects.com/</a>
        </div>
    </div>
</body>
</html>
Posted by Everett in Programming, Web Applications, 3 comments
Arduino Project: HTML to LCD

Arduino Project: HTML to LCD

The original source code for this project can be found Here.

WHAT YOU NEED:

  1. Arduino (I use the UNO, but I think the older revisions should also work)
  2. An Ethernet Shield
  3. 16×2 Basic Character LCD (There are many more colour options than just white on black)
  4. An Ethernet cable (I just used an old one lying around)
  5. A Breadboard  or some other means to connect the LCD to the Arduino and one 2.2 kΩ resistor

WHAT IT DOES:

This project combines the above to turn the Arduino into web server which is hooked up to the LCD. It produces a simple HTML web page, from which the user may see what text is currently displayed on the LCD, and provides them the opportunity to change the text using simple input forms. The hardware side of this project is fairly simple, and there were no physical hacks or modifications that needed to be made. The real challenge to this project were working within the limitations of the Arduino as a computing device.

SETUP AND OPERATION:

As we can see, the Ethernet shield is plugged into the Arduino, and is hooked up to the breadboard and LCD according to the scheme outlined in the comments of the source code. This one is completely set up and running, since there is already some text displayed on the LCD. The Arduino’s digital pins 4, 10, 11, 12 and 13 are left free since they are utilized by the shield. All the remaining pins are used by the LCD, so unfortunately this means all of the digital IO pins are used which restricts future additions to this project . The Arduino is also being powered over the USB cable, since this allows me to use the serial monitor to debug, and also because the Wiznet chip and voltage regulator get very hot when my 9V wall adapter is used. I have read that this may be because the shield draws a fair amount of current, and the linear voltage regulator reduces the voltage in accordance with the equation       P = I*ΔV. The power is proportional to the current draw, and it is dissipated by the voltage regulator in the form of heat.

With those notes aside, we can connect to the web page by typing in the local network IP address of the Arduino, which I have set up to be fixed using my router’s DHCP reservation functionality. The Arduino did not automatically receive an IP address when the Ethernet cable was first plugged in, so it was necessary to manually add the it to the router’s client list using the mac address on the underside of the shield.

On the web page served by the Arduino it tells us what is currently displayed by the LCD, and it provides us with two boxes to enter more text, one for each row. Each row is limited by the HTML code to 16 characters which provides instant feedback about the limitations of our setup to the user.

Typing in a couple of new lines and clicking the submit button causes the page to refresh and update with what we just entered! In a related issue, due to the RAM limitations of the Arduino and the way symbols are encoded for a URL (a % sign followed by two Hex digits), symbols initially use three times as many bytes as normal alpha-numeric characters. Therefore, if a user were to enter nothing but symbols in each field, the Arduino would crash due to memory issues. To solve this problem I added some code to limit the length of the raw text accepted, but this means that if a user enters too many symbols, the second line will be truncated. In fact, if nothing but symbols are entered into both lines, the second line will be truncated to nothing. It is an unsatisfactory solution, but it’s not an issue if the user uses symbols responsibly.

 Putting these issues aside and returning to our example, we can see that the the web page is now displaying what we entered earlier, and that the LCD also reflects the changes made on the web page.

While my Arduino is only hooked up to my local area network, it is possible to release it into the wild world of the Internet, though there is some risk (which I lack the expertise to properly assess) to doing so on your home network. There are some instructions for implementing this at http://sheepdogguides.com/arduino/art5serv.htm involving changing the settings on your home router and utilizing DynDNS to get a readable and static domain name.

The most difficult aspect of this whole project was without a doubt managing the heavy use of strings within the Arduino’s small amount of  RAM. Lots of work went into preventing any major or obvious memory leaks, but I am not entirely sure that I have gotten them all. To ration the memory effectively, I also utilized the ability of the Arduino to store constants in flash memory for the strings of HTML code that are sent to the client’s browser. This means I have at most only a single line of HTML stored in RAM at any time.

That wraps up my first post and first major Arduino project that involved some real coding and which was more than just random experimentation and tinkering. There is still some work that could be done debugging, and I intend to work on some extreme and border conditions to see if there are any bugs remaining that will cause it to crash or do some other bad thing.

(what fun is it if you don’t try to break it?)

As a reminder, the source code for this project can be found Here.

Posted by Everett in Arduino, Electronics, 36 comments