11/14/2005

Backing Up Web Data Using cURL

Filed under: howto, webservices — ryan @ 11:07 pm

More and more of my important data is stored on the web: my bookmarks at del.icio.us, my citations at CiteULike, and my notes at Backpack and Writeboard. I love these services, but I am fully aware that they could disappear at any time, taking my precious data with them. Fortunately, they all make it easy for me to back up my data to the location of my choice (in this case the SIMS servers, which are themselves backed up regularly).

Backing up Backpack data:

#! /bin/sh

curl -s -H 'X-POST_DATA_FORMAT: xml' 
    -o "/home/ryanshaw/backup/backpack/$$.xml" 
    -d '<request><token>[My Backpack API token]</token></request>' 
    http://rybesh.backpackit.com/ws/account/export

Backing up CiteULike data:

#! /bin/sh

URL="http://www.citeulike.org/bibtex/user"
DIR="/home/ryanshaw/backup/citeulike"

cd "$DIR/ryanshaw"
mv -f *.xml backup/
curl -s "$URL/ryanshaw" | /home/ryanshaw/bin/bib2xml -s 2> /dev/null

Backing up del.icio.us data:

#! /bin/sh

curl -s --user rybesh:[my password] 
    -o "/home/ryanshaw/backup/delicious/$$.xml" 
    http://del.icio.us/api/posts/all

Backing up Writeboard data:

#! /bin/sh

url="http://123.writeboard.com"
dir="/home/ryanshaw/backup/writeboard"
cookies="$dir/cookies.txt"

# args: writeboard_id, password
export_writeboard () {
    [ -d "$dir/$1" ] || mkdir "$dir/$1"
    curl -s -L -c $cookies -o /dev/null -d "password=$2" "$url/$1/login"
    curl -s -b $cookies -o "$dir/$1/$$.html" "$url/$1/v/export?format=html"
    rm -f $cookies
}

export_writeboard [writeboard ID from URL] [writeboard password]

One Response to “Backing Up Web Data Using cURL”

  1. David wrote:

    Hi there,

    thanks for the curl script. I was just willing to write on myself, and was browsing for the delicious api, when I found your script. Works like a charm.

Leave a Reply

Powered by WordPress