Seeing The Forest But For The Trees: On Exist.io

So many services, so much data. How can I make sense of it all? You’ve probably had this thought yourself on occasion. You have your Fitbit data, your calendar data, your email data, your task management system, your last.fm music data and so on. All of these things exist in their own silos.

If you’re halfway intrepid, you’ve maybe even made efforts to liberate your data from these prisons. Best case scenario: you’re left with a half-complete .CSV file that is incompatible with all the other .csv files you’ve downloaded from other services. Now, if you want to hook all these files up together, you’ll have to clean them up, massage the various tables and forms so that they match up. And then you’ll have to perform a whole host of calculations in order to figure out how you’re doing, or what the data has to tell you.

Collating all this data and making sense of it can be a full-time job in and of itself. I admire those who give talks at institutions like the Quantified Self, telling tales derived from the past 20 years of financial tracking or weight data.

Personally, I don’t have time to do all of that. I need a service to help me out, something that will serve up platters of charts and correlations that address the main broad questions that I might address of these various data streams, notably:

• how am I feeling?
• how am I sleeping?
• am I moving enough?
• am I getting enough work done?

Now you might say that it’s easy to look at each of the services individually and you’d be right. I can look at a chart over at Fitbit.com and I’ll see something that shows me all the data for the past 12 months:

This is nice, but it doesn’t tell me a great deal, aside from the fact that I seem to be walking less in Amman than when I lived next to a huge forest in a provincial part of Holland. No surprises there. Also, a whole year is a bit too much for me to process. It doesn’t really help me calibrate my current actions and my plan going forward.

Enter, Exist. I’ve been using Exist for a little over a year now, checking in on the site pretty much every day. Their own explanation for what they do is:

“We turn numbers into insights. We collect data from the services you already use and find trends and correlations in the results.”

You hook up all your web services, and your Exist dashboard will be populated with pretty graphs, useful correlations and various other goodies. These are the services currently supported:

Your dashboard page has a bunch of different panels which aggregate data from different parts of your life. For instance, my activity panel for today looks like this:

I usually go for a walk in the evenings, so I’ll most likely end today with a higher count, but you get the idea. There are panels for activity, productivity, sleep, mood, workouts, health, location, social media, music and weather.

So the dashboard is where you can come to get a snapshot of the reality of your current short-term. Exist also offers correlations and suggestions based on its combination and number-crunching of all the various parts of your data stream. This is often most interesting when it comes to your mood data. Check out these recent correlations that Exist crunched from my self-reported mood data alongside the other automated streams:

There are so many different types of charts and graphs; it’s difficult to pick my favourite ones. Here you can see my slow but steady battle to increase the amount of time I sleep every day, for example:

The mood data is self-reported. Exist sends you an email every day at a time of your choosing and you respond with a number from 1–5 alongside some comments about your subjective experience of the day. These comments are also then processed into useful charts and prompts. Weird things I’ve learned from Exist’s number crunching:

• the less I sleep, the more likely I am to have a ‘perfect’ day (i.e. 5 on the 1–5 mood scale)
• the more it rains or snows, the happier I am

You can use Exist in a bunch of different ways, depending on what you hope to get out of it. For my own situation, I love how it encourages awareness of my current patterns on the short- to medium-term timescale (i.e. one week to three months). For things like sleep, where it’s easy to forget whether you’re managing to do what you seek to do (i.e. sleep more, or get up earlier etc), Exist is really excellent.

There's a sizeable userbase who actively contribute to shaping the future of the company / service through feedback. There are a bunch of things I'd love to see happen with Exist -- changing the 1-5 scale for mood feedback to 1-10 as a start -- but the site is actively developed, and I'm really happy with all the new features that have been added in the year or so since I signed up.

You can find out the future / expansion plans that Belle and Josh have for Exist here. There’ll be more services and correlations and integrations will be more usefully presented. All of this is as you’d expect. Belle and Josh are good people (together, they make up HelloCode which is the Australia-based parent entity that produces Exist, among other things). I spoke to Belle for a Sources and Methods podcast episode; that’ll be out soon, I hope, so keep your eyes out for that.

In my interactions with Belle / Josh I’ve been really impressed with the service that they offer through Exist and their other products. I put them in my personal category of ‘nice, kind people seeking to do good things in the world through offering really useful services’. Like the team who run Beeminder and GMB Fitness, Belle and Josh offer something that saves me time and adds meaning on a daily basis to my life. Exist is a paid service. If you already run a bunch of the services listed above in the diagram, you’ll probably benefit from Exist’s charts and correlations. I hope you’ll consider signing up for their free one-month trial.

PhD Tools: Beeminder

[This is part of a series on the tools I used to write my PhD. Check out the other parts here.]

I feel like I've mentioned the end of the PhD several times in recent posts (PHD IS OVER!). It occurred to me that it might be useful to go through some of the tools and principles that I found most useful in completing the doctoral thesis, the research and the work in general. Part of this is by way of giving thanks to the application or methodological creators, and the other part is me thinking that others (future / current PhD students?) might find this useful.

It took me many years to finally settle on these tools. It would probably be unwise to adopt my entire writing style and process for yourself, because everyone's unique. I read a lot of books, blogposts and discussed things in forums and at meetings with others. This is all the product of a lot of procrastination (some active, some just resulting from hanging out on twitter or subscribing to a bunch of productivity-related blogs in my RSS reader).

Each post will vary in size. For some I'll go into a bit more detail because the principle will be somewhat unknown. Others are mega-players in the tech world so I'll just tip my hat in their direction.

Minding the Bees

My first pick is, of course, Beeminder. (I've written about Beeminder before here.) The principle behind this service is pretty simple: you commit to doing a certain thing (or things) by a certain date (or regularly each day etc) and if you don't do them, you're penalised with money taken from your credit/debit card. The amount of money taken depends on whether you're a first-time offender (free, or $5), but then it increases exponentially. Pretty soon you'll be facing $270 or even higher fines.

Needless to say, this is a pretty strong motivator. You can hear about some of the nitty-gritty details in a podcast interview I did with Matt Trevithick and the founders of Beeminder, Bethany Soule and Daniel Reeves.

I have used Beeminder for a really wide variety of things -- not just for my work but for my personal life, too -- but in terms of my PhD, I had three main goals it supported:

1) tracking the amount of time I spent writing. You can hook up RescueTime (a passive activity tracker on your laptop) to feed into Beeminder. I can then say that I want to make sure I do a minimum of 1 hour of writing in Scrivener each day (for example), and Beeminder keeps track of the rest. This is a good thing to track because, ultimately, the PhD is all about keeping writing. You can get lost in the research, but after a certain point you just have to deliver it and ship the damn thing. This keeps you honest about the writing part, the sitting down in the chair and putting words on the page.

2) words drafted -- this one's a bit more delicate, since often when you're starting out, drafting a new section or chapter, the words that come out are useless drivel (or replace with a far less charitable way of describing their quality, and say hi to my inner voice while you're there!). At the beginning, doing basic drafting, it's hard to get started because you feel everything has to be perfect. The best antidote to this is to work on a 'shitty first draft'. Here, the idea is simply to churn out enough thoughts to fill the blank space in the outline, or book, or chapter or wherever.
A specific example: I flew to Karachi in late 2012 to hammer out the first draft of my dissertation. I setup a Beeminder goal of having 100,000 words of text (approx the maximum word count allowed for submission to the university) and a date 6 weeks in the future, and I got writing. Beeminder calculates and tells you how many words you have to get done each day in order to stay ahead of the curve. (There are graphs. They are awesome). As long as you keep writing, you're ok. And I did it. Most was horrible, and some of it was inner conversations between myself and myself about the subject under consideration, almost all of which I had to rewrite in some shape or form later on. But... it was words on the page, and it was me thinking through the issues. It was essential.

3) Sources Read -- this might be unique to me, but at some point I had to return to the newly-gathered sources of the Taliban Sources Project. I looked in my DevonThink database (about which, more to come in a future post) and saw I'd flagged 1000+ articles to reread, catalogue/tag and integrate into the main thesis argument. So I plugged those numbers into Beeminder, gave myself a workable daily rate (50 or 100, I think) and then it calculated the rest and kept me honest.

So, to sum up:

• Beeminder forces you think backwards from your goal if you have a specific endpoint in mind. This is extremely valuable as it makes sure you're not being overambitious.
• Beeminder gives you accountability. It keeps you honest. This is what I initially found was most valuable, but later on I needed this less. YMMV.
• The community of Beeminder users is wonderful. The forum is a great place to get ideas, discuss approaches / failures etc.
• It works! Many people have had great results using Beeminder.

I'm not going to say I couldn't have written my PhD without Beeminder, but I'm almost saying it. Go check it out!