My new book: The Taliban Reader

My new book is out (finally). The Taliban Reader is somehow the culmination of years of work to drive studies of the Taliban back to primary sources. Some of this work was accidental; more recently it was more purposeful. The book I produced (together with Felix Kuehn) is long and detailed.

Comments and feedback prior to publication were extremely positive. It'll presumably take readers a while to start getting some real independent reviews in, but I look forward to feedback and whatever conversation is generated off the back of it all.

You can pick up a copy at any good bookshop or from Amazon here.

Machine Learning with Weka

Learning to program is an infinite process. The field is as open and wide as you can imagine, and you are mostly constrained by your imagination.

I spent much of May getting my mind around Go. I took Todd McLeod's Go course on Greater Commons and learned a great deal. The course was somewhat short on practical implementation, however, and I'm eager to do things with what I learned. More on that in due course.

A parallel strand of my studies has been in statistics and more advanced applications of statistical methods i.e. machine learning. I had done a bit of this in the past, but my poor foundation in basic statistics didn't serve me well. I am now rectifying that through Andy Field's excellent textbook.

For machine learning I decided to take a step back from the programming and use a graphical interface to start with. There are great APIs / tools available for this in most languages you can think of but I wanted more of a solid foundation in workflows around machine learning and the kinds of analysis that get done.

I read through a good deal of Jason Brownlee's blog(https://machinelearningmastery.com/) as well as his book on Weka and he made a good case for why Weka is a good place to start.

I have noted a number of steps to move through in sequence, at the same time recognising that data analysis is often unsequential. I expect this to expand and/or redefine this over time.

Kaggle is one of the major hubs for machine learning practice (and learning) and I wanted to reengage there. The first data set they usually have you work on comes from the Titanic disaster. You take the full roster of people who boarded, including data points like their economic class, where they were staying on board the ship and their age/gender etc and use anything and everything in terms of tools to predict who survived and who didn't. I had used this data set in the past when I was studying ML with Python.

My initial idea, therefore, was to take the .csv files from the Kagge competition and use them in Weka to come up with some predictions. Unfortunately, there are some idiosyncracies about the .csv file that make this difficult. Some of the attributes / columns in the data (like names) use punctuation marks which make parsing the csv data non-trivial. Weka uses ARFF files as standard but has the option to parse CSV data. It ran into quite a few errors when trying to crunch through the Titanic data, and no amount of basic fiddling would fix it.

Reading around a little, it seems that others have noted this problem in the past. One blog post tackled the problem head on but the solution didn't really help me much in the short term. I'm now somewhat stuck, knowing that the fix to the problem is to use another language (Python, perhaps) to range over the data and process it in a form that will be more palatable for Weka. Alternatively, I could use it as an opportunity to build a short Go programme that could perform the same function.

For the moment, i've decided to do neither. I'm going to find an alternative data set which doesn't require wrangling and fiddling. I know wrangling and fiddling is an important skill to master, but it's not the skill I'm trying to focus on right now. Luckily, between the UCI Machine Learning repository and various other places, I'm not exactly lacking for examples / other data sets. Today I'll work with the Pima Indians Diabetes data set which came built-in with Weka.

My First Arabic Book Translation

I’ve been sitting on this for a while. 

Early in 2017, I was lucky to get put in touch with Sami Al-Hajj and al-Jazeera by a good friend. They were looking for someone to translate Sami’s memoirs from Arabic into English. I had done similar work in the past, working Zaeef’s memoirs from his Guantánamo time that later expanded into My Life With the Taliban. I had never done any serious translation from Arabic, however, particularly of this length.

I finished translating a while back. It was both harder and more enjoyable than I had expected. Harder in that it requires an intense focus that can’t really let up while the translation is happening. More enjoyable since I realised the process was something that brought in a lot of creativity to get the language ‘just right’.

Today, Sami’s memoirs have been published. You can download the PDF version via Al-Jazeera here. It looks like Kindle and iBooks versions will be made available in due course as well.

I hope you get a chance to give this book a read. Not only is it some work that I spent a good chunk of time working on, but it’s a really useful and moving account of someone who passed through Guantánamo. In particular, Sami was often on a hunger strike so there is a lot of detail on his mental state and coping mechanisms, as well as the way the medical authorities at Guantánamo attempted to 'deal' with the problem.

Fuzzy Searching and Foreign Name Recognition

Here's something that happens fairly often: I'll be reading something in a book and someone's name is mentioned. I'll think to myself that it'd be useful at this point to get a bit of extra information before I continue reading. I hop over to DevonThink to do a full-text search over all my databases. I let the search compute for a short while, but nothing comes up. I tweak the name slightly to see if a slightly different spelling brings more results. That works a bit better, but I have to tweak the spelling several times until I can really claim the search has been exhaustively performed.

Anyone who's done work in and on a place where a lot of material is generated without fixed spellings for transliteration. In Afghanistan, this ranges from people's names -- Muhammad, Mohammad, Muhammed, Mohammed etc -- to place and province names -- Kunduz, Konduz, Kondoz, Qonduz, Qhunduz etc.

DevonThink actually has a 'fuzzy search' option that you can toggle but it isn't clear to me how it works or whether it's reliable as a replacement for a more systematic approach.

As I'm currently doing more and more work using Python, I was considering what my options would be for making my own fuzzy search emulator.

My first thought was to be prescriptive about the various rules and transformations that happen when people make different spelling choices. The Kunduz example from above reveals that vowels are a key point of contention: the 'u' can also be spelt 'o'. The 'K' at the beginning could also, in certain circumstances, become 'Q' or 'Qh'. These various rules could then be coded in a system that would collect all the possible spelling variations of a particular string and then search the database for all the different variations.

Following a bit of duckduckgo-ing around, I've since learnt that there are quite extensive discussions of this problem as well as approaches to solution that have been proposed. One, commonly referenced, is a Python package called 'FuzzyWuzzy'; it uses a mathematical metric called the Levenshtein distance to measure how similar or not two strings are. I imagine that there are many other possible metrics that one could use to detect how much two strings resemble one another.

I imagine the most accurate solution is a mixture of both approaches. You want something that is agnostic about content in the case of situations where you don't have domain knowledge. (I happen to have read a lot of the materials relating to Afghanistan, so I know that these variations of names exist and that there is a single entity that unites the various spellings of Kunduz, for example). But you probably want to code in some common rules for things which come up often. (See this article, for example, on the confusion over spellings of Muslim names and how this leads to law enforcement mistakes).

I may end up coding up a version that has high accuracy on Afghan names because it's a scenario in which I often find myself, but I'll have to explore the other more mathematically-driven options to see if I can find a happy medium.

Tweeting to the Void

I've previously written about how I turned off Facebook's news feed. I keep an account with Facebook because people occasionally contact me there. It is also an unfortunate truth that many companies in Jordan (where I live) or in the wider Middle East only have representation on Facebook instead of their own website. (Why they insist on doing this baffles me and is perhaps a topic for a future post).

I have long preferred Twitter as a medium for filtering through or touching -- however obliquely -- things going on at any particular moment. I have no pretensions to actively follow every single tweet to pass through my feed. Rather, it's something I dip into every now and then.

Increasingly in recent months, I found myself growing dissatisfied with the pull it often has on me. It has become something of a truism to state that 'twitter isn't what it once was', but there's less and less long-term benefit in following discussions as and when they happen.

RescueTime tells me that I spent 86 hours and 16 minutes on Twitter in 2017 -- just under quarter of an hour each day. That feels like a lot to me.

Enter 'Tweet to the Void'. This is a Chrome extension. (For Firefox and other browsers, I have to imagine things like this exist.) When I visit twitter.com, the feed is not visible. All I see is somewhere to post a tweet if that's what I want to do. (There is still some value in posting blogposts and articles there, since I know some people don't use RSS). Of course, I can always turn off the extension with ease, but adding this extra step has effectively neutralised Twitter for me. 

Try it; see how you feel about having something standing in the way of your social media fix. Let me know how you get on.

Installing PostgreSQL on a Mac

PostgreSQL is a SQL-type database system. It has been around for a while, and is in the middle of a sort of revival. Installing Postgres on your own system can be a little difficult. Last time I tried, I was helped through the process while doing the Udacity Intro to Programming Nanodegree.

Recently I had to reinstall Postgres, and there were some useful improvements to the process when guided through it in my Dataquest lessons.

Postgres.app is an application you can install on your Mac which simplifies a lot of the legwork, particularly when setting up new databases, servers and so on.

When you want to install a commonly used Python library for interfacing with Postgres, psycopg2 is a good option. You can do this easily with Anaconda:

conda install psycopg2

Making and shuffling lists in Python

I discovered some useful functions the other day while trying to solve one of the Dataquest guided projects. These all relate somehow to lists and use Numpy. I'm listing them here mainly as a note for my future-self.

import numpy as np

# this code returns a list of n number of items starting at 0
np.arange(3)
---- returns [0,1,2]

# this code is a variation on the previous one
np.arange(3,7)
---- returns [3,4,5,6]

# this adds the functionality of steps in between values
np.arange(2,9,2)
---- returns [2,4,6,8]

# these are slightly different; they sort lists
# if you want to make list of numbers randomly sorted:

np.random.permutation(10)
---- returns the numbers 1-9 in a list, randomly sorted

# you can also pass non-numeric lists into the `permutation`
list = [a,b,c]
np.random.permutation(list)
---- returns something like [b,a,c]

Tabula for extracting table data from PDFs

Have you ever come across a PDF filled with useful data, but wanted to play around with that data yourself? In the past if I had that problem, I'd type the table out manually. This has some disadvantages:

• it is extremely boring
• it's likely that mistakes will get made, especially if the table is long and extends over several pages
• it takes a long time

I recently discovered a tool that solves this problem: Tabula. It works on Windows and Mac and is very easy and intuitive to use. Simply take your page of data:

View fullsize

A page listing Kandahar's provincial council election polling stations from a few years back. Note the use of English and Dari scripts. Tabula handles all this without problems.

Then import the file into Tabula's web interface. It's surprisingly good at autodetecting where tables and table borders are, but you can do it manually if need be:

Then check that the data has been correctly scraped, select formats for export (from CSV to JSON etc):

And there you have it, all your data in a CSV file ready for use in R or Python or just a simple Excel spreadsheet:

Note that even though the interface runs through a browser, none of your data touches external servers. All the processing and stripping of data from PDFs is done on your computer, and isn't sent for processing to cloud servers. This is a really nice feature and I'm glad they wrote the software this way.

I haven't had any problems using Tabula so far. It's a great time saver. Highly recommended.

Language Learner's Journal: Homestay Edition

[This is a continuation of Taylor's blog series where she details some of the week-in-week-out lessons that she learns through her Arabic studies and coaching work together with me. For other posts in the series, click here.] 

My first time around with immersive language learning (studying for a semester in Brazil ten years ago), it was a no-brainer that I would do a homestay. I was a college student, I had no local networks to tap into to find housing, and forgoing a degree of privacy and independence felt not so different from the dorm life I was living the rest of the year. Furthermore, an employee of my school coordinated an apartment for me from the start, and back then I wasn’t as intentional, organized, and proactive about my learning as I am now.

By the time I got to Amman to bump up my Arabic studies last December, I only briefly considered doing a homestay. I wanted my own place or to at least live with other young people like me — I’d have guests visiting, I like to do yoga in my home, spend long times in the kitchen making vegetarian food and listening to podcasts, stay up late laughing with my mom on skype, etc. I also work part-time remotely each day, and I needed to make sure I’d have a quiet place appropriate for that.

About six months into studying, however, I started to think about a homestay seriously for the first time. I had a persistent feeling that my ability to read native texts and understand a newscast was beyond my ability to make small talk or say “Excuse me, I need to go brush my teeth.” 

I had a home in Amman and wasn’t going to force a homestay to happen, but I also began laying out feelers for families of all women interested in hosting a 30-year-old journalist. I had a happening Amman social life by half a year into my stay in the city, with morning yoga, friends who’d pick me up for running groups, dinner parties out, and meetings at coffee shops with my language partner. If I was going to stay with a family, it would ideally be one that understood my educational goals in Amman and how those various parts of my daily life fit into it.

I lucked out. A friend who coaches a track team broached the idea with a student who lives with her three sisters, parents and grandmother. The sisters range from kindergarten to college-aged and are a lively pack that zips around the city for their own activities each day, be it shopping, going to a sports team practice, or all the daughters going to the balad for a sisters meal out. Their home also had an open-door policy for all the neighbourhood youth, and throughout the day I’d have a smiley primary school girl telling me about the barbecue she ate the day before with her family or an Iraqi pharmaceutical student telling me about the farsi words he uses in dialect back home.  

Alex has emphasized to me the importance of speaking to a variety of people, since it can be deceptive to feel so confident speaking with language teachers and then, say, find groups of old people really challenging. The times when I was speaking with the grandmother of the family and the kindergarten daughter were indeed my hardest moments, but also fun, no-pressure ones Tayta usually wanted to know my whereabouts throughout the day and my travel plans. The little girl generally liked talking about her dog or the puppy show we saw in the circus together.

My favourite time of day was bedtime, when myself and the oldest and youngest daughter all went to bed together in the same room, chattering until we drifted off to sleep. I realized I had never needed to learn to say “Shall I turn off the light?” to anyone in all of my classes (بطفي الضوء؟)

I wish I had worked more with a notebook to write down new words and phrases I was hearing, but that house was full of life, and I floated between rooms and conversations and meals and situations throughout the day. Their conversations amongst themselves were often too fast for me, and sometimes I knew only the main outlines of what they were discussing and missed what specific point was funny or surprising. There was a huge difference between having a one-on-one conversation versus participating in the group chat – the former was a speaking exercise, whereas the latter was largely a listening one. I was once very lost in a conversation about food that involved peals of laughter until I recognized one word I knew (كبد) and realized they were talking about eating the internal organs of animals. 

Even if I hadn’t learned a single new word (thankfully, I did), it was a valuable booster shot to my confidence to be able to know that yes, I can live amongst a family and handle myself in day-to-day situations. I brought up basic things that I wanted to resolve in Amman with them, like explaining I wanted to find a quality and affordable dentist (they took me to a great one) and that I had Brazilian friends looking for an affordable way to send money abroad (hence many discussions about bank accounts and foreign currencies and a final group trip to MoneyGram). 

In the end, I stayed about two very happy weeks with them. I could always have done more toward a laser-focus goal of learning Arabic – I could have stayed with them from day one, I could have refused to speak with anyone in English all of my time in Amman, I could have lived in more remote parts of the country where I would be a rare foreigner. But one takeaway I have from my Brazil years is that, for me as a journalist, an immersive experience in a new place is a big-tent one: the gringos are part of it, the English-speaking locals with a foot abroad are a part of it, my bilingual running group and morning yoga are parts of it, the other language students working toward the same goals that I am are a part of it. The college-aged daughter who took me under the wing for my weeks in her home was about to study abroad in Germany, which I think is why she got my loosely defined “spend time with a family” goal so well. My next hope is that I’ll be able to open my doors to the sisters and offer them a home abroad someday.

Language Learner's Journal: Phone Calls and Timed Breaks

[This is a continuation of Taylor's blog series where she details some of the week-in-week-out lessons that she learns through her Arabic studies and coaching work together with me. For other posts in the series, click here.] 

I haven't written in this space in a bit because I've been taking breaks, both extended ones and daily small ones. At first I was anxious about "wasting" time by taking a sort of staycation holiday, but I'm seeing now it did me well.

Two friends came to visit me around Ramadan, and I threw a lot of my daily habits to the wind to enjoy having them here. One stayed an entire month – having a friend from Rio de Janeiro to pound pavement around this city with me during the unique time which is Ramadan was a fine way to enjoy what is otherwise a rather quiet and still time. I think together we made as many Jordanian friends as I had otherwise made in the six months before she came. We went to multiple yoga classes, including the Indian embassy's "International Day of Yoga," took a makeup shopping spree with two young women we met at Amman's Roman amphitheatre and waded through Wadi Mujib with a guide who seemed happy to have any clients during a slow month. There's something to be said for the "reset" button you can push when you have the novelty and joy of seeing a place through a visitor's eyes. 

Even as I wasn't studying formally and took a month off from classes, I still had useful, spontaneous ways to put my language to use. I got a speeding ticket on the way back from the Baptism site, which taught me some useful new vocabulary words (مخالفة غرامة، رخص). I even tried to argue with the officer that we couldn't see the sign because it was covered by a tree, but I was not successful (that said, the ticket was only 20JOD). During a week-long trip to Greece, a friend who took me to Athens' central fish market told me one of the men working at a booth was Syrian, so I went and introduced myself. It is a motivation super-charger to see how someone lights up to hear their own language when everything else around them sounds like Greek. 

When my trips were finished, I sat down with Alex's "Mastering Arabic" to review and refresh my study methods. I'm doing decently with motivation, a topic Alex covers at length. Learning Arabic has always been a project stemming from intrinsic motivation, something I chose to do for and by myself. I have no hoops to jump through to please an employer or scholarship committee – for now, I define my own success, which usually is satisfaction at being able to make a phone call or recalling a precise and useful word I've studied and putting it to use in the real world. 

"Mastering Arabic" did prompt me to think more about time organization methods, which I had let slide into just some vague sort of maximalism ("I have the evening free and will study as much as possible.") Two methods that have been useful are to divide tasks into discrete parts and then "reward" myself by taking a break afterward. That may mean: Read one story in حكايتهن from the UNRWA and then take a break by chatting on Whatsapp, or make Anki cards for 45 minutes then go do pushups while listening to Despacito. 

I've also made more active use of a language notebook. In the past, I took notebooks to class, but wasn't proactive enough to take it to stores or pull it out as soon as a friend and I part ways. One of my favorite exercises now is to try to recall new words or types of sentences I heard the soonest possible after I finish an activity – that recently included زبيب and شمندر after trips buying groceries, and اخبرني عنك and شو خطة الليل؟ when I listened in on phone conversations. I heard a friend say ...كنت رح اشتغل في محل بس كان عندي مشكلة and jotted it down. I've been feeling that what I lack is many basic speaking forms and daily vocabulary, and as simple as many of these things are, I didn't know them beforehand – I wouldn't have know how to say "I was going to do X" and that it was such a simple construction. I've long been a podcast fan, and now, when circumstances permit, I listen with a notebook beside me to jot down anything new. (My favorite new podcast discovery is عيب، which is in Jordanian colloquial Arabic and on subjects very accessible to me.) I've returned to private classes twice a week with a fabulous new teacher, so I bring her the questions that have accumulated in the days between our meetings. 

Alex is a big proponent of making phone calls, which for me is one of the final frontiers for language learning – it's hard to have no visual cues and introduce yourself coherently when you are a disembodied foreign voice over the phone and ask for things like interviews or deliveries. Thankfully, each successful call boosts my confidence a bit more, though I can't say I'm great – I stutter and stammer a lot, and I speak much more simply than I know how to speak if you gave me a quiet space and time to compose my thoughts. Still, to my delight, I was able to speak several times with a business whose employees I sought to interview, to my landlady to complain that she's been locking our veranda door unnecessarily, and to order food deliveries and try to explain where I live ("between the liquor store and the dukan of fulan"). My comprehension rate is variable – I understand nothing my building's haris tells me on the phone other than a general sense that he is granting or denying my request, but, thankfully, he seems to understand me.

A final reflection on taking breaks: When I was hosting visitors, I had moments of stress about how behind I was getting. I could do some calculation for you about how many words I "lost" by kicking back and speaking more English, Portuguese, and Spanish on a daily basis than Arabic. 

That said, the break served me well. One, it was nice to reflect on what language skills are – not a one-time performance but a world of abilities we keep in our pockets and pull out often at unexpected times. I haven't been in a Spanish class in about ten years, but I still am making friends and enjoying pop music in the language. Many of Alex's techniques are about how to maintain language abilities, i.e., not how to just do well on a test but recall vocabulary even several years from now. 

Two, athletic comparisons have always made sense to me when it comes to tackling this difficult project which is learning Arabic. My Rio de Janeiro guest added a new layer to that analogy when she began to show me what she was learning in her advanced yoga and acrobatics classes back home. My mode of exercise had long been about having time to daydream while I threw my body into motion, muscles and mind unattached while I ran ten, fifteen, twenty miles with little effort and thought. A pushup, on the other hand, or playing a competitive strategy-based sport was always difficult to me. The difference, my guest said, is attention to form and control and knowing that it won't "be natural" and effortless in the beginning. We practiced yoga chaturangas and headstands, two exercises that require total concentration and engagement rather than "zoning out." We also talked about rest. Nobody will run well if they run every day; yoga always ends with a savasana "corpse pose." The rest makes us more energetic and more engaged the next time we work out, pushing forward the frontiers of what one day will become natural for us. 

P.S. – This blog writing was also a timed activity. I went a little over my allotted hour-time block, but still wrote swiftly. Now, I'm off to do some pushups.

Spaced Repetition Without Computers

I wrote a little something over for the Spaced Repetition Foundation about getting the benefits of spaced repetition without relying on technology. Check it out here. Also give Matt's recent post on spaced repetition and creativity a read.

The book you need to read on bin Laden's life post-2001

There is no shortage of books on the September 11 attacks and their aftermath. Some stand back to analyse it in terms of trends and networks, seeking to explain the 'why' through abstractions. Others have written participant accounts or their histories from the sidelines. The Exile offers a fulsome corrective to this trend towards abstraction. Curious what life was like for bin Laden, his commanders and their families? Cathy Scott-Clark and Adrian Levy deliver in spades.

The beating heart of this book are the stories of bin Laden's wives, their children and their life in 'exile'. The authors seem to have managed to achieve as yet unparalleled access to the wives and some other family members of Osama bin Laden, and their tale is both gripping and believable. The second important contribution that the book makes is to reveal Iran's role in hosting the bin Laden families (and commanders) post-2001. The rich detail goes a long way to giving the reader a sense of the day-to-day frustrations of their lives in Tehran (and other places) as well as their interactions with Iranian military and government officials. The book would be worth its price just for these sections alone.

Chapter Eleven tells the story of the night bin Laden died, to a large extent from the perspective of his wives and family members. They also weave in accounts from US soldiers participating in the raid, but this is a perspective we have been denied till now and I think it is an important one. Indeed, the trauma faced by the children on that night (and throughout the years prior, for the most part unable to leave their home) is one of the understated but crucial themes that stand out from 'The Exile'.

Every account of bin Laden's time post-2001 has to grapple with the question of Pakistan's role. The authors take a smart position throughout the book, which is to abstain from abstraction and a strong analytic voice. There are some claims of Pakistani ISI involvement and meetings here and there, but they step back a little before charging the government or senior officials with a state-level conspiracy. Whatever happened, this account holds, was much more an affair of bit players.

The book had the feeling of being rushed to press. It must have cost a lot to research the book, so perhaps the authors simply ran out of funds, but it seemed like there were so many other lines of enquiry that could have been started. The hardcover copy I read still had a fairly large number of typos, and it's a shame that the authors use the word "Afghanis" to refer to Afghans.

The book is extremely readable -- it kept me up until two in the morning as I finished it -- and this is in large part because of the use of dialogue and building up narrative tension through conversations. Unfortunately, the handling of some of these conversations -- reported through interviews with participants -- strains credulity. Study of memory and oral history has shown how these kinds of memories degrade or get reshaped with each telling, and I wish there were more caveats throughout the book that what we're reading is an approximation of what happened in order to better imaginatively enter the situation.

All in all, though, The Exile is an important book, an engrossing read and hopefully the beginning of more enquiries as others follow up on leads and side-stories raised in the telling. I took copious notes and will be digesting its various contents for a while, I think. It seems that scholars of September 11 and its aftermath are doomed to eternally reading and retelling the same events in slightly different contortions as new facts and witnesses emerge. If all the books were as good as this one, I wouldn't mind so so much.

Italian Comprehension: Putting Myself On the Hook

I'm currently scheduled to do some archival work in August that would greatly benefit from being able to understand written Italian. That's a little under two months away from now, so I've been thinking about how best to approach this problem.

I'm only interested in comprehension, and not in production, of language, so this would seem a great opportunity to get a little bit of grammar and then immerse myself in some comprehensible input. I bought a quick-and-basic grammar overview and have set myself up with Clozemaster. (Clozemaster is a great way to learn vocabulary through context. I wrote about it previously here.)

I'm posting about this less-than-two-month mini goal here for a little bit of public accountability. I will in no way be fluent by the end of it, but I hope to be able to make my way through an archive of materials in Italian, or at the very least to know what is worth properly translating and what I can skip over.

There's rumoured to be a Beeminder/Clozemaster direct service connection in the works, but until then I'll use Clozemaster's own daily reminder tool. A più tardi.