internet

Using APIs to make things happen on the web

in

Towards the end of the Launch School core syllabus, we start to work with API calls. These allow us to pass information between servers. It turns out, this is really useful to be able to do.

Many if not most of the things you do online involve API calls. That little widget on the side of the web page that shows today's weather: an API call. Even things like Siri which aren't exactly web pages: using API calls.

Being able to make those calls and to interact with the services available through the internet gives users all sorts of power. The creativity comes, then, in how these things are all combined together.

That said, the dreams of the connected web have been somewhat oversold in the past. What are we on now, web 4.0?.

From a technical perspective, in this part of the course I enjoyed seeing how simple text in the form of JSON objects was behind so much of our communication and interactivity online these days. (All the more reason to have more secure ways of exchanging those plain text data).

I was also conscious of how much creativity and widened thinking has gone into expanding the possibilities of what HTML, CSS and a bit of JavaScript can do over the internet. Some of these capabilities mean that we're straining the possibilities in this area or that, but above all I take away some inspiration in how people made do with what they had instead of feeling like they needed to reinvent the wheel.

How events drive programming for the web

in

As part of my studies for Launch School, last month I was introduced to the idea of 'events' and how we can use the DOM and JavaScript to cause things to happen without having to force a full reload of the page. This is sometimes known as AJAX requests, which stands for Asynchronous JavaScript And XML.

In this mental model, we can think of the HTML and CSS code loaded by the browser (when you visit a website, for example) as the part which we can interact with. Absent any JavaScript code we include, this is somewhat static.

If we add in some JavaScript, we can first make sure that the DOM is fully loaded. For this we add an event listener for the DOMContentLoaded event.

Then after that, the sky is the limit. We can add event listeners to basically anything in our document (or just the whole document object itself). This is the list of events that we can watch for.

From reading outside the course, I know that companies' monitoring of user behaviour via JavaScript events triggering is a really big source of privacy violations or issues. I am not sure how the current model of how the web works serves to limit these violations. For instance, do people know that their mouse movements and clicks and patterns of mouse behaviour can all be monitored while they're on a site. So even if you don't click on an object, the website can still track that you're hovering over a particular element. That, to my mind, is a step too far, but it's the world we live in. You can disable JavaScript completely, but then you break a majority of sites that you're visiting.

Of course, if you're browsing in something like emacs' eww, or in brow.sh, then you bypass this problem somewhat.

Back to the JavaScript, though, the most interesting thing I found was the sheer amount of events that the browser was built to encounter. I'd love to see a chart showing how and when all these events were added to the browser's capabilities.

How the Internet Works

in ,

The internet. It just works. Understanding exactly how is a bit more complicated than many pieces of engineering. The more you examine the different aspects and parts that make it up, the more you see complexity concealed under the surface.

Visiting this website, for instance: it feels like a trivial thing to do, but there are many different parts making that happen, from the parts that actually transport the bits of data across the physical infrastructure, to the pieces that serve it all to you on a secure connection (ensuring that what I've written hasn't been altered by a third-party).

I've just finished Launch School's LS170 module which takes you a decent way down in the weeds to explain exactly how all of these pieces fit together to make up 'the internet'. So today I thought I'd retransmit some of that as a way of cementing it in my own mind.

At a very abstract level, the internet can be thought of as a network of networks. A network itself is a set of two or more computers which are able to communicate between each other. This could be the computers attached to a home network, or the computers that connect through a central server to a particular Internet Service Provider or ISP.

The internet makes use of a series of 'protocols', shared rules and understandings which have been developed or accreted over time. These protocols allow a computer on the other side of the planet to communicate in a mutually comprehensible manner. (If these shared sets of rules didn't exist, communicating with strangers or sending messages from one server to another would be a lot more difficult).

So once we have this top-down understanding of the internet as a bunch of networks that interact with each other, what, then, is the process by which a web browser in the United Kingdom communicates with a web server in China? Or in other words, if I want to access a website hosted on a Chinese webserver, how does that series of communication steps work to make that happen?

At this point, it's useful to make use of another abstraction: communication across the internet happens across a series of layers. There are several different models for these various layers. Two of the more common models — the OSI model and the TCP/IP model — are represented below:

layered-system-osi-tcp-ip-comparison.png

At the top level — "Application" — you have your website or whatever the user comes into contact with that is being served up to your web browser, let's say. All the layers below that are progressively more and more specialised, which is another way of saying that they become progressively less comprehensible if you were to eavesdrop on the data as it were passing over the wire or through the fibre optic cable.

Let's move through the big pieces of how information is communicated, then, starting at the bottom. (I'll mostly follow the TCP/IP model since it's a bit less granular and allows me to split things up in a way that make sense). This chart will help keep all the pieces in your mind:

layersofinternet.png

Note that each layer has something known as a 'protocol data unit' or PDU. A PDU is usually made up of a combination of a header, payload or chunk of data and an optional footer or trailer. The header and footer contain metadata which allows for the appropriate transmission / decoding etc of the data payload.

The PDU of one layer is used by the layer below or above it to make up its own separate PDU. See the following diagram as an illustration:

encapsulation.png

Physical Layer

Before we get into the realm of protocols, it's worth remembering and reminding ourselves that there is a physical layer on which all the subsequent layers rely. There are some constraints relating to the speed or latency with which data can be transmitted over a network which relate to fixed laws of physics. The most notable of those constraints is the fact of the speed of light.

Ethernet is the protocol that enables communication between devices on a single network. (These devices are also known as 'nodes'). The link layer is the interface between the physical network (i.e. the cables and routers) and the more logical layers above it.

The protocols for this layer are mostly concerned with identifying devices on the network, and moving the data among those devices. On this layer, devices are identified by something called a MAC (Media Access Control) address, which is a permanent address burned into every device at the time of manufacturing.

The PDU of the Ethernet layer is known as a 'frame'. Each frame contains a header (made up of a source address and a destination address), a payload of data, and a footer.

Internet Layer — The Internet Protocol (IPv4 or IPv6)

Moving up a layer, part of the Ethernet frame is what becomes the PDU for the internet or network layer, i.e. a packet.

This internet layer uses something known as the internet protocol which facilitates communication between hosts (i.e. different computers) on different networks. The two main versions of this protocol are known as IPv4 and IPv6. They handle routing of data via IP addressing, and the encapsulation of data into packets.

IPv4 was the de facto standard for addresses on the internet until relatively recently. There are around 4.3 billion possible addresses using this protocol, but we are close to having used up all those addresses now. IPv6 was created for this reason and it allows (through the use of 128-bit addresses) for a massive 340 undecillion (billion billion billion billion) different addresses.

Adoption of IPv6 is increasing, but still slow.

There is a complex system of how data makes its way from one end node on the network, through several other networks, and then on to the destination node. When the data is first transmitted, a full plan of how to reach that destination is not formulated before starting the journey. Rather, the journey is constructed ad hoc as it progresses.

Transport Layer — TCP/UDP

There are a number of different problems that the transport layer exists to solve. Primarily, we want to make sure our data is passed reliably and speedily from one node to another through the network.

TCP and UDP are two protocols which are good at different kinds of communication. If the reliability of data transmission is important to us and we need to make sure that every piece of information is transmitted, then TCP (Transmission Control Protocol) is a good choice. If we don't care about every single piece of information — in the case of streaming a video call, perhaps, or watching a film on Netflix — but rather about the speed and the ability to continuously keep that data stream going, then UDP (User Datagram Protocol) is a better choice.

There are differences between the protocols beyond simply their functionality. We can distinguish between so-called 'connection-oriented' and 'connectionless' protocols. For connection-oriented protocols, a dedicated connection is created for each process or strand of communication. The receiving node or computer listens with its undivided attention. With a connectionless protocol, a single port listens to all incoming communication and has do disambiguate between all the incoming conversations.

TCP is a connection-oriented protocol. It first sends a three-way handshake to establish the connection, then sends the data, and sends a four-way handshake to end the connection. The overhead of having to make these handshakes at the beginning and at the end, it's a fairly costly process in terms of performance, but in many parts of internet communication we really do need all the pieces of information. Just think about an email, for example: it wouldn't be acceptable to receive only 70% of the words, would it?

UDP is a connectionless protocol. It is in some ways a simpler protocol compared to TCP, and this simplicity gives it speed and flexibility; you don't need to make a handshake to start transmitting data. On the negative side, though, it doesn't guarantee message delivery, or provide any kind of congestion avoidance or flow control to stop your receiver from being overwhelmed by the data that's being transmitted.

Application Layer — HTTP

HTTP is the primary communication protocol used on the internet. At the application layer, HTTP provides communication of information to applications. This protocol focuses on the structure of the data rather than just how to deliver it. HTTP has its own syntax rules, where you enclose elements in tags using the < data-preserve-html-node="true" and > symbols.

Communication using HTTP takes the form of response and request pairs. A client will make a 'request' and it'll receive (barring some communication barrier) a 'response'. HTTP is known as a 'stateless' protocol in that each request and response is completely independent of the previous one. Web applications have many tricks up their sleeve to make it seem like the web is stateful, but actually the underlying infrastructure is stateless.

When you make an HTTP request, you must supply a path (e.g. the location of the thing or resource you want to request / access) and a request method. Two of the most common request methods are GET and POST, for requesting and amending things from/on the server respectively. You can also send optional request 'headers' which are bits of meta-data which allow for more complicated requests.

The server is obliged to send a HTTP status code in reply. This code tells you whether the request was completed as expected, or if there were any errors along the way. You'll likely have come across a so-called '404' page. This is referring to the 404 status code indicating that a resource or page wasn't found on the server. If the request was successful, then the response may have a payload or body of data (perhaps a chunk of HTML website text, or an image) alongside some other response headers.

Note that all this information is sent as unencrypted plain text. When you're browsing a vanilla http:// website, all the data sent back and forth is just plain text such that anyone (or any government) can read it. This wasn't such a big issue in the early days of the internet, perhaps, but quite soon it became more of a problem, especially when it came to buying things online, or communicating securely. This is where TLS comes in.

TLS or Transport Layer Security is sometimes also known as SSL. It provides a way to exchange messages securely over an unsecured channel. We can conceptually think of it as occupying the space between the TCP and HTTP protocols (at the session layer of the OSI framework above). TLS offers:

  • encryption (encoding a message so only authorised people can decode it)
  • authentication (verifying the identity of a message sender)
  • integrity (checking whether a message has been interfered with)

Not all three are necessarily needed or used at any one time. We're currently on version 1.3 of TLS.

Whew! That was a lot. There are some really good videos which make the topic slightly less dry. Each of these separate sections are extremely complex, but having a broad overview is useful to be able to disambiguate what's going on when you use the internet.

PhD Tools: Turn Off the Internet with Freedom

in , , ,

[This is part of a series on the tools I used to write my PhD. Check out the other parts here.]

Freedom does one thing and it does it well: turning off the internet (or parts of it). It removes temptation by giving you a time slot where the internet is turned off (and no way to turn it back on) on both your laptop and your phone. [Note: at the current moment there is no Android version of Freedom, but it's been a long time coming so I imagine that will be released in the near-term future -- a recent twitter query suggested "end of the summer"].

You can run it on an ad hoc basis -- i.e. you decide that you want 30 minutes of 'freedom' starting now, click, and then you've turned off the internet. OR you can pre-schedule those times (my preference) such that you can say Every Monday-Friday, I want to turn the internet off from 5am-12 noon every day. That time will thus be core time for writing, reading or using in some other kind of productive manner, free from distractions and interruptions.

You can tweak the settings so that you're not turning off the entire internet. You can make your own custom blacklist of sites that you know are kryptonite for you. (RescueTime is a great way of coming up with that list of which sites you're sinking too many hours into, especially when you have a few months of data). I don't particularly like this selective blocking because there's always going to be a new site of some kind or other that I haven't preemptively added to my blacklist. I don't need any access to the internet for my work, actually, so it's easiest to just turn it off completely.

In short, Freedom is great for aligning your goals (i.e. write words for my PhD every day) with a reality in which there are many shiny sites and videos and social media streams to follow. If you can find a way to turn that all off (or down to as minimal a level as possible) you'll get a lot more done and feel better at the same time.

PhD Tools: Save your web links with Pinboard

in ,

[This is part of a series on the tools I used to write my PhD. Check out the other parts here.]

Pinboard is the successor to Delicious and various other social link repositories. It's a service I get a lot of use out of because some parts of following the news, monitoring various government / non-governmental sites etc means reading lots of small articles each day. If, in the future, I want to return to a particular article, I generally don't want to have to go through the hassle of searching for it afresh (sidebar: use DuckDuckGo instead of Google! It's great!) so I just click a button to save a page in Pinboard when I think there's a chance I'll find this useful or I want to preserve it in some way. (Visit the 'tour' part of the site to learn more about how Pinboard works.)

Pinboard also auto-adds links, if I've starred, retweeted or saved anything in Twitter. Also if I've added a link from twitter or elsewhere into my Instapaper (Pocket is also used by some people, and is supported by Pinboard), then these articles are also autosaved into Pinboard. I reason that if I've taken the trouble to save it for reading later, then there was probably something in there that I might find useful in the future, or something that I might want to reference later.

The great thing about searching your repository of pinboard links is that you can do in-text searches. So you're not just searching the name and URL of the link, but you're searching the full text of the page. This is really useful, especially if you have many links saved. I just checked my account stats and see that I have over 60,000 bookmarks saved in my pinboard account. This is over 15 years worth of bookmarking.

Pinboard also offers a paid upgrade service where it will archive copies of a page and store that archived image. That way, even if the site is later taken down, or someone deletes the page, or anything at all happens to the page, then you still have a copy of the page and can search in it, can download it etc. Needless to say, this is really useful for monitoring the Taliban's websites, for example, which are frequently targeted in take-down attempts and where data is periodically deleted from servers or changed in various ways. I often double down and make a manual archive copy of important messages/pages to be stored in DevonThink, but I generally rely on Pinboard to handle the bulk of this work for me.

The creator of Pinboard, Maciej Cegłowski, is a smart guy and who writes interesting things, and the service reflects this. (Check out his interview on the Longform podcast). It's a paid service, but is relatively inexpensive as far as these services go. Moreover, the paid nature of the service means that there's relatively little (if at all?) creep factor to using it. Pinboard isn't selling your link database on to anyone else, they aren't marketing data profiles of their users etc. It's a solid service.

The interface is pretty minimal, which I like, but in case it's not to your taste you can browse your links in any one of a dozen or so apps which can hook up into your Pinboard data. Links are fully taggable, including tag nesting etc, so you're fully covered on that front. There's a social element to pinboard that I don't use much (mainly because I don't know many others who use pinboard for saving links) but I can see that that might be a useful feature if you have a community working on a particular topic or area.

Pinboard is easy to use, reliable and relatively inexpensive. It can save you time and help you find things you read on the web. Check it out here.