Behold, a live visualization of the active content on the various StackExchange communites.

I have dreamt about live data stream visualizations for a long time – but as a (mainly) Python programmer, the web scared me. That is why I usually rely on the old, lovable Matplotlib for my plots.

Such dynamic web content probably involves javascript, and a bit of knowledge about computer networks, which I’ve never dove into, and an understanding of how a browser works, right?

Well, have no fear my fellow Pythonistas – you can get your live data visualization all with just one file and no messing around with Heroku or Flask or whatnot.

The How

Wanna do it yourself? Here’s the only file you need. I’ll explain a high level view of it below. But first, try saving it and opening it with your favorite webbrowser. Voila, your own personal live data visualization. Hell, throw it up on a free GitHub Pages repo, and now it’s on the web!

The Moral

Okay but what’s going on here, and what’s this dark magic. I don’t see any Flask/Django, I didn’t have to struggle with Heroku. There’s gotta be a catch.

Yes, well, you had to sign a deal with Javascript, but there’s a larger lesson to be had here:

Maybe Python can solve all your problems, but it doesn’t mean it is the solution of minimal effort. Especially when it comes to the web.

A High-Level View

Here is the flow:

  • Our server (in my case, Github Pages) sends the client the static html page live_graph.html. That’s the only time the client ever needs to talk to our server.
  • When the client’s browser is rendering the HTML and running the javascript code, it sets up a websocket connection (think of it as a persistent connection between two places) between the client’s computer and StackExchange and tells StackExchange the kind of data we’d like to see.
  • Whenever StackExchange publishs a new active question, it sends it to the client via the websocket in the form of a JSON object.
  • In the client’s browser, whenever a new data JSON object arrives, it triggers some javascript code to parse the object for the data, appends it to its growing data collection and sends the updated collection to Plotly.
  • Plotly responds with a new plot, which the client’s browser then renders in the current html page.

Shoutouts

This solution came from a widely distributed knowledge-base. key players where Lucas Kushner, Alexandre Chabot-Leclerc, and Matt Makai’s talk.