[Thesis 1] Google ABC – Technical Module Prototype

B is for Beyonce, according to Google images
B is for Beyonce, according to Google images

This prototype is still under development, but a functional version can be accessed here.

I’m trying to build a sort of Google ABC — an analogy between alphabet books and Google’s autocomplete engine. The idea is make people reflect about how this system works.
Besides, Google has been our default choice when searching for knowledge for the past years. Choosing an alphabet book as an analogy is a way of commenting on this.

This work is not finished yet, but here’s the process so far:

  1. The old Google Web Search API and Google Images Search API have been deprecated as of 2010 and 2011, respectively. Though still functioning, they will stop operating soon and users are encouraged to replace them by the new Google Custom Search API. I started by implementing the search box using the new engine.
    However, this API is actually intended to perform custom searches inside websites. The purpose of this project, though, was to search from the entire web using the Google Search engine. This is possible through a laborious process: by setting up a custom search with any given website, then editting your search to include results form the entire web, and removing the website you setup at first.
    The main problem though was that the autocomplete did not seem to work for the “entire web” option. To sum up, it does not mimic the regular Google Search as I expected.
  2. I found a Google Autocomplete “API.” It allows access to Google’s autocomplete results. However, it seems to be an engine developed for internal use, not really an API. There is no official documentation from Google.
    After a few more searches, I found a jQuery plugin that allows for searches using various Google services, all through the same non-documented API.
  3. Integrated the autocomplete plugin with the Google Custom Search bar.

    Google Custom Search default layout
    Google Custom Search default layout
  4. The new custom search is based on simple html tags and does not require javaScript knowledge. However, that makes it also harder to customize. Results can only be displayed according to a predefined set of layouts.
    The solution was to scrape the data from the results, erase Google’s injected html elements, and store the pictures sources only.

    Google Custom Search Layouts
  5. I started working on a layout to display the results. My idea was to form the initials using images. To do so, I’d have to grab the letter typed by the user, draw it on a canvas and get the pixel results. I made a short Processing sketch to test the method first.

    Processing sketch to draw pixelated fonts.
  6. Because this small experiment seem to work fine, I decided to implement the page display using P5.js, a javaScript library that utilizes Processing syntax. The process turned out to be laborious, though. There seem to be some flaws in the current implementation of the pixel manipulation methods in P5. Due to browser restrictions — see cross-origin resource sharing and developer Lauren McCarthy’s comments on the P5 Github repository — the images had to be displayed as regular html elements.

Code on Github.

[Thesis 1] Design Brief #2

Thesis Question

In spite of current improvements in technology, data visualization continues to use techniques that date to its origins. Digital interfaces might have changed the way we interact with data, making it possible to filter, zoom-in, and access details of it. However, the basic principles to display data are still based on reductions to geometric shapes, like rectangles, circles, and lines.

When working with cultural data, this gap between representation and artifact might be particularly problematic. Does a bar chart of the most used words of a book reveal more than the book itself? Can a pie chart of the colors of a painting tell us more about a painting than the actual image? Or is it possible to balance both representations? If so, can modern technologies create new visualization techniques to bridge this gap between representation and artifact?

Research

In his paper “Visualizing Vertov,” [@manovich_visualizing_2013] Lev Manovich analyses the work of Russian filmmaker Dziga Vertov using methods from his Software Studies Initiative. The research intends to show alternative ways to visualize media collections and focuses on the movies The Eleventh Year (1928) and Man With a Moving Camera (1929). It utilizes digital copies of Vertov’s movies provided by the Austrian Film Museum, in Vienna. They are the media source from which movie frames are taken, visualized and analysed, using custom software. Also, they provide metadata for the research: frame numbers, shot properties and manual annotations. A secondary source utilised by Manovich is Cinemetrics, a crowdsourced database of shot lengths from more than 10000 movies.

As for the visual representations, traditional techniques, like bars and scatterplots, are complemented by Manovich’s “direct visualization“ approach. In this method, “the data is reorganized into a new visual representation that preserves its original form.” [@manovich_what_2010]

Manovich describes his own method for analysis as analogous to how Google Earth operates: from a “bird’s eye” view of a large collection of movies to a zoom into the details of a single frame. The summary of those steps is as follows:

vertov_01

  • Panorama
    • ‘Birds-eye’ view: 20th Century movies compared by ASL (average shot length). Technique: scatterplot.
    • Timeline of mean shot length of all Russian films in the Cinemetrics database. Technique: line chart.
    • Movies from Vertov and Eisenstein compared to other 20th century movies. Technique: scatterplot.

vertov_02

  • Shot length
    • The Eleventh Year and Man With a Moving Camera compared by shot length. Technique: bars, bubbles, and rectangles.
    • Zoom-in of the same visualisations.

vertov_03

  • Shot
    • Each of 654 shots in The Eleventh Year, represented by its second frame. Technique: direct visualization.
    • Shots rearranged based on content (close-ups of faces). Technique: direct visualization.
    • Shots and their length. Technique: direct visualization and bar chart.

vertov_04

  • Frame
    • First and last frames from each shot compared. Technique: direct visualization.
    • Average amount of visual change in each shot. Technique: direct visualization (2nd frame) and bar chart.
    • Average amount of visual change in each shot. Technique: direct visualization (juxtaposition) and bar chart.
    • Frames organised by visual property. Technique: direct visualization.

Also, Manovich uses contextual notes to draw conclusions from the visualisations. His findings are often compared to facts from the history of cinema, Vertov’s early manifestos, and previous studies, confirming or contradicting them.

Project Concept

This prototype will compare two movies utilising some of the methods from “Visualizing Vertov.” It will combine traditional visualization techniques — charts using 2d primitives — and the “direct visualization” approach.

As for the data, it will use a specific component from films: sound. Because Manovich’s research relies largely on visual artefacts, using sound in this prototype might reveal limitations of his method or point out to new ways of applying it.

Besides, the Cinemetrics project focuses exclusively on a single aspect of movies: ”In verse studies, scholars count syllables, feet and stresses; in film studies, we time shots.” [@cinemetrics] This approach seem to underestimate other quantifiable data that could be used in movie studies.

In spite of using sound instead of time or visuals, this prototype will keep the analytical aspect of Cinemetrics and “Visualizing Vertov.” Then, it will draw conclusions on this approach compared to the supercut method.

To sum up, these are the questions raised by this prototype:

  • Is it possible to apply the “direct visualization“ technique to a non-visual artifact?
  • Which patterns and insights can a sound analysis of a movie reveal as compared to a visual analysis?
  • What are the results of an analytical method as compared to the supercut technique?

Research Methodology

Media

All levels of analysis in “Visualizing Vertov” — movies, shot lengths, shots, and frames — utilise comparisons.This device is largely employed in traditional data visualization, and seems to be even more useful for cultural artefacts. In Vertov’s case, for instance, the shot length measurement would not provide any insight if it was not compared to the Russian avant-garde or the 20th Century movies in general.

Following the same logics, this prototype takes an European movie, Wings of Desire (Der Himmel über Berlin, 1987), by German filmmaker Win Wenders, and its American remake, City of Angels (1998), directed by Brad Silberling.

Data

The following diagram shows the development steps for this prototype:

methodology_diagram

The digital copies of the movies utilised in the first step were not high quality ones. Also, the process by which the data was gathered do not preserve a high resolution sample. Those are limitations of this prototype, which focused on a rapid technique to extract and compare sound data. They will affect the visualization decisions as well.

Visualization

The data exported as XML was read and visualised using D3, a javaScript library for data visualization. D3 provides a fast and reliable way to parse and represent large amounts of data in web documents. Web pages are also able to natively embed media such as sound. Those are the reasons why the final result of this prototype is a web page.

First Iteration

The first iteration of this prototype is a simple area chart depicting the sound variations from the movies. Because of the low quality of the sources, it utilizes a relative scale for each movie: the higher value of each film is represented as the higher point of each scale, and all other values are relative to that.

For this reason, the peaks from each movie might differ in absolute value of decibels. In conclusion, this parameter should not be used for comparison.

prototype_01

Some visual disparities seem to appear in this first iteration: longer areas with high volume in Wings of Desire versus constant medium-volume ones in City of Angels.

However, the sound volume does not seem to provide many insights. Are these variations due to background music? Which patterns represent dialogues? The representation is so highly encoded that leaves no way to answer these questions.

Second Iteration

In order to add some more clarity, the second prototype includes visual marks to represent the parts of the movies when dialogues occur. A computational method to differentiate music from speech would be laborious and not reliable. The solution was to use part of the first prototype developed for this project, which parses subtitles files into different formats. The software generated JSON files with timestamps of the subtitles. Like the sound data, they were read and visualised using D3.

prototype_02

This new iteration seems to shed some light on how the movies compare. While City of Angels shows a constant use of dialogues, the subtitle marks in Wings of Desire have long blanks in several parts of the movie.

prototype_02a

The presence of the red marks also help us understand the sound representation. By comparing the two, it is possible to see that the alternating medium-low volume represents dialogues, while the constant and higher areas might indicate background music.

prototype_02b

Even though this iteration offers more insights about the movies, most of them are not yet verifiable. The tool does not let us access the sound itself.

Third Iteration

The last iteration of this prototype embeds the sound extracted from the movies into the page. It is controlled by a slider that matches the horizontal dimension of the charts.

This final representation is analogous to the shot length one in “Visualizing Vertov.” It combines a 2D chart with the artefact itself, trying to bridge the gap between the two.

prototype_03

Findings and Next Steps

Though the third iteration of the prototype includes the sound, it does not achieve the same results as the display of frames in “Visualizing Vertov.” The access to the sound is mediated through a GUI element, which creates an extra step. On one hand, the overview of the “media collection” (sound fragments) is only accessible through the chart. And on the other hand, the access to the sound through the player does not provide an overview, since it is not possible to listen to all the sound fragments at the same time. As opposed to what happens in Manovich’s visualizations, those two representations were not combined.

Nevertheless, the sound analysis does seem to provide some useful insights for movie studies. Even though this rough prototype relies on low-res sources and the support of subtitle files, yet it reveals some interesting patterns. An analysis of other sound components might include some more interesting findings.

At last, this prototype shows a radically different approach compared to the supercut technique. The analytical method of translating time to space and reducing sound to visuals have a very less entertaining result. In conclusion, the next steps for this project are:

  • Find out the communication purposes of it — data visualization as an analytical tool? A media? A language?
  • Continue the explorations of media collections using a “direct visualization” approach.
  • Expand this approach, as much as possible, to media and representations not yet explored by the Software Studies Initiative.

Bibliography

“Cinemetrics – About.” 2014. Accessed October 10. http://www.cinemetrics.lv/ index.php.

Manovich, Lev. 2010. “What Is Visualization?” Paj:The Journal of the Initiative for Digital Humanities, Media, and Culture 2 (1). https://journals.tdl.org/paj/ index.php/paj/article/view/19.

———. 2013. “Visualizing Vertov.” Russian Journal of Communication 5 (1): 44–55. doi:10.1080/19409419.2013.775546.

[sims 2014] Ribbon drawing prototype (midterm)

Screen Shot 2014-10-06 at 11.18.33 PM

Screen Shot 2014-10-06 at 11.20.42 PM

I’m trying to make this drawing tool that generates ribbons. The final goal is to allow people to draw some interesting things — I can only think of calligraphic shapes, but anyway. And, hopefully, they can apply some physics effects and export everything as a video.
This tool has ONE user in mind: Laura. She needs it to make some letterings.

Everything on the class github.

[Thesis 1] Methodological Module Prototype

city_wings

City of Angels and Wings of Desire: Nicolas Cage might be the only connection between this prototype and ‘internet culture.’

This prototype is a visual comparison of sound and character lines from 2 movies: City of Angels and Wings of Desire. It is an attempt to:
* Apply Lev Manovich’s method from “Visualizing Vertov” to sound, instead of shots.
* Determine if/how patterns emerge from a sound analysis of a movie.

This analysis would be completely useless without data visualizer’s best friend: comparison. That’s the reason why I’m showing 2 movies instead of one. Choosing an European movie and its American remake seemed like a good comparison for insights.

Screen Shot 2014-10-04 at 2.04.34 PM

How to read it
The area chart shows the relative sound volume from each movie. Because I have no access to good digital copies of any of these movies, made no sense to draw a scale with absolute values.
The red rectangles above the chart show dialogues. I used data from subtitles to draw them.
The slider and play button work like in a regular audio player.
Click here to access the prototype.

Browser compatibility
* Safari: best results
* Chrome: slow interface, but fully operational
* Firefox: might not work

Also, bear in mind that the sound files from these movies are huge. I did my best to manage all sound loading events. Expect some delay before playing, though.

Notes
The process and reasons behind this prototype are documented in this presentation. In-depth explanations will be presented in the yet-to-come paper. I’ll update this post as soon as it is ready.

[Thesis 1] Methodological Module Workshop

In this workshop, me, Daniel, and Gabor listed and discussed methods we found in our thesis research so far.
I described the method used by Lev Manovich in his article Visualizing Vertov. The paper applies some of Manovich’s techniques — direct visualization, cultural analytics – to visualize the works of Russian movie director Dziga Vertov.

method

Database

  • digital copies of Vertov’s films provided by The Austrian Film Museum, in Vienna.
  • metadata from these movies.
  • cinemetrics database with shots lengths. This one was used for the comparisons betweens Vertov’s movies with other ones.

Tools

  • software
    • ImageJ (image processing and analysis software)
    • custom software
      • computer vision
      • image analysis
  • visualization

Method

  • starts with a “bird’s eye” view, comparing a set of movies from the 20th century, based on date and average shot length, in a scatterplot
  • zoom in into Vertov’s movies compared to Eisenstein’s (Russian movie director from the same period); uses both mean and median shot length
  • zoom in into Vertov’s movies comparing the shots — Manovich uses the 2nd frame of each shot to represent it
  • zoom in into the shot’s frames to compare average motion

Manovich often refers to other film studies made and also to contextual information, such as:

  • Vertov’s manifestos
  • Russian avant-garde ideas about montage
  • Russian political context