Conference visitors

Richard Rogers, University of Amsterdam

Conference Venue: Trouw Gebouw, Amsterdam

Florian Cramer

Siva Vaidhyanathan

Conference crew and speakers

Konrad Becker Book Presentation: Deep Search. The politics of Search Beyond Google

Evening Program

Photography by Anne Helmond. More pictures on Flickr.

New media theorist Lev Manovich summarized his latest contribution to the field of software studies: cultural analytics. The idea of cultural analytics was first presented by Lev Manovich in 2005, and in 2007 he released a paper at CALIT2 entitled “Cultural Analytics: Analysis and Visualization of Large Cultural Data Sets.” In his talk today Manovich routinely made comparisons between cultural analytics and cultural analysis, and so it was necessary that audience members understand the distinction between these two (similarly sounding) terms: whereas traditional cultural analysis relies on real-world resources (human interpretation and physical storage), cultural analytics relies on the computer and search algorithms in order to discern and interpret culture.

Within cultural analytics Manovich is looking to answer questions such as: Can we create quantitative measures of cultural innovation? Can we visualize flows of cultural ideas, images, and trends? As a new way to study culture he suggested, “Let’s take principles from search engines (and data analysis in general) + web analytics and Google Trends (interactive visualization of patterns) + Google Earth (continuous zoom and navigation) + Manyeyes (visualization, sharing of data and analysis).”

Manovich’s work is perhaps more easily understood through examples that implement these techniques. He presented the “Interactive Visualization of Image Collections for Humanities Research” project which was developed by Manovich and the Software Studies Institute at UC San Diego. The project explores a collection of Mark Rothko’s paintings: turning the paintings into sets of data that can be graphed, and then turning that data into a collection of paintings (see the video below). Manovich argued, “by extracting and graphing this data it will help us understand patterns and explore trends in a painter’s life and work.”

This visualization shows changes in Rothko’s painting’s average brightness over his career.

This visualization organizes the paintings by their brightness and saturation:

Manovich spoke in Amsterdam last May at the Paradiso to which his presentation on cultural analytics raised a great deal criticism [1] [2]. Shortly after his last talk professor and art historian Edward Shanken wrote the following on The University of Amsterdam blog,

“The outcome of the analysis was as underwhelming as the method was problematic. The challenges of accurately capturing the color and tone of a painting in digital form and then representing them on a monitor are well known. The challenges of comparing multiple paintings on monitors is all the more complicated. While there may be insights to be gained by such a method – and I’m not sure how relevant they would be even in the best of circumstances – it appears to be limited to only the most superficial formal aspects of a painting.  And while certain aspects of connoisseurship may be aided by computer analysis of high-resolution digital images, Manovich’s example was far from that. What do we learn about Rothko or about art in general from an analysis of the brightness in his work over time?  Why even bother posing that as a research question?”

This morning Shanken politely asked Manovich,

Your work enables us to ask questions we might have not seen before, but could you tell us more about the particlar insights that this type of data has shown?

Manovich answered,

Every time I make visualizations I see something I have not seen before. The most common ideas about culture get challenged.

He went on to display the following graphic entitled “Seeing How We Play” and commented, “This visualization compares interactivity in ten video games over two decades. It shows very clearly what the relation is per game is: interactive vs. non-interactive times, and patterns of rapid changes.”

Richard Rogers, Director of the University of Amsterdam’s Digital Methods Initiative, poignantly followed up,

Have you been Googlized? A lot of critics say Google has taken over industry, over industry. E.g. libraries. Is Google now taking over humanities?

To which Manovich replied,

Good question. It’s not only Google, it’s general ideas and methods. We re-use and apply them for cultural analytics. At the same time we want to understand the methods so we can critique them better.

Later in the day at the Googlization panel Siva Vaidhyanathan asked Manovich a question that many of us were probably wondering,

Isn’t what you do expensive? It’s more than I’ve seen in any humanities lab…EVER!.

Of course the question was in reference to the 287-Megapixel HIPerSpace Wall seen early in Manovich’s presentation, but either Manovich didn’t completely understand or was trying to avert the question with his short reply,

I do most of my work with my laptop using open-source software.

Whether you agree or disagree with his cultural analytics research, Manovich’s allure is that his ideas are often compelling and provocative. He recently released a digital copy of his upcoming book “Software Takes Command” on his website where more information about Lev Manovich and the Software Studies Institute is also available.

Florian Cramer, head of the Networked Media Master at the Piet Zwart Institute in Rotterdam, ended the last session of The Society of the Query conference. The Alternative Search 2 session presented a few of the latest web technologies as potential directions for the web and search engine design in the near future: RFDa, which would make the shift to what Steven Pemberton named the web 3.0, and semantic search, as implemented in the Europeana project.

Florian Cramer concluded this series of presentations with a critical and somewhat pessimistic evaluation of the current state of the web and the idea of a semantic web and semantic search, as one of its potential futures. His three main arguments revolved around: “why search is not just web search (and not just Google),” “why semantic search is flawed,” and “why the world wide web is broken.”

The first point expressed his frustration with the narrow understanding of the notions of query and search engine on which the conference focused. As he explains, wikis and social networking sites also include the search engine functionalities.

As far as semantic search is concerned, Cramer usefully pointed out to the difference between folksonomies, the currently used form of semantic tagging, and the universal semantic tagging which a semantic web would require. While folksonomies are “unsystematic, ad-hoc, user-generated and site-specific tagging systems,” (Cramer, 2007), like the tagging systems of Flickr for example, the semantic web would require a structured, universal tagging and classification system which would apply to the entire web. Cramer is skeptical of the possibility to create this unified, ‘objective’, meta-tagging system because classifications, or taxonomies, are not arbitrary but expressions of ideologies, which would call for the discussion of the politics of meta-tagging. While meta-tagging may have its advantages, such as arguably empowering the web users and weakening the position of large web services corporations, although still maintaining the necessity of search engines to aggregate data, it also has several potential weaknesses. The semantic web model must be based on trust in order to prevent some predictable problems, such as massive spamming.

In the concluding section, Cramer expressed his concern that the Internet as a medium for publication and information storage is not sustainable and argued for redundancy in web archiving. However this desire for permanence raises questions about the nature of the medium itself.

Steve Pemberton starts off with explaining the Sapir-Whorf hypothesis, which proposes a link between langauge and thought. If you do not have a word for something, you cannot think about it, and if you do not think about it, you will probably not invent a word for it. Pemberton applies this idea to the term of “Web 2.0″, which has been created by a publisher who wanted to organize some conferences about the idea that websites achieve value by users transferring their data to them. But the concept of Web 2.0 already existed in the Web 1.0 era, namely in the form of ebay. Nowadays, when we can utilize the established concept of the Web 2.0 one can talk about and discuss this phenomena.

In his speech he suggests that people should have their own machine readable websites instead of giving their data to middlemen and mediators as the model of Web 2.0 requires it. According to Pemberton the problem is that the network-organization  the Web 2.0 separates the web into several sub-webs. Referring to Metcalfe‘s Law Pemberton states that this separation reduces the value of the web as a whole.

Additionally he mentions further problems prevailing in regards to the Web 2.0. First of all using Web 2.0 applications like social networks and photo-sharing sites forces you to log in to certain kind of organization, you have to adapt to their data format to be able to publish and work on your contents, which is equipollent to a certain commitment. Subsequently there is the question about the case of deletion or closure of your account or even the death of the network (like mp3.com, Google video, Jaiku, Magnolia). One has to rely on the provider that he keeps your account running, so that the data and the work you have put into it do not get lost. Facebook for example closed a woman‘s account because they decided that an uploaded picture showing her breastfeeding is totally inappropriate.

The crucial point of Pemberton‘s view is that we need personal webpages which have to be readable by machines, so that the sites can be scanned and used by an along coming aggregator. According to him the solution to enable machine readable sites is the format RDFa, which Pemberton refers to as the „CSS of meaning“ and which represents the incarnation of the Web 3.0. The advantage of RDFa is that it joins together different data automatically, things do not have to be joined together because RDFa already linked data together from different places.  Futhermore Pemberton states that the usage of machine readable sites has several advantages for users. For example the browser can provide the user with better experiences, if it is able to identify addresses and dates it could directly offer the possibility to find it on the map or inscribe it to the calendar.

Peripheral Forces: On the Relevance of Marginality in Networks
Daniel van der Velden is a is designer and researcher. He organized the Quaero (Latin for ‘I seek’) event at the Jan van Eyck in 2007 and has been working on alternative interfaces for search results, to move away from the ‘top-10-search-result-experience’, and to create new search options and visualizations. At Society of the Query, Daniel is speaking on behalf of the design collective Metahaven.

His talk starts with the song “Drive By Hit”, by the music group ‘The Search Engines’. The Search Engines’ song on Myspace illustrates the fading distinction between search engines and social networking sites. More and more these different intranets deal with one particular problem: how to become and stay a main source of information within ‘the growing pool’ of information sources.

Daniel’s talk is the result of a thesis that was developed at both the Quaero conference and the following discussions during the 3 months of research that lead up to the ISEA 2008 exhibition in Singapore. In this thesis Daniel (and Metahaven) stated that the authoritative sources of print have been exchanged for socially powerful, nodal publishers. This form of power is built on social ties, that replace the old fashioned hierarchical authority. Moreover, the system of ‘peer review’ has been exchanged for a social ‘peer pressure’ of linking (nodes feeling the pressure to link to nodes that consent or correspond to their own opinions), which makes most linked nodes redundant. The interesting challenges to any idea, notion or argument are insteas developed in the sphere that is exists outside the center, in a periphery or bailie. These are often not directly connected to the statement and exist isolated as isolated worlds away from the powerful, reigning opinion. Metahaven wants to develop a search engine that connects these different spheres, to provide different points of views on particular issues and be able to put emphasis on the marginal forces.

Daniel van der Velden states that we need a new world political map that takes the changing, globally distributed power relations into account. The biggest question concerning this statement is what such a map would look like (probably dynamic), and how different forms of power can be registered in such a map?

The internet as a finite architectural object, as is reflected by the data centers, the ‘dark fiber’ networks, is actually a real terrain that is not vague at all. They are real geophysical terrains. the physicality is actually finite. Metahaven would like to argue for a political map that would reveal the networks of power to generate a better oversight of where information exists, to connect these different worlds of information and the geographical world as we know it. They want to open the black boxes of cloud computing.

Such a map would also make clear that our use of Google is quite alike the way we drink Coca Cola. We like the sweet taste of Cola, so we keep on drinking it, but in fact we have no idea about its contents. According to Daniel Google is also very much like Ikea, a firm that provides us with furniture and other tools that are created only to be used in one way. This furniture cannot be changed, or expanded upon for different usage. He concludes that we need to be more aware of Googlekia.

With Google seeping into every nook of the conference – the subject, direct or indirect, of most presentations and discussions – you might ask why Google isn’t here to speak for itself. Unfortunately and unsurprisingly, the company makes it very difficult for staff to speak at events (look at how rarely they attend the industry’s largest conferences: SIGIR, WSDM, WWW.)

Lucky for us here at Society of the Query, we’ve got a company rep in the house, Ingmar Weber, a search engine researcher from Yahoo! Weber rounded out yesterday’s discussion with his lecture “It’s Hard to Rank Without Being Evil: where evil means big centralized and keeping track of a huge query log.” Chock full of metaphors linking data to wealth, his talk proposed an alternative search engine of the future that makes query logs a free public resource.

What’s a query log? Let’s say you’re a designer like Weber and want to pioneer this alternate search engine. First you’d consider ranking, or how to organize, prioritize, and filter the web’s data. You could rank a few ways: by document content, such as a word and where it appears on a page, the most basic ingredient of a search; or by hyperlink structure, using a giant webcrawl to discern hits and inlinks – essentially votes – from other websites. Or you could use query logs. Query logs are quality votes; they show that users who search for x always click on y. They also show relations between pages – page y and z are clicked by the same user. A search engine could use this implicit relevance feedback to infer what people like and direct them there.

Over time, a log of individual search actions becomes powerful resource, a goldmine of data. Put it all together, and we could find out flu patterns or fine tune election predictions, or discover what local bar most people like. But there’s a paradox: if you’re using search data to build a search engine from scratch, you’d need to pull that data from some other, pre-existing search engine. And currently there is no access to major search engines’ query logs. Companies hoard their logs like misers sitting on mounds of gold.

There are other such hidden mounds of gold, or ‘information silos,’ as Weber terms them. Mobility data from mobile phones for instance, could tell us where people are at all times. This would be useful to predict traffic jams, for one. Also shopping basket information, held by credit card companies and stores, could tell us what people are buying, where and when. Imagine a real time snapshot of the amount of junk food consumed.

Weber wants to know if we can unlock these silos and chase the misers away, but still respect obvious privacy issues and potential abuses. How can we all contribute to the query log but protect ourselves from intrusions or misuse of our personal data?

Weber offered a few current examples, such as Ippolita’s SCookies, a site that swaps cookies among Google users; you offer up search information but SCookies makes it anonymous. Data sharing without the creepiness factor. What other legal and technical innovations could open up massive querying data for the public good? There’s no answer yet. But who knows what Weber’s cooking up when he’s outside the office.

French artist Christophe Bruno introduced the audience into some aspects of his art concerning search engines, espacially Google. In order to explain his latest project Dadameter, he initially presented a selection of projects he has worked on over the last years. His career as an aritst started with the project Epiphanies which Bruno established in 2001. It is a Google hack collecting pieces of texts and reconstituting these particular pieces in a new structure. This idea was inspired by James Joyce who walked through Dublin writing down phrase fragments he heard on the streets and called those epiphanies, therefore Bruno calls Google an „ephiphany machine“.

Another project of Bruno is Fascinum from 2001. In this project Bruno developed a program searching through Yahoo! news sites of different countries and select and present those pictures which were looked at the most in each country. In a work of 2002 called Adwords Happening, Bruno depicted the development of a generalized semantic capitalism. He started buying different words at Google‘s AdWords application and presented the price of different words, this creates the awareness that via Google any word of any language has a price and can be bought.

Bruno identified that corporate organizations started to highjack methods formerly applied by conceptual artists and called it „Guerilla Marketing“. This was the origin of a famous work of 2004 called Human Browser, in which persons were verbally displaying search results to other people which were transferred to them in realtime via ear-phones, the individual human being then embodies the world wide web. Logo.Hallucination is a different project of Christophe Bruno in which he scans through pictures circulating in the internet and searches for logos of corporations and organizations that are represented in those pictures. If a logo is detected in a picture Logo.Hallucination automatically sends cease and desist emails complaining about the violation of copyright laws. This selection shows steps leading to Christophe Bruno’s newest project, the Dadameter. It is inspired by the work of the french author Raymond Rousse and is a very ambitious and complex project which cannot be summarized easily, the aim is the production of a map displaying our distance to dada. Due to its complexity it is highly recommended to look up the details about the project here: Detailed information about the Dadameter.

Notes and discussion on The Society of the Query are public on Google Wave:

You can find them here:

with:public “Notes on Society of the Query”

Introduction by Sabine Niederer
Alessandro Ludovico – fresh issue of neural is out, with Yesman on the cover, so promising. Alessandro is a media researcher, media critic, and also a new media artist. He is famous by the work “Google will eat itself”. Also known for the  “Amazon Noir” project.

Alessandro
Thanks a lot for invitation: I will continue with Google discussion, the theories that came out from GWEI project. This project was done by 4 different people; Alessandro was in charge of the theories.  “I want to talk about the GWEI project and about unexpected problems we encountered after this art project. Also I will talk about the Google self referential side. I will try to prove that Google can become/ will establish its role as a public service”.  Therefore the title of the talk is:  “The Google paradigm: for the final dictator it is never enough”

Google establishes monopolies via their pervasiveness, coolness, and attracting functionality. They are error-proof and have an accelerated innovation rate where the word antitrust sounds unattractive.

Google does this by establishing rules that are flexible. Internally, their organizational motto of ‘freedom” turns out to be very effective.  Externally, products are light and convincing. As an example, contextual advertisement is mentioned. Their services are funny and attractive. They make up for a large part of Internet and they want to entertain us forever. There are options to debunk their perfect level of marketing, communication and strategies in our mass-based economy.

About the Google-will-eat-itself project (GWEI)
We started with principally focusing on Google’s way of online marketing. The analysis is that all corporations have to make cultural interfaces in order to have or create a capitol of attention. This becomes more and more precious. As Cory Doctorow stated: we are in an era of distraction.

We created an affective hack by establishing fake websites. This website(s) pretended to be about eCommerce and online marketing. These fake websites were aggregating marketing news websites. After a while, we submitted and subscribed to the Google AdSense program. For those who do not know AdSense: it lets you have textual and visual adds. Google pays money for every click on such an add. What we did then with the ads was that we opened a Swiss bank account that was linked to Google Shares (latest price). So, AdSense income was linked to buying Google shares. Alessandro now shows the actual account (not allowed to show in public, but we will do it anyways). It shows your earnings for every click. Google was giving us the money to buy itself (hence Google will eat itself). Why a Swiss bank account? Because Google is worth more than all the Swiss banks together. Alessandro now shows pictures of the exhibition. The motto of this exhibition was: “lets share their shares”. Google figured out our scheme after a while (via human and software tools they use) they mechanism. They started closing down our Google ads.

The software diagram is now showed. This software makes fraudulent clicks every time a visitor comes to site, sending Google the data as if the user had clicked on the advertisement. So, the software was simulating user behaviour. For us it was a scientific experiment. For me personally, it was also interaction, it was questioning how to define a fraudulent click, because it is the same data as a permitted click would be. It is impossible to distinguish. Who decides that it is a fraudulent action? There is no CCTV on your mouse clicks – but just data from a computer to a computer. GWEI is conceptual artwork, not to practically take over Google. Summarized in a nice calculation: The rate is 23 million years to take over the Google shares (in this project).

Some interesting problems during and after the project
1) We were invited to a conference by Google in half moon bay, California. They said: “Hi guys, we want to learn about what you are doing? Can we arrange a talk?” After a while, they were repeatedly asking the technical details of the software we used and then they disappeared. Maybe the conference never existed.

2) We were approached by a journalist – the chief tech journalist of Reuters – and he was going to make an interview. He was asking Google about our project. Google replied: “We don’t comment on any AdSense account. The journalist said: “Sorry, I don’t have the counter story, so no interview. No further replies.

3) We were also approached by Wired magazine. They were opening an art department.  For this first art department, they want to talk about the “GWEI” project. They were enthusiastic – we did an interview, we made colorful images and sent these images to Wired (page size).  Then they killed the issue. It turned out Google had its influence even on Wired advertisement.

4) We were often approached on Skype by anonymous people. Alessandro now acts out such a Skype conversation:

Guy:  hi
We: hi dear:
Guy: could get you into big trouble
Guy: it is against the law
We: yes, we know, do you know what they collect? So many things are against the law?
Guy: no, just want to know that you know what you are doing.
Guy: fraud is fraud, art or not
We: no we are not stealing. Also, art becomes history
Guy: the judge wont think so
We:  we will take the risk
Guy: Ok, your choice, just wanted to inform you about the risk.

Of course, artists are hoping for these reactions. But it is also sort of a cliche – law firm that defends the big company. Google responded by saying: Oke, we understand its art, but you have to stop now. Of course we never did.

About the Porcelain interface of Google
It is so clean in its interface – everybody likes this interface and it is widely recognized – these interfaces are becoming standard. Clean, rounded, known. But the interface is impenetrable – that is porcelain.  I was in Dublin for a lecture – somebody said to me that if I wanted, I could make a tour inside of Google. I did accept the invitation: in the belly of the beast, so to speak. I had the opportunity to look inside in Google and check the porcelain interface from within. Actually, the type of organization is a recurrent theme (freedom, young, cool to work there). You can see the colors inside the spaces in Google office in Dublin, They are always round- shaped, familiar. The brand is perceivable everywhere and in everything. Especially the G. It struck me. These four colors have influence in our visual life. Alessandro shows the logo of Google wave: it is rounded, smooth.

The Gateway
Beyond the browser interface there are other ways Google is spreading its interface. This gateway to Google becomes self-referential. If we look at Google as a dictator, then how can a dictator be fun for people? By influencing every choice we make? Google knows very well how to entertain Internet users. They periodically release new and effective services; people want them and more of them. It is not Microsoft-like monopoly. Rather, Google, uses their porcelain interface. It is shining and funny and everyone knows it; is familiar with it. Via this interface they are presenting themselves as a public serve. You buy a computer and then there is Google. Everything is light, easy and shining. Fast and undated, the cream of fun and the strawberry of results make the monopoly. The database of Google is very valuable. Most pages on the net are put through page rank algorithms. The website can be located and statistically analyzed; this is the secret dream of every Internet market incentive.

The point is that the user ignores all recorded data. They are hypnotized by interfaces and services. We are giving our data to Facebook without even thinking about it. But, unlike Facebook, this funny empire has another element: advertisement. It is its core business. Everyone can buy in on the AdSense program. Also, tons of people have become publishers. They accent to have ads in exchange for money via clicks. The final scenario is Google as the giant middleman, between advertisement and publishers and thereby sucks all information. Being in the middle, it makes the balance, but is it not a natural system, it is an economical system.

One example: Google mail
Established not because propaganda of 1 GB space, but the effective spam filter, you can be quite sure you can mail safely. Google don’t want to fight spam, because it makes their killer app possible. Googles’ mission: get info and make it accessible universal. This is comparable with the mission of the library of congress. But they are actually a public service. Google will never be. The book- scanning project is mentioned. The aim is to establish a public service. To be more precise: a privatized public service.

I will show a sarcastic video by onion. It is about the outcome of these services. A two minute-video about the opt-out village; http://www.theonion.com/content/video/google_opt_out_feature_lets_users
Who can build a prison like that? Only a public space can do that.

Conclusions
The Google effect: creating constituent on new business. the greatest enemy of a giant is the parasite. Think about creating Google with itself manually. If a parasite would suck money, they will kill the giant. We have to start decoding and disposing these mechanisms. In order to create cult, we need to create antibodies to  Google interfaces.

Questions

q:
How does competition show up in your analysis? We should also think about competition in companies and product. There are principles in these products by Google. It is a competition. Is web product logic in your analysis?

a: Competition is not abstract concept. Competition exists if there are comparable conditions. When you have gained the Google position of monopoly, there is competition anymore. They have a very successful model os searching them expanding this condition, I cannot think about a real competitor of Google. They can really built new services – Google programming languages. It is not only comprehensive, but it is made by Google, so it was on every technology – it is building on its position. The program of Google is adding this position. I would push more on the cultural side. it s funny, we are pleased by Google. I am a little scared on the final step: becoming a public service (example of Google books). Disappearing of libraries:  culturally we can accept a privatized public service. We should not be happy with that!

Cees Snoek, member of the Intelligent Systems Lab of the University of Amsterdam, talked about the future of video search. He starts by explaining how the traditional and familiar video search engines work: via text queries. But Snoek points out that an interface working with text queries is insufficient to produce satisfying results. This way of video searching may work if you have a simple query like „flower“. Yet if you have a more complicated query like „Find shots of one or more helicopters in flight“ the classical textbased search interface would not generate adequate results.

Furthermore he explains that the problem with picture or video search is that human beings as cognitive animals perceive semantic patterns when looking at something. This is an attribute computers do not have, and therefore Snoek speaks of a semantic gap between machines and human beings who have the ability to interpret what they perceive and transfer it into semantic patterns. The aspect that human visual perception is a very complex task which needs a big amount of ressources is supported by the fact that visual percipience needs 50% of our cognitive capacity, while playing chess only requires 5%. In his research Snoek tries to find a way to close the semantic and to find solutions to label and name the world‘s visual information.

Cees Snoek presents a modern form of semantic video search engine which is called MediaMill. In his model he provides the search engine with a huge amount of image fragments which can be connected with a particular search query. The engine then calculates every image oder video in regards to a lot of different distinguishing features as color, texture or shape. After this step the search engine determines a distinctive correlation between the particular distinguishing features and the search query supplied by the user. This analysis is the basis for a statistic model – the semantic concept detector – which can be used to search a database for other pictures fitting to this model (Example Video of Semantic Pathfinder). The results found by this model are presented to the user by something Snoek introduces as a CrossBrowser (Example Video here). The vertical axis shows the parts of a video detected by the search engine, while the horizontal axis presents the timeline of specific video clip.

Besides Cees Snoek also presents the VideOlympics, a contest where search engine researches compete in video searching. In front of a live audience different teams try to get the best results in video retrieval for a certain set of search queries (VideoOlympics showcase video).