Hong Kong Polytechnic University
Abstracts
Keynote speakers
Cees de Bont
Design with a purpose: moving from beauty and functionally to meaning.
The design discipline has gone through many changes in the last decade. From the early days, design is an amalgamation of functionality and aesthetics. Quite often designers presented themselves as “problem solvers”. Designers come up with solutions to the needs of people. These needs can vary from washing your clothes to finding your way at an airport. Nowadays designers are aiming far beyond the making of beautiful objects and the solving of straightforward problems. They aim for: system-based design (e.g. product-service systems), for behavioral change (thereby solving major problems in society), for cultural preservation and for happiness. In the presentation several examples will be given of such design projects. One is about a PhD project on female hygiene in rural India. Another project, that was conducted and supervised by the Jockey Club Design Institute for Social Innovation of the Polytechnic University, is about street hawkers in Hong Kong. The third project will be taken from the Institute of Positive Design of the Delft University of Technology. Through these projects that all serve a different purpose, a very exciting and impactful development of design can be illustrated. More than before design aims at meaningful innovations.
Kay O’Halloran
Multimodal Studies: State of the Art and Future Directions
Multimodal studies has emerged as a significant field of research in linguistics and other areas of language studies (e.g. psychology, education) over the past two decades, resulting in a focus on the interaction of language with other resources (e.g. gesture, gaze, proxemics, visual images, music and so forth). In this talk, I provide an overview of the various theoretical approaches, techniques and methodologies developed for multimodal research over the past two decades. Following Jewitt (2013), these approaches may be broadly classified as social semiotic multimodal analysis (Kress & van Leeuwen, 2001, 2006), multimodal discourse analysis (O'Halloran, 2011; O'Toole, 2011) and multimodal interactional analysis (Norris, 2004; Norris & Jones, 2005); although in reality, these approaches are not discrete areas of study; and furthermore, there are variations of focus in terms of mapping domains of enquiry (e.g. music, displayed art and mathematics) and developing theories and methodologies specific to multimodal research (e.g. definitions of key terms, empirical approaches) (O'Halloran & Smith, 2011). Following an overview of current research, I suggest possible future research directions, focusing on digital technology, which has the capacity to handle the complexity of multi-dimensional analysis over space and time. In this respect, digital technology has not only expanded the potential for multimodal communication and socio-cultural development (in a manner analogous to the printing press) but it also provides the necessary tools and techniques for the study of multimodal phenomenon and the advancement of multimodal research (O'Halloran & Smith, 2013). In essence, it is envisaged that multimodal studies must bridge the ‘two cultures’ of humanist and scientific research (Snow, 1961) to progress beyond general frameworks and descriptions of multimodal phenomena.
W. J. T. Mitchell
Seeing Madness: Insanity, Media, and Visual Culture.
This is a report on a seminar entitled “Seeing Madness: Insanity, Media, and Visual Culture,” that I co-taught with Francoise Meltzer at the University of Chicago in the winter of 2011. The purpose of the course was to put into question the visibility of madness and the madness of visuality, whether in graphic stereotypes, scientific classifications, theatrical spectacle, ritual performance, or cinema and new media. As an exploration of technology, the seminar was equally concerned with techniques and mechanisms for the manipulation of perceptions, mental processes, and psychological states, from mass propaganda to individual minds. The syllabus for the course ranged over the theoretical literature on madness (Freud, Foucault, Lacan, Fanon, Deleuze); classic images of insanity from the pathos formulae of Le Brun to the photographs of Charcot; the DSM IV and the history of psychiatry’s complicity with torture and state power; and finally, what might be called “movie madness,” focusing on a selection of films that do not simply portray insanity, but stage it in a definite relation to the institutions—asylums, psychiatric discourse—that make it visible and intelligible: films discussed included Spellbound, Cuckoo’s Nest, Shock Corridor, A Beautiful Mind, Manchurian Candidate, L’Age D’Or, Now Voyager, Shutter Island, Marat/Sade, and The Snake Pit. This lecture will trace our thinking about two questions: 1) what is at stake in the visibility of madness? 2) what do movies, as well as pre-cinematic media and extended cinema, bring to the subject of madness, and what does madness bring to them? The conclusion of the report is an eccentric speculation on the role of smoke, smoking, and the Cold War in American noir mediations of madness .
Jana Holsanova
Reception of multimodality: Applying eye-tracking methodology in multimodal research
The aim of most producers and designers is to create an aesthetically appealing, effective and user-friendly multimodal text design that makes it easy for the user to find, process and understand information (Holsanova & Nord 2010). However, design is not only a product of the design process, an organized arrangement of one or more elements that has been created in order to serve a certain purpose. It can also be conceived of as a starting point for interpretation processes since it invites the user to a certain interaction (Holsanova 2010). Thus, multimodality can be studied from both a production and a reception perspective. Whereas the production aspect focuses on how information in multimodal messages is presented by the sign-maker or designer, the reception aspect emphasizes how multimodal messages are perceived by the users (Holsanova 2012). In my presentation, I will focus on the reception of multimodality and show how eye-tracking measurements, in combination with verbal protocols, interviews, comprehension tests and questionnaires, can be used in research on multimodality. I will summarise various application areas associated with the use of eye-tracking methodology, account for potentials and limitations of this methodology, and provide examples from eye-tracking studies focusing on the integration of various modes. I will argue that interdisciplinary framework and integrated methods are preferable when studying reception aspects of visual communication and multimodality (Holsanova 2014).
Gunther Kress
On the ‘take-up’ of Multimodality: present work and future agendas
The term (and the concept) of Multimodality has had an astonishing ‘take-up’ across a wide range of domains of academic and intellectual work. The response to the inherent promise – and challenge - of the idea has been enthusiastic and energetic. It has enabled long-standing, persistent problems to be looked at with a different set of lenses. In that, it has extended and altered the ‘reach’ of existing disciplines. Indeed it is going beyond that: it has the effect of softening or even breaking the formerly firm frames of these disciplines, forcing quite fundamental questions to be raised about them.
To a very large extent, this was an idea waiting to be named; it was a question which was ‘in the air’. The reasons for that ‘about-ness’ lie well beyond the semiotic domain: they have their origins in the social, economic and political developments of the last four or five decades. Semiotic practices and changes are shaped by social practices and changes. The strong semiotic frames of the representational world of the last century were the analogues of strong social and economic frames: of community, of class, of nation, of profession, of work, and so on.
If this account has a plausibility, it entails that in order to make sense of the present world of communication and representation, we need to have the best possible view of our present social environments and their organization – politically, economically, technologically. In order to attempt to speculate about trends in the semiotic world, we need to attempt to speculate about the social, economic, political and technological world.
This is the broader frame in which I look at where multimodal work might be going, where it might need to go, and what theoretical developments are needed to allow us to match our work to the pace of the world around us, in all its ways.
Eija Ventola & Francis Low
Commodification, Commercialization and Transformed Semiotization of Angry Birds
Anything can be commodified today. The original is ‘semiotized anew’ as an object, a person, services, etc. and once it is commodified it can be marketed, sold and consumed, i.e. it is commercialized, often globally. This paper explores the semiotic resources that are employed when such commodifications and commercializations have been applied to Angry Birds - originally a computer game created by the Finnish company Rovio. The rise of the popularity of the game has been a remarkable global economic and cultural phenomenon. Thus, it invites a study of the ongoing processes of commodification, commercialization and ‘multimodally transformed semiotization’and how these processes function when the popular game is modified into various products and services, such as stuffed birds, sweets and even newly established tourist destinations of Angry Birds Parks. The Angry Birds-phenomenon – in all of its multimodal transformed semiotizations – is thus an excellent prime object of studying how international business and multimodally realized processes interlink in these ‘transformed semiotizations’. This paper aims to answer the following questions: Into what kind of products, places, events, etc. has Angry Birds been semiotized in various commercial and cultural contexts and how? How to capture the major multimodal transformed semiotizations (from virtual screen game to something ‘real’ and ‘consumable’) and what is the significance of their semiotization anew in product development and business and cultural contexts? The data include various kinds of modifications of Angry Birds and the analyses involve mapping out the multimodal resources used for the modifications. The findings will be relevant to multimodal theory in terms of showing its explanatory extent, to Rovio, the company, and its global partners. The paper discusses the semiotization aspects that have enabled the commodifications of Angry Birds that have become globally commercialized successes and those that remain relatively local phenomenon. It further discusses how the commodifications vary in different contexts. The companies will find the results useful when further engaging in the discourses of this kind and when intermediating between the company goals and the views of the public.
Len Unsworth
Reconstructing viewer stance in animated movie adaptations of literary picture books.
Different styles of character drawing in picture books have been described as minimalist, generic and naturalistic and recognized as key signifiers of a system of reader alignment or PATHOS (Painter, Martin, & Unsworth, 2013). The non-realistic, minimalist style in picture books is associated with an ‘appreciative’, detached observer view of characters, the generic (realistic but not naturalistic) style encourages empathy where the reader ‘stands in the character’s shoes’, while the naturalistic style supports more personalized reader engagement with characters as individuals (Painter et al., 2013). With the minimalist character depiction style in picture books, there is often the concomitant use of relatively long shots and frequent oblique angles, together with observe rather than contact choices from focalization options – creating the ‘appreciative’ reader stance of relative distance from which story events and characters are observed and lessons learned.
But in movie versions of minimalist style picture books there is a shift in PATHOS to a more empathetic interpretive stance constructed through different choices in focalization, social distance and attitude, as well as in the communication of affect, notwithstanding essentially maintaining in the movie the minimalist character depiction style of the picture books. The nature and interpretive significance of this interpersonal shift is examined in animated movie adaptations of picture books including Where the Wild Things Are (Deitch, 1973; Sendak, 1962), traditional Chinese stories such as The Little Stone Lion (Xiong, 2007)and Pangsao Visits Her Mother (He & Bing, 2010), the recent Oscar-winning animation of The Lost Thing (Ruhemann & Tan, 2010; Tan, 2000), and the movie and graphic novel version of the popular children’s novel, Coraline (Gaiman & Russell, 2008; Selick, 2009).
Implications for refining existing systems of focalization choices and their realizations in static and moving images are discussed in relation to issues such as the interaction of social distance, horizontal and vertical angle with rear view portrayal of a character such that the audience point of view is positioned ‘along with’ that of the character, and the subtlety of depicting just the part of the body that could be seen by the focalising character (usually the hands or feet out in front of the unseen body) as a means of inscribing the audience viewpoint as that of the focalizing character. The paper will conclude by briefly emphasizing the significance of further explicating systematic accounts of such meaning-making resources of still and moving images to support the development of explicit multimodal literacy pedagogies.