1. As Chief Scientist of Meta, I'm involved in the creation of the world's most advanced AR (Augmediated Reality) DEG (Digital Eye Glass).

    We're launching the world's first AR App store, "Meta SpaceApp Store"
    and anyone willing to develop an app can get the developer discount:

    Wearable Computing: AR Glass for developers $492 available until Aug 20th (over next 24 hour time period):



    See also Engadget: http://www.engadget.com/2013/08/17/meta-smart-glasses-steve-mann-mediated-reality/

    2

    View comments

  2. The Sousveillance Scenarios

    Steve Mann, 2012 October 22 Presented to at:
    "Identity, Privacy & Security by ReDesign",
    Monday 2012 October 22nd, 4pm to 5:30pm,
    Room 728, Bissell building, 140 St. George Street

    The class was given this file as "sousnario.txt" (plain text)

    Introduction to concept of DASTEM (Design+Art + Science+Technology+Engineering+Mathematics as a unified "discipline"; discipline versus passion, etc.), general discussion...

    One can be in favour of more surveillance (safety) or less surveillance (privacy), or one can be merely conducting studies, in an allegedly objective manner as "surveillance studies". But in all 3 cases, one would be against sousveillance. All three approaches are part of the problem: considering only one side of the argument (one side of the square box below). Those who only consider the surveillance side of the space are inherently biased, i.e. looking at the world from a one-sided point-of-view. We must consider all the veillances, i.e. various combinations of surveillance and sousveillance:
    . .

    The topic of discussion for today's presentation is sousveillance by way of four scenarios which we will discuss and build upon:

    Sousveillance Scenario A

    Allan, a blind man, has an EyeBorg Implantable Camera System (Canadian Patent 2313693). It transmits live video to the "Seeing-Eye People" collective comprised of friends, relatives, and volunteers who take turns to help him see by remotely viewing his video and talking in his telephone earpiece to describe what's in his environment. His wife is confined to a hospital bed and unable to walk, but enjoys "walking" by seeing through his eyes as he walks around the neighbourhood.

    Questions:
    Suppose as in the last sentence above, that Al's body is male, but his vision and seeing is female (his wife). Suppose Al is walking through the park one day, and needs to use the toilet.
    • A1. Which washroom (Men or Women) should he use?
    • A2. An opporunist robber recognizes that Al is blind, and decides to attack and rob him. His wife records the encounter. Other less extreme incidents have also been captured, so as to help Al see and understand the world around him. Does Al have a moral, ethical, or legal duty to let people like the robber, and others, know that they are being video recorded?
    • A3. After a number of years practice seeing this way, and sometimes with the help of a new computer-based Artificial/Humanistic Intelligence system, Al can "recognize" faces of his co-workers, and get around quite independently. Many people don't even know that he is completely blind. Should he notify these coworkers, or anyone else, that he is a camera?
    • A4. Al decides, for moral and ethical reasons, to make it known that he is a camera. He installs a small lapel pin miniature television screen that displays video from his eye, with a small sign analogous to the television screen at the entrance to a department store where you see yourself on TV as you walk in. Subsequently Al is physically assaulted by staff at a fast food restaurant because they don't want a cameras in their business establishment. They argue that since it is their property, they can make up whatever rules they want on their property. Al is met with similar treatment at other places such as a large public art gallery where security guards push him down the stairs and claim he was violating copyright. Al realizes that all of this harrassment stops when he switches off the television display but keeps the camera running as usual. Al tries to contact the restaurant owner and gallery director, but neither respond to him. Should he take legal action against them in order to win the right to openly use a camera? Or, pragmatically, should he simply go back to his original "don't ask, don't tell" approach of keeping the camera hidden. If he decides to be a crusader for the right to openly use his camera (i.e. for the right to wear a television display that shows the output of the otherwise hidden camera), who should bear the cost (legal, etc.) of him doing so?
    • A5. How can the wrongs committed against Al be redressed? Consider swollag.

    Sousveillance Scenario B:

    Bo wears "Glass", otherwise known as "Digital Eye Glass" (EyeTap) to help her see better. The Glass does not record. It simply captures video at different exposures (dark, medium, light) in rapid succession and CEMENTs them together for dynamic range management. Originally Bo only wore the Glass while doing high dynamic range activities like welding, but discovered Glass helped her see in many other situations.

    Bo was physically assaulted on various occasions for wearing Glass.

    Bo was physically assaulted by a security guard at a local art gallery because he was afraid she was violating copyright. The guard pushed her down the stairs and told her she was not welcome in the gallery.

    When she spoke to the gallery director, he said that she was welcome in the gallery as long as the Glass did not record anything.

    Bo's lawyer advised her to record all encounters if possible, to use as evidence in court against people who broke the law (either assault laws or human rights laws broken by those treating her differently than others).
    • B1. Should Bo follow her lawyer's advice?
    • B2. Discuss in terms of swollag (action outside the domain of internal investigation, for example).

    Sousveillance Scenario C

    Cy noticed her company often used video-oral contracts rather than written contracts. Normally both parties get a copy of a written contract or it is not enforcable. But her company had a rule forbidding cameras, while they used surveillance cameras themselves. They refused to give her a copy of the video in which she appeared, but they regularly made verbal contracts with her, while recording their copy of the conversation for possible use in court.
    • C1. Should Cy break company rules and hide a camera to record her copy of the contract?
    • C2. If 2 parties make a contract and only 1 is allowed a copy, is the contract valid?
    • C3. Life-in-general is a "social contract" of sorts. Discuss this in terms of sousveillance versus surveillance and in particular in terms of McVeillance.
    • C4. Discuss in terms of swollag, e.g. action outside the company.

    Sousveillance Scenario D

    Di is involed in a car accident in a McDonalds parking lot. She was not at fault. The Manager of the McDonalds was driving a car that struck her car. When she stepped out to photograph the accident scene, the McDonalds manager told her that photography is prohibited on McDonalds property. He physically assaulted her when she tried to photograph the accident scene, claiming that McDonalds is private property and she was bound by the rules while on private property.
    • D1. Discuss in terms of McVeillance theory.
    • D2. Consider other scenarios like fire exits chained shut and civic duty to report safety violations.
    • D3. Discuss in terms of swollag.

    Swollag

    Swollag is that which gives sousveillance efficacy. Surveillance, by its very nature, is associated with the efficacy of the usually large organizations that bring it on. For example, sousveillance benefits from the "guns, gaols, and gallows" of the state. But sousveillance, being typically "crowdsourced" requires a similar mechanism. Sometimes, for example, the swollag comes from social networking and the "court of public opinion". Each sousveillance scenario should be discussed in terms of swollag.


    13

    View comments

  3. McVeillance: McDonaldized surveillance as a monopolization of sight

    McVeillance:

    How McDonaldized surveillance creates a monopoly on sight that chills AR and smartphone development

    Steve Mann, 2012, 1010 (October 10)


    How to cite this article:

    I published this article yesterday on Blogger but then it mysteriously disappeared, so I am re-posting it. As proof of the 2012 October 10th publication date, here is the article saved from Blogger: (link), here are screen grabs: (link), and here is a more complete archive: (link)


    "Surveillance" is a French word that means "to watch from above" or "to watch over". The closest English word is "oversight". Surveillance (oversight) means, quite literally, that those on top are watching those underneath.

    What happens when surveillance becomes a monopoly that tries to extinguish other forms of sight like "sousveillance" (undersight)?

    For example, 101 days ago, on July 1st, 2012, I was physically assaulted by three McDonalds employees because I was wearing Glass ("Digital Eye Glass" seeing aid). And they admitted to enforcing laws that don't even exist (link) ---- laws that their own surveillance cameras would violate if they did exist!

    More and more people are using cameras as seeing aids, whether to photograph a menu and magnify the text, or to use a smartphone with optical character recognition to translate foreign text to their own language, or to read 2d barcodes on products. However, Penny Sheldon, a travel agent from Boise, Id., was also physically assaulted by McDonalds staff in Paris, France, because she photographed their menu (link).

    Contradictions like these abound in many retail establishments that prohibit cameras while they also use 2-d barcodes that require people to use cameras to read them:


    (Use your camera to scan these watermelons ... break the store rule prohibiting cameras)

    Business establishments, like McDonalds, are also heavy users of surveillance cameras, many of which are hidden and secretly record both customers and employees.

    McVeillance

    George Ritzer coined the term "McDonaldization" to mean "The process by which the principles of the fast-food restaurant are coming to dominate more sectors of our society."

    Imagine a process that not only dominates most sectors of our society, but also extinguishes alternate viewpoints. In describing this much more pervasive force, I coin the word "McVeillance" in reference to "The process by which the principles of surveillance are coming to dominate sight itself."(!)

    "McVeillance" is not merely the mass-production of surveillance, but also its one-sided sight: watching everyone while forbidding them from watching back.

    Here's a definition:
    McVeillance is the installation or using of surveillance cameras while simultaneously prohibiting people from having or using their own cameras, hand-held magnifiers, smartphones, or the like.

    More precisely, McVeillance is the ratio of surveillance to sousveillance.

    If there are eight surveillance cameras in a restaurant and only two customers have cameras then the McVeillance equals eight divided by two which is four, i.e. there are four times as many surveillance cameras as their are sousveillance cameras.

    McVeillance can thus be an objective measure or fact, irrespective of the good or bad purposes to which the surveillance or sousveillance may be applied. Sight can be used for many purposes, both useful and harmful. But it should not be McDonalds staff that are the arbiters and permitters of sight. Nor should it be the role of government to be "thought police" and determine what we are allowed to see and not see or remember.

    As a memory aid for an Alzheimers patient, or a seeing aid for AR, a camera for personal use (i.e. not distributing the images to others) should always be considered fair use.

    But whether or not you agree with this viewpoint, McVeillance can still be a useful construct with which to argue for or against this viewpoint.

    As an equation, McVeillance, M, equals Surveillance, S, divided by sousveillance:

    M  =    S

    s


    A graphical depiction of this equation appears below:

    Use of the McDonalds symbol in satire is fair use, but if the reader prefers something less blatant, here it is replaced by something my 5-year old made (link).

    Surveillance/Sousveillance

    "Surveillance" is a French word that means "to watch from above", taken from the word "veiller" meaning "to watch" and "sur" meaning "above". A direct translation of the French word "surveillance" into the nearest English word would be the word "oversight".

    Examples of surveillance include police watching over citizens (citywide surveillance cameras as well as red-light cameras, traffic laws, etc.), and shopkeepers watching over shoppers. Surveillance suggests a general socio-political and economic "oversight".

    More recently computer vision has entered the realm of surveillance, with such innovations as automatic red-light cameras, as well as energy management. I predict that soon there will be a camera in almost every streetlight (http://intellistreets.com/) and ultimately a camera in every light fixture (http://www.lsgc.com/pixelview/) for energy-management (to meet rising costs of electricity by dimming down the lights when nobody's around or needs them).

    Thus the "eye-in-the-sky" ("God's-Eye View") has such a wide variety of purposes and promises (green-technology, public safety) to "save energy, save lives, and "save the planet", that surveillance is for the most part unstoppable, i.e. a fact-of-life. Surveillance has crept into all facets of our lives, including surveillance cameras in washrooms and locker rooms ("Cameras can stay in Talisman's locker room, says commissioner" http://www.cbc.ca/news/canada/calgary/story/2007/03/22/talisman-privacy.html) and the modern automatic flush toilets, faucets, and sensor-operated showers are starting to use more sophisticated camera-based computer-vision technologies (e.g. U.S. Patent 5828793).

    Although there is some degree of "privacy" and "policy" to safeguard some of the surveillance data, there is also a potential conflict-of-interest in that those on the higher "rungs" of the social "ladder of life" (e.g. governments and large corporations) are often the ones who both conduct the surveillance as well as control, make, and enforce the policies that govern it.

    Much of the surveillance is done in secret (the security profession itself is also very secretive) and cameras are often hidden behind ceiling domes of wine-dark opacity, or concealed in a variety of other ways (completely hidden in light fixtures, or integrated into signage, etc.).

    There are people opposed to surveillance, and others in favour of it. Many of the conferences, symposia, and academic programs on surveillance and surveillance studies view surveillance as this one-dimensional axis: a tradeoff between security and safety (more surveillance) and privacy (less surveillance), as illustrated below:


    Sousveillance (the opposite of surveillance)

    If you or I try to question surveillance, we're told "if you have nothing to hide you should have nothing to fear", as if to imply that only criminals are afraid of cameras.

    But try photographing a policeman in a public place. Though perfectly legal, you might cause the policeman to break the law because he might smash your camera or become violent. (various references, e.g. affirming the right of citizens to photograph the police).

    The simple act of photographing a policeman tells us that "veillance" is not merely a one-dimensional space.

    "Sousveillance" is a French word that means the opposite of "surveillance". In French, "sous" means "below", thus "sousveillance" means "to watch from below". A direct and literal English translation would be the word "undersight". Whereas surveillance is very hierarchical, structured, bureaucratic, and mechanized, sousveillance is more rhizomic (like roots that grow horizontally rather than vertically) and human, like social networking.

    Sousveillance can take many forms, and need not necessarily imply a 20th Century "us versus them" confrontation of citizens photographing police or shoppers photographing shopkeepers. For example, many people carry a camera (perhaps as part of a smartphone) to document their daily lives, and to help them see and remember things better.

    For example, many people use a camera as a hand-held magnifier to help them read small print, or to "zoom in" on signs, such as restaruant menus. With the growing population of the elderly, and those with failing eyesight, the camera represents a useful seeing aid, the digital image often being sharper and clearer than that of bifocals or akward compound-lens eyewear.

    Moreover the camera can do other things like translate a sign to another language (e.g. a smartphone can translate the menu in a French McDonalds to English automatically).

    Unfortunately the same business establishments that are quick to embrace surveillance (e.g. to outfit their business establishment with a large number of surveillance cameras to capture audiovisual recordings of their employees and customers) are often not so open to sousveillance.

    Sousveillance is not counter-surveillance. A person can be in favour of both surveillance and sousveillance, for example. Such a person is simply in favour of more veillance (of both kinds).

    The four veillances and their eight points

    More generally, we see that veillance is no longer a one-dimensional axis of surveillance versus anti-surveillance. Consider the diagram below:
    .

    Surveillance and sousveillance are orthogonal (at right angles to one another), i.e. each can, in principle be increased or decreased independently of the other.

    Veillance is the vector sum of surveillance and sousveillance. Veillance is an agnostic form of seeing, e.g. perhaps simply the total number of cameras (irrespective of whether they're surveillance cameras on the ceiling or hand-held or wearable cameras operated by individuals).

    Counterveillance is the opposition to these cameras (an opposition to both surveillance and sousveillance). The center of the diagram is the origin (zero). Can we have a negative number of cameras? Yes, I believe we can! In some sense McDonalds destroyed my camera, so they owe me a new one, just as someone who borrows (or steals) money, owes it back. If you borrow something and owe it, that's like a negative number. In this sense, McVeillance is the linear combination of surveillance with anti-sousveillance (negative sousveillance). They've installed some surveillance cameras and they've destroyed some sousveillance cameras and therefore "owe" some people, or at least me, some (or at least one) sousveillance camera.

    As an equation we now have:
    McVeillance = Survellance - Sousveillance
    where we consider these axes as logarithmic if we wish to be consistent with the previous equation that defined McVeillance as a ratio rather than a difference.

    Another possible direction on the veillance compass is "Anti-McVeillance". I'm not so violent as to wear a camera into McDonalds while also vandalizing their surveillance cameras, but that would be an example of sousveillance combined with anti-surveillance. This direction of the veillance compass I will call "Anti-McVeillance".

    A society with oversight-only is an oversight on our part

    It would be a gross oversight on our part to construct a society without undersight. To have a survellance-only society is to have a society with inherent and potential conflicts of interest. Consider the example of installing surveillance cameras in only the East side of town (see below).


    This could cause a Westward shift in crime's equilibrium, where some criminals actually move West, or where crimes move West by being stopped ("extinguished") on the east side, and and re-born (as new crimes) in the West-End, due to the greater relative (compared to the East) opportunities for crime there.

    Can surveillance cause corruption?

    Consider the case where surveillance cameras are also placed in the West-End, i.e. where cameras are everywhere. Installing surveillance cameras everywhere means that police are watching citizens, shopkeepers are watching shoppers, taxicab drivers are watching passengers, etc.. Thus, those on the bottommost rung of the social hierarchy are under close scrutiny, whereas those on the next rung, e.g. police, shopkeepers, and taxicab drivers, may escape some of this scrutiny.

    With surveillance to the East and West (and everywhere else), crime could actually be driven Upward, from a lower rung of the ladder of power and political hiearchy, to become corruption at the higher rungs of this ladder.

    In this example, it is not that over-sight (even in the panoptic sense) necessarly results in corruption, but that surveillance provides the opportuntities, precisely because of the power asymmetries built into surveillance systems, for corruption. Thus surveillance can cause "movement" of low-level street crimes up the proverbial ladder, to emerge as higher-level corruption, as illustrated below:

    Surveillance may thus help facilitate corruption at higher levels of observation and information gathering. Of course to say generally that "surveillance causes corruption" would be an oversimplification, but surveillance does make it more difficult to commit petty crimes such as pick pocketing and shoplifting, while maintaining opportunities for police officers, shopkeepers, and others at the next-higher level of the social hierarchy to continue to commit crimes.

    How does this actually work? It is not that unskilled street criminals suddely get a position as a high-ranking police officer. Rather, when surveillance cameras are installed throughout all the streets, street crime is reduced everywhere. Street thugs may be caught and sent to jail, or otherwise shut down, causing a shift in crime's "market equilibrium". For example, the increased effectiveness of law enforcement may create a vacuum in the marketplace for stolen goods. The demand for stolen goods remains, but the reduced supply can drive up the price of the stolen goods. This increased price of stolen goods may cause more Upward inhabitants to consider criminal activity, if the criminal activity is made more lucrative. Moreover, a new possibility of using (or temptation to use) the surveillance cameras for criminal purposes (e.g. police stalking potential victims) could be more tempting to certain members of law enforcement.

    See for example, "Cleaner says she was stalked on CCTV by security guard" - News - Scotsman.com, Monday 14 March 2011 22:53. Here is a short quote from this article:
    A SECURITY guard at one of Edinburgh's best-known visitor attractions used CCTV cameras to stalk a young female worker...

    James Tuff used the camera system ... to track his victim and then radio her with lewd comments.

    Tuff eventually sexually assaulted Dora Alves ... He was fined and placed on the sex offenders register for three years. ... She said: "At first it was just the odd comment about my body; he would say things about me having a real woman's body ... "But soon after he would appear out of nowhere when I was cleaning in the toilets. ... as she walked to the canteen on her break and stopped to collect something from her locker. "Mr Tuff came out of his office and grabbed me from behind. ... She said CCTV footage which could have proved the incident took place had gone missing.
    This case raises some interesting issues, such as the conflict-of-interest in surveillance (e.g. CCTV footage mysteriously disappearing when under the control of police).

    Fruit-based analogy of surveillance-induced crimes

    Let me propose an analogy that explains this phenomenon:

    The fruits of our labours exist within a market equilibrium. When goods are easily stolen, they exist like Low-Hanging Fruit (LHF), and if there were such LHF everywhere, its price would be relatively low.

    But when some or most of this LHF is eliminated, thus reducing the supply of stolen "fruit", the demand for it may continue to exist. When the demand remains and the supply is reduced, the price escalates, creates new opportunities for crime in higher places or insider-trading in stolen fruit. And ladders are needed to reach the higher fruits, there exists both (1) an increased incentive for thieves to climb such ladders; and (2) an increased incentive for those already further up these ladders to consider the possibility of stealing these higher fruits, because of their increased value arising from the fruit shortages created by eliminating LHF.

    There thus exists a possibility for surveillance to facilitate corruption, and there certainly have been some documented instances of this. Quantification of this effect remains a topic of future research on the practices of sousveillance of institutional agents.

    Conflict-of-interest inherent in surveillance

    Typically surveillance is done in secret, e.g. the video from surveillance cameras used or installed by police is not as freely and easily available to ordinary citizens as it is to the police.

    Moreover, there have been numerous documented cases of police destroying evidence that might incriminate them. The Tuff case above, where the CCTV footage disappeared, is not an isolated example. For example, in a case of mistaken identity, London police shot and killed Jean Charles de Menezes, a Brazilian electrician, and then lied about what happened ---- claiming he was running from them, when in fact he was just calmly walking and did nothing threatening. The police seized the hard drives from four separate surveillance systems in the area, and later reported that all four hard drives were blank, i.e. that no data was recorded by any of the surveillance cameras in the area. Despite the fact that no surveillance recordings in the area survived, the incident was captured by sousveillance (i.e. by citizens with camera phones in the area). http://en.wikipedia.org/wiki/Death_of_Jean_Charles_de_Menezes

    Does police oversight cause high-level corruption?

    Oversight committees may minimize this situation, i.e. where employees such as officers are watched by someone further up the ladder.

    Indeed, Foucault essentially asked "who watches the watcher?" (1995). Traditionally, oversight has been the immediate answer. But, more oversight only pushes the potential of corruption even higher up the proverbial ladder-of-life. Oversight can shift crime even further up this ladder, i.e. toward higher-level corruption (see below).

    Foucault (1995) advocated an open society, where the mechanisms of control and surveillance are open and subject to pubic scrutiny. Ideally these would be part of a larger mechanism of "potence" that mediates the tension between undersight and oversight.

    This is not to say that surveillance is the cause of corruption, or that oversight is the cause of high-level corruption, anymore than saying that installing cameras in the East end causes crime in the West end. But surveillance often does leave the door open to corruption. In some sense, surveillance is like locking only some of the doors to crime (e.g. the basement doors), while leaving the upper doors wide-open.

    Moreover high-level oversight, such as congressional oversight investigating a corrupt Chief of Police, has the possibility, by the same means, push this high-level corruption even higher, up to the level of President or Prime minister (see below).

    Of even greater concern is also the possibility that oversight may push crime and corruption to more unseen height obscured by the public eye. Indeed, massive investments in police infrastructure and police oversight may actually create a massively secret and powerful-yet-unaccountable force in society whose crimes are much more far-reaching than any low-level street crime.

    Conclusions

    Surveillance provides an incomplete picture that leaves room for corruption, and this incompleteness cannot be fixed with oversight alone.

    Sousveillance is inevitable and a necessary part of our sensory-world. Whether as a seeing aid, or a window into cyborgspace and Augmediated Reality, sousveillance is a natural part of seeing.

    A society without sousveillance and undersight, i.e. a society with surveillance and oversight-only, may leave itself open to this Upward-migration of crime and corruption.

    I summarize this reality as follows:
    A society with oversight-only is an oversight on our part.


    We need some way to guard against the possibility that surveillance may push crime up into the middle rungs of the "ladder of life", and that oversight may push crime up to the top rungs. A proposed solution to this problem is sousveillance, in particular, inverse-surveillance, as a way of balancing an otherwise one-sided "surveillance-only" society. Inverse surveillance might, for example, include citizens photographing police misconduct, shoppers photographing shopkeepers, and passengers photographing reckless cab drivers from within the very cab that might, for example, be involved in an automobile accident. This sort of information, taken from below, then communicated and spread through a social/media network can achieve a level of sousveillance-efficacy ("swollag") that may result in a change in policy. An important aspect of inverse surveillance is that it emanates from individuals recording their personal experience and their immediate vicinity, rather than the recording/monitoring of individuals by an outside party.

    The upward-gaze of sousveillance (see below) provides a possibility of creating balance (equiveillance) between surveillance and sousveillance.

    Placing those in the upper echelons of society over sousveillance potentially allows all persons, not just those down at street-level, to enjoy the benefits of a society with a more balanced "veillance".

    Thus technologies like Augmediated Reality may itself restore sightlines in a plurality of directions, not just the one dictated by McVeillance.

    Acknowledgements

    The figures and illustrations were done in collaboration with Christina Mann and Mike Zandwyk.
    19

    View comments

  4. Augmediated* Reality and "McVeillance"

    by Steve Mann, 2012 September 13th

    Augmented Reality grew out of an earlier effort called Mediated Reality, which uses wearable computing to augment, deliberately diminish, or otherwise modify our perception of reality.
    Here's a couple of pictures from the www.Interaction-Design.org article on wearable computing:




    Cell phone view, Augmediated Reality on iPhone also displayed to Eye Glass in early 2008.
    I've personally been living in an Augmediated Reality environment for the last 34 years, exploring wearable computing in everyday life. This system has evolved from a cumbersome prototype connected to a network of subdermal (implantable) sensors and permantently attached apparatus, to a more sleek and slender more normal looking eyeglass unit that comes on and off easily. Here is how this system evolved over the years:

    I wrote a textbook on the topic, outlining why the recent concept of augmented reality won't work, and what is needed to make it work (i.e. augmediated reality = mediated reality over a mediation zone). See Chapters 2 and 3 of this book for the details.

    Augmediated versus Augmented Reality and its information overload

    Augmented Reality merely adds new matter on top of reality. There are 2 problems with this:
    • Any vision system with a camera and display will, by necessity, modify our perception of reality whether we want it to or not. If we ignore this simple reality we do so at our own peril. Therefore we must think in terms of mediated reality or augmediated reality, not in terms of augmented reality alone;
    • Augmented reality alone generally results in "information overload". What we must do to make this really work is to also take away something when we add something. If the apparatus we're using is actually a reality mediator then we are foolish to merely underutilize it merely as a reality augmentor. An example of a reality mediator is a smartphone with camera and display. It has the capacity to mediate, and it mediates whether we want it to or not.
    Here's some practical examples: a computerized seeing aid that helps us see better, e.g. while welding (in which case we actually need some deliberatly diminished reality, along with the augmentation). See the following article: Quantigraphic camera promises HDR eyesight from Father of AR
    by Chris Davies, SlashGear, Sep 12th 2012

    Diminished Reality example:



    I've invented a "glass" that reduces a dynamic range of more than a hundred million-to-one down to something the human eye can see comfortably. This is an example of deliberately diminished reality, allowing us to see electric arc welding, or the license plate number on a car in a dark alley when the headlights are shining in your face.

    Augmediated Reality allows us to deliberately diminish, augment, and more generally re-map and modify reality perception, and all this can be done automatically with dynamic range management that helps the visually impaired, partially sighted, etc., see better and have higher quality-of-life.

    The future of Augmediated Reality

    Over the last 34 years of living my life in augmediated reality, I've noticed a growing trend toward what I have previously called inequiveillance, but which I hereby name "McVeillance" as something that people can easily understand. For example, while just simply wearing electric eyeglasses that are not even recording my surroundings, I've been physically assaulted by staff persons at places like McDonalds, because they were afraid that the apparatus might be recording.See Physical assault by McDonald's for wearing Digital Eye Glass

    The irony of this was that they recorded themselves by breaking my computer and preventing it from overwriting its temporary buffer space.

    Image from Panopticon In Reverse: Steve Mann Is Fighting for Your Cyborg Rights, by Claire_Evans, http://motherboard.vice.com/, Tuesday, Sep 11, 2012, which itself credits the original source as Journal de bord, Mardi 17 Julliet, 2012.

    Presently McDonalds has yet to repsond to an open letter, but there seems to be general consensus that McDonalds has committed a crime by taking a non-existent law into their own hands.

    Here's some quotes from members of the public:
    • "[McDonalds] took the interpretation and judicial authority concerning Article 9 into their own hands in violation of the law." (comment on previous posting).
    • "Mac Donalds have just been admitting their guilt by saying: 'We regret that our staff's request that you stop filming in our restaurant offended you. As we previously informed you, our employees were simply attempting to protect the right of privacy of our staff and customers, a right which is protected by Article 9 the French Civil Code.' since none of the Article 9 of the French Civil Code give them the rights of doing any of what they just admitted of doing."
    More on this case as it unfolds, in http://eyetap.blogspot.com.

    McVeillance is Surveillance over Sousveillance

    My 5-year old, who was with me while were were assaulted by McDonalds staff, sketched out the letter "M" and drew the word McVeillance beneath it, in the context of this simple mathematical equation:

    Here is set forth an equation: McVeillance is the monopoly on veillance:

    McVeillance is a highly un-individualized form of veillance, in the same way that a "McMansion" is a mass-produced mansion. But McVeillance I define to also imply a prohibition on individual veillance (sousveillance).

    McVeillance, thus, for example, is the installation of a large number of security cameras in a restaurant while at the same time physically assaulting someone for using their own camera as a seeing aid to photograph (and read) the menu.

    Mathematically speaking, I define McVeillance as the ratio between surveillance and sousveillance. Alternatively we can define it as the ratio between the permissability of surveillance to the permissability of sousveillance.

    As an equation, this would be McVeillance, M, equals Surveillance, S, divided by sousveillance, s:
    M = S/s.

    Widespread adoption of Augmediated Reality

    Augmediated reality by its very necessity involves sensing the environment. I forsee that the primary impediment to its widespread adoption is not so much tecnical as social, with the main problem being McVeillance.

    Commercialization

    I'm not the only one anymore. This technology is coming. Companies like Google and Apple are bringing out products, such as that pictured below. You can even notice the similar physical appearance, e.g. the minimalist eyeglass design with the thin aluminum strip supported on two silicone nose pads, holding a glass over the right eye:

    (rightmost image adapted from Antonio Zugaldia's picture on Wikimedia Commons)

    Conclusions:

    In summary, I'll conclude with two main points: (1) Augmented Reality alone will not and cannot work, for it ignores two important elements: (1a) mediation happens whether we want it to or not; ignore this fact at your own peril; (1b) recognizing and capitalizing on (1a) will help us design a better system that mediates and mitigates information overload; and (2) the sensory aspects of augmediated reality will necessarily imply a shared veillance as a replacement for surveillance alone. This will mark the end of one-sided veillance (i.e. surveillance without sousveillance), and that alone will transform society far more profoundly than augmediated reality itself!

    References:


    *Augmediated Reality is a trademark of author S. Mann.


    5

    View comments


  5. Toujours pas de réponse de Sebastien...

    Steve Mann      Aug 20 (9 days ago)


    to
    Sebastien Perochain, Lisa McComb, Caroleen, Christopher Elliott, Don, Katie Daubs, Jake Edmiston, Nick Bilton

    Cher Sebastien,
    Nombre d'entre nous participent à la conception de système informatisés portés par leur utilisateur et de systèmes d'aide à la vision ayant pour but d'aider les malvoyants dans leur perception du monde et d'améliorer leur qualité de vie.

    Nous sommes inquiets de constater que Mac Donalds fait preuve actuellement et peut-être à l'avenir d'une forme d'activisme justicier à notre encontre ainsi qu'à celui nos clients et des utilisateurs de nos produits.

    Nous vous sommes gré de bien vouloir clarifier votre position sur les points suivants:
    1.  l'interdiction des appareils informatisés d'aide à la vision dans les restaurants Mac Donalds;
    2.  la prise en charge par Mac Donalds de l'application du respect de la loi interdisant cette technologie. Merci par ailleurs de référencer précisément l'article du code civil que vous mentionnez.
    Cordialement,
    Steve

    Steve Mann,
    330 Dundas Street West,
    Toronto, Ontario, Canada,
    M5T 1G5

    -----Message d'origine-----
    Perochain Sebastien Sebastien.Perochain@fr.mcd.com
    Jul 28
    cc: McComb Lisa <Lisa.McComb@us.mcd.com>
    Dear Dr Mann,

    We have now completed our in-depth inquiry regarding your visit to our Paris restaurant on July 1, 2012. I would like to inform you that the findings of this inquiry, based on the declarations of employees involved in the alleged incident and also confirmed by all witness statements, did not verify your version of the events.  In particular, the accounting of these employees and witnesses did not include any physical altercation, damage to your documents, or obstruction of your use of the toilet. Furthermore they confirmed that you and your family were able to complete your meals before you were asked a second time to stop filming in the restaurant.                     

    We regret that our staff’s request that you stop filming in our restaurant offended you.  As we previously informed you, our employees were simply attempting to protect the right of privacy of our staff and customers, a right which is protected by Article 9 the French Civil Code.  We aim to provide all of our customers a great experience in our restaurants and remain committed to complying with the laws that govern our business and of the rights of all of our employees and customers.

    We will be very happy to welcome you again in any of our restaurants and sincerely hope that future visits will leave you with an improved image of our service in France

    Best regards
    Sebastien Perochain
    6

    View comments


  6. "[McDonalds] took the interpretation and judicial authority concerning Article 9 into their own hands in violation of the law." (comment on previous posting).


    Here are some additional pictures as evidence that will refute the claims that McDonalds staff "was polite" in interacting with me on 2012 July 1st....

    I wear a computer vision system called the EyeTap or "Digital Eye Glass" or "GlassEye" (or just Glass). The "Digital Glass" uses wearable computers to help people see better.

    During a recent visit to France with my family, we were walking along the Champs-Élysées.  I chose to stop at McDonalds because I really needed to use the toilet, and I know that French McDonalds have "cyborg-friendly" toilets, e.g. separate private compartments for each person, rather than men's and women's toilets.  In order to be considerate of others, I use the single-person washrooms whenever possible.

    My children were also very much in favour of this choice of restaurant.  (Anyone with children will also no doubt understand choosing McDonalds from among all the fine French restaurants that are available along the Champs-Élysées.)

    Unfortunately I was physically assaulted by McDonalds employees because I was wearing Glass.

    This incident was covered in the media, and became known as "McDoGate" in the French media, as #McDoGate, number 1 Twitter trend of the day, as the top scoring link on reddit (the "front page of the internet"), as well as the world's first "cybernetic hate crime", and a petition has also been started.


    (The above image was taken from Journal de bord, MARDI 17 JUILLET 2012.)

    McDonalds has denied any wrongdoing.  For example, their Head of Customer Services claims that McDonalds staff did not damage any of my documents. Rather than rely on their massive number of surveillance cameras they have relied only on the testimony of their employees (with an obvious possible conflict-of-interest).

    It is not clear whether they destroyed their own surveillance videos, or simply refused to view them to see the truth. But because they damaged my computer vision system, some images were retained in it because the wearable computer buffer was not overwritten with new images as would have happened had they not damaged it.

    In my previous post I presented an image where one of the perpetrators could be seen tearing up one of my documents. In order to prove that this document is from me, here is a new picture that shows Perpetrator 1 in posession of a stack of documents that I handed him:
    .
    You can see that Perpetrator 1 is now holding a document from the stack of my documents. This document is a letter from my physician regarding my computer vision system. .
    Although this picture does not prove that the document is actually a letter from my physician, it does show that the document is one of the documents that I handed Perpetrator 1, and therefore it confirms that the document is my personal property, and not some random piece of paper such as "a napkin" as someone had suggested (i.e. that Perpetrator 2 was merely tearing up a paper napkin or the like).
    Perpetrator 1 then handed this letter to Perpetrator 2, as we can see here:

    This confirms that the document that Perpetrator 2 was tearing up, was in fact one of the documents that I handed Perpetrator 1, i.e. that the document that Perpetrator 2 tore up was my personal property.
    Additionally, Perpetrator 1 crumpled up one of my other documents, a magazine I had handed him,
    and struck me with it.

    I'm not saying this is anything close to being hit with a metal rod (in my research on the topic of assaults, I came across this headline: "McDonald's Cashier Seen Beating Female Patrons With Metal Rod"), but it did also damage the magazine.

    Lastly, we see in the pictures the back of the magazine, but the following picture will help prove that the document is a magazine article about my work, i.e. that the document is my personal property (unless they claim to have been in posession of a copy of this Canadian magazine from five years ago, other than by way of my having handed it to them):
    .
    All along, McDonalds has been ignoring me (failing to respond to my letters), until I bring the matter to the public, and their only response seems to be complete denial of the truth. I don't expect an apology anytime soon, but perhaps this matter can serve as an example to learn from.

    It was once said to "Never hit someone who wears glasses". In the Glassage, this is all the more true!

    If you walked into a building and were violent toward the surveillance cameras in the building, you'd likely find yourself a subject of being photographed. More generally, those who resort to violence against veillance (whether it is surveillance our sousveillance) will be photographed: I predict that in the future, most seeing aids (most eyeglass people wear, for example) will keep a cloud-based logfile so that if there's a problem, such as physical assault, the perpetrators can be identified (and hopefully brought to justice).

    Here's an interesting opinion (comment) from my last posting:

    Unknown20 August 2012 05:41

    I don't think Article 9 protects unpublished images inadvertently taken by staffers who triggered the recordings, else the images they have of you for security purposes are equally illegal. http://www.law.ed.ac.uk/ahrc/personality/france.asp I think the staffers took the interpretation and judicial authority concerning Article 9 into their own hands in violation of the law.


    4

    View comments

  7. Some people have felt that my last letter to McDonalds Head of Customer Services would have more likely gotten a response if it had been shorter.

    Here is a proposed shorter letter to McDonalds.

    Please give your comments, and when I have enough comments I'll re-work into a new draft and send it to McDonalds Head of Customer Services, for either a response, or if not, at least a useful discussion of the issues among French legal scholars, and other thinkers, etc. (so the letter should capture the attention and imagination of others in the event McDonalds chooses to ignore it).

    I would also welcome any thoughts on how best to translate this letter into French, and, for example, how to translate some of these concepts into French (with perhaps some re-working to disambiguate nuances, etc.).


    Proposed subject line:
    Why are computerized seeing aids prohibited at McDonalds, and why has McDonalds become a law-enforcement vigilante to enforce possibly non-existent laws that McDonalds has invented?

    A number of us are creating wearable computers and comuter vision systems (augmediated-reality seeing aids, for example) to help people see better, and live better qualities of lives.

    We are concerned about McDonalds' present and possible future vigilante-style of activisim against us and our customers and product users.

    Please clarify your present position in
    (1) not allowing computerized seeing aids in McDonalds; and
    (2) specifically identify the law that you claim exists against such technology and why you are taking that law (if it exists) into your own hands.

    Regards,

    Steve

    Steve Mann
    330 Dundas Street West
    Toronto, Ontario,
    M5T 1G5

    On Sat, Jul 28, 2012 at 6:32 AM, Perochain Sebastien <Sebastien.Perochain@fr.mcd.
    com> wrote:
    Dear Dr Mann,

    We have now completed our in-depth inquiry regarding your visit to our Paris restaurant on July 1, 2012. I would like to inform you that the findings of this inquiry, based on the declarations of employees involved in the alleged incident and also confirmed by all witness statements, did not verify your version of the events.  In particular, the accounting of these employees and witnesses did not include any physical altercation, damage to your documents, or obstruction of your use of the toilet. Furthermore they confirmed that you and your family were able to complete your meals before you were asked a second time to stop filming in the restaurant.                      

    We regret that our staff’s request that you stop filming in our restaurant offended you.  As we previously informed you, our employees were simply attempting to protect the right of privacy of our staff and customers, a right which is protected by Article 9 the French Civil Code.  We aim to provide all of our customers a great experience in our restaurants and remain committed to complying with the laws that govern our business and of the rights of all of our employees and customers.

    We will be very happy to welcome you again in any of our restaurants and sincerely hope that future visits will leave you with an improved image of our service in France

    Best regards
    Sebastien Perochain
    8

    View comments

  8. After more than 2 weeks, I've still received no response from McDonalds' Head of Customer Services.

    Here is the unanswered letter I wrote to them.

    As always I've given them a chance to respond to me directly before taking the matter public, but as in the past they are simply not responding, until I bring the matter to the public.

    As I continue to work with industry to develop and manufacture wearable computers, it is very important that we prevent physical assaults like this from happening again, so we must at the very least learn from this experience and understand how we can grow this industry by creating a world of tolerance and fairness.

    So my letter to McDonalds goes beyond a resolve of my personal experience there, and extends to all of us as we develop technologies to help people see better and live better and more independent qualities of life.

    And since they have not even taken the courtesy to respond to my letter, I am now open to alternative approaches.  I'm not seeking personal money from them, but if they can be compelled to repair the damage, or to support vision research in some way, or at least to not repeat such attacks in the future, that would be helpful to all of us.

    From: Steve Mann
    Date: Thu, Aug 2, 2012 at 9:57 AM
    Subject: Re: From McDonald's France
    To: Perochain Sebastien <Sebastien.Perochain@fr.mcd.com
    Cc: McComb Lisa <Lisa.McComb@us.mcd.com>, Don Taylor <don.taylor@us.mcd.com>, [...]

    Sebastien,

    I have three points I'd like to cover in response to your 2012 July 28th email, and I am asking for a response from you on or before 2012 August 10th; I've also copied in some others who might be able to help motivate you to respond substantively:

    1. Witness Statements as the Only Evidence in McDonalds Investigation:

    I would like to suggest that the possibility exists that there is a potential conflict-of-interest inherent in your witness statements.  An organization investigating itself is likely to have a potential conflict-of-interest that could bias your "witness statements" from your employees (e.g. the assault perpetrators themselves, who are not likely to give unbiased witness statements).

    I would certainly be happy to review these "witness statements" if you'd be happy to share them.

    With all the video surveillance cameras that you have in your establishment you should be able to do better than merely "witness statements" as the only evidence to stand against the solid evidence that your employees generated in my wearable computing system by damaging it and thus preventing it from overwriting its circular buffer; see http://eyetap.blogspot.ca/
    2012/07/physical-assault-by-
    mcdonalds-for.html

    2. Droit d'Image (Image Rights):

    My understanding of French law regarding image rights pertains to the dissemination of pictures but not to their mere acquisition.  A person with photographic memory who merely remembers something for their own use has not broken any law.  If that person were to make a photorealistic painting of recognizeable persons and exhibit that painting, that is the point where the law is broken.

    A person with some form of assistive technology that mimics the human mind and body should not be treated any differently at law than a person, with, for example, naturally occurring biological memory.  With the growing population of elderly we're going to see more and more examples of what I call "Silicon Brain" (e.g. parts of the mind and body being replaced by computers).  Technologies like the EyeTap that cause the eye itself to function computationally do not and should not automatically be considered by you to be against the law (and certainly should not move you to take the law into your own hands rather than calling the Police).

    Thus I feel that you are enforcing laws that don't exist, and that you are enforcing these non-existent laws in a violent and unlawful manner, through the use of physical assault, against me and against many others as I have encountered in my research.  I found for example that your Paris employees physically assaulted someone last year merely for photographing the menu, and I know many people who photograph signs when that they have trouble reading signage, i.e. many people use a camera as a hand-held magnifier.  See http://www.cbsnews.com/
    8301-505125_162-49140976/
    mcdonalds-messes-up-how-not-to-handle-service-mistakes/
    Article Title: "McDonald's Messes Up: How Not to Handle Service Mistakes"
    Date: August 17, 2011
    Author: Christopher Elliott

    I also came across an example of your employees making video recordings of customers and posting the recordings on the Internet:
    Quote: "The video was shot by a McDonald's employee who can be heard laughing."
    Article title: "McDonald's Beating Caught on Tape: Was it a Hate Crime?"
    ABC News, 2011 April 25th.
    as well as other examples of assault:
    Quote: "Yesterday morning in the early hours a McDonald's employee assaulted a pair of female customers with a metal rod. One was taken to the hospital with a fractured skull and broken arm, while the other sustained only minor cuts and bruises."
    Article title: "Greenwich Village McDonald's Employee Beats Two Unruly Customers With a Metal Rod"
    The Village Voice, 2011 October 14th.

    With these kinds of crimes in progress, perhaps customer-owned cameras could help law enforcement bring the perpetrators (whether McDonalds employees or otherwise) to justice, whereas company-owned surveillance video could end up being "lost" or "misplaced" when it shows the organization to be at fault.  

    Getting back to the topic of Wearable Computing, there's a distinction between mere recording devices (which can be covertly concealed), and computer vision systems that use Wearable Computing to help people live better lives.

    There are plenty of hidden spy cameras that people often use in day-to-day applications ranging from investigative journalism to "mystery shoppers".  See for example, http://www.
    hiddencameraglasses.com
    and
    (neck tie camera, button camera, baseball cap camera, etc.).

    A camera is very easy to conceal, so it certainly is not the camera itself that you can object to, but, rather, it appears that you object only to the part of the wearable computing system that helps people see better, i.e. the eyepiece (which is harder to conceal).  Is it not the camera itself you object to, but the computer vision system?

    The computer vision system has no film in it, so no filming is taking place.  I've have seen people hide a massive Arriflex motion picture film camera in a bowling bag over their shoulder, and they could be filming you and you would not even know it!  But film is obsolete nowadays.

    I also notice that you are now using QR codes in your stores, which require a camera to photograph and read by computer.   So you are now simultaneously prohibiting and requiring photography in your stores.

    This matter must be resolved, and it is important not just to me, but also to others, as I am involved in a number of projects to commercialize computational seeing aids such as the MannVis(TM) product that helps people see better.  Additionally other companies are now manufacturing similar products (see attached example).

    Having invented, designed, built, and worn wearable computing technologies in my day-to-day life for 34 years now, I have come across very few problems with this technology especially recently with its widespread commercial adoption.  But it is important that I learn from the experience in your place of business, as we (myself and others) "invent the future".

    Many of these systems did not, in the past, record data at all times.  But as a result of this McDonalds incident, manufacturers are now planning on building continuous cloud-based (i.e. indestructible by perpetrators of physical assault) recording into these devices so that they function like the "black box" flight recorder on an aircraft, and could thus be used as evidence to bring the perpetrators of any assault to justice.  This should serve as a deterrent, but not as a violation of Droit d'Image, because the data is not published or disseminated other than on a need-to-know basis for law enforcement, the courts or the like.

    We are also implementing Priveillance(TM) technology in manufactured products, as we consider privacy to be of the utmost importance, and would welcome any thoughts you might have in this regard.

    3. Potential resolution of this matter, and making sure that we can learn from it so that the same thing (physical assault by McDonalds employees) does not keep happening:

    This matter needs to be brought to a resolve.

    An apology rather than a denial would go a long way toward mending hurt feelings.

    I believe you also need to re-think the taking of the law (Droit d'Image) into your own hands, especially in view of the growing population of people who use cameras to help them see better, so that they can buy more of your products.  In 2012 it is pretty much impossible to stop photography, especially in a popular tourist location where thousands of people are carrying and wearing cameras.  We no longer live in the surveillance-age, we live in the veillance-age or "glassage" where surveillance and sousveillance co-exist to create transparency in both directions.


    Regards,

    Steve



    Steve Mann
    330 Dundas Street West
    Toronto, Ontario,
    M5T 1G5




    On Sat, Jul 28, 2012 at 6:32 AM, Perochain Sebastien <Sebastien.Perochain@fr.mcd.
    com> wrote:
    Dear Dr Mann,

    We have now completed our in-depth inquiry regarding your visit to our Paris restaurant on July 1, 2012. I would like to inform you that the findings of this inquiry, based on the declarations of employees involved in the alleged incident and also confirmed by all witness statements, did not verify your version of the events.  In particular, the accounting of these employees and witnesses did not include any physical altercation, damage to your documents, or obstruction of your use of the toilet. Furthermore they confirmed that you and your family were able to complete your meals before you were asked a second time to stop filming in the restaurant.                       

    We regret that our staff’s request that you stop filming in our restaurant offended you.  As we previously informed you, our employees were simply attempting to protect the right of privacy of our staff and customers, a right which is protected by Article 9 the French Civil Code.  We aim to provide all of our customers a great experience in our restaurants and remain committed to complying with the laws that govern our business and of the rights of all of our employees and customers.

    We will be very happy to welcome you again in any of our restaurants and sincerely hope that future visits will leave you with an improved image of our service in France

    Best regards
    Sebastien Perochain
    4

    View comments

  9. The Society of Glass

    Firstly, I would like to thank the many people who have shown their support regarding a recent unfortunate incident at a business establishment in France.  Unfortunately they have denied any assault or destruction of my property, despite solid evidence that I have provided to the contrary.



    Wearable computers will fundamentally improve the quality of our lives.

    Here is an article I recently wrote for the encyclopedia of Interaction Design: See http://interaction-design.org ("Feature article" which is Chapter 23).

    Toward the end of that article I discuss briefly the social implications of this technology.  Please read through the arguments presented there, and then revisit this page, where I am going to now explain some of the issues brought to light by this incident.



    When traditional optical eyeglasses were first invented, many wearers of these eyeglasses were treated poorly, and discriminated against.  But as time wore on, society began to accept eyeglasses, even to the point where they have, in some instances become fashionable (you can now even buy "pseudo intellectual eyeglasses" that have zero prescription so that you can look like an "intellectual", i.e. as associated with wearing eyeglasses).

    But we're now in the era of DEG (Digital Eye Glass).

    Especially as we age, many of us will choose DEG, which can correct for more than just focus (e.g. prescriptions can be adaptive and interactive).  And the wearable computer becomes, in effect, like an additional hemisphere of the brain (to use Adam Oranchuk's analogy).  "Eye is a camera" and "mind is a computer".

    Unfortunately, however, some of us have experienced resistance to this technology, as some people are frightened of cameras.  Ironically the people most frightened of cameras seem to be the ones who are pointing cameras at us (e.g. big multinational organizations that use lots of surveillance cameras in their own establishments).  My own experience with EyeTap DEG is that objections raised by individuals are usually appeased by a simple explanation of what the eyeglasses do, and how they help me.  Basically individuals can work things through.  But when a large organization has a policy against cameras, we have a fundamental problem that, on the surface, would seem to have a chilling effect on mass acceptance of DEG.

    This makes us ask: "What is a Camera?"

    Camera is a Latin word that means "room" and it is usually an abbreviation for "Camera Obscura" (which means "dark room"), as that is how pictures (and many paintings) were made from as far back as Leonardo daVinci's day.  A cardboard box with a hole in it is a camera.  The human eye is a camera, and the human brain is a recording device.  A person with a photographic memory can see something and remember it for his own purpose.  But if he makes a detailed painting of what he saw, and puts the painting on exhibit, he may be violating the privacy rights of others.

    Today, my DEG captures 120 pictures per second, in groups of three: one that is underexposed, one that is normally exposed, and one that is overexposed.  The computer then combines these together to help me see better, using something that I invented about 20 years ago called HDR (High Dynamic Range; see US Patent 5828793 and this paper: Realtime HDR (High Dynamic Range) Video for EyeTap Wearable Computers, FPGA-Based Seeing Aids).

    If I see and remember something (whether a temporary short-term image cache, or permanently), and use it for my own use, I have not violated anyone's privacy.  This issue has come up a number of times in various establishments, and we have generally been able to reach an agreement that my personal use of imaging is actually less of a violation of privacy than the surveillance cameras that are already present in most spaces, public or private.

    To try and stop picture-taking in 2012 is almost impossible.  Numerous completely covert cameras can be purchased, e.g. covert eyeglass-based wearable cameras cost less than $40 each now.

    And most people have a camera phone that can surreptitiously take pictures (some smartphone apps can continue to record while making the phone look like it is turned off).

    The camera is therefore easy to completely conceal.  The part that's hard to conceal is the part that functions as a seeing aid.  So attacking someone with a seeing aid is pointless, because if the goal was merely to record (without helping people see better) there already exist plenty of technologies to do that!

    In many ways I feel safer living in a world with so many cameraphones all around me, to counteract the otherwise one-sided effect of what would otherwise be a one-sided surveillance society.

    Fortunately, ubiquitous personal cameras have helped society evolve not so much into an Orwellian surveillance nightmare, but more into a "veillance" society where the boundaries between surveillance (watching from above) and sousveillance (watching from below) have become blurred.

    In many ways this is a good thing, toward what David Brin refers to as "The Transparent Society", for which "glass" is the perfect metaphor.

    Moreover, an organization that tries to mandate what kind of eyeglasses a person wears or does not wear, is overstepping their authority and liable for the result (e.g. if a person trips and falls down the stairs because they were required to remove their eyeglass).  Rules regarding eyewear are a far more fundamental intrusion than, for example, prohibiting people from wearing blue shirts in your establishment.

    We need either a Statutory Declaration that a seeing aid is not a "Camera" in these sense prohibited, or we need to accept its existence (as surveillance cameras are somehow not "cameras" according to the rules that prohibit them).

    Computer vision system (electric seeing aid)

    I wear such a computer vision system to help me see and understand the world around me.

    (link)

    I originally created this technology, and the computer vision algorithms (e.g. HDR = High Dynamic Range), to help people see better. I have also assisted a number of blind and visually impaired (partially sighted) persons with various projects, and I continue to conduct research in this area. I was also part of the team that invented, designed, and built rehabilitation technology for the Canadian National Institute for the Blind, and this technology continues to be used by the CNIB.

    The "Surveillight": See the Light ... and the Light Sees You

    If you're playing a video game (perhaps at home, in your underwear), you're probably on camera. Gesture-sensing cameras capture your every move --- not so much to spy on you, but simply to respond to your movements.

    Many of the new LED streetlights also have a camera in each light fixture, and some have multiple cameras in the light, to get better coverage.

    Motion-sensing streetlights have been a long-standing idea that's only recently coming to fruition because of new technologies like image-based motion sensing. The old kind of ultrasonic or microwave motion sensors didn't work well for outdoor lighting at the scale of a typical street. But new image based motion sensors work much better in this application. The raw cost of a camera is less than $2, i.e. actually less than some of the older motion sensing technologies. And the image-based motion sensor works better because it can determine speed and direction of moving objects, and a rough estimate of the number and type of object (e.g. how many cars, how many bicycles, and how many people). Many cities are installing camera-based motion-sensing streetlights as part of their energy savings efforts. There's a small camera in each light that tracks the street usage and reduces the light output (but never turning it completely off) when nobody's on the street.  When one light "sees" a car it determines the speed and direction of the car and sends messages to the lights up ahead to make a "bubble" of light maybe 10 lamp posts ahead of the car but only 2 or 3 behind the car, for example.

    The cameras in modern streetlights are also used for urban planning, and intelligent communities, etc., beyond merely controlling the lights in which they reside.  Such systems have been installed in many European, Far East, and American cities.  There are various companies installing vision systems in streetlights (i.e. in addition to the skyward pointing light sensor, there's an image array pointing downwards at the street).  Some of these companies like Philips, and Tvilight, are in Europe, whereas others like KMW are located in the Far East, and there are others like SmartSite and Eco City Lights that are located in the United States.  There is also a convergence of different applications of image-based sensing.

    There are three primary purposes for building one or more cameras in every streetlight:
    1. Energy management: motion sensing, occupancy estimation, usage estimation, etc., are used to automatically control the dimming of the lights. Such automated light dimmers, on average, represent a typical savings of 72% [reference; to be added...];
    2. Public safety: many police forces already put cameras throughout a city. But the position of the camera relative to ambient sources of light such as streetlights varies.  Some cameras have a light to the left, casting shadows to the right, or vice-versa.  Other cameras have a light directly above them, whereas still others end up being positioned directly across from a light that's in their field of view resulting in glare.  By integrating streetlighting with surveillance, the picture quality from the cameras is much better because there are no shadows from the primary source of illumination, the streetlight.  Thus documentation of traffic accidents, emergency notification, and crime deterrence, prevention, and evidence-gathering could become greatly improved;
    3. Civic management. Citywide surveillance can answer questions like "how many cars were parked on this street or that street, and what is the peak utilization of this street or that street...". This information will help civic planners decide which roads to enlarge or which roads to put on a "street diet" [http://en.wikipedia.org/wiki/Road_diet].
    Some groups have expressed privacy concerns, but the energy savings alone will make it necessary to address these privacy concerns by technological means (like the Priveillance technology), rather than by removing or not installing the cameras.  I predict that in the very near future, there will be a camera, or at least a place to install one, in every streetlight manufactured.

    Moreover, as this technology is already widely used in streetlights, it is now moving to other lighting applications.  Lighting Science Group has recently released a new product which is a light fixture that has a camera in it: http://www.lsgc.com/pixelview/

    Texas Instruments also now makes a camera-based occupancy sensor for use in "offices, classrooms, copy rooms, restrooms, ..." (T.I. specifications for IOS).

    Should there be surveillance in private places?

    Just about anywhere you go in public you will likely be on camera.  In the past, some places were off limits to surveillance.  Once upon a time, people who installed cameras in changerooms, toilets, or shower areas were arrested and thrown in jail.  But more recently, when, for example, Gym manager Chris Kopp installed surveillance cameras in the men's locker room of Vida Fitness in Washington, and bobby-trapped a locker with white powder, he became a hero in the world of fitness clubs.

    Once upon a time Privacy Commissioners would frown upon locker room surveillance but another recent headline reads "Alberta Commissioner upholds cameras in locker rooms at health club".

    A number of surveillance cameras have also been installed in washroom areas as well, and acceptance is growing.  Moreover, as washrooms are automated with computer vision systems, cameras are being used to automate various plumbing fixtures.  See for example Masco Corporation's use of CCD cameras to automate plumbing fixtures (See for example, US Pat. Application 20060231782, CCD camera element used as actuation detector for electric plumbing products, by inventor Jeffrey Iott).

    I refer to this effect as Conveillance, i.e. Concomittant usages that end up being used for surveillance.  Is ubiquitous surveillance good for society?  It would seem that society has come to accept ubiquitous surveillance without questioning it.


    Regardless of whether or not ubiquitous surveillance is justified, should those people who accept surveillance not also accept sousveillance?


    When we're surrounded by "smart lights", "smart toilets", "smart refrigerators", and the like, what's wrong with having "smart people"?  That is, what is wrong with putting intelligence on people?



    La Verreté: La Verre de la Vérité de la sousveillance

    People who live in Glass buildings should not throw stones at Glass

    In a world where people interact face-to-face, in often crowded spaces, a wearable camera is in itself not necessarily a violation of privacy when the images are used only for personal use.  In fact surveillance is a greater privacy violation than sousveillance because, for example, when you're alone, you might still be on camera.

    The following is a quote from my previous weblog on that basic idea:

    Rémi has left a new comment on [...] "Physical assault by McDonald's for wearing Digital...":
    Le droit à l'image concerne les images diffusées publiquement. Prendre une photo, y compris de personnes non-consentantes, n'est pas interdit! Ce serait une absurdité. Extrait de http://fr.wikipedia.org/wiki/Droit_à_l'image : Contrairement à une idée reçue, le droit à l'image ne concerne que l'utilisation des images, et non leur prise de vue.
    Lots of philosophical implications to come ...
    Keep posted for future writings to this weblog ...



    38

    View comments

  10. Physical assault by McDonald's for wearing Digital Eye Glass

    Digital Eye Glass

    I believe that Digital Eye Glass will ultimately replace glasses, and will help many people see better, and improve the quality of their lives through wearable computers; and Augmediated Reality.

    Personal introduction

    I wear a computer vision system, and carry a letter from my family physician, as well as documentation on this system when I travel.

    I have worn a computer vision system of some kind for 34 years, and am the inventor of the technology that I wear and use in my day-to-day life.

    Although it has varied over the last 34 years, I have worn the present embodiment of this system (pictured below) for 13 years. This simple design which I did in collaboration with designer Chris Aimone, consists of a sleek strip of aluminum that runs across the forehead, with two silicone nose pads. It holds an EyeTap device (computer-controlled laser light source that causes the eye itself to function as if it were both a camera and display, in effect) in front of my right eye. It also gives the wearer the appearance of having a "glass eye", this phenomenon being known as the "glass eye" effect (Presence Connect, 2002). Over the years the EyeTap has also therefore been known as the "Glass Eye" or "Eye Glass", or "Digital Eye Glass", using the word "Glass" in its singular form, rather than its plural form "Glasses" (See figure caption, "EyeTap digital eye glass", Aaron Harris/Canadian Press, Monday Dec. 22, 2003).

    Recent news has described me as "the father of wearable computing" in the context of various commercially manufactured versions of similar eye glass, such as those made by companies like Google, Olympus, and the like (see below), so as this technology becomes mainstream, McDonald's might need to get used to it.


    I originally created this technology, and the computer vision algorithms (e.g. HDR = High Dynamic Range), to help people see better. I have also assisted a number of blind and visually impaired (partially sighted) persons with various projects, and I continue to conduct research in this area. I was also part of the team that invented, designed, and built rehabilitation technology for the Canadian National Institute for the Blind, and this technology continues to be used by the CNIB.

    Physical assault and willful destruction of customer's property by persons acting as representatives of McDonald's

    In June of 2012, my wife, children, and I traveled to Paris, France, for our summer vacation, in order to give our children the opportunity to learn true Parisian French (we have them enrolled in French immersion at school).

    On the evening of 2012 July 1st, my wife and children and I went to McDonalds at 140, Avenue Champs Elysees, Paris, France, after a day of sightseeing (8 museums and other landmark sights, as part of a boat cruise package), and while we were standing in line at McDonalds, I was stopped by a person who subsequently stated that he was a McDonalds employee, and he asked about my eyeglass (digital computer vision system, i.e. EyeTap).

    Because we'd spent the day going to various museums and historical landmark sites guarded by military and police, I had brought with me the letter from my doctor regarding my computer vision eyeglass, along with documentation, etc., although I'd not needed to present any of this at any of the other places I visited (McDonald's was the only establishement that seemed to have any problem with my eyeglass during our entire 2 week trip).

    Since I happened to have it with me, I showed this doctor's letter and the documentation to the purported McDonalds employee who had stopped me in the McDonalds line.

    After reviewing the documentation, the purported McDonalds employee accepted me (and my family) as a customer, and left us to place our order. In what follows, I will refer to this person as "Possible Witness 1".

    We ordered two Ranch Wraps, one burger, and one mango McFlurry, from a cashier who I will refer to as "Possible Witness 2". My daughter handled the cash to pay Possible Witness 2, as my daughter wanted to practice her French. Possible Witness 2 complimented my daughter on her fluency in French.
    Next my family and I seated ourselves in the restaurant right by the entrance, so we could watch people walking along Avenue Champs Elysees while we ate our meal.

    Subsequently another person within McDonalds physically assaulted me, while I was in McDonand's, eating my McDonand's Ranch Wrap that I had just purchased at this McDonald's. He angrily grabbed my eyeglass, and tried to pull it off my head. The eyeglass is permanently attached and does not come off my skull without special tools.

    I tried to calm him down and I showed him the letter from my doctor and the documentation I had brought with me. He (who I will refer to as Perpetrator 1) then brought me to two other persons. He was standing in the middle, right in front of me, and there was another person to my left seated at a table (who I will refer to as Perpetrator 2), and a third person to my right. The third person (who I will refer to as Perpetrator 3) was holding a broom and dustpan, and wearing a shirt with a McDonald's logo on it. The person in the center (Perpetrator 1) handed the materials I had given him to the person to my left (Perpetrator 2), while the three of them reviewed my doctor's letter and the documentation.

    After all three of them reviewed this material, and deliberated on it for some time, Perpetrator 2 angrily crumpled and ripped up the letter from my doctor. My other documentation was also destroyed by Perpetrator 1.

    I noticed that Perpetrator 1 was wearing a name tag clipped to his belt. When I looked down at it, he quickly covered it up with his hand, and pulled it off and turned it around so that it was facing inwards, so that only the blank white backside of it was then facing outwards.

    Perpetrator 1 pushed me out the door, onto the street.

    The computerized eyeglass processes imagery using Augmediated Reality, in order to help the wearer see better, and when the computer is damaged, e.g. by falling and hitting the ground (or by a physical assault), buffered pictures for processing remain in its memory, and are not overwritten with new ones by the then non-functioning computer vision system.

    As a result of Perpetrator 1's actions, therefore images that would not have otherwise been captured were captured. Therefore by damaging the Eye Glass, Perpetrator 1 photographed himself and others within McDonalds.

    The following images, all taken by Perpetrator 1 (i.e. their having been captured was caused by Perpetrator 1's actions), were among those recovered from the damaged computer vision system, and will hopefully help in solving this crime:

    Possible Witness 1 (stated he was a McDonald's employee):


    Possible Witness 2 (the cashier who sold my family our food items):


    Left-to-right: Perpetrator 2, Perpetrator 1, and Perpetrator 3:


    Here Perp. 1 struck my Eye Glass (not a direct hit like a punch in the face, but a side-swipe, grabbing motion):

    Left-to-right: Perpetrator 2 tearing up my doctor's letter, while Perpetrator 3 watches:


    Perpetrator 1's name tag:


    Perpetrator 1's attempt to conceal his name tag:

    Please help

    I tried on many occasions to contact McDonald's but have not received any response. As McDonand's does not publish any direct contact email information, I used the whois database to find some email addresses, e.g. of domains like "mcdonalds.com" and emailed those addresses.

    My attempts included filling out various online forms on mcdonalds.com but to no avail. I also tried calling the main number, at mcdonands.com: 1-800-244-6227, but got a voice recording that was totally unintelligible (very loud and distorted), and it appears this number does not work.

    I also contacted the Embassy, Consulate, Police, etc., without much luck.

    In my research, I came across Penny Sheldon, a travel agent from Boise, Id., who was physically assaulted by McDonalds staff in Paris, France, because she photographed their menu. This seems surprising because many people use a handheld camera as a seeing aid to magnify and read signs, etc. (zooming into a picture to see it on screen).

    Penny Sheldon contacted the Police in Paris, but did not receive much help from them.
    I'm not seeking to be awarded money. I just want my Glass fixed, and it would also be nice if McDonald's would see fit to support vision research.

    I don't have the resources to take on a branch of a large multi-national corporation operating in a distant country, but I could use some help and advice as to how to resolve this matter, how to ensure it doesn't happen again to me or anyone else wearing Eye Glass, and what can be done to advance Digital Eye Glass research in not just the technological realm, but also the realm of social responsibility and "culture and technology.


    Best regards,

    Steve

    Dr. Steve Mann, PhD (MIT '97), PEng (Ontario),
    330 Dundas Street West
    Toronto, Ontario,
    Canada, M5T 1G5.

    Research in Wearable Computing and Augmediated Reality

    The more people that adopt this technology to improve the quality of their lives, the more that McDonald's will become accustomed to it.  You can become involved by building your own wearable computer vision system.  See for example, the following links:


    734

    View comments

Loading