Can your phone keep you mentally well?
Developments in digital phenotyping have brought new attention to forms of behavioural data collection that capitalise on the apparent ubiquity of mobile phone use and the fact that many people are almost constantly connected to digital devices. A phenotype refers to observable characteristics understood to be shaped by genetics and/or the environment. Digital phenotyping is also known as remote sensing, personal sensing, intelligent sensing, body computing, mobile sensing or context sensing (De Angel et al., 2022; Martinez-Martin et al., 2021; Mohr et al., 2020), all of which suggest an increasingly normalized sense of intimacy with technology in situated environments. These expanding relationships of technological intimacy have sparked further interest of the new kinds of digital ecologies which might be represented, formed and enacted by the ways in which we interact with such technologies (Williams and Pykett, 2022). Likewise, medical anthropologists, philosophers, and ethicists have begun to ask questions about the effects of interacting with technologies, particularly those which monitor, quantify and claim to predict people’s behaviour, including their mental states, to produce a digital phenotype (see, for example, Birk et al., 2021; Coghlan and D’Alfonso, 2021; Martinez-Martin et al., 2021; see also Somatosphere’s “Tracking Digital Psy” series). In what follows, we extend concerns about how digital technologies reshape the mind-brain-body nexus, by focusing on environments. Using social theory and work from digital and cultural geographies, we offer ways that research into digital phenotyping could be furthered in the social sciences by thinking about the geographies of digital phenotyping and focussing on questions of interactive data production, mediation and spatial organisation.
The environment and its relation to mental health has had a recent upsurge in attention, particularly around the therapeutic benefits of green space, and conceptualisations of mental health as “ecosystems” or “ecologies.” Environments, like habitats, afford humans certain possibilities, lessening and increasing their capacities to act. Nikolas Rose and colleagues (2022, p.121) have recently suggested the need to “open up the black box of ‘environment’” and work with neuroscience and the life sciences to create a vitalist form of biosocial science. They argue for a neuroecosocial understanding of mental health in adversity, foregrounding the political, economic, social and structural conditions of precarity and mental health, with an approach that brings together social theories of materiality and embodiment with neuroscientific research on pathways and processes of mental health (N. Rose et al., 2022). For Rose et al., this approach “begin[s] by tracing out, empirically, the niches in which those experiencing mental distress strive to make a life for themselves” (N. Rose et al., 2022, p.132, emphasis added). One different way in which these “niches” are beginning to be traced out is through digital phenotyping research.
Every time we even pick up our phones, we produce raw data for digital phenotyping. A recent BBC article reported on research that suggests that people spend a third of their waking hours using smartphone apps, generating an almost-continuous stream of data. The idea with digital phenotyping is that these data represent an “individual’s behaviors, psychological states, and environments, forming a picture of their lived experience” (Mohr et al., 2020, p.1). The supposedly ‘real-time’ and ‘real-world’ nature of data collection for digital phenotyping is forecast by researchers to significantly advance precision and personalised psychiatry, offering a means “to understand the lived experiences of mental health in context” (Torous et al., 2021, p.319). This promise of precision and personalization is heralded as one of the benefits of digital phenotyping for healthcare organisations, governments, mental health services and individuals diagnosed or living with mental illness. Proponents of digital phenotyping argue that this benefit lies in technology’s potential to provide seemingly objective reports of mental health and illness symptoms. Digital phenotyping is applied to mental health research through, for example, harnessing passive data produced from things such as accelerometry (axis-based motion sensing in smartphones – typically used to record step count) and phone utilisation statistics to pinpoint individuals at risk for depression and anxiety (Huckvale et al., 2019). Another example is analysis of keystroke patterns to predict episodes of mania (Martinez-Martin et al., 2021).
Data for digital phenotyping is captured automatically via sensors embedded in smartphones or wearable devices. Sources of passive data range from phone use metrics (Mohr et al., 2020) to global positioning systems (GPS), voice tone (via microphone) and facial expression data (via camera) (Torous et al., 2021). Whilst the data that feeds into digital phenotyping may be construed by its proponents as passive – because they don’t involve direct input from humans but are rather received from sensors or scraped from our everyday interactions with devices such as smartphones – we can problematise this assumed passivity. For some, swiping, clicking and tapping is better thought of as producing interactional data (Coghlan and D’Alfonso, 2021). In addition to understanding how humans interact with digital phenotyping, we also need to consider the non-human: the interactions between digital technologies, sensors, their mechanisms, infrastructures and the data produced. For example, we could consider how the technical mechanisms of digital phenotyping (such as accelerometry) interacts with the smartphone itself; how the data feeds into other smartphone apps (such as Health on iPhone or the cycling app, Strava), and how this connects to other sensors in the device and other devices in the environment. This indicates instead highly active and more-than-human processes at work in the way that sensors produce environments and digital ecologies.
In addition to paying attention to the interactions between the sensors, objects and technical mechanisms involved in digital phenotyping, we argue that understandings of ‘environments’ in digital phenotyping research need to be unpacked.
Clinical psychologists have raised concerns that through digital phenotyping people may become reduced “data points, abstracting them from their environment” (Cosgrove et al., 2020, p.612). Definitions of the ‘environment’ tend to be sparse in much of the digital psychiatry literature on digital phenotyping. Similar critiques have been made of neuroscientific approaches to environment which are said to be ill-defined (N. Rose and Abi-Rached, 2013) or too focused on people’s proximate environments rather than on their wider socio-spatial contexts (Pykett, 2015; Pykett, 2018). Scholars such as Fitzgerald and colleagues (2016) and Winz and Söderström (2021) have highlighted how neurobiological approaches to mental health often render the environment and what it is made up of as static – classified as discrete factors that act as mental stressors. This rendering can ignore the dynamism of the environment and socio-spatial relations. Measuring facets of our environments, such as living conditions, are fundamental characteristics of the development of knowledges of public health and modern Western medicine. Indeed, the environment, and what it’s made up of, has long played a role in biomedical knowledges and governance, its antecedents can be found further back than Foucault’s biopolitics of eighteenth-century Europe (Meloni, 2021). Whilst the idea of environments playing a role in health and wellbeing may not be novel, the methods through which these relationships are beginning to be digitally measured offer new ways to understand this role.
A recent systematic review of methods of digital health tools for the passive monitoring of depression by De Angel and colleagues (2022, p.1) refers to the location of people rather than their environment, measured by GPS. They note:
The first step in generating meaningful clinical information from data derived from digital sensors is to generate features, which are the smallest constructed building blocks, designed to explain the behaviours of interest. For example, GPS data (sensor), can be translated into ‘location type’ data (low-level feature), and ‘increased time at home location’ (high-level behaviour) derived from location data may indicate social withdrawal or lack of energy (symptom), and may therefore be associated with depression severity.
De Angel et al. (2022) suggest the importance of the temporalities of our locations, our movements in and between locations, and the relations between location variables in understanding depression severity. However they do not suggest that technologies – themselves – mediate and produce locations and environments. We argue that environments and locations are not simply containers for action, but full of mediated technological practices.
The concept of ‘mediation’ has been used recently in digital geographies scholarship to describe how cultural and technological objects such as digital media are implicated in how we experience everyday life. In this context, mediation describes the ‘coming together’ of technology, space and society relations (Leszczynski, 2019). For geographer Gillian Rose (2019), the move from the study of fixed computer screens to sensors embedded in environments has shifted the focus of enquiry from representation to mediation:
[Mediation] refuses the distinction between the real and its representation (Kember and Zylinska, 2012). In so far as it is attentive to questions of meaning, mediation assumes that the world is no longer represented by cultural objects, but is produced at multiple sites, between hardware, software, and humans (G. Rose, 2019, p.168, original emphasis)
In other words, mediation denotes the multiple and contingent ways in which people, place, space and technologies come together and produce everyday life (Leszczynski, 2019). As a concept, mediation questions what digital technologies are doing in space and society, rather than what they represent (Parikka, 2011). In particular, the concept points to the ways such technologies not only mediate our social lives but reshape the social life of data to redefine human identity and the practices of knowing and governing human emotions (Armstrong, 2019; Pykett and Paterson, 2022). For example, the smartphone is not simply a device that brings you closer to someone through having a video call, it produces an encounter. A video call reconfigures our understandings of proximity and distance: multiple people based in different geographical locations produce a digital space. Physical space can feel as though it has collapsed. These technologically-produced encounters change the nature of social interactions (and vice versa), while reshaping our understandings of time, space, and distance.
In Program Earth: Environmental Sensing and the Making of a Computational Planet, Jennifer Gabrys (2016) takes inspiration from philosopher of technology, Gilbert Simondon, to see environments as both geographical and technical. We argue that this approach to understanding environments in digital phenotyping research is necessary. Gabrys (2016) argues that technical objects such as sensors need to be considered within their milieu (environment). To understand how both technological objects and environments undergo change or individuation we need to think of them together. Ash (2019, p.115) notes that Gabrys proposes a techno-geography to “move away from a notion that sensing technologies come to be simply laid or spread on top of a pre-existing non-technical space.” The sensors, data, and devices (smartphones and wearables) involved in digital phenotyping are themselves distributive (Hörl, 2017). Environments involve non-human processes, such as relations between sensor technologies, Ash (2019, p.116) notes these environments can “have selective effects on humans, but do not necessarily appear to human perception as such.” Applying this line of argument to digital phenotyping suggests that research needs to consider not only data and human as hybrid entities, but relations between environments (or milieus) and the sensor technologies as well.
Spatial Organisation and Sociotechnical Milieus
Turning to our third question of spatial organisation, digital mental health researcher John Torous describes smartphones as a central node, or ‘hub’ technology in digital phenotyping, evoking a wider network or spatial organisation of technologies, people, infrastructures and environments (Torous et al., 2021). However, this spatiality is yet to be mapped out in existing research on digital phenotyping. One way in which this could be done is shown by those researching “geographies produced through, produced by, and of the digital” (Ash et al., 2018a, p.27, original emphasis). Sam Kinsley (2014), for example, has steered our attention to the material and infrastructural landscapes of technologies. Kinsley (2014, p.370) writes that sensors, circuits and cables are “material expressions of a sociotechnical milieu” that are comprised with bodies, languages, code and software in the production of space. The various technologies and sensors involved in digital phenotyping make up a sociotechnical milieu. Understanding digital phenotyping technologies in this way means unpacking the relations between material technologies (sensors, smartphones, cables, data banks); the various organisations and humans involved (healthcare organisations, technology developers, data analysts, GPS and individuals); and environments.
The questions that this form of mapping may produce are reflected in researchers in bioethics, social sciences and psychiatry’s engagements with the ethics of digital phenotyping in relation to infrastructural networks. This includes concerns about the involvement of Big Tech companies, privacy and data security, discrimination, and surveillance (see for example, Cosgrove et al., 2020; Birk et al., 2021). Building on this, a geographical approach could usefully map the relations, networks and spatialities of sensors, to understand how data moves through, between, and in, technological mechanisms and objects (sensors, smartphones) and environments (milieus). Looking to digital geographies, we could explore digital phenotyping through its: “speed, rhythm, historicity, location, flow, friction, extension, futurity, splintering, distribution, fracturing, and orientation”(G. Rose, 2017, p.789). This geographical vocabulary is a way to articulate the relations between bodies and technologies in digital phenotyping and to better appreciate the temporal and spatial fluxes of mental health and illness.
One way of exploring friction and flow in digital geographies, has focused on the interface of digital technologies such as apps. The post-phenomenological method proposed by Ash and colleagues (2018b) could be used to inform understandings of the data produced when using smartphones – through analysing the interface itself. One of their examples shows how the interfaces for High-Cost Short-Term Credit products are designed to “manage frictions” at various points (or thresholds) of the application process to prevent people from discontinuing with the application for a loan or credit (Ash et al., 2018c, p.1136). Mobile phone apps and digital interfaces – whether for financial products, mental wellbeing services or mental health research – are designed to make people want to interact with them through various means and for various rationales. This undermines their claims to passivity and objectivity, and instead begs the question of how our individual and collective forms of attention are being shaped through these techno-geographies.
Conclusion: Spatialising digital phenotypes
We can also consider human users of digital phenotyping through this spatial glossary. At a population level, the human imagined through digital phenotyping shares similarities to the bodies that are thought to move through smart cities: a “mobile, navigational body, seeing and being seen through screens as well as smartphone signals and sensors” (G. Rose, 2017, p.787). Engaging with smartphone apps is always embodied – apps are with people in their day to day lives, they become, shape and produce routines, habits and behaviours (G. Rose et al., 2021; Pykett, 2022). There are arguably both processes of abstraction and production through digital phenotyping. Sometimes the body becomes amorphous and extensive; data emerging, leaking and leaving trails through various sensor technologies, databases and infrastructures; and other times the body is reduced when for example mania is to be predicted through keystrokes: the body becomes the isolated keystrokes, points of data. As we’ve shown with environments, we also need to ensure that bodies, and how they are already gendered, raced, classed and so on, are not left undifferentiated (G. Rose, 2017) in digital phenotyping research and application. These concerns are starting to emerge in digital phenotyping research: De Angel et al. (2022, p.11) state that some wearable devices used in research have been found to be more accurate on biologically male bodies and people with lighter skin tones. In future research, it could be useful for social scientists to think of bodies in multiple ways moving through, and being produced by, digital ecologies of mental health at a much wider range of scales. Even in the context of the rapid expansion of the availability of personal digital data and advances in predictive computing, we still need to pay attention to the longer term and broader socio-spatial contexts of behaviour, and to take a more circumspect view of what counts as observable behaviour and the complex environmental realities which shape it.
Jessy E. Williams is a doctoral researcher at the School of Geography, Earth and Environmental Sciences at the University of Birmingham. Her PhD research examines the changing technological landscape of mental health provision in England, focusing on the ways that young people use digital technologies for mental health and wellbeing, practices of research and design, and the role digital technologies play in community mental health contexts. Twitter: @jessy_ewilliams
Jessica Pykett is associate professor of geography, co-director of the Centre for Urban Wellbeing, and member of the Institute for Mental Health at the University of Birmingham. Her research investigates policy innovation, knowledge practices, emotion science, and digital futures. Her books include Neuroliberalism: Behavioural Government in the Twenty-First Century (2017), Emotional States: Sites and Spaces of Affective Governance (2017), and Brain Culture: Shaping Policy Through Neuroscience (2015). Twitter: @JessicaPykett
Armstrong D. (2019) ‘The social life of data points: Antecedents of digital technologies’: Social Studies of Science, 49(1), 102-117
Ash J., Kitchin R. and Leszczynski A. (2018a) ‘Digital turn, digital geographies?’: Progress in Human Geography, 42(1), 25-43
Ash, J., Anderson, B., Gordon, R., and Langley, P. (2018b) ‘Unit, vibration, tone: A post-phenomenological method for researching digital interfaces’: cultural geographies, 25(1), 165-181
Ash J., Anderson B., Gordon R. and Langley P. (2018c) ‘Digital interface design and power: Friction, threshold, transition’: Environment and Planning D: Society and Space, 36(6), 1136-1153
Ash J. (2019) ‘For a techno-geography of sensing objects’: Dialogues in Human Geography, 9(1), 115-117
Birk R., Lavis A., Lucivero F. and Samuel G. (2021) ‘For what it’s worth. Unearthing the values embedded in digital phenotyping for mental health’: Big Data & Society, 8(2), https://doi.org/10.1177/20539517211047319
Coghlan S. and D’Alfonso S. (2021) ‘Digital Phenotyping: an Epistemic and Methodological Analysis’: Philosophy & Technology, 34(4), 1905-1928
Cosgrove L., Karter J. M., Morrill Z. and McGinley M. (2020) ‘Psychology and surveillance capitalism: The risk of pushing mental health apps during the COVID-19 pandemic’: Journal of Humanistic Psychology, 60(5), 611-625
De Angel V., Lewis S., White K., Oetzmann C., Leightley D., Oprea E., Lavelle G., Matcham F., Pace A., Mohr D.C. and Dobson R. (2022) ‘Digital health tools for the passive monitoring of depression: a systematic review of methods.’: NPJ Digital Medicine, 5(1), 1-14
Fitzgerald D., Rose N. and Singh I. (2016) ‘Revitalising sociology: Urban life and mental illness between history and the present’: British Journal of Sociology 67 (1), 138–160
Gabrys J. (2016) Program Earth: Environmental Sensing Technology and the Making of a Computational Planet, London: University of Minnesota Press
Hörl E. (2017) ‘Introduction to general ecology: The ecologization of thinking’: Hörl E. and Burton J. (eds.) General ecology: the new ecological paradigm, London: Bloomsbury Academic, 1-74
Huckvale K., Venkatesh S. and Christensen H. (2019) ‘Toward clinical digital phenotyping: a timely opportunity to consider purpose, quality, and safety’: NPJ digital medicine, 2(1), 1-11
Kember S. and Zylinska J. (2012) Life after New Media: Mediation as a Vital Process, Cambridge, MA: MIT Press
Kinsley S. (2014) ‘The matter of ‘virtual’ geographies’: Progress in Human Geography, 38(3), 364-384
Leszczynski A. (2019) ‘Spatialities’: Ash J., Kitchin R. and Leszczynski A. (eds.) Digital Geographies, London: Sage, 13-23
Martinez-Martin N., Greely H. T. and Cho M. K. (2021) ‘Ethical development of digital phenotyping tools for mental health applications: Delphi study’: JMIR mHealth and uHealth, 9(7), e27343, doi: 10.2196/27343
Meloni M. (2021) ‘The politics of environments before the environment: Biopolitics in the longue durée’: Studies in History and Philosophy of Science Part A, 88, 334-344
Mohr D. C., Shilton K. and Hotopf M. (2020) ‘Digital phenotyping, behavioral sensing, or personal sensing: names and transparency in the digital age’: NPJ digital medicine, 3(1), 1-2
Parikka J. (2011) ‘Media ecologies and imaginary media: transversal expansions, contractions, and foldings’: The Fibreculture Journal, 17 – Available: https://seventeen.fibreculturejournal.org/wp-content/pdfs/FCJ-116Jussi%20Parikka.pdf, 34-50
Pykett J. (2015) Brain Culture: Shaping policy through neuroscience, Bristol: Policy Press
Pykett, J. (2018) ‘Geography and neuroscience: Critical engagements with geography’s “neural turn”’: Transactions of the Institute of British Geographers, 43(2), 154-169
Pykett J. (2022) ‘Spatialising happiness economics: Global metrics, urban politics and embodied technologies’: Transactions of the Institute of British Geographers, https://doi.org/10.1111/tran.12528 (early view).
Pykett J. and Paterson M. (2022) ‘Stressing the ‘body electric’: History and psychology of the techno-ecologies of work stress’: History of the Human Sciences, https://doi.org/10.1177/09526951221081754
Rose G. (2017) ‘Posthuman agency in the digitally mediated city: Exteriorization, individuation, reinvention’: Annals of the American Association of Geographers, 107(4), 779-793
Rose G. (2019) ‘Representation and mediation’: Ash J., Kitchin R. and Leszczynski A. (eds.) Digital Geographies, London: Sage, 164-173
Rose G., Raghuram P., Watson S. and Wigley E. (2021) ‘Platform urbanism, smartphone applications and valuing data in a smart city’: Transactions of the Institute of British Geographers, 46(1), 59-72
Rose N. and Abi-Rached J. M. (2013) Neuro: The new brain sciences and the management of the mind, Woodstock: Princeton University Press
Rose N., Birk R. and Manning N. (2022) ‘Towards neuroecosociality: mental health in adversity’: Theory, Culture & Society, 39(3), 121-144
Torous J., Bucci S., Bell I. H., Kessing L. V., Faurholt‐Jepsen M., Whelan P., … and Firth J. (2021) ‘The growing field of digital psychiatry: current evidence and the future of apps, social media, chatbots, and virtual reality’: World Psychiatry, 20(3), 318-335
Wakefield J. (2022) ‘People devote third of waking time to mobile apps’ – Available: https://www.bbc.co.uk/news/technology-59952557
Williams J. E. and Pykett J. (2022) ‘Mental health monitoring apps for depression and anxiety in children and young people: A scoping review and critical ecological analysis’: Social Science & Medicine, 114802, https://doi.org/10.1016/j.socscimed.2022.114802
Winz M. and Söderström O. (2021) ‘How environments get to the skin: biosensory ethnography as a method for investigating the relation between psychosis and the city’: BioSocieties, 16 (2),157–176
- The hype and hope of data for healthcare in Africa
- The subjects of digital psychiatry
- Form, Content, Data, Bodies: Jonathan Zong and Beth Semel on Biometric Sans
- Mapping Algorithmic Assumptions: Reflections from a Society for Psychological Anthropology roundtable
- A Politics of numbers? Digital registration in Kenya’s experiments with universal health coverage