Blog

Posted on

Human Aspects of Cyber-crime and Online Fraud Summer school

Post by Melanie Wilson (2018 Cohort)

This summer school was presented by Canterbury University in collaboration with the Leicester Castle Business School of De Montfort University, Leicester.

I attended on the Monday & Tuesday of this three-day event, as I had a family commitment on the Wednesday.

The presentations on the Monday were specifically addressing ideas around cyber-crime, social engineering and fraud. These are particularly relevant to me as my PhD is around increasing children’s abilities to identify and resist activities and approaches whilst online. I am addressing these from the perspective of enabling children to recognise attempts from others to engage in social engineering and to have the confidence and personal autonomy to reject anything they feel uncomfortable with, and to seek help where it is needed.

I am working with the Northamptonshire Police cyber-crime team on this and as such have a valuable insight into the challenges they have seen children facing as well as my own perspective as a psychotherapist working with children.

The summer school was the first of its kind to be run at Canterbury University and was led with great enthusiasm and skill by Jason Nurse. The summer school involved people from academia, social enterprise and industry which allowed a large variety of input and ideas to be expressed. Jason was skilful in accommodating discussions within the topics and I found that this approach, rather than the “talked at” approach, was very beneficial. Cyber-crime is a fast-growing field and the traditional approaches of academic study, which often take years to complete, are at risk of being overtaken as both technology and its associated exploitation by criminals proceeds at a rate far outstripping the slower traditional progress of academic work.

I feel this pace change was reflected well in the way the summer school was run. Some of the input could have been improved with more industry input to increase the pace and knowledge of the current challenges further, but I believe that Jason is aware of this and plans to address this in future events.

The first session explored the basics of cyber-crime reflecting on what forms it can take and highlighting how insidious it can be. It addressed the aspects of this area of criminology that are rapidly expanding and exploiting the tools that are available to enhance crime via technological means. One big take-home from this introduction and discussion was how fast this is developing in criminal circles where there are few restrictions and great financial gains to be made. This is at odds with the crime fighters and honest technological industry where there are checks and balances to be met in all circumstances which often results in a slower response which the criminal can exploit.

The Cyber Protect and Prevent Officer for Kent then gave us her perspective on how cyber-crime was affecting policing and the tensions between businesses which often wanted  just to solve the issue and move on with business as usual and the desire to pursue cyber-crime as a crime with ramifications for the criminals.

The final session of the day looked at profiling cyber criminals and looking at how these criminals might be led into a perpetrator role. This is particularly relevant to my work because vulnerability leads to both perpetrator and victim activities and the two routes often share common factors.

At the end of the sessions we arranged to meet after dinner in a pub in Canterbury. Doing so was valuable as it provided a relaxed atmosphere in which to talk further with other attendees both about their work and that of others and gave a great deal of insight into the varying field that are involved with this ever-expanding and important field.

Tuesday morning looked at how cyber-crime is often underlined by psychological methodologies that criminals have learnt to use in order to perpetrate their crimes. We explored how social engineering uses a number of methods to elicit cooperation from people, utilising their vulnerabilities and often just their normal desires to help others and be nice. Again, this is an area that I focus on a great deal and feel addressing our ability to say “no” is fundamentally decreasing individuals’ vulnerability to such tactics. There is a noticeable difference in the psychological mechanisms that criminals exploit ruthlessly and the non-criminals’ tendency to trust.

The afternoon sessions addressed the cost to businesses from cyber-crime.  It was led by Edward Cartwright from De Montfort University and Anna Cartwright from Coventry University. It addressed business vulnerabilities and the attacks that businesses face daily, and the routes into the enterprise which are often indirect. The conflicts of security and running businesses were highlighted and discussed. The reality of end users often not rejecting companies following a breach and whether reputational damage is as damaging as is often thought.  In Anna’s session we looked at the financial motivation for attacks and at what level the attacks and demands became profitable for criminals.

Finally, we looked at the problems and advantages in cyber security that Small and Medium Sized Enterprises face, addressing the challenges of this sector where often there are just a few individuals trying to complete multiple roles.

This summer school was fun. Where a learning experience is fun, lively and open to discussion, I feel far more is gained than from a situation where there is just one voice with very little interaction.

It greatly benefited from a range of perspectives and allowing those to be expressed and discussed. I feel everyone learned something from the variety and range of participants at the event and very much look forward to taking part next year.

Posted on

Creativity Sandpit with Imperial, Cambridge, and Nottingham

This past May, 28 PhD students from three CDTs based at Imperial, Cambridge and Nottingham participated in a creativity sandpit.

Post by Natalie Leesakul and Shazmin Majid (2018 Cohort)

One of the perks of being Horizon CDT students is the connection with the network of centres for doctoral training led by the Engineering and Physical Sciences Research Council (EPSRC). Through this connection, Shazmin and I took part in a 3-day creativity sandpit workshop organized by Imperial HiPEDS in collaboration with Nottingham Horizon and Cambridge Sensors CDTs. This workshop was quite a unique summer school experience as it was designed for the students to work in a multidisciplinary environment to solve real world industry based problems using the Creative Problem Solving (CPS) framework to help generate innovative ideas. Industry professionals from companies such as a Formula 1 racing team were also in attendance.

On the first day, the workshop started off with various ice-breaking activities, providing the space for the students and industry professionals to get familiarized with each other and learn about others’ backgrounds. Fortunately, as part of Horizon CDT training, we are prepared to do an elevator speech on the spot; hence, giving a quick snapshot of our PhD research was not too difficult. However, the challenging part was discussing the research in more detail with others who have very different backgrounds and skillsets. While Shazmin and I have a background in Psychology and Law respectively, majority of the students are engineers and computer scientists. It was certainly refreshing to hear from a different perspective. Some of the questions and suggestions we got were something that we have not thought of before. Certainly, one of the crucial skills that we have taken away from this experience is the ability to simplify our PhD projects into something that a mixed audience can understand while maintaining engagement in the discussion. This ability to simplify our work in the wild helped us to connect with our own work and really hone in what we are trying to achieve in terms of aims and objectives.

On the second day, after the ice-breaking activities, we went on to learn about CPS and how it can be applied to different projects from the industry to academic research projects. This framework comes in quite handy for doctoral students as our aim is to find the research gap and produce innovative and original ideas to that field of research. This framework is applicable throughout the whole PhD, from the time when we have writer’s block to the time when we have a lot of ideas and not knowing what to focus on. CPS can help in generating the innovative ideas and also eliminating the ‘not-so-good’ ones; it introduces 4 stages of idea developing: clarify, ideate, develop, and implement. During this activity, we were encouraged to discuss with others and to think about our preferences in the process based on our skills. Following the CPS group work, the next activity was to form a team and explore the real-world challenges provided by the industry ranging from Formula 1 racing to drone solution to ad-tech. This was when we needed to put the learned skills together – maintaining a good rapport in a team consisting of people from different backgrounds and applying the CPS framework.

Natalie’s Experience

For the second day activity, I chose a Formula 1 (F1) racing challenge on how to best make use of the existing data in order to gain more fan base. The problem we received was that F1 fan base started to become stagnant, if not declining. Hence, the goal of this challenge was how to attract the new demographics while maintaining the current fan base.

In term of applying the CPS framework to the sandpit challenge, I’ve learned that the stage of clarifying and ideating was my strongest suit. Coming from the business and law background, I’m very comfortable with unpacking the problem, identifying facts, and generating solutions at the abstract level. In contrast, developing technical solution is not part of my practice. Hence, when forming a group to work on the challenge, I looked for others who had the technical expertise. As mentioned before, working in a multidisciplinary team was certainly a challenge on its own. My team consisted of 5 people – our team dynamic ranged from law to marketing to engineering. Certainly, we had a significant competitive advantage over the other teams given the number of people we had on the team and the wide range of expertise. However, the brainstorming session got very messy and unorganized which was problematic as we had to work in a limited amount of time. We started off with shouting out different ideas. Then, we had to choose the most promising idea to further develop which we struggled with quite a bit at this stage.

Thus, this was when we applied the CPS framework. We revisited the challenge objective to truly understand what was the problem we were trying to solve. From that, we looked at the ideas and started crossing out the ones that did not quite answer the question. Once we narrowed down the ideas to a few that we liked, we took on the big picture approach by bringing different ideas under one main solution – starting a promotional campaign for industry wide with the particular racing team as the major driver of the campaign. We arrived to “Feel the data” campaign which was designed to introduce F1 to a new market segment by playing on the FOMO (fear of missing out) trend. We wanted to create the hype around F1 so the idea was to organize a pop-up event at major attraction in the city and then rotate the event to different cities after a few weeks. The pop-up site would include various activities such as driving simulator, pit stop tire changing, and impersonation of driver’s personal engineer. The activities would be designed based on the data of the drivers and their teams in order to create more immersive experience and also incentivize the diehard fans to join the event given that they would gain more insights into their favourite teams and drivers. There would also be a rewards app for the fans to collect points from visiting different sites to win freebies and to vote for the next location of the pop-up site. We hoped this campaign would increase the fan base as it would expose F1 sport to the new demographics and help the crowd understand this sport better.

I found the creative sandpit challenge quite similar to the PhD journey in a nutshell. First, I needed to understand the problem and the root cause of it. Second, I had to explore various solutions in order to find the most optimal one. Third, it was all about how to convey the findings to a wide range of audience. Similarly, a PhD is about understanding the problem, filling in the knowledge gap, and convincing the community of why this project is worth pursuing. At Horizon, we are trained to go broad and expand our research objectives then strategically narrow down the scope to the specific. With that, the ultimate goal is to provide creative and innovative approach to tackle the challenge – just like the industry R&D projects!

Shazmin’s experience

As part of the challenge we received a short brief on the types of data that the racing team were able to show viewers which ranged from car algorithmic outputs to physiological measures of the driver. However, the company were reluctant to deliberate further about the physiological measures of the driver based on the ethical and legal challenges of showing this data on air. After the brief, all members of the summer school were asked to choose the industry challenge they wanted to tackle and then further split into specific teams which tackled the problem from a specific angle. Naturally, as my PhD looks into health markers, I gravitated towards exploring driver physiological data. We ended up being a small team of three, one other student who also had a background in health markers and another more traditional computer science PhD student.

We had half a day to explore the problem with the ability to ask F1 representatives questions, with the plan to pitch our idea to industry specialists in the morning the next day. We brainstormed our ideas using post-it notes and jointly decided which ideas were most feasible. We came to the decision that showing the following types of physiological metrics of the driver could be of interest to fans: heat levels, hydration, and heart rate. We were told that showing specific metrics would result in an ethical and legal problem so instead of showing specific numbers, we thought we could visually show the data in a less identifiable way. For example for hydration levels, we could have an icon of a water bottle with a categorical (potentially 3 categories) indicator of level of fullness. The company showed interest in our initial idea, especially the Director of Engineering who further formulated that our idea could potentially fit into a database that they are trying to make progress with specifically targeted at the Young Driver Programme. We developed our idea further with this information and prepared our pitch in the morning. The “quantified self” is a term that I learned during my PLP which describes the cultural phenomena of self-knowledge through technological self-tracking. Based on my knowledge of this, I was the one to open the pitch by describing this and asking the audience how many people were wearing some sort of wearable tracking device (a large proportion was!) to describe how we as a generation were gravitating towards becoming the quantified self and how our idea was following this trend.

We received very positive feedback from industry experts on our pitch. I have been in touch with the Director of Engineering since the summer school and hope to arrange a visit to the company in the near future.

Reflection on event

When we were practising how to use CPS, we had to critically reflect on our strengths and weaknesses. Our experiences of group work with Horizon helped us understanding our default behaviours, this allows us to improve ourselves and know what skills that we still lack of and how others can help us out. Because at the end of the day, even though a PhD may feel like a journey of one, in reality, we have a lot of support and it is important to know who and when we should ask for the support. Thus, when forming a team, we were encouraged to look for those who had the skills that we lack to ensure that the team consisted of various skill sets so that everyone had a unique contribution to the team. It was a little stressful at first as we only had the afternoon and the next morning to complete the assignment before presenting it to the industry. Although we were able to generate a lot of ideas, we spent a decent amount of time dwelling on the initial ones that each one of us came up with. This was when the CPS played a big role in helping us filtering out different ideas in order to generate the most optimal solution as a team which we could then develop it further. We were under pressure, however, we had to learn how to work together and liaise with each other in spite of our differences in expertise. Building a good team was definitely a challenge on its own, but it was a very valuable experience. We, as multidisciplinary PhD students, are constantly working with people from different disciplines which it is not always easy. By having more practice working in a diverse team, we have learned to better clarify and communicate our ideas which we can continue to apply throughout our doctoral process.

Last but not least, one of the unplanned bonding activities we (both the students and people from the industry) did together was chasing down the peacocks for pictures – good times!

 

 

 

Posted on

Summer School on The Human Aspects of Cyber-crime and Online Fraud

post by Neeshé Khan (2018 cohort)

This Summer School and workshop was hosted by the Kent Interdisciplinary Research Centre in Cyber Security (KirCCS) and School of Computing at the University Kent, the Institute of Applied Economics and Social Value at De Montfort University and International Association for Research in Economic Psychology (IAREP). It took place at Canterbury between 15th to the 17th of July lead by Dr Jason RC Nurse.

As I’m working on accidental insider threat within cybersecurity to examine human factors that trigger this threat, I was keen to attend this event as it would provide an overview of the issues around social engineering and associated forms of crime in the virtual and physical world – broadly sitting within my own research interests. Recent media has highlighted many cases where fraud and cybercrime have resulted from a mixture of social engineering and human vulnerabilities to gain undesirable outcomes including encryption of data to hold at ransom on an organisational and individual level. Whilst there is literature on cyber-psychology linking to malicious insiders and cybercriminals, there is little literature available that takes an interdisciplinary approach to tackle this problem, especially examining this from a psychological, economics, and cybercrime perspective. So the aim of the summer school was to introduce these disciplines and their relevance to be able to better understand this challenge. This was particularly important to me as I believe that all the global challenges being faced by the world today require collective interdisciplinary action to resolve them.

One of the highlights of attending this school was meeting a diverse range of about 40 attendees, which included different career stages within academia, people from industry, diversity in research being pursued and interests as well as diversity in ethnicity, age and academic backgrounds. Whilst most of the projects weren’t similar, it was still cohesive in terms of disciplines and understanding of cybersecurity. This allowed a space where I shared and received ideas and insights about this issue over workshop discussions and group dinners. Presentations were a mixture of academics from various universities including the University of Bristol and the University of Cambridge as well as law enforcement. I hope my notes below are of interest to anyone from psychology, economics, and cybersecurity fields taking an interdisciplinary approach to exploring cybercriminal and victim behaviour and traits, especially those involving malicious or intentional insiders.

Discussions included how the definition of cybercrime is hard to settle on as it means many different things for researchers, businesses, and individual users. Technology evolving has meant that many of the devices aren’t seen to be within the remit of cybercrime by the general public, for example, cybercrimes that happen through mobile phones or smart wearable devices are seen to be separate from the same crimes that occur through a desktop or a laptop. A way of looking at cybercrime is by categorising attacks that are ‘computer dependent’ (DoD, ransomware, etc) and those that are ‘computer-enabled’ (online fraud, phishing, etc). This can also be categorised through Crime in Technology, Crime against Technology, and Crime through Technology.

Cybercrime is a big challenge being faced by society and whilst there are numerous different types of cybercrimes, currently, popular ones include social engineering, online harassment, identity-related, hacking, and denial of service (DoS) and/or information. Social engineering and phishing attacks are the biggest attacks that are currently taking place. Cybercriminals are getting better at replicating official documents (less spelling mistakes, logos, branding, etc) and use a mixture of techniques that include misdirection and pressurising recipients to take action. Most classifications of cybercriminals are through using early techniques developed by the FBI’s human behaviour department and include the Dark Triad and OCEAN personality traits. Techniques used to investigate crimes in real life such as ‘method of operation’ (MO) and copycats seem to transfer relevantly well to cybercrime investigations.

Law enforcement believes that in their experience there is a strong link between gender, age, and mental ability and cybercriminals. Children test out their coding skills from lessons in schools to attack websites or online gaming platforms. There also appeared to be a strong link between online gaming habits, mental disorders such as ADHD and hacking. Whilst there are more cybercrimes reported to the police than crimes in the physical world, the task force is still suited for ‘boots on the ground’ than cybercrime. All individual reports of cybercrime are done through Action Fraud and involved cybercrimes that came from someone they knew such as disgruntled ex-partners. Threats included a wide spectrum but the most popular ones included fraud, abuse, blackmail, harassment, and defamation of character.

In psychology, cybercriminals are classified in similar ways to that of criminal profiling in real-world crimes. There is also interest in exploring victim traits since individuals who are a victim to an online attack are likely to be a victim to another attack in the future. When looking at cybercriminal profiling psychological and emotional states are key factors. Various online forums are researched to create a cybercriminal’s profile mainly through the following categorization: language used, attitudes towards work (for example in the case of a malicious insider threat), family characteristics, criminal history, aggressiveness, and social skill problems including integrity and historical background. However, this is challenging as personality traits and characteristics are easier to change online especially for narcissistic personality traits. However there is never a 100% certainty of creating a psychological profile of a cybercriminal, with very little research and involves stereotypical profiles such as ‘white, male, geek, like maths, spends a lot of time alone, plays online games, anti-social traits, etc. Often personality traits associated with ‘openness’ of individuals links to individuals being susceptible online to phishing and other scams.

Most important models of profiling are ‘inductive’ and ‘deductive’ criminal profiling. Inductive is using existing data to identify patterns and deductive is starting from the evidence and building up to the profile (deductive cybercriminal profile model). Deductive method is very popular and is designed by Nykodym et al 2005 but there’s also geographical profiling (Canter and Hammond 2003) that is starting to become more popular as a result of social engineering attacks. Economists are applying ‘willingness to pay’ (WTP) and ‘willingness to accept’ (WTA) models and game theory to ransomware attacks.

Overall, the summer school provided a great platform to create a new network, reaffirmed my understanding of the current approaches being adopted, offered insights to some of the research being conducted, and provided a platform to promote my research. 

Posted on

Second EPSRC Impact award for Richard!

Richard Ramchurn (2015 cohort) won the EPSRC “Telling Tales of Engagement” 2018 Award. This is the second year running that Richard has secured this award which will give up to £10,000 of funding to maximise the reach and impact of his research. The “Telling Tales of Engagement” 2017 Award funded Richard’s mobile touring cinema which travelled the UK in 2018 and 2019.

Post by Richard Ramchurn (2015 Cohort)

The MOMENT is a brain controlled film which has been touring since 2018. It has been primarily screened in a converted cinema caravan which has allowed it to travel outside of the cinema circuit. 

The research project adopts a performance-led research in the wild methodology, through which impact though public performance is an inherent factor – the default practice behind this methodology is that real-world artefacts are professionally made, and performed for the public, as the mode of studying their design and implications.

The MOMENT has had over 300 public screenings across the world, at FACT, Lakeside Arts, Sheffield DOC/FEST, Ars Electronica, Kendal Calling, Blue Dot, Arts By The Sea, Leeds, Geneva and Reykjavik International Film Festivals, and Aesthetica. We were also invited to  international events: SPARK British Council event in Hong Kong, Brain Film Festival in Barcelona; Riverside Film Festival in Padua. Substantial work has been made to make the touring of the caravan self-sustainable and there are plans to further exhibit throughout 2019.

I have been invited to present my research and engage in panel discussions with the film and computer game industry at: Creative England’s Proconnect conference; Continue Conference; Picturehouse London with B3 Media; Broadway Cinema Nottingham; Geneva and Reykjavik International Film Festival; Sheffield DOC/FEST and FACT Liverpool. Organisations and individuals have since contacted me to ask to work on upcoming projects, to collaborate, and screen the film at their festivals.

The project has now secured funding to screen Live Score performances in 2020. In these performances musicians Hallvarður Ásgeirsson and Scrubber Fox perform the score to The MOMENT as the film is created live from the brain data of an audience member.  The performances are followed by a Q and A with the musicians and myself. We previewed the Live Score in Reykjavík and Nottingham last year to engaged audiences. We are now planning a UK tour for the summer of 2020.

A live score accompaniment for an interactive film is an unique, timely, and relevant proposition, capable of capturing both the public’s imagination and commercial interest. Our tour offers an alternative engagement proposition: creative, interactive live performances that large audiences can experience collectively at local arts venues. This model fits with the  industries move towards marketing cinema as a live experience, both through streaming theatre and music performances to screening venues (e.g. NT Live), and by creating immersive environments in which screenings take place (e.g. Secret Cinema).

The Live Score has the potential to reach larger audiences, including both a wider film industry audience, and members of the public who may not usually engage in academic research. The performances are set to be an exciting and dynamic way to share my research.

Posted on

The Unbanked and Poverty: Predicting area-level socio-economic vulnerability from M-Money transactions

posted by Gregor Engelmann (2014 cohort)

Emerging economies around the world are often characterised by governments and institutions struggling to keep key demographic data streams up to date. The combination of mass call detail records (CDR) data with machine learning has recently been proposed as a way to obtain this data without the expense required by traditional census and household survey methods. The paper is based on the exploratory analysis of CDR and mobile payment (M-Money) data for Dar Es Salaam, Tanzania, carried out as part of the PhD Research. It forms the basis for a chapter on the potential of mobile phone generated data to supplement traditional surveying methods for socio-economic analysis in urban and peri-urban areas.

The paper was written by N/LAB PhD student Gregor Engelmann in collaboration with the PhD supervisor James Goulding and the N/LAB Data Science Lead Gavin Smith. Moving from an initial analysis of the M-Money data and overcoming both technical and general paper writing hurdles to submitting and ultimately presenting the paper was a prolonged process that took nearly a year.

The major technical hurdle was adding a geographical component to, and cleaning the M-Money data. The dataset is comprised of anonymised logs of mobile payment transactions generated through mobile financial services usage of a Tanzanian Mobile Network Operator. While CDR data has been used in a wide number of areas from epidemiology to mobility and urban analysis with more than 900 papers using CDR data published in the last decade alone, the research on M-Money data is very scarce due to the high difficulty of obtaining such data sets from Mobile Network providers. Organisational hurdles included identifying an appropriate journal or conference venue, making sure that both abstract and paper are ready by the relevant deadlines, and raising enough funding for conference travel. The submission to the initially identified conference was ultimately delayed as the paper required more work before submission to IEEE Big Data in early summer 2018. Being based in the same lab made collaborating on the paper easier as we could have regular in-person meetings to review the paper. In addition to in-person meetings, we extensively used Overleaf, an online LaTeX platform, which allows for both collaborative writing and editing. LaTeX has the advantage of being able to handle equations, figures, indices etc. better than traditional writing software such as Word. LaTex (and by extension BibTex) also offers more effective bibliography and reference management while allowing for changes in formatting and style with a few simple lines of code.

Gregor Engelmann presenting at the IEEE Big Data 2018 conference in Seattle, WA

The reviewers’ response to the paper was positive and it was ultimately accepted as one of 98 regular papers (acceptance rate 18.9%) without the need for major changes. In addition to funding from the Business School and CDT, the paper was successful in securing one of the limited Student Travel Awards offered by the conference organisers to cover travel and registration costs to Seattle. The paper was presented to a mixture of academic and practitioner audiences as part of the Big Data Applications: Society track at the IEEE Big Data 2018 conference in Seattle, WS in December 2018.

N/LAB PhD student Gregor Engelmann’s paper: The Unbanked and Poverty: Predicting area-level socio-economic vulnerability from M-Money transactions, accepted and presented as a regular paper into the IEEE Big Data conference 2018 is available online via the publisher (paywall) https://ieeexplore.ieee.org/abstract/document/8622268/figures#figures and Nottingham University eprints http://eprints.nottingham.ac.uk/55720/

Posted on

IEEE FG 2018 paper & conference experience

Post by Siyang Song (2016 Cohort)

Venue: FG 2018, Xi’an, China https://fg2018.cse.sc.edu/

Paper Title: Human Behaviour-based Automatic Depression Analysis using Hand-crafted Statistics and Deep Learned Spectral Features

The work presented in this paper is the extension of my CDT PLP module. During the PLP, we found that current clinical standards for depression assessment are subjective, and requires extensive participation from experienced psychologists. Therefore, my supervisor (Dr. Michel Valstar) and I explored a video-based automatic depression analysis approach, and find our approach beat state of the art systems in a dataset. As a result, we decided to write a paper about this approach.

This bulk of the paper was written by me. Since I didn’t have much experience about writing paper at that point, my supervisor helped and taught me about how to organize a paper, as well as how to implement ablation studies to show the strengths of the proposed approach. In addition, my external partner (Prof. Linlin Shen) also helped me to check the typos and languages. After, several rounds improvement, we eventually submitted the paper to FG conference which focus on the automatic face and gesture analysis.

Three months after the submission, we received the reviewers comments, where two of them gave ‘weak accept’ and two of them gave ‘borderline’. In the rebuttal, we addressed issues one by one. For some real drawbacks, we made some changes. Also, some comments were gave due to the misunderstanding of the paper. To deal with them, we detailed explained in the rebuttal.

Posters:

This study is the basis of my PhD study. Based on this work, while we continuously developing a better model for practical use, we also looking to other temporal modeling methods.

Building several models for not only depression analysis, but also personality traits and anxiety analysis.

https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8373825

–originally posted on Siyang’s blog

Posted on

VRtefacts Outreach at Derby Museum & Art Gallery

Post by Joseph Hubbard-Bailey (2016 Cohort)

The VRtefVRacts project provides museum and gallery visitors with the opportunity to hold and explore exhibit objects which they would otherwise just look at behind a literal or figurative red rope. Throughout the day, visitors from around the museum were invited to come and put on a VR headset, interact with some 3D-printed VR-augmented models of artefacts, and share their own story or commentary about the objects as they handled them. They then moved into another room for a short interview about the experience, allowing for the next participant to get started with the VR. While previous outreach events I’ve done have felt engaging and productive, none have been as interactive as this VRtefacts trial; others mostly involved having conversations across tables, and the distance and dynamic between researcher and participant felt similar to a campus-based study scenario. Due to the nature of this event, with participants engaging physically and narratively during the session, members of the public seemed much more a part of what was going on, as opposed to passive spectators.

For the visitors who chose to participate in the VRtefacts project, the experience served as both a novel sort of ‘exhibit’ in itself and a novel way to access preexisting materials in the museum’s collection. The latter seemed of particular value in the case of visitors who lived locally and so visited the museum often, offering an unexpected new level of access to familiar objects. The opportunity to contribute or “donate” a story as part of the VRtefacts experience may also have been particularly appealing to those who visit regularly and were keen to ‘give back’ to the museum. Several visitors did fall into this category of ‘regulars’, but there were also plenty of people who were passing through and popped in to pass the time. Visitors across both of these groups commented about how the decision to work with VRtefacts reflected well on Derby Museum, showing its openness to new ideas and resistance to stagnate. For those who were visiting the museum in groups, engaging with the VRtefacts exhibit seemed to provide a great source of interest and conversation, as they emerged and compared experiences. The fact that the corresponding artefacts themselves were available in the museum’s collection also meant that there was a comfortable transition back into the rest of the exhibit, as people could go and find the ‘real thing’ they had just encountered virtually.

Before I left the museum for the day, I sat down on the duct-taped-still chair and had my hairdo sabotaged by the VR headset so that I could have a go at the VRtefacts experience myself. I chose and inspected a small intricate model of a giant jet engine, turning it over and fumbling around the prickly detail of the gaskets while I tried to think of something clever to say for the camera. It reminded me of a frighteningly massive aircraft housed at the RAF Museum in Hendon, where I’d been for relentless school trips as a child due to its proximity to school grounds. I remember cowering through the awful hangar where the scary plane’s wings were so expansive that you had no option but to walk underneath them if you wanted to get out. While this wasn’t a pleasant experience, I think the physicality of being below the Vulcan — which I now know was not just a war plane, but a strategic nuclear bomber — came to mind during VRtefacts because it was a similar example of the power of perspective.

Image credit: Kenneth Griffiths (Ascension Island, 1982)

When an object is in a glass case or on a screen or behind a rope, I think we often instinctively revert to what I can only describe as a ‘flat’ perspective on it. We might press our noses to the glass as children to try and get a closer look, but the glass fogs up and we get told off, so eventually our curiosity wanes and we take a respectful cursory look instead. What this tired perspective gives us is often limited to two-dimensional factual information about the object of interest, without the weight and contour  and color of the object’s life. I’m very glad I decided to have a go with the VRtefacts pilot myself before I left the event, because it made me aware of how cowering under the expanse of the Vulcan’s wings taught me more about the gravity of war than any of my history lessons had. There is a narrative power in an artefact’s physicality which cannot be accessed by simply looking at it — the VRtefacts project has the potential to provide that physicality in a way that protects the original object, which needn’t even be on the same continent as it’s VR counterpart.

Beyond the benefit this technology could offer in enhancing the habitual gallery-goer’s usual experience, there is also potential benefit to those who aren’t so familiar and comfortable with these venues. Having come from a family who didn’t really go to museums or galleries, I still feel quite awkward and out of place in these spaces at times. I don’t think it’s much of a leap to suggest that projects like VRtefacts — which offer more diverse ways of accessing meaning in historical and art objects — have the potential to make galleries and museums not only more engaging for visitors, but more accessible to a diverse range of visitors.

Thanks to Jocelyn Spence and the rest of the VRtefacts team for letting me join in for the day!

VRtefacts is a pilot project developed within the European Union’s Horizon 2020 research and innovation programme under grant agreement No 727040, GIFT: Meaningful Personalization of Hybrid Virtual Museum Experiences Through Gifting and Appropriation.

–originally posted on Joe’s blog

Posted on

Programming in Unity at the DEN Summer School

Post by Joe Strickland (2017 Cohort)

Back in the summer of 2018 I attended the DEN summer school in Bournemouth. One of the big draws of the summer school for me was the programming in Unity course that was being offered. Having come from a psychology background, I had no programming knowledge but it was becoming clearer and clearer that this was going to be something that held me back during my PhD, especially when it came to prototyping ideas for experiences. The course itself was pretty good, we ran through several different elements of using Unity including the basics of building scenes, game object physics, and exporting our scene onto a smartphone and viewing it with a cardboard header as a VR experience. We also started using Vuforia and making basic AR content. This workshop gave me a good basic understanding of Unity, but more importantly, it showed that what I wanted to learn and eventually make was well within my grasp. This was very important for motivating me to carry on learning how to build Unity experiences, as well as code in general.

Once the summer was over, my supervisors and I sat down and started discussing short-term goals to get me learning everything I’d need to learn in order to build interactive AR experiences myself. The first of these goals was to learn Python and C# in order to understand the logic of coding and be able to write my own Unity scripts to control different elements of the software. My supervisor ran me through all the basics in Unity that I might need for the specific things I was going to make, a welcome refresher after the summer school course, and I was sent off to learn my languages. Personally, I found Python quite easy to learn. The logic of the language made sense to me and the online resource I had been recommended taught it in a very hands on a practical way, with many small assignments to try out new coding knowledge and to keep old knowledge fresh and reinforced in your memory. Also, the course was broken up into bite size chunks and I found doing a lesson a day over the course of a month a very productive way of learning this language.

C# scripting was a little harder for me to grasp. I don’t know whether it was the difference between it and Python throwing me off or knowing that having to learn this was going to be more important for my PhD, but it took a lot more to try and figure out what I was doing with it. Learning this was done through some of the Unity provided tutorials, as well as other user generated tutorials on YouTube. I was also learning how to use Unity to specifically make the first short term goal project I had been assigned; making videos plays in Unity. The Unity video player isn’t completely user friendly and it took a lot of trial and error and searching Unity message boards and community sites to find out how to get it to work in the way I wanted it to. Having got it to work I moved on to controlling it a bit more and building an experience where the audience can press keys to trigger the playing of different video clips. I crafted a game object for each video clip we had and had them generated and destroyed whenever we needed that video playing, depending on the input of the audience. What I ended up with was a functional interactive film about a man trying to find his heart medication, where the audience could decide whether he moved left, right, or had a heart attack at various points in the film. When I showed my supervisors they liked it but found how I had made the film incredibly inefficient, so they tasked me with remaking it so that different videos played on the same game object and not on different ones. This next step proved challenging but eventually I managed to write a functioning Unity script which changed the state of the game object and, once a game object was of a certain state, it would play different videos with different audience responses. It would then change its stage again to allow the experience to progress. This experience pleased my supervisors, but they didn’t like how making decisions at the wrong times messed the game up, so I had to add delays into the script that stopped audiences making decisions at the wrong points in the experience. Fortunately, this wasn’t to difficult to do, although trying to use time as a function while coding with the video player in mind did prove confusing.

I was also asked to build a restaurant scene and fill it with moving virtual characters, but this was very similar to the summer school exercises and the Unity developer tutorials so this didn’t prove too tricky. Characters were downloaded from Adobe Maximo, so came with animation cycles attached and a few YouTube tutorials later I had people looking around and being furious at virtual restaurant tables.

Finally, I was asked to build an AR tester experience. I had to place a virtual character, like those from the restaurant scene, into a real world environment and have them occluded by a real world object, specifically sitting behind and hidden by a real world table. This is something that is surprisingly hard to find official Unity information for. There is lots of help for tracking markers and placing AR content in the real world but not so much for having that content blocked by real objects. I eventually found a YouTube tutorial which addressed a similar problem in a way which allowed me to figure out how to solve my own. They showed how there was a depth occluder material that you could use to create invisible game objects that would block the audience’s vision of the virtual content. Creating a cube the size of a table top and placing it over the lap of my sitting virtual character, then using a placemat as a tracking marker in the real world to position the avatar behind a real table allowed the virtual character to appear as if they were sitting in the real world. The illusion was particularly impressive when the character moved and there arms would disappear and reappear below and above the line of the back of the table. See attached photo for a snapshot of the experience.

If I had any advice to any other researchers looking to get into creating XR experiences, or even just learning to code, it is there’s no time like the present to start learning. There are plenty of great resources online for free that go through everything you’ll need to know step by step, while also allowing you to navigate through lessons to learn the specific things that you need for whatever project you might be working on. Though getting an understanding of the basics is fundamental you can pick or choose what of the more specific stuff to learn to suit your needs fairly easily. Also, just like any skill, you’ll need to keep practising. Find some little challenges to work towards, like I had set out for me. There were a few times I’d not focus on coding for a few weeks and then notice that I had forgotten something I definitely knew before and had to go back over previous lessons or code that I had written to find it. Don’t fall for this like I did, keep it up at a steady pace and you’ll be writing code in no time.