Publishing Conference Paper – A valuable experience

post by Keerthy Kusumam (2017 cohort)

I published my conference paper, ”Unsupervised face manipulation via hallucination” in the International Conference on Pattern Recognition. The paper focused on a generative computer vision method to alter the pose and expression of a facial image in an unsupervised manner. I spent several months conducting experiments, analyzing the results, and discussing our findings. I received valuable feedback from my supervisors, which helped us to improve the quality of our work.

After the initial submission, I received comments from reviewers who provided suggestions for revisions. I took these comments into consideration and worked hard to make the necessary changes. This process was challenging as well as rewarding in the end. The paper was accepted to be presented as an oral presentation at the conference. The reception of our paper was quite positive and received several questions and comments from attendees. This was a valuable opportunity for me to network and receive feedback.

The motivation behind writing my conference paper was to explore the current state of face manipulation technology and to identify potential future directions for research in this area. As a 2nd year PhD student, I wanted to demonstrate my knowledge and understanding of the field, as well as contribute to the use of generative AI in face manipulation tasks. My main objective was to present a comprehensive overview of the current state of the field and to identify areas that could benefit from further research, especially behavioural monitoring in affective computing. In these areas, data is limited, and the use of generative AI can synthesize realistic data for further analysis.

I approached the research process by first conducting a thorough literature review to understand the current state of face manipulation technology and to identify gaps in the current research. I then used various research methods, such as conducting interviews with experts in the field and collecting data from various sources, such as academic journals, conference proceedings, and online forums. I also conducted experiments to validate some of my findings.

My key findings showed that the field of face manipulation is rapidly advancing and that there are many promising areas for future research. I discovered that there are various technical and ethical challenges that must be addressed to ensure that face manipulation technology is used responsibly. These findings impacted my original objectives by reinforcing the need for further research in this area and by highlighting the importance of responsible development and use of face manipulation technology.

I presented my research in the conference paper using a clear and concise writing style, and by using various visual aids, such as diagrams, graphs, and tables, to help illustrate my points. I also used a logical structure, with clear introductions, conclusions, and recommendations, to ensure that my ideas were easily understood by the conference audience. I also made sure to clearly state my findings and to provide context for each of the points I was making. The contributions were accompanied by experimental evidence.

One of the main challenges I faced while writing the conference paper was ensuring that my research was comprehensive and up-to-date. To overcome this, I made sure to regularly consult with my supervisors and to gather feedback from my peers. I also took the time to review relevant literature and to stay informed about the latest developments in the field.

As a result of writing the conference paper, my understanding of the topic of generative computer vision methods has deepened, and I have gained a better appreciation for the complex and rapidly evolving nature of this field. I have also gained a deeper understanding of the technical and ethical challenges that must be addressed to ensure responsible development and use of face manipulation technology.

The feedback I received from the conference audience was quite positive. Many attendees commented on the comprehensiveness of the research. Some attendees suggested areas for further research, which I have since incorporated into my future plans, especially in using this method to anonymize face datasets.

Overall, my conference paper on unsupervised face manipulation via hallucination was a valuable experience that allowed me to contribute to the field of generative computer vision and gain valuable insights into the complex nature of this field. The research process allowed me to deepen my understanding of the technical and ethical challenges that must be addressed in order to ensure responsible development and use of face manipulation technology.

Rising Spring – Awakening and Arrivals

post by Pavlos Panagiotidis (2022 cohort)

The Rising Spring: Awakening and Arrivals interactive installation is Makers of Imaginary Worlds (MOIW) latest project and will be presented at Lakeside Arts until May 2023. It is a compelling example of how multisensory interactive art can create a fun and playful experience for children and family audiences. MOIW and Lakeside Arts are the industry partners of my Horizon CDT PhD project, where I explore mixed reality performance-making methods. My task was to identify elements of the event’s narrative design and its impact on the audience’s engagement, with the aim of exploring the potential applications of digitally enhanced narratives in mixed reality theatre production.

The inspiration for the installation was the theme of spring and the feeling of transition to the blossoming season. It was designed to evoke the feeling of rejuvenation. The designers were also inspired by the beautiful landscape of Lakeside Arts, and incorporated aspects of the natural environment into the installation. MOIW designed the installation to engage the audience’s senses beyond sight, with an additional focus on touch and sound.

One of the most exciting parts of the installation is the stepping stones, where children can jump around stone-looking fabric shapes in an imaginary lake. The stones respond to people stepping on them with sounds, allowing participants to cooperate and create small symphonies and encouraging children to work together. The installation also contains giant flowers, some of which are robotic and open when approached by participants. Matt Little, the creative technologist who supported the creation of the installation, focused on ensuring that the event was safe for children and would operate continuously with minimum service. Tom Hartman from the School of Life Sciences of the University provided some 3D anaglyph images of bugs made up of red and cyan filters. By using 3D glasses, the audience could see bugs from a unique perspective, providing more layers of unique sensory engagement.

Another interesting aspect of the installation is the use of scale. The large-scale flowers and objects create an unfamiliar and aesthetically pleasing environment for children to immerse themselves. Additionally, the installation encourages visitors to leave their own mark by sharing their spring drawings. This adds a personal touch to the event and makes visitors feel part of the experience.

A critical consideration in designing interactive installations for children is promoting agency and playfulness while finding the right balance regarding the instructions provided. Too little information may leave some participants without experiencing the full extent of the event, while too much information may strip away the joy of exploration and discovery. The Rising Spring installation strikes the right balance, allowing children to explore and discover at their own pace while still providing enough information to guide them.

Overall, the event allows children and family audiences to engage with interactive art in a fun and playful environment. It encourages exploration and discovery. As the installation evolves over time with new material and activities added, repeat visits offer surprises and discoveries for participants. “Rising Spring” at Lakeside Arts is an example of how interactive installations can offer a memorable experience for young children.

Podcast: Christine Li reflects on Placement

Christine Li, a fourth-year Horizon CDT Phd candidate, is focusing on the design and study of VR prototypes for at-home bodily training, such as performance arts or sports.

Christine’s industry placement took place at KTH Royal Institute of Technology in Stockholm at the Media Technology and Interaction Design (MID) department. Christine reflects on her time at KTH, the lessons she gained there and how it has influenced her research.

Listen to Christine’s podcast:

 

My summer school experience: Inquiry through making and playing with objects, spaces and situations

post by Daniel Swann (2021 cohort)

In early June 2022, I attended a PhD summer school entitled ‘Inquiry through making and playing with objects, spaces and situations.’ The summer school lasted for three days and was organised by Design School Kolding, although was hosted by Aalborg University at their Copenhagen campus. I was looking forward to attending the summer school: I had identified play as an important theme to my research and was also engaging with theoretical approaches to design, particularly in the way that it can inform interaction. The summer school was not specifically focused on children and young people, but I felt this would be a positive as it may allow me to reengage with my research from a different perspective—which I understood as one of the main benefits of attending a summer school as a PhD student.

On the way to Copenhagen, I went over the provided list of reading resources which were curiously broad in scope. It included seminal texts from design theory, research papers from the field of Human-Computer Interaction, experimental autoethnography, and examples of futures studies (an area which was almost entirely new to me). As well as providing the reading list in advance, the organisers asked that each participant brought an object or image that somehow embodied the focus of our research. This provided the basis for introductory conversations with the group on day one of the summer school.

After we had done our introductions, the organisers each made a short presentation that established some of the key themes and ideas of the summer school. This included the many ways we can think about play, especially in the context of making, and a broad overview of design anthropology. These presentations were important as they allowed the group to share their initial interests, doubts, or queries in relation to the central themes of the summer school. Furthermore, it provided us all with a common vocabulary with which to identify the synergies and tensions between the theoretical framework(s) and the practical activities that took place over the following days.

The first day ended with a visit to the Play Lab at the National Museum of Denmark. This innovative laboratory at the heart of the museum is home to a small team that work on reimagining the relationship between the historical assets and the public in an inherently playful way. This was highly relevant to my own research, and I found their presentation and the subsequent discussion to be highly rewarding, as the Play Lab team were engaged in academic debates around playful experiences in cultural institutions as well as the practicalities of running events and exhibitions on a day-to-day basis.

Day two of the summer school represented a deeper dive into the themes as we explored how play can be thought of conceptually and applied to a variety of practical activities. The first topic was ‘Atmospheres and Ambiances.’ Two of the organisers had transformed the seminar room that we were using entirely by changing the layout of the furniture and introducing interactive artefacts, with different lighting and ambient music added also. We were invited to explore the space in silence as a group and then discussed how these changes had altered our perceptions of the atmosphere both individually and collectively. After this, we split up into small groups and explored the campus through our senses: one member of the group would provide prompts and the other would respond through a constant stream-of-consciousness.

After playing with spaces, we then played with empirical data by writing down our experiences and then editing them in a collaborative, creative, and chaotic way. For example:

I remember becoming intrigued, inspired, interested in the elemental world, the spaces we inhabit, use, appropriate, play with
Jeg husker smagen af min farmors saftevand
Jeg husker følelsen af cykelshorts
​​I remember discussing new methods, new ways of thinking
I remember being told that I was going to have a sister.

 

This activity vividly brought to life many of the ideas discussed on the first day. In the following discussion, there was a general feeling that this playful approach to poetry had enabled an entirely different engagement with our memories than was otherwise possible. I found day two of the summer school to be sometimes challenging, and often I felt outside of my comfort zone, but it certainly provided me with many ways of thinking about the central themes of my doctoral research.

On the final day, we focused mainly on making and tinkering. This was particularly rewarding as it began as quite a simple creative task but evolved into an engaging activity that questioned how we can think about both space and time in playful ways. We ended by presenting back to the group as a whole and then discussing some of the emergent themes of the summer school in a round circle debate. It was interesting how the different backgrounds of the participants had influenced the discussions and produced original thoughts and ideas that were thoroughly interdisciplinary.

As I cycled back from the campus to central Copenhagen, I quite literally bumped into another participant who was going the same way. We decided to grab a drink and reflect on the summer school as a learning experience. Without a doubt, we enjoyed our time, particularly the fact that the PhD students came from a variety of backgrounds which ensured that our debates remained dynamic and multidisciplinary throughout. During our conversation, however, we noted that sometimes these debates produced interesting ideas that could not be fully explored due to the time pressures of the summer school.

On further reflection, I felt this was somewhat unfair as it seems to me that this is the very purpose of attending a summer school as a PhD student: to be challenged and leave with fresh ideas that one’s research can attempt to resolve. My time in Denmark not only provided me with a new network of like-minded researchers, but also allowed me to reengage with my studies from a new playful perspective.

Robot dancer interacting with children – Feng’s internship

post by Feng Zhou (2017 cohort)

The robot dancer was one of four exhibits of “Thingamabobas,” a playful, sensory experience where participants meet and interact with a circus troupe of performative hybrid mechanized sculptures crafted from sustainable and recycled materials. The installation space is a place of wonder. It draws on the absurdities of British artists Heath Robinson and Rowland Emett’s contraptions, Calder’s Circus (1930’s), automata, object theatre, puppetry, and circus acts.

The robot dancer was shown in Lakeside Arts at the University of Nottingham. It attracted many families with children to attend.

We chose the low-cost robot arm – Ned as our dancer. It is also safe working with humans. The Intel RealSense Depth Camera was used as “dancer”‘s eyes to conduct facial recognization.

The fan-shaped zone is the range of the camera. It is also the zone in which children interact with the “robot dancer.” When the camera “sees” the face of the nearest child in the green area, it will dance following the child’s face. Since the “robot dancer” has a bad eye condition, it will be hard to tell whether there is anyone around when the nearest child shows in the purple zone. Thus the “robot dancer” will try to look around and search for children. And then, when children stand farther than the purple area, the distance will be over the range of the camera’s detection. So the “robot dancer” will fall asleep. During the same time when children go through “interacting,” ” searching,” and “sleeping” areas, there will be corresponding music being played.

There are still many points that could be improved such as increasing the interacting zone by increasing amount of cameras, switching professional robotic arms to achieve more sufficient movements, and decreasing the delay of interaction.

It was an amazing experience to work with artists and children.

My job was primarily on developing the robotic interacting system base on ROS, including developing the robotic interacting mode based on distance from the users, music playing etc. From this internship, I extended my skill set by learning the ROS system, which enables high flexibility in combining multiple devices into an integrated system. This enabled me to extend my six-axis 3D printing system, which is significant for my PhD research. During the process of developing the interaction mode of the robotic arm, I had a chance to work with dancers, who interacted with the system with dancing motion, through which, I got valuable experience working with dancers who are also aimed users for my PhD research.

Paper reflection – Articulating Soma Experiences using Trajectories

post by Feng Zhou (2017 cohort)

Somaesthetics combines the term ‘soma’ with ‘aesthetics’. The concept of ‘soma’ is predicated on the interconnectedness of mind, body, emotion and social engagement, considering all to be inseparable aspects that together form an embodied, holistic subjectivity. Aesthetics here refers to the ways in which we perceive and interact with the world around us. Somaesthetics is a widely used methodology for user study, which plays a significant role in my PhD research. Researchers who have focused on the research of Somaesthetics for many years and have published a number of prominent papers from the Royal Institute of Technology Stockholm visited the Mixed Reality Lab (MRL) of the University of Nottingham (where I am based) and collaborated with researchers here to run workshops on Somaesthetics. It was an excellent chance for me to learn Somaesthetics deeply through the workshop and explore the application of this methodology to my research.

Researchers were split into four groups to explore different applications. The group I was involved in was to explore the boundaries between humans and technology. The skin is traditionally seen as being a critical boundary of the body and one way of defining the bodily self. We can see, i.e. perceive with our eyes, our “external”, fleshy body – our moving limbs and parts, and our skin as the boundary between our “external” and “internal” body –our organs, cells, muscles etc., – which we cannot see, but instead feel or imagine. However, the boundary may be considered malleable. Take the example of a prosthesis – is this a part of our body or a separate piece of technology?

We attached cloth straps to the dancer’s calf and thigh so other members of the team could control them. Participants had to imagine a limb that had a ‘mind of its own’ – an exploration of dance where a part of one’s body was separated from control. The dance experience became one of negotiating control with one’s own body. This could serve as a conceptual stand-in for novice kinaesthetic skills where one’s body is unable to do what is asked – perhaps lacking the range of motion needed. But this was beyond being simply unable to perform the controlled limb; it actually became a separate performer in its own right, creating an intriguing partnership with a part of one’s own body, and encouraging the dancer to question the boundaries of their body and soma.

As we began to dance, our bodies behaved as we expected and we were unfettered. As our group began to take control of our limbs, we lost some agency over our bodies. The external influence started to exert itself in such a way that it restrained us, or actively pulled us. We were no longer ‘at one’ with our own bodies – rather those who controlled our limbs shared control with us. Over time as we learned how to work together, that action could even be considered a part of us (at least as far as the experience goes). It should be noted that the group members pulling on the straps were a stand-in for a ‘disobedient’ prosthesis – so the notion of it becoming part of us, or perhaps beginning as part of us, separating from us and returning might be more tightly aligned to our own body than the group experience – nevertheless the group does have access and licence to control our limbs.

This workshop was one of the user studies to support our final paper. Questioning the boundaries between humans and technology also invites reflection on the boundary between ‘inside’ and ‘outside’: separated by the skin, breathing in and out, ingesting and excreting. Thinking through these boundaries allows designers to redefine them, and thus challenge not only where the soma begins and where it ends, but also where the boundaries of experience lie. This turned out to significantly support my user workshop with disabled dancers to personalise their prostheses.

My job for the final paper was mainly to describe the activity I was involved in during the workshop. This was a precious experience to learn to write a paper collaboratively with many authors. Our final paper has 14 authors from the Royal Institute of Technology Stockholm and Mixed Reality Lab. Each of us wrote a specific part of the paper on Overleaf. We also have regular meetings to discuss writing up issues. This was also the time l started to learn Latex, which helped a lot in my left writing up on papers and thesis.

Paper Accepted in the Machine Learning, Optimization and Data Science Conference

post by Jimiama Mafeni Mase (2018 cohort)

EFI: A Toolbox for Feature Importance Fusion and Interpretation in Python

Divish Rengasamy (a PhD candidate with the Institute for Aerospace Technology at the University of Nottingham, who recently passed his viva) and Dr. Grazziela Figueredo (my supervisor) published a manuscript to Applied Sciences. Their manuscript proposed a solution to address the lack of agreement among feature importance techniques regarding how they quantify the importance of features to machine learning predictions.  Their solution combined the results from multiple feature importance quantifiers to reduce the variance in estimates and to improve the quality of explanations using crisp information fusion techniques, such as mean and majority vote. A few months later, Grazziela scheduled a meeting to discuss opportunities to improve their solution using a fuzzy logic system (FLS), which is one of the main approaches in my PhD due to its capability to capture and model uncertainties and ambiguity in information.  Divish, Dr. Mercedes Torres Torres (who was my second supervisor at that time, but now works for B-Hive Innovations at Lincoln) and I attended the meeting. In the meeting, Divish went through their proposed solution and together we identified some limitations i.e., significant loss of information as their approach reduced several quantifiers to crisp outputs and difficulty to understand the representation of `importance’ as continuous values (also known as importance coefficients). Next, I described the potential benefits of FLSs, such as using human-understandable linguistic terms to define concepts and using fuzzy sets and rules to handle ambiguous information. They loved the idea and provided synthetic data for experiments.

After obtaining the results from the experiments, we observed that FLSs outperformed the previous solution proposed by Divish in capturing increased variation of feature importance caused by increased data dimensionality, complexity and noise. However, the previous solution still showed remarkable performance in situations with less noise and less variability in the importance of features produced by the multiple machine learning models and feature importance techniques.  It was at this point that we decided to develop an open-source Python toolbox called Ensemble Feature Importance (EFI) consisting of the two solutions i.e., crisp information fusion and FLS.  However, we needed an expert in software modularity and testing to handle the development of the toolbox. Fortunately, Aayush Kumar (a software data scientist), who was doing a master’s in data science at the University of Nottingham, agreed to come on board as the developer of the toolbox. Aayush implemented the toolbox in Python programming language as his master’s dissertation. The toolbox consists of packages to automatically optimise ML algorithms, calculate and visualise the importance of features using various feature importance techniques, aggregate the importance of features from multiple ensemble methods, and create fuzzy logic systems for capturing uncertainties and interpreting importance. The audience for the toolbox is machine learning researchers, end-users of machine learning systems, and decision-makers. After developing and testing the toolbox, we approached Professor David Winkler (a Professor of Biochemistry & Chemistry at La Trobe University, Professor of Pharmacy at the University of Nottingham, and Professor of Medicinal Chemistry at Monash University) to assist in the theoretical motivation of the solutions and as a  second pair of eyes in reviewing the paper.

The paper was written in two weeks and the team was divided into two groups. The first group was made up of Divish, Grazziella and I, who developed the paper structure and decided on the content.  We used `Overleaf’, an open-source online collaborative platform for writing and editing documents, to write the paper. The second group was made up of Mercedes, Benjamin Rothwell (Divish’s supervisor) and David, who reviewed and proofread the paper. Comments were made in ‘Overleaf’ using color-coding and the platform’s chat functionality. After several iterations, we all agreed the paper was ready for submission. A month after submission, we received an email titled ‘Final Decision: Notification of Paper’ with delightful content that our paper had been accepted for presentation at the conference. It was indeed a pleasant message because we were aware of the importance of our toolbox to the machine-learning community. There were a few comments from the reviewers. We were required to address all the comments before uploading a camera-ready version of the paper with a maximum of 15 pages including references. I quickly created a google word document for addressing the comments and shared the link with the other authors. We dedicated a week to addressing the comments, as they were minor. Divish and I did a majority of the revision, as we were more familiar with the methodologies. The most recent version of the toolbox is limited to classification tasks; however, we plan to extend the toolbox to deal with regression tasks. In addition, the toolbox is open-source (i.e., accessible to the public for review, modification and extension)  and we encourage other researchers and ML engineers to contribute to its growth, such as improving the structure of the toolbox, implementing additional state-of-the-art ML algorithms and implementing other feature importance techniques.

Finally, I will like to mention the role of this paper in my PhD. To recap, my PhD aims to develop a reliable, comprehensive and interpretable intelligent driving risk assessment system that considers the simultaneous occurrence of driving behaviours and external conditions. The fuzzy approach implemented in the toolbox for aggregating importance coefficients from multiple machine learning models and feature importance techniques, and to make the importance of features easily understandable has also been explored in my thesis. In my thesis, we explore FLSs for capturing uncertainties in driving data and providing meaningful representations of driving behaviours as linguistic terms using fuzzy sets. In addition, we explore human-understandable fuzzy rules to combine the impact of driving behaviours and external conditions on road safety and incorporate expert knowledge into the system.

Gaining another perspective – a reflection on my internship with Derbyshire County Council (DCC)

post by Kathryn Baguley (2020 Cohort)

My internship opportunity arose from my ongoing work with DCC. Whilst working on another project in early January 2022, I was excited to hear the news that DCC was looking to initiate two large data projects. Hearing this made me wonder whether there might be an opportunity for me to experience some different work within DCC and complete my internship. Upon enquiring, I was delighted to hear that DCC and the CDT thought this would be a good match, and I was given the go-ahead to continue.

As you might expect, DCC, like many large organisations, holds vast amounts of data across many systems. The projects I worked on aim to enable more meaningful use of data for DCC, its employees, and service users. The work looks at aspects such as the accuracy of data, data migration, and the interoperability of new systems moving forward. At the end of the project, DCC hopes that having data in a better place can drive key performance indicators (KPIs). KPIs will assist with better planning, which it hopes will result in a better outcome for service users and generate much-needed savings for the public purse.

I completed my internship over three months on a full-time basis in the Information Technology team. I was excited to be a part of a new team, but it was reassuring that I had worked with some colleagues on previous projects. Overall, my role was part of a larger multidisciplinary team to assist DCC in achieving better management of its data, and enable more agile working.

Going into my internship I hoped for further insights into how a large organisation makes system changes, mainly around automated decision-making. I was also looking to dive deeper into understanding Microsoft systems and work more closely with my new colleagues in Information Technology (IT).

My internship plan was to liaise with IT, the project team and internal clients (various departments) to understand both the technology and the intended context for use. However, we quickly discovered that the scope was too broad, so I concentrated on the technology side and laid foundations to assist the context later. I prepared documents showing the work carried out for accountability of which also acted as a living document moving forward. I also gave ongoing feedback to the project and senior management teams. I recorded my work through a data protection impact assessment (DPIA). The document quickly grew to some 100 pages just to lay down the foundations. I feel the learning acquired here is something that I will definitely carry forward and use in my PhD writing. I am also reflecting on how the Council is carrying out these projects. I am thinking about how risks were assessed relating to automated decision-making and how I can expand my Peer Review Module output to integrate into my PhD writing.

My internship was fully remote, but I do not feel this negated my experience in any way. I felt fully integrated into the organisation and into the project teams from the very beginning. I accept that I may feel this way simply because of my pre-existing relationship with the organisation and colleagues within. It was helpful to me that I had some knowledge of the wider Council and basic knowledge of the IT systems. The experience did build my research skills in both a practical and academic way, since I had to use a combination of speaking with skilled colleagues and online research. It also underlined to me that I find research is much more satisfying if it can be done in a collaborative way with others, as it brings the topic to life. Moving forwards, I am now wondering how I might bring in the practical and collaborative approach to my work, so as to limit the usual loneliness a PhD can bring.

I thoroughly enjoyed the internship, and it is always a pleasure to work on projects where the people in the organisation are passionate about what they do and about doing the right thing. At the end of the internship, I was recognised for doing a good job, and this has helped me to build confidence in an area to which was technically unfamiliar.

On reflection of the module, I feel the biggest challenge was obtaining the right internship in the first place. I envisaged that the process might be more accessible in my case being a qualified and practising solicitor with several years of professional practice experience. In reality I found opportunities scarce, and maybe this appeared to be the case, owing to increased remote working in 2020 with the pandemic and less collaboration in general. I also had my doubts about the module and how I might make it useful for me. As someone with a professional career before starting my PhD, I was keen to ensure the internship gave me more than ‘work experience’ and provided insights that I could use in my PhD.

A further reflection I have is that while there is an emphasis on doing the internship early, I think it is more important to conduct it when the right opportunity comes along with the right organisation. I began to feel pressure to find and do the placement module from the start. For me, I did not feel I had enough detail to understand what I wanted from the experience in the first year. By my second year I did know more about what I wanted from the internship and I am very glad I did wait; the experience has definitely benefited me.

In conclusion, while the internship is complete, I am happy that DCC have provided me with more opportunity to continue working with them to progress these projects. The work carried out over the placement underlines for me the need for industry and academia to work together and how it can produce much better results for all. In particular, I am currently liaising with the IT team about building on the systems work, and risk assessments carried out in the internship.