post by Matthew Yates (2018 cohort)

On 16^th June 2022, my paper “Evaluation of synthetic aerial imagery using unconditional generative adversarial network” was accepted into the following August edition of the journal “ISPRS Journal of Photogrammetry and Remote Sensing”. This was my first paper to get published and also my first time going through the academic peer review process.

Although the paper has only just been published in summer 2022, the research began in 2019 with much of the initial work taking up a large portion of the first year of my PhD. The motivation for the paper was to take all the early research I had been doing for my PhD Thesis and produce a deliverable publication out of it. The reasons I was keen on doing this was that it would give me chance to get ahead of the Thesis writing end stage of the PhD as this paper would more or less cover the first empirical chapter, it would also help introduce me to the process of peer reviewing which I did not really have any prior experience with.

After delays due to the outbreak of COVID and the subsequent lockdown measures, the paper was submitted to the Journal of Machine Learning in summer 2019. The scope of the paper had been stripped back from the original ideas, with fewer models being benchmarked due to the inaccessibility to the Computer Vision GPU cluster at that time. After a couple of months of waiting, the Journal came back with a “Major Revisions” decision along with plenty of comments from the 3 reviewers. The reviewers deemed the paper to be lacking a substantial enough contribution to warrant publication at this stage and there was a general negative sentiment amongst the reviews. I then resubmitted a month later after responding to the reviewer’s comments and making large amendments to the paper only to get a rejection after the next round of reviews. As this was my first paper I had tried to get published I was rather disheartened in receiving this decision after spending so much time on the work. My supervisors were less concerned, having gone through the process many times, they told me this was fairly normal and it would be a good idea to submit to a different venue which may be more appreciative of the work.

In early spring 2021 I sent the paper to the ISPRS Journal of Photogrammetry and Remote Sensing. The decision I received here was another rejection, although this one came with much more constructive criticism and I was advised to revise and resubmit at a later date. As this was over a year on from my original submission attempt I had also been working on additional research for my PhD and then made the decision to incorporate some of these new results into the original paper, significantly increasing the contributions of the work. At this point my primary supervisor Mercedes Torres Torres, brought in Michael Pound to help add a new perspective on the work and give me some new feedback before resubmission. After submitting to his new journal again in October 2021 I was given a “major revisions” decision in December 2021, the reviewers who were a mix of old and new from the last failed attempt at the same journal had responded much more positively to the changes and additional content but thought it still required additional work and depth in some parts. Now in January 2022, I resubmitted, hoping that this would be the last time but received another round of corrections in April. At this point I was getting fairly fatigued with the entire process, having done the bulk of the work years ago and each round of revisions taking months. Luckily the reviews in this last round were positive and only one of the 3 reviewers called for additional work to be done. As I could see I was close to publication I went over all of this final reviewers’ comments in detail and responded accordingly as I did not think I could face another round of revisions and wanted to move on to other research. Luckily the next decision was an acceptance with all the reviewers now satisfied with the work.

The acceptance of the paper was a huge relief as it felt like the time and effort myself and my collaborators had put into the paper was finally vindicated. I was additionally pleased as this is my first publication, something I had been looking to achieve for a few years now. The paper also represents the first part of my PhD project and gives that whole stage of research more credibility now it has gone through the peer review process. Following this publication, I have now been invited to be on the list of reviewers for the journal if anything in my field is submitted. This is something I would be interested in doing to get an insight into the other side of the review process which could feel quite opaque at times. I have also been invited to publish further research in other related journals. These initial responses to the publication have shown that it was worth enduring through the rather lengthy and sometimes unpredictable process of peer reviews.

post by Luke Skarth-Hayley (2018 cohort)  

It’s tough to do outreach during a pandemic. What can you do to help communities or communicate your research to the public when it’s all a bit iffy even being in the same room as another person with whom you don’t live?

So, what can we do?

Well, online is obvious. I could certainly have done an online festival or taught folks to code via Zoom or Skype or Teams or something.

I don’t know about you, but I’ve got a serious case of online meeting fatigue. We’re not meant to sit for hours talking in tiny video windows. I’m strongly of the opinion that digital systems are better for asynchronous communications, not instantaneous.

So, what did I do?

I took my research online, asynchronously, through various means such as releasing my code as an open-source plugin for the Unity game engine and creating tutorial videos for said plugin.

Open-Source Software

What is open source? I’m not going to go into the long and storied history of how folks make software available online for free, but the core of it is that researchers and software developers sometimes want to make the code that forms their software available for free for others to build on, use, and form communities around. A big example of open-source software is the Linux operating system. Linux-based OSes form the foundation on which are built most websites and internet-connected services you use daily. So, sometimes open-source software can have huge impacts.

Anyway, it seems like a great opportunity to give back from my research outputs, right? Just throw my code up somewhere online and tell people and we’re done.

Well, yes and no. Good open-source projects need a lot of work to get them ready. I’m not going to say I have mastered this through my outreach work, but I have learned a lot.

First, where are you going to put your code? I used GitHub in the end, given it is one of the largest sites to find repositories of open-source software. But you might also want to consider GitLab, Source Hut, or many others.

Second, each site tends to have guidance and best practices on how to prepare your repository for the public. In the case of GitHub, when you’ve got research code you want to share, in addition to the code itself you want to:

• • • Write a good readme document.
    • Decide on an open-source license to use and include it in the repository.
    • Create a contribution guide if you want people to contribute to your code.
    • Add a CITATION.cff file to help people cite the code correctly in academic writing.
    • Get a DOI (https://www.doi.org/) for your code via Zenodo (great guide here: https://guides.lib.berkeley.edu/citeyourcode) so it can be cited correctly.
    • Create more detailed documentation, such as via GitHub’s built-in repository wiki, or via another method such as GitHub pages or another hosted documentation website.
    • Bonus: Consider getting a custom domain name to point users to that redirects to your code or that hosts more information.

That’s already a lot to consider. I also discovered a couple of other issues along the way:

Academic Code

Academic Code, according to those in industry, is bad. Proving an idea is not the same as implementing it well and robustly, with all the tests and enterprise code separation, etc. That said, I have seen some enterprise code in my time that seems designed to intentionally make the software engineer’s job harder. But there is a grain of truth regarding academic code. My code was (and still is in places) a bit of a hacked together mess. Pushing myself to prepare it for open source immediately focused the mind on places for improvement. Nothing is ever done, but I did find ways to make the code more adaptable and flexible to diverse needs rather than just my own, fixed some outstanding bugs, and even implemented custom editors in Unity so non-programmers would be able to do more with the plugin without having to edit the scripts that underpin it. In addition to making the code better for others, it made it better for me. Funny how we do what’s best for ourselves sometimes under the guise of what’s best for others.

Documenting a Moving Target

No software is ever done. Through trying to break the Curse of Academic Code as above, I rewrote a lot of my plugin. I had already started documenting it though. Cue rewriting things over and over. My advice is split your docs into atomic elements as much as possible, e.g., if using a wiki use one page per component or whatever smallest yet useful element you can divide your system up into for the purposes of communicating its use. Accept you might have to scrap lots and start again. Track changes in your documentation through version control or some other mechanism.

Publicising Your Open-Source Release!

Oh no, the worst bit. You must put your code child out in the world and share it with others. Plenty of potential points to cringe on. I am not one for blatant self-promotion and rankle at the idea of personal brands, etc. Still, needs must, and needs must mean we dive into “Social Media”. I’m fortunate I can lean on the infrastructure provided by the university and the CDT. I can ask for the promotion of my work through official accounts, ask for retweets, etc. But in general, I guess my advice is put it out there, don’t worry too much, be nice to others. If people give you grief or start tearing apart your code, find ways to disambiguate real feedback from plain nastiness. You are also going to get people submitting bug reports and asking for help using your code, so be prepared for how you balance doing some of that with concentrating on your actual research work.

Tutorial Videos

Though I can quite quickly point back to the last section above, and say, “MOVING TARGET!” there is value in video tutorials. Even if you must re-record them as the system changes. For a start, what you see is what you get, or more accurately what you get to do. Showing people at least one workflow with your code and tools in a way that they can recreate is immediate and useful. It can be quick to produce, with a bit of practice, and beats long textual documentation with the occasional picture (though that isn’t without its place). Showing your thinking and uses of the thing can help you see the problems and opportunities, too. Gaps in your system, gaps in your understanding, gaps in how you explain your system are all useful to flag for yourself.

Another nice get is, depending on the platform, you might get an accidental boost to awareness of your work and your code that you’ve released through The Algorithm that drives the platform. Problems with Algorithms aside, ambient attention on your work through recommended videos e.g., on YouTube can be another entry point for people to discover your work and, their attention and interest willing, prompt them to find out more. This can be converted into making contacts from people who might want to use your tools, who might then participate in studies, or it may make people pay attention to your work, which you can use to nudge them into checking out something you are making with your tools, again making them into a viewer/player/potential study participant.

But how do you go about recording these things? Well, let’s unwrap my toolkit. First up, you’re going to need a decent microphone, maybe a webcam too. Depending on how fancy you want to get, you could use a DSLR, but those can be prohibitively expensive. Maybe your research group has one you can borrow? Same goes for the microphone. I can recommend the Blue Yeti USB microphone. It’s a condenser microphone, which means the sound quality is good, but it can pick up room noise quite a bit. I just use a semi-decent webcam with at least 720p resolution, but I’ve had that for years. This is in case you want to put your face on your videos at any point. Just how YouTube do you want to be?

Anyway, you have some audio and maybe video input. You have a computer. Download OBS Studio is my next recommendation. You can get it at https://obsproject.com/. This is a piece of software that you can use to record or stream video from your computer. It even has a “Virtual Camera” function so you can pipe it into video conferencing software like Zoom or Microsoft Teams. Cue me using it to create funny effects on my webcam and weird echoes on my voice. But, in all seriousness, this is a very flexible piece of freely available software that allows you to set up scenes with multiple video and audio inputs that you can then switch between. Think of it as a sort of home broadcasting/recording kit. It is very popular for people making content for platforms like YouTube and Twitch.tv. I’ll leave the details to you to sort out, but you can quite quickly set up a scene where you can switch between your webcam and a capture of your computer’s desktop or a specific application, and then start recording yourself talking through whatever it is you want to explain. For example, in my tutorial videos I set it up so I could walk through using the plugin I created and open-sourced, showing how each part works and how to use the overall system. Equally, if you aren’t confident talking and doing at the same time you could record your video of the actions to perform, and then later record a separate audio track talking through what you are doing in the video. For the audio, you might want to watch the video as you talk and/or read from a script, using something like Audacity, a free tool for audio recording you can download from https://www.audacityteam.org/.

Which brings me on to my next piece of advice. Editing! This is a bit of a stretch goal, as it is more complex than just straight up recording a video of you talking and doing/showing what you want to communicate. You could just re-record until you get a good take. My advice in that case would be to keep each video short to save you a lot of bother. Editing takes a bit more effort but is useful and can be another skill you can pick up and learn the basics of with reasonable effectiveness. Surprisingly, there is some excellent editing software out there that is freely available. My personal recommendation is Davinci Resolve (https://www.blackmagicdesign.com/products/davinciresolve/), which has been used even to edit major film productions such as Spectre, Prometheus, Jason Bourne, and Star Wars: The Last Jedi. It is a serious bit of kit, but totally free. I also found it relatively simple to use after an initial bit of experimentation, and it allowed me to cut out pauses and errors, add in reshot parts, overdub audio, and so on. This enables things like separating recording the actions you are recording from your voiceover explanation. Very useful.

Next Steps

My public engagement is not finished yet. My research actively benefits from it. Next up I intend to recruit local and remote game developers in game jams that use the plugin, specifically to evaluate the opportunities and issues that arise with its use, as well as to build an annotated portfolio as part of my Research through Design-influenced methodology.

Conclusion

So, there you have it. Ways I’ve tried to get my research out to the public, and what I plan to do next. I hope the various approaches I’ve covered here can inspire other PhD students and early-career researchers to try them out. I think some of the best value in most of these comes from asynchronicity, with a bit of planning we can communicate and share various aspects of our research in ways that allow a healthy work-life balance, that can be shaped around our schedules and circumstances. As a parent to a young child, I know I’ve appreciated being able to stay close to home and work flexible hours that allow me to tackle the unexpected while still doing the work. If there is one thing, I want to impress upon you, it is this: make your work work for you, on your terms, even if it involves communicating your work to tens, hundreds, thousands of people or more. Outreach can take any number of forms, as long as you are doing something that gives back to or benefits some part of society beyond academia.

You can get the Unity plugin here: https://github.com/lukeskt/Reactive-Mise-en-scene

post by Ana Rita Pena (2019 cohort)

Interning at Capital One

Between May and October 2021 I held a part-time internship with my Industry Partner, Capital One. Capital One is a credit card company that launched its operations in the UK in 1996 and has its parent company in the US. The company is known for being technology driven and in the specific UK case focusing on credit building cards as their main product.

My internship with Capital One UK consisted of working on several projects as part of their Responsible AI initiative due to my interest in topics related to “ethical” machine learning and FACct/FATE (Fairness, Accountability, Transparency and Explainability).

The Responsible AI initiative initially consisted of three projects: the Trustworthy Algorithm Checklist, Global Methods and Individual Level Methods. The Trustworthy Algorithm Checklist project was already under way when I joined the company in May 2021. The project consisted of creating a checklist for Model Developers to complete during the model development process, in order to instigate some reflection and mitigation of ethical risks and consequences associated with the new model. The Global Methods project was subdivided in two parts. Generally, the project aimed to evaluate different explainability methods and provide some guidance and recommendation on different methods to be adopted internally by the Data Science team. The first part consisted in interviewing stakeholders from different departments to have a better understanding about which information each of them needed about the model and the second part consisted of a technical evaluation of the tools. Finally, the third project, Individual Level Methods, aimed to explore how consumers understand different explainability methods to represent their individual outcome. This third project never went ahead due to lack of time.

On my day to day, I worked within the Data Science Acquisition team as my manager was based on this team and he spent a percentage of his time working on the Responsible AI Initiative, however my workflow was separate to the rest of the team. Being able to attend the team’s meetings allowed me to have a better understanding of the workings of the company and the processes involved in model development and monitoring.

In the following sections I will describe in more detail the two projects I worked on as well some general reflections and thoughts on the internship.

Trustworthy Algorithms Checklist

Over the last few years there have been several stories in the press of algorithms which are implemented and end up having unintended consequences which negatively affect users, for example the UK’s A-level grading algorithm controversy, which ended up affecting students from lower socio-economic backgrounds more negatively than other students by deflating their grades the most. This has led to research in the areas of “ethical” AI to cross into real world applications. The Trustworthy Algorithm Checklist Project aims to design a checklist which will make model developers actively reflect on unwanted impacts of the model they are building as part of the development process. After the checklist completion it would then go through an ethics panel composed of stakeholders from different business departments within the company, for an approval process. The initial draft checklist was divided into three sections: Technical Robustness, Diversity Non-discrimination and Fairness and, finally Accountability.

The next iteration of the checklist design created more sections which were based on the Ethical Principles for Advanced analytics and Artificial Intelligence in Financial Services created by UK Finance, which is a trade association for the UK banking and Financial Sector. Ending up with the following five sections: Explainability and Transparency, Integrity, Fairness and Alignment to Human Rights, Contestability and Human Empowerment and Responsibility and Accountability. It was at this stage that I joined the company and started working on this project. The project involved stakeholders from the legal and compliance department as well as from the data science department. This second iteration of the checklist was trialed with a model that was in the initial stages of development, from this trial it was noticed that most of the answers to the prompts in the checklist were mentioned to already be covered by existing Capital One Model Policy. In order to avoid the Checklist becoming just another form that needs to be submitted, the team decided to put more emphasis on the ethics panel discussion meeting and have the checklist being an initial prompt in the discussion in order to foster critical reflection which is aided by having stakeholders that come from different backgrounds and hence bring different perspectives.

While this project initially only focused on the algorithmic aspect of decision making, the team involved discussed the possibility of expanding the checklist to the process of development of Credit Policies. It is the combination of the Algorithmic Risk Assessment with the Credit Policy which will end up impacting the consumer and hence the need to critically evaluate both these parts.

Explainability Toolbox

Global methods are a set of tools and visualisations to help better understand the way complex machine learning models work at an overall, rather than individual level. These methods can focus on the variables, for example which variables have bigger impact on the result or how different variables are related among themselves, or the general decision rules of the model, for example summarise a complex method by approximating it to a set of simple decision rules.

Including explainability methods on the machine learning work process is quite important as these allow one to verify the behaviour of our model at any time and check if it is working well and as expected.

When this project was first discussed with me it consisted of implementing and comparing different explainability methods and packages implementations (both developed internally as well as open-source tools) in order to propose a set of tools to uniformise the tools used by different teams within the Data Science Department. Due to my interdisciplinary experience in Horizon and partly working within the Human Factors field, I was aware that the technical components of the explainability methods are not the only factor that affect how well technology is adopted and used in an institutional setting. In order to address the social aspect of this task I suggested having a series of interviews with different Stakeholders that interact with the Data Science team, in order to have a better understanding of what information they needed on the models and would like to better understand and on their views on the methods that were currently implemented. Doing these interviews allowed me to understand better what different roles and departments did and how they interacted, as previously I had mainly only interacted with the Data Science department.

From the interviews I learned that stakeholders from more business-related roles as well as higher level roles were interested in being able to translate how changes in the model’s behaviour impact business, e.g. in terms of number of people given loans or in terms of profit or number of defaults. Stakeholders from the technical departments were also aware of the shortcomings of the methods they currently adopted but had not had the time to test alternatives in their workdays. From the interviews I created a list of guidance when presenting technical materials to stakeholders as well as identified several fields to evaluate on the second part of the project.

In the second part of the project, I compiled different packages/libraries (open source and internal packages developed by different C.O departments and countries) to test their methods and give guidance on what could be beneficial to implement across the Data Science Department. During this process I learned that different branches of C.O. in different countries use different coding languages as well as that different models from different Data Science teams had different characteristics due to their different needs and what historically their department and branch have implemented. This meant that oftentimes different teams had to create their packages from scratch to meet the specificities of their model, even if they were using the same tools which could have been avoided if there was uniformisation of the language or model construction approach.

Final Reflections

This was my first time working in Industry and I was very pleasantly surprised with the importance that Capital One puts in research, running internal conferences and having specialised research centres (in the US, which is their biggest market). This was further encouraged by the very open and collaborative work environment, for example the Responsible AI UK initiative I was involved with had regular meetings with other research teams within C.O. working in the same field.

While the company had very good intentions and initiates in different projects just like the ones I worked in, the reality of the scale of the UK branch meant that all of the team (apart from myself) only worked on the Responsible AI initiative 10% of their time on top of their team ’s roles. The Explainability Toolbox project also showcased the drive to optimise processes across departments, even if hard to accomplish at scale due to logistical constraints.

Overall, my internship at Capital One gave me a better understanding of the Consumer Credit Industry and the way different departments come together to be able to provide a financial product to the consumer.

post by Eliot Jones-Garcia (2019 cohort)

My journey with CIMMYT, the international maize and wheat improvement centre, began shortly before enrolling with the Horizon CDT. In February of 2019, after having recently graduated with an MSc in Rural Development and Innovation from Wageningen University, I found myself in Mexico and in need of a job. I was familiar with the organisation because of its pivotal role in delivering the Green Revolution of the 1960s; an era of significant technological advancement in agriculture enabled by seed breeding innovation achieved by CIMMYT scientists. Since then, they have branched out to engage in several areas of agricultural research, from climate change adaption and mitigation strategies to markets and value chains.

Thus, prior to beginning my PhD research in earnest, I spent 6 months conducting a systematic review of technological change studies for maize systems of the Global South. I worked at the headquarters in Texcoco and gained valuable experience among the various academic disciplines CIMMYT employees use to approach agricultural sustainability. Having forged strong relationships with management staff, they volunteered to support me in my move to Nottingham and transition in research toward ‘digital’ agriculture.

During my first year of research at Horizon, I worked with the CIMMYT staff to conceptualise an internship project. The plan was to head back to Mexico once again in the summer of 2020 to collaborate with scientists there. Unfortunately, however, the unexpected onset of COVID-19 forced me to change plans. At first the work was postponed in the hope the situation would ease but to no avail. I decided to undertake my internship remotely and part-time beginning in January of 2021. In hindsight I was incredibly pleased to have had the initial in-person experience but working at a distance would prove to have its own great lessons.

The goal of my work was to explore different methods of natural language processing, sentiment analysis and emotion classification for analysing interviews with farmers. COVID-19 had not only stunted my travel plans, but all CIMMYT researchers were finding it hard to get to farmers to collect data. These interactions were increasingly taking place remotely, via mobile phones. This removed a significant interpersonal dimension from the research process; without supporting visual context, it became difficult to understand affective elements of conversation. I was given access to a series of interviews with different agricultural stakeholder that had been manually coded according to their use of technology and charged with finding out how these digital tools might aid in analysing audio and textual data.

I approached the task exploring the grounding literature. The first major insight from my internship became how to turn around a thorough and well-argued review for motivating a study in a short time, whilst providing a good understanding for myself and the reader. This yielded a variety of approaches to defining and measuring emotion, selecting audio features for analysis, and modelling tools. I ended up taking a conventional approach, using ready-made R and Python tools and lexicons to analyse text, and a series of widely available labelled datasets to train the model. The second insight from my internship was to engage with different open-source communities and apply available tools to achieve my desired goal.

In combination with working remotely, these activities gave me great confidence to independently deal with tasks, to seek and gain certain skills and to utilise them with support of experts to a high degree of quality. More than anything I feel like this internship taught me to apply my academic abilities to unpack and explore problems in a concise and specific way and to deliver what CIMMYT want in the form of outputs, that is actionable insights that can be applied in a Global South context, and for motivating future research.

In light of this, I produced a structured analysis and report for my CIMMYT supervisors which was then published as an internal discussion paper in December of 2021. Findings from the study indicate that sentiment analysis and emotion classification can indeed support remote interviews and even conventional ethnographic studies. By revealing several biases related to transcription and translation of text, the analyses suggested greater consistency in future study to mitigate any unreliability this may introduce. In terms of affect, there was a clear relationship between different sources of data; dis-adopters of technology, or those who rejected use, were shown to be angrier relative to the rest of the sample, whereas new adopters expressed greater joy and happiness. While this confirmed our expectations there were also unusual insights, for example, female farmers were less fearful in the adoption of technologies. It is expected that in future this research may contribute to better targeted interventions, making technologies available to those who are more likely to make use of them.

Moving forward, I continue to work with my industry partner in smaller projects and look forward to collaborating with them in a professional capacity. This experience has been a great help in my PhD for focusing the direction of my research, highlighting the role of data in shaping how knowledge is created and how that plays into agricultural development. It has helped me to manage tasks and to allocate time wisely, and to produce industry standard work to provide benefit to farmers. The final version of the work is undergoing peer review and I hope to see it published in the near future.

If anyone would like to learn more about this work or would like to contact anyone at CIMMYT, please do not hesitate to contact me at eliot.jones@nottingham.ac.uk.

Many thanks for reading!

Eliot