7/9 Research Software Engineering with Python (COMP233) - Project Management

In this episode I talk to Monika Byrne Svata, Agile Delivery Manager at UCL about project management and in particular Scrum and Kanban. GitHub has features, like Issues and Projects that can help you stay on top of your software projects.

Links

• http://www.linkedin.com/in/monikabyrnesvata Monika's LinkedIn profile
• https://docs.github.com/en/issues/planning-and-tracking-with-projects/learning-about-projects/about-projects GitHub projects documentation
• https://www.atlassian.com/software/jira JIRA
• https://trello.com/home Trello
• https://agilemanifesto.org Agile Manifesto
• https://www.scrum.org/resources/what-scrum-module Scrum, a popular agile methodology
  • https://www.agile42.com/en/blog/scrum-history Scrum history
• https://www.atlassian.com/agile/kanban Kanban methodology
  • https://en.wikipedia.org/wiki/Kanban a bit of background to Kanban
• https://beza1e1.tuxen.de/waterfall.html Waterfall methods
• https://digital.ai/resource-center/analyst-reports/state-of-agile-report/ annual report on all things agile

This podcast is brought to you by the Advanced Research Computing Centre of the University College London, UK. 
Producer and Host: Peter Schmidt

Transcript

Copy Transcript Copied

I guess that by now you have realised that much of what software engineers do in their daily work goes well beyond writing code. There is Git and source code version tracking, there is keeping your code clean and working. And, of course, there is managing software projects as well. Which is the focus of this episode. 

My guest today is Monika Byrne Svata, who is an agile delivery manager at the advanced research computing centre at UCL. What that means and what agile software development methods are will come a bit later. 

First, let’s start with some very practical ways and tools with which you can keep on top of your software projects. And, not surprisingly perhaps, much of that you can do on GitHub. 

One of the GitHub features is well known and used: GitHub Issues. You’ve probably been using that in this course and maybe outside the course. GitHub Issues is a feature that was originally designed to be a bug tracking tool, hence the name. But it’s also used to list features or enhancements you and your team want to make to the code. In that sense, issue is a very broad term that can cover anything you need to or want to do on a software project. 

And the issue list on GitHub gives you a quick overview of all the work that is currently done or still needs doing. Without that list, or backlog as it is often called, you wouldn’t know how much work is actually involved and how far away from completing a software project you are. Creating and maintaining a backlog is a team effort. And I will get back to that in a minute. 

You can create new issues easily on GitHub. And a very nice feature is to attach a label to an issue - and even a delivery date or a milestone. The label is definitely something I would use. Not only does it help you to categorise the work that you need to do - it also helps to find things later. GitHub comes with a set of predefined labels out of the box - but you can easily define your own labels. 

Having a task list or backlog is one thing. But when you manage a project you would also need to look at what the state of each tasks is. The default states of GitHub issues, that is being open or closed, are not really enough. Is the job already being worked on, is it in testing, is it done, is it delivered or is it still waiting to be assigned to a developer? 

For a long time, project managers created clever and complicated spreadsheets to do that. But over the years a number of tools have been made available, such as JIRA from Atlassian Software or Trello. And in 2022, that is little more than a year ago from the time of recording, GitHub also added a project management tool to their site, simply called Projects. You will find the Projects menu in the same navigation bar along with Code and Issues. 

When you create a new Project you have several options: it could create a simple table, a board or a roadmap. For now I want to focus on the Project Board. When you select a Board for a new Project in GitHub, it will create different columns for you by default: such as To do, In Progress and Done. The three you get out of the box with GitHub are a good place to start. You often hear people call these columns ‘swim lanes’. 

The next step is to link issues with the project. You can do this when you create a new issue or you can go through the list of existing issues and put them into the Project. When you do this, you need to specify the state the issue is in: in short is it a To Do, is it being worked on or is it done. There is also a way to automate this: you can set up a project in such a way that any new issue gets automatically linked to a project and set in a default state. Typically that state would be the To Do state. The notes have a link some documentation. 

And with these two features: GitHub Issues and GitHub Project you have the very basic components of a decent project management tool in your hands. With issues you manage your list of tasks and with the project you can see how the work is progressing. 

Again, GitHub is by far not the only one providing basic tools for project management. Many organisations prefer for instance JIRA which comes with a lot more features. 

But project management is much more than just using a bunch of tools. Project management is often about judgement calls and making difficult decisions. What priorities shall I assign to features? What do I need to deliver first? How can I convince the PI that the features they want cannot be delivered in the available time? 

In the next section, I will talk about some of the methodologies that have become very popular and which can help you manage your projects. 

[transition] 

Managing software projects has undergone a bit of a revolution over the last twenty years or so. And this is largely down to a set of software development methodologies that fall under the umbrella of agile. And it is a rather broad umbrella. 

It all started in the late 1990s, when software engineers realised they have a problem. Their software projects are often delivered late, they have bugs and have been running hopelessly over budget. Not a happy state. 

In the winter of 2001, 17 engineers got together in the Wasatach mountains of Utah in the US and discussed what can be done about it. The result was a set of four statements that they would call ‘The Agile Manifesto’. And here they are 

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• And Responding to change over following a plan

The manifesto is underlined by twelve principles, which I am not going to read out now. It’s also important to pay attention to the subleties of the text. It doesn’t say for instance working software instead of comprehensive documentation. It says working software over comprehensive documentation. The four statements are a set of preferences.
 
The next question is: how do you take these rather broad statements and turn them into a concrete and practical method you can actually use in your daily work? Well, you are in luck, because at least some of the signatories of the Agile Manifesto created just set. Take for instance Kent Beck, who created a methodology he called eXtreme Programming, or XP in short.
 
But a far better known method was created by Ken Schwaber and Jeff Sutherland, and is known as Scrum. According to some surveys like the state of agile, Scrum is the most widely used agile methodology. So much so that in most cases when people talk about Agile, they actually mean Scrum.
 
The concept of Scrum actually existed before the agile manifesto. Jeff Sutherland is said to have created the first Scrum team as early as 1993, and Ken Schwaber published a Scrum paper in 1995.
 
There are only a few elements in Scrum and on paper they look easy to implement. That it may not be quite so simple is something Monika and I will talk about in a minute. There is enough material out there on how Scrum is implemented, so I will give only a few key points.
 
The basic concept of Scrum is that a project is delivered in iterations, that Scrum calls sprints. Each iteration has a specific outcome or goal that the team will decide at the beginning of a sprint. The idea being that with each sprint you get closer to the final delivery. Working in iterations has several advantages
 

• the first is, that you focus your efforts on a smaller set of tasks rather than getting distracted by trying to solve everything at once
• the second is, that at the end of each sprint you can see how far away you are from your delivery
• the third is, that new features or changed features can be put in between sprints, which means that Scrum is a reasonable way to hangle changing requirements.

Scrum comes with its own lingo, and it sounds strange at first. Take the backlog for instance. The backlog is in fact nothing but a list of tasks ordered by priority. Scrum has two backlogs: the list of all features called the product backlog and the list of tasks that will be done in a given sprint, the sprint backlog.
 
The other term to get used to is the ceremony. This is a fancy word for meeting. There are three types: the daily meeting, called standup, which should be a 10min review to see if anyone needs help then the longer sprint planning and review meetings.
 
Finally, Scrum has assigned roles and again the words may be a bit confusing: the product owner - in most company that would be the product or portfolio manager or business analyst. In short the person who decides what does or doesn’t go into a software application. In research the product owner is often the primary investigator - or PI. Then there is the Scrum Master, a kind of maitre’d of all the Scrum meetings - sorry ceremonies.
 
In summary: you iterate your way through a project in regular intervals. At the beginning of a sprint you set out the tasks you’ll do. And at the end the team and product manager review what’s been done and sign off on tasks that are completed.
 
Going back to GitHub Issue lists and GitHub Projects: maybe you can see how they can be used effectively in Scrum. The issue list can act as the product backlog for a software project. The project board can then be used to take tasks into a sprint and sign them off at the end of the iteration.
 
So far so good: but at this stage I should say that Scrum is not the only way projects can be managed. And in some cases, using Scrum may not even be the best solution, as Monika will explain in my chat with her.
 
For small teams, say 2 or three developers, you may choose other methodologies, such as for instance Kanban. Kanban is a Japanese expression and apparently means ‘visual signal’ or making something visual.
 
Like Scrum, Kanban is based on a list of tasks - the backlog again. But this time, there are no sprints, no fixed meetings. Rather, the team picks tasks from the prioritised backlog each time they finish something and become available. And as with Scrum, you can use the GitHub Project Board to set up columns for this purpose. Tools like JIRA have project board templates for both Scrum and Kanban.
 
Both Scrum and Kanban are good for making the workload clear and visible. But this doesn’t happen automatically. It’s all very well to create a project board and a backlog. The important thing is to come up with a meaningful list of tasks. Like how important and relevant is the task. Does it depend on anything else. How much can be actually done by the team at any given time and other questions like that.
 
And this, of course, is where the meat of project management is. Which is why implementing any kind of project management tool isn’t as easy as it may sound. And this brings me, finally, to Monika. Monika has a lot of experiences helping companies and developer teams to adopt a systematic way to manage their workload and deliver software. As she explains in the following interview:
 
MONIKA My name is MoniKa Byrne Svata, I have spent my whole career managing software development one way or another, a lot of it was in the commercial and in big enterprises. So I spent my good part of 20 years in consulting and helping managing software delivery various clients through Accenture. That was my consulting company. I was working for that, and I spent couple of years with VPI where I was actually service owner of Finance Upstream Systems and managing software development. There. At that point, Agile really became a thing and we started the transformation. Not going to say how wittingly or unwittingly. From waterfall with every nine months, every 12 months delivery to agile delivery on a quite a monstrous complex financial system that we obviously had. So that it wasn’t trivial sure if it was necessarily the best match, but it was definitely a reality check. And it was my first encounter with Agile management. And after that I went through a lot of training and retraining. At the beginning of the year, I ended up with the UCL, very excited to be part of ARC, which is completely contrary of anything, any companies that I worked for before. So it is very much a little projects, research projects, ended, a lot of times not measured in finances, measured in value that is quite whimsical and hard to put a finger on. And at UCL in this role, my role is agile delivery manager. So my role is very much overseeing delivery of software through projects or even across the board. Coaching is a large part of it, making sure that principles that make sense are available and applied; things that don’t make sense don’t disturb and create unnecessary noise. So basically trying to make sense out of it whilst understanding some of the principles of Agile.

PETER Yeah, That’s quite a lot. Okay. I’m quite interested in your career in the private sector, because, I mean, I have some experience there as well. You said Agile delivery manager. And I always struggled a little bit with that definition of what an Agile delivery manager is. So what is your understanding of what an Agile delivery manager

MONIKA 2:38 I have a couple of colleagues obviously have the same title and I think our work very much differs depending on what our teams need. in some instances it is more heavy on the delivery management. So if we are talking about infrastructure or environments, the management of actual delivery of getting the infrastructure implemented, having the dependencies , So that’s definitely a delivery management at its core and the Agile comes to it much as needed or as possible. On my side, I’m in a team where we have basically hundreds of little projects and a lot of research software engineers or other specialties, data stewards, data scientists, etc. delivering a little research science project sized couple of months or maybe a year or two. it’s on this side. It is much more about coaching, making sure that the teams have access to the principles of Agile, that I can help them. Sometimes I act as a Scrum Master, sometimes I act as advisor. So I would come to the project, help set up the Agile delivery, and then help coach the person who will be the Scrum master. Sometimes it is just a little when the project is not too heavy and it should finish in a couple of months. kind of project management, a little bit of coming on board, a little bit of making sure that people can get together and actually check what is important and what they are doing. So it very much depends from project to project. It is not just Scrum as such, but it is element of that. It is on that of a coach, And So it is very varied and very, very interesting.

PETER There’s quite a lot in there that I think we need to talk in a bit more detail . So you mentioned Scrum. You mentioned Scrum Master. You mentioned combining all these things. So our listeners may not be quite familiar with it, but when we talk about actually different agile methodologies Scrum is one and Kanban is one and I think there is an organizational one as well because Scrum and Kanban work on teams, but then you that is a large organization and you follow a different framework. Could you quickly talk about that?

MONIKA Okay. So let me start from above quickly. The the UCL framework. So we are operating on this. So called scaled Agile framework or SAFE is the acronym. The purpose of that is basically allowing smaller Agile teams to come together and align their delivery into larger products, make sure that the dependencies across those teams are aligned. It has to be said we don’t necessarily work across ISD towards the same product. So so there is a limitation on how much this is used for. Especially in the Collaborations team projects we are working on grants with the various universities on very deadlines that have nothing to do with the, internal UCL deadlines. So we are trying to make the best out of this framework and especially use the regular sessions that are every three months sessions a milestone to pause, make sure that our services are aligned, make sure that we understand what is our project portfolio, make sure that we know what we are doing in the next three months. Is it quite the scaled Agile framework? I don’t think it is, but we are trying to use it reasonably and to go to the best advantage of what we can do.

PETER Yeah. Adapt it basically to your needs. Mm hmm. Okay.

MONIKA yeah, underneath, we have various teams that are doing various types of delivery. A lot of them function more or less as scrums Again, how it is fitting to different teams is depending on what products they are delivering in the collaboration team a whole, the team doesn’t operate as a scrum because it is around 60 team members. So obviously that’s not possible size wise. And also the the portfolio of what we are delivering is various projects still towards various deadlines with various collaborators. So basically the aim is to support skills development, to support the collaborations with external entities, to make sure we do all this better and more efficiently, that we produce reusable assets, that we can apply to other projects or that we can make available outside. So that’s basically the kind of higher aims that within the project we are following methodologies in some of the projects. But they differ very much. We have applied now very successfully on a good few projects principles of Scrum. The reason why we chose Scrum is basically because it is kind of the simplest, understandable framework that works in a sprint cadence. So usually two weeks. So it allows basically the team to focus delivery every two weeks knowing what is important, So it is very useful for the shorter deadlines for some of the projects that are really already in slide just about to finish had some tasks. We are just using Kanban, which basically where we are not using sprints, we are just trying to monitor what has to be done, what is in progress. Basically just focusing minds and making sure that in progress column is not too heavy and that that we are actually planning reasonably to the the capacity we have. So that’s basically the main framework. there are sometimes projects that we have like one person or two people. hardly right or sensible to apply any framework across that.

PETER So what I’m hearing is it’s not a methodology at all costs. You’re looking at what kind of methodology fits the project and fits the case.

MONIKA 8:47 Absolutely. Yeah. what is reasonable. What is the best common sense approach we can find having all the knowledge about agile delivery and or other types of delivery. what helps most Basically,

PETER Mm hmm. And Scrum is, I believe, the most popular form of Agile methodology is being used. So my understanding is that the main difference is that in Scrum, you work in sprints. You mentioned typically two weeks. Some people use three or maybe even four weeks, after which you come together and discuss what you’ve done and you plan for the next sprint. Kanban doesn’t work in Sprint, it works on a backlog and how much the team can do. Is that correct? Is that sort of the main difference or did I get that wrong?

MONIKA Yeah. is the main difference. Yes. Yeah. Scrum we have a five preset ceremonies. It sounds like a lot of ceremonie is what I say. Ceremonies. I mean meetings basically. So at the the end of the sprint we have a Sprint review where we have a look at what we have done and plan for the next sprint. Then there is the constant backlog refinement. And so that’s basically making sure that we understand any work that we still need to do. And it is in the to do this And then there is also Sprint retrospective, which is basically looking back at the process, talking about what works, what doesn’t, what we can improve, And then there is very important and very hard to implement because people don’t like the idea of it. But to me something that makes the most difference most quickly is the daily standups.

We do not do them daily, but what we have quite successfully implemented mostly is every other day, which is better than nothing. And that basically is something that takes ten, maximum 15 minutes depending on the size of the team and makes sure that team gets together. Everybody says, this is what I have done, this is what I’m doing, this is what is stopping me. This is the question I have very short answer from somebody. If there is not a very short answer possible, then there is an answer. Okay, let’s sit together tomorrow at two because I have time. There is no instance where somebody should be waiting or for somebody to answer an email or some other means. It also puts the product owners feeds into the water and makes them understand much better what the team is dealing with rather than coming every two weeks going: Hey? How is it not done? Understanding the detailed questions, understanding need of the collaboration that’s invaluably done in daily stand-ups. The Kanban doesn’t quite have the same rigid ceremonies and also typically doesn’t do sprints. So we we are monitoring there is making sure that we understand the backlog that to do list and making sure that the in progress what we choose to do is reasonable and that we are not basically working at everything at once,

PETER implementing agile in environments where it hasn’t been used before, whether it’s Scrum Kanban or any other methodology, like extreme programming sometimes meets resistance. Is that your experience as well

MONIKA So my experience in private companies versus you see it is almost opposite. There is always some resistance when you want people do
new things. But private companies, because the systems I have supported tended to be really large and convoluted. robust Finance systems lean themselves very well to waterfall or so decoupling from that and go into some kind of new mode where we say, okay, it’s fine if we change this every two weeks and it’s fine if we change something: there was a learning curve and it was quite difficult to even for me to let go of some of the ideas that this is not possible and find kind of the happy medium of what is possible while still not breaking everything. At UCL, the work is very different, especially in my team. When the team are small and there is no sense of rigidity or of, of finite product that that is hard to decouple. It’s quite the contrary. Nonetheless, of course some teams already have ways of working. What is quite a difficult problem, is finding a product owner, which is usually the PI, who would have enough time or who would perceive that his time is well spent giving his input regularly to the team. So challenges are very different, we have managed to very successfully and implement a couple of scrum working projects with UCL and each where we work very closely with the PIs, with the clinicians and it feels like everybody is happy about that cooperation. So I’m hoping we can just continue doing that good news.

PETER Indeed. Talking of which, Have you seen any changes in the way that teams interact with each other, that they do things a little bit differently, that they do things better, What’s your experience in that?

MONIKA So I haven’t been at UCL before Agile was introduced, so that’s not an experience that I have. Just from what I have seen in the individual projects, when we started to introduce Scrum, I know that the collaboration with the PI is increased, especially through the regular stand ups is much closer. So there is better understanding, and just appreciating each other’s input more or even the effort that goes into things, because it is witnessed from much closer distance.

PETER Okay. Well, thank you very much. We’ve run out of time. Thank you very much, Monica. And all the best to you at UCL.

MONIKA Thank you, Peter.
 
OK, I hope this conversation made it clear that even though Scrum and kanban are very popular - they are not the kind of silver bullet solution people may hope for. Which, I confess, is something you hear very often from people these days. Because there is no shortage of Agile advocates, training courses and certificates out there. And there a lot of consultants who will tell you that agile, and by that they mean mostly Scrum, is the bees knees and the solution to all your software problems.
 
Often, they will contrast the benefits of agile with the bogeyman of project management: waterfall. Waterfall has become the catch-all phrase for anything that doesn’t work. It’s loosely based on the concept of delivering software in a very strict and rigid order: you start with a complete list of specifications, then you code, after that you test and then you deliver. One thing leading to another, like water trickling down a hill. Hence the name.
 
Reality, however, was very different. In my career I haven’t seen any company deliver software in a Waterfall sense. And, likewise, using Scrum or Kanban is no guarantee that software deliveries won’t get delayed.
 
Still, I think there are advantages of considering and adopting Scrum and Kanban as long as you think about how they may benefit your work and how they can help with your software deliveries.