6 Lessons Learned as a New Data Science Lead

Whenever I read the reviews for a book (especially if the reviews are mixed), I think that those reviews would be way more useful for me if only I knew a bit more about the authors of the reviews.

What kind of books do they normally like? If it’s a textbook, what level of knowledge do they already have on the topic? What was their goal when they picked up this book? This way I can judge more effectively if their reading advice applies to me and if I should base my future reading decision on these strangers’ opinions.

Along the same line of thought I also want to describe briefly my situation and background, so that as a reader you have more context to judge to what extent the lessons I had learned apply to your situation.

I am a computational linguist and computer scientist by training. After doing a number of internships in both engineering and R&D roles I got into a full-time data scientist position at trivago. I worked in this role in a data science team for little more than two years before I accepted a lead position, initially leading a small team of two data scientists.

This was my first experience in a leadership position.

Since then more than a year has passed and I’m still in this lead position, having seen my team grow, and shrink and grow again now to a team of 6 (5 data scientists and 1 product manager). My own lead is an individual with a business background, who sees a lot of value in data-driven decision making and data-driven products. He is overseeing 4 teams altogether (all very technical, a mix of data science and engineering) and is quite invested in fostering an environment where data scientists get to solve interesting problems while contributing value to the business.

My transition from an individual contributor (IC) role into a leadership role is a quite prototypical move in technical fields. Hence, I believe that experiences very similar to mine can be found in many other organisations where data science teams are growing. So I hope that my reflections on this topic can be of use to some other data scientist who might be feeling sceptical of their new challenge and is looking for someone else’s perspective once sitting in the same boat as they are.

1. Uncertainty is okay. Even more so than before.

If you have worked in a data science team already, probably you are not entirely unfamiliar with uncertainty. Most probably you have worked on some greenfield projects in your past. Maybe you have even led some of them. And some of them might have succeeded, while some others might not have reached the desired objective. As a data scientist you were naturally aware of the uncertainty of exploratory projects and if you worked in an environment that was well calibrated to data science work, the uncertainty was accepted, both by yourself and your lead.

Now, as the lead of a team, you will feel the uncertainty to weigh more heavily on your shoulders than before. What if the projects that my team works on fail? What am I supposed to do as a lead? What if someone figures out that I don’t have a plan (everyone’s inner impostor says hi)?

Breathe.

In a way it’s a good thing that uncertainty feels more daunting now than before. It’s because you can feel the responsibility that was given to you. However, it’s a misunderstanding if you think that your responsibility is to completely get rid of all uncertainties for yourself and your team. That’s impossible.

The first step is to accept that working in the data science domain is still uncertain. This hasn’t changed since you’ve become the lead. Yes, more people are looking to you for guidance, but they won’t expect you to solve all their problems and resolve all uncertainties.

As you grow as a lead, the number and the scope of people that you will interface with will also naturally grow. Remember that this extension of your scope will result in even more uncertainty. So learn to be okay with it, as in time you will learn skills that help you to reduce the uncertainty. But don’t expect this to happen over night.

2. Trust your intuition (and sometimes your reasoning).

This is probably the piece of advice that I have most frequently received from other leads in my organisation. Some went as far as saying that I should become ‘more arrogant’ in some sense, i.e. having the audacity to believe that I know it better (than someone else, maybe even some more experienced co-worker). While this quip can obviously not be fully taken seriously, its underlying message is clear, though.

You know more than you give yourself credit for.

While I am generally a critical person, when it comes to a field that still feels unfamiliar to me I tend to over-rely on people that seem more experienced. Don’t get me wrong. Experienced people’s advice is valuable and you should always seriously consider them. But don’t put them on a pedestal in every situation.

You will be facing issues where there is no clear right or wrong, just different styles and preferences of doing things, or different values competing with each other. These are the situations where your gut feeling might be the most valuable clue for what you (want to) stand for.

If you are still uncertain if you can make the right decision or take the right action: Try it out! Experiment! Gather real-life data points. What’s the worst that can happen?!

3. Mindsets about value contribution change slowly.

In my early development as a lead while figuring out the things I have to do in my day-to-day work, this point was my biggest challenge. I was used to seeing my value contribution as the analyses I did and the models I developed. That’s what I spent most of my working days on.

As a lead I suddenly had a wider variety of tasks on my plate, which left less time for the hands-on work that I did before. Instead I was invited to meetings where topics were discussed that went beyond my own work or even my team’s work. I learned about the bigger picture and about company-wide initiatives. I started to think about how my team fits into that bigger picture.

At the same time, I started to do hiring from scratch, which meant defining a job profile, writing job ads, creating case studies and coming up with a process of interviewing candidates. Before the candidates were hired I created on-boarding tasks for them, related to the job profile according to which they were hired.

These are examples of tasks that you can at least somewhat pinpoint in terms of time spent. There will be other things that you spend your time on, which are almost impossible to pinpoint. There will be days where you will have your meetings spread across the whole day, you will answer some messages here and there, and suddenly the day is over and you are not sure what you actually did that day. You will miss seeing the steady progress of your previous hands-on work. You will notice the conspicuous lack of your name on valuable and challenging projects. And you will start questioning your usefulness to the team, especially if you were a strong IC before you became a lead.

The worst thing is that you might not be the only one in your team wondering about your value contribution. The individual contributors that you used to collaborate with on projects will notice your lack of participation as well. Of course, they know that you have become a lead. But all they will see for a while is that you’ve stopped contributing as you did before. All the other tasks that you have picked up will not be visible to them. If your organisation has a 360º reviewing system, you might only learn about their thoughts in this anonymous way later.

I don’t have a good solution for this other than to have some patience and let time do its magic. Mindsets and perception can change slowly, even (or especially) your own. Give yourself some time to get comfortable with the idea that your value contribution is manifold now. In fact, the wider your scope grows the less you will probably add value by deep-diving into a project yourself. My experience is that also your co-workers’ perception will change in time as they see how you start to build and/or grow your team. Don’t get discouraged by these deceptive feelings in the beginning.

4. Attach your ego to your team’s value contribution.

As a strong IC you probably derived pride from the projects you delivered and the business problems you managed to solve with your ingenuity and perseverance. Get used to the idea that this source of ego boosting will become less and less significant for your daily work and for how you can measure your success within your organisation. Instead, as a people’s leader your focus should be on building a strong team and empowering your team to deliver the most business value while developing themselves along the career trajectory that gets them excited.

Depending on where you are on your ego curve this might be one of the hardest tasks for a new lead. You have to learn to take your own ego one notch (or several notches) down in order to let others shine. While most of us have learned at some point in life not to always put ourselves at the forefront in order to be socially adequate, this is only one of the puzzle pieces. Great leaders don’t only let their team shine, they take pride in the success of their team and viscerally get satisfaction from seeing their people living up to their potential.

I remember the first time when I decided not to contribute a technical talk to our company-internal tech conference. My ego suffered a hit and I had to admit to myself that I had really wanted that spotlight. But I realised before the conference that most of the technical work I had done was in a supportive role. If anyone should speak about those projects, it should be the team members that had contributed most, not me. In the end, all of the people in my team gave a talk at that edition of the conference. And I started to have an inkling about what experienced leaders must feel when they see their protegés kick ass.

5. Give feedback. Mostly acknowledgement.

The first time I gave a piece of behavioural feedback to an intern that I had hired was a revelation. For both of us.

I pointed out to him that he had the habit of interrupting the people who were explaining something to him (not just me, but also other team members). To me this came across as very impatient and even a bit rude.

It was obviously a kind of feedback that he had not expected to receive in a work context, and definitely not from his ‘boss’. For me it was an experiment as I wasn’t sure of the outcome. How would he take it? Would he actually act on this feedback? Would he get offended by it?

What I found out was that he really appreciated the feedback as he hadn’t noticed this behavioural pattern himself. He explained that he probably did it because he really wanted to show that he is up to speed, or in other words, he wanted to prove his worth by not requiring extra explanations. After that conversation he would become much more mindful about interrupting other people, having realised that he didn’t have to prove his worth in that manner. In our following 1:1s I made to sure to acknowledge the changed behaviour as I had observed it.

As an IC you might have never made an effort to make feedback-giving a habit of yours. As a leader, however, you have to take on a more proactive attitude towards feedback. Think about it this way: Now that your value contribution depends on your team’s development, you have a strong interest to do everything to help them to grow. And one of the most effective ways to do that is by providing feedback and following up on it. There have been books written about how to give feedback so I won’t get into it here. My belief is that if you give the feedback with the common understanding that you are invested in your team member’s growth, it will be received in the right way (even when given with radical candour).

Giving feedback also means giving acknowledgement where adequate. It doesn’t mean that you have to praise your team members at every occasion, but let them know that you are paying attention. Acknowledgement at the right place can sometimes go a long way in terms of motivation and positive reinforcement for developing in the right direction.

6. Do hands-on work.

This is the question that I hear most frequently from aspiring data scientists who are considering a leadership role: “How much time will I still have for hands-on data science stuff?” My answer is: It’s up to you, your team and your lead but make sure it’s not zero.

The truth is that it takes some effort to find enough time to do quality hands-on work while leading a team but my belief is that unless you have fully dedicated yourself all the way to the management path (and won’t ever go back to the IC path), you have to stay on top of your data science skills by practising them. Of course your analytical skills won’t disappear overnight but like many skills they need to be maintained in order to not deteriorate too much. Also, if your team is mostly made up of data scientists, keeping your toes wet will allow you to have a better understanding of their work and the challenges they face.

It’s a good idea even if you only think about your own sanity (read: motivation). In leadership you don’t very often get the kind of instant gratification as you can sometimes get with data science (or more technical work in general). Seeds that you plant in leadership take a long time to grow and very often you won’t be able to exactly pinpoint if it was your actions that made the difference.

So do yourself a favour and have a project that can get you into your usual flow state.

However, be mindful about the project you choose. Don’t let yourself become a bottleneck of a crucial and urgent project. If you pick up a highly exploratory project where you also have to pick up a lot of new skills in order to have a chance at success, make sure that your schedule can afford the extra learning time needed (e.g. when your team is in a stable stage and you don’t have to do any hiring).

Summary

Here are my lessons learned in brief:

1. Uncertainty is okay. Even more so than before.
2. Trust your intuition (and sometimes your reasoning).
3. Mindsets about value contribution change slowly.
4. Attach your ego to your team’s value contribution.
5. Give feedback. Mostly acknowledgement.
6. Do hands-on work.

Needless to say these are lessons based on my own experience and your mileage may vary. However, I do believe that some patterns emerge in many places and I’m curious which of these lessons ring true to other data science leads and where they might strongly disagree.