Uncertainty about where your life is going next is tough to deal with. You can sit around and wait for something to happen to you. You can prepare for a particular outcome and then have that be nothing like what you prepared for. Or you can prepare for multiple different plausible outcomes, spread your already-overcommitted energies even thinner, and still have nothing like any of your anticipated alternatives come to pass. I’m starting to better appreciate the extent to which this has driven large parts of my career.
My own history as a programmer
I’m pretty well known within the astronomy community as a software designer — at least, I would say, well enough to be recruited for postdoctoral positions and invited for talks at conferences based on my large-scale data pipeline design expertise, rather than my scientific accomplishments. This is a frustrating thing.
Computers and programming have always been a familiar thing to me: my family got an Atari 800XL when I was around eight years old, which featured programming in BASIC and offline storage on cassette tapes. A succession of newer and more sophisticated home computers, attendance at a high school with a Computer Systems research lab and advanced computer science coursework such as Systems Architecture and Artificial Intelligence, and ongoing exposure through self-study meant I could teach myself pretty much any language and do sophisticated things in it quickly. Together with friends I was designing computer games with simple sprite graphics and exploring object-oriented programming and parallel programming in high school. I was way ahead of the curve as far as computers were concerned; I had every advantage.
Starting in undergrad, I really bent my energies more towards physics and math, and although my computer skills were still coming in handy, I updated them only occasionally or as needed. I learned proper object-oriented C (as C++) in grad school, picked up perl and SQL in my first postdoc and Python in my second one. Here at ANU I’ve learned object-oriented Python, and gotten some experience with Django and CSS. But it’s mostly been little bits of things here and there. The overriding concern in science is getting papers out. What’s more, in my postdocs so far I have usually been forced by necessity to maintain pre-existing software systems that were never really designed so much as accreted, with the accompanying labor overhead associated with maintaining something that wasn’t written to be maintained.
At first I took it personally and blamed specific people for my poor luck. But I’ve come to realize that I’m probably only marginally better, when it comes to really big systems like the ones that make surveys like SkyMapper go forward. The design for the SkyMapper pipeline isn’t terrible; Fang Yuan and I have put a lot of thought into it, and we’re both personally responsible for the fact that it works, and works to spec. But I wonder how much better it could have been if we had both been trained in software design: good coding habits, best practices for design, project management methodologies like Agile, and so forth.
I’m now worrying about this from a career management point of view: I’ve always imagined that I could transfer to software engineering if physics didn’t work out, but this may not be as easy a transition as I had at first imagined. The fact is that opportunities to learn from real software experts as a scientist are probably not that hard to come by. But at least in my past experience, we were rarely encouraged to consult and learn from those people, and perhaps tacitly discouraged from doing so by the incentives at work in our employment environments.
Readers who are experienced with software, let me know whether the following claims are reasonable. As a software developer, assuming your workplace is reasonably functional, your task is clear: design a software product that works, on time and on budget. If your workplace is super-functional, you might also concern yourself with the problem of designing a software product that works well and is easily maintainable and extensible into the future. But you’re being paid to do one thing, and doing that thing exceptionally well will presumably also advance your career and allow you to move to positions of greater influence and responsibility.
In science, if your workplace is functional you’ll have to split your energy between writing software for an experiment and publishing results associated with that experiment. If your workplace is dysfunctional you can easily spend your whole time writing software, which is what you were nominally hired to do, but spend virtually no time publishing first-author papers, which is what you need to do to get your next job. If your experiment is small and may never be repeated, you have no accountability to anyone but yourself to make sure that your code is well-documented, reusable, or maintainable. While it is important to large, ongoing projects (such as wide-area sky surveys) that software be designed well, in practice you rarely get the eyes of a real expert under the hood and so the design can be pretty horrendous. You might know how it works, but heaven help the person who comes after you. Any time you spend developing skills to write good software takes precious time away from writing the papers you’ll need to get your next job. Or even just getting the code working so you can show more tangible results to your supervisor — a plot made, a supernova discovered. The breakneck speed both of scientific discovery and technological progress threatens to keep our view short-term, looking only one project or one job ahead.
Moreover, because you’re learning to write and maintain bad software without the domain knowledge typically used in industry, your skills may not be as immediately transferrable as you think. I’ve been reading about the coding interviews used in top software firms such as Microsoft, Google, and Amazon; similar interview practices are in place for “Data Scientist” positions, not just software engineer positions. They involve (among other things) coding base-level algorithms such as linked lists, hash maps, binary search trees, and quick sort, at a whiteboard under time pressure in front of an interviewer; I haven’t had to do things like that since AP Computer Science in high school. I know what all these data structures are, but (haha) isn’t this kind of tedious quiz-work what the standard library implementations are meant to free you from? I worry about the efficiency of my code, but usually only if it’s slower than I need it to be to solve the problem I’m trying to solve, and that’s a pretty loose criterion. I try to comment my code well, and to make sensible class hierarchies, and to only check working versions of things into the revision control system; I would say this is probably more than about 90% of my scientist colleagues do. But I don’t usually give myself time to implement well-known design patterns like Model-View-Controller. Or write unit tests. Or use distutils to make a Python module that can be easily shared or distributed on GitHub.
In short, I know some of the things I’ll need to know but have only ever taken a very short-term view towards maybe picking them up someday. If I’m looking to make a career switch, now’s probably the time, and I can certainly hope that the act of doing so will accelerate the software I have to write and share. But let’s not pretend that’s easy, or guaranteed.
The factors I’ve mentioned above are slowly changing. Senior scientists are beginning to realize that both the needs of modern, computationally-intensive science and the career landscape for scientists are very different from when they were students and postdocs. The younger generation is adopting more best practices of software design, and is aggressively promoting open-source science as well as open-source software infrastructure for science. There are some groups which are clear models for effective software production in astronomy, which is also discussed on popular professional blogs. Young scientists ignore these resources, and the trends they represent, at their peril.
How this is affecting my plans going forward
I’m still holding out hope for an academic career. Fortunately, since much of what I do is closer to software engineering than the typical postdoc, there’s probably time for me to make some adjustments to be more competitive in an industry setting. What’s more, I may be able to chart a course that does double duty — e.g., publishing papers about applications of machine learning to problems faced by next-generation astronomical surveys. It’s important that I get my big science results out as well — that’s mostly going to happen this year.
I also intend to make a big push this year to publish any piece of code that’s worth sharing. This also will benefit both my scientific and software careers:
As a scientist, putting my code out there will encourage other scientists to use it, make my process more transparent, get my work cited and increase its impact. David Hogg argued strongly at the last winter AAS meeting that the benefits of releasing code publicly far outweigh the disadvantages, for these and other reasons. Many people argue that if they release their code to the public, others will scoop them. But in general, putting your name out on it means you’ve established priority — others will see you as the expert and not try to scoop you (in general)! This might not be true for certain very competitive subfields, but in general, people want to scoop each other on Nature-style results, which I would bet generally aren’t the results enabled by releasing code. Regarding my specific code and the problem it was designed to solve, I want to turn e.g. my bolometric light curve fitter into a general-purpose supernova analysis toolkit, and I can leverage the contributions of others if I release my code (along with a methods paper targeted for PASA, which others can cite when they use or extend my code). The real scientific value is no longer in the code itself, but in the way it’s used — the problems to which it’s applied, and (in the Bayesian analysis case) the particular priors used for the modeling. If these problems and assumptions were obvious, we would already know a lot more about supernovae than we do; people out there have good data, but they’re using the same old assumptions all the time, when a little more thought would go a long way.
As a software developer, making my code open-source puts it out there for potential employers to see. It also puts it out there for other software developers to see, from whom I can learn some of the abovementioned finer points of the software design craft through hands-on problem solving, as what would otherwise be pretty ugly code becomes improved. (“If you’re not embarrassed, you took too long to release it!”) If I’m really lucky, some of them might even have future job leads — but let’s not count those unhatched chickens just yet.