Chapter 4. FlySorter
An Automated Drosophila Classification System
I keep a wine fridge in my spare bedroom, and it’s full of fruit flies. On purpose.
Figure 4-1. The author’s Drosophila storage system
Just a few feet away from these unsuspecting flies is the prototype of a machine designed to efficiently isolate, image, and classify them, all in the name of science.
Drosophila melanogaster—commonly referred to as fruit flies or vinegar flies—are one of the most frequently used model organisms in biological research (Dietrich et al, 2014), in no small part because they are easy to keep and care for. They need little besides an inexpensive food source (which also serves as hydration) and space to grow, and then they multiply quickly. A 12” x 12” tray of small plastic vials can hold thousands of flies, each fly a tiny experimental subject. For under $100 total—for flies, vials, food, and, if you’re feeling rich, a $70 wine fridge from Craigslist—anyone can work with Drosophila.
Flies were first used to study genetics over 100 years ago, and researchers have developed powerful tools and systems that allow them to learn about disease, development, behavior, neuroscience, and much more, all using the humble fly (Stephenson & Metcalfe, 2013). For all the tools at scientists’ disposal, however, there is still a great deal of manual labor involved in fly experiments (Greenspan, 2004). It was a conversation with a friend about fly pushing that pushed this engineer to found a company and develop many prototypes to automatically sort fruit flies by sex, eye color, and other physical characteristics.
Why Sort?
Researchers have isolated thousands of different mutant strains of Drosophila as a way to investigate aspects of genetics, physiology, development, and other biological systems. Because most of these mutations are not readily apparent just looking at a fly, scientists have developed a method of linking other genes—ones that code for an externally visible characteristic like eye color, wing shape, production of a fluorescent protein, etc. (see Figure 4-2)—to the ones they’re interested in studying. This way, they can more easily distinguish flies that carry the mutation of interest from flies that do not.
Figure 4-2. Different fly phenotypes, from left: wild-type female, wild-type male, white eyes, curly wings, yellow body with white eyes
Labs also frequently isolate males and “virgin” females to set up a mating cross or to tally results at the end of an experiment.
Most labs use a similar approach for sorting:
- Flies are anesthetized with carbon dioxide gas, and the vial is emptied onto a small platform. Exposure to CO2 immobilizes Drosophila temporarily; they recover rapidly once returned to a normal atmosphere.1
- The platform—called a flypad—is placed under a stereomicroscope, where flies are examined, one at a time.
- Researchers move flies into piles on the pad according to physical characteristics, using a soft-bristled brush to avoid injury.
- Finally, the different piles of flies are transferred (either with the brush or with a mouth pipette) to appropriate destinations: different vials, various experimental assays, etc.
Though the process is relatively simple, it can be time consuming, especially when experiments include tens or hundreds of thousands of flies. I saw an opportunity to free up researchers’ time for less menial labor and to solve an interesting engineering problem while improving throughput and accuracy. I decided to automate this process.
Is This Even Possible?
Addressing feasibility early on in a project is good practice. You don’t have to build a fully working prototype to know whether something is doable, and more often than not, you shouldn’t—it’s a waste of time, energy, and/or money. I took a few different approaches.
First, I chatted with a number of computer vision and machine learning authorities to learn about imaging and classification technologies that might be helpful. Opinions ranged on the right algorithmic approach, but most agreed it was doable.
And as luck would have it, my brother is one of those experts. Matt Zucker is an Assistant Professor of Engineering at Swarthmore College, and studied computer vision and machine learning extensively in graduate school. Not only was I able to call him up with questions, but I later persuaded him to join FlySorter as a co-founder.
I built a number of prototype mechanisms in an effort to repeatably image and classify flies. I anesthetized and sent them through a vibrating hopper, hoping they would all come out in a particular orientation. I used air pumps to shoot them through tubes. I took apart web cameras to make cheap, high-resolution microscopes. I illuminated flies with infrared and UV light to try to find highly visible markers. And so on.
Finally, instead of trying to build a fully automatic system, I captured and manually classified hundreds of images of flies by sex, vial after vial, and trained machine learning software on the data. My first datasets yielded a 75%–80% success rate. While hardly good enough for a final product, the rate was very promising, considering how rough the images were.
So a few months in, I felt good. I had proved that machine learning software could differentiate between pictures of anesthetized male and female flies, and I had in mind a robotic platform that would capture images of flies on a pad and then use air pressure to sort them into vials, once software had done the classification.
Making Stuff
I’ve always been a maker and tinkerer, but to produce something of this complexity required more than just buying parts from McMaster-Carr or Digi-Key. Having used 3D printers extensively at previous jobs, I knew how powerful they could be as a prototyping tool. I did my research, and I bought the parts to build a Kossel Mini (Rocholl, 2012).
The printer was more than just a way for me to make plastic parts. The Kossel Mini is an open source printer, meaning the design is freely available for others to use and modify. It is a delta robot, a platform also well suited for moving a camera and fly manipulator quickly.
I replaced the plastic extruding head on the printer with one that held a high-resolution digital camera and a set of tubes connected to vacuum pumps, and replaced the print bed with a flypad. After some months of tinkering, I had a machine that could image flies on a flypad automatically, and subsequently move individuals to one of three vials atop the machine, using vacuum pumps. Thus, the first version of the sorter was born (see Figure 4-3).
Figure 4-3. Left: Kossel Mini 3D printer; right: FlySorter DCS, version 1.0
Because we were imaging anesthetized organisms, the flies were oriented more or less randomly on the pad: on their backs, on their sides, on their bellies, or legs all akimbo. To account for this inconsistency, we developed a two-step classification process. First, we used a texture classification technique to identify fly body parts in the image (Varma & Zisserman, 2005).
Using the labeled image, we could then orient the image and narrow our analysis to a region of interest. For example, to classify by sex, we rotated each image so the head pointed up and then cropped out everything but the abdomen (where the most significant differences between males and females would be obvious). We passed that standardized image to a second classifier to determine the sex of the fly (Shan, 2012). See Figure 4-4.
Figure 4-4. Sex classification algorithm
Close, But...
I took a step back to evaluate how well our approach worked. On the plus side, the prototype was relatively compact, not too expensive (around $1,200 in parts), and sorted flies fairly well. After many iterations on the code, Matt and I were able to achieve 97% accuracy sorting flies by sex.
Unfortunately, there were some real limitations. The design required users not only to empty a vial of flies onto the pad, but also to spread them out so that each was a few millimeters away from its neighbors. This step (which prevented two flies from getting sucked up at once) took a minute or two for a vial of 50 flies.
And there was the issue of accuracy. While 97% accuracy doesn’t sound too shabby, that translates to an average of three mistakes per hundred flies. Sorting by sex is commonly followed by making a cross for breeding, and with a spurious male in a vial of females, that resultant cross wouldn’t be worth much. To have a viable product, we needed closer to 99.9%, or better.
We considered our options to improve accuracy, and ultimately decided all the changes that might work involved trade-offs that would render the final product either too expensive, slow, or cumbersome to use. We’d hit a dead end.
A New Strategy
When we reconsidered the decisions leading to this fly sorting cul-de-sac, there was one that begged to be revisited: should we image awake or anesthetized flies? Flies under anesthesia don’t move around, of course, but are challenging to work with. Their legs get stuck together like Velcro, and the flies are difficult to place in a consistent orientation. As we considered ways to image nonanesthetized flies, we learned that someone else—a group at ETH Zurich—had successfully built a sorter! Reading the journal article about their system, dubbed the FlyCatwalk, was simultaneously exciting and gut-wrenching (Medici et al, 2015). Although the article validated the purpose and feasibility of our project, someone had beaten us to the punch.
On the third or fourth read through the paper describing their very clever system—they imaged flies walking in the narrow gap between two clear plastic sheets—a few things became clear. First, using nonanesthetized flies was the way to go. It made the computer vision and machine learning code a lot simpler. And second, the FlyCatwalk had much room for improvement. Their system is quite large, taking up considerable space on a bench. Additionally, the throughput of the FlyCatwalk is quite low, about one fly every 40 seconds, and some flies never even make it out of the vials to be imaged.
Prompted by their success, and considering the limitations of their system, Matt and I discussed different ways we could usher individual, awake flies into an imaging chamber like the FlyCatwalk. We hit on one that sounded promising, essentially a fly “valve” made from soft foam rollers, and thus began another round of designing and building.
More than 50 prototypes later, we’d created a small device to automatically dispense individual flies, awake, one every few seconds. There were, of course, many details to work out—how to avoid dispensing more than one fly at a time, how to avoid damaging flies, how to make it work quickly, etc.—but ultimately the machine hewed very close to our valve idea, using two motorized foam rollers to create a barrier with flow control for flies (see Figure 4-5).
Figure 4-5. Automatic fly dispenser
The dispenser is a big step toward a high throughput classification system. As an added bonus, we’ve connected with researchers who study fly behavior and learned that some are eager to use the dispenser on its own, without any imaging at all. Loading up their experimental assays one fly at a time can take a long time, and this device not only speeds up the process, but it also avoids any potential side effects from anesthesia or human chemistry (by way of the mouth pipette). In start-up terms, we’ve found our minimum viable product, and by the time this article is published, I expect we’ll have made our first sale.
What’s Next?
We’re glad to have built relationships with several high-profile labs around the country; their feedback has been and will continue to be extremely valuable, and the possibilities with collaboration are very exciting. It’s going to be a busy six months at FlySorter. In addition to bringing the dispenser to market, we plan to prototype the next-generation classification system, and we have a few other projects in the works as well.
While I never imagined I would be wrangling fruit flies this way, I’m glad to have found this overlap between engineering and science. It’s satisfying and rewarding to launch the results of two years of work publicly, and I’m excited to take on the challenges ahead.
1 Prolonged exposure can be damaging or lethal, and even with shorter exposures, some effects can linger. For example, CO2 can alter a fly’s sense of taste and smell for hours. Some labs expose flies to cold temperatures as an alternative to CO2, to avoid these side effects.
Get BioCoder #9 now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
