Prologue
In 2007, using large-scale data to train computer algorithms wasn't an obvious idea. If large-scale data was going to become a part of the trifecta of technological trends – alongside GPUs and neural networks – that sparked the deep learning era, in which OpenAI was founded, then, once again, it wasn't such an obvious idea.
As you might be beginning to sense – ImageNet, the large-scale dataset, is hitched to the cross-disciplinary contributions that came before it. And as naturalist John Muir, Father of the National Parks, wrote in My First Summer in the Sierra, "When we try to pick out anything by itself, we find it hitched to everything else in the universe."
The story of ImageNet begins with three words – three words that a twelve-year-old Fei-Fei Li overheard her parents whisper at home one night in 1989. Those three words would culminate two decades later in the publication of ImageNet – which, in 2009, was the largest human-curated dataset in the history of AI.
Fei-Fei Li was born in 1976, in Beijing, China. But she was raised in Chengdu – over a thousand miles south-west of Beijing – until she was fifteen years old. Then, in 1992, Fei-Fei and her mother immigrated to the United States to join her father, who had immigrated three years earlier. That move, as Fei-Fei would later describe it on The Tim Ferriss Show, might as well have been to a different planet.
Finally reunited as a family in Parsippany, New Jersey – about thirty miles north-west of New York City – they began their life in the West. Fei-Fei would later recall her mother saying, "I had no plan coming to New Jersey, but I think I'm going to survive. I just believe I'm going to survive, and I'm going to make sure Fei-Fei survives."
Those three words that a twelve-year-old Fei-Fei Li overheard her parents whisper – education, opportunity, and freedom – were quietly becoming a reality.
The Early Life of Fei-Fei Li
Parsippany High School
Forty-eight hours after arriving in the United States, Fei-Fei had her first day at school. The basic English she had learnt was useless in practice. Learning to express herself, not pursuing her love of physics, would be the first challenge to consume her.
"I really think these public teachers in America are the unsung heroes of our society," Fei-Fei Li would later say on The Tim Ferriss Show. For Fei-Fei, one of those unsung heroes was Mr. Robert 'Bob' Sabella – the head math instructor at Parsippany High School, who started out as just her teacher.
Mr. Sabella's office, unofficially christened the Math Lab, soon became a refuge for Fei-Fei. It was a safe space where Fei-Fei, a self-described nerdy immigrant teenager, could be tutored and rediscover the simple joy of a conversation. When the school couldn't offer a Calculus BC class, it was Mr. Sabella who gave up his lunch hour – his only lunch hour – to tutor Fei-Fei in the Math Lab.
It was those moments in the Math Lab that Fei-Fei credits with helping her walk in the footsteps of her greatest hero – Albert Einstein – who had lived and worked approximately forty miles south-west of Parsippany.
"I think I know what this means," Mr. Sabella said, squinting at the letter Fei-Fei showed him, with an orange and black shield emblazoned on it, "but I want to be sure."
Three words had brought Fei-Fei Li to the United States – education, opportunity, and freedom. Three years later, those same three words helped the only daughter of a poor immigrant family secure an almost full scholarship to Princeton University.
Princeton University
Two years later, in 1997, Garry Kasparov became the first world chess champion to lose to a computer – IBM Deep Blue. Meanwhile, Fei-Fei was halfway through her physics undergraduate degree at Princeton University – a place she describes as her first breath of truly fresh air since immigrating to the United States.
Fei-Fei had become ever more interested in the minds behind the ideas that had captured her imagination. She devoured biographies of the greats in her field of study, physics – Einstein, Feynman, Bohr, and many more. Unexpectedly, she started to notice a pattern among them – one she was noticing in herself. They developed an interest in the mystery of life, and sometimes pivoted towards the study of biology.
"You won't believe what happened today, Mr. Sabella," Fei-Fei told the big bearded math teacher, as her mother affectionately called him. By this point, Mr. Sabella had become more than just her old teacher. He had become her American friend. And he and his wife Jean had become her American family.
In the summer of 1997, Fei-Fei secured a place as an assistant on an experiment in the Department of Cellular and Molecular Biology at the University of California, Berkeley.
The experiment combined two disciplines – neuroscience and biology – in an attempt to understand how vision worked. More specifically, the experiment would project imagery upon a cat's eyes and analyse its neural response, then attempt to reconstruct the imagery that caused it – from inside its brain.
The anaesthetised cat lay on the table with electrodes probing its brain. Wires connected the electrodes to an amplifier that translated spikes of activity in its visual cortex into an audio signal. And what the cat saw, Fei-Fei heard.
As Fei-Fei watched that experiment unfold before her eyes, she could feel something changing within her. That something would soon consume the next decade of her life.
Within two years of submission, their work was published in The Journal of Neuroscience.
A few months after that experiment, in the autumn of 1997, Fei-Fei undertook a project closer to home – one meant to heal old scars and build a more peaceful future.
Fei-Fei commemorated the sixtieth anniversary of the Nanking Massacre with an international conference – the Princeton University Nanking 1937 Conference – that brought together scholars from around the world with different viewpoints on the massacre. The Nanking Massacre – inflicted on the Chinese by the Japanese Imperial Army during World War II – is considered one of the worst wartime atrocities in history.
Fei-Fei Li, Mr. Robert 'Bob' Sabella, and David T. Liu then edited and published a book based on the conference. I somehow managed to get a used copy of that book – Nanking 1937: Memory and Healing – from a local bookseller.
"It really helped me have an understanding of humanity," Fei-Fei said of the conference. "As scientists, we are so involved in formulas and numbers, sometimes we lose sight of human issues."
That human-centred thinking is something I believe she inherited largely from her mother. It's something I've noticed throughout everything of hers I've read and heard.
Beyond the study of physics, physics had helped Fei-Fei learn one of the most important skills – how to ask audacious questions. She found herself searching for that one question – a north star she could chase. Her parents chased their north star – education, opportunity, and freedom. Now Fei-Fei was searching for her own.
In 1999, Fei-Fei Li graduated from Princeton University as a physics major. She also received certificates in engineering physics, and computational and applied mathematics.
Tibet
After graduating from Princeton, Fei-Fei spent a year on the roof of the world – Lhasa, Tibet – working on a project titled 'Exploring Tibetan Medicine, An Untapped Resource'.
"If you ask me what is the number one thing I love the most of Tibet," Fei-Fei wrote of her experience, "I would have answered you without a second thought. It is her people, the sons and daughters of this land of snows. During my year-long stay and the numerous following trips in Tibet, I had the chance to meet and befriend with all kinds of Tibetans, from nomads to farmers, monks to students, urban youth to college professors."
The $20,000 Martin Dale Fellowship she was awarded by Princeton University made that year-long project in Tibet possible. The criteria for the fellowship judged candidates by the potential impact their project could have on their own futures, on American society, or on the international community.
The fellowship panelists weren't to know it – and neither, perhaps, was Fei-Fei – but in time, she was going to have an impact on all three criteria.
Caltech (One-Shot Learning)
"Fei-Fei, is that what you want?" her mother asked.
The dot-boom was at its peak, and Wall Street was competing to hire anyone who came with a stamp of approval from the right school. Self-described geeks like Fei-Fei were their target. They offered packages that would promote Fei-Fei and her parents into the middle class – realising the American dream – less than a decade after immigrating to the country. But in return, they asked that she trade science for finance.
"You know what I want, Mom. I want to be a scientist," Fei-Fei replied.
"So what are we even talking about?" her mother retorted.
The decision was made. In 2000, Fei-Fei started her graduate studies at the California Institute of Technology (Caltech). Again, she was walking in the footsteps of her hero – Einstein – who had spent three winters there as a visiting scientist. She would earn a Master's and a PhD in electrical engineering – combining neuroscience and computation. Caltech, she would later say, is where she became a budding AI scientist.
In her second year at Caltech – during her PhD – Fei-Fei hit a brick wall. It prompted her to swap the warm Pasadena colours for the cool Oxford colours – for three months.
Fei-Fei started working on one-shot learning of objects. She wanted to use one, two, or three images to enable algorithms to recognise objects. However, she struggled with the lack of images. At that time, digital cameras weren't mainstream. The internet as we know it was in its infancy. And the likes of Google Images had only just launched.
In her memoir, The Worlds I See, Fei-Fei Li describes the hypothesis: "By exposing the algorithm to a broad cross section of the visual world first, it would be better equipped to learn something specific."
After being shown hundreds of unrelated images – cats, motorcycles, faces, and more – the algorithm, surprisingly, was able to recognise an airplane, having seen only one. Fei-Fei – alongside Rob Fergus and her advisor Pietro Perona – validated that algorithms, like humans, benefit from seeing more of the visual world.
In hindsight, it seems obvious that algorithms, like humans, benefit from seeing more of the visual world – so they can recognise objects after a single glimpse. The simple reason for that is we bring a lifetime of experience to bear on it.
Consider a child's eye, for example, as a pair of biological cameras that take a picture every two hundred milliseconds – the average time an eye stays still to process what it's looking at. By the age of three, that child would have seen hundreds of millions of pictures. Therefore, it's hard to imagine seeing anything in isolation.
The resulting paper – 'A Bayesian Approach to Unsupervised One-Shot Learning of Object Categories' – was a breakout success. It secured Fei-Fei one of the few oral presentation spots at the 2003 International Conference on Computer Vision (ICCV).
At the conference, Fei-Fei was repeatedly asked about the mathematical core of the algorithm they had chosen – a probabilistic technique called a Bayesian network.
The Bayesian network is named after Reverend Thomas Bayes, a Presbyterian minister, born in England in either 1701 or 1702. Bayes died in 1761, at the age of 59, having published only two works – a treatise on divinity, and a defence of Sir Isaac Newton's calculus. Two years after his death, Bayes's work on probability was published by his friend Reverend Richard Price – considered the greatest Welsh thinker of all time.
The mention of Newton reminds me of a pun from Ted Lasso in season three, episode nine: "You know what my favourite thing was about Sir Isaac Newton? He was so down to Earth."
Pun aside, the story of ImageNet – and, more broadly, the story of innovation – is a non-linear lineage of inputs, influences, and interdependencies that enable future innovations. The Bayesian network is one example.
Returning to the conference, no one was asking about the images that the algorithm was trained on. But it was those images – even though there were only hundreds of them – that enabled the algorithm to recognise objects after as little as a single exposure.
Fei-Fei ruminated on the lack of attention paid to the image dataset at the conference. It raised a question – that one question – she had been searching for since graduating from Princeton University: "If so little data could make such a powerful capability possible, what might more data enable?"
Fei-Fei had found her north star – one way or another, she was going to make the visual world familiar to machines.
Caltech (Caltech 101)
Back at Caltech, Fei-Fei indulged her curiosity. How many categories should her image dataset include? The biggest image dataset curated at Caltech included seven categories.
"Ten? Twenty?" Fei-Fei said.
"Let's do a hundred, Fei-Fei," Pietro said boldly.
Fei-Fei's initial look of death soon gave way to excitement, and she started curating the one-hundred-category image dataset – a number seemingly pulled out of thin air.
The first challenge in curating the dataset was deciding which categories to include – without bias. A Merriam-Webster's Collegiate Dictionary – something Fei-Fei had been hauling around since her days as an English-as-a-second-language student – had now become her most useful tool. Flipping through its pages, she selected a subset of categories that had drawings.
The next challenge was sticking with the tedious process of collecting and labelling images. Fei-Fei, assisted by a few students, queried Google Images to collect as many images as possible per category. It was an unglamorous process that took months.
In 2004, the dataset was completed and christened Caltech 101. At the time – with nine thousand images across one hundred and one categories – it was the largest image dataset ever curated for machine learning. Fei-Fei had added one more category as a playful nod to her advisor, who she describes as being "intelligent, adventurous, and silly, all at once". Some of those traits, I believe, took root in Fei-Fei.
Caltech 101 prompted a follow-on paper to the one presented at the 2003 International Conference on Computer Vision (ICCV) – 'Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories'. Within six months, it had become the computer vision benchmark to beat.
The follow-on paper – similar to the one presented in 2003 at ICCV – again validated that computer vision algorithms performed better after seeing more of the visual world. The field, however, continued to fixate on algorithms that tried to tame complexity. This stood in contrast to Fei-Fei's fixation on the unreasonable effectiveness of data.
WordNet
It was 2005, and Fei-Fei was now an assistant professor at the University of Illinois Urbana-Champaign. She had been invited back to Princeton University to share her latest work with the computer science department.
"Have you heard of a project called WordNet?" asked linguist Christiane Fellbaum.
WordNet, Fei-Fei was about to learn, was the creation of Princeton psychologist George Armitage Miller. Miller was born in 1920, in the United States. He died in 2012, at the age of 92. With WordNet, he had built a structure for language – in contrast to alphabetical dictionaries – where over 150,000 words were shown in relation to one another. For example, "dog" would be nested under "canine", and "canine" under "mammal".
In a 1995 article by Miller on WordNet, one sentence almost reads like a description of the LLMs that would launch to the public over two decades later: "Because meaningful sentences are composed of meaningful words, any system that hopes to process natural languages as people do must have information about words and their meanings."
Christiane then mentioned an abandoned project that attempted to visualise every concept in WordNet with a single image. The project, and its name, was a green light for Fei-Fei in her pursuit of making the visual world familiar to machines. And it made her ponder: "What if a data set like Caltech 101 were created on the scale of WordNet?"
Awkwardly, she asked Christiane, "So... does that mean I can use the name?" – ImageNet.
The Story of ImageNet
Creating ImageNet
After just over a year as an assistant professor at the University of Illinois Urbana-Champaign, Princeton University offered Fei-Fei the same position – assistant professor – in the computer science department. Fei-Fei would later describe that moment as one of the happiest in her life – a moment when she felt validated.
Work on ImageNet started with WordNet as its foundation. The initial step was to cut the 150,000-plus words in WordNet down to a visual subset. Fei-Fei and Jia Deng – whom Fei-Fei credits as the most important person in the history of ImageNet – trimmed it to just over 22,000 object categories. Although considerably reduced, at the time, it was still orders of magnitude larger than any other image dataset for machine learning.
Surprisingly, 22,000 object categories was close to the 30,000 objects that vision scientist Irving Biederman estimated the average human can recognise. It was a figure that Fei-Fei had come across in her cross-disciplinary readings.
With the initial step complete, Fei-Fei attempted to curate ImageNet by adopting a similar manual process she had used for Caltech 101. However, because ImageNet was orders of magnitude larger – a thousand images for each of its 22,000 categories – it wasn't something Fei-Fei and Jia could do alone. Therefore, they started to hire undergraduates at ten dollars an hour to download and accurately label images.
To ensure enough images were downloaded and labelled, they expanded each query by appending the parent category. For example, when querying "Whippet", they would also query "Whippet Dog". To further ensure there were enough images, they queried the translated terms from the Chinese, Spanish, Dutch, and Italian versions of WordNet.
Progress was steady but slow, and some napkin math suggested that continuing to curate ImageNet by adopting this process would take nineteen years. "I believe in this project, I really do," said Jia, "but I can't wait that long for my PhD."
The human element – the undergraduates – was the bottleneck. The only way to alleviate that bottleneck was to hire an army of undergraduates. But that would require more funding – none of which was making its way to the project. Fei-Fei failed to receive the federal grants she had applied for. What she did receive, frustratingly, were comments saying the only strength of her application was that she was a woman.
Fei-Fei and Jia realised the audacity of the project, and then set out to tame it like a wild animal. Jia went through each step in the process, optimising the tedious steps that didn't require human intelligence – scraping image search engines, for example. When his solutions broke, he'd find another – for example, implementing a dynamic IP address so he could continue scraping image search engines without getting blocked.
With those optimisations, the dataset started to grow – thousands of images were being added and labelled. That nineteen-year estimate, however, had barely shrunk. To their dismay, they had only managed to shrink it to eighteen years.
The desperation to shrink the eighteen-year estimate bred alternative solutions. One solution they debated was using Fei-Fei's work on one-shot learning to label images in batches. But after some reasoning and tinkering, they decided against it. After all, if an algorithm could already accurately label images, ImageNet would be redundant.
More importantly, they decided against the alternative solutions for philosophical reasons. As Fei-Fei explains in her memoir, "Our goal was to embed unalloyed human perception in every image, in the hopes that a computer vision model trained on the complete set would be imbued with some similar spark of acumen. Machine influence threatened to dilute that."
As artificial intelligence becomes more widespread and powerful, we'll likely see more of these technical-versus-philosophical debates – I hope, at least. It is, however, reassuring to know that there are people at the frontier of this field – Fei-Fei among them – who bring philosophical considerations to the forefront.
Let's extrapolate and consider this – whether you're listening, reading, or watching it. First, thank you. Second, an LLM can output texts of varying length – and perhaps depth – on Fei-Fei Li and ImageNet. However, I created this without an LLM. I read source material and transferred my notes to index cards – which I'd happily share.
The goal wasn't to flood the internet with more information. As Derek Sivers said, "If more information was the answer, then we'd all be billionaires with perfect abs."
The goal was to tell a deeply researched story on human innovation – by a human. The use of an LLM would dilute that. It would, therefore, be perverse to dumb down any of it, including the writing, in an attempt to convince some anonymous commenter that an LLM didn't create it – I'll own that en dash, thank you.
Bringing philosophical considerations back to the forefront, Fei-Fei and Jia – deflated after multiple failed attempts – were beginning to feel, like many innovators before them, that they were ahead of what was practical for the time.
Feeling down and out about the progress of ImageNet, Fei-Fei drove twelve hundred monotonous miles north-west – from Princeton to Minneapolis – to attend the 2007 Computer Vision and Pattern Recognition (CVPR) conference. At the conference, she confided in her academic grandfather, Jitendra Malik – he had been the advisor of her advisor, Pietro Perona.
"The trick to science is to grow with your field," said Jitendra – knocking the wind out of Fei-Fei – "not to leap so far ahead of it."
Fei-Fei could have pivoted her gaze towards the field's north star – algorithms. But in moments like these, you sometimes – sometimes – need the confidence to override people with more credentials than you.
"Have you heard of crowdsourcing?" Min, a master's student, asked Fei-Fei as she rushed to a faculty meeting. "Amazon is offering it as a service, if you're interested. It's called Mechanical Turk."
Amazon Mechanical Turk (AMT) – as a service – was conceived in a 2001 United States patent disclosure by Indian entrepreneur Venky Harinarayan. Junglee Corp, a pioneer in internet comparison shopping which he previously co-founded, had been acquired by Amazon in 1998 for 1.6 million shares – worth approximately $250 million at the time.
AMT launched as a platform in 2005. Employers, known as requesters, could post HITs (Human Intelligence Tasks) – otherwise known as jobs. And workers, known as turkers, could browse the HITs and complete them for money.
The name of the platform itself – Amazon Mechanical Turk – was apt. It was inspired by the chess-playing automaton – the Mechanical Turk – unveiled by Wolfgang von Kempelen in 1770. The automaton had toured Europe and beaten Benjamin Franklin, Napoleon Bonaparte, and others at chess. It was, however, eventually exposed as a hoax. A chess master hidden in its base had been making the moves.
The opportunity that AMT offered ImageNet met the prepared mind of Fei-Fei – a mind prepared by years of conscious and unconscious work on ImageNet. To say that AMT changed ImageNet's trajectory would be an understatement.
AMT's international team of tens, hundreds, then thousands of turkers shrank the estimated eighteen-year completion time for ImageNet to nine, six, then less than three years. At the same time, it relieved the previous bottleneck of funding an army of labellers. AMT's international workers, hireable for a fraction of the cost of local labellers, were what made it invaluable for the large-scale labelling ImageNet required.
To avoid thinking of turkers as an army of international ghost workers, I turned to research from Panos Ipeirotis titled 'Demographics of Mechanical Turk'. His research collected demographic data from a thousand turkers across sixty-six countries.
The research showed that the United States had the most turkers at 46.8%, trailed by India at 34%. Most turkers in the United States were female, whereas most in India were male. And most of the turkers in the United States considered AMT a secondary source of income, whereas most in India considered it a primary source of income.
On HITs – otherwise known as jobs – the research showed that most turkers completed twenty to a hundred per week. That would earn them a low income of less than a hundred dollars per month. However, the few turkers who considered AMT a primary source of income could earn a thousand dollars or more per month.
In a 2011 Google TechTalk, Fei-Fei revealed they paid turkers four cents to categorise three hundred images. A 2012 New York Times article by John Markoff revealed that the average turker could categorise roughly two hundred and fifty images in five minutes.
Using AMT for the large-scale labelling that ImageNet required, however, wasn't as simple as posting HITs and hiring turkers. As humans are prone to mistakes and shortcuts, the HITs needed to be well-designed to ensure that a large, accurate dataset would emerge. A large inaccurate dataset would be next to useless for making the visual world familiar to machines.
The HIT that Fei-Fei and the wider team designed presented turkers with a set of images and a definition for a target synset – short for synonym set – including a link to its Wikipedia page. With that information provided to eliminate any ambiguities, turkers were tasked with verifying whether each image contained objects of the synset. They were also tasked with labelling the images until a confidence score was reached.
Even after hiring an international team of thousands of turkers, the ImageNet paper wasn't published until 2009, two and a half years after Fei-Fei had heard about AMT – prior to its publication, she had gone West and joined the faculty at Stanford University. At the time of the paper's publication, ImageNet included 3.2 million images across 5,247 categories, sorted into 12 subtrees.
The ImageNet dataset itself remained a work-in-progress. Over the next two years, it would grow to include 14,197,122 images across 21,841 categories.
Completing ImageNet
Despite funding, scale, and technology constraints, Fei-Fei Li, Jia Deng, and the wider team across Princeton and Stanford University had succeeded in creating a dataset like Caltech 101 on the scale of WordNet. At the time of its publication, ImageNet was orders of magnitude larger than any other dataset ever curated for machine learning – it was also more diverse, accurate, and hierarchical.
ImageNet owed its existence to WordNet, similar to how future datasets like ActivityNet, EventNet, MusicNet, and others owed their existence to ImageNet. It also owed its existence to multiple technological threads that serendipitously converged – the internet, large-scale data, cameras, search engines, and crowdsourcing platforms.
Two years after attending the 2007 CVPR conference in Minneapolis, ImageNet was accepted into the 2009 CVPR conference in Miami – albeit as a poster session, where attendees might stop and ask questions, not as an anticipated presentation in a hall. ImageNet, as a result, was largely ignored.
Fei-Fei was prepared for her bet on ImageNet – the bet that she hoped would fulfil her north star of making the visual world familiar to machines – to be proven right or wrong. Either outcome would provide a path forward. But being ignored was a stalemate – it made her question the bet she'd made.
Fei-Fei must have felt enormous pleasure in completing ImageNet. But she must have felt equally enormous pain seeing years of her work ignored. It reminds me of what Leonardo da Vinci wrote on his drawing, the Allegory of Pleasure and Pain – "Pleasure and pain are represented as twins, because there is never one without the other."
She had hoped that ImageNet would become a primary resource for computer vision research – a benchmark, a training resource, and more. Largely ignored as it was, maybe her academic grandfather, Jitendra Malik, had been right all along – "The trick to science is to grow with your field, not to leap so far ahead of it."
Others at the time – notably Shane Legg, who would co-found DeepMind alongside Demis Hassabis and Mustafa Suleyman – were starting to talk about another set of converging technological threads. The convergence of those threads would lead AI into its second spring – unbeknownst to Fei-Fei, ImageNet's large-scale dataset would be one of them.
The ImageNet Challenge
Doing whatever she could to keep ImageNet from slipping into obscurity, Fei-Fei delivered presentations on it at whichever university would have her.
On the way to the 2009 International Conference on Computer Vision (ICCV) in Japan, a conversation with fellow computer vision researcher Alex Berg prompted Fei-Fei to launch a competition alongside ImageNet. Inspired by the PASCAL Visual Object Classes Challenge, the competition would encourage researchers to improve their image recognition algorithms, thus turning ImageNet into a benchmark and training resource.
The competition was christened in 2010 as the ImageNet Large Scale Visual Recognition Challenge. It scaled up the number of object classes and images in the PASCAL Visual Object Classes Challenge by orders of magnitude.
The brewing excitement in the next chapter of ImageNet was quickly dissipated. Though the inaugural competition drew a healthy thirty-five entries, the winning entry was only slightly better than work found elsewhere in the field. Attracting fewer entries the following year, the winning entry barely beat its predecessor. The progress Fei-Fei had hoped the challenge would spawn in image recognition was dimming.
Reluctantly, Fei-Fei ran the competition for a third time in 2012. The winning entry that year, so unexpected, compelled Fei-Fei to part with her newborn child at home in California, and board an uncomfortable overnight flight to the European Conference on Computer Vision (ECCV) in Italy, where it would be announced.
The winning entry, from a team at the University of Toronto, had revived the long-considered-antiquated neural network and submitted what is known as a convolutional neural network. And what compelled Fei-Fei to part with her newborn child was that the winning entry had dwarfed the next best entry by more than 10.8 percentage points, heralding the start of the second AI spring.
The AlexNet Moment
The University of Toronto team – SuperVision, later dubbed AlexNet after its lead author – consisted of Geoffrey Hinton, Ilya Sutskever, and Alex Krizhevsky.
The name Geoffrey Hinton – full name Geoffrey Everest Hinton – had further compelled Fei-Fei to attend the conference in Italy. Hinton's great-great-grandfather, George Boole, invented Boolean logic – the zeroes-and-ones logic computers run on. And his great-great-granduncle, Sir George Everest, the Surveyor General of India, had Mount Everest named after him – which is where Hinton's middle name comes from.
As impressive as his family tree was, the name Geoffrey Hinton meant more than that to Fei-Fei. Ilya Sutskever and Alex Krizhevsky were young researchers yet to make a name for themselves in the field. Hinton, however, had already spent over two decades stubbornly making a name for himself – building on the then-unpopular neural networks and convolutional neural networks that would eventually culminate in the AlexNet moment.
Awarded a PhD from the University of Edinburgh in 1978, Hinton soon moved to the United States. First, he moved to UC San Diego, then to Carnegie Mellon University. At the latter, in 1986, he co-published a pioneering paper which popularised backpropagation – a term coined by Frank Rosenblatt, implementer of the first artificial neural network, in 1962. In 1987, he moved to the University of Toronto in Canada.
AlexNet, a neural network architecture – more specifically a convolutional neural network architecture – managed to perform backpropagation more efficiently, and thrived in image recognition thanks to the explosion in image data, and the collapsing price of computing power, which Moore's Law made possible.
AlexNet had resurrected the neural network, which had been relegated to the sidelines in the short history of artificial intelligence, and made it more deeply layered and connected than ever before. It was as if the potential of neural networks had simply been waiting for a serendipitous trifecta of technological trends to emerge – alongside large-scale data and GPUs – that would unleash it towards human-like capabilities.
The large-scale dataset of ImageNet – part of that trifecta of technological trends – had provided AlexNet with the training data it needed to realise the potential of its convolutional neural network architecture.
Similarly, Nvidia's GPUs and CUDA platform – again, part of that trifecta of technological trends – had provided AlexNet with another crucial ingredient, affordable access to large-scale compute to do matrix multiplications in parallel. Subsequently, this made Nvidia's GPUs the industry standard for training neural networks – at a time when its market cap was shy of a now-seemingly-minuscule ten billion dollars.
More specifically, AlexNet had been trained on two Nvidia GTX 580 GPUs – each with 3GB of memory – which took five to six days. That was much, much faster than using CPUs.
Let's now take a step back and discuss neural networks, backpropagation, and convolutional neural networks so we can gain an understanding – albeit an elementary understanding – of how AlexNet does what it does. Otherwise, I would only be adding to the brandishing of these words which I often came across in my readings.
The simplest example of a neural network I can share, perhaps, is a feedforward network consisting of three layers – an input layer, a middle hidden layer, and an output layer – networked similarly to how synapses connect neurons in the human brain.
Such a neural network, in theory, provided with enough human-labelled images of horses, for example, could learn to identify one through a training process known as supervised learning. It's during this process that the network, through a technique called backpropagation, iteratively adjusts the weights of its synapse-like connections – strengthening or weakening – until it, given an unlabelled image, can identify a horse.
This is, of course, a simple and binary example. It does, however, lead us logically to the convolutional neural network used by AlexNet, which dwarfed the next best entry in the 2012 ImageNet challenge by more than 10.8 percentage points.
AlexNet, in comparison to the simple, binary, three-layer network I described, consisted of eight layers – five convolutional layers, and three fully connected layers. It also consisted of 650,000 individual neurons, networked by 630 million connections, with 60 million weights influencing the strength of those connections.
In practice, when AlexNet was shown an image, neurons in the first layer of its network processed the pixels, searching for simple visual cues like lines and colours. Traversing deeper, subsequent layers pieced those cues together to identify something more comprehensible. Layer by layer, the network recognised specific components until it labelled the object in an image as a whole – for example, a dog.
The math underpinning AlexNet started with processing pixels as numbers. Neurons then multiplied those numbers by weight variables and added bias variables. That result went through a mathematical formula which determined the signal to send to the next layer. This process was repeated until the final layer correctly labelled an image – with the weight variables and bias variables adjusted whenever it incorrectly labelled an image.
AlexNet, after winning the 2012 ImageNet challenge, heralded the start of the second AI spring – and the deep learning era, or deep neural networks era. Geoffrey Hinton, Ilya Sutskever, and Alex Krizhevsky, realising the importance of their creation, formed a company. A few months later, Google acquihired it for $44 million.
In the following years of the ImageNet challenge, nearly all of the entries built upon the deep convolutional neural networks that AlexNet popularised. In the 2014 challenge, a team from Google submitted a twenty-two-layer network that significantly outperformed AlexNet. The ImageNet challenge, having served its purpose as a benchmark and a training resource, was eventually retired in 2017.
Alumni of the ImageNet challenge entered what was almost a Premier League-like seller's market – and the salaries reflected it. Its alumni can be found across the tech world.
Yuanqing Lin later served as Director of Baidu's Institute of Deep Learning. Kai Yu later founded Horizon Robotics. Ilya Sutskever later co-founded OpenAI. Christian Szegedy later co-founded xAI. Yangqing Jia later served as President of Alibaba Cloud's Computing Platform. Dragomir Anguelov later served as Head of AI Foundations at Waymo. Xiangyu Zhang later co-founded StepFun. And those are only a select few of the alumni.
The trifecta of technologies – large-scale data, GPUs, and neural networks – that enabled AlexNet shows us that the story of innovation is often circuitous. Ideas, like neural networks, are pioneered, criticised into obscurity, then rediscovered.
AlexNet also shows us that step changes – lollapalooza effects, as Charlie Munger called them – often come from a cross-pollination of things and people. With AlexNet, it was large-scale data, GPUs, and neural networks. It was also Geoffrey Hinton, Ilya Sutskever, and Alex Krizhevsky. And they were building upon the ideas and work of Fei-Fei Li, Jensen Huang, Frank Rosenblatt, Warren McCulloch, Walter Pitts, and more.
The decade that followed AlexNet – with all the advances in the artificial intelligence field in between – culminated in the release of OpenAI's ChatGPT, a product that captured the attention of everyone not already immersed in the field.
The Aftermath of ImageNet
The ImageNet dataset, however, was not without its criticisms. The dataset, curated to be a representation of the world, was scrutinised for the bias, discrimination, and unfairness that the networks trained on it perpetuated. It led some to argue that comprehensive documentation – including, but not limited to, assumptions, motivations, and values – needed to be written for the datasets themselves.
The ImageNet team was praised for its uncommon openness. They updated problematic categories, like the people categories, which labelled people like objects, deleting over a million images in the process – including an image that labelled a man drinking beer as an alcoholic. That said, the ImageNet dataset had already been downloaded countless times by then, and, perhaps, used in production.
Others argue that simple technical fixes – like deleting problematic images – are not the solution, because they remove the opportunity to investigate how datasets come to normalise a specific worldview. And there is no independent worldview of a dataset. Its worldview is derived from a human worldview. These simple fixes, furthermore, continue to collapse real-world complexities into fixed and singular categories.
The criticisms of the ImageNet dataset – and, in general, the process of dataset curation – become increasingly complex and philosophical. And rightly so, after all, if the goal is to build products that interact with the world, having the least biased, discriminatory, and unfair datasets upstream is non-negotiable.
In a video published on the Stanford University School of Engineering YouTube channel, in the same year as the AlexNet moment, Fei-Fei described some of the other projects that she and her team were working on to make the visual world familiar to machines. In that video, two projects stand out.
One project Fei-Fei describes is smart albums – a forerunner of the photo-categorisation algorithms which now tame the digital mess in the photos application on our smartphones. It's this that enables us to open up the photos application and type in a specific person, event, car, or even colour and see related photos. This is an example of technology solving a seemingly boring, but useful problem.
More broadly, technology like this also helps the likes of Facebook categorise the tens, if not hundreds of thousands of photos uploaded to its platform every minute. That's something a human wouldn't be able to do – nor, perhaps, find any joy in doing.
Another project Fei-Fei describes is storytelling with images – a project that eventually ventured beyond the labelling of objects, towards describing the scene of objects. It graduated from toddler-like labels such as "car", towards adolescent-like descriptions such as "a car parked on the kerbside". And in my case, if that car is a Porsche 911, whoever is with me would have to endure a rundown of its specification.
This project, developed alongside Andrej Karpathy – which, unbeknownst to them, Google was also working on – could be considered a precursor to the LLMs that would later capture our attention with the launch of OpenAI's ChatGPT. Karpathy would later serve as a research scientist at OpenAI.
Fei-Fei, having achieved her more-than-a-decade-old north star – to make the visual world familiar to machines – again felt pleasure and pain. She felt pleasure in having helped machines label and then describe objects – realising the saying that a picture is worth a thousand words, not a thousand disconnected ones. And felt pain in no longer having a north star to pursue, having achieved what she thought would take a lifetime.
She had dedicated her career to pursuing difficult problems that were, perhaps, borderline delusional. But they were problems that, if solved, would leap her ahead of the field. In solving them, she had succeeded in making the visual world familiar to machines, helping them to see – whether they understood what they saw is another topic.
A quote from Sir Winston Churchill – via a presentation delivered by Fei-Fei titled 'ImageNet: Where Have We Been? Where Are We Going?' – aptly captures this point in her career: "This is not the end. It is not even the beginning of the end. But it is, perhaps, the end of the beginning."
Fei-Fei Li Post-ImageNet
In 2024, Fei-Fei Li – alongside Justin Johnson, Christoph Lassner, and Ben Mildenhall – raised $230 million and founded World Labs, a spatial intelligence AI company.
Spatial intelligence is an ability that goes beyond language. For example, something as seemingly simple as loading a dishwasher involves spatial intelligence – the perceiving, the turning of that perceiving into reasoning about the environment, and the turning of that reasoning into an interaction. It's that spatial intelligence, through the generation of environments, that Fei-Fei wants to give machines.
Fei-Fei believes that spatial intelligence, similar to language intelligence, is key to unlocking the next frontier in artificial intelligence – a frontier that will help us to create better, manufacture better, design better, and more. Some may call that artificial general intelligence, a term popularised by DeepMind co-founder Shane Legg, but that's a term Fei-Fei refuses to engage with.
At World Labs, Fei-Fei and the wider team use curated data and algorithmic innovations to create frontier models that have the ability to perceive, generate, reason, and interact with worlds. As with the early days of ImageNet, the curation of data, now multimodal, presents a similar challenge. It's like Fei-Fei looped back to ground zero to tackle the problem that initially consumed her – albeit at a deeper dimension.
In 2026, AMD acquired World Labs for a reported $8.2 billion to accelerate towards their shared vision of the future – solving real problems for human betterment.
Epilogue
Fei-Fei, throughout her career, in both the resource-constrained public sector and the abundant private sector, alongside numerous talented people across a multitude of projects, has remained focussed on a variation of her initial north star – making the visual world familiar to machines. Her career has been a part of the advances that took computer vision from objects, to scenes, and then to worlds.
More broadly, her career has been a part of the advances that took artificial intelligence from a niche field, to the most debated field on the planet. Artificial intelligence is a technology that has, and will continue to have, an outsized impact on civilisation. It's for this reason that having the right level of optimism is hard. Fei-Fei, believing in neither extreme, describes herself as a pragmatic optimist.
A learning we can take from the career of Fei-Fei Li and the story of ImageNet is that the default position of any innovator should be that of humility. As the story of ImageNet shows, progress in innovation is built upon innovations. And step changes in progress are often marked by serendipitous inflection points due to the contribution of many – not the contribution of a lone ambitious genius.
There are so many influences, inputs, and interdependencies that the simplicity of a hero's journey, though appealing, is almost never true in the story of innovation. Therefore, a country, field, or company must continue to foster both public and private centres of excellence, as well as a thriving society of people across disciplines, to keep the story of innovation alive.
It's those people, those innovators, in those places, who will stubbornly find new solutions when the current ones hit a wall. And it's those people, though historically not always the case, who will keep dignity and agency at the heart of innovation.
Another learning we can take is that curiosity, perseverance, and intellectual honesty – traits that Fei-Fei embodied – are prerequisites to innovation that require a special influence. That influence, whether from books, teachers, or elsewhere, is what cultivates those traits and helps you grow a little wiser every day. The explosion in information, after all, has not been accompanied by a similar explosion in wisdom.
The story of ImageNet, in my opinion, resembles a Charlie Munger quote: "Take a simple, basic idea, and take it very seriously."
The problem of curating a large-scale dataset to train computer algorithms, which ImageNet solved, might now be considered simple. After all, it could be said that it's basically a big bunch of labelled images. But Fei-Fei Li, Jia Deng, and the wider team, having taken the problem seriously and interrogated it, were able to answer questions of increasing complexity on quantity, quality, and more that others could not answer.
When colleagues constructively challenged ImageNet – as should be rightly expected and encouraged – it was those answers that encouraged them to continue pursuing ImageNet.
On a personal level, it would have taken a lot of courage for Fei-Fei, Jia, and the wider team to work against their field, which, at the time, was fixated on algorithms. And it would have taken a lot more courage to continue working against their field when the publication of their work was largely ignored.
I started the story of ImageNet with the three words that a twelve-year-old Fei-Fei Li overheard her parents whisper at home one night – education, opportunity, and freedom. Those three words led Fei-Fei towards her north star – making the visual world familiar to machines – which culminated in the publication of ImageNet, which, at the time, was the largest human-curated dataset in the history of artificial intelligence.
Artificial intelligence is an increasingly powerful and useful technology that is sometimes debated in a godlike manner. Godlike depictions are often given to things we simply don't understand. I hope the humanistic thread throughout the story of ImageNet has helped to detach artificial intelligence from godlike depictions.
As Fei-Fei Li said on the tenth anniversary of ImageNet, "There's nothing artificial about artificial intelligence – it's made by humans, intended to work with humans, and it will have the impact for humans."
On The Tim Ferriss Show, Fei-Fei suggested that people who are searching for their north star in the AI era should not choose one that is on a collision course with organisations that have an abundance of compute and data. That, perhaps, is better advice than knowing which north star you should choose. As Charlie Munger famously said, "Invert, always invert."
That leads me to an apt closing from the memoir of Fei-Fei Li: "Stay curious. Stay bold. Be forever willing to ask impossible questions."