Medical board exams are a strange kind of test. The USMLE Step 1 covers roughly 20,000 concepts across anatomy, physiology, pharmacology, pathology, and a dozen other disciplines. The NCLEX draws from a question pool so large that no two test-takers see the same exam. You can’t just study the material. You have to practice applying it under pressure, to questions you haven’t seen before, over and over again.
That’s why question banks have been the backbone of medical exam prep for decades. Services like UWorld and Amboss charge $200-400 for access to curated sets of practice questions written by physicians. They’re good. They’re also generic by design, because they need to work for every student.
This is where AI quiz generators have started to fill a gap that I didn’t expect them to fill.
What AI quiz generators actually do
The concept is simple: you give the tool your study material (notes, textbook chapters, lecture slides) and it generates practice questions from that content. The questions aren’t pulled from a database. They’re created on the fly by large language models that have been trained to understand what makes a good test question.
I’ve been testing a few of these over the past semester, and the quality varies a lot. Some produce questions that are basically vocabulary drills. “What is the function of the mitochondria?” Fine for day one of biology, useless for board prep where you need multi-step clinical reasoning.
The better tools understand context. Quizgecko’s AI quiz generator was one that surprised me. I uploaded a set of pharmacology notes on antiarrhythmics, a topic I kept getting wrong on practice tests, and it generated questions that actually required me to apply the material. Not just “what does amiodarone do” but scenario-based questions about which drug you’d choose given specific patient presentations. It uses a pipeline of LLMs to process the content: understanding the subject matter, identifying what’s worth testing, and generating questions at the right difficulty level.
That said, no AI quiz generator consistently matches the quality of questions written by experienced physicians. The AI doesn’t have clinical judgment. It can produce questions that are technically correct but test the wrong thing, or that have subtle ambiguities a human question writer would catch.
Where this actually helps
The real value isn’t replacing your question bank. It’s filling in the gaps your question bank can’t cover.
Here’s what I mean. I take detailed notes during lectures and while reading First Aid. Those notes are specific to how I’m learning the material, what connections I’m making, what mnemonics I’m using. No commercial question bank can test me on my own notes. But an AI quiz generator can.
I started running my notes through Quizgecko after every major study session, usually generating 15-20 questions from that day’s material. The process of answering questions about content I studied just hours ago forced me to retrieve information in a way that re-reading never does. When I got questions wrong about material I’d literally just covered, it told me exactly which concepts hadn’t stuck.
I also used it to create practice questions from specific research papers and guidelines. When the latest hypertension management guidelines came out, I wanted to quiz myself on the new recommendations. No question bank had updated yet, but I could paste in the guideline summary and have practice questions in minutes.
What it doesn’t do well
Clinical vignette questions at the USMLE Step 2 level are still beyond what these tools produce reliably. The long scenario-based questions that require integrating information across multiple organ systems, weighing competing diagnoses, and considering patient-specific factors need the kind of medical judgment that LLMs don’t have yet.
I also found that the questions work better for some subjects than others. Pharmacology, microbiology, and biochemistry, subjects with lots of discrete facts and mechanisms, produce strong questions. Behavioral science and ethics, where the “right” answer depends on nuanced professional judgment, produce weaker ones.
And you need decent source material to get decent questions. If your notes are sparse or poorly organized, the generated questions will reflect that. I learned to structure my notes more carefully because I knew they’d become source material for practice questions later, which was honestly a useful side effect.
The bigger picture for medical education
What I find interesting about this from an AI perspective is that it’s one of the more practical applications I’ve seen. Not replacing doctors, not diagnosing patients, just helping students practice more efficiently with their own materials.
The combination I’ve landed on is using UWorld or Amboss for standardized, high-quality practice (you still need that baseline), Quizgecko for generating personalized questions from my own notes, and Anki for long-term spaced repetition of facts I keep forgetting. Each tool does something the others can’t.
Medical education has been slow to adopt AI tools compared to other fields, for understandable reasons. But quiz generation is a relatively low-risk application. The stakes of a bad practice question are that you study the topic more carefully, not that a patient gets harmed. It’s the kind of application where AI being “pretty good” is already useful.
Whether these tools improve exam scores in a measurable way is something I’d love to see studied properly. My own experience has been positive, but I’m one person with one study method. What I can say is that it changed how I prepare for exams in a way that feels sustainable, and I haven’t gone back to just re-reading notes and hoping for the best.
