Generative AI in the Real World: Tom Smoker on Getting Started with GraphRAG

Join Ben Lorica and Tom Smoker for a discussion of GraphRAG, one of the hottest topics of the last few months. GraphRAG goes a step beyond RAG to make the output of language models more consistent, accurate, and explainable. But what is a graph? A graph is a way of structuring data. In the end, it’s the structure that’s important, along with the work you do to create that structure.

About the Generative AI in the Real World podcast: In 2023, ChatGPT put AI on everyone’s agenda. In 2024, the challenge will be turning those agendas into reality. In Generative AI in the Real World, Ben Lorica interviews leaders who are building with AI. Learn from their experience to help put AI to work in your enterprise.

Points of Interest

• 0:00: Introduction
• 0:15: GraphRAG is RAG with a knowledge graph. Do you have a more strict definition?
• 1:00: A lot of what I do is the R in RAG: retrieve. Retrieval is better if you have structured data. I’ve yet to find a definition for GraphRAG. You want to bring in structured data.
• 2:03: At the end of the day, the lesson is structure. Sometimes structure is a SQL database. Don’t lose hope if you don’t have a knowledge graph.
• 2:49: A knowledge graph is a knowledge base and a list of axioms (rules). The knowledge base is just a word connected to another word through a third word. Fundamentally, the benefit comes from the list of triples. The value is in having extracted and defined those triples.
• 4:01: Knowledge graphs are cool again. What are your two favorite examples of GraphRag in production?
• 4:57: My examples are people who are structuring their data so that it’s consistent. Then you can bring it into a context window and do something with it.
• 5:18: LinkedIn and Pinterest are the best examples of existing graph structures that work.
• 5:35: A new application is a veterinary radiology example. Without GraphRAG, the LLM kept recommending conditions specific to Labradors not bulldogs. GraphRAG controlled the problem.
• 6:37: The underlying data was almost exclusively text. It’s difficult to build up a consistent dataset for veterinary radiology because animals move.
• 7:12: My favorite examples: Google uses their data commons to build a Q&A application. Metaphor Data: The starting point is structured data, then they create a second graph from the first graph that maps technical terms to business terms. Then they construct a social graph based on who is using the data.
• 9:41: Structured data can be the basis for a graph.
• 10:06: Unstructured data is valuable, but you need a way to navigate and categorize unstructured data.
• 11:04: Where are we on GraphRAG? Do you still have to explain what GraphRAG is?
• 11:28: More people know about it, but I have to explain it more than I did previously. Exactly what are we referring to? Most people want accuracy in the beginning; the value is often that it is more explainable. People may have seen a fantastic example, but what they haven’t seen is the iterative process in schema design. The upfront cost of these systems is nontrivial.
• 13:13: What are the key bottlenecks? How do I get a knowledge graph?
• 13:23: The biggest question is: Do you need a graph in the first place? There’s a whole spectrum. It’s in most people's interest to stop before they get to the end.
• 14:01: For people who come to us brand-new, we say, “You should try vector RAG first. If that doesn’t work, there’s a lot of good that structuring data can provide.”
• 15:01: If the chunks are structured, and a lot of the work is done up front, then it’s possible to navigate through structured information. At that point, you get value out of vector RAG. Academic papers have to follow a certain structure. If you spend time making sure you know what the chunks are, where they’re split and why, and they’re labeled, you can get a lot of value.
• 16:43: What are some of your pointers about how to get started?
• 16:47: The knowledge base is often a compressed representation. That means less tokens. That means better rate limits and less cost. So some people want a graph to help scale. That’s one start. Another is the desire for a system to be explainable. Getting that information into a structured representation and tracing back that structured representation can be very useful.
• 17:57: The first pass is extracting in a consistent way. Pydantic and JSON are often a good first step. You can prompt an LLM and ask it to extract for you.
• 18:24: Are we getting into some sort of automatic knowledge graph construction?
• 18:38: We said, “Automated knowledge graph construction creates bad graphs.”
• 18:58: But can automatic knowledge graph construction be good enough for GraphRAG?
• 19:29: There is no one answer; it’s context-dependent. You need intention. Why didn’t you just add metadata filters? Creating a graph is an iterative process. The value is starting that process.
• 20:19: One of the first lessons from RAG was metadata filtering.
• 20:27: That’s really useful. It can be a way to differentiate between vector providers. We use metadata filers all the time. Any time spent structuring your data requires you to understand your data better. There is business value there.
• 21:31: You’ve been in knowledge graphs a long time. Some companies actually have knowledge graphs: logistics, fraud detection, things like that. In those domains, are you seeing GraphRAG?
• 22:09: Absolutely. People want to increase the ROI from the graphs that they’ve already built and are maintaining. They’re also looking to give more people access to the data if there’s a natural language interface.
• 24:04: There are two communities: semantic web people (RDF) and property graph people. Can you talk about these two groups in light of GraphRAG?
• 24:39: Property graphs are easier, more accessible. The RDF world is more consistent and more specified. I haven’t seen many people change their mind. But when property graphs came out, there was controversy around weakening the value. The value to RDF was the reasoning. But in the age of LLMs, most people are just trying to get started. I don’t see many new use cases coming to RDF. With knowledge graphs, it’s easier to get started. The cool new things are happening in property graphs.
• 27:10: I have seen some interesting work around JSON-LD, which is in a lot of LLM training data. So LLMs are good at understanding JSON-LD. But not many people progress through to RDF.
• 27:42: It seems like GraphRAG is really property graphs?
• 27:49: The bigger question is whether GraphRAG is graph structured at all, as opposed to just structured data. Do I need properties at all?
• 28:28: What are your predictions for GraphRAG in 2025? Will it be so easy that most companies have some sort of GraphRAG system in place?
• 29:00: The RAG world defines that much more than the graph part. Progress in RAG has been slower than people expected. Structured data will definitely be important. Will it be in graphs? There will be some specific use cases. But it won’t necessarily be stored in graphs.
• 30:09: We need more structure and explainability.
• 30:32: “Structure is all you need.”
• 30:50: For people who haven’t tried doing RAG rigorously, they have the impression that RAG will make hallucination go away. But the LLMs have opinions. You can prompt the LLM, and it may still give you its opinion.
• 31:35: Absolutely. Karpathy said, “LLMs don’t have a hallucination problem. LLM assistants have a hallucination problem.” Karpathy also said, “When you are speaking to an LLM, you’re speaking to the average of the model labelers who contributed to the model.” With that in mind, we are much more realistic about what we can build with RAG.
• 33:08: Next year, people will realize that the most interesting data is operational data in live systems. We have to get the right data at the right time, and that will be real time.
• 33:55: When people talk about the gold mine that is unstructured data, it assumes that you captured data well. Data in PDFs isn’t necessarily consistent or valuable. You have to sift through a lot of noise to get signal in PDFs. I see value in data that is already structured and that streams in consistently. The holy grail of RAG is where physical meets digital, like a large-scale multinational supply chain.