book

Unifying Business, Data, and Code

by Ron Itelman, Juan Cruz Viotti

January 2024

Intermediate to advanced

356 pages

9h 53m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Includes

Includes Quizzes

What You Can’t See Can Kill You, and the Same Is True for DataHidden Threats to Organizations: A Modern ParallelYour AI Is Only as Good as Your DataAligning Problem-Solving Strategies, Data, and AIA New Paradigm to Optimize Data Management and Business Strategy for the Age of AIThe Origin Story of UnifyingOrchestrating Alignment at Organizational ScaleConventions Used in This BookO’Reilly Online LearningHow to Contact UsAcknowledgments
Your Quest for Data-Driven Breakthroughs Begins There Are Usually Multiple, Conflicting North StarsThe Good, the Bad, and the Ugly of Data ProblemsThe Problem with ProblemsUnifying Concepts: The Key to InnovationWhat a Unifying Data Strategy Will Do for AgileDefining Being AgileAgile TheaterAgile, Waterfall, and UnifyingDefining a Unifying Data Strategy ApproachUnderstanding the Phrase Being Data DrivenTo Be Data Driven, Be Data CentricBottlenecks Preventing Teams from Being Data DrivenThis Book’s Project: Intelligence.AI Coffee BeansSummary
Introducing JSONA Simple JSON ExampleJSON Viewing and Authoring ToolsOverview of JSON GrammarBooleansNumbersStringsArraysObjectsNullLearning MoreMinificationAlternative RepresentationsCreating a JSON DocumentA Product EntryA Store OrderSummary
Data Transformations Require Data ChampionsThe Rise of the Data Product ManagerAlignment Is a Journey, Not a DestinationEvaluating Alignment from a Holistic PerspectiveThe Goal Isn’t Alignment, It’s Effective AlignmentStrategies for Setting Up Teams for SuccessIncorporating a Product Management MindsetDefining Data Users’ NeedsDefining Product FeaturesDefining and Measuring SuccessUnifying Versus AligningSummary
Packaging and Products: An Example Using CoffeeThe Four Facets of a Data ProductGetting Started with Concept-First DesignA Blueprint for UnifyingMapping the Conceptual Terrain: Assessing ConceptsFacilitating Assessments of Conceptual Alignment Across Technical and Nontechnical TeamsSmooth Is Slow, Slow Is FastSummary
What Is JSON Schema?What Is a Schema?The Building Blocks of JSON SchemaVocabularies and DialectsMeta-Schemas: Schemas That Describe Other SchemasUnderstanding JSON SchemasStep 1: Determining the Schema Dialect: The $schema KeywordStep 2: Determining the Schema VocabulariesStep 3: Understanding Schema VocabulariesStep 4: Understanding Schema KeywordsJSON Schema as a Recursive Data StructureReferencing SchemasWhat does duplication look like?Local referencingRemote referencingYour First JSON Schema ProjectWriting a Schema: Step by StepGenerating a Web FormSummary
Enemies of Alignment: Ambiguity and AssumptionsAmbiguity: The Culprit in the Illusion of CommunicationAssumptions: Ambiguity’s Best FriendDefining Success: Symmetry Between Concepts and JSON Schema Equals Minimal AmbiguityIlluminating Misalignment with a Concept CompassStep 1: Harmonizing the WhatStep 2: Harmonizing the WayStep 3: Harmonizing the HowHarmonized Concepts Validating Concepts: Belief Scoring and HypotheticalsCounterfactualsBelief ScoringSummary
An Introduction to Thinking in NetworksExample of Thinking in Networks: Athletes Versus ArtistsGraphs: The Visual Language of NetworksNetworks of Entities: Knowledge GraphsA Simple Knowledge GraphChallenges with Knowledge GraphsAligning Knowledge for the 99%Fundamentals of CLEAN Data GovernanceCollaborationKnowledgeBusiness LogicActivityCLEAN Data Governance in PracticeThe Four Facets of Data Products and CLEANThe Four Horsemen of Data DeathIgnoranceSiloed IncentivesShortsightednessIndecisivenessThe Power of Design in Collaborative NetworksSummary
Validating the Structure of DataUsing an Online ValidatorValidation ExampleJSON Schema as a Constraints LanguageBoolean SchemasHeterogeneous Data StructuresThe format KeywordUsing Annotations to Define MeaningAnnotation Extraction ExampleA Simple Use Case: DeprecationsRuntime ExtractionStandard Output FormatsRevisiting the format KeywordUsing an Online ValidatorThinking in SchemasSummary
How Ambiguity, Knowledge Gaps, and Blind Spots Influence Decisions and Progress Toward GoalsWhich Is Bigger: Greenland or the US?Mapping Pathways of Processes and ProgressMeasuring Progress Toward GoalsDefining Decisions and Steps with Process MapsHow Process Maps Reveal AmbiguityVisualizing and Removing Ambiguity in ProcessesEnriching Process Maps with AnnotationsProcess Maps Reveal Innovation OpportunitiesSummary

An Introduction to Knowledge FrameworksKnowledge Experiences and PathwaysA Tool for Designing Knowledge ExperiencesFrom Structured Knowledge to Computational KnowledgeSuccess SpectrumsMapping Progress and ValueVisualizing and Adding “Next Best States” Removing Blind SpotsEmbracing Multiperspective Design and Road MapsDefining KPIs for Success Measures and Metrics (Assessments)Using Demons and Magical Thinking for InnovationFaster HorsesImagining Magical PossibilitiesProblem Landscapes: Quantifying Pain Points Threatening ValueNudges: The Right Information at the Right TimeA Real-Life Problem Landscape and Demon Example That Led to a Unified Data Product ModelUnderstanding the Problem LandscapeThe Staggering ImpactA Meeting of Minds and the Birth of a SolutionBeyond Data Products: Data Product ManagementThe Circular Nature of UnifyingSummary
Schemas Over HTTPStep 1: Setting Up a GitHub RepositoryCreating a GitHub RepositoryUploading Your First SchemaStep 2: Deploying to Cloudflare PagesCreating a New Cloudflare Pages Website ProjectStep 3: Configuring HTTP HeadersInspecting the Current HTTP HeadersDeclaring Custom HTTP Headers on Cloudflare PagesChecking the ResultsStep 4: Creating a Landing PageAdding an HTML Entry PointStep 5: Adding a Custom DomainConfiguring a Custom Domain in Cloudflare PagesSetting Up a CNAME DNS RecordChecking the ResultsBest PracticesSchemas Are ImmutableAdopt a Versioning StrategySummary
First Facet: DataAn Example CSV DatasetA JSON Row RepresentationSecond Facet: StructureGeneral-Purpose ConceptsApplication-Specific ConceptsDataset EntriesThe Dataset SchemaThird Facet: MeaningTimestampIP AddressEmailUS StateCurrencyPriceMilestoneAnalytics EntryFourth Facet: ContextThe Signup Analytics SchemaSummaryAutomated Schema ExtractionNext Steps
Simple Case: Unknown KeywordsExtracting Unknown Keywords as AnnotationsPros and Cons of This ApproachComplex Case: Authoring VocabulariesThe JSON Schema Vocabulary SystemStep 1: Writing a SpecificationStep 2: Writing a Vocabulary Meta-SchemaStep 3: Extending an ImplementationConsuming VocabulariesDefining a DialectMaking Use of the DialectExample: Extracting Annotations with HyperjumpSummary
Introducing the Dataset VocabularyRevisiting the Signup Analytics ExampleJSON Schema BundlingThe Bundling ProcessBundling Our Example Data ProductReferencing Remote DataThe Problem of Streaming JSONIntroducing JSON LinesExtracting MeaningA Simple ExampleUsing Logic OperatorsThe Signup Analytics ExampleDataset LineageFilteringTransformingAggregationSummary
Your Unifying Journey So FarA Constellation of Deeper Principles Guides Unifying1. The Principle of AlignmentTransforming the Abstract to ConcreteWhat You See Can Kill You, and the Same Is True in Data2. The Principle of InformationUnderstanding Uncertainty3. The Principle of LearningDefining LearningDefining Errors4. The Principle of Integrated SimplicityComplexity ReductionDecompositionCompressionMemoizationIntegrating in Communication Networks5. The Principle of ContinuumsMaking Things MeasurableThe Dangers of Misusing MeasurementsA Continuum Example for a Control Strategy Problem6. The Principle of State TransitionsA Simple State MachineSimplifying State Transitions7. The Principle of Decidability What Is Decidability?Two Key Approaches to Problem SolvingMaking Informed DecisionsReal-World Decidability to Reduce Misalignment in Teams8. The Principle of HeuristicsAwareness and Ethical ConsiderationsConnection to Decision Making in Business9. The Principle of MasteryLevels of Mastery in KnowledgeStrategies for Mastery10. The Principle of WisdomSummary
Functional Artificial IntelligenceYour AI Is Only as Good as Your DataBeware Illusions Within Vetting ProcessesQuestion AssumptionsCollective IntelligenceCollaborative IntelligenceUnified IntelligenceCollaborative Learning NetworksPersonalized KnowledgeAnticipatory Design: Personalization and Digital TwinsCodifying Principles of IntelligenceContinuous Human–Machine Learning LoopsApplying Wisdom in PracticeConceptual ZoomabilityWisdom Graphs: Connecting Concepts, Actions, and OutcomesCognitive Primitives: Standardizing Cognitive Experience DesignThe Value of UnifyingPrioritize Knowledge Before AIA Tale of Simple Knowledge Versus Complex IntelligenceFollow the Principle of Integrated SimplicitySummaryGoing Beyond This Book

Content preview from Unifying Business, Data, and Code

Chapter 4. Concept-First Design for Data Products

In this chapter, you will learn a new and straightforward definition of a data product, described across four facets. You’ll also discover a method called concept-first design for crafting data products and understand their relationship to schemas.

Concept-first design is an effective strategy that seamlessly bridges the communication gap between nontechnical business teams and technical teams. It employs universally understood tools, such as diagrams and spreadsheets, to represent ideas and challenges.

The beauty of concept-first design lies in its simplicity. One translates business language or logic into a hierarchical format, akin to how JSON and JSON Schema operate—these are digital languages that machines comprehend. By using concept-first design, the effort to convert business logic into code is reduced. This minimizes the errors and inefficiencies often arising when developers and data stewards make assumptions during concept modeling.

Imagine a data product as a box containing a single tool, complete with everything a user needs to accomplish a task. Think of buying a friend an electronic gift. If the gift lacks a battery or instructions, it leaves your friend scrambling to find missing pieces—a frustrating experience. This is precisely how you must think about data products. A data product should be a comprehensive package, providing users with everything they need to utilize the data effectively. The goal is to avoid ...