Mastering GitLab 12

Mastering GitLab 12

by Joost Evertse
Released August 2019
Publisher(s): Packt Publishing
ISBN: 9781789531282

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

An expert guide to helping you use DevOps techniques with the latest GitLab version to optimize and manage your software workflow

Key Features

  • Delve into GitLab's architecture, and install and configure it to fit your environment
  • Learn about the underlying principles of Agile software development and DevOps
  • Explore Gitlab's features to manage enterprise cloud-native applications and services

Book Description

GitLab is an open source repository management and version control toolkit with functions for enterprises and personal software projects. It offers configurability options, extensions, and APIs that make it an ideal tool for enterprises to manage the software development life cycle.

This book begins by explaining GitLab options and the components of the GitLab architecture. You will learn how to install and set up GitLab on-premises and in the cloud, along with understanding how to migrate code bases from different systems, such as GitHub, Concurrent Versions System, Team Foundation Version Control, and Subversion. Later chapters will help you implement DevOps culture by introducing the workflow management tools in GitLab and continuous integration/continuous deployment (CI/CD). In addition to this, the book will guide you through installing GitLab on a range of cloud platforms, monitoring with Prometheus, and deploying an environment with GitLab. You'll also focus on the GitLab CI component to assist you with creating development pipelines and jobs, along with helping you set up GitLab runners for your own project. Finally, you will be able to choose a high availability setup that fits your needs and helps you monitor and act on results obtained after testing.

By the end of this book, you will have gained the expertise you need to use GitLab features effectively, and be able to integrate all phases in the development process.

What you will learn

  • Install GitLab on premises and in the cloud using a variety of configurations
  • Conduct data migration from the SVN, TFS, CVS, and GitHub platforms to GitLab
  • Use GitLab runners to develop different types of configurations in software development
  • Plan and perform CI/CD by using GitLab features
  • Monitor and secure your software architecture using Prometheus and Grafana
  • Implement DevOps culture by introducing workflow management tools in GitLab

Who this book is for

If you are a software developer, DevOps professional, or any developer who wants to master GitLab for productive repository management in your day-to-day tasks, this book is for you. Basic understanding of the software development workflow is assumed.

Table of contents

  1. Title Page
  2. Copyright and Credits
    1. Mastering GitLab 12
  3. Dedication
  4. About Packt
    1. Why subscribe?
  5. Contributors
    1. About the author
    2. About the reviewer
    3. Packt is searching for authors like you
  6. Preface
    1. Who this book is for
    2. What this book covers
    3. To get the most out of this book
      1. Download the example code files
      2. Code in Action
      3. Download the color images
      4. Conventions used
    4. Get in touch
      1. Reviews
  7. Section 1: Install and Set Up GitLab On-Premises or in the Cloud
  8. Introducing the GitLab Architecture
    1. Technical requirements
    2. The origins of GitLab
    3. Exploring GitLab editions – CE and EE
    4. The core system components of GitLab
      1. NGINX
        1. Debugging NGINX
      2. Unicorn
        1. Debugging Unicorn
          1. Timeouts in Unicorn logs
          2. Unicorn processes disappear
          3. Other kinds of errors or 100% CPU load
      3. Sidekiq
        1. Debugging Sidekiq
          1. Sidekiq processes disappear
          2. A Sidekiq process is seemingly doing nothing
          3. Other kind of errors or 100% CPU load
      4. GitLab Shell
        1. Debugging GitLab Shell
      5. Redis
        1. Basic data operations in Redis
      6. Gitaly
        1. Debugging Gitaly
      7. GitLab Workhorse
        1. Debugging GitLab Workhorse
      8. Database
        1. Debugging PostgreSQL
    5. GitLab CI
      1. Pipelines and jobs
    6. GitLab Runners
      1. Issues with the old runner
      2. Switching to Go
    7. Cloud native
    8. Summary
    9. Questions
    10. Further reading
  9. Installing GitLab
    1. Technical requirements
    2. Installing GitLab using omnibus packages
      1. Omnibus structure
        1. Project definition
        2. Individual software definitions
        3. A GitLab configuration template
        4. Chef components
        5. Runit recipe
        6. Tests
        7. gitlab-ctl commands
      2. Running the installer
        1. Browsing to the external URL and login
        2. Upgrade using the omnibus-gitlab package
    3. Running from source
      1. Operating system – Debian 10
        1. Required basic software packages
        2. Required programming languages
          1. Ruby
          2. Go
          3. Node.js
        3. System users
        4. SQL database
        5. Redis memory database
        6. GitLab
        7. Installing GitLab Shell
        8. Installing GitLab-Workhorse
        9. Installing Gitaly
        10. Initializing the database and activating advanced features
        11. Final steps for preparing the system
        12. Preparing to serve
          1. Compiling GetText PO files
          2. Compiling assets
          3. Starting your GitLab instance
        13. NGINX
    4. Using it from Docker
      1. Running the image directly
        1. Configuring GitLab after startup
        2. Starting the container with configuration settings as input
        3. Upgrading GitLab
        4. Run GitLab CE on a different IP address
        5. Debugging the container
      2. Install GitLab using Docker Compose
        1. Updating GitLab using Docker Compose
    5. Deploying GitLab using Kubernetes
      1. GitLab Runner Helm chart
        1. Deploying of a GitLab Runner to Kubernetes
      2. GitLab Helm chart
        1. Deploying GitLab to Kubernetes
        2. Monitoring the deployment
        3. Initial login
          1. Outgoing email
          2. Updating GitLab using the Helm chart
          3. Uninstalling GitLab using the Helm chart
      3. Creating droplets on DigitalOcean
    6. Summary
    7. Questions
    8. Further reading
  10. Configuring GitLab Using the Web UI
    1. Technical requirements
    2. Configuring GitLab settings at the instance level
      1. Menu options
      2. Monitoring
      3. Messages
      4. System hooks
        1. Plugins
      5. Applications
      6. Abuse reports
      7. License
      8. Kubernetes
      9. Push rules
      10. Geo
      11. Deploy Keys
      12. Service templates
      13. Appearance
      14. Settings
        1. General
          1. Visibility and access controls
          2. Account and limit
          3. Diff limits
          4. Sign-up restrictions
          5. Sign-in restrictions
          6. Terms of service and privacy policy
          7. External authentication
          8. Web Terminal
          9. Web IDE
      15. Integrations
        1. Elasticsearch
        2. PlantUML
        3. Third-party offers
        4. Snowplow
      16. Repository
        1. Repository mirror
        2. Repository storage
        3. Repository maintenance
      17. Templates
      18. CI/CD
        1. Auto DevOps settings
        2. Shared runner settings
        3. Container registry
      19. Reporting
        1. Spam and anti-bot protection
        2. Abuse reports
        3. Error reporting and logging
      20. Metrics and profiling
        1. Metrics – InfluxDB
        2. Metrics – Prometheus
        3. Profiling – Performance Bar
        4. Usage statistics
        5. Pseudonymizer data collection
      21. Network
        1. Performance optimization
        2. User and IP rate limits
        3. Outbound requests
        4. Geo
      22. Preferences
        1. Email
        2. Help page
        3. Pages
        4. Real-time features
        5. Gitaly
        6. Localization
    3. Configuring GitLab settings at the group level
    4. Configuring GitLab settings at the project level
      1. General
        1. Naming, topics, avatar
        2. Visibility, project features, permissions
        3. Merge requests
    5. Summary
    6. Questions
    7. Further reading
  11. Configuring GitLab from the Terminal
    1. Technical requirements
    2. Configuring omnibus and GitLab installations from the terminal
    3. Configuring source installations
      1. GitLab app settings
        1. Storing big files
          1. Using object storage
      2. GitLab pages
      3. Mattermost
      4. Gravatar
      5. Sidekiq
        1. GitLab Registry
      6. GitLab CI settings
      7. Auth settings
      8. Advanced settings
      9. Rack Attack
    4. Reconfiguring GitLab Docker containers
    5. Changing GitLab in a Kubernetes environment
      1. Basic configuration
        1. Configuring TLS
        2. Configuring outgoing emails
        3. Other settings
    6. Summary
    7. Questions
    8. Further reading
  12. Section 2: Migrating Data from Different Locations
  13. Importing Your Project from GitHub to GitLab
    1. Technical requirements
    2. Using the GitHub integration feature
      1. Preparing GitHub for export
      2. Preparing GitLab for import
      3. Running the import
    3. Using a GitHub token
      1. Preparing GitHub for export
      2. Running the import
    4. Using a GitLab rake task
      1. Preparing GitLab for import
      2. Running the import
    5. Summary
    6. Questions
    7. Further reading
  14. Migrating from CVS
    1. Technical requirements
    2. CVS versus Git
      1. Filesets versus changesets
      2. Git branching
      3. Creating repositories
      4. Atomic operations
      5. Object naming or referencing versions
      6. Keyword substitution
      7. Binary blobs
      8. Amending commits
      9. A central repository
      10. Accompanying toolset
      11. Detecting file renames
      12. Commit before merge
    3. Preparing to migrate from CVS to Git
      1. Preparing for a conversion using cvs-fast-export
      2. Preparing for a conversion using cvs2git
    4. Running the conversion
      1. Converting data using cvs-fast-export
      2. Converting data using cvs2git
    5. Cleaning up after migration
    6. Summary
    7. Questions
    8. Further reading
  15. Switching from SVN
    1. Technical requirements
    2. The difference between SVN and Git
      1. Security and access control
      2. Space requirements and references
      3. Branching
      4. Handling binaries with SVN and Git
    3. Mirroring SVN and GIT
      1. No sync, just convert
    4. Using svn2git to migrate in one cut
    5. Summary
    6. Questions
    7. Further reading
  16. Moving Repositories from TFS
    1. Technical requirements
    2. TFS versus Git
    3. Centralized or decentralized
      1. Handling changes
        1. Branching and merging capacity
        2. History
        3. Traceability
        4. File handling
    4. The git-tfs tool
      1. Preparing to migrate
    5. Summary
    6. Questions
    7. Further reading
  17. Section 3: Implement the GitLab DevOps Workflow
  18. GitLab Vision - the Whole Toolchain in One Application
    1. Technical requirements
    2. The Agile Manifesto
      1. The initial model – waterfall
      2. Royce's model
      3. The sashimi model
      4. The V-model
      5. DSDM
        1. Timeboxing
        2. MoSCoW
      6. Scrum
      7. Crystal methods
    3. XP
      1. Fine-scale feedback
        1. Planning game
          1. Release planning
          2. Iteration planning
        2. Pair programming
        3. Test Driven Development
      2. Continuous processes
        1. Continuous integration
        2. Refactoring
        3. Short iterations
      3. Everybody owns the code
        1. Shared understanding
        2. Coding standards
        3. Simple design
        4. System metaphor
    4. The DevOps movement
      1. History of the movement
        1. Four Quadrant Model
        2. Four levels of competence
    5. The toolchain
    6. Summary
    7. Questions
    8. Further reading
  19. Create Your Product, Verify, and Package it
    1. Technical requirements
    2. The GitLab workflow
    3. DevOps phase – manage your ideas
      1. Cycle analytics
    4. DevOps phase – plan your feature
      1. Issues
        1. Content
        2. Status of the issue
        3. Meta information
      2. Discussions
      3. Milestones
      4. Epics
      5. Time tracking
      6. Quick actions
      7. The Project Issue board
      8. Todos
    5. DevOps phase – create it
      1. Projects and groups
      2. Snippets
      3. Web IDE
      4. Wiki
        1. Protected branches
      5. Merge requests
    6. DevOps phase – verify your product
      1. Code Quality reports
      2. Review apps
    7. DevOps phase – package it for use
      1. GitLab container registry
    8. Summary
    9. Questions
    10. Further reading
  20. The Release and Configure Phase
    1. Technical requirements
    2. Continuous Deployment
    3. Auto DevOps
      1. Configuring Auto DevOps
        1. Build step
        2. Code quality scan
        3. Container scanning
        4. Dependency scanning
        5. License management
        6. Static application security testing (sast)
        7. The final test step
        8. Production
        9. Performance
    4. Summary
    5. Questions
    6. Further reading
  21. Monitoring with Prometheus
    1. Technical requirements
    2. Setting up Prometheus
      1. Using an external Prometheus host
      2. Enabling the external dashboard link
    3. Customizing monitoring
    4. The static analysis of security vulnerabilities
    5. Dynamic Application Security Testing
    6. Dependency checking
    7. Summary
    8. Questions
    9. Further reading
  22. Integrating GitLab with CI/CD Tools
    1. Technical requirements
    2. Using Jira with GitLab
    3. Connecting Jenkins to GitLab
    4. Integrating with Mattermost
    5. Using webhooks for events
    6. Summary
    7. Questions
    8. Further reading
  23. Section 4: Utilize GitLab CI and CI Runners
  24. Setting Up Your Project for GitLab Continuous Integration
    1. Technical requirements
    2. Pipelines
    3. Jobs
    4. Creating .gitlab-ci.yml
    5. Configuring a runner
      1. GitLab Runner features
    6. Summary
    7. Questions
    8. Further reading
  25. Installing and Configuring GitLab Runners
    1. Technical requirements
    2. The Runner client architecture
      1. Basic architecture
    3. Creating a basic Runner with the shell executor
      1. Installing Runner on Linux
        1. Using a package manager
        2. Using a manual installation
        3. Updating a manually installed runner binary
      2. Installing on Mac
        1. The manual way of installing a runner
        2. Installing and using the Homebrew package manager
        3. Updating a manually installed runner binary
      3. Installing on Windows
      4. Registering a runner
        1. The interactive way of registering a runner
        2. The non-interactive way of registering a runner
      5. Running the nightly version
    4. Summary
    5. Questions
    6. Further reading
  26. Using GitLab Runners with Docker or Kubernetes
    1. Technical requirements
    2. Runner client architecture
    3. Creating your own Dockerized GitLab Runner
    4. Using a prebuilt Docker container to deploy GitLab Runners
    5. Using a Kubernetes cluster to spawn GitLab Runners
    6. Summary
    7. Questions
    8. Further reading
  27. Autoscaling GitLab CI Runners
    1. Technical requirements
    2. Runner client architecture
    3. Setting up the environment
      1. Preparing a bastion host
      2. Deploying the GitLab Runner software
      3. Installing Docker Machine
    4. Configuring the Runner
      1. Off-peak time mode configuration
      2. Distributed runners caching
        1. Setting the cache globally
        2. Setting the cache at the project level
      3. Distributed container registry mirroring
      4. Installing and running a proxy container registry and a caching server
        1. Proxy container registry
        2. Caching server
          1. Creating an S3 bucket in Amazon Web Services
          2. Creating your own MinIO server
      5. Scaling your runners
        1. Using Docker Machines with a local VirtualBox instance
        2. Using docker machines that have been created on Amazon Web Services (EC2)
    5. Summary
    6. Questions
    7. Further reading
  28. Monitoring CI Metrics
    1. Technical requirements
    2. Enabling monitoring for Runners
    3. Editing the GitLab Runner configuration file
    4. Runner business logic metrics
      1. Key metrics to watch
    5. General process metrics
      1. Key metrics to watch
      2. Alert management
    6. Summary
    7. Questions
    8. Further reading
  29. Section 5: Scale the Server Infrastructure (High Availability Setup)
  30. Creating a Basic HA Architecture Using Horizontal Scaling
    1. Technical requirements
    2. The underlying architecture of this solution
      1. Amazon services
        1. Elastic compute cloud (EC2)
        2. Classic load balancer
        3. Virtual private cloud and subnets
        4. SGs
      2. Terraform
        1. Installing Terraform
      3. Ansible
      4. Installing the Ansible Terraform provider
      5. Starting with the code
        1. vpc.tf
        2. subnet.tf
        3. instance.tf
        4. ansible_host.tf
        5. route_table.tf
        6. security_group.tf
        7. variable.tf
        8. keypair.tf
        9. lb.tf
        10. providers.tf
      6. Preparing to run Terraform to deploy the virtual hardware
      7. Running the deployment
    3. Setting up the bastion hosts
    4. Configuring the database nodes
      1. Contents of the gitlab.rb.postgres.j2 template
      2. Configuring the consul nodes
        1. Contents of gitlab.rb.consul.j2
      3. Configuring the PgBouncer node
        1. Contents of gitlab.rb.pgbouncer.j2
    5. The Redis configuration
      1. Contents of gitlab.rb.redis.j2
    6. Connecting the shared filesystem
      1. Contents of nfs_exports.j2
    7. Setting up the application servers
      1. Contents of gitlab.rb.j2
      2. Running all the Ansible playbooks
    8. Summary
    9. Questions
    10. Further reading
  31. Managing a Hybrid HA Environment
    1. Technical requirements
    2. The basic architecture of this solution
    3. A renewed Terraform configuration
      1. instance.tf
      2. ansible_host.tf
    4. Splitting application components into frontend and middleware tiers
      1. Splitting Sidekiq from the frontend
        1. Contents of gitlab.rb.middleware.j2
      2. Creating a monitoring instance
        1. Contents of gitlab.rb.prometheus.j2
      3. Creating a monitoring dashboard with Grafana
        1. Contents of gitlab.rb.grafana.j2
    5. Connecting the shared filesystem
      1. Contents of the gitlab.rb.gitaly.j2 file
    6. Changes in Ansible files
    7. Script enhancements
    8. Summary
    9. Questions
    10. Further reading
  32. Making Your Environment Fully Distributed
    1. Technical requirements
    2. The basic architecture of this solution
    3. Performing changes to the Terraform configuration
      1. instance.tf
      2. ansible_host.tf
    4. Splitting more application components
      1. The third application server for Git SSH
        1. Contents of gitlab.rb.frontend_ssh.j2
      2. The middleware layer – Sidekiq
        1. The ASAP Sidekiq instance
          1. Contents of gitlab.rb.sidekiq_asap.j2
        2. The real-time Sidekiq instance
        3. Contents of gitlab.sidekiq.realtime.j2
        4. The pipeline Sidekiq instance
        5. Contents of gitlab.sidekiq.pipeline.j2
        6. The normal Sidekiq instance
    5. The clustered Redis/Sentinel configuration
      1. Contents of gitlab.rb.redis-cluster.j2
    6. Changes in Ansible files
    7. Script enhancements
    8. Summary
    9. Questions
    10. Further reading
  33. Using Geo to Create Distributed Read-Only Copies of GitLab
    1. Technical requirements
    2. The basic architecture of this solution
    3. Preparing the infrastructure
      1. The root module explained – main.tf
      2. Utilizing the keypair module – modules/services/keypair/main.tf
      3. Variables for the keypair module – modules/services/keypair/vars.tf
      4. Outputs from the keypair module – modules/services/keypair/output.tf
      5. Explaining the GitLab module
        1. The main module file – modules/services/gitlab/main.tf
        2. The variable file – modules/services/gitlab/vars.tf
        3. The outputs for the module – modules/services/gitlab/outputs.tf
    4. Setting up Geo
      1. Installing the GitLab software and license
        1. Contents of gitlab.rb.j2
      2. Preparing the database replicas
        1. Contents of gitlab.rb.primary.j2
        2. Contents of gitlab.rb.primary.j2
      3. Changing the SSH key lookup method to the database
      4. Adding the secondary node via the web UI
        1. Activating hashed storage
        2. Checking the status of the secondary node
    5. Summary
    6. Questions
    7. Further reading
  34. Assessments
    1. Chapter 1: Introducing the GitLab Architecture
    2. Chapter 2: Installing GitLab
    3. Chapter 3: Configuring GitLab Using the Web UI
    4. Chapter 4: Configuring GitLab from the Terminal
    5. Chapter 5: Importing Your Project from GitHub to GitLab
    6. Chapter 6: Migrating From CVS
    7. Chapter 7: Switching from SVN
    8. Chapter 8: Moving Repositories from TFS
    9. Chapter 9: GitLab Vision: The Whole Toolchain in One Application
    10. Chapter 10: Create Your Product, Verify, and Package it
    11. Chapter 11: The Release and Configure Phase
    12. Chapter 12: Monitoring with Prometheus
    13. Chapter 13: Integrating GitLab with CI/CD Tools
    14. Chapter 14: Setting Up Your Project for GitLab Continuous Integration
    15. Chapter 15: Installing and Configuring GitLab Runners
    16. Chapter 16: Using GitLab Runners with Docker or Kubernetes
    17. Chapter 17: Autoscaling GitLab CI Runners
    18. Chapter 18: Monitoring CI Metrics
    19. Chapter 19: Creating a Basic HA Architecture by Using Horizontal Scaling
    20. Chapter 20: Managing a Hybrid HA Environment
    21. Chapter 21: Making Your Environment Fully Distributed
    22. Chapter 22: Using Geo to Create Distributed Read-Only Copies of GitLab
  35. Other Books You May Enjoy
    1. Leave a review - let other readers know what you think

Product information

  • Title: Mastering GitLab 12
  • Author(s): Joost Evertse
  • Release date: August 2019
  • Publisher(s): Packt Publishing
  • ISBN: 9781789531282