Notes on Udacity's Intro to DevOps course. See the course wiki for more.
- Integrates Agile and Lean principles in the operations life cycle
DevOps is the practice of operations and development participating together in the entire service life cycle, from design and development to production support.
A cultural and professional movement, focused on how we build and operate high velocity organizations, born from the experiences of its practitioners.
Agile development has lead to greater development team productivity and product success. However, teams within the organization can still be siloed, and having an optimal silo doesn't guarantee optimal flow for the entire organization:
To fix this DevOps integrates agile and lean principles to break the barrier between teams. This builds a better product idea to customer value flow.
CAMS: Devops is About CAMS
- Culture: People over Process over Tools. People and process first. If you don’t have culture, all automation attempts will be fruitless.
- Automation: DevOps integrates development side automation and operations side automation, bringing these together in deploying automation.
- Measurement: You can't improve what you can't measure. DevOps monitors metrics as much as possible. Metrics relevant to operations, development, business operation, etc.
- Sharing: Comprehensive stake holders create the loop back in the CAMS cycle. Shared input leads to shared responsibility and ownership.
Read the post What Devops Means to Me for more in depth discussion on CAMS. Additionally, read Atlassian's Three ingredients for great software releases for a concise overview of software releases.
Great software development involves the entire team from product management to operations. All your lean planning and iterative development won't mean a thing if you can't ship quickly. DevOps lives in this space.
Differences between development and production environments can be a source of tension between teams. We will use Packer to build deployment images for both dev and production environments.
Golden image: Master image of machine contains all dependencies and can be reused without configuration.
- Fast boot & install
- Large upfront cost
- Immutable image
Configuration Management: Manages dependency installation on machines that have a common OS image.
- Lighter build process
- Easy to update
- Slower startup
Can also use a hybrid of the two approaches.
See project setup and install the software as described.
Per Packer's Getting Started:
Packer is an open source tool for creating identical machine images for multiple platforms from a single source configuration.
Next up, read the Intro to Packer, Packer Terminology, and the template overview.
Now you should have a good understanding of what Packer is and what you can use it to accomplish. Read the following docs to continue:
At this point, you should make sure you've cloned the course repo as explained in setup. Follow the steps in that repo's readme to build your first packer box.
- Local: A developer's workstation.
- Sandbox: A dev server for the dev to work on their branch.
- Integration: Often a CI server where changes are tested and merged to the main branch.
- Staging: A mirror of production and has fairly recent data.
- Production: The environment finally deployed to.
Workflow uses VCS to sync changes (often using a branching strategy). Here's the idea:
Taken from Frederic Dewinne's The Continuous Talk.
CI integrates changes into existing code bases by watches a branch for changes. When changes are committed, the CI server spawns a build process, and then runs tests on the build artifacts. Read this post for more.
Jenkins is the CI tool that we'll use for this course. If you've launched your cloud provider account and built a packer box in lesson 2, you're good to go. If not, follow these instructions to do this now.
Run:
packer build -only=<cloud service target> control-server.json
Navigate to your instance's IP/jenkins url. Eg: 123.456.78.9/jenkins and login with vagrant, vagrant as credentials. Explore your dashboard or the one here to familiarize yourself with the layout. Then read the getting started to become familiar with Jenkins Pipelines. Additionally, you'll want to have a look at the console's build pipeline.
Boxes represent steps to run and their color represent's their status. Green == Passing, Yellow == Running, Blue == Queued, Red == Failed. Note that you can change these under "Configure":
Testing helps identify bugs and ensure the quality of our code.
- Unit Tests: Written alongside code to test behavior of individual functions or classes.
- Regression Tests: Written for debugging to verify that a bug is fixed and not reintroduced.
- Smoke Tests: Preliminary test to make sure the system works before testing further.
- System Integration Tests: Tests whole system including dependencies (apis, databases, etc).
- Acceptance Tests: Verifies the system implements the user facing features as planned.
- Manual QA Tests: Approval process to continue deployment.
For more, see Udacity's Software Testing: How to Make Software Fail.
Bug tracking is also important. Having a unified place to identify, track, and collaborate on bugs makes resolving them faster. Read this article on why to use a issue tracking system.
Now we have automated from deployment process from the developer's local machine to a staging area by building environments and testing them as discussed above. At this point, our code should be in a staging environment. The last step between Continuous Integration and Continuous Deployment is a manual trigger.
Good monitoring is a critical feedback loop for the organization, and should be widely accessible for different organizational units . For marketing and sales, this can provide KPIs such as traffic, growth, pageviews, etc. For development, monitoring can identify bugs. For Ops, it can provide facts about your infrastructure utilization such as CPU usage, response codes, or network bandwidth.
- External Probing: External interaction with system, recording the results. Eg: test queries.
- Application stats: Queries per second, latency, etc
- Environment stats: Memory profiling, etc
- Host stats: Load avg, disk errors, etc
- Logs
- Alerts
- Performance analysis
- Capacity prediction
- Growth measurement
- Debugging metrics
Picking which to use depends on your needs. For small projects, something small and easy to set up is better than a comprehensive suite meant to manage hundreds or thousands of nodes and services. One the other hand, if you have a huge system, you should pick a system that is meant to be very scalable.
Here's a few common tools:
- Nagios and Zabbix - comprehensive solutions for monitoring large infrastructure, but maybe too big and complex for small projects.
- Graphite: Opensource database and a graphing solution for storing and displaying monitoring data.
- InfluxDB: an open-source distributed time series database for metrics, events, and analytics.
- StatsD: Simple daemon for easy stats aggregation, by Etsy. Read about the philosophy behind it in the article by it's creators - Measure Anything, Measure Everything
- Grafana: metrics dashboard and graph editor for Graphite and InfluxDB
- PagerDuty: incident resolution lifecycle management platform that integrates with over 100 other systems to streamline the process for large organisations.
- Logstash: log storage and search system, works well with - Kibana graphing and visualization software.
Congrats! You've reached the end of Intro to DevOps. Be sure to continue learning by checking out the resources in the course wiki.