Observability and SRE Built and Run for Software Teams
Your engineers get fewer pages, each worth acting on, and a written review after every incident. In Dallas-Fort Worth, ITECS covers logs, metrics and traces, objectives set from what customers notice, and alerts that page only when someone must act. It also covers on-call your team can sustain, incident reviews, the observability bill and AI answer quality.
- Who it is for
- Engineering leads whose team carries the pager
- What ITECS builds
- Instrumentation, targets, alerts and runbooks
- What ITECS runs
- Alert rules, the bill report and AI canaries
- Pricing
- Custom-scoped. The assessment's written plan puts the work in order and quotes the first service to instrument. How the quote is set
- After launch
- Your rotation by day, the ITECS NOC at night
Scope
What gets measured and who gets paged
ITECS instruments in your accounts and your tooling: OpenTelemetry where the stack allows it, and the vendor you already pay for where it does not. Every change is a pull request your engineers approve.
- Logs, metrics and traces
- Each service sends the three signals through OpenTelemetry, the open standard most monitoring vendors accept, or through your vendor's agent. Every service also gets a cardinality budget, a cap on the distinct label values its metrics may take, split across its labels. The bill then follows traffic rather than tags.
- Objectives
- A service-level objective (SLO) is a reliability target for something customers notice, such as checkouts that succeed in time. The error budget is the failure that target allows. The service owner, a person you name, signs the policy for when it runs out.
- Alerts that page
- Each alert has one of three severities. A page goes to a person who must act now, a ticket waits for the next working day, and the rest are only recorded. Every page links its runbook, the written steps for it.
- On-call
- The rotation fits the team you have. Your engineering lead sets a ceiling on pages per shift and protects project time for whoever carries the pager. The ITECS NOC can cover the nights and weekends your team cannot.
- Incident reviews
- After an incident, the people involved write down what happened, what they saw and why each step made sense at the time. Each review ends in an action your engineering lead gives an owner.
- Observability bill
- Vendors such as Datadog count a custom metric for each combination of name and tag values, so one tag holding user IDs can multiply the bill. ITECS reads each bill beside the traffic it should follow.
- AI answer quality
- An AI feature gets scored canaries, fixed questions with known answers asked of the live feature, and a drift window that compares recent scores with the accepted version. The service owner sets the score that pages, and a window with too few scored answers reads as unknown.
Scenario
Dashboards that show everything and pages that mean nothing
Illustrative planning scenario, not a reported client result or a named public case study.
- Starting point
- A software company has a dashboard for every service and alerts on most of them. The same three engineers are woken for alerts nobody acts on. The observability bill has doubled while traffic has not, and a new AI feature has no measure of answer quality at all.
- Scoped outcome
- Each customer-facing service gets one objective, signed by its owner. Alerts are cut to the ones a person must act on. The three engineers carry business hours, and the ITECS NOC takes the nights and weekends. The labels that doubled the bill come under a budget, every incident gets a written review and the AI feature gets scored canaries.
Approach
How ITECS brings the pages down
Your current alerts keep paging until their replacements have run silently and your engineering lead agrees they catch the same problems.
Read the pager history
With read access to your monitoring tools, paging tool and bills, ITECS lists every recent page and what the engineer did about it. ITECS then talks with the people on the rotation.
Choose one service
You pick the customer-facing service that hurts most. Its service owner chooses what customers notice and sets a loose first target, tightened later from measured data.
Instrument it
Instrumentation lands as pull requests in your repositories, with the service's budget split across its labels. An AI feature gets its scored canaries, and the bill report starts with the next invoice.
Cut the noise
Alerts nobody acted on become tickets or are deleted once your engineering lead agrees. New alerts run without paging until their firings line up with incidents your team worked.
Staff the hours
The rotation, its page ceiling, the escalation contact and the hours ITECS covers go on one schedule. ITECS writes the first incident reviews with your team, each ending in an action with an owner.
Incident loop
From a signal to a written review
At the diamond a person you name sets the target and signs the error-budget policy. The diamond holds a target, and no change waits at it. Every new alert is written against that target, so none can page before it is set. Each review returns to the diamond, and the same person decides whether the target moves.
Your service
Signal
Logs, metrics and traces leave the service with the labels its budget allows. A trace follows one request from the user's click to the database and back.
Output: Signals within budget
Objective
Your service owner sets the target for what customers notice and signs the error-budget policy. The policy says what the team stops doing when the budget runs out, and who may overrule that.
Output: Signed target and policy
Set by your service owner
On-call
Alert
An alert fires when the error budget is burning fast enough to run out early. A page goes out only at the severity that needs a person now, and it links the runbook.
Output: Alert with its runbook
Page
The on-call engineer gets the page and acknowledges it. In the hours the team cannot staff, the page goes to the ITECS NOC under managed DevOps, with the same runbook.
Output: Acknowledged page
Mitigate
The runbook comes first, to stop the harm, then the fix. The on-call engineer decides when service is restored. In the hours ITECS covers, that is the NOC, which wakes your escalation contact when the runbook falls short.
Output: Restored service
Your team
Review
The people involved write the review without blame. It ends in an action with an owner, and the person who signed the policy decides whether the target changes.
Output: Action with an owner
Review: next target
Dashed: no alert until the target is signed. Left line: reviews go back to it.
Deliverables
What the rotation runs on
- Instrumentation in your repositories, with each service's budget split across its labels and written beside the code.
- One target per customer-facing service, and the error-budget policy its owner signed.
- Alert rules with a severity each, all run without paging before they went live, and a runbook behind every page.
- A rotation schedule with its page ceiling, the escalation contact you name and the hours ITECS covers.
- A review template, and the first reviews written with your team, each one ending in an action with a named owner.
- A bill report with each invoice that sets your metric series, by label, beside your traffic.
- Scored canaries and a drift window for each AI feature, with the score that pages written down.
- Your team keeps admin rights to every dashboard, alert rule and schedule, and the ITECS seats are listed for removal.
Where it fits
Where it connects
Many incidents start with a change. The cluster and the pipeline each feed this loop, and the assessment reads the pager before any work is quoted.
- DevOps assessment
- Reads your pager history, alert rules and observability bill, so you can choose the first service to give a target.
- DevOps services
- Pipelines, platforms, infrastructure code and the managed DevOps retainer that takes your after-hours pages.
- Kubernetes platforms
- Cluster signals such as nodes short of memory, failed rollouts and restarting containers page only when a service's budget is burning. The rest become tickets for your team.
- CI/CD pipelines
- Each deploy is checked against the service's target, so a bad release shows up as spent budget and the on-call engineer can roll it back.
- 24/7 network monitoring
- Under ITECS's network monitoring on itecsonline.com, the same NOC watches the network your services run on.
- Score canaries and watch live drift windows
- The regression-testing guide shows how to score canary questions against a live AI feature and set each drift window before any alarm is turned on.
Pricing
How the quote is set
Observability work is custom-scoped. The quote follows what the assessment finds in your services, your tooling and your pager.
- Assessment
- The first step, priced on a call before it starts. Its written plan records the first service you choose and quotes the work on it.
- What moves the quote
- The services and environments to instrument, the tooling already in place and the on-call hours you want covered.
- Observability build
- Instrumentation, targets, alerts and runbooks, quoted one service at a time in the order you set.
- Managed DevOps
- A retainer for alert-rule upkeep, the bill report with each invoice, canary scoring and the nights and weekends your team cannot staff. The ITECS 24/7 NOC answers those pages, works the runbook and wakes your escalation contact.
- Tooling
- Your observability vendor, paging tool and cloud usage stay on your own invoices. A proposal names anything ITECS passes through instead.
This work is priced apart from the AI packages on this site.
FAQ
Observability and SRE FAQ
What do observability and SRE services include?
Observability makes a production system measurable, and site reliability engineering (SRE) keeps the people on call able to act on what they see. ITECS covers logs, metrics and traces, objectives set from what customers notice, and alerts that page only when someone must act. It also covers on-call your team can sustain, incident reviews, the observability bill and AI answer quality, for teams in Dallas-Fort Worth and across the United States.
Will we have to switch monitoring vendors?
No. ITECS works in the tooling you already pay for. Where your stack allows it, new instrumentation uses OpenTelemetry, so a later move to another vendor needs no new code in your services. When a move would pay for itself, the written plan says so and you decide.
How is an SLO set, and who signs the policy?
Start with one thing customers notice, such as logins that succeed or payments that go through. The indicator (SLI) is good events divided by all events, and the objective is the share that must be good. Start loose and tighten it from measured data. The service owner, a person you name who answers for that service, signs the error-budget policy that goes with it.
How do pages stop waking people for nothing?
Every alert that paged is listed with what the engineer did about it. Alerts nobody acted on become tickets or are deleted, once your engineering lead agrees. New alerts run without paging until their firings line up with incidents your team worked, and every page links the runbook that says what to do.
Can a small team run on-call at all?
Yes, if pages are rare and each one is real. A small team can carry business hours with a ceiling on pages per shift and project time protected for whoever is on call. The nights and weekends it cannot staff go to ITECS under managed DevOps, and your engineers are woken only when the runbook does not restore service.
How do you keep the observability bill in line with traffic?
Most runaway bills come from labels with unbounded values, such as user or request IDs. ITECS sets a cardinality budget per service, split across its labels, and drops or samples the logs your engineers agree they never read. Each bill is read beside the traffic, so growth that traffic does not explain is found and traced to its label.
How do you watch AI features in production?
Like any other service, with a target and an alert. Canary questions with known answers run against the live feature and are scored, and a drift window compares recent scores with the accepted version. The service owner sets the window, how many scored answers it needs and the score that pages. Quality can slip without a release when sources change, so the canaries keep running.
Should we hire an SRE instead?
Hire one if you can, and give them the authority to delete alerts. One SRE cannot carry every night, and when they are on leave the pager falls to whoever is free. ITECS has operated managed IT and security since 2002, with a 24/7 NOC that answers alerts through the night. Under managed DevOps it takes the pages your rotation leaves uncovered. When you hire, the new SRE takes over the runbooks, the alert rules and the policies.
Bring your pager history
Send the alerts that woke someone recently, the dashboards nobody opens and the latest observability bill. You choose the first service to give a target, and the assessment's written plan quotes the work.