huhu's Blog

FLOWER ON THE PRECIPICE

[学习笔记] - Datadog

发表于 更新于
本文字数： 5.3k 阅读时长 ≈ 5 分钟

Datadog

There are two package for datadog write by Go.

There is no log package for datadog, since it collect from file or stdout.

Datadog Terminology

Agent

we need send data to agent, then the agent send data to datadog.

  • Collector
  • DogStatsD
  • Forwarder

StatsD

if originally a simple daemon developed and released by Etsy to aggregate and summarize application metrics.
With StatsD, applications are to be instrumented by developers using language-specific clients libraries. These libraries will then communicate with the StatsD daemon using its dead-simple protocal, and the daemon will then generate aggregate metrics and relay them to virtually any graphing or monitoring backend.

  • Datadog embedded StatsD daemon within the Datadog Agent
  • Datadog extended the StatsD protocal to support tagging

DogStatsD

Datadog accept all three of the major Datadog data types: metrics, event, service checks. In version 6, DogStatsD is a Golang implementation of Etsy’s StatsD metric aggregation daemon. It is used to receive and roll up arbitary metrics over UDP or Unix socket, thus allowing custom code to be instrumented without adding latency.

dd-trace-go

profiler

This is for sending data like CPU, MEM for datadog. A profiler shows how much “work” each function is doing by collecting data about the program as it’s running. For example, if infrastructure monitoring shows your app servers are using 80% of their CPU, you may not know why. Profiling shows a breakdown of the work, for example:

FUNCTION CPU USAGE
doSomeWork 48%
renderGraph 19%
Other 13%
  • setup
1
go get gopkg.in/DataDog/dd-trace-go.v1/profiler
  • code
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
import "gopkg.in/DataDog/dd-trace-go.v1/profiler"

func main() {
    // init profiler
    err := profiler.Start(
        profiler.WithService("<SERVICE_NAME>"),
        profiler.WithEnv("<ENVIRONMENT>"),
        profiler.WithVersion("<APPLICATION_VERSION>"),
        profiler.WithTags("<KEY1>:<VALUE1>,<KEY2>:<VALUE2>"),
        profiler.WithProfileTypes(
        profiler.CPUProfile,
        profiler.HeapProfile,
        // The profiles below are disabled by default to keep overhead
        // low, but can be enabled as needed.

        // profiler.BlockProfile,
        // profiler.MutexProfile,
        // profiler.GoroutineProfile,
        ),
    )
}
if err != nil {
    log.Fatal(err)
}
defer profiler.Stop()

then you are done, the profiler will send the data automatically.

ddtrace

This is for sending tracing data to datadog.

  • setup
1
go get gopkg.in/DataDog/dd-trace-go.v1/ddtracer
  • code
1
2
3
4
5
6
7
8
9
func main() {
    ddtracer.Start(
        ddtracer.WithAgentAddr("host:port"),
        ddtracer.WithServiceName("serviceName"),
        ddtracer.WithGlobalTag("tag", tag),
        ddtracer.WithSampler(ddtracer.NewRateSampler(rate)),
    )
    defer ddtracer.Stop()
}

There also other options like

  • ServiceMapping
  • HTTPClient
  • UDS
  • RuntimeMetrics
  • ServiceVersion
  • …

ddtrace/tracer

There is a concept called span. span is like a tree. we can have child span, parent span.

Bscially we can consider span as a node with some tags.

1
2
3
// Start a root span.
span := tracer.StartSpan("get.data")
defer span.Finish()

if we want to create a child span

1
2
3
4
// Create a child of it, computing the time needed to read a file.
child := tracer.StartSpan("read.file", ChildOf(span.Context()))
...do something
child.Finish()

we have 3 way to create span

  • StartSpanFromContext
    • if span is in context, the span will become parent span, otherwies start new span.
  • SpanFromContext
    • get span from context
  • StartSpan
    • start a new root span

we can set span to context

1
ctx = ddtracer.ContextWithSpan(ctx, span)

we can set tag for span, this can be seen from the dashboard. span use tag to carry message.

1
2
3
4
5
span， ctx := ddtracer.StartSpanFromContext(ctx, "span.name"
    ddtracer.Tag("tag1", tag1),
    ddtracer.Tag("tag2", tag2),
)
span.SetTag("tag", tag)

ddtrace/ext

This is a package for some common constant, for example

1
2
3
4
5
AppTypeWeb = "web"
SpanName = "span.name"
SpanType = "span.type"
ServiceName = "service.name"
Version = "version"

we can use it in SetTag or other place.

ddtrace/tracer sampling

The tracing client can perform trace sampling. While the trace agent already samples traces to reduce bandwidth usage, client sampling reduces performance overhead. To make use of it, the package comes with a ready-to-use rate sampler that can be passed to the tracer. To use it and keep only 30% of the requests, one would do:

1
2
s := tracer.NewRateSampler(0.3)
tracer.Start(tracer.WithSampler(s))

More precise control of sampling rates can be configured using sampling rules. This can be applied based on span name, service or both, and is used to determine the sampling rate to apply.

1
2
3
4
5
6
7
8
9
10
11
12
rules := []tracer.SamplingRule{
      // sample 10% of traces with the span name "web.request"
      tracer.NameRule("web.request", 0.1),
      // sample 20% of traces for the service "test-service"
      tracer.ServiceRule("test-service", 0.2),
      // sample 30% of traces when the span name is "db.query" and the service
      // is "postgres.db"
      tracer.NameServiceRule("db.query", "postgres.db", 0.3),
      // sample 100% of traces when service and name match these regular expressions
      {Service: regexp.MustCompile("^test-"), Name: regexp.MustCompile("http\\..*"), Rate: 1.0},
}
tracer.Start(tracer.WithSamplingRules(rules))

contrib

This package contain some already written code used to tracing some common service/library like

  • GCP PubSub
  • Gin
  • Gorm
  • net/http

datadog-go

datadog-go is a library that provides a DogStatsD client in Golang. DogStatsD accepts custom metrics, events, and service checks over UDP and periodically aggregates and forwards them to Datadog. DogStatsDis already enabled by defaul at agent. Here we use this pakcage send data to agent.