influxdb/gather
Chris Goller bb09a74732 fix(gather): add all subscribers to a single nats queue group
Previously, scrapers would scrape the target 10 times.  This was
because each scraper subscriber was not put into a queue group.

I've added the queue group "metrics" and now we the subcribers
will only scrape the target once.

Additionally, I moved to using nats memory instead of nats file
store.  We don't need durability for scraper runs across restarts.
2019-01-22 23:05:24 -06:00
..
README.md add(gather): add writer interface 2019-01-11 13:09:18 -05:00
handler.go add(gather): add writer interface 2019-01-11 13:09:18 -05:00
metrics.go fix(scraper): remove enriched type label 2019-01-17 10:22:22 -06:00
metrics_test.go fix(scraper): remove enriched type label 2019-01-17 10:22:22 -06:00
prometheus.go add(gather): add writer interface 2019-01-11 13:09:18 -05:00
recorder.go fix(gather): add all subscribers to a single nats queue group 2019-01-22 23:05:24 -06:00
scheduler.go fix(gather): add all subscribers to a single nats queue group 2019-01-22 23:05:24 -06:00
scheduler_test.go add(gather): add writer interface 2019-01-11 13:09:18 -05:00
scraper.go add(gather): add writer interface 2019-01-11 13:09:18 -05:00
scraper_test.go add(gather): add writer interface 2019-01-11 13:09:18 -05:00

README.md

How to use this package

Make sure nats is running. Both publisher and subscriber are open

// NATS streaming server
m.natsServer = nats.NewServer(nats.Config{FilestoreDir: m.natsPath})
if err := m.natsServer.Open(); err != nil {
    m.logger.Error("failed to start nats streaming server", zap.Error(err))
    return err
}

publisher := nats.NewAsyncPublisher("nats-publisher")
if err := publisher.Open(); err != nil {
    m.logger.Error("failed to connect to streaming server", zap.Error(err))
    return err
}

subscriber := nats.NewQueueSubscriber("nats-subscriber")
if err := subscriber.Open(); err != nil {
    m.logger.Error("failed to connect to streaming server", zap.Error(err))
    return err
}

Make sure the scraperTargetStorageService is accessible

scraperTargetSvc influxdb.ScraperTargetStoreService = m.boltClient

Setup recorder, Make sure subscriber subscribes use the correct recorder with the correct write service

recorder := gather.PlatformWriter{
    Timeout: time.Millisecond * 30,
    Writer: writer,
}
subscriber.Subscribe(MetricsSubject, "", &RecorderHandler{
    Logger:   logger,
    Recorder: recorder,
})

Start the scheduler

scraperScheduler, err := gather.NewScheduler(10, m.logger, scraperTargetSvc, publisher, subscriber, 0, 0)
if err != nil {
    m.logger.Error("failed to create scraper subscriber", zap.Error(err))
    return err
}