Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bigquery/storage/managedwriter: provide opt-in dynamic interface for stats tracking #5100

Closed
GlenDC opened this issue Nov 9, 2021 · 4 comments
Assignees
Labels
api: bigquery Issues related to the BigQuery API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.

Comments

@GlenDC
Copy link

GlenDC commented Nov 9, 2021

Is your feature request related to a problem? Please describe.

It is not really a problem in the intuitive sense, it is however extra baggage that isn't always desired or fully utilised.
Currently the stats are being tracked using go.opencensus.io/stats. This hardcoded dependency means that:

  • I either use that platform to get stats I want;
  • or I do not use the platform and than there is no way for me to get the stats;

Describe the solution you'd like

I would have imagined that similar to how you can often implement your own logger,
one could implement their own stats tracker. You could still provide a solution using opencensus,
but at the very least it would also allow me to get get these tracks using a very different platform.

Describe alternatives you've considered

  • fork managedwriter, but that puts a lot of extra burden on my team and myself for not too much gain;
  • live with it and have no stats: not too big of a deal should this be the result of it all, we can probably live without;
  • use opencensus but that is not going to happen as it isn't used anywhere else in our organisation;

Additional context

N/A

@GlenDC GlenDC added the triage me I really want to be triaged. label Nov 9, 2021
@product-auto-label product-auto-label bot added the api: bigquery Issues related to the BigQuery API. label Nov 9, 2021
@meredithslota meredithslota added type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. and removed triage me I really want to be triaged. labels Nov 9, 2021
@shollyman
Copy link
Contributor

Generally, this is solved with a metrics exporter, not by making the instrumentation code itself dynamic/pluggable.

Personally, I'm not super enamored with opencensus, but as its successor (opentelemetry) still has not stabilized their metrics APIs it wasn't a viable alternative. Primarily the reason we're using opencensus is precedent with related services; it's how libraries like pubsub are instrumented as well.

@GlenDC
Copy link
Author

GlenDC commented Nov 9, 2021

I see. In that case I guess we're fine, as long as people can plugin their own system by registering the desired exporter, be it an existing one or a custom one, it should be fine I guess. Thank you for that info :) every day is a school day

@GlenDC
Copy link
Author

GlenDC commented Nov 10, 2021

Shall I close this issue by the way, as for me this is sufficient of an answer. The only thing that I would suggest is to add this information to the documentation, as it might be obvious if you're familiar with the ecosystem, but to me that answer didn't occur to me at all. Either way, thanks for the help :) Definitely helped me out.

@shollyman
Copy link
Contributor

shollyman commented Nov 11, 2021

I'll go ahead and close this. I've already got a task to show this more clearly through some samples.

However, if others come across this report, here's a snippet from a benchmarking server that sets up metrics exporting into cloud monitoring via contrib.go.opencensus.io/exporter/stackdriver (cloud monitoring was formerly branded as stackdriver, hence the package name):

func (s *Server) startExporter() error {
	exporter, err := stackdriver.NewExporter(stackdriver.Options{
		ProjectID: s.projectID,
	})
	if err != nil {
		return err
	}

	if err := view.Register(managedwriter.DefaultOpenCensusViews...); err != nil {
		return err
	}

	if err := exporter.StartMetricsExporter(); err != nil {
		return err
	}

	// Add flush and stop to shutdown tasks.
	s.shutdownFuncs = append(s.shutdownFuncs, func() {
		exporter.Flush()
		exporter.StopMetricsExporter()
	})
	return nil
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the BigQuery API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.
Projects
None yet
Development

No branches or pull requests

3 participants