How to Visualize Google Logs with Grafana

Monitor your Cloud data with Grafana

We have introduced how to write logs to Google Cloud Logging in our Python applications. We have also introduced how to visualize our data with Grafana. Now it’s time time to combine both and try to visualize or monitor our Google Logs with Grafana, which can have a more direct and more vivid way of checking the status of our applications.

Image by neotam on Pixabay.

For Google Logs, there are two types of metrics, namely Counter and Distribution types. Counter metric can be used to monitor the frequency of some types of logs, and Distribution metric can be used to monitor the trend of some numeric values extracted from the logs. With these types of metrics, you can keep a close eye on your applications in Grafana and take actions to deal with potential errors in our applications as soon as possible.

If you have been working with GCP before and have everything set up already, you can proceed directly to the following part of the tutorial.

Remember to set an environment variable for the service account key downloaded:

The name of the environment variable must be GOOGLE_APPLICATION_CREDENTIALS otherwise it won’t work. This environment variable is used by the Google SDK for authentication.

Besides, we need to install the client library for Google Logging. It is recommended to create a virtual environment and install the library there so it won’t mess up system libraries. For simplicity, we will use conda to create the virtual environment.

Now everything is set up properly for writing logs to Google Logging.

In order to visualize the Google Logs in Grafana, we need to send some logs first. We will use log_text and log_struct…