Learn an efficient way to work with large datasets — When you have a large dataset like a big CSV file or a big SQL table, it’s inefficient or even impossible to load all data into memory. Your computer will get stuck and your program will crash. It can be time-consuming and also quite frustrating to debug. Fortunately, iterators and…

Learn a better way to manage your sensitive data — An application normally needs some sensitive data like API keys, database passwords, private keys, etc. It’s common to store them as environment variables. However, this is not a good practice because the values of environment variables are set and stored as plain texts and thus can be revealed easily. If…

Learn to deal with GCP local authentications in minutes — If you work on the Google Cloud Platform (GCP), having your local environment set up properly can make your life much easier. It can be a headache to deal with all the nitty-gritty authentication issues especially when you have just started to work with GCP. Don’t worry, in this post…

Learn different ways to read and write CSV files in Python — Comma-separated values (CSV) files are very commonly used to store data. In this post, we will introduce how to work with CSV files in Python. We will start with a simple example and expand it with some special cases. Both the classical reader/writer and the slightly more advanced DictReader/DictWriter will…

Get started with query_string queries in Elasticsearch — When it goes to boolean operations such as NOT, AND, and OR, we normally use bool queries with must, should, must_not clauses. Yes, bool queries are very powerful and can be used to perform all types of advanced searches. However, for simple searches with basic NOT, AND, and OR conditions…

Learn different ways to index documents in bulk efficiently — When we need to create an Elasticsearch index, the data sources are normally not normalized and cannot be imported directly. The original data can be stored in a database, in raw CSV/XML files, or even obtained from a third-party API. In this case, we need to pre-process the data to…

Some tips to help you deal with breaking changes of Elasticsearch — There are quite a few breaking changes in version 8 of the Elasticsearch Python client library, which will give you a lot of trouble when you update the library from version 7 to 8. Even though it can be a painful task, it is still recommended to update the library…

Start Elasticsearch and Kibana with Docker and Docker Compose in minutes — For the new major version of Elasticsearch (8.x.x), there are significant updates on running Elasticsearch and Kibana on Docker. The commands and syntax for Docker and Docker Compose that used to work for previous versions would need to be updated in order to work for the latest version. In this…

Connect to Cloud SQL instances with private IPs in minutes — If you use the Google Cloud Platform (GCP), it’s common to access Cloud SQL in your Cloud Run services. This is supposed to be an easy task because both Cloud SQL and Cloud Run are in the same network. However, when you do it yourself, you may not be able…