In this article, we are going to talk about how to use Elasticsearch in Python. If you have no knowledge about Elasticsearch yet, you can read this article for a quick introduction.

Elasticsearch and Python

As a data engineer you may need to create Elasticsearch documents in Python with some scripts. As a software engineer, when you design your API in Python, you would need to make REST API calls to Elasticsearch to fetch the data. Therefore, if you are using Elasticsearch in your work or plan to learn it, this article can be useful for you.

If you haven’t installed Elasticsearch yet…


When we write Python programs, we often need to print out the results and also log some exceptions. As a beginner, we would normally use the print function to log everything. It is handy to use the print command but the logs are not stored properly and are difficult to check sometime after the program is finished. As a best practice for logging in python, we should use the built-in logging module. With the logging module, we can log different levels of messages to different destinations. We can also configure the log messages to be sent to some channels so…


A cool new feature called assignment expression was introduced in PEP572 since Python 3.8. Assignment expression uses operator := to assign the value of an expression to a variable. Since operator := resembles the eyes and tusks of a walrus, it is informally known as “the walrus operator”.

Photo by Jay Ruzesky from Unsplash.

The assignment expressions are most commonly used in if and while conditions to simplify code. While they can also be used in other cases as demonstrated in PEP572, it is recommended not to do so in order to improve code readability and avoid confusion.

The major benefit of assignment expressions is that…


Postman is a collaboration platform for API development. If you are a software developer and need to develop, test, and maintain an API or a set of API endpoints, you will find Postman very helpful. In this article, we won’t introduce too many concepts about API and Postman but will focus on some of the most common use cases which shall be handy for your development work.

Photo by Alex Knight from Unsplash.

You can use the web version of Postman directly, or you can download and install it on your computer. …


A decorator is a function that takes another function as input, extends its behavior, and returns a new function as output. This is possible because, in Python, functions are first-class objects, which means they can be passed as arguments to functions and also be returned from functions, just as other types of objects such as string, int, or float.

Photo by laura adai from Unsplash

Let’s first demonstrate how a decorator works. Once you understand it, you won’t use it as a black box anymore.

Let’s create a super simple function:

def func1():
print("Inside func1.")
>>> func1()
Inside func1.

Then let’s create a simple decorator, which…


Python is a dynamically typed programming language, which means the types are only checked at runtime and a variable is allowed to change its type over its lifetime, whereas a statically typed language like Java checks the types at compile-time, and a variable is not allowed to change its type over its lifetime. On the other hand, Python is a strongly typed language because the types cannot be automatically converted at runtime. For example, you cannot have an addition calculation on integer 1 and string "2", while in a weakly typed language such as JavaScript such calculation is allowed.

Photo by Hitesh Choudhary from Unsplash.

Even…


Google is a great source for obtaining data, either in your personal projects or in real business projects. Google has a Custom Search Engine (CSE) API which can be conveniently used in your Python code. It has a free version which is enough for studying purpose or even small projects which have less than 100 search queries every day.

Photo by Benjamin Dada from Unsplash.

To get started using Google Custom Search Engine, you need to have a valid API key, which can be obtained here. You need to log in with your Google account and create a project for it. You can create a project…


Photo by Markus Winkler from unsplash.

In this article, some advanced CRUD and search queries for nested objects in Elasticsearch will be introduced. If you want to learn more about the basics about Elasticsearch you can check these articles for a quick start or refresh:

In case you just want to learn nested objects in Elasticsearch. You can run this command in Kibana to create the index which will be used later.

In this example, the attributes field is a nested…


Photo by Chris Ried from unsplash

Good Cloud Storage can be a convenient option if you want to store your data in the cloud programmatically, especially if you use the Google Cloud Platform. You can store any data in your work, such as plain text files, images, videos, etc. As a beginner you may prefer to use the Google Cloud Console to manage you files, which is very straightforward to use. However, as a developer, you would need to use the command line tool or the client library in your code to deal with Google Cloud Storage more programmatically.

Before we introduce the command line tool…


Python 2.7 is not supported after Jan. 1st, 2020. After the first release in July 3, 2010, Python 2.7 has been active for 10 years and a large number of projects are written in this version. Most Python 2 projects should still be able to work properly as Python 2.7 is a very stable version and most bugs have been fixed in the past 10 years. However, with Python 2.7, you are not able to use many cool features introduced in Python 3, such as f-string, typing and assignment expression. Particularly, typing has become a must-to-have for new Python projects…

Lynn Kwong

Senior data engineer specialized in Python, JavaScript/TypeScript, Java/Scala, MySQL, MongoDB, Elasticsearch, API, Big Data, Cloud Computing, Git, etc.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store