In this article, we are going to talk about how to use Elasticsearch in Python. If you have no knowledge about Elasticsearch yet, you can read this article for a quick introduction.

Elasticsearch and Python

As a data engineer, you may need to create Elasticsearch documents in Python with some scripts. As a software engineer, when you design your API in Python, you would need to make REST API calls to Elasticsearch to fetch the data. Therefore, if you are using Elasticsearch in your work or plan to learn it, this article can be useful for you.

If you haven’t installed Elasticsearch yet…

When we write Python programs, we often need to print out the results and also log some exceptions. As a beginner, we would normally use the print function to log everything. It is handy to use the print command but the logs are not stored properly and are difficult to check sometime after the program is finished. As a best practice for logging in Python, we should use the built-in logging module. With the logging module, we can log different levels of messages to different destinations. We can also configure the log messages to be sent to some channels so…

Kubernetes is an open-source container orchestration system that automates the deployment, scaling, and management of containerized applications. Kubernetes is commonly referred to as k8s for simplicity since there are 8 characters between k and s. Kubernetes is widely used in the modern software development practice and is available in all major cloud providers such as Google Cloud Platform (GCP) and Amazon Web Service (AWS).

Photo by Samuel Sianipar from Unsplash.

Compared to traditional monolithic and virtualized deployment practices, Kubernetes brings revolutionary changes, including self-healing, automated rollouts and rollbacks, automatic resource management, secret and configuration management, service discovery and load balancing, storage orchestration, etc. If you want…

Redis, which stands for Remote Dictionary Server, is an open-source in-memory data structure store that can be used as an in-memory key-value database, caching system, and pub/sub message broker. Redis is special in that all its data is stored in memory rather than on disk which makes it extremely fast and is a popular option for caching.

Photo by Tyler Clemmensen from Unsplash.

Redis is similar to Memcached, which as the name indicates is also an in-memory key-value data storage. However, Redis has more advanced features which makes it applicable to more scenarios. Unlike Memcached which only supports strings, Redis supports a variety of data structures…

In our Python code, we often need to specify some sensitive information such as database username, password, API keys, JWT tokens, etc. We should not store any sensitive data as plain texts in our source code repository because they can get leaked easily. A common practice is to store the credentials as environmental variables or secret files on the machine where the application is running on. The machine is a generic concept and can be a bare-metal machine, a virtual machine, a docker container, a Cloud Run service, a Cloud Function, etc.

Photo by Kristina Flour from Unsplash.

For simplicity, suppose that we have set up…

All Python developers should have some knowledge about how to deal with the date and time in Python. I assume everybody has already used time.sleep() and in their code. However, sometimes you might come across special cases which are out of your knowledge and you have to turn to Google every time to find the best solution for you, which can be a time-consuming process sometimes. …

Photo by Pankaj Patel from Unsplash.

Cloud functions in the Google Cloud Platform (GCP) is a serverless function as a service (FaaS) framework. Cloud functions can automatically run backend code in response to events triggered by HTTP requests or events emitted from your cloud infrastructure and services. Cloud functions have the following benefits:

  • Serverless — no need to manage the underlying infrastructure and runtime for the functions, everything is handled by the cloud platform.
  • Automatic — the service is automatically scaled based on the usage, with integrated monitoring, logging, and debugging capability.
  • Cost friendly — you only pay when the function is used, not when it’s…

Falcon is a WSGI library for building web APIs using the REST architectural style. Before we get started, let’s get familiar with two important terminologies, which can be mentioned in interviews and the Falcon documentations.

Photo by Erik van Dijk from Unsplash
  • WSGIWeb Server Gateway Interface. It is an interface specification by which web server and application communicate. A WSGI application is just a callable with a well-defined signature so that you can host the application with any web server that understands the WSGI protocol.
  • RESTREpresentational State Transfer. It is a software architectural style that was created to guide the design and development…

A cool new feature called assignment expression was introduced in PEP572 since Python 3.8. Assignment expression uses operator := to assign the value of an expression to a variable. Since operator := resembles the eyes and tusks of a walrus, it is informally known as “the walrus operator”.

Photo by Jay Ruzesky from Unsplash.

The assignment expressions are most commonly used in if and while conditions to simplify code. While they can also be used in other cases as demonstrated in PEP572, it is recommended not to do so in order to improve code readability and avoid confusion.

The major benefit of assignment expressions is that…

Postman is a collaboration platform for API development. If you are a software developer and need to develop, test, and maintain an API or a set of API endpoints, you will find Postman very helpful. In this article, we won’t introduce too many concepts about API and Postman but will focus on some of the most common use cases which shall be handy for your development work.

Photo by Alex Knight from Unsplash.

You can use the web version of Postman directly, or you can download and install it on your computer. …

Lynn Kwong

Senior software engineer and data engineer specializing in Python, JS/TS, Java/Scala, API, ML, GCP/AWS, Kafka, Spark, MySQL, MongoDB, Elasticsearch, Redis, etc.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store