Learn the new features and syntaxes of Pydantic V2 — Data validation stands as a cornerstone for robust applications in the ever-evolving field of data engineering and software development. Ensuring data cleanliness and accuracy is essential not only for application reliability but also for user experience. Pydantic is the most widely used data validation library for Python. The core of…

Demystify the mysterious generic types in Python — In this post, we will introduce the basics of type variables and generic functions and classes with simple code examples. Using type variables and generic types can make our code more robust and easier to read. Even if we may not have many chances to use them directly in our…

Learn to use word boundaries and lookahead assumptions to solve practical problems — In this post, we will showcase a selection of special applications for regular expressions in Python, aimed at resolving real-world challenges. As we will see in our examples, we can use word boundaries and lookahead assumptions to achieve more accurate matching. …

Learn a handy tip to test your web application on the phone — When developing a web application with Angular, or any other framework, we need to test it on our phones frequently. Even though we can use the “device toolbar” simulator in Chrome for testing, sometimes we still need to use a real phone for testing. For example, when we want to…

Learn a handy tool to check package dependencies in Python — Writing a requirements.txt file from scratch for your application can be cumbersome because you need to sort out what packages need to be installed and what are their dependencies. Worse still, it’s tedious to resolve conflicting dependencies, especially on legacy systems when an older version of pip is used. Luckily…

Learn how to scrape JavaScript webpages with a smart proxy — In this post, we will introduce how to use the Zyte Smart proxy with Splash which is integrated with the Scrapy web scraping framework. We will learn how to set up the Zyte smart proxy, how to use it with Splash, and how to set up a headless proxy tool…

Learn to build and publish both Python packages and Docker images with Cloud Build — In previous posts, we introduced how to store Python packages and Docker images in Google Artifact Registry, respectively. In this post, we will try to connect these two techniques and introduce how to install private Python packages in Artifact Registry when building Docker images with Cloud Build. Using Cloud Build…

Learn a simple way to perform smooth transitions — Container Registry has been deprecated since May 15, 2023. In order to get the latest updates, features, and support for the services, we should transition from Container Registry to Artifact Registry. In this post, we will first introduce the differences between Container Registry and Artifact Registry and then the procedures…

Learn a simple way to build a production-ready Splash server in the Cloud — In previous posts, we introduced how to start Splash locally using Docker. In production, we would want to put the Splash server somewhere all our scraping projects can access. Since Splash is Dockerized, a simple solution is to start a Splash microservice in Cloud Run. It may seem simple to…