Continuously Generating Code Script for Python

Who Should Read This?

Anyone who writes useful Python scripts or programs and wants a good way to demonize them in a standard, predictable way. If your scripts are fine running as 'one-offs' — then this article will be of limited use. However, if you intend for your programs to run continuously, in the background, this article will be of immense benefit to you.

edit: I have had some readers ask: why not just use Docker? Docker solves any portability issues you may have, but what will you run in your Docker container? Following this tutorial inside of a Docker container will give you the best of both worlds! (and perhaps I will make an article on this in the future).

Github Repo

If you just want the source code, follow this link. Feel free to use it as a starting point for your future projects.

DISCLAIMER: I assumed the reader has a decent knowledge of Python and Linux for this article, but feel free to check out this tutorial if you are just getting started with Python.

Services

A service is a process or group of processes (also known as daemons) that run continuously in the background. They provide a wide range of important functionalities that makes your system work properly. For example, httpd and sshd are the services which provide your system with SSH and HTTP.

In Linux, systemd is the system manager that deals with all of the nitty-gritty service management that makes your system work.

Wouldn't it be cool if your Python scripts were system services?

Well yes, it would, and by doing this, we will be granted several benefits:

disaster recovery
standardized log management
status reporting
share-ability

Starting Point: a simple Python Script

If you have spent any amount of time working with Python, you have surely ended up with a script that you are proud of that accomplishes a useful task. Maybe you wrote a script that converts your documents to PDFs at the end of the day, or automatically zips and emails files in a certain directory.

Whatever it is, you probably ran your program like this:

          python myprogram.py some arguments

And if you wanted to share this program, you could create a requirements.txt , and send it over to coworkers so they could install it on their machine. (edit: some readers have wondered why I didn't mention setup.py. You could use that instead if you want).

If you wanted to run it continuously in the background, maybe you would add a handy while True: and let your program run indefinitely in a terminal session.

There is nothing inherently wrong with this approach — but perhaps we can do better.

Level Up your Program

There are a few features we will add to our simple program, and by the end of the article, you will be a certified 'Python Service' writer.

If our program fails, it should do so gracefully — logging its error code so we can inspect what went wrong.
Upon failure, our program should try to restart itself.
Our program should start on system boot-up.
Our program should be easy to share and install!

All of this can be accomplished using systemd, which is the Linux service manager. It is what we will use to transform our fledgling python script into a well-behaved program.

Step 1: The basic Python Script

Our basic program will simply log file changes in a target directory using watchdog — a python library for filesystem monitoring.

Our simple program loads a couple of environment variables (VERBOSE: do we actually want to output anything, DIRECTORY: the directory to monitor for changes), spawns an Observer, and runs indefinitely.

mypythonservice.py

This script is fine as a standalone program — but as discussed, we are going to make some improvements.

Step 2: Add requirements.txt

This is just good form for any Python script that you create. Simply run:

          pip freeze > requirements.txt

Now, anyone you share this script with will have a list of dependencies to install if they want to run your script.

Step 3: Add program.service

Alright, time for the interesting bits. Systemd will need a .service file — which is a configuration file that describes your program, how to run it, and some other useful details. Here is what ours will look like:

          [Unit] Description=Sample Python System Service  [Service] WorkingDirectory=/usr/local/lib/mypythonservice/ Environment=PYTHONUNBUFFERED=1 EnvironmentFile=/usr/local/etc/mypythonservice/mypythonservice.env ExecStart=/usr/local/lib/mypythonservice/venv/bin/python3 /usr/local/lib/mypythonservice/mypythonservice.py  Restart=on-failure  [Install] WantedBy=default.target

Let's break down some of the important fields we are using:

**Environment: **This simply allows Python print/log statements to show up in our system logs.

**EnvironmentFile: **Loads in our mypythonservice.env file, which we have yet to cover, which contains some user-configured environment variables.

**ExecStart: **The command that gets run when we start our program. Notice how are actually using a python3 interpreter that is in a virtual environment in our program directory? More on that later.

**Restart: **If our program fails: restart it

Step 3: Add mypythonservice.env

As the previous section mentioned, we use an environment file to let the user configure their program. There are many ways to accomplish this, but this works well for simple programs with minimal configuration requirements:

          VERBOSE=1

If you look back at our Python code, this will enable the logging for our script, since if verbose: will evaluate to true.

Step 4: Creating the Makefile

We now have all of the files we need to make our program work, but they need to be installed.

By convention, there are standard directories where we will copy our project files:

Source files will be copied to a program directory in /usr/local/lib/
Configuration files will be copied to a program directory in /usr/local/etc
A virtual environment will be created in /usr/local/lib for our Python interpreter with the proper dependencies installed.

You might be asking yourself — "this sounds tedious, can we just automate this please?"

The answer is yes. This a perfect use-case for GNU Make. Take a look at our Makefile, which is just a set of instructions for installing/uninstalling this program.

Makefile

Making sure it works

Before you run around telling everyone you have this cool new thing you made — let's make sure it works.

Install

If you don't have makealready, run sudo apt install make.

Simply run sudo make install from your project directory to install.

Installation

Looks like everything worked O.K. — some strange warnings about pip cache, but we can't be bothered with such tangential topics in this basic tutorial.

Enable and Start

We can now use systemctl to interact with our program, which is a super handy way to check to make sure everything is working properly.

First, with enable we enable our service, which will now start on reboot.

Second, we start our service, it is now running as a service.

Third, we check up on the status of our service, which looks like it is running fine.

Enable and start our program

We can even see a couple of logs that were created in our monitored /tmp/ directory — it looks like my local Kafka instance is busy even when I am not using it!

Perusing the logs

journalctl can be used to explore logs created by your service.

journalctl logs

Cleanup — uninstalling your service

Uninstall

And just like that, your system is rid of this example program. Pretty convenient eh?

What's Next? Onward and Upward.

I hope at this point you are convinced of the amazing practicality of packaging your Python programs as systemd services. I have really just scratched the service of most of the concepts here — but using this approach allows Python developers to robustly run, monitor, and share their programs.

So what now? Next time you automate something on your home machine or develop a nifty tool for your team — turn it into a systemd service. Your friends and coworkers will be amazed by how easy and intuitive it is to get your software running.

References

https://github.com/adamcyber1/mypythonservice
https://www.freedesktop.org/wiki/Software/systemd/
https://pypi.org/project/watchdog/
https://www.digitalocean.com/community/tutorials/how-to-use-systemctl-to-manage-systemd-services-and-units

lamondwithein.blogspot.com

Source: https://plainenglish.io/blog/turning-your-python-script-into-a-real-program-cb702e16ed02

Lamond Withein