Modular Coding in Python: Finally Solve your Import Errors

A Beginner-Friendly Guide to Understanding and Fixing ModuleNotFoundError and ImportError in Your Python Code

Published in

Level Up Coding

11 min readFeb 9, 2024

Just a cool intro image, we will explain everything going on in it in this article :)

When our code gets too large or complex, a good idea is to break it down into smaller more manageable parts. This is called modular programming. It is a very common and useful way to structure our code. It provides better organization, reusability, and maintainability.

However, many Python programmers (including me) often encounter issues during development, such as ModuleNotFoundError and ImportError.

So, I decided to write this article to provide the foundation, and offer solutions and explanations to deal with these issues once and for all.

Outline

Modules in Python with an Example
Explaining the Import Errors
Steps for Finding and Loading a Module
Solving Module Import Errors
Implicit Imports
Summary

1. Modules in Python with an Example

Modules give us the ability to store functions, variables, arrays, dictionaries, objects, and more, so we can easily use them again in different parts of our code. Instead of writing one-script application with multiple thematic blocks like this example:

my_app/
|
│app.py

app.py:

# This is a script has multiple code blocks
# app.py 

# Configs block
database_url = "example.com/db"
api_key = "your_api_key"
debug_mode = True
config_constant = 10

# Functions block
def add_config_constant(a:int)->int:
    return a + config_constant

# Prints and usage of the previous parameters and funcion blocks
print("Database URL:", database_url)
print("API Key:", api_key)
print("Debug Mode:", debug_mode)

result_add = add_config_constant(5)
print("Add config result:", result_add)

Example repo link.

We can separate those blocks of our code, and add them in specific folders/directories:

my_app/
|
│app.py
|
───inner_folder/
|  |
│  │   config.py
|  |   helper_functions.py

Example_2 repo link

app.py :

print(f"running {__name__}")

from inner_folder import helper_functions

result_add = helper_functions.add_config_constant(5)

print("Add config result:", result_add)

inner_folder/config.py :

print(f"running {__name__}")
# an example of variables that someone could have on config.py

database_url = "example.com/db"
api_key = "your_api_key"
debug_mode = True
config_constant = 10

inner_folder/helper_functions.py :

import config

print(config.database_url)
print(config.config_constant)


def add_config_constant(a: int) -> int:
    return a + config.config_constant

We selected the case where helper_functions.py depends on config.py in folder/directory and app.py depends on helper_functions.py. Such modular structure (and much more complex in real-world cases) is quite common in Python projects.

Figure 1: A simple example of modular structure. `app.py` depends on helper_function.py which is in the /inner_folder. Also, helper_function.py depends on config.py.

However, the Python community often mentions a confusion regarding the correct way to import a module especially when the structure of the project becomes large and complex [1]. Such cases lead to ImportError and ModuleNotFoundError .

For example, if we run app.py we will get ModuleNotFoundError: No module named ‘config' However, running helper_functions.py alone will produce no error.

Let’s see this issue in more detail by editing helper_functions.py and providing two different ways to import config.py using try-except:

# inner_folder/helper_functions.py 

# Idea for the example was taken from: 
# https://stackoverflow.com/questions/43728431/relative-imports-modulenotfounderror-no-module-named-x

print(f"running {__name__}")

# Relative import for config
try:
    from . import config

    print('Relative import was successful')

except (ImportError, ModuleNotFoundError):
    print('Config relative import (from . import config) failed')

# Absolute import for config
try:
    import config

    print('Absolute import was successful')

except (ImportError, ModuleNotFoundError):
    print('Config absolute import (import config) failed')

def add_config_constant(a: int) -> int:
    return a + config.config_constant

Example_3 repo link

Here, we have two kinds of imports a relative: from . import config and an absolute: import config (like the previous version of helper_functions.py)

Note: Relative and Absolute Imports in Python.

Absolute imports use the full path from the project's root directory to 
specify the module's location (e.g., from script_path import module or 
import script_path.module).

Relative imports refer to modules in relation to the current script or module. 
Specifically, a single dot (.) represents the current directory 
(from . import module). You can also use two dots (..) to refer 
to the parent directory.

case a) Running app.py again, we get:

running __main__
running inner_folder.helper_functions
running inner_folder.config
Relative import (from . import config) was successful
Config absolute import (import config) failed       
Add config result: 15

Figure 2: Case a) Running app.py the absolute import of the helper function is successful. The absolute import of config by the helper function fails. However, the relative import succeeds.

In this case the relative import of config.py: from . import config was successful. However, the absolute import: import config failed (as we saw previously).

case b) Running inner_folder/helper_functions.py we get:

running __main__
Config relative import (from . import config) failed
running config
Absolute import (import config) was successful

Figure 3: Case b) Running helper_functions.py directly and keeping the same code as the previous case a). We see that now the absolute import succeeds and the relative import fails of config.py

In this case, the relative import of config.py: from . import config failed. But the absolute import: import config succeeded.

Let’s explain these errors.

2. Explaining the import errors

First of all, notice that when you run a script directly, Python changes the name of the script to __main__.

You probably have seen this before:


# Check if the script is being run as the main program
if __name__ == "__main__":
    # This block will only be executed if the script is run directly,
    # not when it is imported as a module in another script.
    print("This script is being run directly.")

So if we include this in app.py and run it. We will get the print output `This script is being run directly.

If the script is imported as a module, the code under the if __name__ == “__main__” block will be skipped.

So that is why when we run app.py specifically print(f”running {__name__}”) we get: running __main__ and running inner_folder.helper_functions. __main__ is the user-specified Python module that first starts running. Python calls it “top-level” (or entry-point) because it is at the top and then after it, we have all the imports the program specifies [2].

But why do some imports fail to load? For example in case a) where import config fails and the relative import succeeds?

To answer this first we have to answer:

Key question: How and where does Python import these modules?

Python has two functions that run whenever we make an import: a 1) Finder and a 2) Loader

Note: If you have some time check this amazing video explaning about 
importing, finders and, loaders: 
https://www.youtube.com/watch?v=QCSz0j8tGmI&t=0s

A Finder is an object (python class) that identifies the correct loader for the module we request. Why do we need different loaders? There are some packages or modules that have an extension .py and others that are written in other languages like C++. Thus, there should be a different loading procedure for these two packages. We are not going to get into more details about the different loaders for this article.
A Loader is also an object whose job is to locate and load the contents of a module so that Python can use them in your program.

Python can only import correctly the modules and libraries that it can find.

So, where does Python find all the installed modules?

To answer this question, we are going to use the library. Sys is a built-in package that provides access to variables and functions in the Python environment. Specifically, we are going to use the command sys.path to access a list of directory names where Python looks for modules when the import statement is used. Let’s use it :

# importing sys module
import sys
 
# importing sys.path
print(sys.path)

Output:

['c:\\Users\\My-user\\My_computer\\CS_Course\\Foundations-of-Software-Engineering-and-Computer-Science-for-Data-Scientists\\4_python_we_did_not_learn_part_1\\paths_modules_variables', 
'C:\\Users\\My-user\\AppData\\Local\\Programs\\Python\\Python310\\python310.zip',
 'C:\\Users\\My-user\\AppData\\Local\\Programs\\Python\\Python310\\DLLs', 
'C:\\Users\\My-user\\AppData\\Local\\Programs\\Python\\Python310\\lib', 
.......,
 'C:\\Users\\My-user\\My_computer\\CS_Course\\CS_SE_course_env\\lib\\site-packages']

The output is big, but as you can see, it contains a list of directories that Python will search for the required modules in your environment (I skipped some paths in my environment). Notice that most of them are in /Python directory and check the first one: C:\\Users\\My-user\\My_computer\\CS_Course\\...\\paths_modules_variables

This is the current path that I am running the script app.py.

Also, check the last one: C:\\Users\\My-user\\My_computer\\CS_Course\\...\\lib\\site-packages

This is where the site-packages exist and are imported.

Let’s see the path of pandas library that we installed:

import pandas
print(pandas.__file__)

output: C:\Users\My-user\My_computer\...\lib\site-packages\pandas\__init__.py

We see that it exists in the site-packages folder with the name of the module and specifically targets the __init__.py script. Let’s dive more into this list and see how a finder uses it.

3. Steps for Finding and Loading a Module

When a module is imported, the interpreter uses Finder to find the package/module. Let’s see the steps that it takes [4]. First, the Finder tries to find:

1) a built-in module with that name. These module names are listed in sys.builtin_module_names.

If not found, it then searches for a file named config.py in a list of directories given by the variable sys.path that we just saw. sys.path is initialized from these locations:

2) The directory containing the input script (or the current directory when no file is specified).

3) PYTHONPATH (a list of directory names, with the same syntax as the shell variable PATH).

4) The installation-dependent default (by convention including a site-packages directory, handled by the site module).

Let’s understand this with the following examples:

Can you guess in which step(1–4) finds the library or module?

math

When you import the math library, the Python interpreter first looks for a built-in module, and since math is a standard library, it falls under (step 1). So, the interpreter finds it among the built-in modules.

pandas

This is the case that we saw at the end of section 3). The pandas library is an installed external library. So, the interpreter searches for it in the directories specified in sys.path. Remember in our case it was:C:\Users\My-user\My_computer\…\lib\site-packages. This includes locations like the installation-dependent default, possibly including a site-packages directory, which is handled by the site module (step 4).

config.py

If you are importing a local file like config.py, the interpreter first checks the directory containing the input script or the current directory when no file is specified (step 2). If the interpreter can’t find config.py there, it looks in the directories specified in sys.path.

Let’s provide more details on the config.py and case a) (Figure 2):

When we run app.py with the from inner_folder import helper_functions, Python searches for config.py as a top-level module (or in the directory where we run the scripts), not as a sub-module of the areas folder. Since config.py is not in the same directory as __main__ (the app.py), Python can’t find it.

4. Solving Module Import Errors

Solution 1: Relative import

We provide this solution in the code and we show it in Figure 2. We basically make a relative import from helper_functions.py: from . import config the . is indicating the current directory of helper_functions.py. Python knows where helper_functions.py is so, it can make relative imports to that (indirectly).

Pros: It’s simple. It relies in Python to handle path imports.

Cons: Not a good idea to use it with multiple module imports. Not fit for debugging purposes.

However, python (or better as the PEP8 style guide) suggests relative imports as an alternative [3] and prefers in most cases to convert them to absolute.

Solution 2: Absolute import with a parent directory

To convert the previous solution to an absolute import, we can indicate the parent directory:

# instead of from . import config we specify the parent directory:
from inner_folder import config 

def add_config_constant(a: int)-> int:
    return a + config.config_constant

Solution 2 repo link

Figure 4) Using an absolute import that refers to the parent directory, helper_function.py can import successfully config when we run app.py

In this running app.py will be successful (case a) but running helping_function.py directly will fail (case b).

Pros: Recommended by PEP8. It shows the clear path.
Cons: Not independent, requires adjustment when the files or the folder changes.

Solution 3: Add the script to your sys.path (not recommended)

The following code can add any module to Python’s path so you can import it and then use it in your code using sys.path.append(absolute_path_to_module). So in our example, we can edit app.py to be something like this:

#app.py
print(f"running {__name__}")

import sys
import os

# Get the absolute path of the current script's directory
current_script_directory = os.path.dirname(os.path.abspath(__file__))

# Add the parent directory (helper_functions) to sys.path
sys.path.append(os.path.join(current_script_directory, 'inner_folder'))

# Now you can import modules from the added path
from inner_folder import helper_functions

result_add = helper_functions.add_config_constant(5)

print("Add config result:", result_add)

Solution 3 repo link

You will see often this solution on the internet, however, as the Python community mentions:

The Pythonic solution for the import problem is to not add sys.path (or indirectly PYTHONPATH) hacks to any file that could potentially serve as a top-level script (incl. unit tests), since this is what makes your code base difficult to change and maintain [5].

Eventually, you will find yourself adding a lot of files to that “list”. Also, if you change some file’s position this code will have to change. Therefore, it is not considered a scalable solution.

Pros: Quick and easy

Cons: Not scalable solution and easy to lose track. Also, difficult to maintain.

Solution 4: Create a package

By creating a package, you basically make an independent, self-managing code that you can install. Python will add it to the site-packages path, along with other installed libraries. So, your module will be like pandas (if you upload it to the internet). In those cases, you can use the functions or the objects from the package by simple imports.

in this example, it will be an overkill to do it, since /inner_function contains so small scripts. However, when the complexity increases, it is a good scalable strategy.

A package can be developed, tested, and debugged independently making it more scalable and maintainable for future usage. For more information, you can refer to Python’s packaging guide [6].

Pros: Scalable, maintainable, and independent.

Cons: Requires time to maintain. Overkill for simple modules or small-scale projects.

5. Implicit Imports

We left case b) a bit behind in this article. However, this actually has the simplest explanation. So, in case b), when we ran helping_functions.py (as __main__), the absolute import was successful. The relative import did not work __main__ directly why?

The answer here is simple: Python 3 has disabled implicit relative imports altogether to avoid confusion in large and complex codes [7]. In Python 2, implicit relative imports were allowed. However, in Python 3 they disabled them because in larger projects the .notation causes confusion because it does not refer specifically to the module that is imported. Imagine you have ten of those. During debugging or on a late revisit of your code after ten days, how easy would it be to make corrections?

6. Summary

In this article, we explored modular coding in Python, addressing common import errors. We understood cases where relative and absolute imports fail in a common code structure. Pythonistas often mention those issues in the community. So, we offered a theoretical explanation of imports and we provided different solutions to these import issues. We focused on understanding sys.path for locating modules and highlighting the importance of scalable solutions, such as creating packages and absolute imports.