Lock inspect

Multiple pip-compile runs are not coordinated. Without coordination, the output files most likely will contain different package versions for the same package. This issue will occur many times.

Due to the plethora of packages, manually searching for these discrepancies, across .lock files, is error prone; even with grep.

Creates a demand for a output files post processor. To find and report these discrepancies and suggest how to resolve them.

drain-swamp has .in, .unlock, and .lock files. Since the .in are cascading, .unlock files are flattened. So the constraints are easy to find. Rather than digging thru a bunch of .in files.

drain_swamp.lock_inspect.DC_SLOTS: dict[str, bool]

Allows dataclasses.dataclass __slots__ support from py310

drain_swamp.lock_inspect.is_module_debug: bool = False

Flag to turn on module level logging. Should be off in production

drain_swamp.lock_inspect._logger: logging.Logger

Module level logger

class drain_swamp.lock_inspect._T
drain_swamp.lock_inspect._T: TypeVar

Class Pins is a Generic container. Allowing to change which items are allowed into the container

class drain_swamp.lock_inspect.PinsByPkg
drain_swamp.lock_inspect.PinsByPkg: dict[str, set[drain_swamp.lock_inspect.Pin]]

dict of package name by set of Pins or locks. .unlock contains pin. .lock contain locks. Both are stored as Pin

class drain_swamp.lock_inspect.PkgsWithIssues
drain_swamp.lock_inspect.PkgsWithIssues: dict[str, dict[str, packaging.version.Version | set[packaging.version.Version]]]

Packages by a dict containing highest version and other versions

class drain_swamp.lock_inspect.Pin(file_abspath: Path, pkg_name: str)

A pin has a specifier e.g. ‘pip>=24.2’ and may have one or more qualifiers

Package dependencies w/o specifiers are not pins.

Variables:

  • file_abspath (pathlib.Path) – absolute path to requirements file

  • pkg_name (str) – pkg name by itself

  • line (str) –

    Unaltered line. Spacing not normalized. Normalized with double quotes? Will contain the rest. e.g.

    ; python_version < "3.11" ; sys_platform == "win32" `` ;platform_system==”Windows”``

  • specifiers (list[str]) – The package version constraints. Is a pin if a non-empty list.

Raises:

  • KeyError – in requirements file no such package

file_abspath: Path
static is_pin(specifiers)

From a .unlock or .in file, identify a line as containing a pin.

Parameters:

line (list[str]) – raw line from a .unlock or .in file

Returns:

True if a pin otherwise False

Return type:

bool

line: str
pkg_name: str
property qualifiers

From the Pin line, retrieve a clean qualifiers list

Strip whitespace and without semi colon

Returns:

qualifiers

Return type:

list[str]

specifiers: list[str]
class drain_swamp.lock_inspect.Pins(pins)

Pin container.

Variables:

pins (Any) – Expecting either a Sequence[_T] or Set[_T]

_pins: set[drain_swamp.lock_inspect._T]

Container of Pin

_iter: collections.abc.Iterator[drain_swamp.lock_inspect._T]

Holds the reusable iterator

add(item)

Add item to set

Parameters:

item (Any) – Expecting a Pin. Add to set

classmethod by_pkg(loader, venv_path, suffix='.lock', filter_by_pin=True)

Group Pins by pkg_name.

Parameters:

  • loader (drain_swamp.pep518_venvs.VenvMapLoader) – From pyproject.toml, loads venvs, but does not parse the data

  • venv_path (str) – Path to a virtual env. There should be a cooresponding entry in pyproject.toml tool.venvs array of tables.

  • suffix (str | None) – Default .lock. Either .lock or .unlock. Determines which files are read. Looks at last suffix so .shared.[whatever] is supported

  • filter_by_pin (bool | None) – Default True. Filter out entries without specifiers

Returns:

dict which has Pins grouped by package name

Return type:

drain_swamp.lock_inspect.PinsByPkg

classmethod by_pkg_with_issues(loader, venv_path)

Filter out packages without issues. Each pin indicate highest Version and set of other Versions. Which to choose would need to take into account pins located within .unlock files.

Parameters:

  • loader (drain_swamp.pep518_venvs.VenvMapLoader) – From pyproject.toml, loads venvs, but does not parse the data. Expecting loader of the .lock files.

  • venv_path (str) – Path to a virtual env. There should be a cooresponding entry in pyproject.toml tool.venvs array of tables.

Returns:

packages by Pins and packages by dict of highest version and other versions

Return type:

tuple[drain_swamp.lock_inspect.PinsByPkg, drain_swamp.lock_inspect.PkgsWithIssues]

discard(item)

Remove item from set if present

Parameters:

item (Any) – Expecting a Pin. Remove from set

static filter_pins_of_pkg(pins_current: Pins[drain_swamp.lock_inspect._T], pkg_name: str) → Pins[drain_swamp.lock_inspect._T]

Filter unlock Pins by package name.

Parameters:
Returns:

Pins of one package

Return type:

drain_swamp.lock_inspect.Pins[drain_swamp.lock_inspect._T]

static from_loader(loader, venv_path, suffix='.unlock', filter_by_pin=True)

Factory. From a venv, get all Pins.

Parameters:

  • loader (drain_swamp.pep518_venvs.VenvMapLoader) – Contains some paths and loaded not parsed venv reqs

  • venv_path (Any) – Relative path to venv base folder. Acts as a key

  • suffix (str) – Default .unlock. End suffix of compiled requirements file. Either .unlock or .lock

  • filter_by_pin (bool | None) – Default True. Filter out entries without specifiers

Returns:

Feed list[Pin] into class constructor to get an Iterator[Pin]

Return type:

set[drain_swamp.lock_inspect._T]

Raises:
static has_discrepancies(d_by_pkg)

Across .lock files, packages with discrepancies.

Comparison limited to equality

Parameters:

d_by_pkg (drain_swamp.lock_inspect.PinsByPkg) – Within one venv, all lock packages’ set[Pin]

Returns:

pkg name / highest version. Only packages with discrepancies. With the highest version, know which version to nudge to.

Return type:

drain_swamp.lock_inspect.PkgsWithIssues

classmethod qualifiers_by_pkg(loader, venv_path)

Get qualifiers by package. First non-empty qualifiers found wins.

This algo, will fail to discover and fix qualifier disparities. Only gets fixed if there are version disparities

Parameters:

  • loader (drain_swamp.pep518_venvs.VenvMapLoader) – From pyproject.toml, loads venvs, but does not parse the data. Expecting loader of the .lock files.

  • venv_path (str) – Path to a virtual env. There should be a cooresponding entry in pyproject.toml tool.venvs array of tables.

Returns:

dict of package name by qualifiers

Return type:

dict[str, str]

static subset_req(venv_reqs, pins, req_relpath)

Factory. For a venv Pins, create a subset limited to one requirement file.

Parameters:
Returns:

Feed set[T] into class constructor to get an Iterator[drain_swamp.lock_inspect._T]

Return type:

set[drain_swamp.lock_inspect._T]

class drain_swamp.lock_inspect.Resolvable(venv_path: str | Path, pkg_name: str, qualifiers: str, nudge_unlock: str, nudge_lock: str)

Resolvable dependency conflict. Can find the lines for the pkg, in .unlock and .lock files, using (loader and) venv_path and pkg_name.

Limitation: Qualifiers e.g. python_version and os_name

  • haphazard usage

All pkg lines need the same qualifier. Often missing. Make uniform. Like a pair of earings.

  • rigorous usage

There can be one or more qualifiers. In which case, nonobvious which qualifier to use where.

Variables:

  • venv_path (str | pathlib.Path) – Relative or absolute path to venv base folder

  • pkg_name (str) – package name

  • qualifiers (str) – qualifiers joined together into one str. Whitespace before the 1st semicolon not preserved.

  • nudge_unlock (str) – For .unlock files. Nudge pin e.g. pkg_name>=some_version. If pkg_name entry in an .unlock file, replace otherwise add entry

  • nudge_lock (str) – For .lock files. Nudge pin e.g. pkg_name==some_version. If pkg_name entry in a .lock file, replace otherwise add entry

nudge_lock: str
nudge_unlock: str
pkg_name: str
qualifiers: str
venv_path: str | Path
class drain_swamp.lock_inspect.ResolvedMsg(venv_path: str, abspath_f: Path, nudge_pin_line: str)

Fixed dependency version discrepancies (aka issues)

Does not include the original line

Variables:

  • venv_path (str) – venv relative or absolute path

  • abspath_f (pathlib.Path) – Absolute path to requirements file

  • nudge_pin_line (str) – What the line will become

abspath_f: Path
nudge_pin_line: str
venv_path: str
class drain_swamp.lock_inspect.UnResolvable(venv_path: str, pkg_name: str, qualifiers: str, sss: set[SpecifierSet], v_highest: Version, v_others: set[Version], pins: Pins[Pin])

Cannot resolve this dependency conflict.

Go out of our way to clearly and cleanly present sufficient details on the issue.

The most prominent details being the package name and Pins (from relevent .unlock files).

Track down issue

With issue explanation. Look at the .lock to see the affected package’s parent(s). The parents’ package pins may be the cause of the conflict.

The parents’ package pyproject.toml file is the first place to look for strange dependency restrictions. Why a restriction was imposed upon a dependency may not be well documented. Look in the repo issues. Search for the dependency package name

Upgrading

lock inspect is not a dependency upgrader. Priority is to sync .unlock and .lock files.

Recommend always doing a dry run pip compile --dry-run some-requirement.in or looking at upgradable packages within the venv. pip list -o

Variables:

  • venv_path (str | pathlib.Path) – Relative or absolute path to venv base folder

  • pkg_name (str) – package name

  • qualifiers (str) – qualifiers joined together into one str. Whitespace before the 1st semicolon not preserved.

  • sss (set[packaging.specifiers.SpecifierSet]) – Set of SpecifierSet, for this package, are the dependency version restrictions found in .unlock files

  • v_highest (packaging.version.Version) – Hints at the process taken to find a Version which satisfies SpecifierSets. First this highest version was checked

  • v_others (set[packaging.version.Version]) – After highest version, all other potential versions are checked. The potential versions come from the .lock files. So if a version doesn’t exist in one .lock, it’s never tried.

  • pins (drain_swamp.lock_inspect.Pins[drain_swamp.lock_inspect.Pin]) –

    Has the absolute path to each requirements file and the dependency version restriction.

    Make this readable

pins: Pins[Pin]
pkg_name: str
pprint_pins()

Capture pprint and return it.

Returns:

pretty printed representation of the pins

Return type:

str

qualifiers: str
sss: set[SpecifierSet]
v_highest: Version
v_others: set[Version]
venv_path: str
drain_swamp.lock_inspect.filter_by_venv_relpath(loader, venv_current_relpath)

Facilitate call more than once

Could do all in one shot by supplying venv_path=None

Parameters:
Returns:

Container of InFile

Return type:

tuple[tuple[pathlib.Path], drain_swamp.lock_infile.InFiles]

Raises:
drain_swamp.lock_inspect.fix_requirements(loader, venv_relpath, is_dry_run=False)

Iterate thru venv. Treat .unlock / .lock as a pair. Fix requirements files pair(s).

Parameters:

  • loader (drain_swamp.pep518_venvs.VenvMapLoader) – Contains some paths and loaded unparsed mappings

  • venv_relpath (str) – venv relative path is a key. To choose a tools.venvs.req

  • is_dry_run (Any | None) – Default False. Should be a bool. Do not make changes. Merely report what would have been changed

Returns:

list contains tuples. venv path, resolves messages, unresolvable issues, resolvable3 issues dealing with .shared requirements file

Return type:

tuple[dict[str, ResolvedMsg], dict[str, UnResolvable], dict[str, tuple[str, drain_swamp.lock_inspect.Resolvable, drain_swamp.lock_inspect.Pin]]]

Raises:
drain_swamp.lock_inspect.fix_resolvables(resolvables: Sequence[Resolvable], loader, venv_path, is_dry_run=False) → tuple[list[str], list[tuple[str, Resolvable, Pin]]]

Go thru resolvables and fix affected .unlock and .lock files

Assumes target requirements file exists and is a file. This is a post processor. After .in, .unlock, and .lock files have been created.

Parameters:
Returns:

Wrote messages. For shared, tuple of suffix, resolvable, and Pin (of .lock file). This is why the suffix is provided and first within the tuple

Return type:

tuple[list[drain_swamp.lock_inspect.ResolvedMsg], list[tuple[str, str, drain_swamp.lock_inspect.Resolvable, drain_swamp.lock_inspect.Pin]]]

Raises:
drain_swamp.lock_inspect.get_issues(loader, venv_path)

Look thru all the packages with discrepanies. Which have existing nudge(s). Find where those nudges are in the .in files.

Replace or add as appropriate

Parameters:
Returns:

Resolvable and unresolvable issues lists

Return type:

tuple[list[drain_swamp.lock_inspect.Resolvable], list[drain_swamp.lock_inspect.UnResolvable]]

drain_swamp.lock_inspect.get_reqs(loader, venv_path=None, suffix_last='.in')

get absolute path to requirement files

Filtering by venv relative or absolute path is recommended

Parameters:
Returns:

Sequence of absolute path to requirements files

Return type:

tuple[pathlib.Path]

Raises:
drain_swamp.lock_inspect.is_timeout(failures)

lock_compile returns both success and failures. Detect if the cause of the failure was timeout(s)

Parameters:

failures (tuple[str, ...]) – Sequence of verbose error message and traceback

Returns:

True if web (actually SSL) connection timeout occurred

Return type:

bool

drain_swamp.lock_inspect.lock_compile(loader, venv_relpath, timeout=15)

In a subprocess, call pip-compile to create .lock files

Parameters:
Returns:

Generator of abs path to .lock files

Return type:

tuple[tuple[str, …], tuple[str, …]]

Raises:
drain_swamp.lock_inspect.prepare_pairs(t_ins)
drain_swamp.lock_inspect.prepare_pairs(t_ins: tuple[Path])
drain_swamp.lock_inspect.prepare_pairs(in_files: InFiles, path_cwd=None)

Fallback for unsupported data types without implementations

Do not add type annotations. It’s correct as-is.

Raises:
drain_swamp.lock_inspect.unlock_compile(loader, venv_relpath)

.in requirement files can contain -r and -c lines. Relative path to requirement files and constraint files respectively.

Originally thought -c was a pip-compile convention, not a pip convention. Opps!

Resolve the -r and -c to create .unlock file

package dependencies

  • For a package which is an app, lock them.

  • For a normal package, always must be unlocked.

optional dependencies

  • additional feature –> leave unlocked

  • For develop environment –> lock them

Parameters:
Returns:

Generator of abs path to .unlock files

Return type:

collections.abc.Generator[pathlib.Path, None, None]

drain_swamp.lock_inspect.write_to_file_nudge_pin(path_f: Path, pkg_name: str, nudge_pin_line: str) → None

Nudge pin must include a newline (os.linesep) If package line exists in file, overwrite. Otherwise append nudge pin line

Parameters:

  • path_f (pathlib.Path) – Absolute path to either a .unlock or .lock file. Only a comment if no preceding whitespace

  • pkg_name (str) – Package name. Should be lowercase

  • nudge_pin_line (str) – Format [package name][operator][version][qualifiers][os.linesep]